2024-11-28

Title: Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approach

• Authors: Shijian Deng, Wentian Zhao, Yu-Jhe Li, Kun Wan, Daniel Miranda, Ajinkya Kale, Yapeng Tian
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.17760
• Pdf URL: https://arxiv.org/pdf/2411.17760
• Copy Paste: [[2411.17760]] Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approach(https://arxiv.org/abs/2411.17760)
• Keywords: language model, llm, hallucination
• Abstract: Self-improvement in multimodal large language models (MLLMs) is crucial for enhancing their reliability and robustness. However, current methods often rely heavily on MLLMs themselves as judges, leading to high computational costs and potential pitfalls like reward hacking and model collapse. This paper introduces a novel, model-level judge-free self-improvement framework. Our approach employs a controlled feedback mechanism while eliminating the need for MLLMs in the verification loop. We generate preference learning pairs using a controllable hallucination mechanism and optimize data quality by leveraging lightweight, contrastive language-image encoders to evaluate and reverse pairs when necessary. Evaluations across public benchmarks and our newly introduced IC dataset designed to challenge hallucination control demonstrate that our model outperforms conventional techniques. We achieve superior precision and recall with significantly lower computational demands. This method offers an efficient pathway to scalable self-improvement in MLLMs, balancing performance gains with reduced resource requirements.
• 摘要：多模态大型语言模型 (MLLM) 中的自我改进对于增强其可靠性和稳健性至关重要。然而，当前的方法通常严重依赖 MLLM 本身作为评判者，导致计算成本高昂，并可能存在奖励黑客和模型崩溃等陷阱。本文介绍了一种新颖的模型级无评判者自我改进框架。我们的方法采用受控反馈机制，同时消除了验证循环中对 MLLM 的需求。我们使用可控幻觉机制生成偏好学习对，并通过利用轻量级对比语言图像编码器在必要时评估和反转对来优化数据质量。在公共基准和我们新推出的旨在挑战幻觉控制的 IC 数据集上的评估表明，我们的模型优于传统技术。我们实现了卓越的精度和召回率，同时显著降低了计算需求。该方法为 MLLM 中的可扩展自我改进提供了一条有效途径，在性能提升与资源需求减少之间取得平衡。

Title: $H^3$Fusion: Helpful, Harmless, Honest Fusion of Aligned LLMs

• Authors: Selim Furkan Tekin, Fatih Ilhan, Tiansheng Huang, Sihao Hu, Zachary Yahn, Ling Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.17792
• Pdf URL: https://arxiv.org/pdf/2411.17792
• Copy Paste: [[2411.17792]] $H^3$Fusion: Helpful, Harmless, Honest Fusion of Aligned LLMs(https://arxiv.org/abs/2411.17792)
• Keywords: llm
• Abstract: Alignment of pretrained LLMs using instruction-based datasets is critical for creating fine-tuned models that reflect human preference. A growing number of alignment-based fine-tuning algorithms and benchmarks emerged recently, fueling the efforts on effective alignments of pre-trained LLMs to ensure helpful, harmless, and honest answers from both open-source and closed-source LLMs. This paper tackles this problem by developing an alignment fusion approach, coined as $H^3$Fusion, with three unique characteristics. First, $H^3$Fusion ensembles multiple individually aligned LLMs to create a final fine-tuned alignment model with enhanced capabilities beyond those of individual models, delivering robust alignment through promoting helpful, harmless, honest fusion. Second, $H^3$Fusion leverages the mixture-of-experts (MoE) methodology in two steps. We first freeze the multi-head attention weights of each individual model while tuning the FFN layer during alignment fusion. Then we merge the aligned model weights with an expert router according to the type of input instruction and dynamically select a subset of experts that are best suited for producing the output response. Finally, we boost the performance of the resulting $H^3$3Fusion model by introducing gating loss and regularization terms. The former penalizes the selection errors of the expert-router, and the latter mediates the expert weights drifting during fine-tuning and dynamically adjusts the fusion behavior of the resulting model by canalizing the activations on the experts. Extensive evaluations on three benchmark datasets show that $H^3$3Fusion is more helpful, less harmful, and more honest from two aspects: it outperforms each individually aligned model by $11.37\%$, and it provides stronger robustness compared to the state-of-the-art LLM ensemble approaches by $13.77\%$. Code is available at this http URL.
• 摘要：使用基于指令的数据集对预训练的 LLM 进行对齐对于创建反映人类偏好的微调模型至关重要。最近出现了越来越多的基于对齐的微调算法和基准，推动了对预训练的 LLM 进行有效对齐的努力，以确保来自开源和闭源 LLM 的答案有用、无害且诚实。本文通过开发一种对齐融合方法（称为 $H^3$Fusion）来解决这个问题，该方法具有三个独特的特征。首先，$H^3$Fusion 集成多个单独对齐的 LLM 以创建最终的微调对齐模型，该模型的功能超越单个模型，通过促进有用、无害、诚实的融合来实现稳健的对齐。其次，$H^3$Fusion 分两步利用混合专家 (MoE) 方法。我们首先冻结每个单独模型的多头注意力权重，同时在对齐融合期间调整 FFN 层。然后，我们根据输入指令的类型将对齐的模型权重与专家路由器合并，并动态选择最适合产生输出响应的专家子集。最后，我们通过引入门控损失和正则化项来提高生成的 $H^3$3Fusion 模型的性能。前者惩罚专家路由器的选择错误，后者调解专家权重在微调过程中的漂移，并通过引导专家的激活来动态调整生成的模型的融合行为。对三个基准数据集的广泛评估表明，$H^3$3Fusion 从两个方面更有帮助、危害更小、更诚实：它比每个单独对齐的模型高出 $11.37\%$，并且与最先进的 LLM 集成方法相比，它提供了更强的鲁棒性 $13.77\%$。代码可在此 http URL 上获得。

Title: LongKey: Keyphrase Extraction for Long Documents

• Authors: Jeovane Honorio Alves, Radu State, Cinthia Obladen de Almendra Freitas, Jean Paul Barddal
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.17863
• Pdf URL: https://arxiv.org/pdf/2411.17863
• Copy Paste: [[2411.17863]] LongKey: Keyphrase Extraction for Long Documents(https://arxiv.org/abs/2411.17863)
• Keywords: language model
• Abstract: In an era of information overload, manually annotating the vast and growing corpus of documents and scholarly papers is increasingly impractical. Automated keyphrase extraction addresses this challenge by identifying representative terms within texts. However, most existing methods focus on short documents (up to 512 tokens), leaving a gap in processing long-context documents. In this paper, we introduce LongKey, a novel framework for extracting keyphrases from lengthy documents, which uses an encoder-based language model to capture extended text intricacies. LongKey uses a max-pooling embedder to enhance keyphrase candidate representation. Validated on the comprehensive LDKP datasets and six diverse, unseen datasets, LongKey consistently outperforms existing unsupervised and language model-based keyphrase extraction methods. Our findings demonstrate LongKey's versatility and superior performance, marking an advancement in keyphrase extraction for varied text lengths and domains.
• 摘要：在信息过载的时代，手动注释大量且不断增长的文档和学术论文语料库越来越不切实际。自动关键短语提取通过识别文本中的代表性术语解决了这一挑战。然而，大多数现有方法都侧重于短文档（最多 512 个标记），在处理长上下文文档方面存在差距。在本文中，我们介绍了 LongKey，这是一种从长文档中提取关键短语的新框架，它使用基于编码器的语言模型来捕获扩展的文本复杂性。LongKey 使用最大池化嵌入器来增强关键短语候选表示。在综合 LDKP 数据集和六个不同的、看不见的数据集上进行验证，LongKey 始终优于现有的无监督和基于语言模型的关键短语提取方法。我们的研究结果证明了 LongKey 的多功能性和卓越性能，标志着不同文本长度和领域的关键短语提取的进步。

Title: Leveraging Large Language Models and Topic Modeling for Toxicity Classification

• Authors: Haniyeh Ehsani Oskouie, Christina Chance, Claire Huang, Margaret Capetz, Elizabeth Eyeson, Majid Sarrafzadeh
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.17876
• Pdf URL: https://arxiv.org/pdf/2411.17876
• Copy Paste: [[2411.17876]] Leveraging Large Language Models and Topic Modeling for Toxicity Classification(https://arxiv.org/abs/2411.17876)
• Keywords: language model, gpt
• Abstract: Content moderation and toxicity classification represent critical tasks with significant social implications. However, studies have shown that major classification models exhibit tendencies to magnify or reduce biases and potentially overlook or disadvantage certain marginalized groups within their classification processes. Researchers suggest that the positionality of annotators influences the gold standard labels in which the models learned from propagate annotators' bias. To further investigate the impact of annotator positionality, we delve into fine-tuning BERTweet and HateBERT on the dataset while using topic-modeling strategies for content moderation. The results indicate that fine-tuning the models on specific topics results in a notable improvement in the F1 score of the models when compared to the predictions generated by other prominent classification models such as GPT-4, PerspectiveAPI, and RewireAPI. These findings further reveal that the state-of-the-art large language models exhibit significant limitations in accurately detecting and interpreting text toxicity contrasted with earlier methodologies. Code is available at this https URL.
• 摘要：内容审核和毒性分类是具有重大社会影响的关键任务。然而，研究表明，主要的分类模型表现出放大或减少偏见的趋势，并可能在分类过程中忽视或不利于某些边缘群体。研究人员认为，注释者的位置性会影响模型从中学习的黄金标准标签，从而传播注释者的偏见。为了进一步研究注释者位置性的影响，我们深入研究了在使用主题建模策略进行内容审核的同时对数据集上的 BERTweet 和 HateBERT 进行微调。结果表明，与其他著名分类模型（如 GPT-4、PerspectiveAPI 和 RewireAPI）生成的预测相比，对特定主题的模型进行微调可显着提高模型的 F1 得分。这些发现进一步表明，与早期方法相比，最先进的大型语言模型在准确检测和解释文本毒性方面表现出明显的局限性。代码可在此 https URL 上找到。

Title: HOPPR Medical-Grade Platform for Medical Imaging AI

• Authors: Kalina P. Slavkova, Melanie Traughber, Oliver Chen, Robert Bakos, Shayna Goldstein, Dan Harms, Bradley J. Erickson, Khan M. Siddiqui
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.17891
• Pdf URL: https://arxiv.org/pdf/2411.17891
• Copy Paste: [[2411.17891]] HOPPR Medical-Grade Platform for Medical Imaging AI(https://arxiv.org/abs/2411.17891)
• Keywords: language model
• Abstract: Technological advances in artificial intelligence (AI) have enabled the development of large vision language models (LVLMs) that are trained on millions of paired image and text samples. Subsequent research efforts have demonstrated great potential of LVLMs to achieve high performance in medical imaging use cases (e.g., radiology report generation), but there remain barriers that hinder the ability to deploy these solutions broadly. These include the cost of extensive computational requirements for developing large scale models, expertise in the development of sophisticated AI models, and the difficulty in accessing substantially large, high-quality datasets that adequately represent the population in which the LVLM solution is to be deployed. The HOPPR Medical-Grade Platform addresses these barriers by providing powerful computational infrastructure, a suite of foundation models on top of which developers can fine-tune for their specific use cases, and a robust quality management system that sets a standard for evaluating fine-tuned models for deployment in clinical settings. The HOPPR Platform has access to millions of imaging studies and text reports sourced from hundreds of imaging centers from diverse populations to pretrain foundation models and enable use case-specific cohorts for fine-tuning. All data are deidentified and securely stored for HIPAA compliance. Additionally, developers can securely host models on the HOPPR platform and access them via an API to make inferences using these models within established clinical workflows. With the Medical-Grade Platform, HOPPR's mission is to expedite the deployment of LVLM solutions for medical imaging and ultimately optimize radiologist's workflows and meet the growing demands of the field.
• 摘要：人工智能 (AI) 的技术进步推动了大型视觉语言模型 (LVLM) 的开发，这些模型在数百万对图像和文本样本上进行训练。后续的研究表明，LVLM 在医学成像用例（例如，放射学报告生成）中具有实现高性能的巨大潜力，但仍然存在阻碍广泛部署这些解决方案的障碍。这些障碍包括开发大型模型所需的大量计算成本、开发复杂 AI 模型的专业知识，以及难以访问能够充分代表要部署 LVLM 解决方案的人群的大量高质量数据集。HOPPR 医疗级平台通过提供强大的计算基础设施、一套基础模型（开发人员可以在其基础上针对特定用例进行微调）以及一个强大的质量管理系统（为评估在临床环境中部署的微调模型设定标准）来解决这些障碍。 HOPPR 平台可以访问来自数百个不同人群的成像中心的数百万个成像研究和文本报告，以预先训练基础模型并启用特定于用例的群组进行微调。所有数据都经过去识别处理并安全存储，以符合 HIPAA 要求。此外，开发人员可以安全地在 HOPPR 平台上托管模型并通过 API 访问它们，以便在既定的临床工作流程中使用这些模型进行推理。借助医疗级平台，HOPPR 的使命是加快部署用于医学成像的 LVLM 解决方案，并最终优化放射科医生的工作流程并满足该领域日益增长的需求。

Title: QuaLLM-Health: An Adaptation of an LLM-Based Framework for Quantitative Data Extraction from Online Health Discussions

• Authors: Ramez Kouzy, Roxanna Attar-Olyaee, Michael K. Rooney, Comron J. Hassanzadeh, Junyi Jessy Li, Osama Mohamad
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.17967
• Pdf URL: https://arxiv.org/pdf/2411.17967
• Copy Paste: [[2411.17967]] QuaLLM-Health: An Adaptation of an LLM-Based Framework for Quantitative Data Extraction from Online Health Discussions(https://arxiv.org/abs/2411.17967)
• Keywords: language model, gpt, llm, prompt
• Abstract: Health-related discussions on social media like Reddit offer valuable insights, but extracting quantitative data from unstructured text is challenging. In this work, we present an adapted framework from QuaLLM into QuaLLM-Health for extracting clinically relevant quantitative data from Reddit discussions about glucagon-like peptide-1 (GLP-1) receptor agonists using large language models (LLMs). We collected 410k posts and comments from five GLP-1-related communities using the Reddit API in July 2024. After filtering for cancer-related discussions, 2,059 unique entries remained. We developed annotation guidelines to manually extract variables such as cancer survivorship, family cancer history, cancer types mentioned, risk perceptions, and discussions with physicians. Two domain-experts independently annotated a random sample of 100 entries to create a gold-standard dataset. We then employed iterative prompt engineering with OpenAI's "GPT-4o-mini" on the gold-standard dataset to build an optimized pipeline that allowed us to extract variables from the large dataset. The optimized LLM achieved accuracies above 0.85 for all variables, with precision, recall and F1 score macro averaged > 0.90, indicating balanced performance. Stability testing showed a 95% match rate across runs, confirming consistency. Applying the framework to the full dataset enabled efficient extraction of variables necessary for downstream analysis, costing under $3 and completing in approximately one hour. QuaLLM-Health demonstrates that LLMs can effectively and efficiently extract clinically relevant quantitative data from unstructured social media content. Incorporating human expertise and iterative prompt refinement ensures accuracy and reliability. This methodology can be adapted for large-scale analysis of patient-generated data across various health domains, facilitating valuable insights for healthcare research.
• 摘要：Reddit 等社交媒体上的健康相关讨论提供了宝贵的见解，但从非结构化文本中提取定量数据具有挑战性。在这项工作中，我们提出了一个从 QuaLLM 到 QuaLLM-Health 的改编框架，用于使用大型语言模型 (LLM) 从 Reddit 关于胰高血糖素样肽 1 (GLP-1) 受体激动剂的讨论中提取临床相关的定量数据。我们在 2024 年 7 月使用 Reddit API 从五个 GLP-1 相关社区收集了 410k 个帖子和评论。在过滤与癌症相关的讨论后，剩下 2,059 个唯一条目。我们制定了注释指南，以手动提取变量，例如癌症存活率、家族癌症史、提到的癌症类型、风险认知以及与医生的讨论。两位领域专家独立注释了 100 个条目的随机样本，以创建黄金标准数据集。然后，我们在黄金标准数据集上使用 OpenAI 的“GPT-4o-mini”进行迭代提示工程，以构建优化的管道，使我们能够从大型数据集中提取变量。优化后的 LLM 在所有变量上实现了 0.85 以上的准确率，精度、召回率和 F1 得分宏平均值 > 0.90，表明性能均衡。稳定性测试显示，运行期间的匹配率为 95%，证实了一致性。将该框架应用于完整数据集可以有效提取下游分析所需的变量，成本不到 3 美元，大约一小时即可完成。QuaLLM-Health 表明，LLM 可以有效、高效地从非结构化社交媒体内容中提取临床相关的定量数据。结合人类专业知识和迭代快速改进可确保准确性和可靠性。该方法可以适用于跨各个健康领域的患者生成数据的大规模分析，为医疗保健研究提供有价值的见解。

Title: New Faithfulness-Centric Interpretability Paradigms for Natural Language Processing

• Authors: Andreas Madsen
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.17992
• Pdf URL: https://arxiv.org/pdf/2411.17992
• Copy Paste: [[2411.17992]] New Faithfulness-Centric Interpretability Paradigms for Natural Language Processing(https://arxiv.org/abs/2411.17992)
• Keywords: language model
• Abstract: As machine learning becomes more widespread and is used in more critical applications, it's important to provide explanations for these models, to prevent unintended behavior. Unfortunately, many current interpretability methods struggle with faithfulness. Therefore, this Ph.D. thesis investigates the question "How to provide and ensure faithful explanations for complex general-purpose neural NLP models?" The main thesis is that we should develop new paradigms in interpretability. This is achieved by first developing solid faithfulness metrics and then applying the lessons learned from this investigation to develop new paradigms. The two new paradigms explored are faithfulness measurable models (FMMs) and self-explanations. The idea in self-explanations is to have large language models explain themselves, we identify that current models are not capable of doing this consistently. However, we suggest how this could be achieved. The idea of FMMs is to create models that are designed such that measuring faithfulness is cheap and precise. This makes it possible to optimize an explanation towards maximum faithfulness, which makes FMMs designed to be explained. We find that FMMs yield explanations that are near theoretical optimal in terms of faithfulness. Overall, from all investigations of faithfulness, results show that post-hoc and intrinsic explanations are by default model and task-dependent. However, this was not the case when using FMMs, even with the same post-hoc explanation methods. This shows, that even simple modifications to the model, such as randomly masking the training dataset, as was done in FMMs, can drastically change the situation and result in consistently faithful explanations. This answers the question of how to provide and ensure faithful explanations.
• 摘要：随着机器学习变得越来越普遍，并用于越来越关键的应用，为这些模型提供解释以防止意外行为变得非常重要。不幸的是，许多当前的可解释性方法在忠实性方面存在困难。因此，这篇博士论文研究了“如何为复杂的通用神经 NLP 模型提供并确保忠实的解释？”这个问题。主要论点是，我们应该开发可解释性的新范式。这是通过首先开发可靠的忠实度指标，然后应用从这次调查中吸取的经验教训来开发新范式来实现的。探索的两个新范式是忠实度可测量模型 (FMM) 和自我解释。自我解释的理念是让大型语言模型自我解释，我们发现当前的模型无法始终如一地做到这一点。但是，我们提出了如何实现这一点。FMM 的理念是创建设计为能够廉价而精确地测量忠实度的模型。这使得优化解释以实现最大忠实度成为可能，这使得 FMM 被设计为可解释的。我们发现，FMM 给出的解释在忠实度方面接近理论最优。总体而言，从所有对忠实度的调查来看，结果表明事后解释和内在解释默认依赖于模型和任务。然而，使用 FMM 时情况并非如此，即使使用相同的事后解释方法也是如此。这表明，即使是对模型进行简单的修改，例如随机屏蔽训练数据集（如在 FMM 中所做的那样），也可以彻底改变情况并产生始终如一的忠实解释。这回答了如何提供和确保忠实解释的问题。

Title: DRS: Deep Question Reformulation With Structured Output

• Authors: Zhecheng Li, Yiwei Wang, Bryan Hooi, Yujun Cai, Nanyun Peng, Kai-Wei Chang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.17993
• Pdf URL: https://arxiv.org/pdf/2411.17993
• Copy Paste: [[2411.17993]] DRS: Deep Question Reformulation With Structured Output(https://arxiv.org/abs/2411.17993)
• Keywords: language model, gpt, llm
• Abstract: Question answering is a fundamental capability of large language models (LLMs). However, when people encounter completely new knowledge texts, they often ask questions that the text cannot answer due to a lack of understanding of the knowledge. Recent research shows that large language models identify the unanswerability of questions, but they lack the ability to help people reformulate their questions. Even powerful models like GPT-3.5 perform poorly in this regard. To enhance the ability of LLMs to assist humans in reformulating questions to extract relevant knowledge from new documents, we propose a zero-shot method called DRS: Deep Question Reformulation With Structured Output. Our proposed method leverages large language models and the DFS-based algorithm to iteratively search for possible entity combinations and constrain the output with certain entities, effectively improving the capabilities of large language models in this area. Extensive experimental results show that our zero-shot DRS method significantly improves the reformulation accuracy of GPT-3.5 from 23.03% to 70.42% and effectively improves the score of open-source large language models, such as Gemma2-9B, from 26.35% to 56.75%.
• 摘要：问答是大型语言模型 (LLM) 的基本能力。然而，当人们遇到全新的知识文本时，由于对知识的理解不足，他们常常会提出文本无法回答的问题。最近的研究表明，大型语言模型可以识别问题的不可回答性，但缺乏帮助人们重新表述问题的能力。即使是像 GPT-3.5 这样强大的模型在这方面也表现不佳。为了增强 LLM 协助人类重新表述问题以从新文档中提取相关知识的能力，我们提出了一种零样本方法，称为 DRS：具有结构化输出的深度问题重新表述。我们提出的方法利用大型语言模型和基于 DFS 的算法迭代搜索可能的实体组合并用某些实体约束输出，有效地提高了大型语言模型在这方面的能力。大量实验结果表明，我们的零样本 DRS 方法显著提高了 GPT-3.5 的重构准确率，从 23.03% 提高到 70.42%，并有效提高了 Gemma2-9B 等开源大型语言模型的得分，从 26.35% 提高到 56.75%。

Title: Can bidirectional encoder become the ultimate winner for downstream applications of foundation models?

• Authors: Lewen Yang, Xuanyu Zhou, Juao Fan, Xinyi Xie, Shengxin Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.18021
• Pdf URL: https://arxiv.org/pdf/2411.18021
• Copy Paste: [[2411.18021]] Can bidirectional encoder become the ultimate winner for downstream applications of foundation models?(https://arxiv.org/abs/2411.18021)
• Keywords: language model, gpt
• Abstract: Over the past few decades, Artificial Intelligence(AI) has progressed from the initial machine learning stage to the deep learning stage, and now to the stage of foundational models. Foundational models have the characteristics of pre-training, transfer learning, and self-supervised learning, and pre-trained models can be fine-tuned and applied to various downstream tasks. Under the framework of foundational models, models such as Bidirectional Encoder Representations from Transformers(BERT) and Generative Pre-trained Transformer(GPT) have greatly advanced the development of natural language processing(NLP), especially the emergence of many models based on BERT. BERT broke through the limitation of only using one-way methods for language modeling in pre-training by using a masked language model. It can capture bidirectional context information to predict the masked words in the sequence, this can improve the feature extraction ability of the model. This makes the model very useful for downstream tasks, especially for specialized applications. The model using the bidirectional encoder can better understand the domain knowledge and be better applied to these downstream tasks. So we hope to help understand how this technology has evolved and improved model performance in various natural language processing tasks under the background of foundational models and reveal its importance in capturing context information and improving the model's performance on downstream tasks. This article analyzes one-way and bidirectional models based on GPT and BERT and compares their differences based on the purpose of the model. It also briefly analyzes BERT and the improvements of some models based on BERT. The model's performance on the Stanford Question Answering Dataset(SQuAD) and General Language Understanding Evaluation(GLUE) was compared.
• 摘要：在过去的几十年里，人工智能从最初的机器学习阶段发展到深度学习阶段，现在又进入了基础模型阶段。基础模型具有预训练、迁移学习和自监督学习的特点，预训练模型可以进行微调并应用于各种下游任务。在基础模型的框架下，BERT、GPT等模型极大地推动了自然语言处理的发展，尤其是基于BERT的众多模型的出现。BERT通过使用掩码语言模型突破了预训练时只能使用单向方法进行语言建模的限制。它可以捕获双向上下文信息来预测序列中被掩码的单词，从而提高模型的特征提取能力。这使得该模型对于下游任务非常有用，尤其是对于专门的应用。使用双向编码器的模型可以更好地理解领域知识，从而更好地应用于这些下游任务。因此我们希望在基础模型的背景下，帮助理解这项技术是如何在各种自然语言处理任务中演变和提高模型性能的，并揭示其在捕捉上下文信息和提高模型在下游任务上性能方面的重要性。本文分析了基于GPT和BERT的单向和双向模型，并根据模型的用途比较了它们之间的差异。还简要分析了BERT以及一些基于BERT的模型的改进。比较了模型在斯坦福问答数据集(SQuAD)和通用语言理解评估(GLUE)上的表现。

Title: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache

• Authors: Akshat Sharma, Hangliang Ding, Jianping Li, Neel Dani, Minjia Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.18077
• Pdf URL: https://arxiv.org/pdf/2411.18077
• Copy Paste: [[2411.18077]] Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache(https://arxiv.org/abs/2411.18077)
• Keywords: llm, long context
• Abstract: How to efficiently serve LLMs in practice has become exceptionally challenging due to their prohibitive memory and computation requirements. In this study, we investigate optimizing the KV cache, whose memory footprint poses a critical bottleneck in LLM inference, especially when dealing with long context tasks. To tackle the challenge, we introduce MiniKV, a KV cache optimization method that simultaneously preserves long context task accuracy while significantly reducing KV cache size via a novel 2-bit layer-discriminative KV cache. More importantly, we develop specialized CUDA kernels to make MiniKV compatible with FlashAttention. Experiments on a wide range of long context tasks show that MiniKV effectively achieves 86% KV cache compression ratio while recovering over 98.5% of accuracy, outperforming state-of-the-art methods while achieving excellent measured system performance improvements.
• 摘要：由于 LLM 对内存和计算的要求过高，如何在实践中有效地为其提供服务已经变得极具挑战性。在本研究中，我们研究了如何优化 KV 缓存，它的内存占用是 LLM 推理的一个关键瓶颈，尤其是在处理长上下文任务时。为了应对这一挑战，我们引入了 MiniKV，这是一种 KV 缓存优化方法，它通过一种新颖的 2 位层判别 KV 缓存，在保持长上下文任务准确性的同时显著减少了 KV 缓存大小。更重要的是，我们开发了专门的 CUDA 内核，使 MiniKV 与 FlashAttention 兼容。在广泛的长上下文任务上的实验表明，MiniKV 有效地实现了 86% 的 KV 缓存压缩率，同时恢复了 98.5% 以上的准确率，超越了最先进的方法，同时实现了出色的测量系统性能改进。

Title: Training and Evaluating Language Models with Template-based Data Generation

• Authors: Yifan Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.18104
• Pdf URL: https://arxiv.org/pdf/2411.18104
• Copy Paste: [[2411.18104]] Training and Evaluating Language Models with Template-based Data Generation(https://arxiv.org/abs/2411.18104)
• Keywords: language model, gpt, llm
• Abstract: The rapid advancement of large language models (LLMs) such as GPT-3, PaLM, and Llama has significantly transformed natural language processing, showcasing remarkable capabilities in understanding and generating language. However, these models often struggle with tasks requiring complex reasoning, particularly in mathematical problem-solving, due in part to the scarcity of large-scale, high-quality, domain-specific datasets necessary for training sophisticated reasoning abilities. To address this limitation, we introduce Template-based Data Generation (TDG), a novel approach that leverages LLMs (GPT-4) to automatically generate parameterized meta-templates, which are then used to synthesize a vast array of high-quality problems and solutions. Leveraging TDG, we create TemplateMath Part I: TemplateGSM, a dataset comprising over 7 million synthetically generated grade school math problems--each accompanied by code-based and natural language solutions--with the potential to generate an effectively unlimited number more. This dataset alleviates the scarcity of large-scale mathematical datasets and serves as a valuable resource for pre-training, fine-tuning, and evaluating LLMs in mathematical reasoning. Our method not only enables the generation of virtually infinite data but also elevates data augmentation to a new level by using GPT-4 for meta-template generation, ensuring diverse and high-quality problem structures. The TemplateMath Part I: TemplateGSM dataset is publicly available at this https URL. The code is available at this https URL.
• 摘要：GPT-3、PaLM 和 Llama 等大型语言模型 (LLM) 的快速发展极大地改变了自然语言处理，展示了其在理解和生成语言方面的卓越能力。然而，这些模型在执行需要复杂推理的任务时往往举步维艰，尤其是在数学问题解决方面，部分原因是缺乏训练复杂推理能力所需的大规模、高质量、领域特定的数据集。为了解决这一限制，我们引入了基于模板的数据生成 (TDG)，这是一种利用 LLM (GPT-4) 自动生成参数化元模板的新方法，然后使用这些元模板来合成大量高质量的问题和解决方案。利用 TDG，我们创建了 TemplateMath 第 I 部分：TemplateGSM，这是一个包含超过 700 万道合成生成的小学数学问题的数据集——每道题都附有基于代码和自然语言的解决方案——并且有可能生成无限数量的其他问题。该数据集缓解了大规模数学数据集的稀缺性，是预训练、微调和评估数学推理 LLM 的宝贵资源。我们的方法不仅可以生成几乎无限的数据，还可以通过使用 GPT-4 进行元模板生成将数据增强提升到一个新的水平，从而确保多样化和高质量的问题结构。TemplateMath 第一部分：TemplateGSM 数据集在此 https URL 上公开可用。代码可在此 https URL 上找到。

Title: Curriculum Demonstration Selection for In-Context Learning

• Authors: Duc Anh Vu, Nguyen Tran Cong Duy, Xiaobao Wu, Hoang Minh Nhat, Du Mingzhe, Nguyen Thanh Thong, Anh Tuan Luu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.18126
• Pdf URL: https://arxiv.org/pdf/2411.18126
• Copy Paste: [[2411.18126]] Curriculum Demonstration Selection for In-Context Learning(https://arxiv.org/abs/2411.18126)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have shown strong in-context learning (ICL) abilities with a few demonstrations. However, one critical challenge is how to select demonstrations to elicit the full potential of LLMs. In this paper, we propose Curriculum Demonstration Selection (CDS), a novel demonstration selection method for ICL. Instead of merely using similarity, CDS additionally partitions samples by their complexity measurements. Following curriculum learning, CDS then selects demonstrations from easy to difficult. Thus the selected demonstrations cover a wide range of difficulty levels, enabling LLMs to learn from varied complexities within the training set. Experiments demonstrate that our CDS consistently outperforms baseline methods, achieving notable improvements across nine LLMs on three benchmarks. Moreover, CDS proves especially effective in enhancing LLM performance in solving challenging problems.
• 摘要：大型语言模型 (LLM) 通过少量演示就展现出了强大的上下文学习 (ICL) 能力。然而，一个关键挑战是如何选择演示以充分发挥 LLM 的潜力。在本文中，我们提出了课程演示选择 (CDS)，一种新颖的 ICL 演示选择方法。CDS 不仅仅使用相似性，还根据复杂度测量对样本进行划分。在课程学习之后，CDS 会从易到难选择演示。因此，选定的演示涵盖了广泛的难度级别，使 LLM 能够从训练集中的不同复杂性中学习。实验表明，我们的 CDS 始终优于基线方法，在三个基准上的九个 LLM 中取得了显着的改进。此外，CDS 在提高 LLM 解决挑战性问题的性能方面被证明特别有效。

Title: A survey on cutting-edge relation extraction techniques based on language models

• Authors: Jose A. Diaz-Garcia, Julio Amador Diaz Lopez
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.18157
• Pdf URL: https://arxiv.org/pdf/2411.18157
• Copy Paste: [[2411.18157]] A survey on cutting-edge relation extraction techniques based on language models(https://arxiv.org/abs/2411.18157)
• Keywords: language model, llm
• Abstract: This comprehensive survey delves into the latest advancements in Relation Extraction (RE), a pivotal task in natural language processing essential for applications across biomedical, financial, and legal sectors. This study highlights the evolution and current state of RE techniques by analyzing 137 papers presented at the Association for Computational Linguistics (ACL) conferences over the past four years, focusing on models that leverage language models. Our findings underscore the dominance of BERT-based methods in achieving state-of-the-art results for RE while also noting the promising capabilities of emerging large language models (LLMs) like T5, especially in few-shot relation extraction scenarios where they excel in identifying previously unseen relations.
• 摘要：这项全面的调查深入探讨了关系提取 (RE) 的最新进展，这是自然语言处理中的一项关键任务，对于生物医学、金融和法律领域的应用至关重要。这项研究通过分析过去四年在计算语言学协会 (ACL) 会议上发表的 137 篇论文，重点介绍了 RE 技术的发展和现状，重点关注了利用语言模型的模型。我们的研究结果强调了基于 BERT 的方法在实现 RE 的最先进结果方面占据主导地位，同时也注意到了 T5 等新兴大型语言模型 (LLM) 的良好能力，尤其是在少数关系提取场景中，它们擅长识别以前未见过的关系。

Title: SentiXRL: An advanced large language Model Framework for Multilingual Fine-Grained Emotion Classification in Complex Text Environment

• Authors: Jie Wang, Yichen Wang, Zhilin Zhang, Jianhao Zeng, Kaidi Wang, Zhiyang Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.18162
• Pdf URL: https://arxiv.org/pdf/2411.18162
• Copy Paste: [[2411.18162]] SentiXRL: An advanced large language Model Framework for Multilingual Fine-Grained Emotion Classification in Complex Text Environment(https://arxiv.org/abs/2411.18162)
• Keywords: language model, llm
• Abstract: With strong expressive capabilities in Large Language Models(LLMs), generative models effectively capture sentiment structures and deep semantics, however, challenges remain in fine-grained sentiment classification across multi-lingual and complex contexts. To address this, we propose the Sentiment Cross-Lingual Recognition and Logic Framework (SentiXRL), which incorporates two modules,an emotion retrieval enhancement module to improve sentiment classification accuracy in complex contexts through historical dialogue and logical reasoning,and a self-circulating analysis negotiation mechanism (SANM)to facilitates autonomous decision-making within a single model for classification this http URL have validated SentiXRL's superiority on multiple standard datasets, outperforming existing models on CPED and CH-SIMS,and achieving overall better performance on MELD,Emorynlp and IEMOCAP. Notably, we unified labels across several fine-grained sentiment annotation datasets and conducted category confusion experiments, revealing challenges and impacts of class imbalance in standard datasets.
• 摘要：生成模型凭借在大型语言模型（LLM）中的强大表达能力，能够有效捕捉情绪结构和深层语义，但在多语言复杂语境中的细粒度情绪分类方面仍然存在挑战。针对这一问题，我们提出了情绪跨语言识别与逻辑框架（SentiXRL），它包含两个模块，一个情绪检索增强模块，通过历史对话和逻辑推理提高复杂语境中的情绪分类准确率，以及一个自循环分析协商机制（SANM），实现单一模型内的自主决策分类。该http URL已在多个标准数据集上验证了SentiXRL的优势，在CPED和CH-SIMS上优于现有模型，并在MELD，Emorynlp和IEMOCAP上实现了整体更好的性能。值得注意的是，我们统一了几个细粒度情绪标注数据集的标签，并进行了类别混淆实验，揭示了标准数据集中类别不平衡的挑战和影响。

Title: Thai Financial Domain Adaptation of THaLLE -- Technical Report

• Authors: KBTG Labs, Atthakorn Petchsod, Pornchanan Balee, Danupat Khamnuansin, Anuruth Lertpiya, Chanatip Saetia, Tawunrat Chalothorn, Thadpong Pongthawornkamol, Monchai Lertsutthiwong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.18242
• Pdf URL: https://arxiv.org/pdf/2411.18242
• Copy Paste: [[2411.18242]] Thai Financial Domain Adaptation of THaLLE -- Technical Report(https://arxiv.org/abs/2411.18242)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) excel in general tasks but struggle with domain-specific challenges, such as specialized terminology and localized regulations. Existing financial LLMs, like FinGPT and BloombergGPT, lack support for the Thai financial domain. We developed a Thai Financial LLM using the Investment Consultant (IC) exam dataset from the Stock Exchange of Thailand. To address dataset limitations, we applied data augmentation, ReLoRA for efficient training, Continued Pretraining (CPT) for domain knowledge, and Rank-Stabilized LoRA (rsLoRA) for fine-tuning. Supervised Fine-Tuning (SFT) simulated exam scenarios, while Direct Preference Optimization (DPO) refined the model using feedback. The model achieved scores of 72%, 72%, and 84% on IC exam levels P1, P2, and P3, respectively, demonstrating its effectiveness in Thai financial advisory tasks and its potential for specialized applications.
• 摘要：大型语言模型 (LLM) 在一般任务上表现出色，但在特定领域的挑战（例如专业术语和本地化法规）方面却举步维艰。现有的金融 LLM（如 FinGPT 和 BloombergGPT）缺乏对泰国金融领域的支持。我们使用泰国证券交易所的投资顾问 (IC) 考试数据集开发了泰国金融 LLM。为了解决数据集限制，我们应用了数据增强、ReLoRA 进行高效训练、持续预训练 (CPT) 进行领域知识学习以及 Rank-Stabilized LoRA (rsLoRA) 进行微调。监督微调 (SFT) 模拟考试场景，而直接偏好优化 (DPO) 使用反馈完善模型。该模型在 IC 考试级别 P1、P2 和 P3 上分别取得了 72%、72% 和 84% 的分数，证明了其在泰国金融咨询任务中的有效性及其在专业应用中的潜力。

Title: A gentle push funziona benissimo: making instructed models in Italian via contrastive activation steering

• Authors: Daniel Scalena, Elisabetta Fersini, Malvina Nissim
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.18247
• Pdf URL: https://arxiv.org/pdf/2411.18247
• Copy Paste: [[2411.18247]] A gentle push funziona benissimo: making instructed models in Italian via contrastive activation steering(https://arxiv.org/abs/2411.18247)
• Keywords: llm
• Abstract: Adapting models to a language that was only partially present in the pre-training data requires fine-tuning, which is expensive in terms of both data and computational resources. As an alternative to fine-tuning, we explore the potential of activation steering-based techniques to enhance model performance on Italian tasks. Through our experiments we show that Italian steering (i) can be successfully applied to different models, (ii) achieves performances comparable to, or even better than, fine-tuned models for Italian, and (iii) yields higher quality and consistency in Italian generations. We also discuss the utility of steering and fine-tuning in the contemporary LLM landscape where models are anyway getting high Italian performances even if not explicitly trained in this language.
• 摘要：将模型调整为仅部分存在于预训练数据中的语言需要进行微调，这在数据和计算资源方面都很昂贵。作为微调的替代方案，我们探索了基于激活转向的技术在意大利语任务中提高模型性能的潜力。通过我们的实验，我们表明意大利语转向 (i) 可以成功应用于不同的模型，(ii) 实现与意大利语微调模型相当甚至更好的性能，以及 (iii) 在意大利语生成中产生更高的质量和一致性。我们还讨论了转向和微调在当代 LLM 领域中的实用性，即使没有明确用这种语言进行训练，模型也能获得很高的意大利语性能。

Title: Neutralizing Backdoors through Information Conflicts for Large Language Models

• Authors: Chen Chen, Yuchen Sun, Xueluan Gong, Jiaxin Gao, Kwok-Yan Lam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.18280
• Pdf URL: https://arxiv.org/pdf/2411.18280
• Copy Paste: [[2411.18280]] Neutralizing Backdoors through Information Conflicts for Large Language Models(https://arxiv.org/abs/2411.18280)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have seen significant advancements, achieving superior performance in various Natural Language Processing (NLP) tasks, from understanding to reasoning. However, they remain vulnerable to backdoor attacks, where models behave normally for standard queries but generate harmful responses or unintended output when specific triggers are activated. Existing backdoor defenses often suffer from drawbacks that they either focus on detection without removal, rely on rigid assumptions about trigger properties, or prove to be ineffective against advanced attacks like multi-trigger backdoors. In this paper, we present a novel method to eliminate backdoor behaviors from LLMs through the construction of information conflicts using both internal and external mechanisms. Internally, we leverage a lightweight dataset to train a conflict model, which is then merged with the backdoored model to neutralize malicious behaviors by embedding contradictory information within the model's parametric memory. Externally, we incorporate convincing contradictory evidence into the prompt to challenge the model's internal backdoor knowledge. Experimental results on classification and conversational tasks across 4 widely used LLMs demonstrate that our method outperforms 8 state-of-the-art backdoor defense baselines. We can reduce the attack success rate of advanced backdoor attacks by up to 98% while maintaining over 90% clean data accuracy. Furthermore, our method has proven to be robust against adaptive backdoor attacks. The code will be open-sourced upon publication.
• 摘要：大型语言模型 (LLM) 取得了重大进展，在从理解到推理的各种自然语言处理 (NLP) 任务中取得了卓越的表现。然而，它们仍然容易受到后门攻击，即模型对标准查询表现正常，但在激活特定触发器时会产生有害响应或意外输出。现有的后门防御通常存在以下缺点：要么专注于检测而不进行删除，要么依赖于对触发器属性的严格假设，要么被证明对多触发器后门等高级攻击无效。在本文中，我们提出了一种通过使用内部和外部机制构建信息冲突来消除 LLM 中后门行为的新方法。在内部，我们利用轻量级数据集来训练冲突模型，然后将其与后门模型合并，通过在模型的参数内存中嵌入矛盾信息来消除恶意行为。在外部，我们将令人信服的矛盾证据纳入提示中，以挑战模型的内部后门知识。在 4 种广泛使用的 LLM 上进行的分类和对话任务实验结果表明，我们的方法优于 8 种最先进的后门防御基线。我们可以将高级后门攻击的成功率降低高达 98%，同时保持 90% 以上的清洁数据准确率。此外，我们的方法已被证明能够抵御自适应后门攻击。代码将在发布后开源。

Title: Can LLMs assist with Ambiguity? A Quantitative Evaluation of various Large Language Models on Word Sense Disambiguation

• Authors: T.G.D.K. Sumanathilaka, Nicholas Micallef, Julian Hough
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.18337
• Pdf URL: https://arxiv.org/pdf/2411.18337
• Copy Paste: [[2411.18337]] Can LLMs assist with Ambiguity? A Quantitative Evaluation of various Large Language Models on Word Sense Disambiguation(https://arxiv.org/abs/2411.18337)
• Keywords: language model, llm, prompt
• Abstract: Ambiguous words are often found in modern digital communications. Lexical ambiguity challenges traditional Word Sense Disambiguation (WSD) methods, due to limited data. Consequently, the efficiency of translation, information retrieval, and question-answering systems is hindered by these limitations. This study investigates the use of Large Language Models (LLMs) to improve WSD using a novel approach combining a systematic prompt augmentation mechanism with a knowledge base (KB) consisting of different sense interpretations. The proposed method incorporates a human-in-loop approach for prompt augmentation where prompt is supported by Part-of-Speech (POS) tagging, synonyms of ambiguous words, aspect-based sense filtering and few-shot prompting to guide the LLM. By utilizing a few-shot Chain of Thought (COT) prompting-based approach, this work demonstrates a substantial improvement in performance. The evaluation was conducted using FEWS test data and sense tags. This research advances accurate word interpretation in social media and digital communication.
• 摘要：现代数字通信中经常出现歧义词。由于数据有限，词汇歧义对传统的词义消歧 (WSD) 方法提出了挑战。因此，这些限制阻碍了翻译、信息检索和问答系统的效率。本研究调查了使用大型语言模型 (LLM) 来改进 WSD，使用一种新方法将系统提示增强机制与由不同意义解释组成的知识库 (KB) 相结合。所提出的方法结合了人机循环方法进行提示增强，其中提示由词性 (POS) 标记、歧义词的同义词、基于方面的意义过滤和少量提示支持以指导 LLM。通过利用基于少量思路链 (COT) 提示的方法，这项工作展示了性能的显着改进。评估是使用 FEWS 测试数据和意义标签进行的。这项研究促进了社交媒体和数字通信中准确的词语解释。

Title: GPT as ghostwriter at the White House

• Authors: Jacques Savoy
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.18365
• Pdf URL: https://arxiv.org/pdf/2411.18365
• Copy Paste: [[2411.18365]] GPT as ghostwriter at the White House(https://arxiv.org/abs/2411.18365)
• Keywords: language model, gpt, llm, chat
• Abstract: Recently several large language models (LLMs) have demonstrated their capability to generate a message in response to a user request. Such scientific breakthroughs promote new perspectives but also some fears. The main focus of this study is to analyze the written style of one LLM called ChatGPT 3.5 by comparing its generated messages with those of the recent US presidents. To achieve this objective, we compare the State of the Union addresses written by Reagan to Obama with those automatically produced by ChatGPT. We found that ChatGPT tends to overuse the lemma "we" as well as nouns and commas. On the other hand, the generated speeches employ less verbs and include, in mean, longer sentences. Even when imposing a given style to ChatGPT, the resulting speech remains distinct from messages written by the target author. Moreover, ChatGPT opts for a neutral tone with mainly positive emotional expressions and symbolic terms (e.g., freedom, nation). Finally, we show that the GPT's style exposes distinct features compared to real presidential addresses.
• 摘要：最近，几个大型语言模型 (LLM) 已经展示了它们根据用户请求生成消息的能力。这样的科学突破促进了新的观点，但也带来了一些担忧。这项研究的主要重点是分析一个名为 ChatGPT 3.5 的 LLM 的书写风格，将其生成的消息与最近几任美国总统的消息进行比较。为了实现这一目标，我们将里根写给奥巴马的国情咨文与 ChatGPT 自动生成的国情咨文进行了比较。我们发现 ChatGPT 倾向于过度使用词根“我们”以及名词和逗号。另一方面，生成的演讲使用较少的动词，并且包含更长的句子。即使将给定的风格强加给 ChatGPT，生成的演讲仍然与目标作者编写的消息不同。此外，ChatGPT 选择中性语气，主要使用积极的情感表达和象征性术语（例如，自由、国家）。最后，我们表明，与真正的总统演讲相比，GPT 的风格展现出鲜明的特征。

Title: ChatGPT as speechwriter for the French presidents

• Authors: Dominique Labbé, Cyril Labbé, Jacques Savoy
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.18382
• Pdf URL: https://arxiv.org/pdf/2411.18382
• Copy Paste: [[2411.18382]] ChatGPT as speechwriter for the French presidents(https://arxiv.org/abs/2411.18382)
• Keywords: language model, gpt, llm, chat
• Abstract: Generative AI proposes several large language models (LLMs) to automatically generate a message in response to users' requests. Such scientific breakthroughs promote new writing assistants but with some fears. The main focus of this study is to analyze the written style of one LLM called ChatGPT by comparing its generated messages with those of the recent French presidents. To achieve this, we compare end-of-the-year addresses written by Chirac, Sarkozy, Hollande, and Macron with those automatically produced by ChatGPT. We found that ChatGPT tends to overuse nouns, possessive determiners, and numbers. On the other hand, the generated speeches employ less verbs, pronouns, and adverbs and include, in mean, too standardized sentences. Considering some words, one can observe that ChatGPT tends to overuse "to must" (devoir), "to continue" or the lemma "we" (nous). Moreover, GPT underuses the auxiliary verb "to be" (^etre), or the modal verbs "to will" (vouloir) or "to have to" (falloir). In addition, when a short text is provided as example to ChatGPT, the machine can generate a short message with a style closed to the original wording. Finally, we reveal that ChatGPT style exposes distinct features compared to real presidential speeches.
• 摘要：生成式人工智能提出了几种大型语言模型 (LLM)，用于根据用户的请求自动生成消息。这样的科学突破促进了新的写作助手的发展，但也带来了一些担忧。这项研究的主要重点是分析一个名为 ChatGPT 的 LLM 的写作风格，将其生成的消息与最近几位法国总统的消息进行比较。为此，我们将希拉克、萨科齐、奥朗德和马克龙撰写的年终演讲与 ChatGPT 自动生成的演讲进行了比较。我们发现 ChatGPT 倾向于过度使用名词、所有格限定词和数字。另一方面，生成的演讲使用较少的动词、代词和副词，并且包含过于标准化的句子。考虑一些单词，可以观察到 ChatGPT 倾向于过度使用“必须”(devoir)、“继续”或词干“我们”(nous)。此外，GPT 很少使用助动词“是”（^etre），或情态动词“将”（vouloir）或“不得不”（falloir）。此外，当向 ChatGPT 提供一段简短的文本作为示例时，机器可以生成一条风格接近原始措辞的短信。最后，我们发现 ChatGPT 风格与真正的总统演讲相比具有鲜明的特征。

Title: Topic Modeling and Sentiment Analysis on Japanese Online Media's Coverage of Nuclear Energy

• Authors: Yifan Sun, Hirofumi Tsuruta, Masaya Kumagai, Ken Kurosaki
• Subjects: cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2411.18383
• Pdf URL: https://arxiv.org/pdf/2411.18383
• Copy Paste: [[2411.18383]] Topic Modeling and Sentiment Analysis on Japanese Online Media's Coverage of Nuclear Energy(https://arxiv.org/abs/2411.18383)
• Keywords: language model
• Abstract: Thirteen years after the Fukushima Daiichi nuclear power plant accident, Japan's nuclear energy accounts for only approximately 6% of electricity production, as most nuclear plants remain shut down. To revitalize the nuclear industry and achieve sustainable development goals, effective communication with Japanese citizens, grounded in an accurate understanding of public sentiment, is of paramount importance. While nationwide surveys have traditionally been used to gauge public views, the rise of social media in recent years has provided a promising new avenue for understanding public sentiment. To explore domestic sentiment on nuclear energy-related issues expressed online, we analyzed the content and comments of over 3,000 YouTube videos covering topics related to nuclear energy. Topic modeling was used to extract the main topics from the videos, and sentiment analysis with large language models classified user sentiments towards each topic. Additionally, word co-occurrence network analysis was performed to examine the shift in online discussions during August and September 2023 regarding the release of treated water. Overall, our results provide valuable insights into the online discourse on nuclear energy and contribute to a more comprehensive understanding of public sentiment in Japan.
• 摘要：福岛第一核电站事故发生十三年后，日本的核能仅占电力产量的约 6%，因为大多数核电站仍处于关闭状态。为了振兴核工业并实现可持续发展目标，在准确了解公众情绪的基础上与日本公民进行有效沟通至关重要。虽然全国范围的调查传统上被用来衡量公众观点，但近年来社交媒体的兴起为了解公众情绪提供了一条有希望的新途径。为了探索国内对网上核能相关问题的情绪，我们分析了 3,000 多个涉及核能主题的 YouTube 视频的内​​容和评论。主题建模用于从视频中提取主要主题，并使用大型语言模型进行情绪分析对用户对每个主题的情绪进行分类。此外，还进行了词共现网络分析，以检查 2023 年 8 月和 9 月有关排放处理过的水的在线讨论的变化。总体而言，我们的研究结果为核能的在线讨论提供了宝贵的见解，并有助于更全面地了解日本的公众情绪。

Title: Politicians vs ChatGPT. A study of presuppositions in French and Italian political communication

• Authors: Davide Garassino, Vivana Masia, Nicola Brocca, Alice Delorme Benites
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2411.18403
• Pdf URL: https://arxiv.org/pdf/2411.18403
• Copy Paste: [[2411.18403]] Politicians vs ChatGPT. A study of presuppositions in French and Italian political communication(https://arxiv.org/abs/2411.18403)
• Keywords: language model, gpt, chat
• Abstract: This paper aims to provide a comparison between texts produced by French and Italian politicians on polarizing issues, such as immigration and the European Union, and their chatbot counterparts created with ChatGPT 3.5. In this study, we focus on implicit communication, in particular on presuppositions and their functions in discourse, which have been considered in the literature as a potential linguistic feature of manipulation. This study also aims to contribute to the emerging literature on the pragmatic competences of Large Language Models.
• 摘要：本文旨在对法国和意大利政客就移民和欧盟等两极化问题所发表的文本与使用 ChatGPT 3.5 创建的聊天机器人进行比较。在本研究中，我们重点关注隐性沟通，特别是预设及其在话语中的功能，这些预设在文献中被视为操纵的潜在语言特征。本研究还旨在为大型语言模型的语用能力方面的新兴文献做出贡献。

Title: Is my Meeting Summary Good? Estimating Quality with a Multi-LLM Evaluator

• Authors: Frederic Kirstein, Terry Ruas, Bela Gipp
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.18444
• Pdf URL: https://arxiv.org/pdf/2411.18444
• Copy Paste: [[2411.18444]] Is my Meeting Summary Good? Estimating Quality with a Multi-LLM Evaluator(https://arxiv.org/abs/2411.18444)
• Keywords: language model, gpt, llm, agent
• Abstract: The quality of meeting summaries generated by natural language generation (NLG) systems is hard to measure automatically. Established metrics such as ROUGE and BERTScore have a relatively low correlation with human judgments and fail to capture nuanced errors. Recent studies suggest using large language models (LLMs), which have the benefit of better context understanding and adaption of error definitions without training on a large number of human preference judgments. However, current LLM-based evaluators risk masking errors and can only serve as a weak proxy, leaving human evaluation the gold standard despite being costly and hard to compare across studies. In this work, we present MESA, an LLM-based framework employing a three-step assessment of individual error types, multi-agent discussion for decision refinement, and feedback-based self-training to refine error definition understanding and alignment with human judgment. We show that MESA's components enable thorough error detection, consistent rating, and adaptability to custom error guidelines. Using GPT-4o as its backbone, MESA achieves mid to high Point-Biserial correlation with human judgment in error detection and mid Spearman and Kendall correlation in reflecting error impact on summary quality, on average 0.25 higher than previous methods. The framework's flexibility in adapting to custom error guidelines makes it suitable for various tasks with limited human-labeled data.
• 摘要：自然语言生成 (NLG) 系统生成的会议摘要的质量很难自动衡量。诸如 ROUGE 和 BERTScore 等既定指标与人类判断的相关性相对较低，无法捕捉到细微的错误。最近的研究表明，使用大型语言模型 (LLM) 具有更好的上下文理解和错误定义调整优势，而无需对大量人类偏好判断进行训练。然而，当前基于 LLM 的评估器存在掩盖错误的风险，只能充当弱代理，尽管成本高昂且难以在各个研究中进行比较，但人工评估仍是黄金标准。在这项工作中，我们提出了 MESA，这是一个基于 LLM 的框架，采用三步评估单个错误类型、多代理讨论以改进决策以及基于反馈的自我训练来改进错误定义理解并与人类判断保持一致。我们表明，MESA 的组件能够彻底检测错误、进行一致评级并适应自定义错误指南。 MESA 以 GPT-4o 为基础，在错误检测方面实现了与人类判断的中高 Point-Biserial 相关性，在反映错误对摘要质量的影响方面实现了中等 Spearman 和 Kendall 相关性，平均比以前的方法高 0.25。该框架在适应自定义错误指南方面的灵活性使其适用于人工标记数据有限的各种任务。

Title: Draft Model Knows When to Stop: A Self-Verification Length Policy for Speculative Decoding

• Authors: Ziyin Zhang, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Rui Wang, Zhaopeng Tu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.18462
• Pdf URL: https://arxiv.org/pdf/2411.18462
• Copy Paste: [[2411.18462]] Draft Model Knows When to Stop: A Self-Verification Length Policy for Speculative Decoding(https://arxiv.org/abs/2411.18462)
• Keywords: language model
• Abstract: Speculative Decoding (SD) has become an important technique in accelerating the inference speed of large language models. Conventional SD methods employ a fixed draft length, which ignores the token generation difficulty across tasks. Consequently, in this paper, we address such an issue and introduce SVIP - a difficulty-aware dynamic draft length policy for speculative decoding systems. Based on a theoretical lower bound of draft token acceptance rate and its inference-time approximation, SVIP adaptively determines the lengths of draft sequences based on the entropy of each draft token distribution. Experimental results on mainstream SD benchmarks and frameworks demonstrate the superior performance of SVIP, achieving up to 20\% walltime speedup on SpecBench over baseline SD methods and 60\% speedup on MT-Bench for long-form generation of up to 8K tokens. Moreover, SVIP is totally training-free and compatible with any existing SD methods that generate draft tokens autoregressively. Experimental results also show that SVIP yields consistent walltime improvement on top of GliDe & CaPE and EAGLE-2.
• 摘要：推测解码 (SD) 已成为加速大型语言模型推理速度的重要技术。传统的 SD 方法采用固定的草稿长度，忽略了跨任务的 token 生成难度。因此，在本文中，我们解决了这个问题，并引入了 SVIP——一种用于推测解码系统的难度感知动态草稿长度策略。基于草稿 token 接受率的理论下限及其推理时间近似值，SVIP 根据每个草稿 token 分布的熵自适应地确定草稿序列的长度。主流 SD 基准和框架上的实验结果证明了 SVIP 的卓越性能，与基线 SD 方法相比，在 SpecBench 上实现了高达 20% 的挂机时间加速，在 MT-Bench 上实现了 60% 的加速，用于生成多达 8K 个 token 的长格式。此外，SVIP 完全无需训练，并且与任何现有的自回归生成草稿 token 的 SD 方法兼容。实验结果还表明，SVIP 在 GliDe & CaPE 和 EAGLE-2 的基础上实现了持续的挂机时间改进。

Title: Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTS

• Authors: Jinyang Wu, Mingkuan Feng, Shuai Zhang, Feihu Che, Zengqi Wen, Jianhua Tao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.18478
• Pdf URL: https://arxiv.org/pdf/2411.18478
• Copy Paste: [[2411.18478]] Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTS(https://arxiv.org/abs/2411.18478)
• Keywords: language model, gpt, llm, prompt
• Abstract: In-context Learning (ICL) enables large language models (LLMs) to tackle downstream tasks through sophisticated prompting and high-quality demonstrations. However, this traditional ICL paradigm shows limitations when facing complex mathematical reasoning tasks, primarily due to its heavy dependence on example quality and the necessity for human intervention in challenging scenarios. To address these limitations, this paper presents HiAR-ICL, a \textbf{Hi}gh-level \textbf{A}utomated \textbf{R}easoning paradigm in \textbf{ICL} that shifts focus from specific examples to abstract thinking patterns, extending the conventional concept of context in ICL. HiAR-ICL introduces five atomic reasoning actions as fundamental components for constructing chain-structured patterns. Using Monte Carlo Tree Search, we explore reasoning paths and construct thought cards to guide subsequent inference. We then develop a cognitive complexity framework that dynamically matches problems with appropriate thought cards. Experimental results demonstrate HiAR-ICL's effectiveness, achieving state-of-the-art accuracy (79.6$\%$) on the MATH benchmark with Qwen2.5-7B-Instruct, surpassing GPT-4o (76.6$\%$) and Claude 3.5 (71.1$\%$).
• 摘要：情境学习 (ICL) 使大型语言模型 (LLM) 能够通过复杂的提示和高质量的演示来解决下游任务。然而，这种传统的 ICL 范式在面对复杂的数学推理任务时显示出局限性，主要是因为它严重依赖示例质量，并且在具有挑战性的场景中需要人工干预。为了解决这些限制，本文提出了 HiAR-ICL，这是 \textbf{ICL} 中的一种 \textbf{高级} \textbf{自动} \textbf{推理}范式，它将重点从具体示例转移到抽象思维模式，扩展了 ICL 中传统的上下文概念。HiAR-ICL 引入了五个原子推理动作作为构建链式结构模式的基本组成部分。使用蒙特卡洛树搜索，我们探索推理路径并构建思维卡来指导后续推理。然后，我们开发了一个认知复杂性框架，可以动态地将问题与适当的思维卡匹配。实验结果证明了 HiAR-ICL 的有效性，在 Qwen2.5-7B-Instruct 的 MATH 基准上实现了最佳准确率 (79.6%$)，超过了 GPT-4o (76.6%$) 和 Claude 3.5 (71.1%$)。

Title: Emergence of Self-Identity in AI: A Mathematical Framework and Empirical Study with Generative Large Language Models

• Authors: Minhyeok Lee
• Subjects: cs.CL, math.MG
• Abstract URL: https://arxiv.org/abs/2411.18530
• Pdf URL: https://arxiv.org/pdf/2411.18530
• Copy Paste: [[2411.18530]] Emergence of Self-Identity in AI: A Mathematical Framework and Empirical Study with Generative Large Language Models(https://arxiv.org/abs/2411.18530)
• Keywords: language model
• Abstract: This paper introduces a mathematical framework for defining and quantifying self-identity in artificial intelligence (AI) systems, addressing a critical gap in the theoretical foundations of artificial consciousness. While existing approaches to artificial self-awareness often rely on heuristic implementations or philosophical abstractions, we present a formal framework grounded in metric space theory, measure theory, and functional analysis. Our framework posits that self-identity emerges from two mathematically quantifiable conditions: the existence of a connected continuum of memories $C \subseteq \mathcal{M}$ in a metric space $(\mathcal{M}, d_{\mathcal{M}})$, and a continuous mapping $I: \mathcal{M} \to \mathcal{S}$ that maintains consistent self-recognition across this continuum, where $(\mathcal{S}, d_{\mathcal{S}})$ represents the metric space of possible self-identities. To validate this theoretical framework, we conducted empirical experiments using the Llama 3.2 1B model, employing Low-Rank Adaptation (LoRA) for efficient fine-tuning. The model was trained on a synthetic dataset containing temporally structured memories, designed to capture the complexity of coherent self-identity formation. Our evaluation metrics included quantitative measures of self-awareness, response consistency, and linguistic precision. The experimental results demonstrate substantial improvements in measurable self-awareness metrics, with the primary self-awareness score increasing from 0.276 to 0.801. This enables the structured creation of AI systems with validated self-identity features. The implications of our study are immediately relevant to the fields of humanoid robotics and autonomous systems.
• 摘要：本文介绍了一个数学框架，用于定义和量化人工智能 (AI) 系统中的自我认同，以解决人工智能理论基础中的一个关键空白。虽然现有的人工智能自我意识方法通常依赖于启发式实现或哲学抽象，但我们提出了一个基于度量空间理论、测度论和功能分析的形式框架。我们的框架假定自我认同来自两个数学上可量化的条件：在度量空间 $(\mathcal{M}, d_{\mathcal{M}})$ 中存在一个连通的记忆连续体 $C \subseteq \mathcal{M}$，以及一个连续映射 $I:\mathcal{M} \to \mathcal{S}$，该映射可在整个连续体中保持一致的自我认知，其中 $(\mathcal{S}, d_{\mathcal{S}})$ 表示可能的自我认同的度量空间。为了验证这一理论框架，我们使用 Llama 3.2 1B 模型进行了实证实验，采用低秩自适应 (LoRA) 进行有效微调。该模型在包含时间结构化记忆的合成数据集上进行训练，旨在捕捉连贯自我认同形成的复杂性。我们的评估指标包括自我意识、反应一致性和语言精确度的定量测量。实验结果表明，可测量的自我意识指标有了显著改善，主要自我意识得分从 0.276 提高到 0.801。这使得能够结构化地创建具有经过验证的自我认同特征的人工智能系统。我们的研究意义与人形机器人和自主系统领域息息相关。

Title: Retrofitting (Large) Language Models with Dynamic Tokenization

• Authors: Darius Feher, Benjamin Minixhofer, Ivan Vulić
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.18553
• Pdf URL: https://arxiv.org/pdf/2411.18553
• Copy Paste: [[2411.18553]] Retrofitting (Large) Language Models with Dynamic Tokenization(https://arxiv.org/abs/2411.18553)
• Keywords: language model
• Abstract: Current language models (LMs) use a fixed, static subword tokenizer. This choice, often taken for granted, typically results in degraded efficiency and capabilities in languages other than English, and makes it challenging to apply LMs to new domains or languages. To address these issues, we propose retrofitting LMs with dynamic tokenization: a way to dynamically decide on token boundaries based on the input text. For encoder-style models, we introduce a subword-merging algorithm inspired by byte-pair encoding (BPE), but at a batch level. We merge frequent subword sequences in a batch, then apply a pretrained embedding-prediction hypernetwork to compute the token embeddings on-the-fly. When applied with word-level boundaries, this on average reduces token sequence lengths by >20% across 14 languages on XNLI with XLM-R while degrading its task performance by less than 2%. For decoder-style models, we apply dynamic tokenization in two ways: 1) for prefilling, maintaining performance of Mistral-7B almost completely with up to 40% sequence reduction - relative to the word-level; and 2) via an approximate nearest neighbor index, achieving fast generation with a one million token vocabulary, demonstrating scalability to even larger, dynamic vocabularies. Overall, our findings show that dynamic tokenization substantially improves inference speed and promotes fairness across languages, making a leap towards overcoming the limitations of static tokenization and enabling more equitable and adaptable LMs.
• 摘要：当前的语言模型 (LM) 使用固定的静态子词标记器。这种选择通常被视为理所当然，通常会导致除英语以外的其他语言的效率和能力下降，并且使将 LM 应用于新领域或语言变得具有挑战性。为了解决这些问题，我们建议使用动态标记化来改造 LM：一种根据输入文本动态决定标记边界的方法。对于编码器样式的模型，我们引入了一种受字节对编码 (BPE) 启发的子词合并算法，但在批处理级别。我们批量合并频繁子词序列，然后应用预训练的嵌入预测超网络来动态计算标记嵌入。当应用词级边界时，这平均可将 XNLI 上 14 种语言的标记序列长度减少 20% 以上，同时将其任务性能降低不到 2%。对于解码器样式的模型，我们以两种方式应用动态标记化：1) 用于预填充，几乎完全保持 Mistral-7B 的性能，同时相对于单词级别，序列减少高达 40%；2) 通过近似最近邻索引，使用一百万个标记词汇表实现快速生成，展示了对更大的动态词汇表的可扩展性。总体而言，我们的研究结果表明，动态标记化大大提高了推理速度并促进了跨语言的公平性，朝着克服静态标记化的局限性和实现更公平、适应性更强的 LM 迈出了一大步。

Title: Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning

• Authors: Omkar Khade, Shruti Jagdale, Abhishek Phaltankar, Gauri Takalikar, Raviraj Joshi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.18571
• Pdf URL: https://arxiv.org/pdf/2411.18571
• Copy Paste: [[2411.18571]] Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning(https://arxiv.org/abs/2411.18571)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated remarkable multilingual capabilities, yet challenges persist in adapting these models for low-resource languages. In this study, we investigate the effects of Low-Rank Adaptation (LoRA) Parameter-Efficient Fine-Tuning (PEFT) on multilingual Gemma models for Marathi, a language with limited resources. Using a translated Alpaca dataset with 52,000 instruction-response pairs, our findings reveal that while evaluation metrics often show a performance decline post-fine-tuning, manual assessments frequently suggest that the fine-tuned models outperform their original counterparts. The observations indicate improvements in target language generation capabilities but a reduction in reasoning abilities following language adaptation. These results underscore the need for improved evaluation methodologies and the creation of high-quality native datasets to accurately assess language-specific model performance in low-resource settings.
• 摘要：大型语言模型 (LLM) 已展示出卓越的多语言能力，但在将这些模型应用于资源匮乏的语言方面仍然存在挑战。在本研究中，我们研究了低秩自适应 (LoRA) 参数高效微调 (PEFT) 对资源有限的语言马拉地语的多语言 Gemma 模型的影响。使用包含 52,000 个指令-响应对的翻译 Alpaca 数据集，我们的研究结果表明，虽然评估指标通常显示微调后性能下降，但手动评估通常表明微调后的模型优于原始模型。观察结果表明，目标语言生成能力有所提高，但语言适应后推理能力有所下降。这些结果强调需要改进评估方法并创建高质量的本机数据集，以准确评估资源匮乏环境中特定于语言的模型性能。

Title: Automated Literature Review Using NLP Techniques and LLM-Based Retrieval-Augmented Generation

• Authors: Nurshat Fateh Ali, Md. Mahdi Mohtasim, Shakil Mosharrof, T. Gopi Krishna
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.18583
• Pdf URL: https://arxiv.org/pdf/2411.18583
• Copy Paste: [[2411.18583]] Automated Literature Review Using NLP Techniques and LLM-Based Retrieval-Augmented Generation(https://arxiv.org/abs/2411.18583)
• Keywords: language model, gpt, llm, retrieval-augmented generation
• Abstract: This research presents and compares multiple approaches to automate the generation of literature reviews using several Natural Language Processing (NLP) techniques and retrieval-augmented generation (RAG) with a Large Language Model (LLM). The ever-increasing number of research articles provides a huge challenge for manual literature review. It has resulted in an increased demand for automation. Developing a system capable of automatically generating the literature reviews from only the PDF files as input is the primary objective of this research work. The effectiveness of several Natural Language Processing (NLP) strategies, such as the frequency-based method (spaCy), the transformer model (Simple T5), and retrieval-augmented generation (RAG) with Large Language Model (GPT-3.5-turbo), is evaluated to meet the primary objective. The SciTLDR dataset is chosen for this research experiment and three distinct techniques are utilized to implement three different systems for auto-generating the literature reviews. The ROUGE scores are used for the evaluation of all three systems. Based on the evaluation, the Large Language Model GPT-3.5-turbo achieved the highest ROUGE-1 score, 0.364. The transformer model comes in second place and spaCy is at the last position. Finally, a graphical user interface is created for the best system based on the large language model.
• 摘要：本研究介绍并比较了多种使用多种自然语言处理 (NLP) 技术和具有大型语言模型 (LLM) 的检索增强生成 (RAG) 自动生成文献综述的方法。不断增加的研究文章数量对手动文献综述提出了巨大挑战。这导致了对自动化的需求增加。开发一个能够仅从 PDF 文件作为输入自动生成文献综述的系统是本研究工作的主要目标。为满足主要目标，评估了几种自然语言处理 (NLP) 策略的有效性，例如基于频率的方法 (spaCy)、Transformer 模型 (Simple T5) 和具有大型语言模型 (GPT-3.5-turbo) 的检索增强生成 (RAG)。本研究实验选择了 SciTLDR 数据集，并使用三种不同的技术来实现三种不同的文献综述自动生成系统。ROUGE 分数用于评估这三个系统。根据评估，大型语言模型 GPT-3.5-turbo 获得了最高的 ROUGE-1 分数，为 0.364。Transformer 模型位居第二，spaCy 位居最后。最后，为基于大型语言模型的最佳系统创建了一个图形用户界面。