2024-08-14

Title: Evaluating Language Models on Entity Disambiguation in Tables

• Authors: Federico Belotti, Fabio Dadda, Marco Cremaschi, Roberto Avogadro, Riccardo Pozzi, Matteo Palmonari
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06423
• Pdf URL: https://arxiv.org/pdf/2408.06423
• Copy Paste: [[2408.06423]] Evaluating Language Models on Entity Disambiguation in Tables(https://arxiv.org/abs/2408.06423)
• Keywords: language model, llm
• Abstract: Tables are crucial containers of information, but understanding their meaning may be challenging. Indeed, recently, there has been a focus on Semantic Table Interpretation (STI), i.e., the task that involves the semantic annotation of tabular data to disambiguate their meaning. Over the years, there has been a surge in interest in data-driven approaches based on deep learning that have increasingly been combined with heuristic-based approaches. In the last period, the advent of Large Language Models (LLMs) has led to a new category of approaches for table annotation. The interest in this research field, characterised by multiple challenges, has led to a proliferation of approaches employing different techniques. However, these approaches have not been consistently evaluated on a common ground, making evaluation and comparison difficult. This work proposes an extensive evaluation of four state-of-the-art (SOTA) approaches - Alligator (formerly s-elBat), Dagobah, TURL, and TableLlama; the first two belong to the family of heuristic-based algorithms, while the others are respectively encoder-only and decoder-only LLMs. The primary objective is to measure the ability of these approaches to solve the entity disambiguation task, with the ultimate aim of charting new research paths in the field.
• 摘要：表格是重要的信息容器，但理解其含义可能具有挑战性。事实上，最近人们一直关注语义表解释（STI），即涉及对表格数据进行语义注释以消除其含义歧义的任务。多年来，人们对基于深度学习的数据驱动方法的兴趣激增，这些方法越来越多地与基于启发式的方法相结合。在过去一段时间里，大型语言模型 (LLM) 的出现导致了一种新的表格注释方法。这个以多重挑战为特点的研究领域引起了人们的兴趣，导致采用不同技术的方法激增。然而，这些方法并没有在共同的基础上进行一致的评估，使得评估和比较变得困难。这项工作提出了对四种最先进（SOTA）方法的广泛评估——Alligator（以前称为 s-elBat）、Dagobah、TURL 和 TableLlama；前两种属于启发式算法，其他分别是仅编码器和仅解码器的 LLM。主要目标是衡量这些方法解决实体消歧任务的能力，最终目的是在该领域开辟新的研究路径。

Title: Cross-Lingual Conversational Speech Summarization with Large Language Models

• Authors: Max Nelson, Shannon Wotherspoon, Francis Keith, William Hartmann, Matthew Snover
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06484
• Pdf URL: https://arxiv.org/pdf/2408.06484
• Copy Paste: [[2408.06484]] Cross-Lingual Conversational Speech Summarization with Large Language Models(https://arxiv.org/abs/2408.06484)
• Keywords: language model, gpt, llm
• Abstract: Cross-lingual conversational speech summarization is an important problem, but suffers from a dearth of resources. While transcriptions exist for a number of languages, translated conversational speech is rare and datasets containing summaries are non-existent. We build upon the existing Fisher and Callhome Spanish-English Speech Translation corpus by supplementing the translations with summaries. The summaries are generated using GPT-4 from the reference translations and are treated as ground truth. The task is to generate similar summaries in the presence of transcription and translation errors. We build a baseline cascade-based system using open-source speech recognition and machine translation models. We test a range of LLMs for summarization and analyze the impact of transcription and translation errors. Adapting the Mistral-7B model for this task performs significantly better than off-the-shelf models and matches the performance of GPT-4.
• 摘要：跨语言对话语音摘要是一个重要问题，但资源匮乏。虽然许多语言都有转录本，但翻译的对话语音却很少，而且没有包含摘要的数据集。我们在现有的 Fisher 和 Callhome 西班牙语-英语语音翻译语料库的基础上，为翻译补充了摘要。摘要是使用 GPT-4 从参考翻译中生成的，并被视为基本事实。任务是在存在转录和翻译错误的情况下生成类似的摘要。我们使用开源语音识别和机器翻译模型构建了一个基于基线级联的系统。我们测试了一系列 LLM 以进行摘要，并分析了转录和翻译错误的影响。针对此任务调整 Mistral-7B 模型的性能明显优于现成的模型，并且与 GPT-4 的性能相当。

Title: Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models

• Authors: Hila Gonen, Terra Blevins, Alisa Liu, Luke Zettlemoyer, Noah A. Smith
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06518
• Pdf URL: https://arxiv.org/pdf/2408.06518
• Copy Paste: [[2408.06518]] Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models(https://arxiv.org/abs/2408.06518)
• Keywords: language model, prompt
• Abstract: Despite their wide adoption, the biases and unintended behaviors of language models remain poorly understood. In this paper, we identify and characterize a phenomenon never discussed before, which we call semantic leakage, where models leak irrelevant information from the prompt into the generation in unexpected ways. We propose an evaluation setting to detect semantic leakage both by humans and automatically, curate a diverse test suite for diagnosing this behavior, and measure significant semantic leakage in 13 flagship models. We also show that models exhibit semantic leakage in languages besides English and across different settings and generation scenarios. This discovery highlights yet another type of bias in language models that affects their generation patterns and behavior.
• 摘要：尽管语言模型被广泛采用，但人们对其偏见和意外行为仍然了解甚少。在本文中，我们识别并描述了一种从未讨论过的现象，我们称之为语义泄漏，即模型以意想不到的方式将提示中的不相关信息泄漏到生成中。我们提出了一种评估设置来检测人工和自动的语义泄漏，策划了一个用于诊断这种行为的多样化测试套件，并测量了 13 个旗舰模型中的显著语义泄漏。我们还表明，模型在英语以外的语言中以及在不同的环境和生成场景中都表现出语义泄漏。这一发现凸显了语言模型中另一种影响其生成模式和行为的偏见。

Title: Chain-of-Strategy Planning with LLMs: Aligning the Generation of Psychotherapy Dialogue with Strategy in Motivational Interviewing

• Authors: Xin Sun, Xiao Tang, Abdallah El Ali, Zhuying Li, Xiaoyu Shen, Pengjie Ren, Jan de Wit, Jiahuan Pei, Jos A.Bosch
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06527
• Pdf URL: https://arxiv.org/pdf/2408.06527
• Copy Paste: [[2408.06527]] Chain-of-Strategy Planning with LLMs: Aligning the Generation of Psychotherapy Dialogue with Strategy in Motivational Interviewing(https://arxiv.org/abs/2408.06527)
• Keywords: language model, llm
• Abstract: Recent advancements in large language models (LLMs) have shown promise in generating psychotherapeutic dialogues, especially in Motivational Interviewing (MI). However, how to employ strategies, a set of motivational interviewing (MI) skills, to generate therapeutic-adherent conversations with explainability is underexplored. We propose an approach called strategy-aware dialogue generation with Chain-of-Strategy (CoS) planning, which first predicts MI strategies as reasoning and utilizes these strategies to guide the subsequent dialogue generation. It brings the potential for controllable and explainable generation in psychotherapy by aligning the generated MI dialogues with therapeutic strategies. Extensive experiments including automatic and human evaluations are conducted to validate the effectiveness of the MI strategy. Our findings demonstrate the potential of LLMs in producing strategically aligned dialogues and suggest directions for practical applications in psychotherapeutic settings.
• 摘要：大型语言模型 (LLM) 的最新进展已显示出在生成心理治疗对话方面的潜力，尤其是在动机访谈 (MI) 中。然而，如何运用策略（一套动机访谈 (MI) 技能）来生成具有可解释性的治疗性对话尚未得到充分探索。我们提出了一种称为策略感知对话生成的方法，该方法使用策略链 (CoS) 规划，首先将 MI 策略预测为推理，然后利用这些策略来指导后续的对话生成。通过将生成的 MI 对话与治疗策略相结合，它为心理治疗中的可控制和可解释生成带来了潜力。进行了包括自动和人工评估在内的大量实验，以验证 MI 策略的有效性。我们的研究结果证明了 LLM 在产生战略一致的对话方面的潜力，并为心理治疗环境中的实际应用提供了方向。

Title: Introducing the NewsPaLM MBR and QE Dataset: LLM-Generated High-Quality Parallel Data Outperforms Traditional Web-Crawled Data

• Authors: Mara Finkelstein, David Vilar, Markus Freitag
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06537
• Pdf URL: https://arxiv.org/pdf/2408.06537
• Copy Paste: [[2408.06537]] Introducing the NewsPaLM MBR and QE Dataset: LLM-Generated High-Quality Parallel Data Outperforms Traditional Web-Crawled Data(https://arxiv.org/abs/2408.06537)
• Keywords: llm
• Abstract: Recent research in neural machine translation (NMT) has shown that training on high-quality machine-generated data can outperform training on human-generated data. This work accompanies the first-ever release of a LLM-generated, MBR-decoded and QE-reranked dataset with both sentence-level and multi-sentence examples. We perform extensive experiments to demonstrate the quality of our dataset in terms of its downstream impact on NMT model performance. We find that training from scratch on our (machine-generated) dataset outperforms training on the (web-crawled) WMT'23 training dataset (which is 300 times larger), and also outperforms training on the top-quality subset of the WMT'23 training dataset. We also find that performing self-distillation by finetuning the LLM which generated this dataset outperforms the LLM's strong few-shot baseline. These findings corroborate the quality of our dataset, and demonstrate the value of high-quality machine-generated data in improving performance of NMT models.
• 摘要：神经机器翻译 (NMT) 领域的最新研究表明，对高质量机器生成数据进行训练的效果可以优于对人工生成数据进行训练。这项工作伴随着首次发布的 LLM 生成、MBR 解码和 QE 重新排序的数据集，其中包含句子级和多句子示例。我们进行了大量实验，以证明我们的数据集在下游对 NMT 模型性能的影响方面质量。我们发现，从头开始对我们的（机器生成的）数据集进行训练的效果优于对（网络爬取的）WMT'23 训练数据集（大 300 倍）进行训练，并且也优于对 WMT'23 训练数据集的顶级质量子集进行训练。我们还发现，通过微调生成此数据集的 LLM 进行自我提炼的效果优于 LLM 强大的少量样本基线。这些发现证实了我们数据集的质量，并证明了高质量机器生成数据在提高 NMT 模型性能方面的价值。

Title: AquilaMoE: Efficient Training for MoE Models with Scale-Up and Scale-Out Strategies

• Authors: Bo-Wen Zhang, Liangdong Wang, Ye Yuan, Jijie Li, Shuhao Gu, Mengdi Zhao, Xinya Wu, Guang Liu, Chengwei Wu, Hanyu Zhao, Li Du, Yiming Ju, Quanyue Ma, Yulong Ao, Yingli Zhao, Songhe Zhu, Zhou Cao, Dong Liang, Yonghua Lin, Ming Zhang, Shunfei Wang, Yanxin Zhou, Min Ye, Xuekai Chen, Xinyang Yu, Xiangjun Huang, Jian Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06567
• Pdf URL: https://arxiv.org/pdf/2408.06567
• Copy Paste: [[2408.06567]] AquilaMoE: Efficient Training for MoE Models with Scale-Up and Scale-Out Strategies(https://arxiv.org/abs/2408.06567)
• Keywords: language model, llm
• Abstract: In recent years, with the rapid application of large language models across various fields, the scale of these models has gradually increased, and the resources required for their pre-training have grown exponentially. Training an LLM from scratch will cost a lot of computation resources while scaling up from a smaller model is a more efficient approach and has thus attracted significant attention. In this paper, we present AquilaMoE, a cutting-edge bilingual 8*16B Mixture of Experts (MoE) language model that has 8 experts with 16 billion parameters each and is developed using an innovative training methodology called EfficientScale. This approach optimizes performance while minimizing data requirements through a two-stage process. The first stage, termed Scale-Up, initializes the larger model with weights from a pre-trained smaller model, enabling substantial knowledge transfer and continuous pretraining with significantly less data. The second stage, Scale-Out, uses a pre-trained dense model to initialize the MoE experts, further enhancing knowledge transfer and performance. Extensive validation experiments on 1.8B and 7B models compared various initialization schemes, achieving models that maintain and reduce loss during continuous pretraining. Utilizing the optimal scheme, we successfully trained a 16B model and subsequently the 8*16B AquilaMoE model, demonstrating significant improvements in performance and training efficiency.
• 摘要：近年来，随着大型语言模型在各个领域的快速应用，这些模型的规模逐渐增大，其预训练所需的资源也呈指数级增长。从零开始训练 LLM 将耗费大量的计算资源，而从较小的模型向上扩展是一种更有效的方法，因此引起了广泛关注。在本文中，我们提出了 AquilaMoE，这是一种前沿的双语 8*16B 混合专家 (MoE) 语言模型，拥有 8 位专家，每位专家有 160 亿个参数，并使用一种名为 EfficientScale 的创新训练方法开发。该方法通过两阶段过程优化性能并最小化数据需求。第一阶段称为 Scale-Up，使用预训练的小模型中的权重初始化较大的模型，从而能够以明显更少的数据实现大量的知识迁移和持续的预训练。第二阶段称为 Scale-Out，使用预训练的密集模型初始化 MoE 专家，进一步增强知识迁移和性能。在 1.8B 和 7B 模型上进行了大量的验证实验，比较了各种初始化方案，实现了在持续预训练过程中保持并减少损失的模型。利用最优方案，我们成功训练了 16B 模型，随后又训练了 8*16B AquilaMoE 模型，性能和训练效率均有显著提升。

Title: Social Debiasing for Fair Multi-modal LLMs

• Authors: Harry Cheng, Yangyang Guo, Qingpei Guo, Ming Yang, Tian Gan, Liqiang Nie
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06569
• Pdf URL: https://arxiv.org/pdf/2408.06569
• Copy Paste: [[2408.06569]] Social Debiasing for Fair Multi-modal LLMs(https://arxiv.org/abs/2408.06569)
• Keywords: language model, llm
• Abstract: Multi-modal Large Language Models (MLLMs) have advanced significantly, offering powerful vision-language understanding capabilities. However, these models often inherit severe social biases from their training datasets, leading to unfair predictions based on attributes like race and gender. This paper addresses the issue of social biases in MLLMs by i) Introducing a comprehensive Counterfactual dataset with Multiple Social Concepts (CMSC), which provides a more diverse and extensive training set compared to existing datasets. ii) Proposing an Anti-Stereotype Debiasing strategy (ASD). Our method works by revisiting the MLLM training process, rescaling the autoregressive loss function, and improving data sampling methods to counteract biases. Through extensive experiments on various MLLMs, our CMSC dataset and ASD method demonstrate a significant reduction in social biases while maintaining the models' original performance.
• 摘要：多模态大型语言模型 (MLLM) 取得了长足进步，提供了强大的视觉语言理解能力。然而，这些模型往往从其训练数据集中继承了严重的社会偏见，导致基于种族和性别等属性的不公平预测。本文通过以下方式解决了 MLLM 中的社会偏见问题：i) 引入一个全面的具有多种社会概念的反事实数据集 (CMSC)，与现有数据集相比，它提供了更加多样化和广泛的训练集；ii) 提出一种反刻板印象去偏见策略 (ASD)。我们的方法通过重新审视 MLLM 训练过程、重新调整自回归损失函数和改进数据采样方法来抵消偏见。通过对各种 MLLM 进行大量实验，我们的 CMSC 数据集和 ASD 方法在保持模型原有性能的同时，显著减少了社会偏见。

Title: SparkRA: A Retrieval-Augmented Knowledge Service System Based on Spark Large Language Model

• Authors: Dayong Wu, Jiaqi Li, Baoxin Wang, Honghong Zhao, Siyuan Xue, Yanjie Yang, Zhijun Chang, Rui Zhang, Li Qian, Bo Wang, Shijin Wang, Zhixiong Zhang, Guoping Hu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06574
• Pdf URL: https://arxiv.org/pdf/2408.06574
• Copy Paste: [[2408.06574]] SparkRA: A Retrieval-Augmented Knowledge Service System Based on Spark Large Language Model(https://arxiv.org/abs/2408.06574)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have shown remarkable achievements across various language this http URL enhance the performance of LLMs in scientific literature services, we developed the scientific literature LLM (SciLit-LLM) through pre-training and supervised fine-tuning on scientific literature, building upon the iFLYTEK Spark LLM. Furthermore, we present a knowledge service system Spark Research Assistant (SparkRA) based on our SciLit-LLM. SparkRA is accessible online and provides three primary functions: literature investigation, paper reading, and academic writing. As of July 30, 2024, SparkRA has garnered over 50,000 registered users, with a total usage count exceeding 1.3 million.
• 摘要：大型语言模型（LLM）在各种语言领域都取得了显著的成就，为了提高LLM在科学文献服务中的表现，我们在科大讯飞Spark LLM的基础上，通过对科学文献进行预训练和监督微调，开发了科学文献LLM（SciLit-LLM）。此外，我们还基于我们的SciLit-LLM提出了一个知识服务系统Spark Research Assistant（SparkRA）。SparkRA可在线访问，提供三大主要功能：文献调查、论文阅读和学术写作。截至2024年7月30日，SparkRA已拥有超过5万名注册用户，总使用次数超过130万次。

Title: OpenEP: Open-Ended Future Event Prediction

• Authors: Yong Guan, Hao Peng, Xiaozhi Wang, Lei Hou, Juanzi Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06578
• Pdf URL: https://arxiv.org/pdf/2408.06578
• Copy Paste: [[2408.06578]] OpenEP: Open-Ended Future Event Prediction(https://arxiv.org/abs/2408.06578)
• Keywords: llm
• Abstract: Future event prediction (FEP) is a long-standing and crucial task in the world, as understanding the evolution of events enables early risk identification, informed decision-making, and strategic planning. Existing work typically treats event prediction as classification tasks and confines the outcomes of future events to a fixed scope, such as yes/no questions, candidate set, and taxonomy, which is difficult to include all possible outcomes of future events. In this paper, we introduce OpenEP (an Open-Ended Future Event Prediction task), which generates flexible and diverse predictions aligned with real-world scenarios. This is mainly reflected in two aspects: firstly, the predictive questions are diverse, covering different stages of event development and perspectives; secondly, the outcomes are flexible, without constraints on scope or format. To facilitate the study of this task, we construct OpenEPBench, an open-ended future event prediction dataset. For question construction, we pose questions from seven perspectives, including location, time, event development, event outcome, event impact, event response, and other, to facilitate an in-depth analysis and understanding of the comprehensive evolution of events. For outcome construction, we collect free-form text containing the outcomes as ground truth to provide semantically complete and detail-enriched outcomes. Furthermore, we propose StkFEP, a stakeholder-enhanced future event prediction framework, that incorporates event characteristics for open-ended settings. Our method extracts stakeholders involved in events to extend questions to gather diverse information. We also collect historically events that are relevant and similar to the question to reveal potential evolutionary patterns. Experiment results indicate that accurately predicting future events in open-ended settings is challenging for existing LLMs.
• 摘要：未来事件预测（FEP）是世界上一项长期存在且至关重要的任务，因为了解事件的演变有助于及早识别风险、做出明智的决策和制定战略规划。现有工作通常将事件预测视为分类任务，并将未来事件的结果限制在固定范围内，例如是/否问题、候选集和分类法，这很难涵盖未来事件的所有可能结果。在本文中，我们引入了 OpenEP（开放式未来事件预测任务），它可以生成与现实场景相符的灵活多样的预测。这主要体现在两个方面：首先，预测问题多样，涵盖事件发展的不同阶段和视角；其次，结果灵活，不受范围或格式的限制。为了方便研究该任务，我们构建了开放式未来事件预测数据集 OpenEPBench。在问题构建方面，我们从地点、时间、事件发展、事件结果、事件影响、事件响应和其他七个角度提出问题，以便于深入分析和理解事件的全面演变。对于结果构建，我们收集包含结果的自由格式文本作为基本事实，以提供语义完整且细节丰富的结果。此外，我们提出了 StkFEP，这是一个利益相关者增强的未来事件预测框架，它结合了开放式环境中的事件特征。我们的方法提取事件中涉及的利益相关者来扩展问题以收集各种信息。我们还收集与问题相关且相似的历史事件，以揭示潜在的进化模式。实验结果表明，准确预测开放式环境中的未来事件对于现有的 LLM 来说具有挑战性。

Title: Biomedical Event Extraction via Structure-aware Generation

• Authors: Haohan Yuan, Siu Cheung Hui, Haopeng Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06583
• Pdf URL: https://arxiv.org/pdf/2408.06583
• Copy Paste: [[2408.06583]] Biomedical Event Extraction via Structure-aware Generation(https://arxiv.org/abs/2408.06583)
• Keywords: language model, llm, prompt
• Abstract: Biomedical Event Extraction (BEE) is a critical task that involves modeling complex relationships between fine-grained entities in biomedical text data. However, most existing BEE models rely on classification methods that neglect the label semantics and argument dependency structure within the data. To address these limitations, we propose GenBEE, a generative model enhanced with a structure-aware prefix for biomedical event extraction. GenBEE constructs event prompts that leverage knowledge distilled from large language models (LLMs), thereby incorporating both label semantics and argument dependency relationships. Additionally, GenBEE introduces a structural prefix learning module that generates structure-aware prefixes with structural prompts, enriching the generation process with structural features. Extensive experiments on three benchmark datasets demonstrate the effectiveness of GenBEE and it achieves state-of-the-art performance on the MLEE and GE11 datasets. Furthermore, our analysis shows that the structural prefixes effectively bridge the gap between structural prompts and the representation space of generative models, enabling better integration of event structural information.
• 摘要：生物医学事件提取 (BEE) 是一项关键任务，涉及对生物医学文本数据中细粒度实体之间的复杂关系进行建模。然而，大多数现有的 BEE 模型都依赖于忽略数据中的标签语义和参数依赖结构的分类方法。为了解决这些限制，我们提出了 GenBEE，这是一种增强了结构感知前缀的生成模型，用于生物医学事件提取。GenBEE 构建事件提示，利用从大型语言模型 (LLM) 中提取的知识，从而结合标签语义和参数依赖关系。此外，GenBEE 引入了一个结构前缀学习模块，该模块使用结构提示生成结构感知前缀，通过结构特征丰富生成过程。在三个基准数据集上进行的大量实验证明了 GenBEE 的有效性，并且它在 MLEE 和 GE11 数据集上实现了最先进的性能。此外，我们的分析表明，结构前缀有效地弥合了结构提示和生成模型的表示空间之间的差距，从而能够更好地整合事件结构信息。

Title: A Perspective on Large Language Models, Intelligent Machines, and Knowledge Acquisition

• Authors: Vladimir Cherkassky, Eng Hock Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06598
• Pdf URL: https://arxiv.org/pdf/2408.06598
• Copy Paste: [[2408.06598]] A Perspective on Large Language Models, Intelligent Machines, and Knowledge Acquisition(https://arxiv.org/abs/2408.06598)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) are known for their remarkable ability to generate synthesized 'knowledge', such as text documents, music, images, etc. However, there is a huge gap between LLM's and human capabilities for understanding abstract concepts and reasoning. We discuss these issues in a larger philosophical context of human knowledge acquisition and the Turing test. In addition, we illustrate the limitations of LLMs by analyzing GPT-4 responses to questions ranging from science and math to common sense reasoning. These examples show that GPT-4 can often imitate human reasoning, even though it lacks understanding. However, LLM responses are synthesized from a large LLM model trained on all available data. In contrast, human understanding is based on a small number of abstract concepts. Based on this distinction, we discuss the impact of LLMs on acquisition of human knowledge and education.
• 摘要：大型语言模型 (LLM) 以其生成合成“知识”的卓越能力而闻名，例如文本文档、音乐、图像等。然而，LLM 与人类理解抽象概念和推理的能力之间存在巨大差距。我们在人类知识获取和图灵测试的更大哲学背景下讨论这些问题。此外，我们通过分析 GPT-4 对从科学和数学到常识推理等问题的回答来说明 LLM 的局限性。这些例子表明，GPT-4 通常可以模仿人类的推理，即使它缺乏理解。然而，LLM 响应是从对所有可用数据进行训练的大型 LLM 模型合成的。相比之下，人类的理解基于少数抽象概念。基于这种区别，我们讨论了 LLM 对人类知识和教育获取的影响。

Title: IFShip: A Large Vision-Language Model for Interpretable Fine-grained Ship Classification via Domain Knowledge-Enhanced Instruction Tuning

• Authors: Mingning Guo, Mengwei Wu, Yuxiang Shen, Haifeng Li, Chao Tao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06631
• Pdf URL: https://arxiv.org/pdf/2408.06631
• Copy Paste: [[2408.06631]] IFShip: A Large Vision-Language Model for Interpretable Fine-grained Ship Classification via Domain Knowledge-Enhanced Instruction Tuning(https://arxiv.org/abs/2408.06631)
• Keywords: language model, gpt, prompt, chat, chain-of-thought
• Abstract: End-to-end interpretation is currently the prevailing paradigm for remote sensing fine-grained ship classification (RS-FGSC) task. However, its inference process is uninterpretable, leading to criticism as a black box model. To address this issue, we propose a large vision-language model (LVLM) named IFShip for interpretable fine-grained ship classification. Unlike traditional methods, IFShip excels in interpretability by accurately conveying the reasoning process of FGSC in natural language. Specifically, we first design a domain knowledge-enhanced Chain-of-Thought (COT) prompt generation mechanism. This mechanism is used to semi-automatically construct a task-specific instruction-following dataset named TITANIC-FGS, which emulates human-like logical decision-making. We then train the IFShip model using task instructions tuned with the TITANIC-FGS dataset. Building on IFShip, we develop an FGSC visual chatbot that redefines the FGSC problem as a step-by-step reasoning task and conveys the reasoning process in natural language. Experimental results reveal that the proposed method surpasses state-of-the-art FGSC algorithms in both classification interpretability and accuracy. Moreover, compared to LVLMs like LLaVA and MiniGPT-4, our approach demonstrates superior expertise in the FGSC task. It provides an accurate chain of reasoning when fine-grained ship types are recognizable to the human eye and offers interpretable explanations when they are not.
• 摘要：端到端解释是目前遥感细粒度船舶分类 (RS-FGSC) 任务的主流范式。然而，它的推理过程是不可解释的，导致它作为一种黑箱模型受到批评。为了解决这个问题，我们提出了一个名为 IFShip 的大型视觉语言模型 (LVLM)，用于可解释的细粒度船舶分类。与传统方法不同，IFShip 通过用自然语言准确地传达 FGSC 的推理过程，在可解释性方面表现出色。具体而言，我们首先设计了一种领域知识增强的思路链 (COT) 提示生成机制。该机制用于半自动构建一个名为 TITANIC-FGS 的任务特定指令跟踪数据集，该数据集模拟类似人类的逻辑决策。然后，我们使用针对 TITANIC-FGS 数据集调整的任务指令来训练 IFShip 模型。基于 IFShip，我们开发了一个 FGSC 可视化聊天机器人，将 FGSC 问题重新定义为分步推理任务，并以自然语言传达推理过程。实验结果表明，所提出的方法在分类可解释性和准确性方面都超越了最先进的 FGSC 算法。此外，与 LLaVA 和 MiniGPT-4 等 LVLM 相比，我们的方法在 FGSC 任务中表现出卓越的专业性。当人眼可以识别细粒度的船舶类型时，它可以提供准确的推理链，而当人眼无法识别时，它可以提供可解释的解释。

Title: Amuro & Char: Analyzing the Relationship between Pre-Training and Fine-Tuning of Large Language Models

• Authors: Kaiser Sun, Mark Dredze
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06663
• Pdf URL: https://arxiv.org/pdf/2408.06663
• Copy Paste: [[2408.06663]] Amuro & Char: Analyzing the Relationship between Pre-Training and Fine-Tuning of Large Language Models(https://arxiv.org/abs/2408.06663)
• Keywords: language model, prompt
• Abstract: The development of large language models leads to the formation of a pre-train-then-align paradigm, in which the model is typically pre-trained on a large text corpus and undergoes a tuning stage to align the model with human preference or downstream tasks. In this work, we investigate the relationship between pre-training and fine-tuning by fine-tuning multiple intermediate pre-trained model checkpoints. Our results on 18 datasets suggest that i) continual pre-training improves the model in a latent way that unveils after fine-tuning; ii) with extra fine-tuning, the datasets that the model does not demonstrate capability gain much more than those that the model performs well during the pre-training stage; iii) although model benefits significantly through supervised fine-tuning, it may forget previously known domain knowledge and the tasks that are not seen during fine-tuning; iv) the model resembles high sensitivity to evaluation prompts after supervised fine-tuning, but this sensitivity can be alleviated by more pre-training.
• 摘要：大型语言模型的发展导致了先预训练后对齐范式的形成，其中模型通常在大型文本语料库上进行预训练，并经过调整阶段以使模型与人类偏好或下游任务保持一致。在这项工作中，我们通过微调多个中间预训练模型检查点来研究预训练和微调之间的关系。我们在 18 个数据集上的结果表明：i）持续的预训练以一种潜在的方式改进了模型，这种改进在微调后会显着显现出来；ii）通过额外的微调，模型在预训练阶段没有表现出能力的数据集比模型表现良好的数据集获得更大的收益；iii）虽然模型通过监督微调获得了显着的好处，但它可能会忘记以前已知的领域知识和微调期间未看到的任务；iv）在监督微调后，模型对评估提示具有很高的敏感性，但这种敏感性可以通过更多的预训练来缓解。

Title: Pragmatic inference of scalar implicature by LLMs

• Authors: Ye-eun Cho, Seong mook Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06673
• Pdf URL: https://arxiv.org/pdf/2408.06673
• Copy Paste: [[2408.06673]] Pragmatic inference of scalar implicature by LLMs(https://arxiv.org/abs/2408.06673)
• Keywords: language model, gpt, llm
• Abstract: This study investigates how Large Language Models (LLMs), particularly BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019), engage in pragmatic inference of scalar implicature, such as some. Two sets of experiments were conducted using cosine similarity and next sentence/token prediction as experimental methods. The results in experiment 1 showed that, both models interpret some as pragmatic implicature not all in the absence of context, aligning with human language processing. In experiment 2, in which Question Under Discussion (QUD) was presented as a contextual cue, BERT showed consistent performance regardless of types of QUDs, while GPT-2 encountered processing difficulties since a certain type of QUD required pragmatic inference for implicature. The findings revealed that, in terms of theoretical approaches, BERT inherently incorporates pragmatic implicature not all within the term some, adhering to Default model (Levinson, 2000). In contrast, GPT-2 seems to encounter processing difficulties in inferring pragmatic implicature within context, consistent with Context-driven model (Sperber and Wilson, 2002).
• 摘要：本研究探讨了大型语言模型 (LLM)，尤其是 BERT (Devlin 等人，2019) 和 GPT-2 (Radford 等人，2019) 如何进行标量含义（例如 some）的语用推理。以余弦相似度和下一句/标记预测为实验方法，进行了两组实验。实验 1 的结果表明，在没有上下文的情况下，两个模型都将 some 解释为语用含义（不是全部），与人类语言处理一致。在实验 2 中，将讨论中的问题 (QUD) 作为上下文线索呈现，无论 QUD 类型如何，BERT 都表现出一致的表现，而 GPT-2 遇到了处理困难，因为某种类型的 QUD 需要对含义进行语用推理。研究结果表明，在理论方法方面，BERT 固有地将语用含义（不是全部）包含在术语 some 中，遵循默认模型 (Levinson，2000)。相比之下，GPT-2 似乎在推断上下文中的语用含义时遇到了处理困难，这与语境驱动模型（Sperber and Wilson，2002）一致。

Title: Unlock the Power of Frozen LLMs in Knowledge Graph Completion

• Authors: Bo Xue, Yi Xu, Yunchong Song, Yiming Pang, Yuyang Ren, Jiaxin Ding, Luoyi Fu, Xinbing Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06787
• Pdf URL: https://arxiv.org/pdf/2408.06787
• Copy Paste: [[2408.06787]] Unlock the Power of Frozen LLMs in Knowledge Graph Completion(https://arxiv.org/abs/2408.06787)
• Keywords: language model, llm, prompt
• Abstract: Classical knowledge graph completion (KGC) methods rely solely on structural information, struggling with the inherent sparsity of knowledge graphs (KGs). Large Language Models (LLMs) learn extensive knowledge from large corpora with powerful context modeling, which is ideal for mitigating the limitations of previous methods. Directly fine-tuning LLMs offers great capability but comes at the cost of huge time and memory consumption, while utilizing frozen LLMs yields suboptimal results. In this work, we aim to leverage LLMs for KGC effectively and efficiently. We capture the context-aware hidden states of knowledge triples by employing prompts to stimulate the intermediate layers of LLMs. We then train a data-efficient classifier on these hidden states to harness the inherent capabilities of frozen LLMs in KGC. We also generate entity descriptions with subgraph sampling on KGs, reducing the ambiguity of triplets and enriching the knowledge representation. Extensive experiments on standard benchmarks showcase the efficiency and effectiveness of our approach. We outperform classical KGC methods on most datasets and match the performance of fine-tuned LLMs. Additionally, compared to fine-tuned LLMs, we boost GPU memory efficiency by \textbf{$188\times$} and speed up training+inference by \textbf{$13.48\times$}.
• 摘要：经典的知识图谱补全 (KGC) 方法仅依赖于结构信息，难以应对知识图谱 (KG) 固有的稀疏性。大型语言模型 (LLM) 可通过强大的上下文建模从大型语料库中学习大量知识，这对于缓解以前方法的局限性非常理想。直接微调 LLM 提供了强大的功能，但代价是大量时间和内存消耗，而使用冻结的 LLM 会产生次优结果。在这项工作中，我们旨在有效且高效地利用 LLM 进行 KGC。我们通过使用提示来刺激 LLM 的中间层，从而捕获知识三元组的上下文感知隐藏状态。然后，我们在这些隐藏状态下训练一个数据高效的分类器，以利用 KGC 中冻结 LLM 的固有功能。我们还使用 KG 上的子图采样生成实体描述，从而减少三元组的歧义性并丰富知识表示。在标准基准上进行的大量实验展示了我们方法的效率和有效性。我们在大多数数据集上的表现都优于经典的 KGC 方法，并且与经过微调的 LLM 的性能相当。此外，与经过微调的 LLM 相比，我们将 GPU 内存效率提高了 \textbf{$188\times$}，并将训练+推理速度提高了 \textbf{$13.48\times$}。

Title: Layerwise Recurrent Router for Mixture-of-Experts

• Authors: Zihan Qiu, Zeyu Huang, Shuang Cheng, Yizhi Zhou, Zili Wang, Ivan Titov, Jie Fu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06793
• Pdf URL: https://arxiv.org/pdf/2408.06793
• Copy Paste: [[2408.06793]] Layerwise Recurrent Router for Mixture-of-Experts(https://arxiv.org/abs/2408.06793)
• Keywords: language model, llm
• Abstract: The scaling of large language models (LLMs) has revolutionized their capabilities in various tasks, yet this growth must be matched with efficient computational strategies. The Mixture-of-Experts (MoE) architecture stands out for its ability to scale model size without significantly increasing training costs. Despite their advantages, current MoE models often display parameter inefficiency. For instance, a pre-trained MoE-based LLM with 52 billion parameters might perform comparably to a standard model with 6.7 billion parameters. Being a crucial part of MoE, current routers in different layers independently assign tokens without leveraging historical routing information, potentially leading to suboptimal token-expert combinations and the parameter inefficiency problem. To alleviate this issue, we introduce the Layerwise Recurrent Router for Mixture-of-Experts (RMoE). RMoE leverages a Gated Recurrent Unit (GRU) to establish dependencies between routing decisions across consecutive layers. Such layerwise recurrence can be efficiently parallelly computed for input tokens and introduces negotiable costs. Our extensive empirical evaluations demonstrate that RMoE-based language models consistently outperform a spectrum of baseline models. Furthermore, RMoE integrates a novel computation stage orthogonal to existing methods, allowing seamless compatibility with other MoE architectures. Our analyses attribute RMoE's gains to its effective cross-layer information sharing, which also improves expert selection and diversity. Our code is at this https URL
• 摘要：大型语言模型 (LLM) 的扩展彻底改变了它们在各种任务中的能力，但这种增长必须与高效的计算策略相匹配。混合专家 (MoE) 架构因其能够在不显著增加训练成本的情况下扩展模型大小而脱颖而出。尽管具有优势，但当前的 MoE 模型通常表现出参数效率低下。例如，具有 520 亿个参数的预训练的基于 MoE 的 LLM 的性能可能与具有 67 亿个参数的标准模型相当。作为 MoE 的关键部分，当前不同层的路由器独立分配令牌而不利用历史路由信息，这可能导致次优的令牌专家组合和参数效率低下问题。为了缓解这个问题，我们引入了混合专家 (RMoE) 的分层循环路由器。RMoE 利用门控循环单元 (GRU) 在连续层之间建立路由决策之间的依赖关系。这种分层循环可以有效地并行计算输入令牌并引入可协商的成本。我们广泛的实证评估表明，基于 RMoE 的语言模型始终优于一系列基线模型。此外，RMoE 集成了一个与现有方法正交的新型计算阶段，从而可以与其他 MoE 架构无缝兼容。我们的分析将 RMoE 的优势归因于其有效的跨层信息共享，这也改善了专家选择和多样性。我们的代码位于此 https URL

Title: LoRA$^2$ : Multi-Scale Low-Rank Approximations for Fine-Tuning Large Language Models

• Authors: Jia-Chen Zhang, Yu-Jie Xiong, He-Xi Qiu, Dong-Hai Zhu, Chun-Ming Xia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06854
• Pdf URL: https://arxiv.org/pdf/2408.06854
• Copy Paste: [[2408.06854]] LoRA$^2$ : Multi-Scale Low-Rank Approximations for Fine-Tuning Large Language Models(https://arxiv.org/abs/2408.06854)
• Keywords: language model, llm
• Abstract: Fine-tuning large language models (LLMs) with high parameter efficiency for downstream tasks has become a new paradigm. Low-Rank Adaptation (LoRA) significantly reduces the number of trainable parameters for fine-tuning. Although it has demonstrated commendable performance, updating parameters within a single scale may not be the optimal choice for complex downstream this http URL this paper, we extend the LoRA to multiple scales, dubbed as LoRA$^2$. We first combine orthogonal projection theory to train a set of LoRAs in two mutually orthogonal planes. Then, we improve the importance score algorithm, which reduce parameter sensitivity score calculations by approximately 98.5\%. By pruning singular values with lower importance scores, thereby enhancing adaptability to various downstream tasks. Extensive experiments are conducted on two widely used pre-trained models to validate the effectiveness of LoRA$^2$. Results show that it significantly reduces the number of trainable parameters to just 0.72\% compared to full fine-tuning, while still delivering highly impressive performance. Even when the parameters are further reduced to 0.17M, it still achieves comparable results to the baseline with 8 times more parameters. Our code is available here: https://anonymous.4open.science/r/LoRA-2-5B4C
• 摘要：对具有高参数效率的大型语言模型 (LLM) 进行下游任务的微调已成为一种新范式。低秩自适应 (LoRA) 显著减少了微调所需的可训练参数数量。尽管它已经表现出了令人称赞的性能，但在单一尺度内更新参数可能不是复杂下游任务的最佳选择。在本文中，我们将 LoRA 扩展到多个尺度，称为 LoRA$^2$。我们首先结合正交投影理论在两个相互正交的平面上训练一组 LoRA。然后，我们改进了重要性得分算法，将参数敏感度得分计算减少了约 98.5\%。通过修剪重要性得分较低的奇异值，从而增强了对各种下游任务的适应性。在两个广泛使用的预训练模型上进行了广泛的实验，以验证 LoRA$^2$ 的有效性。结果表明，与完全微调相比，它将可训练参数的数量显著减少到仅 0.72\%，同时仍然提供了令人印象深刻的性能。即使参数进一步减少到 0.17M，它仍然可以实现与基线相当的结果，但参数多 8 倍。我们的代码可在此处获取：https://anonymous.4open.science/r/LoRA-2-5B4C

Title: Leveraging Language Models for Emotion and Behavior Analysis in Education

• Authors: Kaito Tanaka, Benjamin Tan, Brian Wong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06874
• Pdf URL: https://arxiv.org/pdf/2408.06874
• Copy Paste: [[2408.06874]] Leveraging Language Models for Emotion and Behavior Analysis in Education(https://arxiv.org/abs/2408.06874)
• Keywords: language model, gpt, llm, prompt, chat, chain-of-thought
• Abstract: The analysis of students' emotions and behaviors is crucial for enhancing learning outcomes and personalizing educational experiences. Traditional methods often rely on intrusive visual and physiological data collection, posing privacy concerns and scalability issues. This paper proposes a novel method leveraging large language models (LLMs) and prompt engineering to analyze textual data from students. Our approach utilizes tailored prompts to guide LLMs in detecting emotional and engagement states, providing a non-intrusive and scalable solution. We conducted experiments using Qwen, ChatGPT, Claude2, and GPT-4, comparing our method against baseline models and chain-of-thought (CoT) prompting. Results demonstrate that our method significantly outperforms the baselines in both accuracy and contextual understanding. This study highlights the potential of LLMs combined with prompt engineering to offer practical and effective tools for educational emotion and behavior analysis.
• 摘要：分析学生的情绪和行为对于提高学习成果和个性化教育体验至关重要。传统方法通常依赖于侵入性视觉和生理数据收集，这会带来隐私问题和可扩展性问题。本文提出了一种利用大型语言模型 (LLM) 和提示工程来分析学生文本数据的新方法。我们的方法利用定制的提示来指导 LLM 检测情绪和参与状态，从而提供一种非侵入性和可扩展的解决方案。我们使用 Qwen、ChatGPT、Claude2 和 GPT-4 进行了实验，将我们的方法与基线模型和思路链 (CoT) 提示进行了比较。结果表明，我们的方法在准确性和上下文理解方面都明显优于基线。这项研究强调了 LLM 与提示工程相结合的潜力，可以为教育情感和行为分析提供实用有效的工具。

Title: Re-TASK: Revisiting LLM Tasks from Capability, Skill, and Knowledge Perspectives

• Authors: Zhihu Wang, Shiwan Zhao, Yu Wang, Heyuan Huang, Jiaxin Shi, Sitao Xie, Zhixing Wang, Yubo Zhang, Hongyan Li, Junchi Yan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06904
• Pdf URL: https://arxiv.org/pdf/2408.06904
• Copy Paste: [[2408.06904]] Re-TASK: Revisiting LLM Tasks from Capability, Skill, and Knowledge Perspectives(https://arxiv.org/abs/2408.06904)
• Keywords: language model, llm, prompt
• Abstract: As large language models (LLMs) continue to scale, their enhanced performance often proves insufficient for solving domain-specific tasks. Systematically analyzing their failures and effectively enhancing their performance remain significant challenges. This paper introduces the Re-TASK framework, a novel theoretical model that Revisits LLM Tasks from cApability, Skill, Knowledge perspectives, guided by the principles of Bloom's Taxonomy and Knowledge Space Theory. The Re-TASK framework provides a systematic methodology to deepen our understanding, evaluation, and enhancement of LLMs for domain-specific tasks. It explores the interplay among an LLM's capabilities, the knowledge it processes, and the skills it applies, elucidating how these elements are interconnected and impact task performance. Our application of the Re-TASK framework reveals that many failures in domain-specific tasks can be attributed to insufficient knowledge or inadequate skill adaptation. With this insight, we propose structured strategies for enhancing LLMs through targeted knowledge injection and skill adaptation. Specifically, we identify key capability items associated with tasks and employ a deliberately designed prompting strategy to enhance task performance, thereby reducing the need for extensive fine-tuning. Alternatively, we fine-tune the LLM using capability-specific instructions, further validating the efficacy of our framework. Experimental results confirm the framework's effectiveness, demonstrating substantial improvements in both the performance and applicability of LLMs.
• 摘要：随着大型语言模型 (LLM) 的规模不断扩大，其增强的性能通常不足以解决特定领域的任务。系统地分析它们的失败并有效地提高它们的性能仍然是重大挑战。本文介绍了 Re-TASK 框架，这是一种新颖的理论模型，它以布鲁姆分类学和知识空间理论的原则为指导，从能力、技能、知识的角度重新审视 LLM 任务。Re-TASK 框架提供了一种系统的方法，可以加深我们对特定领域任务的 LLM 的理解、评估和增强。它探讨了 LLM 的能力、它处理的知识和它应用的技能之间的相互作用，阐明了这些元素如何相互关联并影响任务性能。我们对 Re-TASK 框架的应用表明，许多特定领域任务的失败可以归因于知识不足或技能适应不足。基于这一见解，我们提出了通过有针对性的知识注入和技能适应来增强 LLM 的结构化策略。具体来说，我们识别与任务相关的关键能力项目，并采用精心设计的提示策略来提高任务性能，从而减少大量微调的需要。或者，我们使用特定于能力的指令对 LLM 进行微调，进一步验证了我们框架的有效性。实验结果证实了该框架的有效性，表明 LLM 的性能和适用性都有了显着的改进。

Title: Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personas

• Authors: Louis Kwok, Michal Bravansky, Lewis D. Griffin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06929
• Pdf URL: https://arxiv.org/pdf/2408.06929
• Copy Paste: [[2408.06929]] Evaluating Cultural Adaptability of a Large Language Model via Simulation of Synthetic Personas(https://arxiv.org/abs/2408.06929)
• Keywords: language model, gpt, llm, prompt
• Abstract: The success of Large Language Models (LLMs) in multicultural environments hinges on their ability to understand users' diverse cultural backgrounds. We measure this capability by having an LLM simulate human profiles representing various nationalities within the scope of a questionnaire-style psychological experiment. Specifically, we employ GPT-3.5 to reproduce reactions to persuasive news articles of 7,286 participants from 15 countries; comparing the results with a dataset of real participants sharing the same demographic traits. Our analysis shows that specifying a person's country of residence improves GPT-3.5's alignment with their responses. In contrast, using native language prompting introduces shifts that significantly reduce overall alignment, with some languages particularly impairing performance. These findings suggest that while direct nationality information enhances the model's cultural adaptability, native language cues do not reliably improve simulation fidelity and can detract from the model's effectiveness.
• 摘要：大型语言模型 (LLM) 在多元文化环境中的成功取决于它们理解用户不同文化背景的能力。我们通过让 LLM 在问卷式心理实验范围内模拟代表不同国籍的人类资料来衡量这种能力。具体来说，我们使用 GPT-3.5 来重现来自 15 个国家/地区的 7,286 名参与者对有说服力的新闻文章的反应；将结果与具有相同人口统计特征的真实参与者的数据集进行比较。我们的分析表明，指定一个人的居住国家/地区可以提高 GPT-3.5 与其回答的一致性。相比之下，使用母语提示会引入变化，从而显着降低整体一致性，某些语言尤其会损害性能。这些发现表明，虽然直接的国籍信息可以增强模型的文化适应性，但母语提示并不能可靠地提高模拟保真度，并且会降低模型的有效性。

Title: The advantages of context specific language models: the case of the Erasmian Language Model

• Authors: João Gonçalves, Nick Jelicic, Michele Murgia, Evert Stamhuis
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06931
• Pdf URL: https://arxiv.org/pdf/2408.06931
• Copy Paste: [[2408.06931]] The advantages of context specific language models: the case of the Erasmian Language Model(https://arxiv.org/abs/2408.06931)
• Keywords: language model, gpt
• Abstract: The current trend to improve language model performance seems to be based on scaling up with the number of parameters (e.g. the state of the art GPT4 model has approximately 1.7 trillion parameters) or the amount of training data fed into the model. However this comes at significant costs in terms of computational resources and energy costs that compromise the sustainability of AI solutions, as well as risk relating to privacy and misuse. In this paper we present the Erasmian Language Model (ELM) a small context specific, 900 million parameter model, pre-trained and fine-tuned by and for Erasmus University Rotterdam. We show how the model performs adequately in a classroom context for essay writing, and how it achieves superior performance in subjects that are part of its context. This has implications for a wide range of institutions and organizations, showing that context specific language models may be a viable alternative for resource constrained, privacy sensitive use cases.
• 摘要：目前，提高语言模型性能的趋势似乎是基于增加参数数量（例如，最先进的 GPT4 模型大约有 1.7 万亿个参数）或输入模型的训练数据量。然而，这会在计算资源和能源成本方面付出巨大代价，从而损害 AI 解决方案的可持续性，以及与隐私和滥用相关的风险。在本文中，我们介绍了 Erasmian 语言模型 (ELM)，这是一个小型的上下文特定的、9 亿个参数的模型，由鹿特丹伊拉斯姆斯大学预先训练和微调。我们展示了该模型在课堂环境中如何充分表现为写作论文，以及它如何在属于其上下文的科目中取得优异的表现。这对广泛的机构和组织具有影响，表明上下文特定的语言模型可能是资源受限、隐私敏感用例的可行替代方案。

Title: Generative AI for automatic topic labelling

• Authors: Diego Kozlowski, Carolina Pradier, Pierre Benz
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07003
• Pdf URL: https://arxiv.org/pdf/2408.07003
• Copy Paste: [[2408.07003]] Generative AI for automatic topic labelling(https://arxiv.org/abs/2408.07003)
• Keywords: gpt, llm
• Abstract: Topic Modeling has become a prominent tool for the study of scientific fields, as they allow for a large scale interpretation of research trends. Nevertheless, the output of these models is structured as a list of keywords which requires a manual interpretation for the labelling. This paper proposes to assess the reliability of three LLMs, namely flan, GPT-4o, and GPT-4 mini for topic labelling. Drawing on previous research leveraging BERTopic, we generate topics from a dataset of all the scientific articles (n=34,797) authored by all biology professors in Switzerland (n=465) between 2008 and 2020, as recorded in the Web of Science database. We assess the output of the three models both quantitatively and qualitatively and find that, first, both GPT models are capable of accurately and precisely label topics from the models' output keywords. Second, 3-word labels are preferable to grasp the complexity of research topics.
• 摘要：主题建模已成为研究科学领域的重要工具，因为它们可以大规模地解释研究趋势。然而，这些模型的输出结构为关键字列表，需要手动解释标签。本文提出评估三个 LLM（即 flan、GPT-4o 和 GPT-4 mini）在主题标记方面的可靠性。借鉴之前利用 BERTopic 的研究，我们从 2008 年至 2020 年期间瑞士所有生物学教授（n=465）撰写的所有科学文章（n=34,797）的数据集中生成主题，这些文章记录在 Web of Science 数据库中。我们对这三个模型的输出进行了定量和定性评估，发现首先，这两个 GPT 模型都能够准确而精确地从模型的输出关键字中标记主题。其次，3 字标签更适合掌握研究主题的复杂性。

Title: LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs

• Authors: Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07055
• Pdf URL: https://arxiv.org/pdf/2408.07055
• Copy Paste: [[2408.07055]] LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs(https://arxiv.org/abs/2408.07055)
• Keywords: language model, llm, long context, agent
• Abstract: Current long context large language models (LLMs) can process inputs up to 100,000 tokens, yet struggle to generate outputs exceeding even a modest length of 2,000 words. Through controlled experiments, we find that the model's effective generation length is inherently bounded by the sample it has seen during supervised fine-tuning (SFT). In other words, their output limitation is due to the scarcity of long-output examples in existing SFT datasets. To address this, we introduce AgentWrite, an agent-based pipeline that decomposes ultra-long generation tasks into subtasks, enabling off-the-shelf LLMs to generate coherent outputs exceeding 20,000 words. Leveraging AgentWrite, we construct LongWriter-6k, a dataset containing 6,000 SFT data with output lengths ranging from 2k to 32k words. By incorporating this dataset into model training, we successfully scale the output length of existing models to over 10,000 words while maintaining output quality. We also develop LongBench-Write, a comprehensive benchmark for evaluating ultra-long generation capabilities. Our 9B parameter model, further improved through DPO, achieves state-of-the-art performance on this benchmark, surpassing even much larger proprietary models. In general, our work demonstrates that existing long context LLM already possesses the potential for a larger output window--all you need is data with extended output during model alignment to unlock this capability. Our code & models are at: this https URL.
• 摘要：当前的长上下文大型语言模型 (LLM) 可以处理多达 100,000 个标记的输入，但很难生成超过 2,000 个单词的输出。通过受控实验，我们发现模型的有效生成长度本质上受其在监督微调 (SFT) 期间看到的样本的限制。换句话说，它们的输出限制是由于现有 SFT 数据集中长输出示例稀缺。为了解决这个问题，我们引入了 AgentWrite，这是一个基于代理的管道，它将超长生成任务分解为子任务，使现成的 LLM 能够生成超过 20,000 个单词的连贯输出。利用 AgentWrite，我们构建了 LongWriter-6k，这是一个包含 6,000 个 SFT 数据的数据集，输出长度从 2k 到 32k 个单词不等。通过将此数据集纳入模型训练，我们成功地将现有模型的输出长度扩展到 10,000 个单词以上，同时保持了输出质量。我们还开发了 LongBench-Write，这是一个用于评估超长生成能力的综合基准。我们的 9B 参数模型通过 DPO 进一步改进，在此基准上实现了最先进的性能，甚至超越了更大的专有模型。总的来说，我们的工作表明，现有的长上下文 LLM 已经具备更大输出窗口的潜力——您所需要的只是在模型对齐期间具有扩展输出的数据，以解锁此功能。我们的代码和模型位于：此 https URL。