2024-07-30

Title: Optimising Hard Prompts with Few-Shot Meta-Prompting

• Authors: Sayash Raaj Hiraou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.18920
• Pdf URL: https://arxiv.org/pdf/2407.18920
• Copy Paste: [[2407.18920]] Optimising Hard Prompts with Few-Shot Meta-Prompting(https://arxiv.org/abs/2407.18920)
• Keywords: language model, llm, prompt
• Abstract: Prompting is a flexible and adaptable way of providing instructions to a Large Language Model (LLM). Contextual prompts include context in the form of a document or dialogue along with the natural language instructions to the LLM, often constraining the LLM to restrict facts to that of the given context while complying with the instructions. Masking the context, it acts as template for prompts. In this paper, we present an iterative method to generate better templates using an LLM from an existing set of prompt templates without revealing the context to the LLM. Multiple methods of optimising prompts using the LLM itself are explored to check the effect of few shot sampling methods on iterative propagation while maintaining linguistic styles and syntax on optimisation of prompt templates, yielding a 103.87% improvement using the best performing method. Comparison of the results of multiple contextual tasks demonstrate the ability of LLMs to maintain syntax while learning to replicate linguistic styles. Additionally, the effect on the output with different methods of prompt template generation is shown.
• 摘要：提示是一种灵活且适应性强的向大型语言模型 (LLM) 提供指令的方式。上下文提示包括以文档或对话形式呈现的上下文以及对 LLM 的自然语言指令，通常会限制 LLM 在遵守指令的同时将事实限制在给定上下文中。通过屏蔽上下文，它充当提示的模板。在本文中，我们提出了一种迭代方法，使用 LLM 从现有的一组提示模板中生成更好的模板，而无需向 LLM 透露上下文。探索了使用 LLM 本身优化提示的多种方法，以检查少量样本采样方法对迭代传播的影响，同时保持语言风格和语法对提示模板的优化，使用最佳表现方法可获得 103.87% 的改进。对多个上下文任务结果的比较表明 LLM 能够在学习复制语言风格的同时保持语法。此外，还展示了使用不同提示模板生成方法对输出的影响。

Title: OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automation

• Authors: Zilong Wang, Yuedong Cui, Li Zhong, Zimin Zhang, Da Yin, Bill Yuchen Lin, Jingbo Shang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.19056
• Pdf URL: https://arxiv.org/pdf/2407.19056
• Copy Paste: [[2407.19056]] OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automation(https://arxiv.org/abs/2407.19056)
• Keywords: gpt, llm, hallucination, agent
• Abstract: Office automation significantly enhances human productivity by automatically finishing routine tasks in the workflow. Beyond the basic information extraction studied in much of the prior document AI literature, the office automation research should be extended to more realistic office tasks which require to integrate various information sources in the office system and produce outputs through a series of decision-making processes. We introduce OfficeBench, one of the first office automation benchmarks for evaluating current LLM agents' capability to address office tasks in realistic office workflows. OfficeBench requires LLM agents to perform feasible long-horizon planning, proficiently switch between applications in a timely manner, and accurately ground their actions within a large combined action space, based on the contextual demands of the workflow. Applying our customized evaluation methods on each task, we find that GPT-4 Omni achieves the highest pass rate of 47.00%, demonstrating a decent performance in handling office tasks. However, this is still far below the human performance and accuracy standards required by real-world office workflows. We further observe that most issues are related to operation redundancy and hallucinations, as well as limitations in switching between multiple applications, which may provide valuable insights for developing effective agent frameworks for office automation.
• 摘要：办公自动化通过自动完成工作流程中的日常任务，显著提高了人类的生产力。除了之前许多文档 AI 文献中研究的基本信息提取之外，办公自动化研究还应扩展到更现实的办公任务，这些任务需要整合办公系统中的各种信息源，并通过一系列决策过程产生输出。我们引入了 OfficeBench，这是首批用于评估当前 LLM 代理在现实办公工作流程中处理办公任务的能力的办公自动化基准之一。OfficeBench 要求 LLM 代理根据工作流程的上下文需求，执行可行的长期规划，及时熟练地在应用程序之间切换，并准确地在大型组合动作空间内执行其操作。将我们定制的评估方法应用于每个任务，我们发现 GPT-4 Omni 的最高通过率为 47.00%，在处理办公任务方面表现出色。然而，这仍然远远低于现实世界办公工作流程所要求的人类表现和准确性标准。我们进一步观察到，大多数问题都与操作冗余和幻觉以及在多个应用程序之间切换的限制有关，这可能为开发有效的办公自动化代理框架提供宝贵的见解。

Title: Many-Shot In-Context Learning for Molecular Inverse Design

• Authors: Saeed Moayedpour, Alejandro Corrochano-Navarro, Faryad Sahneh, Shahriar Noroozizadeh, Alexander Koetter, Jiri Vymetal, Lorenzo Kogler-Anele, Pablo Mas, Yasser Jangjou, Sizhen Li, Michael Bailey, Marc Bianciotto, Hans Matter, Christoph Grebner, Gerhard Hessler, Ziv Bar-Joseph, Sven Jager
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.19089
• Pdf URL: https://arxiv.org/pdf/2407.19089
• Copy Paste: [[2407.19089]] Many-Shot In-Context Learning for Molecular Inverse Design(https://arxiv.org/abs/2407.19089)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated great performance in few-shot In-Context Learning (ICL) for a variety of generative and discriminative chemical design tasks. The newly expanded context windows of LLMs can further improve ICL capabilities for molecular inverse design and lead optimization. To take full advantage of these capabilities we developed a new semi-supervised learning method that overcomes the lack of experimental data available for many-shot ICL. Our approach involves iterative inclusion of LLM generated molecules with high predicted performance, along with experimental data. We further integrated our method in a multi-modal LLM which allows for the interactive modification of generated molecular structures using text instructions. As we show, the new method greatly improves upon existing ICL methods for molecular design while being accessible and easy to use for scientists.
• 摘要：大型语言模型 (LLM) 在各种生成性和判别性化学设计任务的少样本上下文学习 (ICL) 中表现出色。LLM 新扩展的上下文窗口可以进一步提高 ICL 的分子逆向设计和先导优化能力。为了充分利用这些能力，我们开发了一种新的半监督学习方法，克服了多样本 ICL 缺乏实验数据的问题。我们的方法包括迭代包含具有高预测性能的 LLM 生成分子以及实验数据。我们进一步将我们的方法集成到多模 LLM 中，允许使用文本指令交互式修改生成的分子结构。正如我们所展示的，新方法大大改进了现有的分子设计 ICL 方法，同时方便科学家使用。

Title: FarSSiBERT: A Novel Transformer-based Model for Semantic Similarity Measurement of Persian Social Networks Informal Texts

• Authors: Seyed Mojtaba Sadjadi, Zeinab Rajabi, Leila Rabiei, Mohammad-Shahram Moin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.19173
• Pdf URL: https://arxiv.org/pdf/2407.19173
• Copy Paste: [[2407.19173]] FarSSiBERT: A Novel Transformer-based Model for Semantic Similarity Measurement of Persian Social Networks Informal Texts(https://arxiv.org/abs/2407.19173)
• Keywords: language model
• Abstract: One fundamental task for NLP is to determine the similarity between two texts and evaluate the extent of their likeness. The previous methods for the Persian language have low accuracy and are unable to comprehend the structure and meaning of texts effectively. Additionally, these methods primarily focus on formal texts, but in real-world applications of text processing, there is a need for robust methods that can handle colloquial texts. This requires algorithms that consider the structure and significance of words based on context, rather than just the frequency of words. The lack of a proper dataset for this task in the Persian language makes it important to develop such algorithms and construct a dataset for Persian text. This paper introduces a new transformer-based model to measure semantic similarity between Persian informal short texts from social networks. In addition, a Persian dataset named FarSSiM has been constructed for this purpose, using real data from social networks and manually annotated and verified by a linguistic expert team. The proposed model involves training a large language model using the BERT architecture from scratch. This model, called FarSSiBERT, is pre-trained on approximately 104 million Persian informal short texts from social networks, making it one of a kind in the Persian language. Moreover, a novel specialized informal language tokenizer is provided that not only performs tokenization on formal texts well but also accurately identifies tokens that other Persian tokenizers are unable to recognize. It has been demonstrated that our proposed model outperforms ParsBERT, laBSE, and multilingual BERT in the Pearson and Spearman's coefficient criteria. Additionally, the pre-trained large language model has great potential for use in other NLP tasks on colloquial text and as a tokenizer for less-known informal words.
• 摘要：NLP 的一项基本任务是确定两段文本之间的相似性并评估它们的相似程度。以前用于波斯语的方法准确率较低，无法有效理解文本的结构和含义。此外，这些方法主要侧重于正式文本，但在文本处理的实际应用中，需要能够处理口语文本的稳健方法。这需要算法能够根据上下文考虑单词的结构和意义，而不仅仅是单词的频率。由于波斯语缺乏适合此任务的数据集，因此开发此类算法并构建波斯语文本数据集非常重要。本文介绍了一种新的基于 Transformer 的模型，用于测量来自社交网络的波斯语非正式短文本之间的语义相似性。此外，为此目的构建了一个名为 FarSSiM 的波斯语数据集，该数据集使用来自社交网络的真实数据，并由语言专家团队手动注释和验证。提出的模型涉及从头开始使用 BERT 架构训练大型语言模型。该模型名为 FarSSiBERT，已在社交网络中大约 1.04 亿条波斯语非正式短文本上进行了预训练，是波斯语中独一无二的模型。此外，还提供了一种新颖的专用非正式语言标记器，它不仅可以很好地对正式文本进行标记，还可以准确识别其他波斯语标记器无法识别的标记。事实证明，我们提出的模型在 Pearson 和 Spearman 系数标准中优于 ParsBERT、laBSE 和多语言 BERT。此外，预训练的大型语言模型在口语文本的其他 NLP 任务中以及作为鲜为人知的非正式词汇的标记器方面具有巨大潜力。

Title: On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs

• Authors: Nitay Calderon, Roi Reichart
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.19200
• Pdf URL: https://arxiv.org/pdf/2407.19200
• Copy Paste: [[2407.19200]] On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMs(https://arxiv.org/abs/2407.19200)
• Keywords: llm
• Abstract: Recent advancements in NLP systems, particularly with the introduction of LLMs, have led to widespread adoption of these systems by a broad spectrum of users across various domains, impacting decision-making, the job market, society, and scientific research. This surge in usage has led to an explosion in NLP model interpretability and analysis research, accompanied by numerous technical surveys. Yet, these surveys often overlook the needs and perspectives of explanation stakeholders. In this paper, we address three fundamental questions: Why do we need interpretability, what are we interpreting, and how? By exploring these questions, we examine existing interpretability paradigms, their properties, and their relevance to different stakeholders. We further explore the practical implications of these paradigms by analyzing trends from the past decade across multiple research fields. To this end, we retrieved thousands of papers and employed an LLM to characterize them. Our analysis reveals significant disparities between NLP developers and non-developer users, as well as between research fields, underscoring the diverse needs of stakeholders. For example, explanations of internal model components are rarely used outside the NLP field. We hope this paper informs the future design, development, and application of methods that align with the objectives and requirements of various stakeholders.
• 摘要：NLP 系统的最新进展，尤其是 LLM 的引入，已导致各个领域的广泛用户广泛采用这些系统，影响决策、就业市场、社会和科学研究。使用量的激增导致 NLP 模型可解释性和分析研究的激增，并伴随着大量技术调查。然而，这些调查往往忽视了解释利益相关者的需求和观点。在本文中，我们讨论了三个基本问题：我们为什么需要可解释性，我们在解释什么，以及如何解释？通过探索这些问题，我们研究了现有的可解释性范式、它们的属性及其与不同利益相关者的相关性。我们通过分析过去十年多个研究领域的趋势，进一步探索这些范式的实际意义。为此，我们检索了数千篇论文并使用 LLM 来描述它们。我们的分析揭示了 NLP 开发人员和非开发人员用户之间以及研究领域之间存在显著差异，强调了利益相关者的不同需求。例如，内部模型组件的解释很少在 NLP 领域之外使用。我们希望本文能够为未来符合各利益相关者的目标和要求的方法的设计、开发和应用提供参考。

Title: Understanding Memorisation in LLMs: Dynamics, Influencing Factors, and Implications

• Authors: Till Speicher, Mohammad Aflah Khan, Qinyuan Wu, Vedant Nanda, Soumi Das, Bishwamittra Ghosh, Krishna P. Gummadi, Evimaria Terzi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.19262
• Pdf URL: https://arxiv.org/pdf/2407.19262
• Copy Paste: [[2407.19262]] Understanding Memorisation in LLMs: Dynamics, Influencing Factors, and Implications(https://arxiv.org/abs/2407.19262)
• Keywords: language model, llm
• Abstract: Understanding whether and to what extent large language models (LLMs) have memorised training data has important implications for the reliability of their output and the privacy of their training data. In order to cleanly measure and disentangle memorisation from other phenomena (e.g. in-context learning), we create an experimental framework that is based on repeatedly exposing LLMs to random strings. Our framework allows us to better understand the dynamics, i.e., the behaviour of the model, when repeatedly exposing it to random strings. Using our framework, we make several striking observations: (a) we find consistent phases of the dynamics across families of models (Pythia, Phi and Llama2), (b) we identify factors that make some strings easier to memorise than others, and (c) we identify the role of local prefixes and global context in memorisation. We also show that sequential exposition to different random strings has a significant effect on memorisation. Our results, often surprising, have significant downstream implications in the study and usage of LLMs.
• 摘要：了解大型语言模型 (LLM) 是否以及在多大程度上记忆了训练数据，对于其输出的可靠性和训练数据的隐私性具有重要意义。为了清晰地测量并将记忆与其他现象（例如上下文学习）区分开来，我们创建了一个基于反复将 LLM 暴露于随机字符串的实验框架。我们的框架使我们能够更好地理解动态，即模型在反复暴露于随机字符串时的行为。使用我们的框架，我们做出了几个惊人的观察：(a) 我们发现模型系列（Pythia、Phi 和 Llama2）的动态阶段是一致的，(b) 我们确定了使某些字符串比其他字符串更容易记忆的因素，以及 (c) 我们确定了局部前缀和全局上下文在记忆中的作用。我们还表明，连续接触不同的随机字符串对记忆有显著的影响。我们的结果通常令人惊讶，对 LLM 的研究和使用具有重要的下游影响。

Title: The Impact of LoRA Adapters for LLMs on Clinical NLP Classification Under Data Limitations

• Authors: Thanh-Dung Le, Ti Ti Nguyen, Vu Nguyen Ha
• Subjects: cs.CL, eess.SP
• Abstract URL: https://arxiv.org/abs/2407.19299
• Pdf URL: https://arxiv.org/pdf/2407.19299
• Copy Paste: [[2407.19299]] The Impact of LoRA Adapters for LLMs on Clinical NLP Classification Under Data Limitations(https://arxiv.org/abs/2407.19299)
• Keywords: language model, llm
• Abstract: Fine-tuning Large Language Models (LLMs) for clinical Natural Language Processing (NLP) poses significant challenges due to the domain gap and limited data availability. This study investigates the effectiveness of various adapter techniques, equivalent to Low-Rank Adaptation (LoRA), for fine-tuning LLMs in a resource-constrained hospital environment. We experimented with four structures-Adapter, Lightweight, TinyAttention, and Gated Residual Network (GRN)-as final layers for clinical notes classification. We fine-tuned biomedical pre-trained models, including CamemBERT-bio, AliBERT, and DrBERT, alongside two Transformer-based models. Our extensive experimental results indicate that i) employing adapter structures does not yield significant improvements in fine-tuning biomedical pre-trained LLMs, and ii) simpler Transformer-based models, trained from scratch, perform better under resource constraints. Among the adapter structures, GRN demonstrated superior performance with accuracy, precision, recall, and an F1 score of 0.88. Moreover, the total training time for LLMs exceeded 1000 hours, compared to under 6 hours for simpler transformer-based models, highlighting that LLMs are more suitable for environments with extensive computational resources and larger datasets. Consequently, this study demonstrates that simpler Transformer-based models can be effectively trained from scratch, providing a viable solution for clinical NLP tasks in low-resource environments with limited data availability. By identifying the GRN as the most effective adapter structure, we offer a practical approach to enhance clinical note classification without requiring extensive computational resources.
• 摘要：由于领域差距和数据可用性有限，对用于临床自然语言处理 (NLP) 的大型语言模型 (LLM) 进行微调面临重大挑战。本研究调查了各种适配器技术（相当于低秩自适应 (LoRA)）在资源受限的医院环境中对 LLM 进行微调的有效性。我们尝试了四种结构 - 适配器、轻量级、TinyAttention 和门控残差网络 (GRN) - 作为临床笔记分类的最终层。我们对生物医学预训练模型进行了微调，包括 CamemBERT-bio、AliBERT 和 DrBERT，以及两个基于 Transformer 的模型。我们广泛的实验结果表明：i) 采用适配器结构不会在微调生物医学预训练 LLM 方面带来显着改进，ii) 从头开始​​训练的更简单的基于 Transformer 的模型在资源受限的情况下表现更好。在适配器结构中，GRN 表现出卓越的性能，准确率、精确度、召回率和 F1 得分为 0.88。此外，LLM 的总训练时间超过 1000 小时，而基于 Transformer 的简单模型则不到 6 小时，这突显出 LLM 更适合具有大量计算资源和较大数据集的环境。因此，本研究表明，基于 Transformer 的简单模型可以从头开始进行有效训练，为数据可用性有限的低资源环境中的临床 NLP 任务提供可行的解决方案。通过将 GRN 确定为最有效的适配器结构，我们提供了一种实用的方法来增强临床记录分类，而无需大量计算资源。

Title: Do Language Models Have a Critical Period for Language Acquisition?

• Authors: Ionut Constantinescu, Tiago Pimentel, Ryan Cotterell, Alex Warstadt
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.19325
• Pdf URL: https://arxiv.org/pdf/2407.19325
• Copy Paste: [[2407.19325]] Do Language Models Have a Critical Period for Language Acquisition?(https://arxiv.org/abs/2407.19325)
• Keywords: language model
• Abstract: Humans appear to have a critical period (CP) for language acquisition: Second language (L2) acquisition becomes harder after early childhood, and ceasing exposure to a first language (L1) after this period (but not before) typically does not lead to substantial loss of L1 proficiency. It is unknown whether these CP effects result from innately determined brain maturation or as a stabilization of neural connections naturally induced by experience. In this study, we use language models (LMs) to test the extent to which these phenomena are peculiar to humans, or shared by a broader class of language learners. We vary the age of exposure by training LMs on language pairs in various experimental conditions, and find that LMs, which lack any direct analog to innate maturational stages, do not show CP effects when trained sequentially on L1 and L2. Our results contradict the claim that CP effects are an inevitable result of learning in statistical learners, and they are consistent with an innate mechanism for CP effects. We show that we can reverse-engineer the CP by introducing a regularizer partway through training to simulate a maturational decrease in plasticity. All in all, our results suggest that L1 learning on its own may not be enough to induce a CP, and additional engineering is necessary to make language models more cognitively plausible.
• 摘要：人类似乎有一个语言习得的关键期 (CP)：第二语言 (L2) 习得在幼儿期后变得更加困难，而在此时期之后（而不是之前）停止接触第一语言 (L1) 通常不会导致 L1 能力的大幅下降。目前尚不清楚这些 CP 效应是由先天决定的大脑成熟度还是由经验自然引起的神经连接稳定所致。在本研究中，我们使用语言模型 (LM) 来测试这些现象在多大程度上是人类所特有的，还是更广泛的语言学习者所共有的。我们通过在各种实验条件下对语言对训练 LM 来改变接触年龄，并发现 LM 缺乏与先天成熟阶段的任何直接类似物，在 L1 和 L2 上进行顺序训练时不会显示 CP 效应。我们的结果与 CP 效应是统计学习者学习的必然结果的说法相矛盾，并且它们与 CP 效应的先天机制相一致。我们表明，我们可以通过在训练过程中引入正则化器来对 CP 进行逆向工程，以模拟可塑性的成熟下降。总而言之，我们的结果表明，仅靠 L1 学习可能不足以诱导 CP，需要进行额外的工程来使语言模型在认知上更合理。

Title: LLAVADI: What Matters For Multimodal Large Language Models Distillation

• Authors: Shilin Xu, Xiangtai Li, Haobo Yuan, Lu Qi, Yunhai Tong, Ming-Hsuan Yang
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.19409
• Pdf URL: https://arxiv.org/pdf/2407.19409
• Copy Paste: [[2407.19409]] LLAVADI: What Matters For Multimodal Large Language Models Distillation(https://arxiv.org/abs/2407.19409)
• Keywords: language model, llm
• Abstract: The recent surge in Multimodal Large Language Models (MLLMs) has showcased their remarkable potential for achieving generalized intelligence by integrating visual understanding into Large Language Models.Nevertheless, the sheer model size of MLLMs leads to substantial memory and computational demands that hinder their widespread deployment. In this work, we do not propose a new efficient model structure or train small-scale MLLMs from scratch. Instead, we focus on what matters for training small-scale MLLMs through knowledge distillation, which is the first step from the multimodal distillation perspective. Our extensive studies involve training strategies, model choices, and distillation algorithms in the knowledge distillation process. These results show that joint alignment for both tokens and logit alignment plays critical roles in teacher-student frameworks. In addition, we draw a series of intriguing observations from this study. By evaluating different benchmarks and proper strategy, even a 2.7B small-scale model can perform on par with larger models with 7B or 13B parameters. Our code and models will be publicly available for further research.
• 摘要：最近，多模态大型语言模型 (MLLM) 的激增展示了它们通过将视觉理解集成到大型语言模型中实现广义智能的巨大潜力。然而，MLLM 的庞大模型规模导致了大量内存和计算需求，阻碍了它们的广泛部署。在这项工作中，我们并没有提出一种新的高效模型结构，也没有从头开始训练小型 MLLM。相反，我们专注于通过知识蒸馏训练小型 MLLM 的重要内容，这是从多模态蒸馏角度来看的第一步。我们广泛的研究涉及知识蒸馏过程中的训练策略、模型选择和蒸馏算法。这些结果表明，在师生框架中，token 和 logit 对齐的联合对齐起着关键作用。此外，我们从这项研究中得出了一系列有趣的观察结果。通过评估不同的基准和适当的策略，即使是 2.7B 的小规模模型也可以与具有 7B 或 13B 参数的大型模型相媲美。我们的代码和模型将公开供进一步研究。

Title: Impact of Decoding Methods on Human Alignment of Conversational LLMs

• Authors: Shaz Furniturewala, Kokil Jaidka, Yashvardhan Sharma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.19526
• Pdf URL: https://arxiv.org/pdf/2407.19526
• Copy Paste: [[2407.19526]] Impact of Decoding Methods on Human Alignment of Conversational LLMs(https://arxiv.org/abs/2407.19526)
• Keywords: language model, llm, chat
• Abstract: To be included into chatbot systems, Large language models (LLMs) must be aligned with human conversational conventions. However, being trained mainly on web-scraped data gives existing LLMs a voice closer to informational text than actual human speech. In this paper, we examine the effect of decoding methods on the alignment between LLM-generated and human conversations, including Beam Search, Top K Sampling, and Nucleus Sampling. We present new measures of alignment in substance, style, and psychometric orientation, and experiment with two conversation datasets. Our results provide subtle insights: better alignment is attributed to fewer beams in Beam Search and lower values of P in Nucleus Sampling. We also find that task-oriented and open-ended datasets perform differently in terms of alignment, indicating the significance of taking into account the context of the interaction.
• 摘要：要纳入聊天机器人系统，大型语言模型 (LLM) 必须与人类对话惯例保持一致。然而，主要在网络抓取的数据上进行训练，使得现有的 LLM 的声音更接近信息文本，而不是实际的人类语音。在本文中，我们研究了解码方法对 LLM 生成的对话与人类对话之间的对齐的影响，包括 Beam Search、Top K Sampling 和 Nucleus Sampling。我们提出了内容、风格和心理测量取向的新对齐度量，并使用两个对话数据集进行了实验。我们的结果提供了微妙的见解：更好的对齐归因于 Beam Search 中较少的波束和 Nucleus Sampling 中较低的 P 值。我们还发现面向任务和开放式数据集在对齐方面的表现不同，这表明考虑交互上下文的重要性。

Title: Motamot: A Dataset for Revealing the Supremacy of Large Language Models over Transformer Models in Bengali Political Sentiment Analysis

• Authors: Fatema Tuj Johora Faria, Mukaffi Bin Moin, Rabeya Islam Mumu, Md Mahabubul Alam Abir, Abrar Nawar Alfy, Mohammad Shafiul Alam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.19528
• Pdf URL: https://arxiv.org/pdf/2407.19528
• Copy Paste: [[2407.19528]] Motamot: A Dataset for Revealing the Supremacy of Large Language Models over Transformer Models in Bengali Political Sentiment Analysis(https://arxiv.org/abs/2407.19528)
• Keywords: language model, gpt, llm
• Abstract: Sentiment analysis is the process of identifying and categorizing people's emotions or opinions regarding various topics. Analyzing political sentiment is critical for understanding the complexities of public opinion processes, especially during election seasons. It gives significant information on voter preferences, attitudes, and current trends. In this study, we investigate political sentiment analysis during Bangladeshi elections, specifically examining how effectively Pre-trained Language Models (PLMs) and Large Language Models (LLMs) capture complex sentiment characteristics. Our study centers on the creation of the "Motamot" dataset, comprising 7,058 instances annotated with positive and negative sentiments, sourced from diverse online newspaper portals, forming a comprehensive resource for political sentiment analysis. We meticulously evaluate the performance of various PLMs including BanglaBERT, Bangla BERT Base, XLM-RoBERTa, mBERT, and sahajBERT, alongside LLMs such as Gemini 1.5 Pro and GPT 3.5 Turbo. Moreover, we explore zero-shot and few-shot learning strategies to enhance our understanding of political sentiment analysis methodologies. Our findings underscore BanglaBERT's commendable accuracy of 88.10% among PLMs. However, the exploration into LLMs reveals even more promising results. Through the adept application of Few-Shot learning techniques, Gemini 1.5 Pro achieves an impressive accuracy of 96.33%, surpassing the remarkable performance of GPT 3.5 Turbo, which stands at 94%. This underscores Gemini 1.5 Pro's status as the superior performer in this comparison.
• 摘要：情绪分析是识别和分类人们对各种主题的情绪或意见的过程。分析政治情绪对于理解舆论过程的复杂性至关重要，尤其是在选举季。它提供了有关选民偏好、态度和当前趋势的重要信息。在本研究中，我们调查了孟加拉国选举期间的政治情绪分析，特别是研究了预训练语言模型 (PLM) 和大型语言模型 (LLM) 如何有效地捕捉复杂的情绪特征。我们的研究以创建“Motamot”数据集为中心，该数据集包含 7,058 个带有正面和负面情绪注释的实例，这些实例来自不同的在线报纸门户网站，形成了政治情绪分析的综合资源。我们仔细评估了各种 PLM 的性能，包括 BanglaBERT、Bangla BERT Base、XLM-RoBERTa、mBERT 和 sahajBERT，以及 Gemini 1.5 Pro 和 GPT 3.5 Turbo 等 LLM。此外，我们探索了零样本和少样本学习策略，以增强我们对政治情绪分析方法的理解。我们的研究结果强调了 BanglaBERT 在 PLM 中的 88.10% 的惊人准确率。然而，对 LLM 的探索揭示了更有希望的结果。通过熟练应用少样本学习技术，Gemini 1.5 Pro 实现了令人印象深刻的 96.33% 的准确率，超过了 GPT 3.5 Turbo 的出色表现，后者为 94%。这强调了 Gemini 1.5 Pro 在本次比较中表现优异的地位。

Title: Are LLMs Good Annotators for Discourse-level Event Relation Extraction?

• Authors: Kangda Wei, Aayush Gautam, Ruihong Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.19568
• Pdf URL: https://arxiv.org/pdf/2407.19568
• Copy Paste: [[2407.19568]] Are LLMs Good Annotators for Discourse-level Event Relation Extraction?(https://arxiv.org/abs/2407.19568)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) have demonstrated proficiency in a wide array of natural language processing tasks. However, its effectiveness over discourse-level event relation extraction (ERE) tasks remains unexplored. In this paper, we assess the effectiveness of LLMs in addressing discourse-level ERE tasks characterized by lengthy documents and intricate relations encompassing coreference, temporal, causal, and subevent types. Evaluation is conducted using an commercial model, GPT-3.5, and an open-source model, LLaMA-2. Our study reveals a notable underperformance of LLMs compared to the baseline established through supervised learning. Although Supervised Fine-Tuning (SFT) can improve LLMs performance, it does not scale well compared to the smaller supervised baseline model. Our quantitative and qualitative analysis shows that LLMs have several weaknesses when applied for extracting event relations, including a tendency to fabricate event mentions, and failures to capture transitivity rules among relations, detect long distance relations, or comprehend contexts with dense event mentions.
• 摘要：大型语言模型 (LLM) 已在各种自然语言处理任务中表现出色。然而，它在语篇级事件关系提取 (ERE) 任务中的有效性仍未得到探索。在本文中，我们评估了 LLM 在解决语篇级 ERE 任务中的有效性，这些任务的特点是冗长的文档和复杂的关系，包括共指、时间、因果和子事件类型。评估使用商业模型 GPT-3.5 和开源模型 LLaMA-2 进行。我们的研究表明，与通过监督学习建立的基线相比，LLM 的表现明显不佳。虽然监督微调 (SFT) 可以提高 LLM 的性能，但与较小的监督基线模型相比，它的可扩展性不佳。我们的定量和定性分析表明，LLM 在用于提取事件关系时存在几个弱点，包括倾向于捏造事件提及，无法捕捉关系之间的传递性规则、检测长距离关系或理解具有密集事件提及的上下文。

Title: SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain

• Authors: Pierre Colombo, Telmo Pires, Malik Boudiaf, Rui Melo, Dominic Culver, Sofia Morgado, Etienne Malaboeuf, Gabriel Hautreux, Johanne Charpentier, Michael Desa
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.19584
• Pdf URL: https://arxiv.org/pdf/2407.19584
• Copy Paste: [[2407.19584]] SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain(https://arxiv.org/abs/2407.19584)
• Keywords: language model, llm
• Abstract: In this paper, we introduce SaulLM-54B and SaulLM-141B, two large language models (LLMs) tailored for the legal sector. These models, which feature architectures of 54 billion and 141 billion parameters, respectively, are based on the Mixtral architecture. The development of SaulLM-54B and SaulLM-141B is guided by large-scale domain adaptation, divided into three strategies: (1) the exploitation of continued pretraining involving a base corpus that includes over 540 billion of legal tokens, (2) the implementation of a specialized legal instruction-following protocol, and (3) the alignment of model outputs with human preferences in legal interpretations. The integration of synthetically generated data in the second and third steps enhances the models' capabilities in interpreting and processing legal texts, effectively reaching state-of-the-art performance and outperforming previous open-source models on LegalBench-Instruct. This work explores the trade-offs involved in domain-specific adaptation at this scale, offering insights that may inform future studies on domain adaptation using strong decoder models. Building upon SaulLM-7B, this study refines the approach to produce an LLM better equipped for legal tasks. We are releasing base, instruct, and aligned versions on top of SaulLM-54B and SaulLM-141B under the MIT License to facilitate reuse and collaborative research.
• 摘要：在本文中，我们介绍了两个专为法律领域量身定制的大型语言模型 (LLM) SaulLM-54B 和 SaulLM-141B。这些模型分别具有 540 亿和 1410 亿个参数的架构，基于 Mixtral 架构。SaulLM-54B 和 SaulLM-141B 的开发以大规模领域自适应为指导，分为三个策略：(1) 利用包含超过 5400 亿个法律标记的基础语料库进行持续预训练，(2) 实施专门的法律指令遵循协议，以及 (3) 将模型输出与人类对法律解释的偏好相一致。第二步和第三步中合成生成的数据的集成增强了模型解释和处理法律文本的能力，有效地达到了最先进的性能，并超越了 LegalBench-Instruct 上以前的开源模型。这项研究探索了在这种规模下特定领域适应所涉及的权衡，提供了一些见解，可能为未来使用强解码器模型进行领域适应的研究提供参考。在 SaulLM-7B 的基础上，这项研究改进了方法，以生成更适合法律任务的 LLM。我们将在 MIT 许可下在 SaulLM-54B 和 SaulLM-141B 之上发布基础、指导和对齐版本，以促进重复使用和协作研究。

Title: Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge

• Authors: Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, Sainbayar Sukhbaatar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.19594
• Pdf URL: https://arxiv.org/pdf/2407.19594
• Copy Paste: [[2407.19594]] Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge(https://arxiv.org/abs/2407.19594)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are rapidly surpassing human knowledge in many domains. While improving these models traditionally relies on costly human data, recent self-rewarding mechanisms (Yuan et al., 2024) have shown that LLMs can improve by judging their own responses instead of relying on human labelers. However, existing methods have primarily focused on improving model responses rather than judgment capabilities, resulting in rapid saturation during iterative training. To address this issue, we introduce a novel Meta-Rewarding step to the self-improvement process, where the model judges its own judgements and uses that feedback to refine its judgment skills. Surprisingly, this unsupervised approach improves the model's ability to judge {\em and} follow instructions, as demonstrated by a win rate improvement of Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2, and 20.6% to 29.1% on Arena-Hard. These results strongly suggest the potential for self-improving models without human supervision.
• 摘要：大型语言模型 (LLM) 正在许多领域迅速超越人类知识。虽然改进这些模型传统上依赖于昂贵的人工数据，但最近的自我奖励机制 (Yuan 等人，2024) 表明，LLM 可以通过判断自己的反应而不是依靠人工标注者来改进。然而，现有方法主要侧重于改进模型反应而不是判断能力，导致迭代训练期间快速饱和。为了解决这个问题，我们在自我改进过程中引入了一个新颖的元奖励步骤，模型会判断自己的判断并使用该反馈来改进其判断技能。令人惊讶的是，这种无监督方法提高了模型判断和遵循指令的能力，正如 Llama-3-8B-Instruct 在 AlpacaEval 2 上的胜率从 22.9% 提高到 39.4% 以及在 Arena-Hard 上的胜率从 20.6% 提高到 29.1% 所证明的那样。这些结果强烈表明了无需人工监督的自我改进模型的潜力。

Title: From Pre-training Corpora to Large Language Models: What Factors Influence LLM Performance in Causal Discovery Tasks?

• Authors: Tao Feng, Lizhen Qu, Niket Tandon, Zhuang Li, Xiaoxi Kang, Gholamreza Haffari
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.19638
• Pdf URL: https://arxiv.org/pdf/2407.19638
• Copy Paste: [[2407.19638]] From Pre-training Corpora to Large Language Models: What Factors Influence LLM Performance in Causal Discovery Tasks?(https://arxiv.org/abs/2407.19638)
• Keywords: language model, llm
• Abstract: Recent advances in artificial intelligence have seen Large Language Models (LLMs) demonstrate notable proficiency in causal discovery tasks. This study explores the factors influencing the performance of LLMs in causal discovery tasks. Utilizing open-source LLMs, we examine how the frequency of causal relations within their pre-training corpora affects their ability to accurately respond to causal discovery queries. Our findings reveal that a higher frequency of causal mentions correlates with better model performance, suggesting that extensive exposure to causal information during training enhances the models' causal discovery capabilities. Additionally, we investigate the impact of context on the validity of causal relations. Our results indicate that LLMs might exhibit divergent predictions for identical causal relations when presented in different contexts. This paper provides the first comprehensive analysis of how different factors contribute to LLM performance in causal discovery tasks.
• 摘要：人工智能的最新进展已经看到大型语言模型 (LLM) 在因果发现任务中表现出显著的能力。本研究探讨了影响 LLM 在因果发现任务中表现的因素。利用开源 LLM，我们研究了其预训练语料库中的因果关系频率如何影响其准确响应因果发现查询的能力。我们的研究结果表明，因果提及的频率越高，模型性能越好，这表明在训练期间广泛接触因果信息可以增强模型的因果发现能力。此外，我们还研究了上下文对因果关系有效性的影响。我们的结果表明，当在不同上下文中呈现时，LLM 可能会对相同的因果关系表现出不同的预测。本文首次全面分析了不同因素如何影响 LLM 在因果发现任务中的表现。

Title: SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languages

• Authors: Wenxuan Zhang, Hou Pong Chan, Yiran Zhao, Mahani Aljunied, Jianyu Wang, Chaoqun Liu, Yue Deng, Zhiqiang Hu, Weiwen Xu, Yew Ken Chia, Xin Li, Lidong Bing
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.19672
• Pdf URL: https://arxiv.org/pdf/2407.19672
• Copy Paste: [[2407.19672]] SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languages(https://arxiv.org/abs/2407.19672)
• Keywords: language model, llm, hallucination, chat
• Abstract: Large Language Models (LLMs) have shown remarkable abilities across various tasks, yet their development has predominantly centered on high-resource languages like English and Chinese, leaving low-resource languages underserved. To address this disparity, we present SeaLLMs 3, the latest iteration of the SeaLLMs model family, tailored for Southeast Asian languages. This region, characterized by its rich linguistic diversity, has lacked adequate language technology support. SeaLLMs 3 aims to bridge this gap by covering a comprehensive range of languages spoken in this region, including English, Chinese, Indonesian, Vietnamese, Thai, Tagalog, Malay, Burmese, Khmer, Lao, Tamil, and Javanese. Leveraging efficient language enhancement techniques and a specially constructed instruction tuning dataset, SeaLLMs 3 significantly reduces training costs while maintaining high performance and versatility. Our model excels in tasks such as world knowledge, mathematical reasoning, translation, and instruction following, achieving state-of-the-art performance among similarly sized models. Additionally, we prioritized safety and reliability by addressing both general and culture-specific considerations and incorporated mechanisms to reduce hallucinations. This work underscores the importance of inclusive AI, showing that advanced LLM capabilities can benefit underserved linguistic and cultural communities.
• 摘要：大型语言模型 (LLM) 在各种任务中都表现出非凡的能力，但它们的开发主要集中在英语和中文等资源丰富的语言上，而资源匮乏的语言却得不到充分的服务。为了解决这一差距，我们推出了 SeaLLMs 3，这是 SeaLLMs 模型系列的最新版本，专为东南亚语言量身定制。该地区以其丰富的语言多样性为特点，缺乏足够的语言技术支持。SeaLLMs 3 旨在通过覆盖该地区使用的广泛语言来弥补这一差距，包括英语、中文、印尼语、越南语、泰语、他加禄语、马来语、缅甸语、高棉语、老挝语、泰米尔语和爪哇语。利用高效的语言增强技术和专门构建的指令调整数据集，SeaLLMs 3 显著降低了训练成本，同时保持了高性能和多功能性。我们的模型在世界知识、数学推理、翻译和指令遵循等任务中表现出色，在类似规模的模型中实现了最先进的性能。此外，我们优先考虑安全性和可靠性，同时考虑一般性和文化特定性，并结合机制来减少幻觉。这项工作强调了包容性人工智能的重要性，表明高级 LLM 能力可以使服务不足的语言和文化社区受益。

Title: CollectiveSFT: Scaling Large Language Models for Chinese Medical Benchmark with Collective Instructions in Healthcare

• Authors: Jingwei Zhu, Minghuan Tan, Min Yang, Ruixue Li, Hamid Alinejad-Rokny
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.19705
• Pdf URL: https://arxiv.org/pdf/2407.19705
• Copy Paste: [[2407.19705]] CollectiveSFT: Scaling Large Language Models for Chinese Medical Benchmark with Collective Instructions in Healthcare(https://arxiv.org/abs/2407.19705)
• Keywords: language model, llm, prompt
• Abstract: The rapid progress in Large Language Models (LLMs) has prompted the creation of numerous benchmarks to evaluate their capabilities.This study focuses on the Comprehensive Medical Benchmark in Chinese (CMB), showcasing how dataset diversity and distribution in supervised fine-tuning (SFT) may enhance LLM performance.Remarkably, We successfully trained a smaller base model to achieve scores comparable to larger models, indicating that a diverse and well-distributed dataset can optimize performance regardless of model size.This study suggests that even smaller models may reach high performance levels with carefully curated and varied this http URL integrating a wide range of instructional content, our approach addresses potential issues such as data quality inconsistencies. Our results imply that a broader spectrum of training data may enhance a model's ability to generalize and perform effectively across different medical scenarios, highlighting the importance of dataset quality and diversity in fine-tuning processes.
• 摘要：大型语言模型 (LLM) 的快速发展促使人们创建了大量基准来评估其能力。本研究重点关注中文综合医学基准 (CMB)，展示了监督微调 (SFT) 中的数据集多样性和分布如何提高 LLM 性能。值得注意的是，我们成功地训练了一个较小的基础模型，使其获得与较大模型相当的分数，这表明无论模型大小如何，多样化且分布良好的数据集都可以优化性能。这项研究表明，即使是较小的模型也可以通过精心策划和多样化的 http URL 达到高性能水平，我们的方法整合了广泛的教学内容，解决了数据质量不一致等潜在问题。我们的结果表明，更广泛的训练数据可以增强模型在不同医疗场景中的泛化和有效执行的能力，凸显了数据集质量和多样性在微调过程中的重要性。

Title: Do Text-to-Vis Benchmarks Test Real Use of Visualisations?

• Authors: Hy Nguyen, Xuefei He, Andrew Reeson, Cecile Paris, Josiah Poon, Jonathan K. Kummerfeld
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2407.19726
• Pdf URL: https://arxiv.org/pdf/2407.19726
• Copy Paste: [[2407.19726]] Do Text-to-Vis Benchmarks Test Real Use of Visualisations?(https://arxiv.org/abs/2407.19726)
• Keywords: language model
• Abstract: Large language models are able to generate code for visualisations in response to user requests. This is a useful application, and an appealing one for NLP research because plots of data provide grounding for language. However, there are relatively few benchmarks, and it is unknown whether those that exist are representative of what people do in practice. This paper aims to answer that question through an empirical study comparing benchmark datasets and code from public repositories. Our findings reveal a substantial gap in datasets, with evaluations not testing the same distribution of chart types, attributes, and the number of actions. The only representative dataset requires modification to become an end-to-end and practical benchmark. This shows that new, more benchmarks are needed to support the development of systems that truly address users' visualisation needs. These observations will guide future data creation, highlighting which features hold genuine significance for users.
• 摘要：大型语言模型能够根据用户请求生成可视化代码。这是一个有用的应用程序，也是 NLP 研究的一个有吸引力的应用程序，因为数据图为语言提供了基础。然而，基准相对较少，而且尚不清楚现有的基准是否代表了人们在实践中的行为。本文旨在通过一项实证研究来回答这个问题，该研究比较了基准数据集和来自公共存储库的代码。我们的研究结果揭示了数据集之间存在巨大差距，评估没有测试相同的图表类型、属性和操作数量的分布。唯一具有代表性的数据集需要修改才能成为端到端和实用的基准。这表明需要新的、更多的基准来支持真正满足用户可视化需求的系统的开发。这些观察结果将指导未来的数据创建，突出哪些功能对用户具有真正的意义。

Title: Legal Minds, Algorithmic Decisions: How LLMs Apply Constitutional Principles in Complex Scenarios

• Authors: Camilla Bignotti, Carolina Camassa
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2407.19760
• Pdf URL: https://arxiv.org/pdf/2407.19760
• Copy Paste: [[2407.19760]] Legal Minds, Algorithmic Decisions: How LLMs Apply Constitutional Principles in Complex Scenarios(https://arxiv.org/abs/2407.19760)
• Keywords: language model, gpt, llm
• Abstract: In this paper, we conduct an empirical analysis of how large language models (LLMs), specifically GPT-4, interpret constitutional principles in complex decision-making scenarios. We examine rulings from the Italian Constitutional Court on bioethics issues that involve trade-offs between competing values and compare model-generated legal arguments on these issues to those presented by the State, the Court, and the applicants. Our results indicate that GPT-4 consistently aligns more closely with progressive interpretations of the Constitution, often overlooking competing values and mirroring the applicants' views rather than the more conservative perspectives of the State or the Court's moderate positions. Our experiments reveal a distinct tendency of GPT-4 to favor progressive legal interpretations, underscoring the influence of underlying data biases. We thus underscore the importance of testing alignment in real-world scenarios and considering the implications of deploying LLMs in decision-making processes.
• 摘要：在本文中，我们对大型语言模型 (LLM)（特别是 GPT-4）如何在复杂的决策场景中解释宪法原则进行了实证分析。我们研究了意大利宪法法院关于涉及相互竞争的价值观之间权衡的生物伦理问题的裁决，并将模型生成的法律论据与国家、法院和申请人提出的论据进行比较。我们的结果表明，GPT-4 始终与宪法的进步解释更加一致，往往忽视相互竞争的价值观并反映申请人的观点，而不是国家或法院温和立场的更保守的观点。我们的实验揭示了 GPT-4 明显倾向于支持进步的法律解释，强调了潜在数据偏差的影响。因此，我们强调在现实世界场景中测试一致性的重要性，并考虑在决策过程中部署 LLM 的影响。

Title: Synthesizing Scientific Summaries: An Extractive and Abstractive Approach

• Authors: Grishma Sharma, Aditi Paretkar, Deepak Sharma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.19779
• Pdf URL: https://arxiv.org/pdf/2407.19779
• Copy Paste: [[2407.19779]] Synthesizing Scientific Summaries: An Extractive and Abstractive Approach(https://arxiv.org/abs/2407.19779)
• Keywords: language model
• Abstract: The availability of a vast array of research papers in any area of study, necessitates the need of automated summarisation systems that can present the key research conducted and their corresponding findings. Scientific paper summarisation is a challenging task for various reasons including token length limits in modern transformer models and corresponding memory and compute requirements for long text. A significant amount of work has been conducted in this area, with approaches that modify the attention mechanisms of existing transformer models and others that utilise discourse information to capture long range dependencies in research papers. In this paper, we propose a hybrid methodology for research paper summarisation which incorporates an extractive and abstractive approach. We use the extractive approach to capture the key findings of research, and pair it with the introduction of the paper which captures the motivation for research. We use two models based on unsupervised learning for the extraction stage and two transformer language models, resulting in four combinations for our hybrid approach. The performances of the models are evaluated on three metrics and we present our findings in this paper. We find that using certain combinations of hyper parameters, it is possible for automated summarisation systems to exceed the abstractiveness of summaries written by humans. Finally, we state our future scope of research in extending this methodology to summarisation of generalised long documents.
• 摘要：任何研究领域都有大量的研究论文，这就需要有自动摘要系统来呈现所进行的关键研究及其相应的发现。科学论文摘要是一项具有挑战性的任务，原因有很多，包括现代 Transformer 模型中的标记长度限制以及长文本相应的内存和计算要求。该领域已经进行了大量工作，其中的方法包括修改现有 Transformer 模型的注意机制，以及利用话语信息来捕获研究论文中的长距离依赖关系的其他方法。在本文中，我们提出了一种混合研究论文摘要方法，该方法结合了提取和抽象方法。我们使用提取方法来捕捉研究的关键发现，并将其与论文的介绍相结合，以捕捉研究的动机。我们在提取阶段使用两个基于无监督学习的模型，并使用两个 Transformer 语言模型，从而为我们的混合方法提供了四种组合。根据三个指标评估模型的性能，并在本文中介绍我们的研究结果。我们发现，使用某些超参数组合，自动摘要系统可以超越人类撰写的摘要的抽象性。最后，我们陈述了未来的研究范围，即将此方法扩展到广义长文档的摘要。

Title: Introducing a new hyper-parameter for RAG: Context Window Utilization

• Authors: Kush Juvekar, Anupam Purwar
• Subjects: cs.CL, cs.ET
• Abstract URL: https://arxiv.org/abs/2407.19794
• Pdf URL: https://arxiv.org/pdf/2407.19794
• Copy Paste: [[2407.19794]] Introducing a new hyper-parameter for RAG: Context Window Utilization(https://arxiv.org/abs/2407.19794)
• Keywords: retrieval-augmented generation
• Abstract: This paper introduces a new hyper-parameter for Retrieval-Augmented Generation (RAG) systems called Context Window Utilization. RAG systems enhance generative models by incorporating relevant information retrieved from external knowledge bases, improving the factual accuracy and contextual relevance of generated responses. The size of the text chunks retrieved and processed is a critical factor influencing RAG performance. This study aims to identify the optimal chunk size that maximizes answer generation quality. Through systematic experimentation, we analyze the effects of varying chunk sizes on the efficiency and effectiveness of RAG frameworks. Our findings reveal that an optimal chunk size balances the trade-off between providing sufficient context and minimizing irrelevant information. These insights are crucial for enhancing the design and implementation of RAG systems, underscoring the importance of selecting an appropriate chunk size to achieve superior performance.
• 摘要：本文介绍了一种新的检索增强生成 (RAG) 系统超参数，即上下文窗口利用率。RAG 系统通过整合从外部知识库检索到的相关信息来增强生成模型，从而提高生成的响应的事实准确性和上下文相关性。检索和处理的文本块的大小是影响 RAG 性能的关键因素。本研究旨在确定最大限度提高答案生成质量的最佳块大小。通过系统实验，我们分析了不同块大小对 RAG 框架效率和有效性的影响。我们的研究结果表明，最佳块大小平衡了提供足够上下文和最小化不相关信息之间的权衡。这些见解对于增强 RAG 系统的设计和实施至关重要，强调了选择合适的块大小以实现卓越性能的重要性。

Title: VolDoGer: LLM-assisted Datasets for Domain Generalization in Vision-Language Tasks

• Authors: Juhwan Choi, Junehyoung Kwon, JungMin Yun, Seunguk Yu, YoungBin Kim
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2407.19795
• Pdf URL: https://arxiv.org/pdf/2407.19795
• Copy Paste: [[2407.19795]] VolDoGer: LLM-assisted Datasets for Domain Generalization in Vision-Language Tasks(https://arxiv.org/abs/2407.19795)
• Keywords: language model, llm
• Abstract: Domain generalizability is a crucial aspect of a deep learning model since it determines the capability of the model to perform well on data from unseen domains. However, research on the domain generalizability of deep learning models for vision-language tasks remains limited, primarily because of the lack of required datasets. To address these challenges, we propose VolDoGer: Vision-Language Dataset for Domain Generalization, a dedicated dataset designed for domain generalization that addresses three vision-language tasks: image captioning, visual question answering, and visual entailment. We constructed VolDoGer by extending LLM-based data annotation techniques to vision-language tasks, thereby alleviating the burden of recruiting human annotators. We evaluated the domain generalizability of various models, ranging from fine-tuned models to a recent multimodal large language model, through VolDoGer.
• 摘要：领域通用性是深度学习模型的一个重要方面，因为它决定了模型在未知领域的数据上表现良好的能力。然而，对深度学习模型在视觉语言任务领域的通用性的研究仍然有限，主要是因为缺乏所需的数据集。为了应对这些挑战，我们提出了 VolDoGer：用于领域通用性的视觉语言数据集，这是一个专门为领域通用性设计的数据集，可解决三个视觉语言任务：图像字幕、视觉问答和视觉蕴涵。我们通过将基于 LLM 的数据注释技术扩展到视觉语言任务来构建 VolDoGer，从而减轻了招募人工注释者的负担。我们通过 VolDoGer 评估了各种模型的领域通用性，从微调模型到最近的多模态大型语言模型。

Title: Teaching LLMs at Charles University: Assignments and Activities

• Authors: Jindřich Helcl, Zdeněk Kasner, Ondřej Dušek, Tomasz Limisiewicz, Dominik Macháček, Tomáš Musil, Jindřich Libovický
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.19798
• Pdf URL: https://arxiv.org/pdf/2407.19798
• Copy Paste: [[2407.19798]] Teaching LLMs at Charles University: Assignments and Activities(https://arxiv.org/abs/2407.19798)
• Keywords: language model, llm
• Abstract: This paper presents teaching materials, particularly assignments and ideas for classroom activities, from a new course on large language models (LLMs) taught at Charles University. The assignments include experiments with LLM inference for weather report generation and machine translation. The classroom activities include class quizzes, focused research on downstream tasks and datasets, and an interactive "best paper" session aimed at reading and comprehension of research papers.
• 摘要：本文介绍了查尔斯大学开设的大型语言模型 (LLM) 新课程的教学材料，特别是作业和课堂活动的想法。作业包括使用 LLM 推理进行天气报告生成和机器翻译的实验。课堂活动包括课堂测验、针对下游任务和数据集的重点研究，以及旨在阅读和理解研究论文的互动式“最佳论文”课程。

Title: Cool-Fusion: Fuse Large Language Models without Training

• Authors: Cong Liu, Xiaojun Quan, Yan Pan, Liang Lin, Weigang Wu, Xu Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.19807
• Pdf URL: https://arxiv.org/pdf/2407.19807
• Copy Paste: [[2407.19807]] Cool-Fusion: Fuse Large Language Models without Training(https://arxiv.org/abs/2407.19807)
• Keywords: language model, llm
• Abstract: We focus on the problem of fusing two or more heterogeneous large language models (LLMs) to facilitate their complementary strengths. One of the challenges on model fusion is high computational load, i.e. to fine-tune or to align vocabularies via combinatorial optimization. To this end, we propose \emph{Cool-Fusion}, a simple yet effective approach that fuses the knowledge of heterogeneous source LLMs to leverage their complementary strengths. \emph{Cool-Fusion} is the first method that does not require any type of training like the ensemble approaches. But unlike ensemble methods, it is applicable to any set of source LLMs that have different vocabularies. The basic idea is to have each source LLM individually generate tokens until the tokens can be decoded into a text segment that ends at word boundaries common to all source LLMs. Then, the source LLMs jointly rerank the generated text segment and select the best one, which is the fused text generation in one step. Extensive experiments are conducted across a variety of benchmark datasets. On \emph{GSM8K}, \emph{Cool-Fusion} increases accuracy from three strong source LLMs by a significant 8\%-17.8\%.
• 摘要：我们专注于融合两个或多个异构大型语言模型 (LLM) 以发挥其互补优势的问题。模型融合的挑战之一是高计算负荷，即通过组合优化微调或对齐词汇表。为此，我们提出了 \emph{Cool-Fusion}，这是一种简单而有效的方法，它融合了异构源 LLM 的知识以利用它们的互补优势。 \emph{Cool-Fusion} 是第一种不需要任何类型训练的方法，如集成方法。但与集成方法不同，它适用于任何具有不同词汇表的源 LLM 集。基本思想是让每个源 LLM 单独生成标记，直到标记可以解码为以所有源 LLM 共有的单词边界结束的文本段。然后，源 LLM 联合重新排序生成的文本段并选择最佳文本段，即一步完成融合文本生成。在各种基准数据集上进行了广泛的实验。在 \emph{GSM8K} 上，\emph{Cool-Fusion} 将三个强源 LLM 的准确率显著提高了 8\%-17.8\%。

Title: Improving Retrieval Augmented Language Model with Self-Reasoning

• Authors: Yuan Xia, Jingbo Zhou, Zhenhui Shi, Jun Chen, Haifeng Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.19813
• Pdf URL: https://arxiv.org/pdf/2407.19813
• Copy Paste: [[2407.19813]] Improving Retrieval Augmented Language Model with Self-Reasoning(https://arxiv.org/abs/2407.19813)
• Keywords: language model, gpt, llm, hallucination
• Abstract: The Retrieval-Augmented Language Model (RALM) has shown remarkable performance on knowledge-intensive tasks by incorporating external knowledge during inference, which mitigates the factual hallucinations inherited in large language models (LLMs). Despite these advancements, challenges persist in the implementation of RALMs, particularly concerning their reliability and traceability. To be specific, the irrelevant document retrieval may result in unhelpful response generation or even deteriorate the performance of LLMs, while the lack of proper citations in generated outputs complicates efforts to verify the trustworthiness of the models. To this end, we propose a novel self-reasoning framework aimed at improving the reliability and traceability of RALMs, whose core idea is to leverage reasoning trajectories generated by the LLM itself. The framework involves constructing self-reason trajectories with three processes: a relevance-aware process, an evidence-aware selective process, and a trajectory analysis process. We have evaluated our framework across four public datasets (two short-form QA datasets, one long-form QA dataset, and one fact verification dataset) to demonstrate the superiority of our method, which can outperform existing state-of-art models and can achieve comparable performance with GPT-4, while only using 2,000 training samples.
• 摘要：检索增强语言模型 (RALM) 通过在推理过程中结合外部知识，减轻了大型语言模型 (LLM) 中继承的事实幻觉，在知识密集型任务上表现出色。尽管取得了这些进步，但在实施 RALM 方面仍然存在挑战，特别是在其可靠性和可追溯性方面。具体来说，不相关的文档检索可能会导致无用的响应生成，甚至降低 LLM 的性能，而生成的输出中缺乏适当的引用会使验证模型可信度的努力变得复杂。为此，我们提出了一种新颖的自推理框架，旨在提高 RALM 的可靠性和可追溯性，其核心思想是利用 LLM 本身生成的推理轨迹。该框架涉及通过三个过程构建自推理轨迹：相关性感知过程、证据感知选择过程和轨迹分析过程。我们在四个公共数据集（两个短格式 QA 数据集、一个长格式 QA 数据集和一个事实验证数据集）上评估了我们的框架，以证明我们方法的优越性，它可以超越现有的最先进模型，并且可以实现与 GPT-4 相当的性能，同时仅使用 2,000 个训练样本。

Title: Comparative Analysis of Encoder-Based NER and Large Language Models for Skill Extraction from Russian Job Vacancies

• Authors: Nikita Matkin, Aleksei Smirnov, Mikhail Usanin, Egor Ivanov, Kirill Sobyanin, Sofiia Paklina, Petr Parshakov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.19816
• Pdf URL: https://arxiv.org/pdf/2407.19816
• Copy Paste: [[2407.19816]] Comparative Analysis of Encoder-Based NER and Large Language Models for Skill Extraction from Russian Job Vacancies(https://arxiv.org/abs/2407.19816)
• Keywords: language model, llm
• Abstract: The labor market is undergoing rapid changes, with increasing demands on job seekers and a surge in job openings. Identifying essential skills and competencies from job descriptions is challenging due to varying employer requirements and the omission of key skills. This study addresses these challenges by comparing traditional Named Entity Recognition (NER) methods based on encoders with Large Language Models (LLMs) for extracting skills from Russian job vacancies. Using a labeled dataset of 4,000 job vacancies for training and 1,472 for testing, the performance of both approaches is evaluated. Results indicate that traditional NER models, especially DeepPavlov RuBERT NER tuned, outperform LLMs across various metrics including accuracy, precision, recall, and inference time. The findings suggest that traditional NER models provide more effective and efficient solutions for skill extraction, enhancing job requirement clarity and aiding job seekers in aligning their qualifications with employer expectations. This research contributes to the field of natural language processing (NLP) and its application in the labor market, particularly in non-English contexts.
• 摘要：劳动力市场正在经历快速变化，对求职者的需求不断增加，职位空缺数量激增。由于雇主的要求各不相同，而且关键技能被遗漏，从职位描述中识别出必要的技能和能力是一项挑战。本研究通过比较基于编码器的传统命名实体识别 (NER) 方法与大型语言模型 (LLM) 来应对这些挑战，以从俄罗斯职位空缺中提取技能。使用 4,000 个职位空缺的标记数据集进行训练，1,472 个职位空缺进行测试，评估两种方法的性能。结果表明，传统的 NER 模型，尤其是 DeepPavlov RuBERT NER 调整，在准确率、精确率、召回率和推理时间等各种指标上都优于 LLM。研究结果表明，传统的 NER 模型为技能提取提供了更有效、更高效的解决方案，提高了职位要求的清晰度，并帮助求职者将自己的资格与雇主的期望相匹配。这项研究为自然语言处理 (NLP) 领域及其在劳动力市场中的应用做出了贡献，特别是在非英语环境中。

Title: Concise Thoughts: Impact of Output Length on LLM Reasoning and Cost

• Authors: Sania Nayab, Giulio Rossolini, Giorgio Buttazzo, Nicolamaria Manes, Fabrizio Giacomelli
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.19825
• Pdf URL: https://arxiv.org/pdf/2407.19825
• Copy Paste: [[2407.19825]] Concise Thoughts: Impact of Output Length on LLM Reasoning and Cost(https://arxiv.org/abs/2407.19825)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Today's large language models (LLMs) can solve challenging question-answering tasks, and prompt engineering techniques, such as chain-of-thought (CoT), have gained attention for enhancing the explanation and correctness of outputs. Nevertheless, models require significant time to generate answers augmented with lengthy reasoning details. To address this issue, this paper analyzes the impact of output lengths on LLM inference pipelines and proposes novel metrics to evaluate them in terms of \textit{correct conciseness}. It also examines the impact of controlling output length through a refined prompt engineering strategy, Constrained-CoT (CCoT), which encourages the model to limit output length. Experiments on pre-trained LLMs demonstrated the benefit of the proposed metrics and the effectiveness of CCoT across different models. For instance, constraining the reasoning of LLaMA2-70b to 100 words improves the accuracy from 36.01\% (CoT) to 41.07\% (CCoT) on the GSM8K dataset, while reducing the average output length by 28 words.
• 摘要：当今的大型语言模型 (LLM) 可以解决具有挑战性的问答任务，而诸如思路链 (CoT) 之类的快速工程技术因增强输出的解释性和正确性而受到关注。然而，模型需要大量时间来生成带有冗长推理细节的答案。为了解决这个问题，本文分析了输出长度对 LLM 推理流程的影响，并提出了新的指标来评估它们在 \textit{正确简洁性} 方面的效果。它还研究了通过改进的快速工程策略 Constrained-CoT (CCoT) 控制输出长度的影响，该策略鼓励模型限制输出长度。在预训练的 LLM 上进行的实验证明了所提出的指标的好处以及 CCoT 在不同模型中的有效性。例如，将 LLaMA2-70b 的推理限制为 100 个单词，可将 GSM8K 数据集上的准确率从 36.01%（CoT）提高到 41.07%（CCoT），同时将平均输出长度减少 28 个单词。

Title: ATHAR: A High-Quality and Diverse Dataset for Classical Arabic to English Translation

• Authors: Mohammed Khalil, Mohammed Sabry
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.19835
• Pdf URL: https://arxiv.org/pdf/2407.19835
• Copy Paste: [[2407.19835]] ATHAR: A High-Quality and Diverse Dataset for Classical Arabic to English Translation(https://arxiv.org/abs/2407.19835)
• Keywords: language model, llm
• Abstract: Classical Arabic represents a significant era, encompassing the golden age of Arab culture, philosophy, and scientific literature. With a broad consensus on the importance of translating these literatures to enrich knowledge dissemination across communities, the advent of large language models (LLMs) and translation systems offers promising tools to facilitate this goal. However, we have identified a scarcity of translation datasets in Classical Arabic, which are often limited in scope and topics, hindering the development of high-quality translation systems. In response, we present the ATHAR dataset, comprising 66,000 high-quality Classical Arabic to English translation samples that cover a wide array of subjects including science, culture, and philosophy. Furthermore, we assess the performance of current state-of-the-art LLMs under various settings, concluding that there is a need for such datasets in current systems. Our findings highlight how models can benefit from fine-tuning or incorporating this dataset into their pretraining pipelines. The dataset is publicly available on the HuggingFace Data Hub at \url{this https URL}.
• 摘要：古典阿拉伯语代表了一个重要的时代，涵盖了阿拉伯文化、哲学和科学文献的黄金时代。人们普遍认为翻译这些文献对于丰富社区间的知识传播非常重要，大型语言模型 (LLM) 和翻译系统的出现为实现这一目标提供了有希望的工具。然而，我们发现古典阿拉伯语的翻译数据集很少，这些数据集的范围和主题往往有限，阻碍了高质量翻译系统的发展。为此，我们提出了 ATHAR 数据集，其中包含 66,000 个高质量的古典阿拉伯语到英语的翻译样本，涵盖了包括科学、文化和哲学在内的广泛主题。此外，我们在各种环境下评估了当前最先进的 LLM 的性能，得出结论，当前系统需要这样的数据集。我们的研究结果强调了模型如何从微调或将此数据集纳入其预训练管道中受益。该数据集在 HuggingFace Data Hub 上公开提供，网址为 \url{this https URL}。

Title: Preliminary WMT24 Ranking of General MT Systems and LLMs

• Authors: Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondrej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Kenton Murray, Masaaki Nagata, Martin Popel, Maja Popovic, Mariya Shmatova, Steinþór Steingrímsson, Vilém Zouhar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.19884
• Pdf URL: https://arxiv.org/pdf/2407.19884
• Copy Paste: [[2407.19884]] Preliminary WMT24 Ranking of General MT Systems and LLMs(https://arxiv.org/abs/2407.19884)
• Keywords: llm
• Abstract: This is the preliminary ranking of WMT24 General MT systems based on automatic metrics. The official ranking will be a human evaluation, which is superior to the automatic ranking and supersedes it. The purpose of this report is not to interpret any findings but only provide preliminary results to the participants of the General MT task that may be useful during the writing of the system submission.
• 摘要：这是基于自动指标的 WMT24 通用机器翻译系统的初步排名。官方排名将由人工评估，优于并取代自动排名。本报告的目的不是解释任何发现，而只是向通用机器翻译任务的参与者提供初步结果，这些结果可能在编写系统提交时有用。

Title: Sentiment Analysis of Lithuanian Online Reviews Using Large Language Models

• Authors: Brigita Vileikytė, Mantas Lukoševičius, Lukas Stankevičius
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.19914
• Pdf URL: https://arxiv.org/pdf/2407.19914
• Copy Paste: [[2407.19914]] Sentiment Analysis of Lithuanian Online Reviews Using Large Language Models(https://arxiv.org/abs/2407.19914)
• Keywords: language model, gpt, llm
• Abstract: Sentiment analysis is a widely researched area within Natural Language Processing (NLP), attracting significant interest due to the advent of automated solutions. Despite this, the task remains challenging because of the inherent complexity of languages and the subjective nature of sentiments. It is even more challenging for less-studied and less-resourced languages such as Lithuanian. Our review of existing Lithuanian NLP research reveals that traditional machine learning methods and classification algorithms have limited effectiveness for the task. In this work, we address sentiment analysis of Lithuanian five-star-based online reviews from multiple domains that we collect and clean. We apply transformer models to this task for the first time, exploring the capabilities of pre-trained multilingual Large Language Models (LLMs), specifically focusing on fine-tuning BERT and T5 models. Given the inherent difficulty of the task, the fine-tuned models perform quite well, especially when the sentiments themselves are less ambiguous: 80.74% and 89.61% testing recognition accuracy of the most popular one- and five-star reviews respectively. They significantly outperform current commercial state-of-the-art general-purpose LLM GPT-4. We openly share our fine-tuned LLMs online.
• 摘要：情感分析是自然语言处理 (NLP) 中一个被广泛研究的领域，由于自动化解决方案的出现而引起了人们的极大兴趣。尽管如此，由于语言固有的复杂性和情感的主观性，这项任务仍然具有挑战性。对于立陶宛语等研究较少、资源较少的语言来说，这更具挑战性。我们对现有的立陶宛语 NLP 研究的回顾表明，传统的机器学习方法和分类算法对这项任务的有效性有限。在这项工作中，我们针对从我们收集和清理的多个领域收集的立陶宛语五星在线评论进行情感分析。我们首次将 Transformer 模型应用于这项任务，探索预训练多语言大型语言模型 (LLM) 的功能，特别关注微调 BERT 和 T5 模型。考虑到这项任务固有的难度，微调后的模型表现相当出色，尤其是当情感本身不太模糊时：对最受欢迎的一星和五星评论的测试识别准确率分别为 80.74% 和 89.61%。它们的表现明显优于目前商用的最先进的通用 LLM GPT-4。我们在网上公开分享我们经过微调的 LLM。

Title: Inference acceleration for large language models using "stairs" assisted greedy generation

• Authors: Domas Grigaliūnas, Mantas Lukoševičius
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2407.19947
• Pdf URL: https://arxiv.org/pdf/2407.19947
• Copy Paste: [[2407.19947]] Inference acceleration for large language models using "stairs" assisted greedy generation(https://arxiv.org/abs/2407.19947)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) with billions of parameters are known for their impressive predicting capabilities but require lots of resources to run. With their massive rise in popularity, even a small reduction in required resources could have an impact on environment. On the other hand, smaller models require fewer resources but may sacrifice accuracy. In this work, we are proposing an implementation of ``stairs'' assisted greedy generation. It is a modified assisted generation methodology that makes use of a smaller model's fast generation, large model's batch prediction, and "stairs" validation in order to achieve a speed up in prediction generation. Results show between 9.58 and 17.24 percent inference time reduction compared to a stand-alone large LLM prediction in a text generation task without a loss in accuracy.
• 摘要：具有数十亿个参数的大型语言模型 (LLM) 以其令人印象深刻的预测能力而闻名，但运行时需要大量资源。随着它们越来越受欢迎，即使所需资源的少量减少也会对环境产生影响。另一方面，较小的模型需要较少的资源，但可能会牺牲准确性。在这项工作中，我们提出了一种“阶梯式”辅助贪婪生成的实现。这是一种改进的辅助生成方法，利用较小模型的快速生成、大型模型的批量预测和“阶梯式”验证来加快预测生成速度。结果表明，与文本生成任务中的独立大型 LLM 预测相比，推理时间减少了 9.58% 到 17.24%，而准确性没有损失。

Title: Do LLMs Really Adapt to Domains? An Ontology Learning Perspective

• Authors: Huu Tan Mai, Cuong Xuan Chu, Heiko Paulheim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.19998
• Pdf URL: https://arxiv.org/pdf/2407.19998
• Copy Paste: [[2407.19998]] Do LLMs Really Adapt to Domains? An Ontology Learning Perspective(https://arxiv.org/abs/2407.19998)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated unprecedented prowess across various natural language processing tasks in various application domains. Recent studies show that LLMs can be leveraged to perform lexical semantic tasks, such as Knowledge Base Completion (KBC) or Ontology Learning (OL). However, it has not effectively been verified whether their success is due to their ability to reason over unstructured or semi-structured data, or their effective learning of linguistic patterns and senses alone. This unresolved question is particularly crucial when dealing with domain-specific data, where the lexical senses and their meaning can completely differ from what a LLM has learned during its training stage. This paper investigates the following question: Do LLMs really adapt to domains and remain consistent in the extraction of structured knowledge, or do they only learn lexical senses instead of reasoning? To answer this question and, we devise a controlled experiment setup that uses WordNet to synthesize parallel corpora, with English and gibberish terms. We examine the differences in the outputs of LLMs for each corpus in two OL tasks: relation extraction and taxonomy discovery. Empirical results show that, while adapting to the gibberish corpora, off-the-shelf LLMs do not consistently reason over semantic relationships between concepts, and instead leverage senses and their frame. However, fine-tuning improves the performance of LLMs on lexical semantic tasks even when the domain-specific terms are arbitrary and unseen during pre-training, hinting at the applicability of pre-trained LLMs for OL.
• 摘要：大型语言模型 (LLM) 在各种应用领域的各种自然语言处理任务中展现出了前所未有的威力。最近的研究表明，LLM 可用于执行词汇语义任务，例如知识库完成 (KBC) 或本体学习 (OL)。然而，尚未有效验证它们的成功是归功于它们对非结构化或半结构化数据的推理能力，还是它们对语言模式和意义的有效学习。这个悬而未决的问题在处理领域特定数据时尤其重要，因为领域特定数据的词汇意义及其含义可能与 LLM 在训练阶段学到的完全不同。本文探讨了以下问题：LLM 是否真的适应领域并在结构化知识的提取中保持一致，还是它们只学习词汇意义而不是推理？为了回答这个问题，我们设计了一个受控实验设置，使用 WordNet 合成平行语料库，其中包含英语和胡言乱语术语。我们研究了两个 OL 任务中 LLM 对每个语料库的输出差异：关系提取和分类发现。实证结果表明，在适应乱码语料库时，现成的 LLM 不会一致地推理概念之间的语义关系，而是利用意义及其框架。然而，即使领域特定术语在预训练期间是任意的和看不见的，微调也会提高 LLM 在词汇语义任务上的性能，这暗示了预训练 LLM 对 OL 的适用性。

Title: Exploring Large Language Models to generate Easy to Read content

• Authors: Paloma Martínez, Lourdes Moreno, Alberto Ramos
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2407.20046
• Pdf URL: https://arxiv.org/pdf/2407.20046
• Copy Paste: [[2407.20046]] Exploring Large Language Models to generate Easy to Read content(https://arxiv.org/abs/2407.20046)
• Keywords: language model, llm
• Abstract: Ensuring text accessibility and understandability are essential goals, particularly for individuals with cognitive impairments and intellectual disabilities, who encounter challenges in accessing information across various mediums such as web pages, newspapers, administrative tasks, or health documents. Initiatives like Easy to Read and Plain Language guidelines aim to simplify complex texts; however, standardizing these guidelines remains challenging and often involves manual processes. This work presents an exploratory investigation into leveraging Artificial Intelligence (AI) and Natural Language Processing (NLP) approaches to systematically simplify Spanish texts into Easy to Read formats, with a focus on utilizing Large Language Models (LLMs) for simplifying texts, especially in generating Easy to Read content. The study contributes a parallel corpus of Spanish adapted for Easy To Read format, which serves as a valuable resource for training and testing text simplification systems. Additionally, several text simplification experiments using LLMs and the collected corpus are conducted, involving fine-tuning and testing a Llama2 model to generate Easy to Read content. A qualitative evaluation, guided by an expert in text adaptation for Easy to Read content, is carried out to assess the automatically simplified texts. This research contributes to advancing text accessibility for individuals with cognitive impairments, highlighting promising strategies for leveraging LLMs while responsibly managing energy usage.
• 摘要：确保文本的可访问性和可理解性是基本目标，特别是对于有认知障碍和智力障碍的人来说，他们在通过各种媒介（例如网页、报纸、行政任务或健康文件）获取信息时遇到挑战。诸如“易读”和“通俗语言”指南之类的举措旨在简化复杂的文本；然而，标准化这些指南仍然具有挑战性，而且通常需要手动流程。这项研究提出了一项探索性研究，利用人工智能 (AI) 和自然语言处理 (NLP) 方法系统地将西班牙语文本简化为易读格式，重点是利用大型语言模型 (LLM) 简化文本，特别是在生成易读内容方面。该研究提供了一个适合易读格式的西班牙语平行语料库，它是训练和测试文本简化系统的宝贵资源。此外，还使用 ​​LLM 和收集的语料库进行了几项文本简化实验，包括微调和测试 Llama2 模型以生成易读内容。在易读内容文本改编专家的指导下，研究人员进行了定性评估，以评估自动简化的文本。这项研究有助于提高认知障碍人士的文本可访问性，并强调了利用 LLM 的同时负责任地管理能源使用的有前景的策略。

Title: MindSearch: Mimicking Human Minds Elicits Deep AI Searcher

• Authors: Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, Feng Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2407.20183
• Pdf URL: https://arxiv.org/pdf/2407.20183
• Copy Paste: [[2407.20183]] MindSearch: Mimicking Human Minds Elicits Deep AI Searcher(https://arxiv.org/abs/2407.20183)
• Keywords: language model, gpt, llm, chat, agent
• Abstract: Information seeking and integration is a complex cognitive task that consumes enormous time and effort. Inspired by the remarkable progress of Large Language Models, recent works attempt to solve this task by combining LLMs and search engines. However, these methods still obtain unsatisfying performance due to three challenges: (1) complex requests often cannot be accurately and completely retrieved by the search engine once (2) corresponding information to be integrated is spread over multiple web pages along with massive noise, and (3) a large number of web pages with long contents may quickly exceed the maximum context length of LLMs. Inspired by the cognitive process when humans solve these problems, we introduce MindSearch to mimic the human minds in web information seeking and integration, which can be instantiated by a simple yet effective LLM-based multi-agent framework. The WebPlanner models the human mind of multi-step information seeking as a dynamic graph construction process: it decomposes the user query into atomic sub-questions as nodes in the graph and progressively extends the graph based on the search result from WebSearcher. Tasked with each sub-question, WebSearcher performs hierarchical information retrieval with search engines and collects valuable information for WebPlanner. The multi-agent design of MindSearch enables the whole framework to seek and integrate information parallelly from larger-scale (e.g., more than 300) web pages in 3 minutes, which is worth 3 hours of human effort. MindSearch demonstrates significant improvement in the response quality in terms of depth and breadth, on both close-set and open-set QA problems. Besides, responses from MindSearch based on InternLM2.5-7B are preferable by humans to ChatGPT-Web and this http URL applications, which implies that MindSearch can already deliver a competitive solution to the proprietary AI search engine.
• 摘要：信息查找和整合是一项复杂的认知任务，需要耗费大量的时间和精力。受大型语言模型的显著进步的启发，最近的研究尝试通过结合 LLM 和搜索引擎来解决此任务。然而，这些方法仍然无法令人满意，因为存在三个挑战：（1）搜索引擎通常无法准确、完整地检索复杂的请求；（2）需要整合的相应信息分散在多个网页上，并伴有大量噪音；（3）大量内容较长的网页可能很快就会超过 LLM 的最大上下文长度。受人类解决这些问题时的认知过程的启发，我们引入了 MindSearch 来模仿人类在网络信息查找和整合中的思维，它可以通过一个简单而有效的基于 LLM 的多智能体框架来实例化。WebPlanner 将人类多步骤信息查找的思维建模为动态图构建过程：它将用户查询分解为原子子问题作为图中的节点，并根据 WebSearcher 的搜索结果逐步扩展图。 WebSearcher 负责处理每个子问题，使用搜索引擎进行分层信息检索，并为 WebPlanner 收集有价值的信息。MindSearch 的多智能体设计使整个框架能够在 3 分钟内从更大规模（例如超过 300 个）网页并行查找和整合信息，这相当于人类 3 小时的努力。MindSearch 在封闭集和开放集 QA 问题上都表现出深度和广度方面的响应质量显著提高。此外，基于 InternLM2.5-7B 的 MindSearch 的响应比 ChatGPT-Web 和此 http URL 应用程序更受人类青睐，这意味着 MindSearch 已经可以为专有 AI 搜索引擎提供有竞争力的解决方案。

Title: QAEA-DR: A Unified Text Augmentation Framework for Dense Retrieval

• Authors: Hongming Tan, Shaoxiong Zhan, Hai Lin, Hai-Tao Zheng, Wai Kin (Victor)Chan
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2407.20207
• Pdf URL: https://arxiv.org/pdf/2407.20207
• Copy Paste: [[2407.20207]] QAEA-DR: A Unified Text Augmentation Framework for Dense Retrieval(https://arxiv.org/abs/2407.20207)
• Keywords: language model, llm, prompt
• Abstract: In dense retrieval, embedding long texts into dense vectors can result in information loss, leading to inaccurate query-text matching. Additionally, low-quality texts with excessive noise or sparse key information are unlikely to align well with relevant queries. Recent studies mainly focus on improving the sentence embedding model or retrieval process. In this work, we introduce a novel text augmentation framework for dense retrieval. This framework transforms raw documents into information-dense text formats, which supplement the original texts to effectively address the aforementioned issues without modifying embedding or retrieval methodologies. Two text representations are generated via large language models (LLMs) zero-shot prompting: question-answer pairs and element-driven events. We term this approach QAEA-DR: unifying question-answer generation and event extraction in a text augmentation framework for dense retrieval. To further enhance the quality of generated texts, a scoring-based evaluation and regeneration mechanism is introduced in LLM prompting. Our QAEA-DR model has a positive impact on dense retrieval, supported by both theoretical analysis and empirical experiments.
• 摘要：在密集检索中，将长文本嵌入密集向量会导致信息丢失，从而导致查询-文本匹配不准确。此外，含有过多噪声或稀疏关键信息的低质量文本不太可能与相关查询很好地匹配。最近的研究主要集中于改进句子嵌入模型或检索过程。在这项工作中，我们引入了一个用于密集检索的新型文本增强框架。该框架将原始文档转换为信息密集的文本格式，对原始文本进行补充，从有效解决上述问题开始，而无需修改嵌入或检索方法。通过大型语言模型 (LLM) 零样本提示生成两种文本表示：问答对和元素驱动事件。我们将这种方法称为 QAEA-DR：在密集检索的文本增强框架中统一问答生成和事件提取。为了进一步提高生成文本的质量，在 LLM 提示中引入了基于评分的评估和再生机制。我们的 QAEA-DR 模型对密集检索有积极影响，这得到了理论分析和实证实验的支持。

Title: Can Editing LLMs Inject Harm?

• Authors: Canyu Chen, Baixiang Huang, Zekun Li, Zhaorun Chen, Shiyang Lai, Xiongxiao Xu, Jia-Chen Gu, Jindong Gu, Huaxiu Yao, Chaowei Xiao, Xifeng Yan, William Yang Wang, Philip Torr, Dawn Song, Kai Shu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2407.20224
• Pdf URL: https://arxiv.org/pdf/2407.20224
• Copy Paste: [[2407.20224]] Can Editing LLMs Inject Harm?(https://arxiv.org/abs/2407.20224)
• Keywords: language model, llm
• Abstract: Knowledge editing techniques have been increasingly adopted to efficiently correct the false or outdated knowledge in Large Language Models (LLMs), due to the high cost of retraining from scratch. Meanwhile, one critical but under-explored question is: can knowledge editing be used to inject harm into LLMs? In this paper, we propose to reformulate knowledge editing as a new type of safety threat for LLMs, namely Editing Attack, and conduct a systematic investigation with a newly constructed dataset EditAttack. Specifically, we focus on two typical safety risks of Editing Attack including Misinformation Injection and Bias Injection. For the risk of misinformation injection, we first categorize it into commonsense misinformation injection and long-tail misinformation injection. Then, we find that editing attacks can inject both types of misinformation into LLMs, and the effectiveness is particularly high for commonsense misinformation injection. For the risk of bias injection, we discover that not only can biased sentences be injected into LLMs with high effectiveness, but also one single biased sentence injection can cause a high bias increase in general outputs of LLMs, which are even highly irrelevant to the injected sentence, indicating a catastrophic impact on the overall fairness of LLMs. Then, we further illustrate the high stealthiness of editing attacks, measured by their impact on the general knowledge and reasoning capacities of LLMs, and show the hardness of defending editing attacks with empirical evidence. Our discoveries demonstrate the emerging misuse risks of knowledge editing techniques on compromising the safety alignment of LLMs.
• 摘要：由于从头开始重新训练的成本高昂，知识编辑技术被越来越多地采用以有效地纠正大型语言模型（LLM）中的错误或过时知识。同时，一个关键但未被充分探索的问题是：知识编辑是否会对LLM造成危害？在本文中，我们提出将知识编辑重新表述为LLM的一种新型安全威胁，即编辑攻击，并使用新构建的数据集EditAttack进行系统调查。具体而言，我们关注编辑攻击的两种典型安全风险，包括错误信息注入和偏见注入。对于错误信息注入的风险，我们首先将其分为常识性错误信息注入和长尾错误信息注入。然后，我们发现编辑攻击可以将这两种类型的错误信息注入LLM，并且对于常识性错误信息注入的有效性尤其高。对于偏见注入的风险，我们发现，不仅可以高效地将有偏见的句子注入 LLM，而且单个有偏见的句子注入就会导致 LLM 的一般输出出现高偏差，这些输出甚至与注入的句子无关，这表明对 LLM 的整体公平性产生了灾难性的影响。然后，我们进一步说明了编辑攻击的高度隐蔽性，以它们对 LLM 的一般知识和推理能力的影响来衡量，并用经验证据展示了防御编辑攻击的难度。我们的发现表明，知识编辑技术在损害 LLM 安全性方面存在滥用风险。