2024-08-08

Title: ULLME: A Unified Framework for Large Language Model Embeddings with Generation-Augmented Learning

• Authors: Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Thien Huu Nguyen
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.03402
• Pdf URL: https://arxiv.org/pdf/2408.03402
• Copy Paste: [[2408.03402]] ULLME: A Unified Framework for Large Language Model Embeddings with Generation-Augmented Learning(https://arxiv.org/abs/2408.03402)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) excel in various natural language processing tasks, but leveraging them for dense passage embedding remains challenging. This is due to their causal attention mechanism and the misalignment between their pre-training objectives and the text ranking tasks. Despite some recent efforts to address these issues, existing frameworks for LLM-based text embeddings have been limited by their support for only a limited range of LLM architectures and fine-tuning strategies, limiting their practical application and versatility. In this work, we introduce the Unified framework for Large Language Model Embedding (ULLME), a flexible, plug-and-play implementation that enables bidirectional attention across various LLMs and supports a range of fine-tuning strategies. We also propose Generation-augmented Representation Learning (GRL), a novel fine-tuning method to boost LLMs for text embedding tasks. GRL enforces consistency between representation-based and generation-based relevance scores, leveraging LLMs' powerful generative abilities for learning passage embeddings. To showcase our framework's flexibility and effectiveness, we release three pre-trained models from ULLME with different backbone architectures, ranging from 1.5B to 8B parameters, all of which demonstrate strong performance on the Massive Text Embedding Benchmark. Our framework is publicly available at: this https URL. A demo video for ULLME can also be found at this https URL.
• 摘要：大型语言模型 (LLM) 在各种自然语言处理任务中表现出色，但利用它们进行密集段落嵌入仍然具有挑战性。这是由于它们的因果注意机制以及它们的预训练目标与文本排名任务之间的不一致。尽管最近做出了一些努力来解决这些问题，但现有的基于 LLM 的文本嵌入框架受到其仅支持有限范围的 LLM 架构和微调策略的限制，限制了它们的实际应用和多功能性。在这项工作中，我们引入了大型语言模型嵌入统一框架 (ULLME)，这是一种灵活的即插即用实现，可在各种 LLM 之间实现双向注意并支持一系列微调策略。我们还提出了生成增强表示学习 (GRL)，这是一种新颖的微调方法，可提升 LLM 的文本嵌入任务。GRL 强制基于表示和基于生成的相关性分数之间的一致性，利用 LLM 强大的生成能力来学习段落嵌入。为了展示我们框架的灵活性和有效性，我们发布了三个来自 ULLME 的预训练模型，它们具有不同的骨干架构，参数范围从 15 亿到 80 亿，所有这些模型在 Massive Text Embedding Benchmark 上都表现出色。我们的框架可在此 https URL 上公开获取。您还可以在此 https URL 上找到 ULLME 的演示视频。

Title: Logistic Regression makes small LLMs strong and explainable "tens-of-shot" classifiers

• Authors: Marcus Buckmann, Edward Hill
• Subjects: cs.CL, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.03414
• Pdf URL: https://arxiv.org/pdf/2408.03414
• Copy Paste: [[2408.03414]] Logistic Regression makes small LLMs strong and explainable "tens-of-shot" classifiers(https://arxiv.org/abs/2408.03414)
• Keywords: language model, llm
• Abstract: For simple classification tasks, we show that users can benefit from the advantages of using small, local, generative language models instead of large commercial models without a trade-off in performance or introducing extra labelling costs. These advantages, including those around privacy, availability, cost, and explainability, are important both in commercial applications and in the broader democratisation of AI. Through experiments on 17 sentence classification tasks (2-4 classes), we show that penalised logistic regression on the embeddings from a small LLM equals (and usually betters) the performance of a large LLM in the "tens-of-shot" regime. This requires no more labelled instances than are needed to validate the performance of the large LLM. Finally, we extract stable and sensible explanations for classification decisions.
• 摘要：对于简单的分类任务，我们表明用户可以从使用小型、本地、生成语言模型（而不是大型商业模型）的优势中受益，而无需牺牲性能或引入额外的标签成本。这些优势（包括隐私、可用性、成本和可解释性方面的优势）对于商业应用和更广泛的人工智能民主化都很重要。通过对 17 个句子分类任务（2-4 个类别）的实验，我们表明，在“数十次”模式下，小型 LLM 嵌入的惩罚逻辑回归等于（并且通常更好）大型 LLM 的性能。这不需要比验证大型 LLM 性能所需的更多标记实例。最后，我们为分类决策提取了稳定且合理的解释。

Title: Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation

• Authors: Weiqi Feng, Yangrui Chen, Shaoyu Wang, Yanghua Peng, Haibin Lin, Minlan Yu
• Subjects: cs.CL, cs.AI, cs.DC
• Abstract URL: https://arxiv.org/abs/2408.03505
• Pdf URL: https://arxiv.org/pdf/2408.03505
• Copy Paste: [[2408.03505]] Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation(https://arxiv.org/abs/2408.03505)
• Keywords: language model, gpt, llm
• Abstract: Multimodal large language models (MLLMs) have extended the success of large language models (LLMs) to multiple data types, such as image, text and audio, achieving significant performance in various domains, including multimodal translation, visual question answering and content generation. Nonetheless, existing systems are inefficient to train MLLMs due to substantial GPU bubbles caused by the heterogeneous modality models and complex data dependencies in 3D parallelism. This paper proposes Optimus, a distributed MLLM training system that reduces end-to-end MLLM training time. Optimus is based on our principled analysis that scheduling the encoder computation within the LLM bubbles can reduce bubbles in MLLM training. To make scheduling encoder computation possible for all GPUs, Optimus searches the separate parallel plans for encoder and LLM, and adopts a bubble scheduling algorithm to enable exploiting LLM bubbles without breaking the original data dependencies in the MLLM model architecture. We further decompose encoder layer computation into a series of kernels, and analyze the common bubble pattern of 3D parallelism to carefully optimize the sub-millisecond bubble scheduling, minimizing the overall training time. Our experiments in a production cluster show that Optimus accelerates MLLM training by 20.5%-21.3% with ViT-22B and GPT-175B model over 3072 GPUs compared to baselines.
• 摘要：多模态大型语言模型 (MLLM) 将大型语言模型 (LLM) 的成功扩展到多种数据类型，例如图像、文本和音频，并在多模态翻译、视觉问答和内容生成等各个领域取得了显著的性能。尽管如此，现有系统在训练 MLLM 方面效率低下，因为异构模态模型和 3D 并行中复杂的数据依赖关系导致了大量 GPU 气泡。本文提出了 Optimus，一种分布式 MLLM 训练系统，可减少端到端 MLLM 训练时间。Optimus 基于我们的原则分析，即在 LLM 气泡内安排编码器计算可以减少 MLLM 训练中的气泡。为了使所有 GPU 都能安排编码器计算，Optimus 搜索编码器和 LLM 的单独并行计划，并采用气泡调度算法来利用 LLM 气泡，而不会破坏 MLLM 模型架构中原始的数据依赖关系。我们进一步将编码器层计算分解为一系列内核，并分析 3D 并行的常见气泡模式，以精心优化亚毫秒气泡调度，从而最大限度地缩短整体训练时间。我们在生产集群中的实验表明，与基线相比，Optimus 在 3072 个 GPU 上使用 ViT-22B 和 GPT-175B 模型将 MLLM 训练速度提高了 20.5%-21.3%。

Title: 1.5-Pints Technical Report: Pretraining in Days, Not Months -- Your Language Model Thrives on Quality Data

• Authors: Calvin Tan, Jerome Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.03506
• Pdf URL: https://arxiv.org/pdf/2408.03506
• Copy Paste: [[2408.03506]] 1.5-Pints Technical Report: Pretraining in Days, Not Months -- Your Language Model Thrives on Quality Data(https://arxiv.org/abs/2408.03506)
• Keywords: language model, llm
• Abstract: This paper presents a compute-efficient approach to pre-training a Language Model-the "1.5-Pints"-in only 9 days, while outperforming state-of-the-art models as an instruction-following assistant.Based on MT-Bench (a benchmark that emulates human judgments), 1.5-Pints outperforms Apple's OpenELM and Microsoft's Phi.This is achieved by a carefully curated pre-training dataset of 57 billion tokens, using a mix of automated workflows and manual human review. The selection of the dataset prioritizes content that is considered expository and "textbook-like" to aid the model in reasoning and logical deduction, culminating in its overall ability as a strong and versatile AI model. In terms of the model architecture, we employed a modified Mistral tokenizer, alongside a Llama-2 architecture for wider compatibility. For training, we adopted the methodologies used by StableLM, TinyLlama, and Huggingface Zephyr. 1.5-Pints demonstrates that by focusing on data quality over quantity in LLM training, we can significantly reduce training time and resources required. We believe this approach will not only make pre-training more accessible but also reduce our carbon footprint. Our findings and resources from this research are open-sourced, aiming to facilitate further advancements in the field. The 1.5-Pints model is available in two versions: 2K and 16K context windows.
• 摘要：本文介绍了一种计算效率高的方法，仅用 9 天就预训练了语言模型“1.5-Pints”，同时作为指令遵循助手，其表现优于最先进的模型。基于 MT-Bench（一种模拟人类判断的基准），1.5-Pints 的表现优于 Apple 的 OpenELM 和 Microsoft 的 Phi。这是通过精心策划的 570 亿个标记的预训练数据集实现的，该数据集使用了自动化工作流程和人工审核相结合的方式。数据集的选择优先考虑被认为是说明性和“教科书式”的内容，以帮助模型进行推理和逻辑推理，最终使其整体能力成为强大而多功能的 AI 模型。在模型架构方面，我们采用了经过修改的 Mistral 标记器，以及 Llama-2 架构，以实现更广泛的兼容性。对于训练，我们采用了 StableLM、TinyLlama 和 Huggingface Zephyr 使用的方法。 1.5-Pints 表明，通过在 LLM 培训中注重数据质量而非数量，我们可以显著减少所需的培训时间和资源。我们相信这种方法不仅会让预训练更容易实现，而且还会减少我们的碳足迹。我们从这项研究中获得的发现和资源都是开源的，旨在促进该领域的进一步发展。1.5-Pints 模型有两个版本：2K 和 16K 上下文窗口。

Title: EgyBERT: A Large Language Model Pretrained on Egyptian Dialect Corpora

• Authors: Faisal Qarah
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.03524
• Pdf URL: https://arxiv.org/pdf/2408.03524
• Copy Paste: [[2408.03524]] EgyBERT: A Large Language Model Pretrained on Egyptian Dialect Corpora(https://arxiv.org/abs/2408.03524)
• Keywords: language model
• Abstract: This study presents EgyBERT, an Arabic language model pretrained on 10.4 GB of Egyptian dialectal texts. We evaluated EgyBERT's performance by comparing it with five other multidialect Arabic language models across 10 evaluation datasets. EgyBERT achieved the highest average F1-score of 84.25% and an accuracy of 87.33%, significantly outperforming all other comparative models, with MARBERTv2 as the second best model achieving an F1-score 83.68% and an accuracy 87.19%. Additionally, we introduce two novel Egyptian dialectal corpora: the Egyptian Tweets Corpus (ETC), containing over 34.33 million tweets (24.89 million sentences) amounting to 2.5 GB of text, and the Egyptian Forums Corpus (EFC), comprising over 44.42 million sentences (7.9 GB of text) collected from various Egyptian online forums. Both corpora are used in pretraining the new model, and they are the largest Egyptian dialectal corpora to date reported in the literature. Furthermore, this is the first study to evaluate the performance of various language models on Egyptian dialect datasets, revealing significant differences in performance that highlight the need for more dialect-specific models. The results confirm the effectiveness of EgyBERT model in processing and analyzing Arabic text expressed in Egyptian dialect, surpassing other language models included in the study. EgyBERT model is publicly available on \url{this https URL}.
• 摘要：本研究介绍了 EgyBERT，这是一种基于 10.4 GB 埃及方言文本进行预训练的阿拉伯语语言模型。我们通过将 EgyBERT 与 10 个评估数据集中的其他五种多方言阿拉伯语语言模型进行比较来评估其性能。EgyBERT 的平均 F1 得分最高，为 84.25%，准确率为 87.33%，远远优于所有其他比较模型，MARBERTv2 作为第二好的模型，F1 得分为 83.68%，准确率为 87.19%。此外，我们引入了两个新的埃及方言语料库：埃及推文语料库 (ETC)，包含超过 3433 万条推文（2489 万句），共计 2.5 GB 文本，以及埃及论坛语料库 (EFC)，包含从各种埃及在线论坛收集的超过 4442 万句（7.9 GB 文本）。这两个语料库都用于对新模型进行预训练，它们是迄今为止文献中报道的最大的埃及方言语料库。此外，这是首次评估各种语言模型在埃及方言数据集上的表现的研究，揭示了性能上的显著差异，凸显了对更多方言特定模型的需求。结果证实了 EgyBERT 模型在处理和分析用埃及方言表达的阿拉伯语文本方面的有效性，超过了研究中包括的其他语言模型。EgyBERT 模型在 \url{此 https URL} 上公开可用。

Title: EXAONE 3.0 7.8B Instruction Tuned Language Model

• Authors: LG AI Research, Soyoung An, Kyunghoon Bae, Eunbi Choi, Stanley Jungkyu Choi, Yemuk Choi, Seokhee Hong, Yeonjung Hong, Junwon Hwang, Hyojin Jeon, Gerrard Jeongwon Jo, Hyunjik Jo, Jiyeon Jung, Yountae Jung, Euisoon Kim, Hyosang Kim, Joonkee Kim, Seonghwan Kim, Soyeon Kim, Sunkyoung Kim, Yireun Kim, Youchul Kim, Edward Hwayoung Lee, Haeju Lee, Honglak Lee, Jinsik Lee, Kyungmin Lee, Moontae Lee, Seungjun Lee, Woohyung Lim, Sangha Park, Sooyoun Park, Yongmin Park, Boseong Seo, Sihoon Yang, Heuiyeen Yeen, Kyungjae Yoo, Hyeongu Yun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03541
• Pdf URL: https://arxiv.org/pdf/2408.03541
• Copy Paste: [[2408.03541]] EXAONE 3.0 7.8B Instruction Tuned Language Model(https://arxiv.org/abs/2408.03541)
• Keywords: language model, llm
• Abstract: We introduce EXAONE 3.0 instruction-tuned language model, the first open model in the family of Large Language Models (LLMs) developed by LG AI Research. Among different model sizes, we publicly release the 7.8B instruction-tuned model to promote open research and innovations. Through extensive evaluations across a wide range of public and in-house benchmarks, EXAONE 3.0 demonstrates highly competitive real-world performance with instruction-following capability against other state-of-the-art open models of similar size. Our comparative analysis shows that EXAONE 3.0 excels particularly in Korean, while achieving compelling performance across general tasks and complex reasoning. With its strong real-world effectiveness and bilingual proficiency, we hope that EXAONE keeps contributing to advancements in Expert AI. Our EXAONE 3.0 instruction-tuned model is available at this https URL
• 摘要：我们推出了 EXAONE 3.0 指令调优语言模型，这是 LG AI Research 开发的大型语言模型 (LLM) 系列中的第一个开放模型。在不同的模型大小中，我们公开发布了 7.8B 指令调优模型，以促进开放研究和创新。通过对各种公共和内部基准的广泛评估，EXAONE 3.0 与其他类似规模的先进开放模型相比，在指令跟踪能力方面表现出了极具竞争力的实际性能。我们的比较分析表明，EXAONE 3.0 在韩语方面尤其出色，同时在一般任务和复杂推理方面也取得了令人信服的表现。凭借其强大的实际有效性和双语能力，我们希望 EXAONE 能够继续为专家 AI 的进步做出贡献。我们的 EXAONE 3.0 指令调优模型可在此 https URL 上找到

Title: Unlocking the Non-Native Language Context Limitation: Native Language Prompting Facilitates Knowledge Elicitation

• Authors: Baixuan Li, Yunlong Fan, Zhiqiang Gao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03544
• Pdf URL: https://arxiv.org/pdf/2408.03544
• Copy Paste: [[2408.03544]] Unlocking the Non-Native Language Context Limitation: Native Language Prompting Facilitates Knowledge Elicitation(https://arxiv.org/abs/2408.03544)
• Keywords: language model, llm, prompt
• Abstract: Multilingual large language models (MLLMs) struggle to answer questions posed in non-dominant languages, even though they have already acquired the relevant knowledge from their dominant language corpus. In contrast, human multilinguals can overcome this issue by invoking the relatively rich knowledge acquired from native language texts through Positive Native Language Transfer (PNLT). Inspired by this, we analogize the dominant language of MLLMs to the native language of human multilinguals, and propose Native Language Prompting (NatLan) to simulate the PNLT observed in human multilinguals. It explicitly creates native language contexts for MLLMs to facilitate the elicitation of the rich native language knowledge during question-answering, unlocking the limitations imposed by non-native language contexts on the effective application of knowledge. By employing multi-MLLM collaboration, NatLan reduces the workload on each MLLM in simulating PNLT and refines semantic transfer. On the C-Eval benchmark, NatLan provides up to a 10.1% average accuracy improvement and up to a 5.0% increase in the hard-level subset across five MLLMs, surpassing all top-notch related methods. Our code is available at this https URL.
• 摘要：多语言大型语言模型 (MLLM) 很难回答非主导语言提出的问题，即使它们已经从主导语言语料库中获得了相关知识。相比之下，人类多语言者可以通过积极母语迁移 (PNLT) 调用从母语文本中获得的相对丰富的知识来克服这个问题。受此启发，我们将 MLLM 的主导语言类比为人类多语言者的母语，并提出母语提示 (NatLan) 来模拟人类多语言者中观察到的 PNLT。它明确为 MLLM 创建母语上下文，以促进在问答过程中引出丰富的母语知识，从而打破非母语上下文对有效应用知识的限制。通过采用多 MLLM 协作，NatLan 减少了每个 MLLM 在模拟 PNLT 时的工作量并改进了语义迁移。在 C-Eval 基准测试中，NatLan 在五个 MLLM 中的平均准确率提高了 10.1%，在硬级子集上提高了 5.0%，超越了所有顶级相关方法。我们的代码可在此 https URL 上找到。

Title: Empirical Analysis of Large Vision-Language Models against Goal Hijacking via Visual Prompt Injection

• Authors: Subaru Kimura, Ryota Tanaka, Shumpei Miyawaki, Jun Suzuki, Keisuke Sakaguchi
• Subjects: cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03554
• Pdf URL: https://arxiv.org/pdf/2408.03554
• Copy Paste: [[2408.03554]] Empirical Analysis of Large Vision-Language Models against Goal Hijacking via Visual Prompt Injection(https://arxiv.org/abs/2408.03554)
• Keywords: language model, gpt, prompt
• Abstract: We explore visual prompt injection (VPI) that maliciously exploits the ability of large vision-language models (LVLMs) to follow instructions drawn onto the input image. We propose a new VPI method, "goal hijacking via visual prompt injection" (GHVPI), that swaps the execution task of LVLMs from an original task to an alternative task designated by an attacker. The quantitative analysis indicates that GPT-4V is vulnerable to the GHVPI and demonstrates a notable attack success rate of 15.8%, which is an unignorable security risk. Our analysis also shows that successful GHVPI requires high character recognition capability and instruction-following ability in LVLMs.
• 摘要：我们探索了视觉提示注入 (VPI)，它恶意利用大型视觉语言模型 (LVLM) 遵循绘制在输入图像上的指令的能力。我们提出了一种新的 VPI 方法，即“通过视觉提示注入进行目标劫持”(GHVPI)，它将 LVLM 的执行任务从原始任务交换为攻击者指定的替代任务。定量分析表明，GPT-4V 容易受到 GHVPI 的攻击，攻击成功率高达 15.8%，这是一个不容忽视的安全风险。我们的分析还表明，成功的 GHVPI 需要 LVLM 具有较高的字符识别能力和指令遵循能力。

Title: A Comparison of LLM Finetuning Methods & Evaluation Metrics with Travel Chatbot Use Case

• Authors: Sonia Meyer, Shreya Singh, Bertha Tam, Christopher Ton, Angel Ren
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03562
• Pdf URL: https://arxiv.org/pdf/2408.03562
• Copy Paste: [[2408.03562]] A Comparison of LLM Finetuning Methods & Evaluation Metrics with Travel Chatbot Use Case(https://arxiv.org/abs/2408.03562)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: This research compares large language model (LLM) fine-tuning methods, including Quantized Low Rank Adapter (QLoRA), Retrieval Augmented fine-tuning (RAFT), and Reinforcement Learning from Human Feedback (RLHF), and additionally compared LLM evaluation methods including End to End (E2E) benchmark method of "Golden Answers", traditional natural language processing (NLP) metrics, RAG Assessment (Ragas), OpenAI GPT-4 evaluation metrics, and human evaluation, using the travel chatbot use case. The travel dataset was sourced from the the Reddit API by requesting posts from travel-related subreddits to get travel-related conversation prompts and personalized travel experiences, and augmented for each fine-tuning method. We used two pretrained LLMs utilized for fine-tuning research: LLaMa 2 7B, and Mistral 7B. QLoRA and RAFT are applied to the two pretrained models. The inferences from these models are extensively evaluated against the aforementioned metrics. The best model according to human evaluation and some GPT-4 metrics was Mistral RAFT, so this underwent a Reinforcement Learning from Human Feedback (RLHF) training pipeline, and ultimately was evaluated as the best model. Our main findings are that: 1) quantitative and Ragas metrics do not align with human evaluation, 2) Open AI GPT-4 evaluation most aligns with human evaluation, 3) it is essential to keep humans in the loop for evaluation because, 4) traditional NLP metrics insufficient, 5) Mistral generally outperformed LLaMa, 6) RAFT outperforms QLoRA, but still needs postprocessing, 7) RLHF improves model performance significantly. Next steps include improving data quality, increasing data quantity, exploring RAG methods, and focusing data collection on a specific city, which would improve data quality by narrowing the focus, while creating a useful product.
• 摘要：本研究比较了大型语言模型 (LLM) 微调方法，包括量化低秩适配器 (QLoRA)、检索增强微调 (RAFT) 和从人类反馈中强化学习 (RLHF)，此外还比较了 LLM 评估方法，包括“黄金答案”的端到端 (E2E) 基准方法、传统自然语言处理 (NLP) 指标、RAG 评估 (Ragas)、OpenAI GPT-4 评估指标和人类评估，使用旅行聊天机器人用例。旅行数据集来自 Reddit API，通过请求旅行相关子版块的帖子来获取旅行相关的对话提示和个性化旅行体验，并针对每种微调方法进行增强。我们使用了两个用于微调研究的预训练 LLM：LLaMa 2 7B 和 Mistral 7B。QLoRA 和 RAFT 应用于这两个预训练模型。根据上述指标对这些模型的推论进行了广泛的评估。根据人工评估和一些 GPT-4 指标，最佳模型是 Mistral RAFT，因此该模型经过了人工反馈强化学习 (RLHF) 训练流程，最终被评为最佳模型。我们的主要发现是：1) 定量和 Ragas 指标与人工评估不一致，2) Open AI GPT-4 评估与人工评估最一致，3) 让人类参与评估至关重要，因为 4) 传统的 NLP 指标不足，5) Mistral 的表现通常优于 LLaMa，6) RAFT 的表现优于 QLoRA，但仍需要后处理，7) RLHF 显著提高了模型性能。接下来的步骤包括提高数据质量、增加数据量、探索 RAG 方法以及将数据收集重点放在特定城市，这将通过缩小焦点来提高数据质量，同时创建有用的产品。

Title: Is Child-Directed Speech Effective Training Data for Language Models?

• Authors: Steven Y. Feng, Noah D. Goodman, Michael C. Frank
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03617
• Pdf URL: https://arxiv.org/pdf/2408.03617
• Copy Paste: [[2408.03617]] Is Child-Directed Speech Effective Training Data for Language Models?(https://arxiv.org/abs/2408.03617)
• Keywords: language model, gpt
• Abstract: While high-performing language models are typically trained on hundreds of billions of words, human children become fluent language users with a much smaller amount of data. What are the features of the data they receive, and how do these features support language modeling objectives? To investigate this question, we train GPT-2 models on 29M words of English-language child-directed speech and a new matched, synthetic dataset (TinyDialogues), comparing to a heterogeneous blend of datasets from the BabyLM challenge. We evaluate both the syntactic and semantic knowledge of these models using developmentally-inspired evaluations. Through pretraining experiments, we test whether the global developmental ordering or the local discourse ordering of children's training data support high performance relative to other datasets. The local properties of the data affect model results, but somewhat surprisingly, global properties do not. Further, child language input is not uniquely valuable for training language models. These findings support the hypothesis that, rather than proceeding from better data, children's learning is instead substantially more efficient than current language modeling techniques.
• 摘要：虽然高性能语言模型通常需要数千亿个单词进行训练，但人类儿童只需要很少的数据就能成为流利的语言使用者。他们收到的数据有哪些特征，这些特征如何支持语言建模目标？为了研究这个问题，我们在 2900 万个英语儿童导向语音单词和一个新的匹配合成数据集 (TinyDialogues) 上训练 GPT-2 模型，并与 BabyLM 挑战赛中的异构混合数据集进行比较。我们使用受发展启发的评估来评估这些模型的句法和语义知识。通过预训练实验，我们测试儿童训练数据的全局发展排序或局部话语排序是否支持相对于其他数据集的高性能。数据的局部属性会影响模型结果，但有点令人惊讶的是，全局属性不会。此外，儿童语言输入对于训练语言模型并不是独一无二的。这些发现支持了这样的假设：儿童的学习不是从更好的数据开始，而是比当前的语言建模技术效率高得多。

Title: A Logical Fallacy-Informed Framework for Argument Generation

• Authors: Luca Mouchel, Debjit Paul, Shaobo Cui, Robert West, Antoine Bosselut, Boi Faltings
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03618
• Pdf URL: https://arxiv.org/pdf/2408.03618
• Copy Paste: [[2408.03618]] A Logical Fallacy-Informed Framework for Argument Generation(https://arxiv.org/abs/2408.03618)
• Keywords: language model, llm
• Abstract: Despite the remarkable performance of Large Language Models (LLMs), they still struggle with generating logically sound arguments, resulting in potential risks such as spreading misinformation. An important factor contributing to LLMs' suboptimal performance in generating coherent arguments is their oversight of logical fallacies. To address this issue, we introduce FIPO, a fallacy-informed framework that leverages preference optimization methods to steer LLMs toward logically sound arguments. FIPO includes a classification loss, to capture the fine-grained information on fallacy categories. Our results on argumentation datasets show that our method reduces the fallacy errors by up to 17.5%. Furthermore, our human evaluation results indicate that the quality of the generated arguments by our method significantly outperforms the fine-tuned baselines, as well as prior preference optimization methods, such as DPO. These findings highlight the importance of ensuring models are aware of logical fallacies for effective argument generation.
• 摘要：尽管大型语言模型 (LLM) 性能卓越，但它们在生成逻辑合理的论证方面仍存在困难，从而导致传播错误信息等潜在风险。导致 LLM 在生成连贯论证方面表现不佳的一个重要因素是它们忽视了逻辑谬误。为了解决这个问题，我们引入了 FIPO，这是一个谬误知情框架，它利用偏好优化方法来引导 LLM 走向逻辑合理的论证。FIPO 包括分类损失，以捕获谬误类别的细粒度信息。我们在论证数据集上的结果表明，我们的方法将谬误错误减少了多达 17.5%。此外，我们的人工评估结果表明，我们的方法生成的论证质量明显优于微调基线以及之前的偏好优化方法，例如 DPO。这些发现强调了确保模型意识到逻辑谬误对于有效生成论证的重要性。

Title: PAGED: A Benchmark for Procedural Graphs Extraction from Documents

• Authors: Weihong Du, Wenrui Liao, Hongru Liang, Wenqiang Lei
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.03630
• Pdf URL: https://arxiv.org/pdf/2408.03630
• Copy Paste: [[2408.03630]] PAGED: A Benchmark for Procedural Graphs Extraction from Documents(https://arxiv.org/abs/2408.03630)
• Keywords: language model, llm
• Abstract: Automatic extraction of procedural graphs from documents creates a low-cost way for users to easily understand a complex procedure by skimming visual graphs. Despite the progress in recent studies, it remains unanswered: whether the existing studies have well solved this task (Q1) and whether the emerging large language models (LLMs) can bring new opportunities to this task (Q2). To this end, we propose a new benchmark PAGED, equipped with a large high-quality dataset and standard evaluations. It investigates five state-of-the-art baselines, revealing that they fail to extract optimal procedural graphs well because of their heavy reliance on hand-written rules and limited available data. We further involve three advanced LLMs in PAGED and enhance them with a novel self-refine strategy. The results point out the advantages of LLMs in identifying textual elements and their gaps in building logical structures. We hope PAGED can serve as a major landmark for automatic procedural graph extraction and the investigations in PAGED can offer insights into the research on logic reasoning among non-sequential elements.
• 摘要：自动从文档中提取过程图为用户提供了一种低成本的方式，通过浏览可视化图表，用户可以轻松理解复杂的过程。尽管最近的研究取得了进展，但仍有未解之谜：现有研究是否很好地解决了这一任务（问题 1），新兴的大型语言模型 (LLM) 是否可以为这一任务带来新的机会（问题 2）。为此，我们提出了一个新的基准 PAGED，配备了大量高质量数据集和标准评估。它调查了五个最先进的基线，发现它们由于严重依赖手写规则和有限的可用数据而无法很好地提取最佳过程图。我们进一步在 PAGED 中引入了三个高级 LLM，并通过一种新颖的自我改进策略对其进行了增强。结果指出了 LLM 在识别文本元素方面的优势及其在构建逻辑结构方面的差距。我们希望 PAGED 可以成为自动过程图提取的重要里程碑，并且 PAGED 中的研究可以为非序列元素之间的逻辑推理研究提供见解。

Title: NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time

• Authors: Yilong Chen, Guoxia Wang, Junyuan Shang, Shiyao Cui, Zhenyu Zhang, Tingwen Liu, Shuohuan Wang, Yu Sun, Dianhai Yu, Hua Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.03675
• Pdf URL: https://arxiv.org/pdf/2408.03675
• Copy Paste: [[2408.03675]] NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time(https://arxiv.org/abs/2408.03675)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have ignited an innovative surge of AI applications, marking a new era of exciting possibilities equipped with extended context windows. However, hosting these models is cost-prohibitive mainly due to the extensive memory consumption of KV Cache involving long-context modeling. Despite several works proposing to evict unnecessary tokens from the KV Cache, most of them rely on the biased local statistics of accumulated attention scores and report performance using unconvincing metric like perplexity on inadequate short-text evaluation. In this paper, we propose NACL, a general framework for long-context KV cache eviction that achieves more optimal and efficient eviction in a single operation during the encoding phase. Due to NACL's efficiency, we combine more accurate attention score statistics in PROXY TOKENS EVICTION with the diversified random eviction strategy of RANDOM EVICTION, aiming to alleviate the issue of attention bias and enhance the robustness in maintaining pivotal tokens for long-context modeling tasks. Notably, our method significantly improves the performance on short- and long-text tasks by 80% and 76% respectively, reducing KV Cache by up to 50% with over 95% performance maintenance. The code is available at https: //github.com/PaddlePaddle/Research/ tree/master/NLP/ACL2024-NACL.
• 摘要：大型语言模型 (LLM) 引发了 AI 应用的创新浪潮，标志着一个配备扩展上下文窗口的激动人心的新时代。然而，托管这些模型的成本高昂，主要是因为涉及长上下文建模的 KV Cache 会消耗大量内存。尽管有几篇论文提出从 KV Cache 中逐出不必要的标记，但它们中的大多数依赖于累积注意力分数的有偏差的局部统计数据，并使用难以信服的指标（如困惑度）在不充分的短文本评估上报告性能。在本文中，我们提出了 NACL，这是一个用于长上下文 KV 缓存逐出的通用框架，可在编码阶段通过单个操作实现更优化和更高效的逐出。由于 NACL 的效率，我们将 PROXY TOKENS EVICTION 中更准确的注意力分数统计数据与 RANDOM EVICTION 的多样化随机逐出策略相结合，旨在缓解注意力偏差问题并增强在长上下文建模任务中维护关键标记的稳健性。值得注意的是，我们的方法在短文本和长文本任务上的性能分别显著提升了 80% 和 76%，在性能保持率超过 95% 的情况下，KV Cache 的使用量减少了 50%。代码可从 https://github.com/PaddlePaddle/Research/tree/master/NLP/ACL2024-NACL 获取。

Title: Local Topology Measures of Contextual Language Model Latent Spaces With Applications to Dialogue Term Extraction

• Authors: Benjamin Matthias Ruppik, Michael Heck, Carel van Niekerk, Renato Vukovic, Hsien-chin Lin, Shutong Feng, Marcus Zibrowius, Milica Gašić
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03706
• Pdf URL: https://arxiv.org/pdf/2408.03706
• Copy Paste: [[2408.03706]] Local Topology Measures of Contextual Language Model Latent Spaces With Applications to Dialogue Term Extraction(https://arxiv.org/abs/2408.03706)
• Keywords: language model
• Abstract: A common approach for sequence tagging tasks based on contextual word representations is to train a machine learning classifier directly on these embedding vectors. This approach has two shortcomings. First, such methods consider single input sequences in isolation and are unable to put an individual embedding vector in relation to vectors outside the current local context of use. Second, the high performance of these models relies on fine-tuning the embedding model in conjunction with the classifier, which may not always be feasible due to the size or inaccessibility of the underlying feature-generation model. It is thus desirable, given a collection of embedding vectors of a corpus, i.e., a datastore, to find features of each vector that describe its relation to other, similar vectors in the datastore. With this in mind, we introduce complexity measures of the local topology of the latent space of a contextual language model with respect to a given datastore. The effectiveness of our features is demonstrated through their application to dialogue term extraction. Our work continues a line of research that explores the manifold hypothesis for word embeddings, demonstrating that local structure in the space carved out by word embeddings can be exploited to infer semantic properties.
• 摘要：基于上下文词表示的序列标记任务的常用方法是直接在这些嵌入向量上训练机器学习分类器。这种方法有两个缺点。首先，这种方法孤立地考虑单个输入序列，无法将单个嵌入向量与当前本地使用上下文之外的向量联系起来。其次，这些模型的高性能依赖于与分类器一起微调嵌入模型，由于底层特征生成模型的大小或不可访问性，这可能并不总是可行的。因此，给定一个语料库（即数据存储）的嵌入向量集合，最好找到每个向量的特征来描述它与数据存储中其他类似向量的关系。考虑到这一点，我们引入了上下文语言模型潜在空间局部拓扑相对于给定数据存储的复杂性度量。通过将它们应用于对话术语提取，证明了我们的特征的有效性。我们的工作延续了探索词嵌入的流形假设的研究方向，证明了词嵌入所划分的空间中的局部结构可以用来推断语义属性。

Title: Question Rephrasing for Quantifying Uncertainty in Large Language Models: Applications in Molecular Chemistry Tasks

• Authors: Zizhang Chen, Pengyu Hong, Sandeep Madireddy
• Subjects: cs.CL, cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2408.03732
• Pdf URL: https://arxiv.org/pdf/2408.03732
• Copy Paste: [[2408.03732]] Question Rephrasing for Quantifying Uncertainty in Large Language Models: Applications in Molecular Chemistry Tasks(https://arxiv.org/abs/2408.03732)
• Keywords: language model, llm
• Abstract: Uncertainty quantification enables users to assess the reliability of responses generated by large language models (LLMs). We present a novel Question Rephrasing technique to evaluate the input uncertainty of LLMs, which refers to the uncertainty arising from equivalent variations of the inputs provided to LLMs. This technique is integrated with sampling methods that measure the output uncertainty of LLMs, thereby offering a more comprehensive uncertainty assessment. We validated our approach on property prediction and reaction prediction for molecular chemistry tasks.
• 摘要：不确定性量化使用户能够评估大型语言模型 (LLM) 生成的响应的可靠性。我们提出了一种新颖的问题改写技术来评估 LLM 的输入不确定性，这是指由提供给 LLM 的输入的等效变化引起的不确定性。该技术与测量 LLM 输出不确定性的采样方法相结合，从而提供更全面的不确定性评估。我们在分子化学任务的性质预测和反应预测方面验证了我们的方法。

Title: 'Finance Wizard' at the FinLLM Challenge Task: Financial Text Summarization

• Authors: Meisin Lee, Soon Lay-Ki
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.03762
• Pdf URL: https://arxiv.org/pdf/2408.03762
• Copy Paste: [[2408.03762]] 'Finance Wizard' at the FinLLM Challenge Task: Financial Text Summarization(https://arxiv.org/abs/2408.03762)
• Keywords: llm, agent
• Abstract: This paper presents our participation under the team name `Finance Wizard' in the FinNLP-AgentScen 2024 shared task #2: Financial Text Summarization. It documents our pipeline approach of fine-tuning a foundation model into a task-specific model for Financial Text Summarization. It involves (1) adapting Llama3 8B, a foundation model, to the Finance domain via continued pre-training, (2) multi-task instruction-tuning to further equip the model with more finance-related capabilities, (3) finally fine-tuning the model into a task-specific `expert'. Our model, FinLlama3\_sum, yielded commendable results, securing the third position in its category with a ROUGE-1 score of 0.521.
• 摘要：本文介绍了我们以“金融奇才”的名义参与 FinNLP-AgentScen 2024 共享任务 #2：金融文本摘要的情况。它记录了我们将基础模型微调为金融文本摘要任务特定模型的流水线方法。它涉及 (1) 通过持续的预训练将基础模型 Llama3 8B 调整到金融领域，(2) 多任务指令调整以进一步为模型配备更多与金融相关的功能，(3) 最后将模型微调为任务特定的“专家”。我们的模型 FinLlama3\_sum 取得了令人称赞的结果，以 0.521 的 ROUGE-1 得分在其类别中排名第三。

Title: Generative Language Models with Retrieval Augmented Generation for Automated Short Answer Scoring

• Authors: Zifan Wang, Christopher Ormerod
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.03811
• Pdf URL: https://arxiv.org/pdf/2408.03811
• Copy Paste: [[2408.03811]] Generative Language Models with Retrieval Augmented Generation for Automated Short Answer Scoring(https://arxiv.org/abs/2408.03811)
• Keywords: language model, prompt, retrieval augmented generation
• Abstract: Automated Short Answer Scoring (ASAS) is a critical component in educational assessment. While traditional ASAS systems relied on rule-based algorithms or complex deep learning methods, recent advancements in Generative Language Models (GLMs) offer new opportunities for improvement. This study explores the application of GLMs to ASAS, leveraging their off-the-shelf capabilities and performance in various domains. We propose a novel pipeline that combines vector databases, transformer-based encoders, and GLMs to enhance short answer scoring accuracy. Our approach stores training responses in a vector database, retrieves semantically similar responses during inference, and employs a GLM to analyze these responses and determine appropriate scores. We further optimize the system through fine-tuned retrieval processes and prompt engineering. Evaluation on the SemEval 2013 dataset demonstrates a significant improvement on the SCIENTSBANK 3-way and 2-way tasks compared to existing methods, highlighting the potential of GLMs in advancing ASAS technology.
• 摘要：自动简答题评分 (ASAS) 是教育评估中的关键组成部分。虽然传统的 ASAS 系统依赖于基于规则的算法或复杂的深度学习方法，但生成语言模型 (GLM) 的最新进展提供了新的改进机会。本研究探讨了 GLM 在 ASAS 中的应用，利用其在各个领域的现成功能和性能。我们提出了一种新颖的管道，结合了矢量数据库、基于转换器的编码器和 GLM，以提高简答题评分的准确性。我们的方法将训练响应存储在矢量数据库中，在推理过程中检索语义相似的响应，并使用 GLM 分析这些响应并确定适当的分数。我们通过微调检索过程和快速工程进一步优化系统。在 SemEval 2013 数据集上的评估表明，与现有方法相比，SCIENTSBANK 3 向和 2 向任务有显著改进，凸显了 GLM 在推进 ASAS 技术方面的潜力。

Title: WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language Models

• Authors: Prannaya Gupta, Le Qi Yau, Hao Han Low, I-Shiang Lee, Hugo Maximus Lim, Yu Xin Teoh, Jia Hng Koh, Dar Win Liew, Rishabh Bhardwaj, Rajat Bhardwaj, Soujanya Poria
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03837
• Pdf URL: https://arxiv.org/pdf/2408.03837
• Copy Paste: [[2408.03837]] WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language Models(https://arxiv.org/abs/2408.03837)
• Keywords: language model, llm, prompt
• Abstract: WalledEval is a comprehensive AI safety testing toolkit designed to evaluate large language models (LLMs). It accommodates a diverse range of models, including both open-weight and API-based ones, and features over 35 safety benchmarks covering areas such as multilingual safety, exaggerated safety, and prompt injections. The framework supports both LLM and judge benchmarking, and incorporates custom mutators to test safety against various text-style mutations such as future tense and paraphrasing. Additionally, WalledEval introduces WalledGuard, a new, small and performant content moderation tool, and SGXSTest, a benchmark for assessing exaggerated safety in cultural contexts. We make WalledEval publicly available at this https URL.
• 摘要：WalledEval 是一款全面的 AI 安全测试工具包，旨在评估大型语言模型 (LLM)。它适用于各种模型，包括开放权重和基于 API 的模型，并具有 35 多个安全基准，涵盖多语言安全性、夸大安全性和提示注入等领域。该框架支持 LLM 和 Judge 基准测试，并结合自定义变量来测试安全性，以应对各种文本样式的突变，例如将来时态和释义。此外，WalledEval 还引入了 WalledGuard（一种新的、小型且性能良好的内容审核工具）和 SGXSTest（一种用于评估文化背景下的夸大安全性的基准）。我们在此 https URL 上公开提供 WalledEval。

Title: BeeManc at the PLABA Track of TAC-2023: Investigating LLMs and Controllable Attributes for Improving Biomedical Text Readability

• Authors: Zihao Li, Samuel Belkadi, Nicolo Micheletti, Lifeng Han, Matthew Shardlow, Goran Nenadic
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03871
• Pdf URL: https://arxiv.org/pdf/2408.03871
• Copy Paste: [[2408.03871]] BeeManc at the PLABA Track of TAC-2023: Investigating LLMs and Controllable Attributes for Improving Biomedical Text Readability(https://arxiv.org/abs/2408.03871)
• Keywords: gpt, llm, prompt, chat
• Abstract: In this system report, we describe the models and methods we used for our participation in the PLABA2023 task on biomedical abstract simplification, part of the TAC 2023 tracks. The system outputs we submitted come from the following three categories: 1) domain fine-tuned T5-like models including Biomedical-T5 and Lay-SciFive; 2) fine-tuned BARTLarge model with controllable attributes (via tokens) BART-w-CTs; 3) ChatGPTprompting. We also present the work we carried out for this task on BioGPT finetuning. In the official automatic evaluation using SARI scores, BeeManc ranks 2nd among all teams and our model LaySciFive ranks 3rd among all 13 evaluated systems. In the official human evaluation, our model BART-w-CTs ranks 2nd on Sentence-Simplicity (score 92.84), 3rd on Term-Simplicity (score 82.33) among all 7 evaluated systems; It also produced a high score 91.57 on Fluency in comparison to the highest score 93.53. In the second round of submissions, our team using ChatGPT-prompting ranks the 2nd in several categories including simplified term accuracy score 92.26 and completeness score 96.58, and a very similar score on faithfulness score 95.3 to re-evaluated PLABA-base-1 (95.73) via human evaluations. Our codes, fine-tuned models, prompts, and data splits from the system development stage will be available at this https URL HECTA-UoM/PLABA-MU
• 摘要：在此系统报告中，我们描述了我们参与 PLABA2023 生物医学摘要简化任务（TAC 2023 轨道的一部分）时使用的模型和方法。我们提交的系统输出来自以下三个类别：1）领域微调的 T5 类模型，包括 Biomedical-T5 和 Lay-SciFive；2）具有可控属性（通过 token）BART-w-CTs 的微调 BARTLarge 模型；3）ChatGPTprompting。我们还介绍了我们为此任务在 BioGPT 微调方面开展的工作。在使用 SARI 分数的官方自动评估中，BeeManc 在所有团队中排名第二，我们的模型 LaySciFive 在所有 13 个评估系统中排名第三。在官方人工评估中，我们的模型 BART-w-CTs 在句子简单性方面排名第二（得分 92.84），在术语简单性方面排名第三（得分 82.33），在所有 7 个评估系统中；它在流畅度方面也获得了 91.57 的高分，而最高分是 93.53。在第二轮提交中，我们的团队使用 ChatGPT-prompting 在几个类别中排名第二，包括简化术语准确度得分 92.26 和完整性得分 96.58，以及与通过人工评估重新评估的 PLABA-base-1（95.73）非常相似的忠实度得分 95.3。我们的代码、微调模型、提示和系统开发阶段的数据分割将在此 https URL HECTA-UoM/PLABA-MU 上提供

Title: Personalized Clinical Note Generation from Doctor-Patient Conversations

• Authors: Nathan Brake, Thomas Schaaf
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.03874
• Pdf URL: https://arxiv.org/pdf/2408.03874
• Copy Paste: [[2408.03874]] Personalized Clinical Note Generation from Doctor-Patient Conversations(https://arxiv.org/abs/2408.03874)
• Keywords: llm
• Abstract: In this work, we present a novel technique to improve the quality of draft clinical notes for physicians. This technique is concentrated on the ability to model implicit physician conversation styles and note preferences. We also introduce a novel technique for the enrollment of new physicians when a limited number of clinical notes paired with conversations are available for that physician, without the need to re-train a model to support them. We show that our technique outperforms the baseline model by improving the ROUGE-2 score of the History of Present Illness section by 13.8%, the Physical Examination section by 88.6%, and the Assessment & Plan section by 50.8%.
• 摘要：在这项工作中，我们提出了一种新技术来提高医生临床笔记草稿的质量。该技术专注于模拟隐性医生对话风格和笔记偏好的能力。我们还介绍了一种新技术，用于在医生只有有限数量的临床笔记和对话配对时招募新医生，而无需重新训练模型来支持他们。我们表明，我们的技术优于基线模型，将现病史部分的 ROUGE-2 评分提高了 13.8%，体格检查部分提高了 88.6%，评估和计划部分提高了 50.8%。

Title: Simplifying Scholarly Abstracts for Accessible Digital Libraries

• Authors: Haining Wang, Jason Clark
• Subjects: cs.CL, cs.AI, cs.CY, cs.DL
• Abstract URL: https://arxiv.org/abs/2408.03899
• Pdf URL: https://arxiv.org/pdf/2408.03899
• Copy Paste: [[2408.03899]] Simplifying Scholarly Abstracts for Accessible Digital Libraries(https://arxiv.org/abs/2408.03899)
• Keywords: language model
• Abstract: Standing at the forefront of knowledge dissemination, digital libraries curate vast collections of scientific literature. However, these scholarly writings are often laden with jargon and tailored for domain experts rather than the general public. As librarians, we strive to offer services to a diverse audience, including those with lower reading levels. To extend our services beyond mere access, we propose fine-tuning a language model to rewrite scholarly abstracts into more comprehensible versions, thereby making scholarly literature more accessible when requested. We began by introducing a corpus specifically designed for training models to simplify scholarly abstracts. This corpus consists of over three thousand pairs of abstracts and significance statements from diverse disciplines. We then fine-tuned four language models using this corpus. The outputs from the models were subsequently examined both quantitatively for accessibility and semantic coherence, and qualitatively for language quality, faithfulness, and completeness. Our findings show that the resulting models can improve readability by over three grade levels, while maintaining fidelity to the original content. Although commercial state-of-the-art models still hold an edge, our models are much more compact, can be deployed locally in an affordable manner, and alleviate the privacy concerns associated with using commercial models. We envision this work as a step toward more inclusive and accessible libraries, improving our services for young readers and those without a college degree.
• 摘要：数字图书馆站在知识传播的最前沿，收集了大量科学文献。然而，这些学术著作往往充斥着行话，是为领域专家而非普通大众量身定制的。作为图书管理员，我们努力为不同的受众提供服务，包括阅读水平较低的人。为了将我们的服务扩展到单纯的访问之外，我们建议对语言模型进行微调，将学术摘要重写为更易理解的版本，从而使学术文献在需要时更容易访问。我们首先引入了一个专门为训练模型设计的语料库，以简化学术摘要。这个语料库包含来自不同学科的三千多对摘要和重要性陈述。然后，我们使用该语料库对四个语言模型进行了微调。随后，对模型的输出进行了定量检查，以了解其可访问性和语义连贯性，并定性检查其语言质量、忠实性和完整性。我们的研究结果表明，生成的模型可以将可读性提高三个等级以上，同时保持对原始内容的忠实度。尽管商业最先进的模型仍然占有优势，但我们的模型更加紧凑，可以以经济实惠的方式在本地部署，并减轻了使用商业模型带来的隐私问题。我们设想这项工作是朝着更具包容性和可访问性的图书馆迈出的一步，改善我们为年轻读者和没有大学学位的人提供的服务。

Title: Speech-MASSIVE: A Multilingual Speech Dataset for SLU and Beyond

• Authors: Beomseok Lee, Ioan Calapodescu, Marco Gaido, Matteo Negri, Laurent Besacier
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.03900
• Pdf URL: https://arxiv.org/pdf/2408.03900
• Copy Paste: [[2408.03900]] Speech-MASSIVE: A Multilingual Speech Dataset for SLU and Beyond(https://arxiv.org/abs/2408.03900)
• Keywords: llm, prompt
• Abstract: We present Speech-MASSIVE, a multilingual Spoken Language Understanding (SLU) dataset comprising the speech counterpart for a portion of the MASSIVE textual corpus. Speech-MASSIVE covers 12 languages from different families and inherits from MASSIVE the annotations for the intent prediction and slot-filling tasks. Our extension is prompted by the scarcity of massively multilingual SLU datasets and the growing need for versatile speech datasets to assess foundation models (LLMs, speech encoders) across languages and tasks. We provide a multimodal, multitask, multilingual dataset and report SLU baselines using both cascaded and end-to-end architectures in various training scenarios (zero-shot, few-shot, and full fine-tune). Furthermore, we demonstrate the suitability of Speech-MASSIVE for benchmarking other tasks such as speech transcription, language identification, and speech translation. The dataset, models, and code are publicly available at: this https URL
• 摘要：我们推出了 Speech-MASSIVE，这是一个多语言口语理解 (SLU) 数据集，包含 MASSIVE 文本语料库一部分的语音对应部分。Speech-MASSIVE 涵盖了来自不同语系的 12 种语言，并从 MASSIVE 继承了意图预测和槽位填充任务的注释。我们的扩展是由于大规模多语言 SLU 数据集的稀缺以及对多功能语音数据集的日益增长的需求，以评估跨语言和任务的基础模型（LLM、语音编码器）。我们提供多模式、多任务、多语言数据集，并在各种训练场景（零样本、少量样本和完全微调）中使用级联和端到端架构报告 SLU 基线。此外，我们展示了 Speech-MASSIVE 对其他任务（如语音转录、语言识别和语音翻译）的基准测试的适用性。数据集、模型和代码可在以下网址公开获取：此 https URL

Title: Decoding Biases: Automated Methods and LLM Judges for Gender Bias Detection in Language Models

• Authors: Shachi H Kumar, Saurav Sahay, Sahisnu Mazumder, Eda Okur, Ramesh Manuvinakurike, Nicole Beckage, Hsuan Su, Hung-yi Lee, Lama Nachman
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.03907
• Pdf URL: https://arxiv.org/pdf/2408.03907
• Copy Paste: [[2408.03907]] Decoding Biases: Automated Methods and LLM Judges for Gender Bias Detection in Language Models(https://arxiv.org/abs/2408.03907)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have excelled at language understanding and generating human-level text. However, even with supervised training and human alignment, these LLMs are susceptible to adversarial attacks where malicious users can prompt the model to generate undesirable text. LLMs also inherently encode potential biases that can cause various harmful effects during interactions. Bias evaluation metrics lack standards as well as consensus and existing methods often rely on human-generated templates and annotations which are expensive and labor intensive. In this work, we train models to automatically create adversarial prompts to elicit biased responses from target LLMs. We present LLM- based bias evaluation metrics and also analyze several existing automatic evaluation methods and metrics. We analyze the various nuances of model responses, identify the strengths and weaknesses of model families, and assess where evaluation methods fall short. We compare these metrics to human evaluation and validate that the LLM-as-a-Judge metric aligns with human judgement on bias in response generation.
• 摘要：大型语言模型 (LLM) 在语言理解和生成人类水平的文本方面表现出色。然而，即使经过监督训练和人工校准，这些 LLM 也容易受到对抗性攻击，恶意用户可以提示模型生成不良文本。LLM 还固有地编码了潜在的偏见，这些偏见可能在交互过程中造成各种有害影响。偏见评估指标缺乏标准和共识，现有方法通常依赖于人工生成的模板和注释，这些模板和注释既昂贵又费力。在这项工作中，我们训练模型自动创建对抗性提示，以从目标 LLM 中引出有偏见的响应。我们提出了基于 LLM 的偏见评估指标，并分析了几种现有的自动评估方法和指标。我们分析了模型响应的各种细微差别，确定了模型系列的优势和劣势，并评估了评估方法的不足之处。我们将这些指标与人工评估进行比较，并验证了 LLM-as-a-Judge 指标与人类对响应生成偏见的判断一致。

Title: From Words to Worth: Newborn Article Impact Prediction with LLM

• Authors: Penghai Zhao, Qinghua Xing, Kairan Dou, Jinyu Tian, Ying Tai, Jian Yang, Ming-Ming Cheng, Xiang Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.03934
• Pdf URL: https://arxiv.org/pdf/2408.03934
• Copy Paste: [[2408.03934]] From Words to Worth: Newborn Article Impact Prediction with LLM(https://arxiv.org/abs/2408.03934)
• Keywords: llm
• Abstract: As the academic landscape expands, the challenge of efficiently identifying potentially high-impact articles among the vast number of newly published works becomes critical. This paper introduces a promising approach, leveraging the capabilities of fine-tuned LLMs to predict the future impact of newborn articles solely based on titles and abstracts. Moving beyond traditional methods heavily reliant on external information, the proposed method discerns the shared semantic features of highly impactful papers from a large collection of title-abstract and potential impact pairs. These semantic features are further utilized to regress an improved metric, TNCSI_SP, which has been endowed with value, field, and time normalization properties. Additionally, a comprehensive dataset has been constructed and released for fine-tuning the LLM, containing over 12,000 entries with corresponding titles, abstracts, and TNCSI_SP. The quantitative results, with an NDCG@20 of 0.901, demonstrate that the proposed approach achieves state-of-the-art performance in predicting the impact of newborn articles when compared to competitive counterparts. Finally, we demonstrate a real-world application for predicting the impact of newborn journal articles to demonstrate its noteworthy practical value. Overall, our findings challenge existing paradigms and propose a shift towards a more content-focused prediction of academic impact, offering new insights for assessing newborn article impact.
• 摘要：随着学术领域的扩大，在大量新发表的作品中有效识别潜在高影响力文章的挑战变得至关重要。本文介绍了一种有前途的方法，利用微调 LLM 的功能，仅根据标题和摘要预测新生文章的未来影响力。与严重依赖外部信息的传统方法不同，所提出的方法从大量标题-摘要和潜在影响力对中辨别出高影响力论文的共同语义特征。这些语义特征进一步用于回归改进的度量 TNCSI_SP，该度量具有值、字段和时间规范化属性。此外，还构建并发布了一个全面的数据集以微调 LLM，其中包含 12,000 多个条目以及相应的标题、摘要和 TNCSI_SP。定量结果（NDCG@20 为 0.901）表明，与竞争方法相比，所提出的方法在预测新生文章的影响力方面实现了最先进的性能。最后，我们展示了一个预测新生期刊文章影响力的实际应用，以证明其值得注意的实用价值。总体而言，我们的研究结果挑战了现有范式，并提出了转向更注重内容的学术影响力预测的转变，为评估新生文章的影响力提供了新的见解。

Title: SLIM-RAFT: A Novel Fine-Tuning Approach to Improve Cross-Linguistic Performance for Mercosur Common Nomenclature

• Authors: Vinícius Di Oliveira, Yuri Façanha Bezerra, Li Weigang, Pedro Carvalho Brom, Victor Rafael R. Celestino
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.03936
• Pdf URL: https://arxiv.org/pdf/2408.03936
• Copy Paste: [[2408.03936]] SLIM-RAFT: A Novel Fine-Tuning Approach to Improve Cross-Linguistic Performance for Mercosur Common Nomenclature(https://arxiv.org/abs/2408.03936)
• Keywords: language model, gpt, llm, prompt, chat, chain-of-thought
• Abstract: Natural language processing (NLP) has seen significant advancements with the advent of large language models (LLMs). However, substantial improvements are still needed for languages other than English, especially for specific domains like the applications of Mercosur Common Nomenclature (NCM), a Brazilian Harmonized System (HS). To address this gap, this study uses TeenyTineLLaMA, a foundational Portuguese LLM, as an LLM source to implement the NCM application processing. Additionally, a simplified Retrieval-Augmented Fine-Tuning (RAFT) technique, termed SLIM-RAFT, is proposed for task-specific fine-tuning of LLMs. This approach retains the chain-of-thought (CoT) methodology for prompt development in a more concise and streamlined manner, utilizing brief and focused documents for training. The proposed model demonstrates an efficient and cost-effective alternative for fine-tuning smaller LLMs, significantly outperforming TeenyTineLLaMA and ChatGPT-4 in the same task. Although the research focuses on NCM applications, the methodology can be easily adapted for HS applications worldwide.
• 摘要：随着大型语言模型 (LLM) 的出现，自然语言处理 (NLP) 取得了重大进展。然而，对于英语以外的语言，仍然需要进行实质性改进，特别是在特定领域，例如南方共同市场通用命名法 (NCM) 的应用，巴西统一命名系统 (HS)。为了解决这一差距，本研究使用基础葡萄牙语 LLM TeenyTineLLaMA 作为 LLM 源来实现 NCM 应用程序处理。此外，提出了一种简化的检索增强微调 (RAFT) 技术，称为 SLIM-RAFT，用于针对特定任务的 LLM 微调。这种方法保留了思路链 (CoT) 方法，以更简洁、更简化的方式快速开发，利用简短而有针对性的文档进行训练。所提出的模型展示了一种高效且经济高效的微调小型 LLM 的替代方案，在同一任务中的表现明显优于 TeenyTineLLaMA 和 ChatGPT-4。虽然该研究的重点是 NCM 应用，但该方法可轻松适应全球 HS 应用。