2024-09-18

Title: Language Models and Retrieval Augmented Generation for Automated Structured Data Extraction from Diagnostic Reports

• Authors: Mohamed Sobhi Jabal, Pranav Warman, Jikai Zhang, Kartikeye Gupta, Ayush Jain, Maciej Mazurowski, Walter Wiggins, Kirti Magudia, Evan Calabrese
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.10576
• Pdf URL: https://arxiv.org/pdf/2409.10576
• Copy Paste: [[2409.10576]] Language Models and Retrieval Augmented Generation for Automated Structured Data Extraction from Diagnostic Reports(https://arxiv.org/abs/2409.10576)
• Keywords: language model, prompt, retrieval augmented generation
• Abstract: Purpose: To develop and evaluate an automated system for extracting structured clinical information from unstructured radiology and pathology reports using open-weights large language models (LMs) and retrieval augmented generation (RAG), and to assess the effects of model configuration variables on extraction performance. Methods and Materials: The study utilized two datasets: 7,294 radiology reports annotated for Brain Tumor Reporting and Data System (BT-RADS) scores and 2,154 pathology reports annotated for isocitrate dehydrogenase (IDH) mutation status. An automated pipeline was developed to benchmark the performance of various LMs and RAG configurations. The impact of model size, quantization, prompting strategies, output formatting, and inference parameters was systematically evaluated. Results: The best performing models achieved over 98% accuracy in extracting BT-RADS scores from radiology reports and over 90% for IDH mutation status extraction from pathology reports. The top model being medical fine-tuned llama3. Larger, newer, and domain fine-tuned models consistently outperformed older and smaller models. Model quantization had minimal impact on performance. Few-shot prompting significantly improved accuracy. RAG improved performance for complex pathology reports but not for shorter radiology reports. Conclusions: Open LMs demonstrate significant potential for automated extraction of structured clinical data from unstructured clinical reports with local privacy-preserving application. Careful model selection, prompt engineering, and semi-automated optimization using annotated data are critical for optimal performance. These approaches could be reliable enough for practical use in research workflows, highlighting the potential for human-machine collaboration in healthcare data extraction.
• 摘要：目的：开发和评估一种使用开放权重大型语言模型 (LM) 和检索增强生成 (RAG) 从非结构化放射学和病理报告中提取结构化临床信息的自动化系统，并评估模型配置变量对提取性能的影响。方法和材料：该研究使用了两个数据集：7,294 份带有脑肿瘤报告和数据系统 (BT-RADS) 评分注释的放射学报告和 2,154 份带有异柠檬酸脱氢酶 (IDH) 突变状态注释的病理报告。开发了一个自动化流程来对各种 LM 和 RAG 配置的性能进行基准测试。系统地评估了模型大小、量化、提示策略、输出格式和推理参数的影响。结果：表现最佳的模型在从放射学报告中提取 BT-RADS 评分时实现了 98% 以上的准确率，在从病理报告中提取 IDH 突变状态时实现了 90% 以上的准确率。顶级模型是医学微调的 llama3。更大、更新和领域微调的模型始终优于旧模型和更小模型。模型量化对性能的影响微乎其微。小样本提示显著提高了准确性。RAG 提高了复杂病理报告的性能，但没有提高较短的放射学报告的性能。结论：开放式 LM 显示出在本地隐私保护应用中从非结构化临床报告中自动提取结构化临床数据的巨大潜力。仔细的模型选择、及时的工程设计和使用注释数据的半自动化优化对于获得最佳性能至关重要。这些方法足够可靠，可用于研究工作流程中的实际应用，凸显了人机协作在医疗保健数据提取中的潜力。

Title: Exploring Fine-tuned Generative Models for Keyphrase Selection: A Case Study for Russian

• Authors: Anna Glazkova, Dmitry Morozov
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.10640
• Pdf URL: https://arxiv.org/pdf/2409.10640
• Copy Paste: [[2409.10640]] Exploring Fine-tuned Generative Models for Keyphrase Selection: A Case Study for Russian(https://arxiv.org/abs/2409.10640)
• Keywords: gpt
• Abstract: Keyphrase selection plays a pivotal role within the domain of scholarly texts, facilitating efficient information retrieval, summarization, and indexing. In this work, we explored how to apply fine-tuned generative transformer-based models to the specific task of keyphrase selection within Russian scientific texts. We experimented with four distinct generative models, such as ruT5, ruGPT, mT5, and mBART, and evaluated their performance in both in-domain and cross-domain settings. The experiments were conducted on the texts of Russian scientific abstracts from four domains: mathematics \& computer science, history, medicine, and linguistics. The use of generative models, namely mBART, led to gains in in-domain performance (up to 4.9\% in BERTScore, 9.0\% in ROUGE-1, and 12.2\% in F1-score) over three keyphrase extraction baselines for the Russian language. Although the results for cross-domain usage were significantly lower, they still demonstrated the capability to surpass baseline performances in several cases, underscoring the promising potential for further exploration and refinement in this research field.
• 摘要：关键词选择在学术文本领域起着关键作用，有助于实现高效的信息检索、摘要和索引。在这项工作中，我们探索了如何将经过微调的基于生成变压器的模型应用于俄语科学文本中关键词选择的特定任务。我们尝试了四种不同的生成模型，例如 ruT5、ruGPT、mT5 和 mBART，并评估了它们在域内和跨域设置中的性能。实验是在四个领域的俄语科学摘要文本上进行的：数学和计算机科学、历史、医学和语言学。使用生成模型 mBART 可提高俄语语言的三个关键词提取基线的域内性能（BERTScore 高达 4.9%，ROUGE-1 高达 9.0%，F1 得分高达 12.2%）。虽然跨域使用的结果明显较低，但它们仍表现出在某些情况下超越基线性能的能力，凸显了该研究领域进一步探索和改进的巨大潜力。

Title: Improving Multi-candidate Speculative Decoding

• Authors: Xiaofan Lu, Yixiao Zeng, Feiyang Ma, Zixu Yu, Marco Levorato
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.10644
• Pdf URL: https://arxiv.org/pdf/2409.10644
• Copy Paste: [[2409.10644]] Improving Multi-candidate Speculative Decoding(https://arxiv.org/abs/2409.10644)
• Keywords: language model, llm
• Abstract: Speculative Decoding (SD) is a technique to accelerate the inference of Large Language Models (LLMs) by using a lower complexity draft model to propose candidate tokens verified by a larger target model. To further improve efficiency, Multi-Candidate Speculative Decoding (MCSD) improves upon this by sampling multiple candidate tokens from the draft model at each step and verifying them in parallel, thus increasing the chances of accepting a token and reducing generation time. Existing MCSD methods rely on the draft model to initialize the multi-candidate sequences and use static length and tree attention structure for draft generation. However, such an approach suffers from the draft and target model's output distribution differences, especially in dynamic generation context. In this work, we introduce an improved version of MCSD that includes a target model initialized multi-candidate process, dynamic sliced topology-aware causal mask for dynamic length adjustment, and decision models to optimize early stopping. Our framework improves the acceptance rate, defined as the ratio of the longest draft sequence length accepted by the target model over the maximum draft sequence length, by a maximum of 164% and gains a maximum of 75% generation speed up over the MCSD baseline. We also conduct an ablation study to evaluate the impact of the decision model.
• 摘要：推测解码 (SD) 是一种加速大型语言模型 (LLM) 推理的技术，它使用复杂度较低的草稿模型来提出由更大的目标模型验证的候选标记。为了进一步提高效率，多候选推测解码 (MCSD) 在此基础上进行了改进，在每个步骤中从草稿模型中抽取多个候选标记并并行验证它们，从而增加了接受标记的机会并减少了生成时间。现有的 MCSD 方法依靠草稿模型来初始化多候选序列，并使用静态长度和树注意结构进行草稿生成。然而，这种方法受到草稿和目标模型的输出分布差异的影响，尤其是在动态生成环境中。在这项工作中，我们介绍了 MCSD 的改进版本，其中包括目标模型初始化的多候选过程、用于动态长度调整的动态切片拓扑感知因果掩码以及用于优化早期停止的决策模型。我们的框架将接受率（定义为目标模型接受的最长草稿序列长度与最大草稿序列长度之比）提高了最多 164%，并且与 MCSD 基线相比，生成速度最多提高了 75%。我们还进行了一项消融研究来评估决策模型的影响。

Title: Self-Attention Limits Working Memory Capacity of Transformer-Based Models

• Authors: Dongyu Gong, Hantao Zhang
• Subjects: cs.CL, cs.AI, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2409.10715
• Pdf URL: https://arxiv.org/pdf/2409.10715
• Copy Paste: [[2409.10715]] Self-Attention Limits Working Memory Capacity of Transformer-Based Models(https://arxiv.org/abs/2409.10715)
• Keywords: language model, llm
• Abstract: Recent work on Transformer-based large language models (LLMs) has revealed striking limits in their working memory capacity, similar to what has been found in human behavioral studies. Specifically, these models' performance drops significantly on N-back tasks as N increases. However, there is still a lack of mechanistic interpretability as to why this phenomenon would arise. Inspired by the executive attention theory from behavioral sciences, we hypothesize that the self-attention mechanism within Transformer-based models might be responsible for their working memory capacity limits. To test this hypothesis, we train vanilla decoder-only transformers to perform N-back tasks and find that attention scores gradually aggregate to the N-back positions over training, suggesting that the model masters the task by learning a strategy to pay attention to the relationship between the current position and the N-back position. Critically, we find that the total entropy of the attention score matrix increases as N increases, suggesting that the dispersion of attention scores might be the cause of the capacity limit observed in N-back tasks.
• 摘要：最近对基于 Transformer 的大型语言模型 (LLM) 的研究揭示了它们的工作记忆容量存在显著限制，这与人类行为研究中发现的情况类似。具体而言，随着 N 的增加，这些模型在 N-back 任务上的表现会显著下降。然而，对于这种现象出现的原因，仍然缺乏机制上的可解释性。受行为科学的执行注意力理论的启发，我们假设基于 Transformer 的模型中的自我注意力机制可能是造成它们的工作记忆容量限制的原因。为了验证这一假设，我们训练了 vanilla 解码器专用 Transformer 来执行 N-back 任务，发现注意力得分在训练过程中逐渐聚集到 N-back 位置，这表明模型通过学习一种关注当前位置和 N-back 位置之间关系的策略来掌握任务。至关重要的是，我们发现注意力得分矩阵的总熵随着 N 的增加而增加，这表明注意力得分的分散可能是 N-back 任务中观察到的容量限制的原因。

Title: Generalized Measures of Anticipation and Responsivity in Online Language Processing

• Authors: Mario Giulianelli, Andreas Opedal, Ryan Cotterell
• Subjects: cs.CL, cs.AI, cs.IT
• Abstract URL: https://arxiv.org/abs/2409.10728
• Pdf URL: https://arxiv.org/pdf/2409.10728
• Copy Paste: [[2409.10728]] Generalized Measures of Anticipation and Responsivity in Online Language Processing(https://arxiv.org/abs/2409.10728)
• Keywords: language model
• Abstract: We introduce a generalization of classic information-theoretic measures of predictive uncertainty in online language processing, based on the simulation of expected continuations of incremental linguistic contexts. Our framework provides a formal definition of anticipatory and responsive measures, and it equips experimenters with the tools to define new, more expressive measures beyond standard next-symbol entropy and surprisal. While extracting these standard quantities from language models is convenient, we demonstrate that using Monte Carlo simulation to estimate alternative responsive and anticipatory measures pays off empirically: New special cases of our generalized formula exhibit enhanced predictive power compared to surprisal for human cloze completion probability as well as ELAN, LAN, and N400 amplitudes, and greater complementarity with surprisal in predicting reading times.
• 摘要：我们引入了在线语言处理中预测不确定性的经典信息理论度量的泛化，该度量基于增量语言上下文的预期延续模拟。我们的框架提供了预期和响应度量的正式定义，并为实验者提供了定义新、更具表现力的度量的工具，超越了标准的下一个符号熵和意外性。虽然从语言模型中提取这些标准量很方便，但我们证明使用蒙特卡罗模拟来估计替代的响应和预期度量在经验上是值得的：与意外性相比，我们的广义公式的新特例对人类完形填空概率以及 ELAN、LAN 和 N400 幅度表现出增强的预测能力，并且在预测阅读时间方面与意外性具有更大的互补性。

Title: Semantics Preserving Emoji Recommendation with Large Language Models

• Authors: Zhongyi Qiu, Kangyi Qiu, Hanjia Lyu, Wei Xiong, Jiebo Luo
• Subjects: cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2409.10760
• Pdf URL: https://arxiv.org/pdf/2409.10760
• Copy Paste: [[2409.10760]] Semantics Preserving Emoji Recommendation with Large Language Models(https://arxiv.org/abs/2409.10760)
• Keywords: language model, gpt, llm, prompt
• Abstract: Emojis have become an integral part of digital communication, enriching text by conveying emotions, tone, and intent. Existing emoji recommendation methods are primarily evaluated based on their ability to match the exact emoji a user chooses in the original text. However, they ignore the essence of users' behavior on social media in that each text can correspond to multiple reasonable emojis. To better assess a model's ability to align with such real-world emoji usage, we propose a new semantics preserving evaluation framework for emoji recommendation, which measures a model's ability to recommend emojis that maintain the semantic consistency with the user's text. To evaluate how well a model preserves semantics, we assess whether the predicted affective state, demographic profile, and attitudinal stance of the user remain unchanged. If these attributes are preserved, we consider the recommended emojis to have maintained the original semantics. The advanced abilities of Large Language Models (LLMs) in understanding and generating nuanced, contextually relevant output make them well-suited for handling the complexities of semantics preserving emoji recommendation. To this end, we construct a comprehensive benchmark to systematically assess the performance of six proprietary and open-source LLMs using different prompting techniques on our task. Our experiments demonstrate that GPT-4o outperforms other LLMs, achieving a semantics preservation score of 79.23%. Additionally, we conduct case studies to analyze model biases in downstream classification tasks and evaluate the diversity of the recommended emojis.
• 摘要：表情符号已成为数字通信中不可或缺的一部分，通过传达情感、语气和意图来丰富文本。现有的表情符号推荐方法主要基于其与用户在原始文本中选择的表情符号的精确匹配能力进行评估。然而，它们忽略了用户在社交媒体上的行为本质，即每条文本可以对应多个合理的表情符号。为了更好地评估模型与现实世界中的表情符号使用情况保持一致的能力，我们提出了一种新的表情符号推荐语义保留评估框架，该框架衡量模型推荐与用户文本保持语义一致性的表情符号的能力。为了评估模型保留语义的效果，我们评估预测的用户情感状态、人口统计资料和态度立场是否保持不变。如果这些属性得以保留，我们认为推荐的表情符号保持了原始语义。大型语言模型 (LLM) 在理解和生成细微、上下文相关的输出方面具有先进的能力，这使得它们非常适合处理语义保留表情符号推荐的复杂性。为此，我们构建了一个全面的基准，以系统地评估六种专有和开源 LLM 使用不同的提示技术在我们的任务上的表现。我们的实验表明，GPT-4o 的表现优于其他 LLM，语义保留得分达到 79.23%。此外，我们还进行了案例研究，以分析下游分类任务中的模型偏差并评估推荐表情符号的多样性。

Title: Model Tells Itself Where to Attend: Faithfulness Meets Automatic Attention Steering

• Authors: Qingru Zhang, Xiaodong Yu, Chandan Singh, Xiaodong Liu, Liyuan Liu, Jianfeng Gao, Tuo Zhao, Dan Roth, Hao Cheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.10790
• Pdf URL: https://arxiv.org/pdf/2409.10790
• Copy Paste: [[2409.10790]] Model Tells Itself Where to Attend: Faithfulness Meets Automatic Attention Steering(https://arxiv.org/abs/2409.10790)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have demonstrated remarkable performance across various real-world tasks. However, they often struggle to fully comprehend and effectively utilize their input contexts, resulting in responses that are unfaithful or hallucinated. This difficulty increases for contexts that are long or contain distracting information, which can divert LLMs from fully capturing essential evidence. To address this issue, many works use prompting to help LLMs utilize contextual information more faithfully. For instance, iterative prompting highlights key information in two steps that first ask the LLM to identify important pieces of context and then derive answers accordingly. However, prompting methods are constrained to highlighting key information implicitly in token space, which is often insufficient to fully steer the model's attention. To improve model faithfulness more reliably, we propose AutoPASTA, a method that automatically identifies key contextual information and explicitly highlights it by steering an LLM's attention scores. Like prompting, AutoPASTA is applied at inference time and does not require changing any model parameters. Our experiments on open-book QA demonstrate that AutoPASTA effectively enables models to grasp essential contextual information, leading to substantially improved model faithfulness and performance, e.g., an average improvement of 7.95% for LLAMA3-70B-Instruct. Code will be publicly available at this https URL .
• 摘要：大型语言模型 (LLM) 在各种实际任务中都表现出色。然而，它们往往难以完全理解和有效利用输入上下文，导致响应不真实或产生幻觉。对于较长或包含分散注意力的信息的上下文，这种难度会增加，这会使 LLM 无法完全捕获基本证据。为了解决这个问题，许多研究使用提示来帮助 LLM 更忠实地利用上下文信息。例如，迭代提示分两步突出显示关键信息，首先要求 LLM 识别重要的上下文，然后据此得出答案。然而，提示方法仅限于在标记空间中隐式突出显示关键信息，这通常不足以完全引导模型的注意力。为了更可靠地提高模型的忠实度，我们提出了 AutoPASTA，这种方法可以自动识别关键上下文信息并通过引导 LLM 的注意力分数明确地突出显示它。与提示一样，AutoPASTA 在推理时应用，不需要更改任何模型参数。我们在开放式问答上的实验表明，AutoPASTA 可以有效地使模型掌握必要的上下文信息，从而显著提高模型的忠实度和性能，例如，LLAMA3-70B-Instruct 的平均改进率为 7.95%。代码将在此 https URL 上公开提供。

Title: ReXErr: Synthesizing Clinically Meaningful Errors in Diagnostic Radiology Reports

• Authors: Vishwanatha M. Rao, Serena Zhang, Julian N. Acosta, Subathra Adithan, Pranav Rajpurkar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.10829
• Pdf URL: https://arxiv.org/pdf/2409.10829
• Copy Paste: [[2409.10829]] ReXErr: Synthesizing Clinically Meaningful Errors in Diagnostic Radiology Reports(https://arxiv.org/abs/2409.10829)
• Keywords: language model
• Abstract: Accurately interpreting medical images and writing radiology reports is a critical but challenging task in healthcare. Both human-written and AI-generated reports can contain errors, ranging from clinical inaccuracies to linguistic mistakes. To address this, we introduce ReXErr, a methodology that leverages Large Language Models to generate representative errors within chest X-ray reports. Working with board-certified radiologists, we developed error categories that capture common mistakes in both human and AI-generated reports. Our approach uses a novel sampling scheme to inject diverse errors while maintaining clinical plausibility. ReXErr demonstrates consistency across error categories and produces errors that closely mimic those found in real-world scenarios. This method has the potential to aid in the development and evaluation of report correction algorithms, potentially enhancing the quality and reliability of radiology reporting.
• 摘要：准确解读医学图像和撰写放射学报告是医疗保健领域一项关键但具有挑战性的任务。人工撰写的报告和人工智能生成的报告都可能包含错误，从临床错误到语言错误。为了解决这个问题，我们引入了 ReXErr，这是一种利用大型语言模型在胸部 X 光报告中生成代表性错误的方法。我们与委员会认证的放射科医生合作，开发了错误类别，以捕获人工和人工智能生成的报告中的常见错误。我们的方法使用一种新颖的采样方案来注入各种错误，同时保持临床合理性。ReXErr 在各种错误类别中都表现出一致性，并且产生的错误与现实世界场景中的错误非常相似。这种方法有可能有助于开发和评估报告校正算法，从而有可能提高放射学报告的质量和可靠性。

Title: Adaptive Large Language Models By Layerwise Attention Shortcuts

• Authors: Prateek Verma, Mert Pilanci
• Subjects: cs.CL, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.10870
• Pdf URL: https://arxiv.org/pdf/2409.10870
• Copy Paste: [[2409.10870]] Adaptive Large Language Models By Layerwise Attention Shortcuts(https://arxiv.org/abs/2409.10870)
• Keywords: language model, gpt, llm
• Abstract: Transformer architectures are the backbone of the modern AI revolution. However, they are based on simply stacking the same blocks in dozens of layers and processing information sequentially from one block to another. In this paper, we propose to challenge this and introduce adaptive computations for LLM-like setups, which allow the final layer to attend to all of the intermediate layers as it deems fit through the attention mechanism, thereby introducing computational \textbf{attention shortcuts}. These shortcuts can thus make the architecture depth and context adaptive. We showcase four different datasets, namely acoustic tokens, natural language, and symbolic music, and we achieve superior performance for GPT-like architecture. We give evidence via attention maps that the models learn complex dependencies across layers that are adaptive in context and depth depending on the input tokens.
• 摘要：Transformer 架构是现代 AI 革命的支柱。然而，它们只是简单地将相同的块堆叠在数十层中，并按顺序从一个块到另一个块处理信息。在本文中，我们建议挑战这一点，并为类似 LLM 的设置引入自适应计算，这允许最后一层通过注意力机制按照其认为合适的方式关注所有中间层，从而引入计算 \textbf{注意力捷径}。这些捷径可以使架构具有深度和上下文自适应性。我们展示了四个不同的数据集，即声学标记、自然语言和符号音乐，并且我们在类似 GPT 的架构中实现了卓越的性能。我们通过注意力图提供证据，证明模型可以学习跨层的复杂依赖关系，这些依赖关系可以根据输入标记在上下文和深度上进行自适应。

Title: CREAM: Comparison-Based Reference-Free ELO-Ranked Automatic Evaluation for Meeting Summarization

• Authors: Ziwei Gong, Lin Ai, Harshsaiprasad Deshpande, Alexander Johnson, Emmy Phung, Zehui Wu, Ahmad Emami, Julia Hirschberg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.10883
• Pdf URL: https://arxiv.org/pdf/2409.10883
• Copy Paste: [[2409.10883]] CREAM: Comparison-Based Reference-Free ELO-Ranked Automatic Evaluation for Meeting Summarization(https://arxiv.org/abs/2409.10883)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Large Language Models (LLMs) have spurred interest in automatic evaluation methods for summarization, offering a faster, more cost-effective alternative to human evaluation. However, existing methods often fall short when applied to complex tasks like long-context summarizations and dialogue-based meeting summarizations. In this paper, we introduce CREAM (Comparison-Based Reference-Free Elo-Ranked Automatic Evaluation for Meeting Summarization), a novel framework that addresses the unique challenges of evaluating meeting summaries. CREAM leverages a combination of chain-of-thought reasoning and key facts alignment to assess conciseness and completeness of model-generated summaries without requiring reference. By employing an ELO ranking system, our approach provides a robust mechanism for comparing the quality of different models or prompt configurations.
• 摘要：大型语言模型 (LLM) 激发了人们对摘要自动评估方法的兴趣，它提供了一种比人工评估更快、更具成本效益的替代方案。然而，现有方法在应用于长上下文摘要和基于对话的会议摘要等复杂任务时往往表现不佳。在本文中，我们介绍了 CREAM（基于比较的无参考 Elo 排名会议摘要自动评估），这是一个新颖的框架，可解决评估会议摘要的独特挑战。CREAM 利用思路链推理和关键事实对齐的组合来评估模型生成的摘要的简洁性和完整性，而无需参考。通过采用 ELO 排名系统，我们的方法提供了一种强大的机制来比较不同模型或提示配置的质量。

Title: Attention-Seeker: Dynamic Self-Attention Scoring for Unsupervised Keyphrase Extraction

• Authors: Erwin D. López Z., Cheng Tang, Atsushi Shimada
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.10907
• Pdf URL: https://arxiv.org/pdf/2409.10907
• Copy Paste: [[2409.10907]] Attention-Seeker: Dynamic Self-Attention Scoring for Unsupervised Keyphrase Extraction(https://arxiv.org/abs/2409.10907)
• Keywords: language model, prompt
• Abstract: This paper proposes Attention-Seeker, an unsupervised keyphrase extraction method that leverages self-attention maps from a Large Language Model to estimate the importance of candidate phrases. Our approach identifies specific components - such as layers, heads, and attention vectors - where the model pays significant attention to the key topics of the text. The attention weights provided by these components are then used to score the candidate phrases. Unlike previous models that require manual tuning of parameters (e.g., selection of heads, prompts, hyperparameters), Attention-Seeker dynamically adapts to the input text without any manual adjustments, enhancing its practical applicability. We evaluate Attention-Seeker on four publicly available datasets: Inspec, SemEval2010, SemEval2017, and Krapivin. Our results demonstrate that, even without parameter tuning, Attention-Seeker outperforms most baseline models, achieving state-of-the-art performance on three out of four datasets, particularly excelling in extracting keyphrases from long documents.
• 摘要：本文提出了 Attention-Seeker，这是一种无监督的关键短语提取方法，它利用大型语言模型的自注意力图来估计候选短语的重要性。我们的方法确定了特定的组件（例如层、头部和注意力向量），模型在这些组件中高度关注文本的关键主题。然后使用这些组件提供的注意力权重对候选短语进行评分。与以前需要手动调整参数（例如选择头部、提示、超参数）的模型不同，Attention-Seeker 可以动态适应输入文本而无需任何手动调整，从而增强了其实际适用性。我们在四个公开可用的数据集上评估了 Attention-Seeker：Inspec、SemEval2010、SemEval2017 和 Krapivin。我们的结果表明，即使没有参数调整，Attention-Seeker 的表现也优于大多数基线模型，在四个数据集中的三个上实现了最先进的性能，尤其是在从长文档中提取关键短语方面表现出色。

Title: Propulsion: Steering LLM with Tiny Fine-Tuning

• Authors: Md Kowsher, Nusrat Jahan Prottasha, Prakash Bhat
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.10927
• Pdf URL: https://arxiv.org/pdf/2409.10927
• Copy Paste: [[2409.10927]] Propulsion: Steering LLM with Tiny Fine-Tuning(https://arxiv.org/abs/2409.10927)
• Keywords: language model, llm
• Abstract: The rapid advancements in Large Language Models (LLMs) have revolutionized natural language processing (NLP) and related fields. However, fine-tuning these models for specific tasks remains computationally expensive and risks degrading pre-learned features. To address these challenges, we propose Propulsion, a novel parameter efficient fine-tuning (PEFT) method designed to optimize task-specific performance while drastically reducing computational overhead. Inspired by the concept of controlled adjustments in physical motion, Propulsion selectively re-scales specific dimensions of a pre-trained model, guiding output predictions toward task objectives without modifying the model's parameters. By introducing lightweight, trainable Propulsion parameters at the pre-trained layer, we minimize the number of parameters updated during fine-tuning, preventing overfitting or overwriting of existing knowledge. Our theoretical analysis, supported by Neural Tangent Kernel (NTK) theory, shows that Propulsion approximates the performance of full fine-tuning with far fewer trainable parameters. Empirically, Propulsion reduces the parameter count from 355.3 million to just 0.086 million, achieving over a 10x reduction compared to standard approaches like LoRA while maintaining competitive performance across benchmarks.
• 摘要：大型语言模型 (LLM) 的快速发展彻底改变了自然语言处理 (NLP) 和相关领域。然而，针对特定任务对这些模型进行微调仍然需要耗费大量计算资源，并且可能会降低预先学习的特征。为了应对这些挑战，我们提出了 Propulsion，这是一种新颖的参数高效微调 (PEFT) 方法，旨在优化特定任务的性能，同时大幅降低计算开销。受物理运动受控调整概念的启发，Propulsion 有选择地重新缩放预训练模型的特定维度，在不修改模型参数的情况下将输出预测引导至任务目标。通过在预训练层引入轻量级、可训练的 Propulsion 参数，我们最大限度地减少了微调期间更新的参数数量，从而防止过度拟合或覆盖现有知识。我们的理论分析由神经切线核 (NTK) 理论支持，表明 Propulsion 以更少的可训练参数近似于完全微调的性能。从经验上看，Propulsion 将参数数量从 3.553 亿减少到仅 8.6 亿，与 LoRA 等标准方法相比减少了 10 倍以上，同时在基准测试中保持了有竞争力的性能。

Title: Investigating Context-Faithfulness in Large Language Models: The Roles of Memory Strength and Evidence Style

• Authors: Yuepei Li, Kang Zhou, Qiao Qiao, Bach Nguyen, Qing Wang, Qi Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.10955
• Pdf URL: https://arxiv.org/pdf/2409.10955
• Copy Paste: [[2409.10955]] Investigating Context-Faithfulness in Large Language Models: The Roles of Memory Strength and Evidence Style(https://arxiv.org/abs/2409.10955)
• Keywords: language model, gpt, llm, retrieval-augmented generation
• Abstract: Retrieval-augmented generation (RAG) improves Large Language Models (LLMs) by incorporating external information into the response generation process. However, how context-faithful LLMs are and what factors influence LLMs' context-faithfulness remain largely unexplored. In this study, we investigate the impact of memory strength and evidence presentation on LLMs' receptiveness to external evidence. We introduce a method to quantify the memory strength of LLMs by measuring the divergence in LLMs' responses to different paraphrases of the same question, which is not considered by previous works. We also generate evidence in various styles to evaluate the effects of evidence in different styles. Two datasets are used for evaluation: Natural Questions (NQ) with popular questions and popQA featuring long-tail questions. Our results show that for questions with high memory strength, LLMs are more likely to rely on internal memory, particularly for larger LLMs such as GPT-4. On the other hand, presenting paraphrased evidence significantly increases LLMs' receptiveness compared to simple repetition or adding details.
• 摘要：检索增强生成 (RAG) 通过将外部信息纳入响应生成过程来改进大型语言模型 (LLM)。然而，LLM 的上下文忠实度如何以及哪些因素影响 LLM 的上下文忠实度仍未得到充分探索。在本研究中，我们研究了记忆强度和证据呈现对 LLM 对外部证据的接受度的影响。我们引入了一种通过测量 LLM 对同一问题的不同释义的回答差异来量化 LLM 记忆强度的方法，这是以前的研究没有考虑到的。我们还生成各种风格的证据来评估不同风格证据的影响。两个数据集用于评估：包含热门问题的自然问题 (NQ) 和包含长尾问题的 popQA。我们的结果表明，对于记忆强度高的问题，LLM 更有可能依赖内部记忆，尤其是对于 GPT-4 等较大的 LLM。另一方面，与简单的重复或添加细节相比，提供释义证据可显著提高 LLM 的接受度。

Title: Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMs

• Authors: Dingjie Song, Wenjun Wang, Shunian Chen, Xidong Wang, Michael Guan, Benyou Wang
• Subjects: cs.CL, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.10994
• Pdf URL: https://arxiv.org/pdf/2409.10994
• Copy Paste: [[2409.10994]] Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMs(https://arxiv.org/abs/2409.10994)
• Keywords: language model, llm
• Abstract: The rapid advancement of Multimodal Large Language Models (MLLMs) has led to remarkable performances across various domains. However, this progress is accompanied by a substantial surge in the resource consumption of these models. We address this pressing issue by introducing a new approach, Token Reduction using CLIP Metric (TRIM), aimed at improving the efficiency of MLLMs without sacrificing their performance. Inspired by human attention patterns in Visual Question Answering (VQA) tasks, TRIM presents a fresh perspective on the selection and reduction of image tokens. The TRIM method has been extensively tested across 12 datasets, and the results demonstrate a significant reduction in computational overhead while maintaining a consistent level of performance. This research marks a critical stride in efficient MLLM development, promoting greater accessibility and sustainability of high-performing models.
• 摘要：多模态大型语言模型 (MLLM) 的快速发展已在各个领域取得了令人瞩目的成绩。然而，这一进步也伴随着这些模型资源消耗的大幅增加。为了解决这一紧迫问题，我们引入了一种新方法，即使用 CLIP 度量的标记减少 (TRIM)，旨在提高 MLLM 的效率而不牺牲其性能。TRIM 受到视觉问答 (VQA) 任务中人类注意力模式的启发，为图像标记的选择和减少提供了全新的视角。TRIM 方法已在 12 个数据集上进行了广泛测试，结果表明，在保持一致的性能水平的同时，计算开销显著减少。这项研究标志着高效 MLLM 开发的关键一步，提高了高性能模型的可访问性和可持续性。

Title: Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Models

• Authors: Potsawee Manakul, Guangzhi Sun, Warit Sirichotedumrong, Kasima Tharnpipitchai, Kunat Pipatanakul
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.10999
• Pdf URL: https://arxiv.org/pdf/2409.10999
• Copy Paste: [[2409.10999]] Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Models(https://arxiv.org/abs/2409.10999)
• Keywords: language model, llm, prompt
• Abstract: Audio language models can understand audio inputs and perform a range of audio-related tasks based on instructions, such as speech recognition and audio captioning, where the instructions are usually textual prompts. Audio language models are mostly initialized from pre-trained audio encoders and large language models (LLMs). Although these pre-trained components were developed to support multiple languages, audio-language models are trained predominantly on English data, which may limit their usability to only English instructions or English speech inputs. First, this paper examines the performance of existing audio language models in an underserved language using Thai as an example. This paper demonstrates that, despite being built on multilingual backbones, audio language models do not exhibit cross-lingual emergent abilities to low-resource languages. Second, this paper studies data mixture for developing audio language models that are optimized for a target language as well as English. In addition. this paper integrates audio comprehension and speech instruction-following capabilities into a single unified model. Our experiments provide insights into data mixture for enhancing instruction-following capabilities in both a low-resource language and English. Our model, Typhoon-Audio, outperforms existing open-source audio language models by a considerable margin, and it is comparable to state-of-the-art Gemini-1.5-Pro in both English and Thai languages.
• 摘要：音频语言模型可以理解音频输入并根据指令执行一系列与音频相关的任务，例如语音识别和音频字幕，其中指令通常是文本提示。音频语言模型大多由预训练的音频编码器和大型语言模型 (LLM) 初始化。尽管这些预训练组件是为了支持多种语言而开发的，但音频语言模型主要在英语数据上进行训练，这可能会限制它们仅适用于英语指令或英语语音输入。首先，本文以泰语为例，研究了现有音频语言模型在服务不足的语言中的表现。本文表明，尽管建立在多语言主干上，但音频语言模型并不表现出对资源匮乏的语言的跨语言新兴能力。其次，本文研究了数据混合，以开发针对目标语言和英语优化的音频语言模型。此外，本文将音频理解和语音指令遵循功能集成到一个统一的模型中。我们的实验为数据混合提供了见解，以增强低资源语言和英语的指令遵循能力。我们的模型 Typhoon-Audio 远远优于现有的开源音频语言模型，并且在英语和泰语方面可与最先进的 Gemini-1.5-Pro 相媲美。

Title: CAST: Cross-modal Alignment Similarity Test for Vision Language Models

• Authors: Gautier Dagan, Olga Loginova, Anil Batra
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.11007
• Pdf URL: https://arxiv.org/pdf/2409.11007
• Copy Paste: [[2409.11007]] CAST: Cross-modal Alignment Similarity Test for Vision Language Models(https://arxiv.org/abs/2409.11007)
• Keywords: language model, hallucination
• Abstract: Vision Language Models (VLMs) are typically evaluated with Visual Question Answering (VQA) tasks which assess a model's understanding of scenes. Good VQA performance is taken as evidence that the model will perform well on a broader range of tasks that require both visual and language inputs. However, scene-aware VQA does not fully capture input biases or assess hallucinations caused by a misalignment between modalities. To address this, we propose a Cross-modal Alignment Similarity Test (CAST) to probe VLMs for self-consistency across modalities. This test involves asking the models to identify similarities between two scenes through text-only, image-only, or both and then assess the truthfulness of the similarities they generate. Since there is no ground-truth to compare against, this evaluation does not focus on objective accuracy but rather on whether VLMs are internally consistent in their outputs. We argue that while not all self-consistent models are capable or accurate, all capable VLMs must be self-consistent.
• 摘要：视觉语言模型 (VLM) 通常通过视觉问答 (VQA) 任务进行评估，该任务评估模型对场景的理解。良好的 VQA 性能被视为该模型将在需要视觉和语言输入的更广泛任务中表现良好的证据。但是，场景感知 VQA 不能完全捕获输入偏差或评估由模态错位引起的幻觉。为了解决这个问题，我们提出了一种跨模态对齐相似性测试 (CAST) 来探测 VLM 在不同模态之间的自洽性。该测试涉及要求模型通过纯文本、纯图像或两者识别两个场景之间的相似性，然后评估它们生成的相似性的真实性。由于没有可供比较的基准，因此此评估不关注客观准确性，而是关注 VLM 在其输出中是否具有内部一致性。我们认为，虽然并非所有自洽模型都具有能力或准确性，但所有有能力的 VLM 都必须具有自洽性。

Title: GEIC: Universal and Multilingual Named Entity Recognition with Large Language Models

• Authors: Hanjun Luo, Yibing Jin, Xuecheng Liu, Tong Shang, Ruizhe Chen, Zuozhu Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11022
• Pdf URL: https://arxiv.org/pdf/2409.11022
• Copy Paste: [[2409.11022]] GEIC: Universal and Multilingual Named Entity Recognition with Large Language Models(https://arxiv.org/abs/2409.11022)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have supplanted traditional methods in numerous natural language processing tasks. Nonetheless, in Named Entity Recognition (NER), existing LLM-based methods underperform compared to baselines and require significantly more computational resources, limiting their application. In this paper, we introduce the task of generation-based extraction and in-context classification (GEIC), designed to leverage LLMs' prior knowledge and self-attention mechanisms for NER tasks. We then propose CascadeNER, a universal and multilingual GEIC framework for few-shot and zero-shot NER. CascadeNER employs model cascading to utilize two small-parameter LLMs to extract and classify independently, reducing resource consumption while enhancing accuracy. We also introduce AnythingNER, the first NER dataset specifically designed for LLMs, including 8 languages, 155 entity types and a novel dynamic categorization system. Experiments show that CascadeNER achieves state-of-the-art performance on low-resource and fine-grained scenarios, including CrossNER and FewNERD. Our work is openly accessible.
• 摘要：大型语言模型 (LLM) 已取代了众多自然语言处理任务中的传统方法。然而，在命名实体识别 (NER) 中，现有的基于 LLM 的方法与基线相比表现不佳，并且需要更多的计算资源，从而限制了它们的应用。在本文中，我们介绍了基于生成的提取和上下文分类 (GEIC) 任务，旨在利用 LLM 的先验知识和自注意力机制来完成 NER 任务。然后，我们提出了 CascadeNER，这是一个通用的多语言 GEIC 框架，用于少样本和零样本 NER。CascadeNER 采用模型级联来利用两个小参数 LLM 独立提取和分类，从而减少资源消耗并提高准确性。我们还介绍了 AnythingNER，这是第一个专为 LLM 设计的 NER 数据集，包括 8 种语言、155 种实体类型和一个新颖的动态分类系统。实验表明，CascadeNER 在低资源和细粒度场景中实现了最佳性能，包括 CrossNER 和 FewNERD。我们的工作是公开的。

Title: Hierarchical Narrative Analysis: Unraveling Perceptions of Generative AI

• Authors: Riona Matsuoka, Hiroki Matsumoto, Takahiro Yoshida, Tomohiro Watanabe, Ryoma Kondo, Ryohei Hisano
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11032
• Pdf URL: https://arxiv.org/pdf/2409.11032
• Copy Paste: [[2409.11032]] Hierarchical Narrative Analysis: Unraveling Perceptions of Generative AI(https://arxiv.org/abs/2409.11032)
• Keywords: language model, llm
• Abstract: Written texts reflect an author's perspective, making the thorough analysis of literature a key research method in fields such as the humanities and social sciences. However, conventional text mining techniques like sentiment analysis and topic modeling are limited in their ability to capture the hierarchical narrative structures that reveal deeper argumentative patterns. To address this gap, we propose a method that leverages large language models (LLMs) to extract and organize these structures into a hierarchical framework. We validate this approach by analyzing public opinions on generative AI collected by Japan's Agency for Cultural Affairs, comparing the narratives of supporters and critics. Our analysis provides clearer visualization of the factors influencing divergent opinions on generative AI, offering deeper insights into the structures of agreement and disagreement.
• 摘要：书面文本反映了作者的观点，因此对文献的彻底分析成为人文和社会科学等领域的重要研究方法。然而，情绪分析和主题建模等传统文本挖掘技术在捕捉揭示更深层次论证模式的分层叙事结构方面能力有限。为了解决这一差距，我们提出了一种利用大型语言模型 (LLM) 提取这些结构并将其组织成分层框架的方法。我们通过分析日本文化厅收集的关于生成式人工智能的公众意见来验证这种方法，比较了支持者和批评者的叙述。我们的分析更清晰地可视化了影响生成式人工智能不同意见的因素，为了解一致和不一致的结构提供了更深入的见解。

Title: Towards No-Code Programming of Cobots: Experiments with Code Synthesis by Large Code Models for Conversational Programming

• Authors: Kranti Chalamalasetti, Sherzod Hakimov, David Schlangen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11041
• Pdf URL: https://arxiv.org/pdf/2409.11041
• Copy Paste: [[2409.11041]] Towards No-Code Programming of Cobots: Experiments with Code Synthesis by Large Code Models for Conversational Programming(https://arxiv.org/abs/2409.11041)
• Keywords: language model, llm
• Abstract: While there has been a lot of research recently on robots in household environments, at the present time, most robots in existence can be found on shop floors, and most interactions between humans and robots happen there. ``Collaborative robots'' (cobots) designed to work alongside humans on assembly lines traditionally require expert programming, limiting ability to make changes, or manual guidance, limiting expressivity of the resulting programs. To address these limitations, we explore using Large Language Models (LLMs), and in particular, their abilities of doing in-context learning, for conversational code generation. As a first step, we define RATS, the ``Repetitive Assembly Task'', a 2D building task designed to lay the foundation for simulating industry assembly scenarios. In this task, a `programmer' instructs a cobot, using natural language, on how a certain assembly is to be built; that is, the programmer induces a program, through natural language. We create a dataset that pairs target structures with various example instructions (human-authored, template-based, and model-generated) and example code. With this, we systematically evaluate the capabilities of state-of-the-art LLMs for synthesising this kind of code, given in-context examples. Evaluating in a simulated environment, we find that LLMs are capable of generating accurate `first order code' (instruction sequences), but have problems producing `higher-order code' (abstractions such as functions, or use of loops).
• 摘要：虽然最近有很多关于家用机器人的研究，但目前，大多数现有的机器人都可以在车间找到，人类和机器人之间的大多数互动都发生在那里。设计用于在装配线上与人类一起工作的“协作机器人”（cobots）传统上需要专家编程，限制了进行更改的能力，或需要手动指导，限制了生成程序的表达能力。为了解决这些限制，我们探索使用大型语言模型（LLM），特别是它们进行上下文学习的能力，进行对话式代码生成。作为第一步，我们定义了 RATS，即“重复组装任务”，这是一个 2D 构建任务，旨在为模拟工业组装场景奠定基础。在这个任务中，“程序员”使用自然语言指导 cobot 如何构建某个组件；也就是说，程序员通过自然语言诱导程序。我们创建了一个数据集，将目标结构与各种示例指令（人类编写的、基于模板的和模型生成的）和示例代码配对。通过此方法，我们系统地评估了最先进的 LLM 综合此类代码的能力，并给出了上下文示例。在模拟环境中进行评估后，我们发现 LLM 能够生成准确的“一阶代码”（指令序列），但在生成“高阶代码”（函数等抽象或循环使用）方面存在问题。

Title: A Comprehensive Evaluation of Quantized Instruction-Tuned Large Language Models: An Experimental Analysis up to 405B

• Authors: Jemin Lee, Sihyeong Park, Jinse Kwon, Jihun Oh, Yongin Kwon
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11055
• Pdf URL: https://arxiv.org/pdf/2409.11055
• Copy Paste: [[2409.11055]] A Comprehensive Evaluation of Quantized Instruction-Tuned Large Language Models: An Experimental Analysis up to 405B(https://arxiv.org/abs/2409.11055)
• Keywords: language model, gpt, llm, hallucination
• Abstract: Prior research works have evaluated quantized LLMs using limited metrics such as perplexity or a few basic knowledge tasks and old datasets. Additionally, recent large-scale models such as Llama 3.1 with up to 405B have not been thoroughly examined. This paper evaluates the performance of instruction-tuned LLMs across various quantization methods (GPTQ, AWQ, SmoothQuant, and FP8) on models ranging from 7B to 405B. Using 13 benchmarks, we assess performance across six task types: commonsense Q\&A, knowledge and language understanding, instruction following, hallucination detection, mathematics, and dialogue. Our key findings reveal that (1) quantizing a larger LLM to a similar size as a smaller FP16 LLM generally performs better across most benchmarks, except for hallucination detection and instruction following; (2) performance varies significantly with different quantization methods, model size, and bit-width, with weight-only methods often yielding better results in larger models; (3) task difficulty does not significantly impact accuracy degradation due to quantization; and (4) the MT-Bench evaluation method has limited discriminatory power among recent high-performing LLMs.
• 摘要：先前的研究工作使用有限的指标（例如困惑度或一些基础知识任务和旧数据集）来评估量化的 LLM。此外，最近的大规模模型（例如高达 405B 的 Llama 3.1）尚未得到彻底研究。本文评估了在 7B 到 405B 的模型上使用各种量化方法（GPTQ、AWQ、SmoothQuant 和 FP8）的指令调整的 LLM 的性能。使用 13 个基准，我们评估了六种任务类型的性能：常识问答、知识和语言理解、指令遵循、幻觉检测、数学和对话。我们的主要发现表明：（1）将较大的 LLM 量化为与较小的 FP16 LLM 相似的大小通常在大多数基准测试中表现更好，幻觉检测和指令遵循除外；（2）性能因量化方法、模型大小和位宽的不同而有很大差异，仅使用权重的方法通常会在较大的模型中产生更好的结果；（3）任务难度不会显着影响量化导致的准确度下降（4）MT-Bench 评估方法对近期表现优异的 LLM 的辨别能力有限。

Title: Large Language Models are Good Multi-lingual Learners : When LLMs Meet Cross-lingual Prompts

• Authors: Teng Wang, Zhenqi He, Wing-Yin Yu, Xiaojin Fu, Xiongwei Han
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11056
• Pdf URL: https://arxiv.org/pdf/2409.11056
• Copy Paste: [[2409.11056]] Large Language Models are Good Multi-lingual Learners : When LLMs Meet Cross-lingual Prompts(https://arxiv.org/abs/2409.11056)
• Keywords: language model, llm, long context, prompt
• Abstract: With the advent of Large Language Models (LLMs), generating rule-based data for real-world applications has become more accessible. Due to the inherent ambiguity of natural language and the complexity of rule sets, especially in long contexts, LLMs often struggle to follow all specified rules, frequently omitting at least one. To enhance the reasoning and understanding of LLMs on long and complex contexts, we propose a novel prompting strategy Multi-Lingual Prompt, namely MLPrompt, which automatically translates the error-prone rule that an LLM struggles to follow into another language, thus drawing greater attention to it. Experimental results on public datasets across various tasks have shown MLPrompt can outperform state-of-the-art prompting methods such as Chain of Thought, Tree of Thought, and Self-Consistency. Additionally, we introduce a framework integrating MLPrompt with an auto-checking mechanism for structured data generation, with a specific case study in text-to-MIP instances. Further, we extend the proposed framework for text-to-SQL to demonstrate its generation ability towards structured data synthesis.
• 摘要：随着大型语言模型 (LLM) 的出现，为实际应用生成基于规则的数据变得更加容易。由于自然语言固有的歧义性和规则集的复杂性（尤其是在长上下文中），LLM 通常难以遵循所有指定的规则，经常会遗漏至少一条规则。为了增强 LLM 在长而复杂的上下文中的推理和理解能力，我们提出了一种新颖的提示策略多语言提示，即 MLPrompt，它会自动将 LLM 难以遵循的易错规则翻译成另一种语言，从而引起人们对它的更多关注。在各种任务的公共数据集上的实验结果表明，MLPrompt 的表现可以优于最先进的提示方法，例如思维链、思维树和自洽性。此外，我们介绍了一个将 MLPrompt 与自动检查机制相结合的框架，用于结构化数据生成，并在文本到 MIP 实例中进行了具体的案例研究。此外，我们扩展了所提出的文本到 SQL 框架，以展示其对结构化数据合成的生成能力。

Title: KVPruner: Structural Pruning for Faster and Memory-Efficient Large Language Models

• Authors: Bo Lv, Quan Zhou, Xuanang Ding, Yan Wang, Zeming Ma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11057
• Pdf URL: https://arxiv.org/pdf/2409.11057
• Copy Paste: [[2409.11057]] KVPruner: Structural Pruning for Faster and Memory-Efficient Large Language Models(https://arxiv.org/abs/2409.11057)
• Keywords: language model
• Abstract: The bottleneck associated with the key-value(KV) cache presents a significant challenge during the inference processes of large language models. While depth pruning accelerates inference, it requires extensive recovery training, which can take up to two weeks. On the other hand, width pruning retains much of the performance but offers slight speed gains. To tackle these challenges, we propose KVPruner to improve model efficiency while maintaining performance. Our method uses global perplexity-based analysis to determine the importance ratio for each block and provides multiple strategies to prune non-essential KV channels within blocks. Compared to the original model, KVPruner reduces runtime memory usage by 50% and boosts throughput by over 35%. Additionally, our method requires only two hours of LoRA fine-tuning on small datasets to recover most of the performance.
• 摘要：与键值 (KV) 缓存相关的瓶颈在大型语言模型的推理过程中带来了重大挑战。虽然深度修剪可以加速推理，但它需要大量的恢复训练，这可能需要长达两周的时间。另一方面，宽度修剪保留了大部分性能，但速度略有提升。为了应对这些挑战，我们提出了 KVPruner 来提高模型效率，同时保持性能。我们的方法使用基于全局困惑度的分析来确定每个块的重要性比率，并提供多种策略来修剪块内非必要的 KV 通道。与原始模型相比，KVPruner 将运行时内存使用量减少了 50%，并将吞吐量提高了 35% 以上。此外，我们的方法只需要在小数据集上进行两个小时的 LoRA 微调即可恢复大部分性能。

Title: RoMath: A Mathematical Reasoning Benchmark in Romanian

• Authors: Adrian Cosma, Ana-Maria Bucur, Emilian Radoi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11074
• Pdf URL: https://arxiv.org/pdf/2409.11074
• Copy Paste: [[2409.11074]] RoMath: A Mathematical Reasoning Benchmark in Romanian(https://arxiv.org/abs/2409.11074)
• Keywords: language model
• Abstract: Mathematics has long been conveyed through natural language, primarily for human understanding. With the rise of mechanized mathematics and proof assistants, there is a growing need to understand informal mathematical text, yet most existing benchmarks focus solely on English, overlooking other languages. This paper introduces RoMath, a Romanian mathematical reasoning benchmark suite comprising three datasets: RoMath-Baccalaureate, RoMath-Competitions and RoMath-Synthetic, which cover a range of mathematical domains and difficulty levels, aiming to improve non-English language models and promote multilingual AI development. By focusing on Romanian, a low-resource language with unique linguistic features, RoMath addresses the limitations of Anglo-centric models and emphasizes the need for dedicated resources beyond simple automatic translation. We benchmark several open-weight language models, highlighting the importance of creating resources for underrepresented languages. We make the code and dataset available.
• 摘要：长期以来，数学都是通过自然语言来传达的，主要是为了便于人类理解。随着机械化数学和证明助手的兴起，人们越来越需要理解非正式的数学文本，但现有的大多数基准测试只关注英语，而忽略了其他语言。本文介绍了罗马尼亚数学推理基准套件 RoMath，它包含三个数据集：RoMath-Baccalaureate、RoMath-Competitions 和 RoMath-Synthetic，涵盖了一系列数学领域和难度级别，旨在改进非英语语言模型并促进多语言人工智能的发展。通过专注于罗马尼亚语（一种具有独特语言特征的资源匮乏的语言），RoMath 解决了以英语为中心的模型的局限性，并强调了除了简单的自动翻译之外还需要专用资源。我们对几种开放权重语言模型进行了基准测试，强调了为代表性不足的语言创建资源的重要性。我们提供代码和数据集。

Title: Strategic Insights in Human and Large Language Model Tactics at Word Guessing Games

• Authors: Matīss Rikters, Sanita Reinsone
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2409.11112
• Pdf URL: https://arxiv.org/pdf/2409.11112
• Copy Paste: [[2409.11112]] Strategic Insights in Human and Large Language Model Tactics at Word Guessing Games(https://arxiv.org/abs/2409.11112)
• Keywords: language model, hallucination
• Abstract: At the beginning of 2022, a simplistic word-guessing game took the world by storm and was further adapted to many languages beyond the original English version. In this paper, we examine the strategies of daily word-guessing game players that have evolved during a period of over two years. A survey gathered from 25% of frequent players reveals their strategies and motivations for continuing the daily journey. We also explore the capability of several popular open-access large language model systems and open-source models at comprehending and playing the game in two different languages. Results highlight the struggles of certain models to maintain correct guess length and generate repetitions, as well as hallucinations of non-existent words and inflections.
• 摘要：2022 年初，一款简单的猜词游戏风靡全球，并被改编为除原始英语版本之外的多种语言。在本文中，我们研究了日常猜词游戏玩家在两年多的时间里演变的策略。一项对 25% 的常客进行的调查揭示了他们继续日常旅程的策略和动机。我们还探讨了几种流行的开放式大型语言模型系统和开源模型在理解和玩两种不同语言的游戏方面的能力。结果突出了某些模型在保持正确的猜测长度和产生重复以及对不存在的单词和词形变化产生幻觉方面所面临的困难。

Title: Diversity-grounded Channel Prototypical Learning for Out-of-Distribution Intent Detection

• Authors: Bo Liu, Liming Zhan, Yujie Feng, Zexin Lu, Chengqiang Xie, Lei Xue, Xiao-Ming Wu, Albert Y.S. Lam
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11114
• Pdf URL: https://arxiv.org/pdf/2409.11114
• Copy Paste: [[2409.11114]] Diversity-grounded Channel Prototypical Learning for Out-of-Distribution Intent Detection(https://arxiv.org/abs/2409.11114)
• Keywords: language model, llm, prompt
• Abstract: In the realm of task-oriented dialogue systems, a robust intent detection mechanism must effectively handle malformed utterances encountered in real-world scenarios. This study presents a novel fine-tuning framework for large language models (LLMs) aimed at enhancing in-distribution (ID) intent classification and out-of-distribution (OOD) intent detection, which utilizes semantic matching with prototypes derived from ID class names. By harnessing the highly distinguishable representations of LLMs, we construct semantic prototypes for each ID class using a diversity-grounded prompt tuning approach. We rigorously test our framework in a challenging OOD context, where ID and OOD classes are semantically close yet distinct, referred to as \emph{near} OOD detection. For a thorough assessment, we benchmark our method against the prevalent fine-tuning approaches. The experimental findings reveal that our method demonstrates superior performance in both few-shot ID intent classification and near-OOD intent detection tasks.
• 摘要：在面向任务的对话系统领域，强大的意图检测机制必须有效处理在现实场景中遇到的畸形话语。本研究提出了一种用于大型语言模型 (LLM) 的新型微调框架，旨在增强分布内 (ID) 意图分类和分布外 (OOD) 意图检测，该框架利用从 ID 类名派生的原型进行语义匹配。通过利用 LLM 的高度可区分表示，我们使用基于多样性的快速调整方法为每个 ID 类构建语义原型。我们在具有挑战性的 OOD 环境中严格测试我们的框架，其中 ID 和 OOD 类在语义上接近但又不同，称为 \emph{near} OOD 检测。为了进行全面评估，我们将我们的方法与流行的微调方法进行了对比。实验结果表明，我们的方法在小样本 ID 意图分类和近 OOD 意图检测任务中都表现出色。

Title: Semformer: Transformer Language Models with Semantic Planning

• Authors: Yongjing Yin, Junran Ding, Kai Song, Yue Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11143
• Pdf URL: https://arxiv.org/pdf/2409.11143
• Copy Paste: [[2409.11143]] Semformer: Transformer Language Models with Semantic Planning(https://arxiv.org/abs/2409.11143)
• Keywords: language model
• Abstract: Next-token prediction serves as the dominant component in current neural language models. During the training phase, the model employs teacher forcing, which predicts tokens based on all preceding ground truth tokens. However, this approach has been found to create shortcuts, utilizing the revealed prefix to spuriously fit future tokens, potentially compromising the accuracy of the next-token predictor. In this paper, we introduce Semformer, a novel method of training a Transformer language model that explicitly models the semantic planning of response. Specifically, we incorporate a sequence of planning tokens into the prefix, guiding the planning token representations to predict the latent semantic representations of the response, which are induced by an autoencoder. In a minimal planning task (i.e., graph path-finding), our model exhibits near-perfect performance and effectively mitigates shortcut learning, a feat that standard training methods and baseline models have been unable to accomplish. Furthermore, we pretrain Semformer from scratch with 125M parameters, demonstrating its efficacy through measures of perplexity, in-context learning, and fine-tuning on summarization tasks.
• 摘要：下一个标记预测是当前神经语言模型中的主要组成部分。在训练阶段，该模型采用教师强制，根据所有先前的地面实况标记预测标记。然而，这种方法被发现会创建捷径，利用已揭示的前缀来虚假地拟合未来的标记，可能会损害下一个标记预测器的准确性。在本文中，我们介绍了 Semformer，这是一种训练 Transformer 语言模型的新方法，该模型明确模拟响应的语义规划。具体来说，我们将一系列规划标记合并到前缀中，引导规划标记表示预测响应的潜在语义表示，这些表示由自动编码器诱导。在最小规划任务（即图路径查找）中，我们的模型表现出近乎完美的性能并有效地缓解了捷径学习，这是标准训练方法和基线模型无法实现的壮举。此外，我们使用 125M 个参数从头开始对 Semformer 进行预训练，通过困惑度测量、上下文学习和总结任务的微调来证明其有效性。

Title: Reasoning Graph Enhanced Exemplars Retrieval for In-Context Learning

• Authors: Yukang Lin, Bingchen Zhong, Shuoran Jiang, Joanna Siebert, Qingcai Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11147
• Pdf URL: https://arxiv.org/pdf/2409.11147
• Copy Paste: [[2409.11147]] Reasoning Graph Enhanced Exemplars Retrieval for In-Context Learning(https://arxiv.org/abs/2409.11147)
• Keywords: language model, llm
• Abstract: Large language models(LLMs) have exhibited remarkable few-shot learning capabilities and unified the paradigm of NLP tasks through the in-context learning(ICL) technique. Despite the success of ICL, the quality of the exemplar demonstrations can significantly influence the LLM's performance. Existing exemplar selection methods mainly focus on the semantic similarity between queries and candidate exemplars. On the other hand, the logical connections between reasoning steps can be beneficial to depict the problem-solving process as well. In this paper, we proposes a novel method named Reasoning Graph-enhanced Exemplar Retrieval(RGER). RGER first quires LLM to generate an initial response, then expresses intermediate problem-solving steps to a graph structure. After that, it employs graph kernel to select exemplars with semantic and structural similarity. Extensive experiments demonstrate the structural relationship is helpful to the alignment of queries and candidate exemplars. The efficacy of RGER on math and logit reasoning tasks showcases its superiority over state-of-the-art retrieval-based approaches. Our code is released at this https URL.
• 摘要：大型语言模型 (LLM) 表现出卓越的小样本学习能力，并通过上下文学习 (ICL) 技术统一了 NLP 任务的范式。尽管 ICL 取得了成功，但范例演示的质量会显著影响 LLM 的性能。现有的范例选择方法主要关注查询和候选范例之间的语义相似性。另一方面，推理步骤之间的逻辑联系也有助于刻画问题解决的过程。在本文中，我们提出了一种名为推理图增强范例检索 (RGER) 的新方法。RGER 首先要求 LLM 生成初始响应，然后将中间问题解决步骤表示为图结构。之后，它使用图核来选择具有语义和结构相似性的范例。大量实验表明结构关系有助于查询和候选范例的对齐。 RGER 在数学和逻辑推理任务上的有效性表明它比最先进的基于检索的方法更优越。我们的代码发布在此 https URL 上。

Title: Improving the Efficiency of Visually Augmented Language Models

• Authors: Paula Ontalvilla, Aitor Ormazabal, Gorka Azkune
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11148
• Pdf URL: https://arxiv.org/pdf/2409.11148
• Copy Paste: [[2409.11148]] Improving the Efficiency of Visually Augmented Language Models(https://arxiv.org/abs/2409.11148)
• Keywords: language model
• Abstract: Despite the impressive performance of autoregressive Language Models (LM) it has been shown that due to reporting bias, LMs lack visual knowledge, i.e. they do not know much about the visual world and its properties. To augment LMs with visual knowledge, existing solutions often rely on explicit images, requiring time-consuming retrieval or image generation systems. This paper shows that explicit images are not necessary to visually augment an LM. Instead, we use visually-grounded text representations obtained from the well-known CLIP multimodal system. For a fair comparison, we modify VALM, a visually-augmented LM which uses image retrieval and representation, to work directly with visually-grounded text representations. We name this new model BLIND-VALM. We show that BLIND-VALM performs on par with VALM for Visual Language Understanding (VLU), Natural Language Understanding (NLU) and Language Modeling tasks, despite being significantly more efficient and simpler. We also show that scaling up our model within the compute budget of VALM, either increasing the model or pre-training corpus size, we outperform VALM for all the evaluation tasks.
• 摘要：尽管自回归语言模型 (LM) 的性能令人印象深刻，但事实证明由于报告偏差，LM 缺乏视觉知识，即它们对视觉世界及其属性了解不多。为了用视觉知识增强 LM，现有的解决方案通常依赖于显式图像，需要耗时的检索或图像生成系统。本文表明，显式图像对于视觉增强 LM 来说不是必需的。相反，我们使用从著名的 CLIP 多模态系统获得的基于视觉的文本表示。为了进行公平的比较，我们修改了 VALM（一种使用图像检索和表示的视觉增强 LM），以直接使用基于视觉的文本表示。我们将这个新模型命名为 BLIND-VALM。我们表明，尽管 BLIND-VALM 效率更高、更简单，但在视觉语言理解 (VLU)、自然语言理解 (NLU) 和语言建模任务方面的表现与 VALM 相当。我们还表明，在 VALM 的计算预算内扩大我们的模型，无论是增加模型还是预训练语料库的大小，我们在所有评估任务中的表现都优于 VALM。

Title: SAGED: A Holistic Bias-Benchmarking Pipeline for Language Models with Customisable Fairness Calibration

• Authors: Xin Guan, Nathaniel Demchak, Saloni Gupta, Ze Wang, Ediz Ertekin Jr., Adriano Koshiyama, Emre Kazim, Zekun Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11149
• Pdf URL: https://arxiv.org/pdf/2409.11149
• Copy Paste: [[2409.11149]] SAGED: A Holistic Bias-Benchmarking Pipeline for Language Models with Customisable Fairness Calibration(https://arxiv.org/abs/2409.11149)
• Keywords: language model, prompt
• Abstract: The development of unbiased large language models is widely recognized as crucial, yet existing benchmarks fall short in detecting biases due to limited scope, contamination, and lack of a fairness baseline. SAGED(-Bias) is the first holistic benchmarking pipeline to address these problems. The pipeline encompasses five core stages: scraping materials, assembling benchmarks, generating responses, extracting numeric features, and diagnosing with disparity metrics. SAGED includes metrics for max disparity, such as impact ratio, and bias concentration, such as Max Z-scores. Noticing that assessment tool bias and contextual bias in prompts can distort evaluation, SAGED implements counterfactual branching and baseline calibration for mitigation. For demonstration, we use SAGED on G20 Countries with popular 8b-level models including Gemma2, Llama3.1, Mistral, and Qwen2. With sentiment analysis, we find that while Mistral and Qwen2 show lower max disparity and higher bias concentration than Gemma2 and Llama3.1, all models are notably biased against countries like Russia and (except for Qwen2) China. With further experiments to have models role-playing U.S. (vice-/former-) presidents, we see bias amplifies and shifts in heterogeneous directions. Moreover, we see Qwen2 and Mistral not engage in role-playing, while Llama3.1 and Gemma2 role-play Trump notably more intensively than Biden and Harris, indicating role-playing performance bias in these models.
• 摘要：人们普遍认为，开发无偏大型语言模型至关重要，但现有的基准由于范围有限、污染和缺乏公平性基线而无法检测到偏差。SAGED(-Bias) 是第一个解决这些问题的整体基准测试流程。该流程包含五个核心阶段：抓取材料、组装基准、生成响应、提取数字特征以及使用差异指标进行诊断。SAGED 包括最大差异指标（例如影响率）和偏差集中度指标（例如最大 Z 分数）。SAGED 注意到评估工具偏差和提示中的上下文偏差会扭曲评估，因此实施了反事实分支和基线校准以缓解这种情况。为了演示，我们在 G20 国家/地区使用 SAGED，其中包括流行的 8b 级模型，包括 Gemma2、Llama3.1、Mistral 和 Qwen2。通过情绪分析，我们发现，虽然 Mistral 和 Qwen2 的最大差异较低，偏见集中度高于 Gemma2 和 Llama3.1，但所有模型都明显偏向俄罗斯和中国（Qwen2 除外）等国家。通过进一步实验让模型扮演美国（副/前）总统，我们发现偏见在不断扩大，并朝着不同的方向转变。此外，我们发现 Qwen2 和 Mistral 没有参与角色扮演，而 Llama3.1 和 Gemma2 扮演特朗普的次数明显多于拜登和哈里斯，这表明这些模型在角色扮演方面存在表现偏差。

Title: Self-Evolutionary Large Language Models through Uncertainty-Enhanced Preference Optimization

• Authors: Jianing Wang, Yang Zhou, Xiaocheng Zhang, Mengjiao Bao, Peng Yan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11212
• Pdf URL: https://arxiv.org/pdf/2409.11212
• Copy Paste: [[2409.11212]] Self-Evolutionary Large Language Models through Uncertainty-Enhanced Preference Optimization(https://arxiv.org/abs/2409.11212)
• Keywords: language model, llm
• Abstract: Iterative preference optimization has recently become one of the de-facto training paradigms for large language models (LLMs), but the performance is still underwhelming due to too much noisy preference data yielded in the loop. To combat this issue, we present an \textbf{U}ncertainty-enhanced \textbf{P}reference \textbf{O}ptimization (UPO) framework to make the LLM self-evolve with reliable feedback. The key idea is mitigating the noisy preference data derived from the current policy and reward models by performing pair-wise uncertainty estimation and judiciously reliable feedback sampling. To reach this goal, we thus introduce an estimator model, which incorporates Monte Carlo (MC) dropout in Bayesian neural network (BNN) to perform uncertainty estimation for the preference data derived from the LLM policy. Compared to the existing methods that directly filter generated responses based on the reward score, the estimator focuses on the model uncertainty in a pair-wise manner and effectively bypasses the confirmation bias problem of the reward model. Additionally, we also propose an uncertainty-enhanced self-evolution algorithm to improve the robustness of preference optimization and encourage the LLM to generate responses with both high reward and certainty. Extensive experiments over multiple benchmarks demonstrate that our framework substantially alleviates the noisy problem and improves the performance of iterative preference optimization.
• 摘要：迭代偏好优化最近已成为大型语言模型 (LLM) 的事实上的训练范例之一，但由于循环中产生的偏好数据过多，其性能仍然不尽如人意。为了解决这个问题，我们提出了一个 \textbf{U} 不确定性增强 \textbf{P} 参考 \textbf{O} 优化 (UPO) 框架，使 LLM 能够通过可靠的反馈进行自我进化。关键思想是通过执行成对不确定性估计和明智可靠的反馈采样来减轻从当前策略和奖励模型中得出的嘈杂偏好数据。为了实现这一目标，我们引入了一个估计器模型，该模型将蒙特卡洛 (MC) 辍学纳入贝叶斯神经网络 (BNN) 中，以对从 LLM 策略中得出的偏好数据进行不确定性估计。与现有的直接根据奖励分数过滤生成的响应的方法相比，该估计器以成对的方式关注模型不确定性，并有效地绕过了奖励模型的确认偏差问题。此外，我们还提出了一种不确定性增强的自进化算法，以提高偏好优化的鲁棒性，并鼓励 LLM 生成具有高奖励和确定性的响应。在多个基准上进行的大量实验表明，我们的框架大大缓解了噪声问题并提高了迭代偏好优化的性能。

Title: Exploring ChatGPT-based Augmentation Strategies for Contrastive Aspect-based Sentiment Analysis

• Authors: Lingling Xu, Haoran Xie, S. Joe Qin, Fu Lee Wang, Xiaohui Tao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11218
• Pdf URL: https://arxiv.org/pdf/2409.11218
• Copy Paste: [[2409.11218]] Exploring ChatGPT-based Augmentation Strategies for Contrastive Aspect-based Sentiment Analysis(https://arxiv.org/abs/2409.11218)
• Keywords: language model, gpt, llm, chat
• Abstract: Aspect-based sentiment analysis (ABSA) involves identifying sentiment towards specific aspect terms in a sentence and allows us to uncover nuanced perspectives and attitudes on particular aspects of a product, service, or topic. However, the scarcity of labeled data poses a significant challenge to training high-quality models. To address this issue, we explore the potential of data augmentation using ChatGPT, a well-performing large language model (LLM), to enhance the sentiment classification performance towards aspect terms. Specifically, we explore three data augmentation strategies based on ChatGPT: context-focused, aspect-focused, and context-aspect data augmentation techniques. Context-focused data augmentation focuses on changing the word expression of context words in the sentence while keeping aspect terms unchanged. In contrast, aspect-focused data augmentation aims to change aspect terms but keep context words unchanged. Context-Aspect data augmentation integrates the above two data augmentations to generate augmented samples. Furthermore, we incorporate contrastive learning into the ABSA tasks to improve performance. Extensive experiments show that all three data augmentation techniques lead to performance improvements, with the context-aspect data augmentation strategy performing best and surpassing the performance of the baseline models.
• 摘要：基于方面的情绪分析 (ABSA) 涉及识别对句子中特定方面术语的情绪，并使我们能够发现对产品、服务或主题的特定方面的细微观点和态度。然而，标记数据的稀缺对训练高质量模型构成了重大挑战。为了解决这个问题，我们探索了使用性能良好的大型语言模型 (LLM) ChatGPT 进行数据增强的潜力，以增强对方面术语的情绪分类性能。具体来说，我们探索了基于 ChatGPT 的三种数据增强策略：以上下文为中心、以方面为中心和上下文-方面数据增强技术。以上下文为中心的数据增强侧重于改变句子中上下文词的词语表达，同时保持方面术语不变。相反，以方面为中心的数据增强旨在改变方面术语但保持上下文词不变。上下文-方面数据增强集成了上述两种数据增强以生成增强样本。此外，我们将对比学习纳入 ABSA 任务以提高性能。大量实验表明，三种数据增强技术均能提高性能，其中上下文方面数据增强策略表现最佳，超越了基线模型的性能。

Title: Fast Analysis of the OpenAI O1-Preview Model in Solving Random K-SAT Problem: Does the LLM Solve the Problem Itself or Call an External SAT Solver?

• Authors: Raffaele Marino
• Subjects: cs.CL, cond-mat.dis-nn, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11232
• Pdf URL: https://arxiv.org/pdf/2409.11232
• Copy Paste: [[2409.11232]] Fast Analysis of the OpenAI O1-Preview Model in Solving Random K-SAT Problem: Does the LLM Solve the Problem Itself or Call an External SAT Solver?(https://arxiv.org/abs/2409.11232)
• Keywords: llm
• Abstract: In this manuscript I present an analysis on the performance of OpenAI O1-preview model in solving random K-SAT instances for K$\in {2,3,4}$ as a function of $\alpha=M/N$ where $M$ is the number of clauses and $N$ is the number of variables of the satisfiable problem. I show that the model can call an external SAT solver to solve the instances, rather than solving them directly. Despite using external solvers, the model reports incorrect assignments as output. Moreover, I propose and present an analysis to quantify whether the OpenAI O1-preview model demonstrates a spark of intelligence or merely makes random guesses when outputting an assignment for a Boolean satisfiability problem.
• 摘要：在本文中，我分析了 OpenAI O1 预览模型在解决随机 K-SAT 实例（K$\in {2,3,4}$）时的性能，该模型是 $\alpha=M/N$ 函数，其中 $M$ 是子句数，$N$ 是可满足问题的变量数。我表明，该模型可以调用外部 SAT 求解器来解决实例，而不是直接解决它们。尽管使用了外部求解器，但该模型仍会将错误的分配报告为输出。此外，我提出并展示了一项分析，以量化 OpenAI O1 预览模型在输出布尔可满足性问题的分配时是表现出了智慧的火花还是仅仅是随机猜测。

Title: Evaluating the Impact of Compression Techniques on Task-Specific Performance of Large Language Models

• Authors: Bishwash Khanal, Jeffery M. Capone
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11233
• Pdf URL: https://arxiv.org/pdf/2409.11233
• Copy Paste: [[2409.11233]] Evaluating the Impact of Compression Techniques on Task-Specific Performance of Large Language Models(https://arxiv.org/abs/2409.11233)
• Keywords: language model, gpt, llm
• Abstract: Large language models (LLMs) offer powerful capabilities but incur substantial computational costs, driving the need for efficient compression techniques. This study evaluates the impact of popular compression methods - Magnitude Pruning, SparseGPT, and Wanda - on the LLaMA-2-7B model, focusing on the trade-offs between model size reduction, downstream task performance, and the role of calibration data. Our findings reveal that while SparseGPT and Wanda preserve perplexity even at 50% sparsity, they suffer significant degradation on downstream tasks, highlighting the inadequacy of perplexity as the sole evaluation metric. To address this, we introduce Jensen-Shannon (JS) Divergence as a more comprehensive metric that captures nuanced changes in model behavior post-compression. We further demonstrate that task-specific calibration data significantly enhances the downstream performance of compressed models compared to general calibration data. This research underscores the necessity for diverse evaluation metrics and careful calibration data selection to fully understand the complexities of LLM compression and its implications for practical applications.
• 摘要：大型语言模型 (LLM) 提供了强大的功能，但会产生大量的计算成本，因此需要高效的压缩技术。本研究评估了流行的压缩方法（Magnitude Pruning、SparseGPT 和 Wanda）对 LLaMA-2-7B 模型的影响，重点关注模型大小减小、下游任务性能和校准数据的作用之间的权衡。我们的研究结果表明，虽然 SparseGPT 和 Wanda 即使在 50% 的稀疏度下也能保持困惑度，但它们在下游任务中的表现会显著下降，这凸显了困惑度作为唯一评估指标的不足。为了解决这个问题，我们引入了 Jensen-Shannon (JS) 散度作为一个更全面的指标，可以捕捉压缩后模型行为的细微变化。我们进一步证明，与一般校准数据相比，特定于任务的校准数据显著提高了压缩模型的下游性能。这项研究强调了多样化评估指标和谨慎选择校准数据的必要性，以充分理解 LLM 压缩的复杂性及其对实际应用的影响。

Title: LLM-as-a-Judge & Reward Model: What They Can and Cannot Do

• Authors: Guijin Son, Hyunwoo Ko, Hoyoung Lee, Yewon Kim, Seunghyeok Hong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11239
• Pdf URL: https://arxiv.org/pdf/2409.11239
• Copy Paste: [[2409.11239]] LLM-as-a-Judge & Reward Model: What They Can and Cannot Do(https://arxiv.org/abs/2409.11239)
• Keywords: language model, llm
• Abstract: LLM-as-a-Judge and reward models are widely used alternatives of multiple-choice questions or human annotators for large language model (LLM) evaluation. Their efficacy shines in evaluating long-form responses, serving a critical role as evaluators of leaderboards and as proxies to align LLMs via reinforcement learning. However, despite their popularity, their effectiveness outside of English remains largely unexplored. In this paper, we conduct a comprehensive analysis on automated evaluators, reporting key findings on their behavior in a non-English environment. First, we discover that English evaluation capabilities significantly influence language-specific capabilities, often more than the language proficiency itself, enabling evaluators trained in English to easily transfer their skills to other languages. Second, we identify critical shortcomings, where LLMs fail to detect and penalize errors, such as factual inaccuracies, cultural misrepresentations, and the presence of unwanted language. Finally, we release Kudge, the first non-English meta-evaluation dataset containing 5,012 human annotations in Korean.
• 摘要：LLM-as-a-Judge 和奖励模型是用于大型语言模型 (LLM) 评估的多项选择题或人工注释者的广泛使用的替代方案。它们在评估长篇回答方面非常有效，在排行榜评估者和通过强化学习对齐 LLM 的代理中发挥着关键作用。然而，尽管它们很受欢迎，但它们在英语之外的有效性仍未得到充分探索。在本文中，我们对自动评估器进行了全面分析，报告了它们在非英语环境中行为的关键发现。首先，我们发现英语评估能力显著影响语言特定能力，通常比语言能力本身更重要，这使得接受过英语培训的评估者能够轻松地将他们的技能转移到其他语言上。其次，我们发现了关键的缺陷，即 LLM 无法检测和惩罚错误，例如事实不准确、文化误解和存在不必要的语言。最后，我们发布了 Kudge，这是第一个非英语元评估数据集，包含 5,012 条韩语人工注释。

Title: Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse

• Authors: Maojia Song, Shang Hong Sim, Rishabh Bhardwaj, Hai Leong Chieu, Navonil Majumder, Soujanya Poria
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11242
• Pdf URL: https://arxiv.org/pdf/2409.11242
• Copy Paste: [[2409.11242]] Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse(https://arxiv.org/abs/2409.11242)
• Keywords: llm, prompt, retrieval-augmented generation
• Abstract: LLMs are an integral part of retrieval-augmented generation (RAG) systems. While many studies focus on evaluating the quality of end-to-end RAG systems, there is a lack of research on understanding the appropriateness of an LLM for the RAG task. Thus, we introduce a new metric, Trust-Score, that provides a holistic evaluation of the trustworthiness of LLMs in an RAG framework. We show that various prompting methods, such as in-context learning, fail to adapt LLMs effectively to the RAG task. Thus, we propose Trust-Align, a framework to align LLMs for higher Trust-Score. LLaMA-3-8b, aligned with our method, significantly outperforms open-source LLMs of comparable sizes on ASQA (up 10.7), QAMPARI (up 29.2) and ELI5 (up 14.9). We release our code at: this https URL.
• 摘要：LLM 是检索增强生成 (RAG) 系统不可或缺的一部分。虽然许多研究都侧重于评估端到端 RAG 系统的质量，但缺乏对理解 LLM 是否适合 RAG 任务的研究。因此，我们引入了一个新的指标 Trust-Score，它提供了对 RAG 框架中 LLM 可信度的整体评估。我们表明，各种提示方法（例如情境学习）都无法有效地使 LLM 适应 RAG 任务。因此，我们提出了 Trust-Align，这是一个对齐 LLM 以获得更高 Trust-Score 的框架。与我们的方法一致，LLaMA-3-8b 在 ASQA（上升 10.7）、QAMPARI（上升 29.2）和 ELI5（上升 14.9）上的表现明显优于同等规模的开源 LLM。我们在以下 https URL 上发布了我们的代码。

Title: Linear Recency Bias During Training Improves Transformers' Fit to Reading Times

• Authors: Christian Clark, Byung-Doh Oh, William Schuler
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11250
• Pdf URL: https://arxiv.org/pdf/2409.11250
• Copy Paste: [[2409.11250]] Linear Recency Bias During Training Improves Transformers' Fit to Reading Times(https://arxiv.org/abs/2409.11250)
• Keywords: language model
• Abstract: Recent psycholinguistic research has compared human reading times to surprisal estimates from language models to study the factors shaping human sentence processing difficulty. Previous studies have shown a strong fit between surprisal values from Transformers and reading times. However, standard Transformers work with a lossless representation of the entire previous linguistic context, unlike models of human language processing that include memory decay. To bridge this gap, this paper evaluates a modification of the Transformer model that uses ALiBi (Press et al., 2022), a recency bias added to attention scores. Surprisal estimates with ALiBi show an improved fit to human reading times compared to a standard Transformer baseline. A subsequent analysis of attention heads suggests that ALiBi's mixture of slopes -- which determine the rate of memory decay in each attention head -- may play a role in the improvement by helping models with ALiBi to track different kinds of linguistic dependencies.
• 摘要：最近的心理语言学研究将人类阅读时间与语言模型的意外估计值进行了比较，以研究影响人类句子处理难度的因素。先前的研究表明，Transformers 的意外值与阅读时间之间存在很强的契合度。然而，与包括记忆衰退的人类语言处理模型不同，标准 Transformers 使用的是整个先前语言环境的无损表示。为了弥合这一差距，本文评估了使用 ALiBi（Press 等人，2022 年）的 Transformer 模型的修改，这是一种添加到注意力分数中的新近偏差。与标准 Transformer 基线相比，ALiBi 的意外估计值显示出与人类阅读时间的更好的契合度。随后对注意力头的分析表明，ALiBi 的斜率混合（决定了每个注意力头的记忆衰退率）可能在改进中发挥作用，帮助使用 ALiBi 的模型跟踪不同类型的语言依赖关系。

Title: The Art of Storytelling: Multi-Agent Generative AI for Dynamic Multimodal Narratives

• Authors: Samee Arif, Taimoor Arif, Aamina Jamal Khan, Muhammad Saad Haroon, Agha Ali Raza, Awais Athar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11261
• Pdf URL: https://arxiv.org/pdf/2409.11261
• Copy Paste: [[2409.11261]] The Art of Storytelling: Multi-Agent Generative AI for Dynamic Multimodal Narratives(https://arxiv.org/abs/2409.11261)
• Keywords: agent
• Abstract: This paper introduces the concept of an education tool that utilizes Generative Artificial Intelligence (GenAI) to enhance storytelling for children. The system combines GenAI-driven narrative co-creation, text-to-speech conversion, and text-to-video generation to produce an engaging experience for learners. We describe the co-creation process, the adaptation of narratives into spoken words using text-to-speech models, and the transformation of these narratives into contextually relevant visuals through text-to-video technology. Our evaluation covers the linguistics of the generated stories, the text-to-speech conversion quality, and the accuracy of the generated visuals.
• 摘要：本文介绍了一种利用生成式人工智能 (GenAI) 增强儿童讲故事能力的教育工具的概念。该系统结合了 GenAI 驱动的叙事共同创作、文本转语音转换和文本转视频生成，为学习者带来引人入胜的体验。我们描述了共同创作过程、使用文本转语音模型将叙事改编为口语，以及通过文本转视频技术将这些叙事转换为上下文相关的视觉效果。我们的评估涵盖了生成故事的语言学、文本转语音转换质量以及生成的视觉效果的准确性。

Title: LOLA -- An Open-Source Massively Multilingual Large Language Model

• Authors: Nikit Srivastava, Denis Kuchelev, Tatiana Moteu, Kshitij Shetty, Michael Roeder, Diego Moussallem, Hamada Zahera, Axel-Cyrille Ngonga Ngomo
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11272
• Pdf URL: https://arxiv.org/pdf/2409.11272
• Copy Paste: [[2409.11272]] LOLA -- An Open-Source Massively Multilingual Large Language Model(https://arxiv.org/abs/2409.11272)
• Keywords: language model
• Abstract: This paper presents LOLA, a massively multilingual large language model trained on more than 160 languages using a sparse Mixture-of-Experts Transformer architecture. Our architectural and implementation choices address the challenge of harnessing linguistic diversity while maintaining efficiency and avoiding the common pitfalls of multilinguality. Our analysis of the evaluation results shows competitive performance in natural language generation and understanding tasks. Additionally, we demonstrate how the learned expert-routing mechanism exploits implicit phylogenetic linguistic patterns to potentially alleviate the curse of multilinguality. We provide an in-depth look at the training process, an analysis of the datasets, and a balanced exploration of the model's strengths and limitations. As an open-source model, LOLA promotes reproducibility and serves as a robust foundation for future research. Our findings enable the development of compute-efficient multilingual models with strong, scalable performance across languages.
• 摘要：本文介绍了 LOLA，这是一种大规模多语言大型语言模型，使用稀疏混合专家 Transformer 架构对 160 多种语言进行训练。我们的架构和实施选择解决了利用语言多样性的挑战，同时保持了效率并避免了多语言的常见陷阱。我们对评估结果的分析表明，它在自然语言生成和理解任务中具有竞争力。此外，我们还展示了学习专家路由机制如何利用隐式系统发育语言模式来潜在地缓解多语言的诅咒。我们深入研究了训练过程、数据集分析以及对模型的优势和局限性的平衡探索。作为一个开源模型，LOLA 促进了可重复性并为未来的研究奠定了坚实的基础。我们的研究结果使开发具有跨语言强大、可扩展性能的计算效率高的多语言模型成为可能。

Title: Task Arithmetic for Language Expansion in Speech Translation

• Authors: Yao-Fei Cheng, Hayato Futami, Yosuke Kashiwagi, Emiru Tsunoo, Wen Shen Teo, Siddhant Arora, Shinji Watanabe
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11274
• Pdf URL: https://arxiv.org/pdf/2409.11274
• Copy Paste: [[2409.11274]] Task Arithmetic for Language Expansion in Speech Translation(https://arxiv.org/abs/2409.11274)
• Keywords: language model, llm
• Abstract: Recent advances in large language models (LLMs) have gained interest in speech-text multimodal foundation models, achieving strong performance on instruction-based speech translation (ST). However, expanding language pairs from an existing instruction-tuned ST system is costly due to the necessity of re-training on a combination of new and previous datasets. We propose to expand new language pairs by merging the model trained on new language pairs and the existing model, using task arithmetic. We find that the direct application of task arithmetic for ST causes the merged model to fail to follow instructions; thus, generating translation in incorrect languages. To eliminate language confusion, we propose an augmented task arithmetic method that merges an additional language control model. It is trained to generate the correct target language token following the instructions. Our experiments demonstrate that our proposed language control model can achieve language expansion by eliminating language confusion. In our MuST-C and CoVoST-2 experiments, it shows up to 4.66 and 4.92 BLEU scores improvement, respectively. In addition, we demonstrate the use of our task arithmetic framework can expand to a language pair where neither paired ST training data nor a pre-trained ST model is available. We first synthesize the ST system from machine translation (MT) systems via task analogy, then merge the synthesized ST system to the existing ST model.
• 摘要：大型语言模型 (LLM) 的最新进展引起了人们对语音文本多模态基础模型的兴趣，在基于指令的语音翻译 (ST) 上取得了出色的表现。但是，由于需要在新的和以前的数据集上重新训练，因此从现有的指令调整的 ST 系统中扩展语言对的成本很高。我们建议使用任务算法，将对新语言对进行训练的模型与现有模型合并，以扩展新的语言对。我们发现，直接将任务算法应用于 ST 会导致合并后的模型无法遵循指令；从而生成不正确语言的翻译。为了消除语言混淆，我们提出了一种增强任务算法方法，该方法合并了额外的语言控制模型。它经过训练可以按照指令生成正确的目标语言标记。我们的实验表明，我们提出的语言控制模型可以通过消除语言混淆来实现语言扩展。在我们的 MuST-C 和 CoVoST-2 实验中，它分别显示出高达 4.66 和 4.92 的 BLEU 分数提升。此外，我们证明了我们的任务算法框架的使用可以扩展到既没有成对的 ST 训练数据也没有预训练的 ST 模型的语言对。我们首先通过任务类比从机器翻译 (MT) 系统中合成 ST 系统，然后将合成的 ST 系统合并到现有的 ST 模型中。

Title: Leveraging Distillation Techniques for Document Understanding: A Case Study with FLAN-T5

• Authors: Marcel Lamott, Muhammad Armaghan Shakir
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11282
• Pdf URL: https://arxiv.org/pdf/2409.11282
• Copy Paste: [[2409.11282]] Leveraging Distillation Techniques for Document Understanding: A Case Study with FLAN-T5(https://arxiv.org/abs/2409.11282)
• Keywords: language model, gpt, llm, chat
• Abstract: The surge of digital documents in various formats, including less standardized documents such as business reports and environmental assessments, underscores the growing importance of Document Understanding. While Large Language Models (LLMs) have showcased prowess across diverse natural language processing tasks, their direct application to Document Understanding remains a challenge. Previous research has demonstrated the utility of LLMs in this domain, yet their significant computational demands make them challenging to deploy effectively. Additionally, proprietary Blackbox LLMs often outperform their open-source counterparts, posing a barrier to widespread accessibility. In this paper, we delve into the realm of document understanding, leveraging distillation methods to harness the power of large LLMs while accommodating computational limitations. Specifically, we present a novel approach wherein we distill document understanding knowledge from the proprietary LLM ChatGPT into FLAN-T5. Our methodology integrates labeling and curriculum-learning mechanisms to facilitate efficient knowledge transfer. This work contributes to the advancement of document understanding methodologies by offering a scalable solution that bridges the gap between resource-intensive LLMs and practical applications. Our findings underscore the potential of distillation techniques in facilitating the deployment of sophisticated language models in real-world scenarios, thereby fostering advancements in natural language processing and document comprehension domains.
• 摘要：各种格式的数字文档激增，包括商业报告和环境评估等不太标准化的文档，凸显了文档理解日益增长的重要性。虽然大型语言模型 (LLM) 已经在各种自然语言处理任务中展示了其威力，但它们直接应用于文档理解仍然是一个挑战。先前的研究已经证明了 LLM 在该领域的实用性，但它们对计算的巨大需求使得它们难以有效部署。此外，专有的 Blackbox LLM 往往比其开源同类产品表现更好，这对其广泛使用构成了障碍。在本文中，我们深入研究了文档理解领域，利用蒸馏方法来利用大型 LLM 的功能，同时适应计算限制。具体来说，我们提出了一种新颖的方法，其中我们将专有 LLM ChatGPT 中的文档理解知识提炼到 FLAN-T5 中。我们的方法集成了标签和课程学习机制，以促进有效的知识转移。这项工作通过提供可扩展的解决方案来弥合资源密集型 LLM 与实际应用之间的差距，从而促进了文档理解方法的进步。我们的研究结果强调了提炼技术在促进复杂语言模型在现实场景中的部署方面的潜力，从而促进自然语言处理和文档理解领域的进步。

Title: Zero-resource Hallucination Detection for Text Generation via Graph-based Contextual Knowledge Triples Modeling

• Authors: Xinyue Fang, Zhen Huang, Zhiliang Tian, Minghui Fang, Ziyi Pan, Quntian Fang, Zhihua Wen, Hengyue Pan, Dongsheng Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11283
• Pdf URL: https://arxiv.org/pdf/2409.11283
• Copy Paste: [[2409.11283]] Zero-resource Hallucination Detection for Text Generation via Graph-based Contextual Knowledge Triples Modeling(https://arxiv.org/abs/2409.11283)
• Keywords: llm, hallucination
• Abstract: LLMs obtain remarkable performance but suffer from hallucinations. Most research on detecting hallucination focuses on the questions with short and concrete correct answers that are easy to check the faithfulness. Hallucination detections for text generation with open-ended answers are more challenging. Some researchers use external knowledge to detect hallucinations in generated texts, but external resources for specific scenarios are hard to access. Recent studies on detecting hallucinations in long text without external resources conduct consistency comparison among multiple sampled outputs. To handle long texts, researchers split long texts into multiple facts and individually compare the consistency of each pairs of facts. However, these methods (1) hardly achieve alignment among multiple facts; (2) overlook dependencies between multiple contextual facts. In this paper, we propose a graph-based context-aware (GCA) hallucination detection for text generations, which aligns knowledge facts and considers the dependencies between contextual knowledge triples in consistency comparison. Particularly, to align multiple facts, we conduct a triple-oriented response segmentation to extract multiple knowledge triples. To model dependencies among contextual knowledge triple (facts), we construct contextual triple into a graph and enhance triples' interactions via message passing and aggregating via RGCN. To avoid the omission of knowledge triples in long text, we conduct a LLM-based reverse verification via reconstructing the knowledge triples. Experiments show that our model enhances hallucination detection and excels all baselines.
• 摘要：LLM 取得了卓越的性能，但存在幻觉。大多数关于检测幻觉的研究都集中在具有简短而具体的正确答案的问题上，这些答案易于检查真实性。对于具有开放式答案的文本生成的幻觉检测更具挑战性。一些研究人员使用外部知识来检测生成文本中的幻觉，但特定场景的外部资源难以获取。最近在没有外部资源的情况下检测长文本中幻觉的研究在多个采样输出之间进行一致性比较。为了处理长文本，研究人员将长文本分成多个事实，并单独比较每对事实的一致性。然而，这些方法 (1) 很难实现多个事实之间的对齐；(2) 忽略了多个上下文事实之间的依赖关系。在本文中，我们提出了一种基于图的上下文感知 (GCA) 文本生成幻觉检测，它对齐知识事实并在一致性比较中考虑上下文知识三元组之间的依赖关系。具体而言，为了对齐多个事实，我们进行面向三元组的响应分割以提取多个知识三元组。为了对上下文知识三元组（事实）之间的依赖关系进行建模，我们将上下文三元组构建成图，并通过消息传递和通过 RGCN 聚合来增强三元组的交互。为了避免长文本中知识三元组的遗漏，我们通过重建知识三元组进行基于 LLM 的反向验证。实验表明，我们的模型增强了幻觉检测并优于所有基线。

Title: THaMES: An End-to-End Tool for Hallucination Mitigation and Evaluation in Large Language Models

• Authors: Mengfei Liang, Archish Arun, Zekun Wu, Cristian Munoz, Jonathan Lutch, Emre Kazim, Adriano Koshiyama, Philip Treleaven
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11353
• Pdf URL: https://arxiv.org/pdf/2409.11353
• Copy Paste: [[2409.11353]] THaMES: An End-to-End Tool for Hallucination Mitigation and Evaluation in Large Language Models(https://arxiv.org/abs/2409.11353)
• Keywords: language model, gpt, llm, hallucination, retrieval augmented generation
• Abstract: Hallucination, the generation of factually incorrect content, is a growing challenge in Large Language Models (LLMs). Existing detection and mitigation methods are often isolated and insufficient for domain-specific needs, lacking a standardized pipeline. This paper introduces THaMES (Tool for Hallucination Mitigations and EvaluationS), an integrated framework and library addressing this gap. THaMES offers an end-to-end solution for evaluating and mitigating hallucinations in LLMs, featuring automated test set generation, multifaceted benchmarking, and adaptable mitigation strategies. It automates test set creation from any corpus, ensuring high data quality, diversity, and cost-efficiency through techniques like batch processing, weighted sampling, and counterfactual validation. THaMES assesses a model's ability to detect and reduce hallucinations across various tasks, including text generation and binary classification, applying optimal mitigation strategies like In-Context Learning (ICL), Retrieval Augmented Generation (RAG), and Parameter-Efficient Fine-tuning (PEFT). Evaluations of state-of-the-art LLMs using a knowledge base of academic papers, political news, and Wikipedia reveal that commercial models like GPT-4o benefit more from RAG than ICL, while open-weight models like Llama-3.1-8B-Instruct and Mistral-Nemo gain more from ICL. Additionally, PEFT significantly enhances the performance of Llama-3.1-8B-Instruct in both evaluation tasks.
• 摘要：幻觉，即产生事实上不正确的内容，是大型语言模型 (LLM) 面临的一个日益严峻的挑战。现有的检测和缓解方法往往是孤立的，不足以满足特定领域的需求，缺乏标准化的流程。本文介绍了 THaMES（幻觉缓解和评估工具），这是一个解决这一差距的集成框架和库。THaMES 为评估和缓解 LLM 中的幻觉提供了端到端解决方案，具有自动测试集生成、多方面基准测试和适应性缓解策略。它可以自动从任何语料库创建测试集，通过批处理、加权抽样和反事实验证等技术确保高数据质量、多样性和成本效益。 THaMES 评估模型在各种任务（包括文本生成和二元分类）中检测和减少幻觉的能力，应用最佳缓解策略，如情境学习 (ICL)、检索增强生成 (RAG) 和参数高效微调 (PEFT)。使用学术论文、政治新闻和维基百科知识库对最先进的 LLM 进行评估表明，GPT-4o 等商业模型从 RAG 中受益多于 ICL，而 Llama-3.1-8B-Instruct 和 Mistral-Nemo 等开放权重模型从 ICL 中受益更多。此外，PEFT 在两个评估任务中都显著提高了 Llama-3.1-8B-Instruct 的性能。

Title: CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark

• Authors: Zachary S. Siegel, Sayash Kapoor, Nitya Nagdir, Benedikt Stroebl, Arvind Narayanan
• Subjects: cs.CL, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2409.11363
• Pdf URL: https://arxiv.org/pdf/2409.11363
• Copy Paste: [[2409.11363]] CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark(https://arxiv.org/abs/2409.11363)
• Keywords: language model, gpt, agent
• Abstract: AI agents have the potential to aid users on a variety of consequential tasks, including conducting scientific research. To spur the development of useful agents, we need benchmarks that are challenging, but more crucially, directly correspond to real-world tasks of interest. This paper introduces such a benchmark, designed to measure the accuracy of AI agents in tackling a crucial yet surprisingly challenging aspect of scientific research: computational reproducibility. This task, fundamental to the scientific process, involves reproducing the results of a study using the provided code and data. We introduce CORE-Bench (Computational Reproducibility Agent Benchmark), a benchmark consisting of 270 tasks based on 90 scientific papers across three disciplines (computer science, social science, and medicine). Tasks in CORE-Bench consist of three difficulty levels and include both language-only and vision-language tasks. We provide an evaluation system to measure the accuracy of agents in a fast and parallelizable way, saving days of evaluation time for each run compared to a sequential implementation. We evaluated two baseline agents: the general-purpose AutoGPT and a task-specific agent called CORE-Agent. We tested both variants using two underlying language models: GPT-4o and GPT-4o-mini. The best agent achieved an accuracy of 21% on the hardest task, showing the vast scope for improvement in automating routine scientific tasks. Having agents that can reproduce existing work is a necessary step towards building agents that can conduct novel research and could verify and improve the performance of other research agents. We hope that CORE-Bench can improve the state of reproducibility and spur the development of future research agents.
• 摘要：AI 代理有潜力帮助用户完成各种重要的任务，包括进行科学研究。为了促进有用代理的开发，我们需要具有挑战性的基准，但更重要的是，这些基准要与现实世界中感兴趣的任务直接对应。本文介绍了这样一个基准，旨在衡量 AI 代理在解决科学研究中一个至关重要但又出乎意料的挑战性方面时的准确性：计算可重复性。这项任务是科学过程的基础，涉及使用提供的代码和数据重现研究结果。我们推出了 CORE-Bench（计算可重复性代理基准），这是一个基准，由 270 项任务组成，这些任务基于三个学科（计算机科学、社会科学和医学）的 90 篇科学论文。CORE-Bench 中的任务有三个难度级别，包括纯语言任务和视觉语言任务。我们提供了一个评估系统，以快速且可并行的方式测量代理的准确性，与顺序实施相比，每次运行可节省数天的评估时间。我们评估了两个基线代理：通用的 AutoGPT 和名为 CORE-Agent 的任务专用代理。我们使用两个底层语言模型测试了这两个变体：GPT-4o 和 GPT-4o-mini。最好的代理在最困难的任务上实现了 21% 的准确率，这表明在自动化常规科学任务方面有很大的改进空间。拥有可以重现现有工作的代理是构建可以进行新颖研究并可以验证和改进其他研究代理性能的代理的必要步骤。我们希望 CORE-Bench 可以改善可重复性状态并促进未来研究代理的发展。

Title: CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibration

• Authors: Jiahui Gao, Renjie Pi, Tianyang Han, Han Wu, Lanqing Hong, Lingpeng Kong, Xin Jiang, Zhenguo Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.11365
• Pdf URL: https://arxiv.org/pdf/2409.11365
• Copy Paste: [[2409.11365]] CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibration(https://arxiv.org/abs/2409.11365)
• Keywords: language model, llm
• Abstract: The deployment of multimodal large language models (MLLMs) has demonstrated remarkable success in engaging in conversations involving visual inputs, thanks to the superior power of large language models (LLMs). Those MLLMs are typically built based on the LLMs, with an image encoder to process images into the token embedding space of the LLMs. However, the integration of visual modality has introduced a unique vulnerability: the MLLM becomes susceptible to malicious visual inputs and prone to generating sensitive or harmful responses, even though the LLM has been trained on textual dataset to align with human value. In this paper, we first raise the question: ``Do the MLLMs possess safety-awareness against malicious image inputs?". We find that after adding a principle that specifies the safety requirement into the input of the MLLM, the model's safety awareness becomes boosted. This phenomenon verifies the existence of MLLM's safety-awareness against image inputs, it is only weakened by the modality gap. We then introduce a simple yet effective technique termed CoCA, which amplifies the safety-awareness of the MLLM by calibrating its output distribution. Our proposed strategy helps the model reclaim its original safety awareness without losing its original capabilities. We verify the effectiveness of our approach on both multimodal safety and understanding benchmarks.
• 摘要：由于大型语言模型 (LLM) 的强大功能，多模态大型语言模型 (MLLM) 的部署在涉及视觉输入的对话中取得了显著的成功。这些 MLLM 通常基于 LLM 构建，并使用图像编码器将图像处理到 LLM 的标记嵌入空间中。然而，视觉模态的集成带来了一个独特的漏洞：MLLM 容易受到恶意视觉输入的影响，并且容易产生敏感或有害的反应，即使 LLM 已在文本数据集上进行训练以符合人类价值观。在本文中，我们首先提出一个问题：“MLLM 是否具备针对恶意图像输入的安全意识？”。我们发现，在 MLLM 的输入中添加指定安全要求的原则后，模型的安全意识得到增强。这一现象验证了 MLLM 对图像输入的安全意识的存在，它只会因模态差距而减弱。然后，我们引入了一种简单而有效的技术，称为 CoCA，它通过校准 MLLM 的输出分布来放大其安全意识。我们提出的策略有助于模型在不失去其原有能力的情况下恢复其原有的安全意识。我们在多模态安全和理解基准上验证了我们方法的有效性。

Title: Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinement

• Authors: Simon Yu, Liangyu Chen, Sara Ahmadian, Marzieh Fadaee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11378
• Pdf URL: https://arxiv.org/pdf/2409.11378
• Copy Paste: [[2409.11378]] Diversify and Conquer: Diversity-Centric Data Selection with Iterative Refinement(https://arxiv.org/abs/2409.11378)
• Keywords: language model, llm
• Abstract: Finetuning large language models on instruction data is crucial for enhancing pre-trained knowledge and improving instruction-following capabilities. As instruction datasets proliferate, selecting optimal data for effective training becomes increasingly important. This work addresses the question: How can we determine the optimal subset of data for effective training? While existing research often emphasizes local criteria like instance quality for subset selection, we argue that a global approach focused on data diversity is more critical. Our method employs k-means clustering to ensure the selected subset effectively represents the full dataset. We propose an iterative refinement method inspired by active learning techniques to resample instances from clusters, reassessing each cluster's importance and sampling weight in every training iteration. This approach reduces the effect of outliers and automatically filters out clusters containing low-quality data. Through extensive evaluation across natural language reasoning, general world knowledge, code and math reasoning tasks, and by fine-tuning models from various families, we observe consistent improvements, achieving a 7% increase over random selection and a 3.8% improvement over state-of-the-art sampling methods. Our work highlights the significance of diversity-first sampling when finetuning LLMs to enhance performance across a broad array of evaluation tasks. Our code is available at this https URL.
• 摘要：在指令数据上微调大型语言模型对于增强预训练知识和提高指令遵循能力至关重要。随着指令数据集的激增，选择最佳数据进行有效训练变得越来越重要。这项工作解决了以下问题：我们如何确定有效训练的最佳数据子集？虽然现有研究通常强调子集选择的局部标准（如实例质量），但我们认为，专注于数据多样性的全局方法更为关键。我们的方法采用 k 均值聚类来确保所选子集有效地代表整个数据集。我们提出了一种受主动学习技术启发的迭代细化方法，从集群中重新采样实例，在每次训练迭代中重新评估每个集群的重要性和采样权重。这种方法减少了异常值的影响，并自动过滤掉包含低质量数据的集群。通过对自然语言推理、一般世界知识、代码和数学推理任务进行广泛的评估，以及通过微调来自不同系列的模型，我们观察到持续的改进，比随机选择提高了 7%，比最先进的采样方法提高了 3.8%。我们的工作强调了多样性优先采样在微调 LLM 以提高广泛评估任务的性能时的重要性。我们的代码可在此 https URL 上找到。

Title: Says Who? Effective Zero-Shot Annotation of Focalization

• Authors: Rebecca M. M. Hicke, Yuri Bizzoni, Pascale Feldkamp, Ross Deans Kristensen-McLachlan
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.11390
• Pdf URL: https://arxiv.org/pdf/2409.11390
• Copy Paste: [[2409.11390]] Says Who? Effective Zero-Shot Annotation of Focalization(https://arxiv.org/abs/2409.11390)
• Keywords: language model, llm
• Abstract: Focalization, the perspective through which narrative is presented, is encoded via a wide range of lexico-grammatical features and is subject to reader interpretation. Moreover, trained readers regularly disagree on interpretations, suggesting that this problem may be computationally intractable. In this paper, we provide experiments to test how well contemporary Large Language Models (LLMs) perform when annotating literary texts for focalization mode. Despite the challenging nature of the task, LLMs show comparable performance to trained human annotators in our experiments. We provide a case study working with the novels of Stephen King to demonstrate the usefulness of this approach for computational literary studies, illustrating how focalization can be studied at scale.
• 摘要：聚焦是叙事呈现的视角，通过广泛的词汇语法特征进行编码，并受读者解读的影响。此外，受过训练的读者经常对解读产生分歧，这表明这个问题在计算上可能是难以解决的。在本文中，我们提供了实验来测试当代大型语言模型 (LLM) 在为聚焦模式注释文学文本时的表现如何。尽管这项任务具有挑战性，但在我们的实验中，LLM 表现出与受过训练的人类注释者相当的表现。我们提供了一个与斯蒂芬·金的小说合作的案例研究，以证明这种方法对计算文学研究的实用性，说明如何大规模研究聚焦。

Title: NVLM: Open Frontier-Class Multimodal LLMs

• Authors: Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuoling Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, Wei Ping
• Subjects: cs.CL, cs.AI, cs.CV, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.11402
• Pdf URL: https://arxiv.org/pdf/2409.11402
• Copy Paste: [[2409.11402]] NVLM: Open Frontier-Class Multimodal LLMs(https://arxiv.org/abs/2409.11402)
• Keywords: language model, gpt, llm
• Abstract: We introduce NVLM 1.0, a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks, rivaling the leading proprietary models (e.g., GPT-4o) and open-access models (e.g., Llama 3-V 405B and InternVL 2). Remarkably, NVLM 1.0 shows improved text-only performance over its LLM backbone after multimodal training. In terms of model design, we perform a comprehensive comparison between decoder-only multimodal LLMs (e.g., LLaVA) and cross-attention-based models (e.g., Flamingo). Based on the strengths and weaknesses of both approaches, we propose a novel architecture that enhances both training efficiency and multimodal reasoning capabilities. Furthermore, we introduce a 1-D tile-tagging design for tile-based dynamic high-resolution images, which significantly boosts performance on multimodal reasoning and OCR-related tasks. Regarding training data, we meticulously curate and provide detailed information on our multimodal pretraining and supervised fine-tuning datasets. Our findings indicate that dataset quality and task diversity are more important than scale, even during the pretraining phase, across all architectures. Notably, we develop production-grade multimodality for the NVLM-1.0 models, enabling them to excel in vision-language tasks while maintaining and even improving text-only performance compared to their LLM backbones. To achieve this, we craft and integrate a high-quality text-only dataset into multimodal training, alongside a substantial amount of multimodal math and reasoning data, leading to enhanced math and coding capabilities across modalities. To advance research in the field, we are releasing the model weights and will open-source the code for the community: this https URL.
• 摘要：我们推出了 NVLM 1.0，这是一系列前沿级多模态大型语言模型 (LLM)，在视觉语言任务上取得了最先进的成果，可与领先的专有模型（例如 GPT-4o）和开放访问模型（例如 Llama 3-V 405B 和 InternVL 2）相媲美。值得注意的是，经过多模态训练后，NVLM 1.0 在其 LLM 主干上表现出更好的纯文本性能。在模型设计方面，我们对仅解码器的多模态 LLM（例如 LLaVA）和基于交叉注意的模型（例如 Flamingo）进行了全面比较。基于这两种方法的优缺点，我们提出了一种新颖的架构，可以同时提高训练效率和多模态推理能力。此外，我们为基于图块的动态高分辨率图像引入了一种 1-D 图块标记设计，这显著提高了多模态推理和 OCR 相关任务的性能。关于训练数据，我们精心策划并提供有关多模态预训练和监督微调数据集的详细信息。我们的研究结果表明，在所有架构中，即使在预训练阶段，数据集质量和任务多样性也比规模更重要。值得注意的是，我们为 NVLM-1.0 模型开发了生产级多模态，使它们能够在视觉语言任务中表现出色，同时与 LLM 主干相比保持甚至提高纯文本性能。为了实现这一点，我们制作并集成了一个高质量的纯文本数据集到多模态训练中，同时还集成了大量多模态数学和推理数据，从而增强了跨模态的数学和编码能力。为了推动该领域的研究，我们将发布模型权重并将代码开放给社区：此 https URL。

Title: AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs

• Authors: Basel Mousi, Nadir Durrani, Fatema Ahmad, Md. Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, Firoj Alam
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.11404
• Pdf URL: https://arxiv.org/pdf/2409.11404
• Copy Paste: [[2409.11404]] AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs(https://arxiv.org/abs/2409.11404)
• Keywords: language model, gpt, llm
• Abstract: Arabic, with its rich diversity of dialects, remains significantly underrepresented in Large Language Models, particularly in dialectal variations. We address this gap by introducing seven synthetic datasets in dialects alongside Modern Standard Arabic (MSA), created using Machine Translation (MT) combined with human post-editing. We present AraDiCE, a benchmark for Arabic Dialect and Cultural Evaluation. We evaluate LLMs on dialect comprehension and generation, focusing specifically on low-resource Arabic dialects. Additionally, we introduce the first-ever fine-grained benchmark designed to evaluate cultural awareness across the Gulf, Egypt, and Levant regions, providing a novel dimension to LLM evaluation. Our findings demonstrate that while Arabic-specific models like Jais and AceGPT outperform multilingual models on dialectal tasks, significant challenges persist in dialect identification, generation, and translation. This work contributes ~45K post-edited samples, a cultural benchmark, and highlights the importance of tailored training to improve LLM performance in capturing the nuances of diverse Arabic dialects and cultural contexts. We will release the dialectal translation models and benchmarks curated in this study.
• 摘要：阿拉伯语方言丰富多样，但在大型语言模型中仍然严重缺乏代表性，尤其是在方言变体方面。我们通过引入七个合成的方言数据集以及使用机器翻译 (MT) 结合人工后期编辑创建的现代标准阿拉伯语 (MSA) 来解决这一差距。我们提出了 AraDiCE，这是阿拉伯方言和文化评估的基准。我们评估 LLM 的方言理解和生成能力，特别关注资源匮乏的阿拉伯方言。此外，我们引入了有史以来第一个细粒度基准，旨在评估海湾、埃及和黎凡特地区的文化意识，为 LLM 评估提供了一个新的维度。我们的研究结果表明，虽然 Jais 和 AceGPT 等阿拉伯语特定模型在方言任务上的表现优于多语言模型，但在方言识别、生成和翻译方面仍然存在重大挑战。这项工作贡献了约 45K 个后期编辑样本，这是一个文化基准，并强调了量身定制的培训对于提高 LLM 在捕捉不同阿拉伯方言和文化背景细微差别方面的表现的重要性。我们将发布本次研究中整理的方言翻译模型和基准。