2024-08-27

Title: Retrieval-Augmented Generation Meets Data-Driven Tabula Rasa Approach for Temporal Knowledge Graph Forecasting

• Authors: Geethan Sannidhi, Sagar Srinivas Sakhinana, Venkataramana Runkana
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.13273
• Pdf URL: https://arxiv.org/pdf/2408.13273
• Copy Paste: [[2408.13273]] Retrieval-Augmented Generation Meets Data-Driven Tabula Rasa Approach for Temporal Knowledge Graph Forecasting(https://arxiv.org/abs/2408.13273)
• Keywords: language model, gpt, llm, hallucination, prompt, chat, retrieval-augmented generation
• Abstract: Pre-trained large language models (PLLMs) like OpenAI ChatGPT and Google Gemini face challenges such as inaccurate factual recall, hallucinations, biases, and future data leakage for temporal Knowledge Graph (tKG) forecasting. To address these issues, we introduce sLA-tKGF (small-scale language assistant for tKG forecasting), which utilizes Retrieval-Augmented Generation (RAG) aided, custom-trained small-scale language models through a tabula rasa approach from scratch for effective tKG forecasting. Our framework constructs knowledge-infused prompts with relevant historical data from tKGs, web search results, and PLLMs-generated textual descriptions to understand historical entity relationships prior to the target time. It leverages these external knowledge-infused prompts for deeper understanding and reasoning of context-specific semantic and temporal information to zero-shot prompt small-scale language models for more accurate predictions of future events within tKGs. It reduces hallucinations and mitigates distributional shift challenges through comprehending changing trends over time. As a result, it enables more accurate and contextually grounded forecasts of future events while minimizing computational demands. Rigorous empirical studies demonstrate our framework robustness, scalability, and state-of-the-art (SOTA) performance on benchmark datasets with interpretable and trustworthy tKG forecasting.
• 摘要：OpenAI ChatGPT 和 Google Gemini 等预训练大型语言模型 (PLLM) 面临着诸如事实回忆不准确、幻觉、偏见和时间知识图谱 (tKG) 预测的未来数据泄露等挑战。为了解决这些问题，我们引入了 sLA-tKGF（用于 tKG 预测的小型语言助手），它利用检索增强生成 (RAG) 辅助、定制训练的小型语言模型，通过从头开始的白板方法进行有效的 tKG 预测。我们的框架使用来自 tKG、网络搜索结果和 PLLM 生成的文本描述的相关历史数据构建知识注入提示，以了解目标时间之前的历史实体关系。它利用这些外部知识注入提示来更深入地理解和推理上下文特定的语义和时间信息，以零样本提示小型语言模型，以更准确地预测 tKG 中的未来事件。它通过理解随时间变化的趋势来减少幻觉并缓解分布转移挑战。因此，它可以更准确、更符合情境地预测未来事件，同时最大限度地减少计算需求。严格的实证研究表明，我们的框架在基准数据集上具有可解释和值得信赖的 tKG 预测的稳健性、可扩展性和最先进 (SOTA) 性能。

Title: Question answering system of bridge design specification based on large language model

• Authors: Leye Zhang, Xiangxiang Tian, Hongjun Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.13282
• Pdf URL: https://arxiv.org/pdf/2408.13282
• Copy Paste: [[2408.13282]] Question answering system of bridge design specification based on large language model(https://arxiv.org/abs/2408.13282)
• Keywords: language model
• Abstract: This paper constructs question answering system for bridge design specification based on large language model. Three implementation schemes are tried: full fine-tuning of the Bert pretrained model, parameter-efficient fine-tuning of the Bert pretrained model, and self-built language model from scratch. Through the self-built question and answer task dataset, based on the tensorflow and keras deep learning platform framework, the model is constructed and trained to predict the start position and end position of the answer in the bridge design specification given by the user. The experimental results show that full fine-tuning of the Bert pretrained model achieves 100% accuracy in the training-dataset, validation-dataset and test-dataset, and the system can extract the answers from the bridge design specification given by the user to answer various questions of the user; While parameter-efficient fine-tuning of the Bert pretrained model and self-built language model from scratch perform well in the training-dataset, their generalization ability in the test-dataset needs to be improved. The research of this paper provides a useful reference for the development of question answering system in professional field.
• 摘要：本文构建了基于大型语言模型的桥梁设计规范问答系统，尝试了三种实现方案：完全微调Bert预训练模型、参数高效微调Bert预训练模型、从头开始自建语言模型。通过自建问答任务数据集，基于tensorflow和keras深度学习平台框架构建并训练模型，预测用户给出的桥梁设计规范中答案的起始位置和终止位置。实验结果表明，完全微调Bert预训练模型在训练集、验证集和测试集均达到100%准确率，系统可以从用户给出的桥梁设计规范中提取答案，回答用户的各种问题；参数高效微调Bert预训练模型和从头开始自建语言模型在训练集上表现良好，但在测试集上的泛化能力有待提高。本论文的研究为专业领域问答系统的开发提供了有益的参考。

Title: Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler

• Authors: Yikang Shen, Matthew Stallone, Mayank Mishra, Gaoyuan Zhang, Shawn Tan, Aditya Prasad, Adriana Meza Soria, David D. Cox, Rameswar Panda
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.13359
• Pdf URL: https://arxiv.org/pdf/2408.13359
• Copy Paste: [[2408.13359]] Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler(https://arxiv.org/abs/2408.13359)
• Keywords: language model
• Abstract: Finding the optimal learning rate for language model pretraining is a challenging task. This is not only because there is a complicated correlation between learning rate, batch size, number of training tokens, model size, and other hyperparameters but also because it is prohibitively expensive to perform a hyperparameter search for large language models with Billions or Trillions of parameters. Recent studies propose using small proxy models and small corpus to perform hyperparameter searches and transposing the optimal parameters to large models and large corpus. While the zero-shot transferability is theoretically and empirically proven for model size related hyperparameters, like depth and width, the zero-shot transfer from small corpus to large corpus is underexplored. In this paper, we study the correlation between optimal learning rate, batch size, and number of training tokens for the recently proposed WSD scheduler. After thousands of small experiments, we found a power-law relationship between variables and demonstrated its transferability across model sizes. Based on the observation, we propose a new learning rate scheduler, Power scheduler, that is agnostic about the number of training tokens and batch size. The experiment shows that combining the Power scheduler with Maximum Update Parameterization (muP) can consistently achieve impressive performance with one set of hyperparameters regardless of the number of training tokens, batch size, model size, and even model architecture. Our 3B dense and MoE models trained with the Power scheduler achieve comparable performance as state-of-the-art small language models. We open-source these pretrained models at this https URL.
• 摘要：寻找语言模型预训练的最佳学习率是一项具有挑战性的任务。这不仅是因为学习率、批量大小、训练标记数量、模型大小和其他超参数之间存在复杂的相关性，还因为对具有数十亿或数万亿个参数的大型语言模型进行超参数搜索的成本过高。最近的研究建议使用小型代理模型和小型语料库进行超参数搜索，并将最佳参数转置到大型模型和大型语料库。虽然零样本可迁移性在理论和经验上已得到证明，适用于与模型大小相关的超参数，如深度和宽度，但从小型语料库到大型语料库的零样本迁移尚未得到充分探索。在本文中，我们研究了最近提出的 WSD 调度程序的最佳学习率、批量大小和训练标记数量之间的相关性。经过数千次小实验，我们发现变量之间存在幂律关系，并证明了其跨模型大小的可迁移性。根据观察，我们提出了一种新的学习率调度程序 Power 调度程序，它与训练标记的数量和批处理大小无关。实验表明，将 Power 调度程序与最大更新参数化 (muP) 相结合，无论训练标记的数量、批处理大小、模型大小甚至模型架构如何，都可以使用一组超参数始终如一地实现令人印象深刻的性能。使用 Power 调度程序训练的 3B 密集和 MoE 模型实现了与最先进的小型语言模型相当的性能。我们在此 https URL 上开源了这些预训练模型。

Title: CodeRefine: A Pipeline for Enhancing LLM-Generated Code Implementations of Research Papers

• Authors: Ekaterina Trofimova, Emil Sataev, Abhijit Singh Jowhari
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.13366
• Pdf URL: https://arxiv.org/pdf/2408.13366
• Copy Paste: [[2408.13366]] CodeRefine: A Pipeline for Enhancing LLM-Generated Code Implementations of Research Papers(https://arxiv.org/abs/2408.13366)
• Keywords: language model, llm, prompt, retrieval-augmented generation
• Abstract: This paper presents CodeRefine, a novel framework for automatically transforming research paper methodologies into functional code using Large Language Models (LLMs). Our multi-step approach first extracts and summarizes key text chunks from papers, analyzes their code relevance, and creates a knowledge graph using a predefined ontology. Code is then generated from this structured representation and enhanced through a proposed retrospective retrieval-augmented generation approach. CodeRefine addresses the challenge of bridging theoretical research and practical implementation, offering a more accurate alternative to LLM zero-shot prompting. Evaluations on diverse scientific papers demonstrate CodeRefine's ability to improve code implementation from the paper, potentially accelerating the adoption of cutting-edge algorithms in real-world applications.
• 摘要：本文介绍了 CodeRefine，这是一种使用大型语言模型 (LLM) 自动将研究论文方法转换为功能代码的新型框架。我们的多步骤方法首先从论文中提取和总结关键文本块，分析其代码相关性，并使用预定义的本体创建知识图谱。然后从这种结构化表示生成代码，并通过提出的回顾性检索增强生成方法进行增强。CodeRefine 解决了连接理论研究和实际实施的挑战，为 LLM 零样本提示提供了更准确的替代方案。对各种科学论文的评估​​表明，CodeRefine 能够改进论文中的代码实现，从而有可能加速尖端算法在实际应用中的采用。

Title: Integrating Multi-Head Convolutional Encoders with Cross-Attention for Improved SPARQL Query Translation

• Authors: Yi-Hui Chen, Eric Jui-Lin Lu, Kwan-Ho Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13432
• Pdf URL: https://arxiv.org/pdf/2408.13432
• Copy Paste: [[2408.13432]] Integrating Multi-Head Convolutional Encoders with Cross-Attention for Improved SPARQL Query Translation(https://arxiv.org/abs/2408.13432)
• Keywords: language model
• Abstract: The main task of the KGQA system (Knowledge Graph Question Answering) is to convert user input questions into query syntax (such as SPARQL). With the rise of modern popular encoders and decoders like Transformer and ConvS2S, many scholars have shifted the research direction of SPARQL generation to the Neural Machine Translation (NMT) architecture or the generative AI field of Text-to-SPARQL. In NMT-based QA systems, the system treats knowledge base query syntax as a language. It uses NMT-based translation models to translate natural language questions into query syntax. Scholars use popular architectures equipped with cross-attention, such as Transformer, ConvS2S, and BiLSTM, to train translation models for query syntax. To achieve better query results, this paper improved the ConvS2S encoder and added multi-head attention from the Transformer, proposing a Multi-Head Conv encoder (MHC encoder) based on the n-gram language model. The principle is to use convolutional layers to capture local hidden features in the input sequence with different receptive fields, using multi-head attention to calculate dependencies between them. Ultimately, we found that the translation model based on the Multi-Head Conv encoder achieved better performance than other encoders, obtaining 76.52\% and 83.37\% BLEU-1 (BiLingual Evaluation Understudy) on the QALD-9 and LC-QuAD-1.0 datasets, respectively. Additionally, in the end-to-end system experiments on the QALD-9 and LC-QuAD-1.0 datasets, we achieved leading results over other KGQA systems, with Macro F1-measures reaching 52\% and 66\%, respectively. Moreover, the experimental results show that with limited computational resources, if one possesses an excellent encoder-decoder architecture and cross-attention, experts and scholars can achieve outstanding performance equivalent to large pre-trained models using only general embeddings.
• 摘要：KGQA 系统（知识图谱问答系统）的主要任务是将用户输入的问题转换成查询语法（如 SPARQL）。随着 Transformer、ConvS2S 等现代流行编码器和解码器的兴起，很多学者将 SPARQL 生成的研究方向转向神经机器翻译（NMT）架构或 Text-to-SPARQL 的生成式 AI 领域。在基于 NMT 的问答系统中，系统将知识库查询语法视为一种语言，利用基于 NMT 的翻译模型将自然语言问题翻译成查询语法。学者们利用流行的配备交叉注意力的架构，如 Transformer、ConvS2S、BiLSTM，来训练针对查询语法的翻译模型。为了达到更好的查询效果，本文改进了 ConvS2S 编码器并加入了来自 Transformer 的多头注意力，提出了一种基于 n​​-gram 语言模型的多头 Conv 编码器（MHC 编码器）。其原理是利用卷积层捕获输入序列中具有不同感受野的局部隐藏特征，利用多头注意力机制计算它们之间的依赖关系。最终我们发现基于多头卷积编码器的翻译模型取得了优于其他编码器的性能，在 QALD-9 和 LC-QuAD-1.0 数据集上分别获得了 76.52% 和 83.37% 的 BLEU-1（BiLingual Evaluation Understudy）。此外，在 QALD-9 和 LC-QuAD-1.0 数据集上的端到端系统实验中，我们取得了领先于其他 KGQA 系统的结果，Macro F1-measures 分别达到了 52% 和 66%。此外，实验结果表明，在有限的计算资源下，如果拥有优秀的编解码器架构和交叉注意力机制，专家学者仅使用一般的嵌入就可以取得与大型预训练模型相当的出色性能。

Title: Make Every Penny Count: Difficulty-Adaptive Self-Consistency for Cost-Efficient Reasoning

• Authors: Xinglin Wang, Shaoxiong Feng, Yiwei Li, Peiwen Yuan, Yueqi Zhang, Boyuan Pan, Heda Wang, Yao Hu, Kan Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.13457
• Pdf URL: https://arxiv.org/pdf/2408.13457
• Copy Paste: [[2408.13457]] Make Every Penny Count: Difficulty-Adaptive Self-Consistency for Cost-Efficient Reasoning(https://arxiv.org/abs/2408.13457)
• Keywords: chain-of-thought
• Abstract: Self-consistency (SC), a widely used decoding strategy for chain-of-thought reasoning, shows significant gains across various multi-step reasoning tasks but comes with a high cost due to multiple sampling with the preset size. Its variants, Adaptive self-consistency (ASC) and Early-stopping self-consistency (ESC), dynamically adjust the number of samples based on the posterior distribution of a set of pre-samples, reducing the cost of SC with minimal impact on performance. Both methods, however, do not exploit the prior information about question difficulty. It often results in unnecessary repeated sampling for easy questions that could be accurately answered with just one attempt, wasting resources. To tackle this problem, we propose Difficulty-Adaptive Self-Consistency (DSC), which leverages the difficulty information from both prior and posterior perspectives to adaptively allocate inference resources, further reducing the cost of SC. To demonstrate the effectiveness of DSC, we conduct extensive experiments on three popular categories of reasoning tasks: arithmetic, commonsense and symbolic reasoning on six benchmarks. The empirical results show that DSC consistently surpasses the strong baseline ASC and ESC in terms of costs by a significant margin, while attaining comparable performances.
• 摘要：自洽 (SC) 是一种广泛用于思路链推理的解码策略，它在各种多步骤推理任务中显示出显著的收益，但由于按预设大小进行多次采样，因此成本较高。它的变体自适应自洽 (ASC) 和早停自洽 (ESC) 根据一组预样本的后验分布动态调整样本数量，在对性能影响最小的情况下降低了 SC 的成本。然而，这两种方法都没有利用有关问题难度的先验信息。这通常会导致对只需一次尝试即可准确回答的简单问题进行不必要的重复抽样，浪费资源。为了解决这个问题，我们提出了难度自适应自洽 (DSC)，它利用先验和后验角度的难度信息来自适应地分配推理资源，进一步降低 SC 的成本。为了证明 DSC 的有效性，我们在六个基准上对三种常见的推理任务类别（算术、常识和符号推理）进行了广泛的实验。实证结果表明，DSC 在成本方面始终远远超过强大的基准 ASC 和 ESC，同时实现了相当的性能。

Title: Utilizing Large Language Models for Named Entity Recognition in Traditional Chinese Medicine against COVID-19 Literature: Comparative Study

• Authors: Xu Tong, Nina Smirnova, Sharmila Upadhyaya, Ran Yu, Jack H. Culbert, Chao Sun, Wolfgang Otto, Philipp Mayr
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.13501
• Pdf URL: https://arxiv.org/pdf/2408.13501
• Copy Paste: [[2408.13501]] Utilizing Large Language Models for Named Entity Recognition in Traditional Chinese Medicine against COVID-19 Literature: Comparative Study(https://arxiv.org/abs/2408.13501)
• Keywords: language model, gpt, llm, chat
• Abstract: Objective: To explore and compare the performance of ChatGPT and other state-of-the-art LLMs on domain-specific NER tasks covering different entity types and domains in TCM against COVID-19 literature. Methods: We established a dataset of 389 articles on TCM against COVID-19, and manually annotated 48 of them with 6 types of entities belonging to 3 domains as the ground truth, against which the NER performance of LLMs can be assessed. We then performed NER tasks for the 6 entity types using ChatGPT (GPT-3.5 and GPT-4) and 4 state-of-the-art BERT-based question-answering (QA) models (RoBERTa, MiniLM, PubMedBERT and SciBERT) without prior training on the specific task. A domain fine-tuned model (GSAP-NER) was also applied for a comprehensive comparison. Results: The overall performance of LLMs varied significantly in exact match and fuzzy match. In the fuzzy match, ChatGPT surpassed BERT-based QA models in 5 out of 6 tasks, while in exact match, BERT-based QA models outperformed ChatGPT in 5 out of 6 tasks but with a smaller F-1 difference. GPT-4 showed a significant advantage over other models in fuzzy match, especially on the entity type of TCM formula and the Chinese patent drug (TFD) and ingredient (IG). Although GPT-4 outperformed BERT-based models on entity type of herb, target, and research method, none of the F-1 scores exceeded 0.5. GSAP-NER, outperformed GPT-4 in terms of F-1 by a slight margin on RM. ChatGPT achieved considerably higher recalls than precisions, particularly in the fuzzy match. Conclusions: The NER performance of LLMs is highly dependent on the entity type, and their performance varies across application scenarios. ChatGPT could be a good choice for scenarios where high recall is favored. However, for knowledge acquisition in rigorous scenarios, neither ChatGPT nor BERT-based QA models are off-the-shelf tools for professional practitioners.
• 摘要：目的：探索和比较 ChatGPT 与其他最先进的 LLM 在特定领域 NER 任务上的性能，这些任务涵盖中医药对抗 COVID-19 文献中不同实体类型和领域。方法：我们建立了一个包含 389 篇中医药对抗 COVID-19 文章的数据集，并以属于 3 个领域的 6 种实体类型作为基本事实，手动注释了其中 48 篇，以此来评估 LLM 的 NER 性能。然后，我们使用 ChatGPT（GPT-3.5 和 GPT-4）和 4 个最先进的基于 BERT 的问答 (QA) 模型（RoBERTa、MiniLM、PubMedBERT 和 SciBERT）对 6 种实体类型执行 NER 任务，而无需事先进行特定任务的训练。还采用了领域微调模型（GSAP-NER）进行全面比较。结果：LLM 的整体性能在精确匹配和模糊匹配方面存在显著差异。在模糊匹配中，ChatGPT 在 6 项任务中的 5 项中超越了基于 BERT 的 QA 模型，而在精确匹配中，基于 BERT 的 QA 模型在 6 项任务中的 5 项中优于 ChatGPT，但 F-1 差异较小。GPT-4 在模糊匹配中表现出比其他模型明显的优势，尤其是在中药方剂实体类型和中成药（TFD）和成分（IG）上。虽然 GPT-4 在草药、目标和研究方法实体类型上优于基于 BERT 的模型，但 F-1 得分均未超过 0.5。GSAP-NER 在 F-1 方面略胜于 GPT-4，RM 略胜一筹。ChatGPT 的召回率远高于精确率，尤其是在模糊匹配中。结论：LLM 的 NER 性能高度依赖于实体类型，并且它们的性能因应用场景而异。对于高召回率的场景，ChatGPT 可能是一个不错的选择。然而，对于严格场景下的知识获取，ChatGPT 和基于 BERT 的 QA 模型都不是专业从业者的现成工具。

Title: Selective Preference Optimization via Token-Level Reward Function Estimation

• Authors: Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Erxue Min, Sophia Ananiadou
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.13518
• Pdf URL: https://arxiv.org/pdf/2408.13518
• Copy Paste: [[2408.13518]] Selective Preference Optimization via Token-Level Reward Function Estimation(https://arxiv.org/abs/2408.13518)
• Keywords: language model
• Abstract: Recent advancements in large language model alignment leverage token-level supervisions to perform fine-grained preference optimization. However, existing token-level alignment methods either optimize on all available tokens, which can be noisy and inefficient, or perform selective training with complex and expensive key token selection strategies. In this work, we propose Selective Preference Optimization (SePO), a novel selective alignment strategy that centers on efficient key token selection. SePO proposes the first token selection method based on Direct Preference Optimization (DPO), which trains an oracle model to estimate a token-level reward function on the target data. This method applies to any existing alignment datasets with response-level annotations and enables cost-efficient token selection with small-scale oracle models and training data. The estimated reward function is then utilized to score all tokens within the target dataset, where only the key tokens are selected to supervise the target policy model with a reference model-free contrastive objective function. Extensive experiments on three public evaluation benchmarks show that SePO significantly outperforms competitive baseline methods by only optimizing 30% key tokens on the target dataset. SePO applications on weak-to-strong generalization show that weak oracle models effectively supervise strong policy models with up to 16.8x more parameters. SePO also effectively selects key tokens from out-of-distribution data to enhance strong policy models and alleviate the over-optimization problem.
• 摘要：大型语言模型对齐方面的最新进展利用 token 级监督来执行细粒度的偏好优化。然而，现有的 token 级对齐方法要么对所有可用 token 进行优化，这可能很嘈杂且效率低下，要么使用复杂且昂贵的关键 token 选择策略进行选择性训练。在这项工作中，我们提出了选择性偏好优化 (SePO)，这是一种以高效关键 token 选择为中心的新型选择性对齐策略。SePO 提出了第一种基于直接偏好优化 (DPO) 的 token 选择方法，该方法训练一个 oracle 模型来估计目标数据的 token 级奖励函数。此方法适用于任何具有响应级注释的现有对齐数据集，并能够使用小规模 oracle 模型和训练数据进行经济高效的 token 选择。然后利用估计的奖励函数对目标数据集内的所有 token 进行评分，其中仅选择关键 token 来使用无参考模型的对比目标函数来监督目标策略模型。在三个公开评估基准上进行的大量实验表明，SePO 仅通过优化目标数据集上的 30% 关键标记，就显著优于竞争性基线方法。SePO 在弱到强泛化方面的应用表明，弱预言机模型有效地监督了参数多 16.8 倍的强策略模型。SePO 还可以有效地从分布外的数据中选择关键标记，以增强强策略模型并缓解过度优化问题。

Title: HRGraph: Leveraging LLMs for HR Data Knowledge Graphs with Information Propagation-based Job Recommendation

• Authors: Azmine Toushik Wasi
• Subjects: cs.CL, cs.IR, cs.IT, cs.SI
• Abstract URL: https://arxiv.org/abs/2408.13521
• Pdf URL: https://arxiv.org/pdf/2408.13521
• Copy Paste: [[2408.13521]] HRGraph: Leveraging LLMs for HR Data Knowledge Graphs with Information Propagation-based Job Recommendation(https://arxiv.org/abs/2408.13521)
• Keywords: language model, llm
• Abstract: Knowledge Graphs (KGs) serving as semantic networks, prove highly effective in managing complex interconnected data in different domains, by offering a unified, contextualized, and structured representation with flexibility that allows for easy adaptation to evolving knowledge. Processing complex Human Resources (HR) data, KGs can help in different HR functions like recruitment, job matching, identifying learning gaps, and enhancing employee retention. Despite their potential, limited efforts have been made to implement practical HR knowledge graphs. This study addresses this gap by presenting a framework for effectively developing HR knowledge graphs from documents using Large Language Models. The resulting KG can be used for a variety of downstream tasks, including job matching, identifying employee skill gaps, and many more. In this work, we showcase instances where HR KGs prove instrumental in precise job matching, yielding advantages for both employers and employees. Empirical evidence from experiments with information propagation in KGs and Graph Neural Nets, along with case studies underscores the effectiveness of KGs in tasks such as job and employee recommendations and job area classification. Code and data are available at : this https URL
• 摘要：知识图谱 (KG) 是一种语义网络，通过提供统一、情境化和结构化的表示形式，以及易于适应不断发展的知识的灵活性，在管理不同领域的复杂互连数据方面被证明是非常有效的。通过处理复杂的人力资源 (HR) 数据，KG 可以帮助完成不同的人力资源职能，如招聘、职位匹配、识别学习差距和提高员工保留率。尽管它们具有潜力，但在实施实用的人力资源知识图谱方面所做的努力有限。本研究通过提出一个使用大型语言模型从文档中有效开发人力资源知识图谱的框架来解决这一差距。由此产生的 KG 可用于各种下游任务，包括职位匹配、识别员工技能差距等等。在这项工作中，我们展示了人力资源 KG 在精确职位匹配方面发挥重要作用的案例，为雇主和雇员带来了优势。来自 KG 和图神经网络中信息传播实验的经验证据以及案例研究强调了 KG 在职位和员工推荐以及职位领域分类等任务中的有效性。代码和数据可在以下网址获取：此 https URL

Title: Pandora's Box or Aladdin's Lamp: A Comprehensive Analysis Revealing the Role of RAG Noise in Large Language Models

• Authors: Jinyang Wu, Feihu Che, Chuyuan Zhang, Jianhua Tao, Shuai Zhang, Pengpeng Shao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13533
• Pdf URL: https://arxiv.org/pdf/2408.13533
• Copy Paste: [[2408.13533]] Pandora's Box or Aladdin's Lamp: A Comprehensive Analysis Revealing the Role of RAG Noise in Large Language Models(https://arxiv.org/abs/2408.13533)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) has emerged as a crucial method for addressing hallucinations in large language models (LLMs). While recent research has extended RAG models to complex noisy scenarios, these explorations often confine themselves to limited noise types and presuppose that noise is inherently detrimental to LLMs, potentially deviating from real-world retrieval environments and restricting practical applicability. In this paper, we define seven distinct noise types from a linguistic perspective and establish a Noise RAG Benchmark (NoiserBench), a comprehensive evaluation framework encompassing multiple datasets and reasoning tasks. Through empirical evaluation of eight representative LLMs with diverse architectures and scales, we reveal that these noises can be further categorized into two practical groups: noise that is beneficial to LLMs (aka beneficial noise) and noise that is harmful to LLMs (aka harmful noise). While harmful noise generally impairs performance, beneficial noise may enhance several aspects of model capabilities and overall performance. Our analysis offers insights for developing more robust, adaptable RAG solutions and mitigating hallucinations across diverse retrieval scenarios.
• 摘要：检索增强生成 (RAG) 已成为解决大型语言模型 (LLM) 中的幻觉问题的重要方法。虽然最近的研究已将 RAG 模型扩展到复杂的噪声场景，但这些探索通常局限于有限的噪声类型，并假设噪声本质上对 LLM 有害，可能偏离现实世界的检索环境并限制实际适用性。在本文中，我们从语言学的角度定义了七种不同的噪声类型，并建立了噪声 RAG 基准 (NoiserBench)，这是一个涵盖多个数据集和推理任务的综合评估框架。通过对具有不同架构和规模的八个代表性 LLM 进行实证评估，我们发现这些噪声可以进一步分为两类实际噪声：对 LLM 有益的噪声（又名有益噪声）和对 LLM 有害的噪声（又名有害噪声）。虽然有害噪声通常会损害性能，但有益噪声可能会增强模型能力和整体性能的几个方面。我们的分析为开发更为强大、适应性更强的 RAG 解决方案以及减轻不同检索场景中的幻觉提供了见解。

Title: Cultural Adaptation of Menus: A Fine-Grained Approach

• Authors: Zhonghe Zhang, Xiaoyu He, Vivek Iyer, Alexandra Birch
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13534
• Pdf URL: https://arxiv.org/pdf/2408.13534
• Copy Paste: [[2408.13534]] Cultural Adaptation of Menus: A Fine-Grained Approach(https://arxiv.org/abs/2408.13534)
• Keywords: language model, gpt, llm, prompt
• Abstract: Machine Translation of Culture-Specific Items (CSIs) poses significant challenges. Recent work on CSI translation has shown some success using Large Language Models (LLMs) to adapt to different languages and cultures; however, a deeper analysis is needed to examine the benefits and pitfalls of each method. In this paper, we introduce the ChineseMenuCSI dataset, the largest for Chinese-English menu corpora, annotated with CSI vs Non-CSI labels and a fine-grained test set. We define three levels of CSI figurativeness for a more nuanced analysis and develop a novel methodology for automatic CSI identification, which outperforms GPT-based prompts in most categories. Importantly, we are the first to integrate human translation theories into LLM-driven translation processes, significantly improving translation accuracy, with COMET scores increasing by up to 7 points.
• 摘要：文化特定项目 (CSI) 的机器翻译带来了重大挑战。CSI 翻译的最新研究表明，使用大型语言模型 (LLM) 来适应不同的语言和文化取得了一些成功；然而，需要进行更深入的分析来研究每种方法的优点和缺点。在本文中，我们介绍了 ChineseMenuCSI 数据集，这是最大的中英文菜单语料库，带有 CSI 与非 CSI 标签和细粒度测试集。我们定义了三个级别的 CSI 形象性以进行更细致的分析，并开发了一种新颖的自动 CSI 识别方法，该方法在大多数类别中都优于基于 GPT 的提示。重要的是，我们是第一个将人工翻译理论融入 LLM 驱动的翻译过程的人，显著提高了翻译准确性，COMET 分数提高了 7 分。

Title: IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering

• Authors: Ruosen Li, Barry Wang, Ruochen Li, Xinya Du
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13545
• Pdf URL: https://arxiv.org/pdf/2408.13545
• Copy Paste: [[2408.13545]] IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering(https://arxiv.org/abs/2408.13545)
• Keywords: language model, gpt, llm, agent
• Abstract: To evaluate Large Language Models (LLMs) for question answering (QA), traditional methods typically focus on directly assessing the immediate responses generated by the models based on the given question and context. In the common use case of humans seeking AI assistant's help in finding information, these non-interactive evaluations do not account for the dynamic nature of human-model conversations, and interaction-aware evaluations have shown that accurate QA models are preferred by humans (Lee et al., 2023). Recent works in human-computer interaction (HCI) have employed human evaluators to conduct interactions and evaluations, but they are often prohibitively expensive and time-consuming to scale. In this work, we introduce an automatic evaluation framework IQA-EVAL to Interactive Question Answering Evaluation. More specifically, we introduce LLM-based Evaluation Agent (LEA) that can: (1) simulate human behaviors to generate interactions with IQA models; (2) automatically evaluate the generated interactions. Moreover, we propose assigning personas to LEAs to better simulate groups of real human evaluators. We show that: (1) our evaluation framework with GPT-4 (or Claude) as the backbone model achieves a high correlation with human evaluations on the IQA task; (2) assigning personas to LEA to better represent the crowd further significantly improves correlations. Finally, we use our automatic metric to evaluate five recent representative LLMs with over 1000 questions from complex and ambiguous question answering tasks, which comes with a substantial cost of $5k if evaluated by humans.
• 摘要：为了评估用于问答 (QA) 的大型语言模型 (LLM)，传统方法通常侧重于直接评估模型根据给定的问题和上下文生成的即时响应。在人类寻求人工智能助手帮助查找信息的常见用例中，这些非交互式评估没有考虑到人类模型对话的动态性质，而交互感知评估表明，人类更喜欢准确的 QA 模型 (Lee et al., 2023)。人机交互 (HCI) 领域的最新研究已聘请人类评估员进行交互和评估，但它们通常成本高昂且耗时长。在这项工作中，我们引入了一个自动评估框架 IQA-EVAL 来进行交互式问答评估。更具体地说，我们引入了基于 LLM 的评估代理 (LEA)，它可以：(1) 模拟人类行为以生成与 IQA 模型的交互；(2) 自动评估生成的交互。此外，我们建议将角色分配给 LEA，以更好地模拟真实的人类评估者群体。我们表明：（1）以 GPT-4（或 Claude）为骨干模型的评估框架在 IQA 任务上与人工评估具有高度相关性；（2）将角色分配给 LEA 以更好地代表人群，可进一步显著提高相关性。最后，我们使用我们的自动指标来评估最近五个具有代表性的 LLM，其中包含来自复杂且模糊的问答任务的 1000 多个问题，如果由人工评估，则需要花费 5,000 美元。

Title: Balancing Diversity and Risk in LLM Sampling: How to Select Your Method and Parameter for Open-Ended Text Generation

• Authors: Yuxuan Zhou, Margret Keuper, Mario Fritz
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.13586
• Pdf URL: https://arxiv.org/pdf/2408.13586
• Copy Paste: [[2408.13586]] Balancing Diversity and Risk in LLM Sampling: How to Select Your Method and Parameter for Open-Ended Text Generation(https://arxiv.org/abs/2408.13586)
• Keywords: language model, llm
• Abstract: Sampling-based decoding strategies have been widely adopted for Large Language Models (LLMs) in numerous applications, which target a balance between diversity and quality via temperature tuning and tail truncation (e.g., top-k and top-p sampling). Considering the high dynamic range of the candidate next-token given different prefixes, recent studies propose to adaptively truncate the tail of LLM's predicted distribution. Although improved results haven been reported with these methods on open-ended text generation tasks, the results are highly dependent on the curated truncation parameters and exemplar text. In this paper, we propose a systematic way to estimate the intrinsic capacity of a truncation sampling method by considering the trade-off between diversity and risk at each decoding step, based on our collected prefix tree which preserves the context of a full sentence. Our work provides a comprehensive comparison between existing truncation sampling methods, as well as their recommended parameters as a guideline for users.
• 摘要：基于采样的解码策略已广泛应用于众多应用中的大型语言模型 (LLM)，其目标是通过温度调整和尾部截断（例如，top-k 和 top-p 采样）在多样性和质量之间取得平衡。考虑到给定不同前缀的候选下一个标记的高动态范围，最近的研究提出自适应地截断 LLM 预测分布的尾部。尽管这些方法在开放式文本生成任务上尚未报告改进的结果，但结果高度依赖于精选的截断参数和示例文本。在本文中，我们提出了一种系统的方法，通过考虑每个解码步骤中的多样性和风险之间的权衡，基于我们收集的保留完整句子上下文的前缀树。我们的工作对现有的截断采样方法进行了全面的比较，并为其推荐了参数作为用户指南。

Title: No Dataset Needed for Downstream Knowledge Benchmarking: Response Dispersion Inversely Correlates with Accuracy on Domain-specific QA

• Authors: Robert L Simione II
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.13624
• Pdf URL: https://arxiv.org/pdf/2408.13624
• Copy Paste: [[2408.13624]] No Dataset Needed for Downstream Knowledge Benchmarking: Response Dispersion Inversely Correlates with Accuracy on Domain-specific QA(https://arxiv.org/abs/2408.13624)
• Keywords: llm, prompt, chat
• Abstract: This research seeks to obviate the need for creating QA datasets and grading (chatbot) LLM responses when comparing LLMs' knowledge in specific topic domains. This is done in an entirely end-user centric way without need for access to any inner workings of the LLM, so long as it can be prompted and given a random seed to create different generations to the same prompt. The paper does this by, for a given topic domain, defining the "response dispersion" of an LLM by repeatedly asking an LLM the same opinion question about that topic domain. Namely, the response dispersion is the count of singular values needed to explain 95% of the variance in the embedding matrix of the LLM's responses. It is found that the response dispersion is inversely correlated with accuracy on relevant QA evaluations (average spearman rank correlation stronger than -.59). A use-case analysis shows that when comparing two different LLMs on the same topic domain, comparing their response dispersion is a suitable replacement for comparing their QA accuracy between 74% and 89% of the time, the range depending on certain reasonable accuracy-difference tolerances that may be acceptable to an end-user in exchange for the labor being saved using response dispersion instead of QA accuracy for comparison. Two response embeddings are studied for creating the embedding matrix in this study, one is from OpenAI's APIs and one is a novel embedding, here named reference sentence similarity embeddings, that can be computed locally and performs very nearly as well in calculating response dispersion. Also in this research, a pre-existing dataset called the IRC-Wiki Trivia dataset, originally developed for trivia games, has been re-purposed, curated, and the curation, called IRC-WikiTriviaQA, is made available for the purpose of this research.
• 摘要：本研究旨在消除在比较 LLM 在特定主题领域的知识时创建 QA 数据集和对（聊天机器人）LLM 响应进行评分的需要。这是以完全以最终用户为中心的方式完成的，无需访问 LLM 的任何内部工作，只要可以提示并给出随机种子以创建针对同一提示的不同代即可。本文通过针对给定主题域，通过反复向 LLM 询问有关该主题域的相同意见问题来定义 LLM 的“响应分散度”。即，响应分散度是解释 LLM 响应嵌入矩阵中 95% 方差所需的奇异值计数。研究发现，响应分散度与相关 QA 评估的准确性呈负相关（平均斯皮尔曼等级相关性强于 -.59）。用例分析表明，在比较同一主题领域的两个不同 LLM 时，比较它们的响应分散度是比较 74% 到 89% 的 QA 准确度的合适替代方法，范围取决于某些合理的准确度差异公差，最终用户可以接受这些公差，以换取使用响应分散度而不是 QA 准确度进行比较所节省的劳动。本研究研究了两种响应嵌入以创建嵌入矩阵，一种来自 OpenAI 的 API，一种是新型嵌入，此处称为参考句子相似度嵌入，它可以在本地计算，并且在计算响应分散度方面表现非常出色。此外，在本研究中，一个预先存在的数据集（称为 IRC-Wiki Trivia 数据集，最初为琐事游戏开发）已被重新利用和整理，并且该整理的数据集（称为 IRC-WikiTriviaQA）可用于本研究。

Title: Symbolic Working Memory Enhances Language Models for Complex Rule Application

• Authors: Siyuan Wang, Zhongyu Wei, Yejin Choi, Xiang Ren
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13654
• Pdf URL: https://arxiv.org/pdf/2408.13654
• Copy Paste: [[2408.13654]] Symbolic Working Memory Enhances Language Models for Complex Rule Application(https://arxiv.org/abs/2408.13654)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have shown remarkable reasoning performance but struggle with multi-step deductive reasoning involving a series of rule application steps, especially when rules are presented non-sequentially. Our preliminary analysis shows that while LLMs excel in single-step rule application, their performance drops significantly in multi-step scenarios due to the challenge in rule grounding. It requires anchoring the applicable rule and supporting facts at each step, amidst multiple input rules, facts, and inferred facts. To address this, we propose augmenting LLMs with external working memory and introduce a neurosymbolic framework for rule application. The memory stores facts and rules in both natural language and symbolic forms, enabling precise tracking. Utilizing this memory, our framework iteratively performs symbolic rule grounding and LLM-based rule implementation. The former matches predicates and variables of symbolic rules and facts to ground applicable rules at each step. Experiments indicate our framework's effectiveness in rule application and its robustness across various steps and settings~\footnote{Code and data are available at \url{this https URL}.}.
• 摘要：大型语言模型 (LLM) 表现出卓越的推理性能，但在涉及一系列规则应用步骤的多步骤演绎推理方面却举步维艰，尤其是在规则非连续呈现时。我们的初步分析表明，虽然 LLM 在单步规则应用中表现出色，但由于规则基础的挑战，它们在多步骤场景中的表现会显著下降。它需要在多个输入规则、事实和推断事实中，在每个步骤中锚定适用规则和支持事实。为了解决这个问题，我们建议用外部工作记忆增强 LLM，并引入一个用于规则应用的神经符号框架。记忆以自然语言和符号形式存储事实和规则，从而实现精确跟踪。利用这种记忆，我们的框架迭代地执行符号规则基础和基于 LLM 的规则实现。前者将符号规则和事实的谓词和变量与每个步骤的基础适用规则相匹配。实验表明我们的框架在规则应用中的有效性及其在各种步骤和设置中的稳健性~\footnote{代码和数据可在 \url{此 https URL} 处获得。}。

Title: DHP Benchmark: Are LLMs Good NLG Evaluators?

• Authors: Yicheng Wang, Jiayi Yuan, Yu-Neng Chuang, Zhuoer Wang, Yingchi Liu, Mark Cusick, Param Kulkarni, Zhengping Ji, Yasser Ibrahim, Xia Hu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.13704
• Pdf URL: https://arxiv.org/pdf/2408.13704
• Copy Paste: [[2408.13704]] DHP Benchmark: Are LLMs Good NLG Evaluators?(https://arxiv.org/abs/2408.13704)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are increasingly serving as evaluators in Natural Language Generation (NLG) tasks. However, the capabilities of LLMs in scoring NLG quality remain inadequately explored. Current studies depend on human assessments and simple metrics that fail to capture the discernment of LLMs across diverse NLG tasks. To address this gap, we propose the Discernment of Hierarchical Perturbation (DHP) benchmarking framework, which provides quantitative discernment scores for LLMs utilizing hierarchically perturbed text data and statistical tests to measure the NLG evaluation capabilities of LLMs systematically. We have re-established six evaluation datasets for this benchmark, covering four NLG tasks: Summarization, Story Completion, Question Answering, and Translation. Our comprehensive benchmarking of five major LLM series provides critical insight into their strengths and limitations as NLG evaluators.
• 摘要：大型语言模型 (LLM) 越来越多地充当自然语言生成 (NLG) 任务的评估者。然而，LLM 在评估 NLG 质量方面的能力仍未得到充分探索。当前的研究依赖于人工评估和简单的指标，无法捕捉到 LLM 在各种 NLG 任务中的辨别能力。为了弥补这一差距，我们提出了分层扰动辨别 (DHP) 基准测试框架，该框架利用分层扰动的文本数据和统计测试为 LLM 提供定量辨别分数，以系统地衡量 LLM 的 NLG 评估能力。我们为该基准重新建立了六个评估数据集，涵盖四个 NLG 任务：摘要、故事完成、问答和翻译。我们对五个主要 LLM 系列进行了全面的基准测试，深入了解了它们作为 NLG 评估者的优势和局限性。

Title: Poor-Supervised Evaluation for SuperLLM via Mutual Consistency

• Authors: Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Boyuan Pan, Heda Wang, Yao Hu, Kan Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13738
• Pdf URL: https://arxiv.org/pdf/2408.13738
• Copy Paste: [[2408.13738]] Poor-Supervised Evaluation for SuperLLM via Mutual Consistency(https://arxiv.org/abs/2408.13738)
• Keywords: llm
• Abstract: The guidance from capability evaluations has greatly propelled the progress of both human society and Artificial Intelligence. However, as LLMs evolve, it becomes challenging to construct evaluation benchmarks for them with accurate labels on hard tasks that approach the boundaries of human capabilities. To credibly conduct evaluation without accurate labels (denoted as poor-supervised evaluation), we propose the PoEM framework. We first prove that the capability of a model can be equivalently assessed by the consistency between it and certain reference model, when their prediction distributions are independent and the sample size is infinite. To alleviate the insufficiencies of the conditions in reality, we further introduce an algorithm that treats humans (when available) and the models under evaluation as reference models, alternately conducting model weights calibration and filtering during E-step and M-step. Comprehensive experiments across 3 types of tasks with 16 mainstream LLMs have shown that PoEM under poor supervision can achieve an average of 0.98 Pearson correlation coefficient with supervised evaluation results, demonstrating good effectiveness, efficiency and generalizability. More generally, PoEM has advanced the evaluation paradigm evolution from human-centric to human&model-centric by treating both of them as reference models, mitigating the limitations of human evaluation in the era of LLMs.
• 摘要：能力评估的指导极大地推动了人类社会和人工智能的进步。然而随着LLM的不断发展，在接近人类能力边界的困难任务上，为LLM构建具有准确标签的评估基准变得越来越具有挑战性。为了可靠地进行无准确标签的评估（称为弱监督评估），我们提出了PoEM框架。我们首先证明，当预测分布独立且样本量无限大时，模型的能力可以通过它与某个参考模型的一致性来等效地评估。为了缓解现实条件的不足，我们进一步引入一种算法，将人类（如果有）和被评估模型视为参考模型，在E步和M步交替进行模型权重校准和过滤。使用16个主流LLM在3类任务上的综合实验表明，弱监督下的PoEM与监督评估结果的皮尔逊相关系数平均可达0.98，具有良好的有效性、效率和泛化性。更普遍地说，PoEM 通过将评估范式与模型一起视为参考模型，推动了评估范式从以人为中心向以人和模型为中心的演变，减轻了 LLM 时代人工评估的局限性。

Title: DOCE: Finding the Sweet Spot for Execution-Based Code Generation

• Authors: Haau-Sing Li, Patrick Fernandes, Iryna Gurevych, André F.T. Martins
• Subjects: cs.CL, cs.AI, cs.PL
• Abstract URL: https://arxiv.org/abs/2408.13745
• Pdf URL: https://arxiv.org/pdf/2408.13745
• Copy Paste: [[2408.13745]] DOCE: Finding the Sweet Spot for Execution-Based Code Generation(https://arxiv.org/abs/2408.13745)
• Keywords: llm
• Abstract: Recently, a diverse set of decoding and reranking procedures have been shown effective for LLM-based code generation. However, a comprehensive framework that links and experimentally compares these methods is missing. We address this by proposing Decoding Objectives for Code Execution, a comprehensive framework that includes candidate generation, $n$-best reranking, minimum Bayes risk (MBR) decoding, and self-debugging as the core components. We then study the contributions of these components through execution-based evaluation metrics. Our findings highlight the importance of execution-based methods and the difference gap between execution-based and execution-free methods. Furthermore, we assess the impact of filtering based on trial unit tests, a simple and effective strategy that has been often overlooked in prior works. We also propose self-debugging on multiple candidates, obtaining state-of-the-art performance on reranking for code generation. We expect our framework to provide a solid guideline for future research on code generation.
• 摘要：最近，一组多样化的解码和重新排序程序已被证明对基于 LLM 的代码生成有效。然而，缺少一个将这些方法联系起来并通过实验进行比较的综合框架。我们通过提出代码执行的解码目标来解决这个问题，这是一个综合框架，包括候选生成、$n$ 最佳重新排序、最小贝叶斯风险 (MBR) 解码和自调试作为核心组件。然后，我们通过基于执行的评估指标研究这些组件的贡献。我们的研究结果强调了基于执行的方法的重要性以及基于执行和无执行方法之间的差异差距。此外，我们评估了基于试验单元测试的过滤的影响，这是一种简单有效的策略，在以前的工作中经常被忽视。我们还建议对多个候选进行自调试，以获得代码生成重新排序的最新性能。我们希望我们的框架能够为未来的代码生成研究提供坚实的指导。

Title: Towards Reliable Medical Question Answering: Techniques and Challenges in Mitigating Hallucinations in Language Models

• Authors: Duy Khoa Pham, Bao Quoc Vo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13808
• Pdf URL: https://arxiv.org/pdf/2408.13808
• Copy Paste: [[2408.13808]] Towards Reliable Medical Question Answering: Techniques and Challenges in Mitigating Hallucinations in Language Models(https://arxiv.org/abs/2408.13808)
• Keywords: language model, llm, hallucination, prompt, retrieval-augmented generation
• Abstract: The rapid advancement of large language models (LLMs) has significantly impacted various domains, including healthcare and biomedicine. However, the phenomenon of hallucination, where LLMs generate outputs that deviate from factual accuracy or context, poses a critical challenge, especially in high-stakes domains. This paper conducts a scoping study of existing techniques for mitigating hallucinations in knowledge-based task in general and especially for medical domains. Key methods covered in the paper include Retrieval-Augmented Generation (RAG)-based techniques, iterative feedback loops, supervised fine-tuning, and prompt engineering. These techniques, while promising in general contexts, require further adaptation and optimization for the medical domain due to its unique demands for up-to-date, specialized knowledge and strict adherence to medical guidelines. Addressing these challenges is crucial for developing trustworthy AI systems that enhance clinical decision-making and patient safety as well as accuracy of biomedical scientific research.
• 摘要：大型语言模型 (LLM) 的快速发展对医疗保健和生物医学等各个领域产生了重大影响。然而，幻觉现象（LLM 产生的输出偏离事实准确性或上下文）构成了严峻挑战，尤其是在高风险领域。本文对现有的缓解知识型任务中的幻觉的技术进行了范围界定研究，特别是针对医学领域。本文涵盖的主要方法包括基于检索增强生成 (RAG) 的技术、迭代反馈循环、监督微调和快速工程。这些技术虽然在一般情况下很有前景，但由于医学领域对最新、专业知识和严格遵守医疗指南的独特需求，因此需要进一步调整和优化。应对这些挑战对于开发可信赖的 AI 系统至关重要，这些系统可以增强临床决策和患者安全以及生物医学科学研究的准确性。

Title: Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Data

• Authors: Felix J. Dorfner, Amin Dada, Felix Busch, Marcus R. Makowski, Tianyu Han, Daniel Truhn, Jens Kleesiek, Madhumita Sushil, Jacqueline Lammert, Lisa C. Adams, Keno K. Bressem
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13833
• Pdf URL: https://arxiv.org/pdf/2408.13833
• Copy Paste: [[2408.13833]] Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Data(https://arxiv.org/abs/2408.13833)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Large language models (LLMs) have shown potential in biomedical applications, leading to efforts to fine-tune them on domain-specific data. However, the effectiveness of this approach remains unclear. This study evaluates the performance of biomedically fine-tuned LLMs against their general-purpose counterparts on a variety of clinical tasks. We evaluated their performance on clinical case challenges from the New England Journal of Medicine (NEJM) and the Journal of the American Medical Association (JAMA) and on several clinical tasks (e.g., information extraction, document summarization, and clinical coding). Using benchmarks specifically chosen to be likely outside the fine-tuning datasets of biomedical models, we found that biomedical LLMs mostly perform inferior to their general-purpose counterparts, especially on tasks not focused on medical knowledge. While larger models showed similar performance on case tasks (e.g., OpenBioLLM-70B: 66.4% vs. Llama-3-70B-Instruct: 65% on JAMA cases), smaller biomedical models showed more pronounced underperformance (e.g., OpenBioLLM-8B: 30% vs. Llama-3-8B-Instruct: 64.3% on NEJM cases). Similar trends were observed across the CLUE (Clinical Language Understanding Evaluation) benchmark tasks, with general-purpose models often performing better on text generation, question answering, and coding tasks. Our results suggest that fine-tuning LLMs to biomedical data may not provide the expected benefits and may potentially lead to reduced performance, challenging prevailing assumptions about domain-specific adaptation of LLMs and highlighting the need for more rigorous evaluation frameworks in healthcare AI. Alternative approaches, such as retrieval-augmented generation, may be more effective in enhancing the biomedical capabilities of LLMs without compromising their general knowledge.
• 摘要：大型语言模型 (LLM) 在生物医学应用中显示出巨大的潜力，人们开始努力针对特定领域的数据对其进行微调。然而，这种方法的有效性仍不清楚。这项研究评估了生物医学微调 LLM 与通用 LLM 在各种临床任务上的表现。我们评估了它们在《新英格兰医学杂志》(NEJM) 和《美国医学会杂志》(JAMA) 的临床病例挑战以及几项临床任务（例如信息提取、文档摘要和临床编码）中的表现。使用专门选择的可能超出生物医学模型微调数据集的基准，我们发现生物医学 LLM 的表现大多不如通用 LLM，尤其是在不关注医学知识的任务上。虽然较大的模型在案例任务上表现出相似的性能（例如，OpenBioLLM-70B：66.4% vs. Llama-3-70B-Instruct：JAMA 案例中的 65%），但较小的生物医学模型表现出更明显的表现不佳（例如，OpenBioLLM-8B：30% vs. Llama-3-8B-Instruct：NEJM 案例中的 64.3%）。在 CLUE（临床语言理解评估）基准任务中也观察到了类似的趋势，通用模型在文本生成、问答和编码任务上的表现通常更好。我们的结果表明，将 LLM 微调到生物医学数据可能无法提供预期的好处，并且可能会导致性能下降，挑战了关于 LLM 领域特定适应性的普遍假设，并凸显了医疗 AI 需要更严格的评估框架。检索增强生成等替代方法可能更有效地提高 LLM 的生物医学能力，同时又不损害他们的一般知识。

Title: CodeGraph: Enhancing Graph Reasoning of LLMs with Code

• Authors: Qiaolong Cai, Zhaowei Wang, Shizhe Diao, James Kwok, Yangqiu Song
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.13863
• Pdf URL: https://arxiv.org/pdf/2408.13863
• Copy Paste: [[2408.13863]] CodeGraph: Enhancing Graph Reasoning of LLMs with Code(https://arxiv.org/abs/2408.13863)
• Keywords: language model, gpt, llm
• Abstract: With the increasing popularity of large language models (LLMs), reasoning on basic graph algorithm problems is an essential intermediate step in assessing their abilities to process and infer complex graph reasoning tasks. Existing methods usually convert graph-structured data to textual descriptions and then use LLMs for reasoning and computation. However, LLMs often produce computation errors on arithmetic parts in basic graph algorithm problems, such as counting number of edges. In addition, they struggle to control or understand the output of the reasoning process, raising concerns about whether LLMs are simply guessing. In this paper, we introduce CodeGraph, a method that encodes graph problem solutions as code. The methods solve new graph problems by learning from exemplars, generating programs, and executing them via a program interpreter. Using the few-shot setting, we evaluate CodeGraph with the base LLM being GPT-3.5 Turbo, Llama3-70B Instruct, Mixtral-8x22B Instruct, and Mixtral-8x7B Instruct. Experimental results on six tasks with six graph encoding methods in the GraphQA dataset demonstrate that CodeGraph can boost performance on graph reasoning tasks inside LLMs by 1.3% to 58.6%, depending on the task. Compared to the existing methods, CodeGraph demonstrates strong performance on arithmetic problems in graph tasks and offers a more controllable and interpretable approach to the reasoning process.
• 摘要：随着大型语言模型 (LLM) 的日益普及，对基本图算法问题的推理是评估其处理和推断复杂图推理任务的能力的重要中间步骤。现有方法通常将图结构数据转换为文本描述，然后使用 LLM 进行推理和计算。然而，LLM 经常在基本图算法问题的算术部分产生计算错误，例如计算边数。此外，它们难以控制或理解推理过程的输出，这引发了人们对 LLM 是否只是猜测的担忧。在本文中，我们介绍了 CodeGraph，一种将图问题解决方案编码为代码的方法。这些方法通过从样例中学习、生成程序并通过程序解释器执行它们来解决新的图问题。使用少样本设置，我们评估 CodeGraph，基础 LLM 是 GPT-3.5 Turbo、Llama3-70B Instruct、Mixtral-8x22B Instruct 和 Mixtral-8x7B Instruct。在 GraphQA 数据集上对六种图编码方法的六项任务进行的实验结果表明，CodeGraph 可将 LLM 中的图推理任务的性能提升 1.3% 至 58.6%，具体取决于任务。与现有方法相比，CodeGraph 在图任务中的算术问题上表现出色，并为推理过程提供了更可控和更可解释的方法。

Title: LLM with Relation Classifier for Document-Level Relation Extraction

• Authors: Xingzuo Li, Kehai Chen, Yunfei Long, Min Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13889
• Pdf URL: https://arxiv.org/pdf/2408.13889
• Copy Paste: [[2408.13889]] LLM with Relation Classifier for Document-Level Relation Extraction(https://arxiv.org/abs/2408.13889)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) create a new paradigm for natural language processing. Despite their advancement, LLM-based methods still lag behind traditional approaches in document-level relation extraction (DocRE), a critical task for understanding complex entity relations. This paper investigates the causes of this performance gap, identifying the dispersion of attention by LLMs due to entity pairs without relations as a primary factor. We then introduce a novel classifier-LLM approach to DocRE. The proposed approach begins with a classifier specifically designed to select entity pair candidates exhibiting potential relations and thereby feeds them to LLM for the final relation extraction. This method ensures that during inference, the LLM's focus is directed primarily at entity pairs with relations. Experiments on DocRE benchmarks reveal that our method significantly outperforms recent LLM-based DocRE models and achieves competitive performance with several leading traditional DocRE models.
• 摘要：大型语言模型 (LLM) 为自然语言处理创造了一种新范式。尽管基于 LLM 的方法取得了进步，但它们在文档级关系提取 (DocRE) 方面仍然落后于传统方法，而文档级关系提取是理解复杂实体关系的关键任务。本文探讨了造成这种性能差距的原因，发现 LLM 注意力因没有关系的实体对而分散是主要因素。然后，我们为 DocRE 引入了一种新颖的分类器-LLM 方法。所提出的方法首先使用专门设计的分类器来选择表现出潜在关系的实体对候选，然后将它们提供给 LLM 进行最终的关系提取。此方法可确保在推理过程中，LLM 的注意力主要集中在具有关系的实体对上。在 DocRE 基准上的实验表明，我们的方法明显优于最近基于 LLM 的 DocRE 模型，并且与几种领先的传统 DocRE 模型相比具有竞争力。

Title: SpeechCaps: Advancing Instruction-Based Universal Speech Models with Multi-Talker Speaking Style Captioning

• Authors: Chien-yu Huang, Min-Han Shih, Ke-Han Lu, Chi-Yuan Hsiao, Hung-yi Lee
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.13891
• Pdf URL: https://arxiv.org/pdf/2408.13891
• Copy Paste: [[2408.13891]] SpeechCaps: Advancing Instruction-Based Universal Speech Models with Multi-Talker Speaking Style Captioning(https://arxiv.org/abs/2408.13891)
• Keywords: language model
• Abstract: Instruction-based speech processing is becoming popular. Studies show that training with multiple tasks boosts performance, but collecting diverse, large-scale tasks and datasets is expensive. Thus, it is highly desirable to design a fundamental task that benefits other downstream tasks. This paper introduces a multi-talker speaking style captioning task to enhance the understanding of speaker and prosodic information. We used large language models to generate descriptions for multi-talker speech. Then, we trained our model with pre-training on this captioning task followed by instruction tuning. Evaluation on Dynamic-SUPERB shows our model outperforming the baseline pre-trained only on single-talker tasks, particularly in speaker and emotion recognition. Additionally, tests on a multi-talker QA task reveal that current models struggle with attributes such as gender, pitch, and speaking rate. The code and dataset are available at this https URL.
• 摘要：基于指令的语音处理正变得越来越流行。研究表明，使用多个任务进行训练可以提高性能，但收集多样化、大规模的任务和数据集的成本很高。因此，设计一个有利于其他下游任务的基本任务是非常可取的。本文介绍了一种多说话者说话风格字幕任务，以增强对说话者和韵律信息的理解。我们使用大型语言模型来生成多说话者语音的描述。然后，我们对我们的模型进行了预训练，该字幕任务随后进行了指令调整。在 Dynamic-SUPERB 上的评估表明，我们的模型仅在单说话者任务上优于预训练的基线，特别是在说话者和情感识别方面。此外，在多说话者 QA 任务上的测试表明，当前模型在性别、音调和语速等属性方面存在困难。代码和数据集可在此 https URL 上找到。

Title: LLMs are Superior Feedback Providers: Bootstrapping Reasoning for Lie Detection with Self-Generated Feedback

• Authors: Tanushree Banerjee, Richard Zhu, Runzhe Yang, Karthik Narasimhan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.13915
• Pdf URL: https://arxiv.org/pdf/2408.13915
• Copy Paste: [[2408.13915]] LLMs are Superior Feedback Providers: Bootstrapping Reasoning for Lie Detection with Self-Generated Feedback(https://arxiv.org/abs/2408.13915)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) excel at generating human-like dialogues and comprehending text. However, understanding the subtleties of complex exchanges in language remains a challenge. We propose a bootstrapping framework that leverages self-generated feedback to enhance LLM reasoning capabilities for lie detection. The framework consists of three stages: suggestion, feedback collection, and modification. In the suggestion stage, a cost-effective language model generates initial predictions based on game state and dialogue. The feedback-collection stage involves a language model providing feedback on these predictions. In the modification stage, a more advanced language model refines the initial predictions using the auto-generated feedback. We investigate the application of the proposed framework for detecting betrayal and deception in Diplomacy games, and compare it with feedback from professional human players. The LLM-generated feedback exhibits superior quality and significantly enhances the performance of the model. Our approach achieves a 39% improvement over the zero-shot baseline in lying-F1 without the need for any training data, rivaling state-of-the-art supervised learning results.
• 摘要：大型语言模型 (LLM) 擅长生成类似人类的对话和理解文本。然而，理解语言中复杂交流的微妙之处仍然是一个挑战。我们提出了一个引导框架，利用自生成的反馈来增强 LLM 的谎言检测推理能力。该框架包括三个阶段：建议、反馈收集和修改。在建议阶段，经济高效的语言模型根据游戏状态和对话生成初始预测。反馈收集阶段涉及语言模型对这些预测提供反馈。在修改阶段，更先进的语言模型使用自动生成的反馈来改进初始预测。我们研究了所提出的框架在外交游戏中检测背叛和欺骗的应用，并将其与专业人类玩家的反馈进行比较。LLM 生成的反馈表现出卓越的质量，并显著提高了模型的性能。我们的方法在不需要任何训练数据的情况下，在谎言-F1 中比零样本基线提高了 39%，可与最先进的监督学习结果相媲美。

Title: MobileQuant: Mobile-friendly Quantization for On-device Language Models

• Authors: Fuwen Tan, Royson Lee, Łukasz Dudziak, Shell Xu Hu, Sourav Bhattacharya, Timothy Hospedales, Georgios Tzimiropoulos, Brais Martinez
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13933
• Pdf URL: https://arxiv.org/pdf/2408.13933
• Copy Paste: [[2408.13933]] MobileQuant: Mobile-friendly Quantization for On-device Language Models(https://arxiv.org/abs/2408.13933)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have revolutionized language processing, delivering outstanding results across multiple applications. However, deploying LLMs on edge devices poses several challenges with respect to memory, energy, and compute costs, limiting their widespread use in devices such as mobile phones. A promising solution is to reduce the number of bits used to represent weights and activations. While existing works have found partial success at quantizing LLMs to lower bitwidths, e.g. 4-bit weights, quantizing activations beyond 16 bits often leads to large computational overheads due to poor on-device quantization support, or a considerable accuracy drop. Yet, 8-bit activations are very attractive for on-device deployment as they would enable LLMs to fully exploit mobile-friendly hardware, e.g. Neural Processing Units (NPUs). In this work, we make a first attempt to facilitate the on-device deployment of LLMs using integer-only quantization. We first investigate the limitations of existing quantization methods for on-device deployment, with a special focus on activation quantization. We then address these limitations by introducing a simple post-training quantization method, named MobileQuant, that extends previous weight equivalent transformation works by jointly optimizing the weight transformation and activation range parameters in an end-to-end manner. MobileQuant demonstrates superior capabilities over existing methods by 1) achieving near-lossless quantization on a wide range of LLM benchmarks, 2) reducing latency and energy consumption by 20\%-50\% compared to current on-device quantization strategies, 3) requiring limited compute budget, 4) being compatible with mobile-friendly compute units, e.g. NPU.
• 摘要：大型语言模型 (LLM) 彻底改变了语言处理，在多个应用程序中都取得了出色的成果。然而，在边缘设备上部署 LLM 在内存、能源和计算成本方面带来了一些挑战，限制了它们在手机等设备上的广泛使用。一个有希望的解决方案是减少用于表示权重和激活的位数。虽然现有研究已经发现将 LLM 量化为较低位宽（例如 4 位权重）取得了部分成功，但量化超过 16 位的激活通常会导致计算开销很大，因为设备上量化支持不佳或准确率大幅下降。然而，8 位激活对于设备上部署非常有吸引力，因为它们将使 LLM 能够充分利用移动友好型硬件，例如神经处理单元 (NPU)。在这项工作中，我们首次尝试使用仅整数量化来促进 LLM 的设备部署。我们首先研究现有量化方法在设备上部署的局限性，特别关注激活量化。然后，我们通过引入一种简单的训练后量化方法（名为 MobileQuant）来解决这些限制，该方法通过以端到端方式联合优化权重变换和激活范围参数来扩展以前的权重等效变换工作。MobileQuant 展示了优于现有方法的卓越能力，具体表现在：1) 在广泛的 LLM 基准上实现近乎无损的量化，2) 与当前的设备上量化策略相比，延迟和能耗降低了 20\%-50\%，3) 所需的计算预算有限，4) 与移动友好型计算单元（例如 NPU）兼容。

Title: CoT Rerailer: Enhancing the Reliability of Large Language Models in Complex Reasoning Tasks through Error Detection and Correction

• Authors: Guangya Wan, Yuqi Wu, Jie Chen, Sheng Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13940
• Pdf URL: https://arxiv.org/pdf/2408.13940
• Copy Paste: [[2408.13940]] CoT Rerailer: Enhancing the Reliability of Large Language Models in Complex Reasoning Tasks through Error Detection and Correction(https://arxiv.org/abs/2408.13940)
• Keywords: language model, llm, hallucination, prompt, chain-of-thought, agent
• Abstract: Chain-of-Thought (CoT) prompting enhances Large Language Models (LLMs) complex reasoning abilities by generating intermediate steps. However, these steps can introduce hallucinations and accumulate errors. We propose the CoT Rerailer to address these challenges, employing self-consistency and multi-agent debate systems to identify and rectify errors in the reasoning process. The CoT Rerailer first selects the most logically correct Reasoning Path (RP) using consistency checks and critical evaluation by automated agents. It then engages a multi-agent debate system to propose and validate corrections to ensure the generation of an error-free intermediate logical path. The corrected steps are then used to generate a revised reasoning chain to further reduce hallucinations and enhance answer quality. We demonstrate the effectiveness of our approach across diverse question-answering datasets in various knowledge domains. The CoT Rerailer enhances the reliability of LLM-generated reasoning, contributing to more trustworthy AI driven decision-making processes.
• 摘要：思路链 (CoT) 提示通过生成中间步骤增强了大型语言模型 (LLM) 的复杂推理能力。然而，这些步骤可能会引入幻觉并积累错误。我们提出了 CoT Rerailer 来解决这些挑战，采用自洽和多智能体辩论系统来识别和纠正推理过程中的错误。CoT Rerailer 首先使用一致性检查和自动代理的批判性评估来选择最合乎逻辑的正确推理路径 (RP)。然后，它使用多智能体辩论系统来提出和验证更正，以确保生成无错误的中间逻辑路径。然后使用更正后的步骤生成修订的推理链，以进一步减少幻觉并提高答案质量。我们在各个知识领域的各种问答数据集中证明了我们的方法的有效性。CoT Rerailer 增强了 LLM 生成的推理的可靠性，有助于实现更值得信赖的 AI 驱动决策过程。

Title: Bidirectional Awareness Induction in Autoregressive Seq2Seq Models

• Authors: Jia Cheng Hu, Roberto Cavicchioli, Alessandro Capotondi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13959
• Pdf URL: https://arxiv.org/pdf/2408.13959
• Copy Paste: [[2408.13959]] Bidirectional Awareness Induction in Autoregressive Seq2Seq Models(https://arxiv.org/abs/2408.13959)
• Keywords: gpt, llm
• Abstract: Autoregressive Sequence-To-Sequence models are the foundation of many Deep Learning achievements in major research fields such as Vision and Natural Language Processing. Despite that, they still present significant limitations. For instance, when errors occur in the early steps of the prediction, the whole output is severely affected. Such reliance on previously predicted tokens and the inherent computational unfriendliness of sequential algorithms, motivated researchers to explore different architectures and methods in the search for bidirectional approaches. In this work, we introduce the Bidirectional Awareness Induction (BAI), a training method that leverages a subset of elements in the network, the Pivots, to perform bidirectional learning without breaking the autoregressive constraints. To showcase its flexibility, we apply the method to three architectures, the Transformer, ExpansionNet v2 and GPT, then perform experiments over three tasks. Experimental results showcase BAI's effectiveness on all selected tasks and architectures. In particular, we observed an increase of up to 2.4 CIDEr in Image-Captioning, 4.96 BLEU in Neural Machine Translation, and 1.16 ROUGE in Text Summarization compared to the respective baselines. Notably, BAI not only has a positive impact on models trained from scratch but on pre-trained models as well. Such an aspect, combined with the absence of architectural requirements synergizes well with the current trend of LLMs.
• 摘要：自回归序列到序列模型是视觉和自然语言处理等主要研究领域中许多深度学习成果的基础。尽管如此，它们仍然存在重大局限性。例如，当预测的早期步骤出现错误时，整个输出将受到严重影响。这种对先前预测的标记的依赖以及顺序算法固有的计算不友好性促使研究人员探索不同的架构和方法以寻找双向方法。在这项工作中，我们引入了双向意识诱导 (BAI)，这是一种利用网络中元素子集 Pivots 进行双向学习而不破坏自回归约束的训练方法。为了展示其灵活性，我们将该方法应用于三种架构，即 Transformer、ExpansionNet v2 和 GPT，然后在三个任务上进行实验。实验结果展示了 BAI 在所有选定任务和架构上的有效性。具体而言，我们观察到与相应基线相比，图像字幕中的 CIDEr 增加了 2.4，神经机器翻译中的 BLEU 增加了 4.96，文本摘要中的 ROUGE 增加了 1.16。值得注意的是，BAI 不仅对从头开始训练的模型有积极影响，而且对预训练模型也有积极影响。这样的方面，加上没有架构要求，与 LLM 的当前趋势很好地协同作用。

Title: Reducing the Cost: Cross-Prompt Pre-Finetuning for Short Answer Scoring

• Authors: Hiroaki Funayama, Yuya Asazuma, Yuichiroh Matsubayashi, Tomoya Mizumoto, Kentaro Inui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13966
• Pdf URL: https://arxiv.org/pdf/2408.13966
• Copy Paste: [[2408.13966]] Reducing the Cost: Cross-Prompt Pre-Finetuning for Short Answer Scoring(https://arxiv.org/abs/2408.13966)
• Keywords: prompt
• Abstract: Automated Short Answer Scoring (SAS) is the task of automatically scoring a given input to a prompt based on rubrics and reference answers. Although SAS is useful in real-world applications, both rubrics and reference answers differ between prompts, thus requiring a need to acquire new data and train a model for each new prompt. Such requirements are costly, especially for schools and online courses where resources are limited and only a few prompts are used. In this work, we attempt to reduce this cost through a two-phase approach: train a model on existing rubrics and answers with gold score signals and finetune it on a new prompt. Specifically, given that scoring rubrics and reference answers differ for each prompt, we utilize key phrases, or representative expressions that the answer should contain to increase scores, and train a SAS model to learn the relationship between key phrases and answers using already annotated prompts (i.e., cross-prompts). Our experimental results show that finetuning on existing cross-prompt data with key phrases significantly improves scoring accuracy, especially when the training data is limited. Finally, our extensive analysis shows that it is crucial to design the model so that it can learn the task's general property.
• 摘要：自动简答评分 (SAS) 是根据评分标准和参考答案自动对提示的给定输入进行评分的任务。尽管 SAS 在实际应用中很有用，但评分标准和参考答案在不同的提示之间有所不同，因此需要获取新数据并为每个新提示训练模型。这样的要求成本高昂，尤其是对于资源有限且仅使用少数提示的学校和在线课程而言。在这项工作中，我们尝试通过两阶段方法降低成本：使用黄金分数信号在现有评分标准和答案上训练模型，并在新提示上对其进行微调。具体而言，鉴于每个提示的评分标准和参考答案都不同，我们利用答案应包含的关键短语或代表性表达来提高分数，并使用已注释的提示（即交叉提示）训练 SAS 模型以学习关键短语和答案之间的关系。我们的实验结果表明，使用关键短语对现有的交叉提示数据进行微调可显著提高评分准确性，尤其是在训练数据有限的情况下。最后，我们的广泛分析表明，设计模型以便学习任务的一般属性至关重要。

Title: TF-Attack: Transferable and Fast Adversarial Attacks on Large Language Models

• Authors: Zelin Li, Kehai Chen, Xuefeng Bai, Lemao Liu, Mingming Yang, Yang Xiang, Min Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13985
• Pdf URL: https://arxiv.org/pdf/2408.13985
• Copy Paste: [[2408.13985]] TF-Attack: Transferable and Fast Adversarial Attacks on Large Language Models(https://arxiv.org/abs/2408.13985)
• Keywords: language model, llm
• Abstract: With the great advancements in large language models (LLMs), adversarial attacks against LLMs have recently attracted increasing attention. We found that pre-existing adversarial attack methodologies exhibit limited transferability and are notably inefficient, particularly when applied to LLMs. In this paper, we analyze the core mechanisms of previous predominant adversarial attack methods, revealing that 1) the distributions of importance score differ markedly among victim models, restricting the transferability; 2) the sequential attack processes induces substantial time overheads. Based on the above two insights, we introduce a new scheme, named TF-Attack, for Transferable and Fast adversarial attacks on LLMs. TF-Attack employs an external LLM as a third-party overseer rather than the victim model to identify critical units within sentences. Moreover, TF-Attack introduces the concept of Importance Level, which allows for parallel substitutions of attacks. We conduct extensive experiments on 6 widely adopted benchmarks, evaluating the proposed method through both automatic and human metrics. Results show that our method consistently surpasses previous methods in transferability and delivers significant speed improvements, up to 20 times faster than earlier attack strategies.
• 摘要：随着大型语言模型 (LLM) 的巨大进步，针对 LLM 的对抗性攻击最近引起了越来越多的关注。我们发现，现有的对抗性攻击方法表现出有限的可迁移性，并且效率特别低，尤其是应用于 LLM 时。在本文中，我们分析了以前主要对抗性攻击方法的核心机制，发现 1) 重要性得分的分布在受害者模型之间差异很大，限制了可迁移性；2) 顺序攻击过程会导致大量时间开销。基于以上两个见解，我们引入了一种新的方案，称为 TF-Attack，用于对 LLM 进行可迁移和快速对抗性攻击。TF-Attack 使用外部 LLM 作为第三方监督者而不是受害者模型来识别句子中的关键单元。此外，TF-Attack 引入了重要性级别的概念，允许并行替换攻击。我们对 6 个广泛采用的基准进行了广泛的实验，通过自动和人工指标评估了所提出的方法。结果表明，我们的方法在可转移性方面始终超越以前的方法，并且速度显著提高，比以前的攻击策略快 20 倍。

Title: Focused Large Language Models are Stable Many-Shot Learners

• Authors: Peiwen Yuan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Chuyi Tan, Boyuan Pan, Heda Wang, Yao Hu, Kan Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.13987
• Pdf URL: https://arxiv.org/pdf/2408.13987
• Copy Paste: [[2408.13987]] Focused Large Language Models are Stable Many-Shot Learners(https://arxiv.org/abs/2408.13987)
• Keywords: language model, llm
• Abstract: In-Context Learning (ICL) enables large language models (LLMs) to achieve rapid task adaptation by learning from demonstrations. With the increase in available context length of LLMs, recent experiments have shown that the performance of ICL does not necessarily scale well in many-shot (demonstration) settings. We theoretically and experimentally confirm that the reason lies in more demonstrations dispersing the model attention from the query, hindering its understanding of key content. Inspired by how humans learn from examples, we propose a training-free method FocusICL, which conducts triviality filtering to avoid attention being diverted by unimportant contents at token-level and operates hierarchical attention to further ensure sufficient attention towards current query at demonstration-level. We also design an efficient hyperparameter searching strategy for FocusICL based on model perplexity of demonstrations. Comprehensive experiments validate that FocusICL achieves an average performance improvement of 5.2% over vanilla ICL and scales well with many-shot demonstrations.
• 摘要：上下文学习 (ICL) 使大型语言模型 (LLM) 能够通过从演示中学习实现快速的任务适应。随着 LLM 可用上下文长度的增加，最近的实验表明，ICL 的性能在多镜头（演示）设置中不一定能很好地扩展。我们从理论和实验中证实，原因在于更多的演示分散了模型对查询的注意力，阻碍了其对关键内容的理解。受人类从示例中学习的方式的启发，我们提出了一种无需训练的方法 FocusICL，它在 token 级别进行琐碎性过滤以避免注意力被不重要的内容分散，并在演示级别操作分层注意力以进一步确保对当前查询的足够注意力。我们还基于演示的模型困惑度为 FocusICL 设计了一种高效的超参数搜索策略。全面的实验验证了 FocusICL 比 vanilla ICL 实现了 5.2% 的平均性能提升，并且在多镜头演示中具有很好的扩展性。

Title: Enhancing Depression Diagnosis with Chain-of-Thought Prompting

• Authors: Elysia Shi, Adithri Manda, London Chowdhury, Runeema Arun, Kevin Zhu, Michael Lam
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.14053
• Pdf URL: https://arxiv.org/pdf/2408.14053
• Copy Paste: [[2408.14053]] Enhancing Depression Diagnosis with Chain-of-Thought Prompting(https://arxiv.org/abs/2408.14053)
• Keywords: prompt, chain-of-thought
• Abstract: When using AI to detect signs of depressive disorder, AI models habitually draw preemptive conclusions. We theorize that using chain-of-thought (CoT) prompting to evaluate Patient Health Questionnaire-8 (PHQ-8) scores will improve the accuracy of the scores determined by AI models. In our findings, when the models reasoned with CoT, the estimated PHQ-8 scores were consistently closer on average to the accepted true scores reported by each participant compared to when not using CoT. Our goal is to expand upon AI models' understanding of the intricacies of human conversation, allowing them to more effectively assess a patient's feelings and tone, therefore being able to more accurately discern mental disorder symptoms; ultimately, we hope to augment AI models' abilities, so that they can be widely accessible and used in the medical field.
• 摘要：在使用 AI 检测抑郁症迹象时，AI 模型通常会得出先发制人的结论。我们认为，使用思路链 (CoT) 提示来评估患者健康问卷 8 (PHQ-8) 分数将提高 AI 模型确定分数的准确性。在我们的研究结果中，当模型使用 CoT 进行推理时，与不使用 CoT 时相比，估计的 PHQ-8 分数平均始终更接近每个参与者报告的可接受真实分数。我们的目标是扩展 AI 模型对人类对话复杂性的理解，使它们能够更有效地评估患者的感受和语气，从而能够更准确地辨别精神障碍症状；最终，我们希望增强 AI 模型的能力，使它们能够在医学领域得到广泛使用。

Title: Investigating the effect of Mental Models in User Interaction with an Adaptive Dialog Agent

• Authors: Lindsey Vanderlyn, Dirk Väth, Ngoc Thang Vu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.14154
• Pdf URL: https://arxiv.org/pdf/2408.14154
• Copy Paste: [[2408.14154]] Investigating the effect of Mental Models in User Interaction with an Adaptive Dialog Agent(https://arxiv.org/abs/2408.14154)
• Keywords: agent
• Abstract: Mental models play an important role in whether user interaction with intelligent systems, such as dialog systems is successful or not. Adaptive dialog systems present the opportunity to align a dialog agent's behavior with heterogeneous user expectations. However, there has been little research into what mental models users form when interacting with a task-oriented dialog system, how these models affect users' interactions, or what role system adaptation can play in this process, making it challenging to avoid damage to human-AI partnership. In this work, we collect a new publicly available dataset for exploring user mental models about information seeking dialog systems. We demonstrate that users have a variety of conflicting mental models about such systems, the validity of which directly impacts the success of their interactions and perceived usability of system. Furthermore, we show that adapting a dialog agent's behavior to better align with users' mental models, even when done implicitly, can improve perceived usability, dialog efficiency, and success. To this end, we argue that implicit adaptation can be a valid strategy for task-oriented dialog systems, so long as developers first have a solid understanding of users' mental models.
• 摘要：心智模型在用户与智能系统（如对话系统）的交互是否成功方面起着重要作用。自适应对话系统提供了将对话代理的行为与不同用户期望相一致的机会。然而，很少有研究涉及用户在与面向任务的对话系统交互时会形成什么样的心智模型、这些模型如何影响用户的交互，或者系统适应性在此过程中可以发挥什么作用，这使得避免损害人机伙伴关系变得具有挑战性。在这项工作中，我们收集了一个新的公开数据集，用于探索用户关于信息搜索对话系统的心智模型。我们证明，用户对此类系统有各种相互冲突的心智模型，这些模型的有效性直接影响他们交互的成功和系统的感知可用性。此外，我们表明，调整对话代理的行为以更好地与用户的心智模型保持一致，即使是隐式地进行，也可以提高感知可用性、对话效率和成功率。为此，我们认为，只要开发人员首先对用户的心智模型有扎实的了解，隐式适应就可以成为面向任务的对话系统的有效策略。

Title: DSTI at LLMs4OL 2024 Task A: Intrinsic versus extrinsic knowledge for type classification

• Authors: Hanna Abi Akl
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.14236
• Pdf URL: https://arxiv.org/pdf/2408.14236
• Copy Paste: [[2408.14236]] DSTI at LLMs4OL 2024 Task A: Intrinsic versus extrinsic knowledge for type classification(https://arxiv.org/abs/2408.14236)
• Keywords: language model, llm
• Abstract: We introduce semantic towers, an extrinsic knowledge representation method, and compare it to intrinsic knowledge in large language models for ontology learning. Our experiments show a trade-off between performance and semantic grounding for extrinsic knowledge compared to a fine-tuned model intrinsic knowledge. We report our findings on the Large Language Models for Ontology Learning (LLMs4OL) 2024 challenge.
• 摘要：我们引入了语义塔，一种外部知识表示方法，并将其与用于本体学习的大型语言模型中的内在知识进行比较。我们的实验表明，与微调模型内在知识相比，外部知识的性能和语义基础之间存在权衡。我们报告了我们在 2024 年用于本体学习的大型语言模型 (LLMs4OL) 挑战赛上的发现。

Title: Predictability and Causality in Spanish and English Natural Language Generation

• Authors: Andrea Busto-Castiñeira, Francisco J. González-Castaño, Silvia García-Méndez, Francisco de Arriba-Pérez
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.14283
• Pdf URL: https://arxiv.org/pdf/2408.14283
• Copy Paste: [[2408.14283]] Predictability and Causality in Spanish and English Natural Language Generation(https://arxiv.org/abs/2408.14283)
• Keywords: language model
• Abstract: In recent years, the field of Natural Language Generation (NLG) has been boosted by the recent advances in deep learning technologies. Nonetheless, these new data-intensive methods introduce language-dependent disparities in NLG as the main training data sets are in English. Also, most neural NLG systems use decoder-only (causal) transformer language models, which work well for English, but were not designed with other languages in mind. In this work we depart from the hypothesis that they may introduce generation bias in target languages with less rigid word ordering, subject omission, or different attachment preferences for relative clauses, so that for these target languages other language generation strategies may be more desirable. This paper first compares causal and non-causal language modeling for English and Spanish, two languages with different grammatical structures and over 1.5 billion and 0.5 billion speakers, respectively. For this purpose, we define a novel metric of average causal and non-causal context-conditioned entropy of the grammatical category distribution for both languages as an information-theoretic a priori approach. The evaluation of natural text sources (such as training data) in both languages reveals lower average non-causal conditional entropy in Spanish and lower causal conditional entropy in English. According to this experiment, Spanish is more predictable than English given a non-causal context. Then, by applying a conditional relative entropy metric to text generation experiments, we obtain as insights that the best performance is respectively achieved with causal NLG in English, and with non-causal NLG in Spanish. These insights support further research in NLG in Spanish using bidirectional transformer language models.
• 摘要：近年来，深度学习技术的最新进展推动了自然语言生成 (NLG) 领域的发展。尽管如此，这些新的数据密集型方法在 NLG 中引入了语言相关的差异，因为主要的训练数据集是英语。此外，大多数神经 NLG 系统使用仅解码器（因果）转换器语言模型，这些模型对英语效果很好，但在设计时并未考虑其他语言。在这项工作中，我们脱离了这样的假设：它们可能会在目标语言中引入生成偏差，包括不太严格的词序、主语省略或对关系从句的不同附着偏好，因此对于这些目标语言，其他语言生成策略可能更可取。本文首先比较了英语和西班牙语的因果和非因果语言建模，这两种语言的语法结构不同，使用者分别超过 15 亿和 5 亿。为此，我们定义了一种新的度量标准，即两种语言的语法类别分布的平均因果和非因果上下文条件熵，作为一种信息论先验方法。对两种语言的自然文本源（例如训练数据）的评估表明，西班牙语的平均非因果条件熵较低，英语的因果条件熵较低。根据这项实验，在非因果背景下，西班牙语比英语更容易预测。然后，通过将条件相对熵度量应用于文本生成实验，我们得出的结论是，英语的因果 NLG 和非因果 NLG 分别实现了最佳性能。这些见解支持使用双向转换器语言模型进一步研究西班牙语的 NLG。

Title: LLM-3D Print: Large Language Models To Monitor and Control 3D Printing

• Authors: Yayati Jadhav, Peter Pak, Amir Barati Farimani
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.14307
• Pdf URL: https://arxiv.org/pdf/2408.14307
• Copy Paste: [[2408.14307]] LLM-3D Print: Large Language Models To Monitor and Control 3D Printing(https://arxiv.org/abs/2408.14307)
• Keywords: language model, llm, agent
• Abstract: Industry 4.0 has revolutionized manufacturing by driving digitalization and shifting the paradigm toward additive manufacturing (AM). Fused Deposition Modeling (FDM), a key AM technology, enables the creation of highly customized, cost-effective products with minimal material waste through layer-by-layer extrusion, posing a significant challenge to traditional subtractive methods. However, the susceptibility of material extrusion techniques to errors often requires expert intervention to detect and mitigate defects that can severely compromise product quality. While automated error detection and machine learning models exist, their generalizability across diverse 3D printer setups, firmware, and sensors is limited, and deep learning methods require extensive labeled datasets, hindering scalability and adaptability. To address these challenges, we present a process monitoring and control framework that leverages pre-trained Large Language Models (LLMs) alongside 3D printers to detect and address printing defects. The LLM evaluates print quality by analyzing images captured after each layer or print segment, identifying failure modes and querying the printer for relevant parameters. It then generates and executes a corrective action plan. We validated the effectiveness of the proposed framework in identifying defects by comparing it against a control group of engineers with diverse AM expertise. Our evaluation demonstrated that LLM-based agents not only accurately identify common 3D printing errors, such as inconsistent extrusion, stringing, warping, and layer adhesion, but also effectively determine the parameters causing these failures and autonomously correct them without any need for human intervention.
• 摘要：工业 4.0 通过推动数字化和将范式转向增材制造 (AM) 彻底改变了制造业。熔融沉积成型 (FDM) 是一种关键的 AM 技术，它能够通过逐层挤压以最少的材料浪费创建高度定制、经济高效的产品，这对传统的减材方法提出了重大挑战。然而，材料挤压技术容易出错，通常需要专家干预才能检测和缓解可能严重损害产品质量的缺陷。虽然存在自动错误检测和机器学习模型，但它们在各种 3D 打印机设置、固件和传感器中的通用性有限，而深度学习方法需要大量标记数据集，从而阻碍了可扩展性和适应性。为了应对这些挑战，我们提出了一个过程监控和控制框架，该框架利用预先训练的大型语言模型 (LLM) 和 3D 打印机来检测和解决打印缺陷。LLM 通过分析每层或打印段后捕获的图​​像、识别故障模式并向打印机查询相关参数来评估打印质量。然后它生成并执行纠正措施计划。我们通过将所提出的框架与具有不同 AM 专业知识的工程师对照组进行比较，验证了该框架在识别缺陷方面的有效性。我们的评估表明，基于 LLM 的代理不仅可以准确识别常见的 3D 打印错误，例如挤压不一致、拉丝、翘曲和层粘附，还可以有效确定导致这些故障的参数并自动纠正它们，而无需任何人工干预。

Title: Claim Verification in the Age of Large Language Models: A Survey

• Authors: Alphaeus Dmonte, Roland Oruche, Marcos Zampieri, Prasad Calyam, Isabelle Augenstein
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.14317
• Pdf URL: https://arxiv.org/pdf/2408.14317
• Copy Paste: [[2408.14317]] Claim Verification in the Age of Large Language Models: A Survey(https://arxiv.org/abs/2408.14317)
• Keywords: language model, llm, prompt, retrieval augmented generation
• Abstract: The large and ever-increasing amount of data available on the Internet coupled with the laborious task of manual claim and fact verification has sparked the interest in the development of automated claim verification systems. Several deep learning and transformer-based models have been proposed for this task over the years. With the introduction of Large Language Models (LLMs) and their superior performance in several NLP tasks, we have seen a surge of LLM-based approaches to claim verification along with the use of novel methods such as Retrieval Augmented Generation (RAG). In this survey, we present a comprehensive account of recent claim verification frameworks using LLMs. We describe the different components of the claim verification pipeline used in these frameworks in detail including common approaches to retrieval, prompting, and fine-tuning. Finally, we describe publicly available English datasets created for this task.
• 摘要：互联网上可用的数据量巨大且不断增加，再加上手动声明和事实验证的繁重任务，引发了人们对开发自动化声明验证系统的兴趣。多年来，已经为这项任务提出了几种基于深度学习和变压器的模型。随着大型语言模型 (LLM) 的引入及其在多项 NLP 任务中的出色表现，我们看到基于 LLM 的声明验证方法激​​增，同时还看到了检索增强生成 (RAG) 等新方法的使用。在这篇综述中，我们全面介绍了最近使用 LLM 的声明验证框架。我们详细描述了这些框架中使用的声明验证流程的不同组件，包括常见的检索、提示和微调方法。最后，我们描述了为此任务创建的公开可用的英语数据集。

Title: Assessing Contamination in Large Language Models: Introducing the LogProber method

• Authors: Nicolas Yax, Pierre-Yves Oudeyer, Stefano Palminteri
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.14352
• Pdf URL: https://arxiv.org/pdf/2408.14352
• Copy Paste: [[2408.14352]] Assessing Contamination in Large Language Models: Introducing the LogProber method(https://arxiv.org/abs/2408.14352)
• Keywords: language model, llm
• Abstract: In machine learning, contamination refers to situations where testing data leak into the training set. The issue is particularly relevant for the evaluation of the performance of Large Language Models (LLMs), which are generally trained on gargantuan, and generally opaque, corpora of text scraped from the world wide web. Developing tools to detect contamination is therefore crucial to be able to fairly and properly track the evolution of the performance of LLMs. Most recent works in the field are not tailored to quantify contamination on short sequences of text like we find in psychology questionnaires. In the present paper we introduce LogProber, a novel, efficient, algorithm that we show able to detect contamination using token probability in given sentences. In the second part we investigate the limitations of the method and discuss how different training methods can contaminate models without leaving traces in the token probabilities.
• 摘要：在机器学习中，污染是指测试数据泄漏到训练集中的情况。这个问题对于大型语言模型 (LLM) 的性能评估尤其重要，这些模型通常是在从万维网上抓取的庞大且通常不透明的文本语料库上进行训练的。因此，开发检测污染的工具对于公平正确地跟踪 LLM 性能的演变至关重要。该领域最近的大多数研究都不是专门针对量化短文本序列的污染（如我们在心理学问卷中发现的）。在本文中，我们介绍了 LogProber，这是一种新颖、高效的算法，我们展示了它能够使用给定句子中的标记概率来检测污染。在第二部分中，我们研究了该方法的局限性，并讨论了不同的训练方法如何污染模型而不在标记概率中留下痕迹。

Title: Probing Causality Manipulation of Large Language Models

• Authors: Chenyang Zhang, Haibo Tong, Bin Zhang, Dongyu Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.14380
• Pdf URL: https://arxiv.org/pdf/2408.14380
• Copy Paste: [[2408.14380]] Probing Causality Manipulation of Large Language Models(https://arxiv.org/abs/2408.14380)
• Keywords: language model, gpt, llm, retrieval augmented generation
• Abstract: Large language models (LLMs) have shown various ability on natural language processing, including problems about causality. It is not intuitive for LLMs to command causality, since pretrained models usually work on statistical associations, and do not focus on causes and effects in sentences. So that probing internal manipulation of causality is necessary for LLMs. This paper proposes a novel approach to probe causality manipulation hierarchically, by providing different shortcuts to models and observe behaviors. We exploit retrieval augmented generation (RAG) and in-context learning (ICL) for models on a designed causality classification task. We conduct experiments on mainstream LLMs, including GPT-4 and some smaller and domain-specific models. Our results suggest that LLMs can detect entities related to causality and recognize direct causal relationships. However, LLMs lack specialized cognition for causality, merely treating them as part of the global semantic of the sentence.
• 摘要：大型语言模型 (LLM) 在自然语言处理方面表现出了多种能力，包括因果关系问题。LLM 无法直观地控制因果关系，因为预训练模型通常处理统计关联，而不关注句子中的因果关系。因此，探究因果关系的内部操纵对于 LLM 来说是必要的。本文提出了一种新方法，通过为模型提供不同的快捷方式并观察行为，来分层探测因果关系操纵。我们在设计的因果关系分类任务中利用模型的检索增强生成 (RAG) 和上下文学习 (ICL)。我们对主流 LLM 进行了实验，包括 GPT-4 和一些较小且特定于领域的模型。我们的结果表明，LLM 可以检测与因果关系相关的实体并识别直接的因果关系。然而，LLM 缺乏对因果关系的专门认知，仅仅将它们视为句子全局语义的一部分。

Title: Language-specific Calibration for Pruning Multilingual Language Models

• Authors: Simon Kurz, Zhixue Zhao, Jian-Jia Chen, Lucie Flek
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.14398
• Pdf URL: https://arxiv.org/pdf/2408.14398
• Copy Paste: [[2408.14398]] Language-specific Calibration for Pruning Multilingual Language Models(https://arxiv.org/abs/2408.14398)
• Keywords: language model, llm
• Abstract: Recent advances in large language model (LLM) pruning have shown state-of-the-art compression results in post-training and retraining-free settings while maintaining high predictive performance. However, such research mainly considers calibrating pruning using English text, despite the multilingual nature of modern LLMs and their frequent uses in non-English languages. In this paper, we set out to explore effective strategies for calibrating the pruning of multilingual language models. We present the first comprehensive empirical study, comparing different calibration languages for pruning multilingual models across diverse tasks, models, and state-of-the-art pruning techniques. Our results present practical suggestions, for example, calibrating in the target language can efficiently yield lower perplexity, but does not necessarily benefit downstream tasks. Our further analysis experiments unveil that calibration in the target language mainly contributes to preserving language-specific features related to fluency and coherence, but might not contribute to capturing language-agnostic features such as language understanding and reasoning. Last, we provide practical recommendations for future practitioners.
• 摘要：大型语言模型 (LLM) 修剪的最新进展已在训练后和无需再训练的设置中显示出最先进的压缩结果，同时保持了较高的预测性能。然而，尽管现代 LLM 具有多语言性质并且经常用于非英语语言，但此类研究主要考虑使用英语文本校准修剪。在本文中，我们着手探索校准多语言语言模型修剪的有效策略。我们提出了第一项全面的实证研究，比较了不同校准语言在不同任务、模型和最先进的修剪技术中修剪多语言模型的效果。我们的结果提出了实用的建议，例如，在目标语言中进行校准可以有效地降低困惑度，但不一定对下游任务有益。我们进一步的分析实验表明，在目标语言中进行校准主要有助于保留与流利度和连贯性相关的语言特定特征，但可能无助于捕捉语言理解和推理等与语言无关的特征。最后，我们为未来的从业者提供了实用的建议。

Title: MEDSAGE: Enhancing Robustness of Medical Dialogue Summarization to ASR Errors with LLM-generated Synthetic Dialogues

• Authors: Kuluhan Binici, Abhinav Ramesh Kashyap, Viktor Schlegel, Andy T. Liu, Vijay Prakash Dwivedi, Thanh-Tung Nguyen, Xiaoxue Gao, Nancy F. Chen, Stefan Winkler
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.14418
• Pdf URL: https://arxiv.org/pdf/2408.14418
• Copy Paste: [[2408.14418]] MEDSAGE: Enhancing Robustness of Medical Dialogue Summarization to ASR Errors with LLM-generated Synthetic Dialogues(https://arxiv.org/abs/2408.14418)
• Keywords: language model, llm
• Abstract: Automatic Speech Recognition (ASR) systems are pivotal in transcribing speech into text, yet the errors they introduce can significantly degrade the performance of downstream tasks like summarization. This issue is particularly pronounced in clinical dialogue summarization, a low-resource domain where supervised data for fine-tuning is scarce, necessitating the use of ASR models as black-box solutions. Employing conventional data augmentation for enhancing the noise robustness of summarization models is not feasible either due to the unavailability of sufficient medical dialogue audio recordings and corresponding ASR transcripts. To address this challenge, we propose MEDSAGE, an approach for generating synthetic samples for data augmentation using Large Language Models (LLMs). Specifically, we leverage the in-context learning capabilities of LLMs and instruct them to generate ASR-like errors based on a few available medical dialogue examples with audio recordings. Experimental results show that LLMs can effectively model ASR noise, and incorporating this noisy data into the training process significantly improves the robustness and accuracy of medical dialogue summarization systems. This approach addresses the challenges of noisy ASR outputs in critical applications, offering a robust solution to enhance the reliability of clinical dialogue summarization.
• 摘要：自动语音识别 (ASR) 系统在将语音转录为文本方面发挥着关键作用，但它们引入的错误会严重降低下游任务（如摘要）的性能。这个问题在临床对话摘要中尤为明显，这是一个资源匮乏的领域，用于微调的监督数据稀缺，因此必须使用 ASR 模型作为黑盒解决方案。由于缺乏足够的医学对话录音和相应的 ASR 转录本，使用传统的数据增强来增强摘要模型的噪声鲁棒性也是不可行的。为了应对这一挑战，我们提出了 MEDSAGE，这是一种使用大型语言模型 (LLM) 生成数据增强的合成样本的方法。具体来说，我们利用 LLM 的上下文学习功能，并指示它们根据一些可用的带有录音的医学对话示例生成类似 ASR 的错误。实验结果表明，LLM 可以有效地模拟 ASR 噪声，将这些噪声数据纳入训练过程可显著提高医学对话摘要系统的鲁棒性和准确性。这种方法解决了关键应用中嘈杂的 ASR 输出的挑战，提供了增强临床对话摘要可靠性的强大解决方案。

Title: Evaluating Large Language Models on Spatial Tasks: A Multi-Task Benchmarking Study

• Authors: Liuchang Xu Shuo Zhao, Qingming Lin, Luyao Chen, Qianqian Luo, Sensen Wu, Xinyue Ye, Hailin Feng, Zhenhong Du
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.14438
• Pdf URL: https://arxiv.org/pdf/2408.14438
• Copy Paste: [[2408.14438]] Evaluating Large Language Models on Spatial Tasks: A Multi-Task Benchmarking Study(https://arxiv.org/abs/2408.14438)
• Keywords: language model, gpt, prompt, chat, chain-of-thought
• Abstract: The advent of large language models such as ChatGPT, Gemini, and others has underscored the importance of evaluating their diverse capabilities, ranging from natural language understanding to code generation. However, their performance on spatial tasks has not been comprehensively assessed. This study addresses this gap by introducing a novel multi-task spatial evaluation dataset, designed to systematically explore and compare the performance of several advanced models on spatial tasks. The dataset encompasses twelve distinct task types, including spatial understanding and path planning, each with verified, accurate answers. We evaluated multiple models, including OpenAI's gpt-3.5-turbo, gpt-4o, and ZhipuAI's glm-4, through a two-phase testing approach. Initially, we conducted zero-shot testing, followed by categorizing the dataset by difficulty and performing prompt tuning tests. Results indicate that gpt-4o achieved the highest overall accuracy in the first phase, with an average of 71.3%. Although moonshot-v1-8k slightly underperformed overall, it surpassed gpt-4o in place name recognition tasks. The study also highlights the impact of prompt strategies on model performance in specific tasks. For example, the Chain-of-Thought (COT) strategy increased gpt-4o's accuracy in path planning from 12.4% to 87.5%, while a one-shot strategy enhanced moonshot-v1-8k's accuracy in mapping tasks from 10.1% to 76.3%.
• 摘要：ChatGPT、Gemini 等大型语言模型的出现凸显了评估其从自然语言理解到代码生成的各种能力的重要性。然而，它们在空间任务上的表现尚未得到全面评估。本研究通过引入一种新颖的多任务空间评估数据集来解决这一差距，该数据集旨在系统地探索和比较几种高级模型在空间任务上的表现。该数据集包含十二种不同的任务类型，包括空间理解和路径规划，每种任务类型都有经过验证的准确答案。我们通过两阶段测试方法评估了多个模型，包括 OpenAI 的 gpt-3.5-turbo、gpt-4o 和 ZhipuAI 的 glm-4。首先，我们进行了零样本测试，然后按难度对数据集进行分类并执行即时调整测试。结果表明，gpt-4o 在第一阶段实现了最高的总体准确率，平均为 71.3%。虽然 moonshot-v1-8k 的整体表现略差，但它在地名识别任务中超越了 gpt-4o。研究还重点介绍了提示策略对模型在具体任务中表现的影响。例如，思维链 (COT) 策略将 GPT-4O 的路径规划准确率从 12.4% 提高到了 87.5%，而一次性策略将 Moonshot-v1-8K 的绘图任务准确率从 10.1% 提升到了 76.3%。

Title: Explicit Inductive Inference using Large Language Models

• Authors: Tianyang Liu, Tianyi Li, Liang Cheng, Mark Steedman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.14467
• Pdf URL: https://arxiv.org/pdf/2408.14467
• Copy Paste: [[2408.14467]] Explicit Inductive Inference using Large Language Models(https://arxiv.org/abs/2408.14467)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are reported to hold undesirable attestation bias on inference tasks: when asked to predict if a premise P entails a hypothesis H, instead of considering H's conditional truthfulness entailed by P, LLMs tend to use the out-of-context truth label of H as a fragile proxy. In this paper, we propose a pipeline that exploits this bias to do explicit inductive inference. Our pipeline uses an LLM to transform a premise into a set of attested alternatives, and then aggregate answers of the derived new entailment inquiries to support the original inference prediction. On a directional predicate entailment benchmark, we demonstrate that by applying this simple pipeline, we can improve the overall performance of LLMs on inference and substantially alleviate the impact of their attestation bias.
• 摘要：据报道，大型语言模型 (LLM) 在推理任务中存在不良的证明偏差：当被要求预测前提 P 是否蕴含假设 H 时，LLM 倾向于使用 H 的脱离上下文的真值标签作为脆弱代理，而不是考虑 P 蕴含的 H 的条件真实性。在本文中，我们提出了一种利用这种偏差进行显式归纳推理的管道。我们的管道使用 LLM 将前提转换为一组经过证明的替代方案，然后汇总派生的新蕴含查询的答案以支持原始推理预测。在定向谓词蕴含基准上，我们证明通过应用这个简单的管道，我们可以提高 LLM 在推理方面的整体性能，并大大减轻其证明偏差的影响。

Title: Step-by-Step Unmasking for Parameter-Efficient Fine-tuning of Large Language Models

• Authors: Aradhye Agarwal, Suhas K Ramesh, Ayan Sengupta, Tanmoy Chakraborty
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.14470
• Pdf URL: https://arxiv.org/pdf/2408.14470
• Copy Paste: [[2408.14470]] Step-by-Step Unmasking for Parameter-Efficient Fine-tuning of Large Language Models(https://arxiv.org/abs/2408.14470)
• Keywords: language model, llm
• Abstract: Fine-tuning large language models (LLMs) on downstream tasks requires substantial computational resources. A class of parameter-efficient fine-tuning (PEFT) aims to mitigate these computational challenges by selectively fine-tuning only a small fraction of the model parameters. Although computationally efficient, these techniques often fail to match the performance of fully fine-tuned models, primarily due to inherent biases introduced during parameter selection. Traditional selective PEFT techniques use a fixed set of parameters based on a predefined budget (a process also known as unmasking), failing to capture parameter importance dynamically and often ending up exceeding the budget. We introduce $\text{ID}^3$, a novel selective PEFT method that calculates parameter importance continually and dynamically unmasks parameters by balancing exploration and exploitation in parameter selection. Our empirical study on 15 tasks spanning natural language understanding and generative tasks demonstrates the effectiveness of our method compared to fixed-masking-based PEFT techniques. We analytically show that $\text{ID}^3$ reduces the number of gradient updates by a factor of two, enhancing computational efficiency. $\text{ID}^3$ is robust to random initialization of neurons and, therefore, can be seamlessly integrated into existing additive and reparametrization-based PEFT modules such as adapters and LoRA for dynamic sparsification.
• 摘要：在下游任务上微调大型语言模型 (LLM) 需要大量计算资源。一类参数高效微调 (PEFT) 旨在通过选择性地仅微调一小部分模型参数来缓解这些计算挑战。尽管这些技术在计算上是高效的，但它们通常无法与完全微调模型的性能相匹配，这主要是由于在参数选择过程中引入了固有偏差。传统的选择性 PEFT 技术使用基于预定义预算的一组固定参数（该过程也称为揭开掩蔽），无法动态捕获参数重要性，并且经常最终超出预算。我们引入了一种新颖的选择性 PEFT 方法 $\text{ID}^3$，它通过平衡参数选择中的探索和利用来持续计算参数重要性并动态揭开参数的掩蔽。我们对 15 项涵盖自然语言理解和生成任务的任务进行的实证研究表明，与基于固定掩蔽的 PEFT 技术相比，我们的方法更有效。我们通过分析表明，$\text{ID}^3$ 将梯度更新次数减少了一半，从而提高了计算效率。 $\text{ID}^3$ 对于神经元的随机初始化具有很强的鲁棒性，因此可以无缝集成到现有的基于加法和重新参数化的 PEFT 模块（如适配器和 LoRA）中，以实现动态稀疏化。