2024-08-23

Title: The Mechanics of Conceptual Interpretation in GPT Models: Interpretative Insights

• Authors: Nura Aljaafari, Danilo S. Carvalho, André Freitas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11827
• Pdf URL: https://arxiv.org/pdf/2408.11827
• Copy Paste: [[2408.11827]] The Mechanics of Conceptual Interpretation in GPT Models: Interpretative Insights(https://arxiv.org/abs/2408.11827)
• Keywords: language model, gpt, llm
• Abstract: Locating and editing knowledge in large language models (LLMs) is crucial for enhancing their accuracy, safety, and inference rationale. We introduce ``concept editing'', an innovative variation of knowledge editing that uncovers conceptualisation mechanisms within these models. Using the reverse dictionary task, inference tracing, and input abstraction, we analyse the Multi-Layer Perceptron (MLP), Multi-Head Attention (MHA), and hidden state components of transformer models. Our results reveal distinct patterns: MLP layers employ key-value retrieval mechanism and context-dependent processing, which are highly associated with relative input tokens. MHA layers demonstrate a distributed nature with significant higher-level activations, suggesting sophisticated semantic integration. Hidden states emphasise the importance of the last token and top layers in the inference process. We observe evidence of gradual information building and distributed representation. These observations elucidate how transformer models process semantic information, paving the way for targeted interventions and improved interpretability techniques. Our work highlights the complex, layered nature of semantic processing in LLMs and the challenges of isolating and modifying specific concepts within these models.
• 摘要：在大型语言模型 (LLM) 中定位和编辑知识对于提高其准确性、安全性和推理原理至关重要。我们引入了“概念编辑”，这是知识编辑的一种创新变体，它揭示了这些模型中的概念化机制。使用反向词典任务、推理跟踪和输入抽象，我们分析了 Transformer 模型的多层感知器 (MLP)、多头注意力 (MHA) 和隐藏状态组件。我们的结果揭示了不同的模式：MLP 层采用键值检索机制和上下文相关处理，这些处理与相对输入标记高度相关。MHA 层表现出分布式特性，具有显着的高级激活，表明了复杂的语义集成。隐藏状态强调了最后一个标记和顶层在推理过程中的重要性。我们观察到逐步信息构建和分布式表示的证据。这些观察阐明了 Transformer 模型如何处理语义信息，为有针对性的干预和改进的可解释性技术铺平了道路。我们的工作强调了 LLM 中语义处理的复杂性、分层性，以及在这些模型中隔离和修改特定概念的挑战。

Title: OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs

• Authors: Hasan Iqbal, Yuxia Wang, Minghan Wang, Georgi Georgiev, Jiahui Geng, Iryna Gurevych, Preslav Nakov
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11832
• Pdf URL: https://arxiv.org/pdf/2408.11832
• Copy Paste: [[2408.11832]] OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs(https://arxiv.org/abs/2408.11832)
• Keywords: language model, llm
• Abstract: The increased use of large language models (LLMs) across a variety of real-world applications calls for automatic tools to check the factual accuracy of their outputs, as LLMs often hallucinate. This is difficult as it requires assessing the factuality of free-form open-domain responses. While there has been a lot of research on this topic, different papers use different evaluation benchmarks and measures, which makes them hard to compare and hampers future progress. To mitigate these issues, we developed OpenFactCheck, a unified framework, with three modules: (i) RESPONSEEVAL, which allows users to easily customize an automatic fact-checking system and to assess the factuality of all claims in an input document using that system, (ii) LLMEVAL, which assesses the overall factuality of an LLM, and (iii) CHECKEREVAL, a module to evaluate automatic fact-checking systems. OpenFactCheck is open-sourced (this https URL) and publicly released as a Python library (this https URL) and also as a web service (this https URL). A video describing the system is available at this https URL.
• 摘要：大型语言模型 (LLM) 在各种实际应用中的使用越来越多，因此需要使用自动工具来检查其输出的事实准确性，因为 LLM 经常会产生幻觉。这很困难，因为它需要评估自由格式开放域响应的真实性。虽然已经有很多关于这个主题的研究，但不同的论文使用不同的评估基准和措施，这使得它们很难进行比较并阻碍了未来的进展。为了缓解这些问题，我们开发了一个统一的框架 OpenFactCheck，它有三个模块：(i) RESPONSEEVAL，它允许用户轻松定制自动事实核查系统并使用该系统评估输入文档中所有声明的真实性，(ii) LLMEVAL，它评估 LLM 的整体真实性，以及 (iii) CHECKEREVAL，一个用于评估自动事实核查系统的模块。OpenFactCheck 是开源的（此 https URL），并作为 Python 库（此 https URL）和 Web 服务（此 https URL）公开发布。您可以在此 https URL 上观看描述该系统的视频。

Title: Editable Fairness: Fine-Grained Bias Mitigation in Language Models

• Authors: Ruizhe Chen, Yichen Li, Jianfei Yang, Joey Tianyi Zhou, Zuozhu Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11843
• Pdf URL: https://arxiv.org/pdf/2408.11843
• Copy Paste: [[2408.11843]] Editable Fairness: Fine-Grained Bias Mitigation in Language Models(https://arxiv.org/abs/2408.11843)
• Keywords: language model, llm
• Abstract: Generating fair and accurate predictions plays a pivotal role in deploying large language models (LLMs) in the real world. However, existing debiasing methods inevitably generate unfair or incorrect predictions as they are designed and evaluated to achieve parity across different social groups but leave aside individual commonsense facts, resulting in modified knowledge that elicits unreasonable or undesired predictions. In this paper, we first establish a new bias mitigation benchmark, BiaScope, which systematically assesses performance by leveraging newly constructed datasets and metrics on knowledge retention and generalization. Then, we propose a novel debiasing approach, Fairness Stamp (FAST), which enables fine-grained calibration of individual social biases. FAST identifies the decisive layer responsible for storing social biases and then calibrates its outputs by integrating a small modular network, considering both bias mitigation and knowledge-preserving demands. Comprehensive experiments demonstrate that FAST surpasses state-of-the-art baselines with superior debiasing performance while not compromising the overall model capability for knowledge retention and downstream predictions. This highlights the potential of fine-grained debiasing strategies to achieve fairness in LLMs. Code will be publicly available.
• 摘要：生成公平准确的预测对于在现实世界中部署大型语言模型 (LLM) 起着关键作用。然而，现有的去偏方法不可避免地会产生不公平或不正确的预测，因为它们的设计和评估是为了实现不同社会群体之间的平等，但却忽略了个人的常识事实，导致修改后的知识引发不合理或不受欢迎的预测。在本文中，我们首先建立了一个新的偏差缓解基准 BiaScope，它利用新构建的数据集和知识保留和泛化指标来系统地评估性能。然后，我们提出了一种新颖的去偏方法 Fairness Stamp (FAST)，它可以对单个社会偏见进行细粒度校准。FAST 确定负责存储社会偏见的决定层，然后通过集成一个小型模块化网络来校准其输出，同时考虑偏差缓解和知识保存需求。全面的实验表明，FAST 以卓越的去偏性能超越了最先进的基线，同时不损害模型的整体知识保留和下游预测能力。这凸显了细粒度去偏策略在 LLM 中实现公平性的潜力。代码将公开发布。

Title: LLaMA based Punctuation Restoration With Forward Pass Only Decoding

• Authors: Yutong Pang, Debjyoti Paul, Kevin Jiang, Xuedong Zhang, Xin Lei
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11845
• Pdf URL: https://arxiv.org/pdf/2408.11845
• Copy Paste: [[2408.11845]] LLaMA based Punctuation Restoration With Forward Pass Only Decoding(https://arxiv.org/abs/2408.11845)
• Keywords: language model, hallucination
• Abstract: This paper introduces two advancements in the field of Large Language Model Annotation with a focus on punctuation restoration tasks. Our first contribution is the application of LLaMA for punctuation restoration, which demonstrates superior performance compared to the established benchmark. Despite its impressive quality, LLaMA faces challenges regarding inference speed and hallucinations. To address this, our second contribution presents Forward Pass Only Decoding (FPOD), a novel decoding approach for annotation tasks. This innovative method results in a substantial 19.8x improvement in inference speed, effectively addressing a critical bottleneck and enhancing the practical utility of LLaMA for large-scale data annotation tasks without hallucinations. The combination of these contributions not only solidifies LLaMA as a powerful tool for punctuation restoration but also highlights FPOD as a crucial strategy for overcoming speed constraints.
• 摘要：本文介绍了大型语言模型注释领域的两项进展，重点介绍了标点符号恢复任务。我们的第一个贡献是将 LLaMA 应用于标点符号恢复，与既定基准相比，它表现出了卓越的性能。尽管质量令人印象深刻，但 LLaMA 仍面临着推理速度和幻觉方面的挑战。为了解决这个问题，我们的第二个贡献提出了仅前向传递解码 (FPOD)，这是一种用于注释任务的新型解码方法。这种创新方法将推理速度提高了 19.8 倍，有效地解决了关键瓶颈，并增强了 LLaMA 在无幻觉的大规模数据注释任务中的实用性。这些贡献的结合不仅巩固了 LLaMA 作为标点符号恢复的强大工具的地位，而且还突出了 FPOD 作为克服速度限制的关键策略。

Title: Density Matrices for Metaphor Understanding

• Authors: Jay Owers, Ekaterina Shutova, Martha Lewis
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11846
• Pdf URL: https://arxiv.org/pdf/2408.11846
• Copy Paste: [[2408.11846]] Density Matrices for Metaphor Understanding(https://arxiv.org/abs/2408.11846)
• Keywords: language model
• Abstract: In physics, density matrices are used to represent mixed states, i.e. probabilistic mixtures of pure states. This concept has previously been used to model lexical ambiguity. In this paper, we consider metaphor as a type of lexical ambiguity, and examine whether metaphorical meaning can be effectively modelled using mixtures of word senses. We find that modelling metaphor is significantly more difficult than other kinds of lexical ambiguity, but that our best-performing density matrix method outperforms simple baselines as well as some neural language models.
• 摘要：在物理学中，密度矩阵用于表示混合状态，即纯状态的概率混合。这个概念以前曾用于对词汇歧义进行建模。在本文中，我们将隐喻视为一种词汇歧义，并研究是否可以使用词义混合有效地对隐喻意义进行建模。我们发现，对隐喻进行建模比其他类型的词汇歧义困难得多，但我们表现最佳的密度矩阵方法优于简单的基线以及一些神经语言模型。

Title: Prompto: An open source library for asynchronous querying of LLM endpoints

• Authors: Ryan Sze-Yin Chan, Federico Nanni, Edwin Brown, Ed Chapman, Angus R. Williams, Jonathan Bright, Evelina Gabasova
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11847
• Pdf URL: https://arxiv.org/pdf/2408.11847
• Copy Paste: [[2408.11847]] Prompto: An open source library for asynchronous querying of LLM endpoints(https://arxiv.org/abs/2408.11847)
• Keywords: language model, llm, prompt
• Abstract: Recent surge in Large Language Model (LLM) availability has opened exciting avenues for research. However, efficiently interacting with these models presents a significant hurdle since LLMs often reside on proprietary or self-hosted API endpoints, each requiring custom code for interaction. Conducting comparative studies between different models can therefore be time-consuming and necessitate significant engineering effort, hindering research efficiency and reproducibility. To address these challenges, we present prompto, an open source Python library which facilitates asynchronous querying of LLM endpoints enabling researchers to interact with multiple LLMs concurrently, while maximising efficiency and utilising individual rate limits. Our library empowers researchers and developers to interact with LLMs more effectively and enabling faster experimentation and evaluation. prompto is released with an introductory video (this https URL) under MIT License and is available via GitHub (this https URL).
• 摘要：大型语言模型 (LLM) 可用性的近期激增为研究开辟了令人兴奋的途径。然而，与这些模型的有效交互存在重大障碍，因为 LLM 通常驻留在专有或自托管的 API 端点上，每个端点都需要自定义代码进行交互。因此，对不同模型进行比较研究可能非常耗时，并且需要大量的工程工作，从而阻碍研究效率和可重复性。为了应对这些挑战，我们推出了 prompto，这是一个开源 Python 库，它促进了 LLM 端点的异步查询，使研究人员能够同时与多个 LLM 交互，同时最大限度地提高效率并利用单独的速率限制。我们的库使研究人员和开发人员能够更有效地与 LLM 交互，并实现更快的实验和评估。prompto 在 MIT 许可下发布了介绍视频（此 https URL），可通过 GitHub（此 https URL）获取。

Title: MGH Radiology Llama: A Llama 3 70B Model for Radiology

• Authors: Yucheng Shi, Peng Shu, Zhengliang Liu, Zihao Wu, Quanzheng Li, Xiang Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11848
• Pdf URL: https://arxiv.org/pdf/2408.11848
• Copy Paste: [[2408.11848]] MGH Radiology Llama: A Llama 3 70B Model for Radiology(https://arxiv.org/abs/2408.11848)
• Keywords: language model, gpt, llm
• Abstract: In recent years, the field of radiology has increasingly harnessed the power of artificial intelligence (AI) to enhance diagnostic accuracy, streamline workflows, and improve patient care. Large language models (LLMs) have emerged as particularly promising tools, offering significant potential in assisting radiologists with report generation, clinical decision support, and patient communication. This paper presents an advanced radiology-focused large language model: MGH Radiology Llama. It is developed using the Llama 3 70B model, building upon previous domain-specific models like Radiology-GPT and Radiology-Llama2. Leveraging a unique and comprehensive dataset from Massachusetts General Hospital, comprising over 6.5 million de-identified medical reports across various imaging modalities, the model demonstrates significant improvements in generating accurate and clinically relevant radiology impressions given the corresponding findings. Our evaluation, incorporating both traditional metrics and a GPT-4-based assessment, highlights the enhanced performance of this work over general-purpose LLMs.
• 摘要：近年来，放射学领域越来越多地利用人工智能 (AI) 的力量来提高诊断准确性、简化工作流程和改善患者护理。大型语言模型 (LLM) 已成为特别有前途的工具，在协助放射科医生生成报告、临床决策支持和患者沟通方面具有巨大潜力。本文介绍了一种先进的以放射学为重点的大型语言模型：MGH Radiology Llama。它是使用 Llama 3 70B 模型开发的，以之前的领域特定模型（如 Radiology-GPT 和 Radiology-Llama2）为基础。利用来自麻省总医院的独特而全面的数据集，该模型包含超过 650 万份跨各种成像模式的去识别医疗报告，在根据相应发现生成准确且具有临床相关性的放射学印象方面表现出显着的改进。我们的评估结合了传统指标和基于 GPT-4 的评估，突出了这项工作相对于通用 LLM 的增强性能。

Title: Style-Talker: Finetuning Audio Language Model and Style-Based Text-to-Speech Model for Fast Spoken Dialogue Generation

• Authors: Yinghao Aaron Li, Xilin Jiang, Jordan Darefsky, Ge Zhu, Nima Mesgarani
• Subjects: cs.CL, cs.AI, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.11849
• Pdf URL: https://arxiv.org/pdf/2408.11849
• Copy Paste: [[2408.11849]] Style-Talker: Finetuning Audio Language Model and Style-Based Text-to-Speech Model for Fast Spoken Dialogue Generation(https://arxiv.org/abs/2408.11849)
• Keywords: language model, llm, chat
• Abstract: The rapid advancement of large language models (LLMs) has significantly propelled the development of text-based chatbots, demonstrating their capability to engage in coherent and contextually relevant dialogues. However, extending these advancements to enable end-to-end speech-to-speech conversation bots remains a formidable challenge, primarily due to the extensive dataset and computational resources required. The conventional approach of cascading automatic speech recognition (ASR), LLM, and text-to-speech (TTS) models in a pipeline, while effective, suffers from unnatural prosody because it lacks direct interactions between the input audio and its transcribed text and the output audio. These systems are also limited by their inherent latency from the ASR process for real-time applications. This paper introduces Style-Talker, an innovative framework that fine-tunes an audio LLM alongside a style-based TTS model for fast spoken dialog generation. Style-Talker takes user input audio and uses transcribed chat history and speech styles to generate both the speaking style and text for the response. Subsequently, the TTS model synthesizes the speech, which is then played back to the user. While the response speech is being played, the input speech undergoes ASR processing to extract the transcription and speaking style, serving as the context for the ensuing dialogue turn. This novel pipeline accelerates the traditional cascade ASR-LLM-TTS systems while integrating rich paralinguistic information from input speech. Our experimental results show that Style-Talker significantly outperforms the conventional cascade and speech-to-speech baselines in terms of both dialogue naturalness and coherence while being more than 50% faster.
• 摘要：大型语言模型 (LLM) 的快速发展极大地推动了基于文本的聊天机器人的发展，展示了它们进行连贯且与上下文相关的对话的能力。然而，将这些进步扩展到端到端语音到语音对话机器人仍然是一项艰巨的挑战，主要是因为需要大量的数据集和计算资源。传统的将自动语音识别 (ASR)、LLM 和文本到语音 (TTS) 模型级联在一个管道中的方法虽然有效，但由于缺乏输入音频及其转录文本和输出音频之间的直接交互，导致韵律不自然。这些系统还受到 ASR 过程固有的延迟限制，无法用于实时应用。本文介绍了 Style-Talker，这是一个创新框架，它可以微调音频 LLM 以及基于风格的 TTS 模型，以快速生成口语对话。Style-Talker 获取用户输入的音频，并使用转录的聊天记录和语音风格来生成响应的说话风格和文本。随后，TTS 模型会合成语音，然后播放给用户。在播放响应语音的同时，输入语音会经过 ASR 处理，以提取转录和说话风格，作为随后对话轮次的背景。这种新颖的管道加速了传统的级联 ASR-LLM-TTS 系统，同时集成了来自输入语音的丰富副语言信息。我们的实验结果表明，Style-Talker 在对话自然度和连贯性方面明显优于传统的级联和语音到语音基线，同时速度提高了 50% 以上。

Title: Parallel Speculative Decoding with Adaptive Draft Length

• Authors: Tianyu Liu, Yun Li, Qitan Lv, Kai Liu, Jianchen Zhu, Winston Hu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11850
• Pdf URL: https://arxiv.org/pdf/2408.11850
• Copy Paste: [[2408.11850]] Parallel Speculative Decoding with Adaptive Draft Length(https://arxiv.org/abs/2408.11850)
• Keywords: llm
• Abstract: Speculative decoding (SD), where an extra draft model is employed to provide multiple \textit{draft} tokens first and then the original target model verifies these tokens in parallel, has shown great power for LLM inference acceleration. However, existing SD methods suffer from the mutual waiting problem, i.e., the target model gets stuck when the draft model is \textit{guessing} tokens, and vice versa. This problem is directly incurred by the asynchronous execution of the draft model and the target model, and is exacerbated due to the fixed draft length in speculative decoding. To address these challenges, we propose a conceptually simple, flexible, and general framework to boost speculative decoding, namely \textbf{P}arallel sp\textbf{E}culative decoding with \textbf{A}daptive d\textbf{R}aft \textbf{L}ength (PEARL). Specifically, PEARL proposes \textit{pre-verify} to verify the first draft token in advance during the drafting phase, and \textit{post-verify} to generate more draft tokens during the verification phase. PEARL parallels the drafting phase and the verification phase via applying the two strategies, and achieves adaptive draft length for different scenarios, which effectively alleviates the mutual waiting problem. Moreover, we theoretically demonstrate that the mean accepted tokens of PEARL is more than existing \textit{draft-then-verify} works. Experiments on various text generation benchmarks demonstrate the effectiveness of our \name, leading to a superior speedup performance up to \textbf{3.79$\times$} and \textbf{1.52$\times$}, compared to auto-regressive decoding and vanilla speculative decoding, respectively.
• 摘要：推测解码 (SD) 已显示出强大的 LLM 推理加速能力，即首先使用额外的草稿模型提供多个 \textit{draft} 标记，然后原始目标模型并行验证这些标记。然而，现有的 SD 方法存在相互等待问题，即当草稿模型 \textit{猜测} 标记时，目标模型会卡住，反之亦然。该问题直接由草稿模型和目标模型的异步执行引起，并且由于推测解码中的固定草稿长度而加剧。为了应对这些挑战，我们提出了一个概念上简单、灵活且通用的框架来增强推测解码，即具有 \textbf{A}daptive d\textbf{R}aft \textbf{L}ength 的 \textbf{P}arallel sp\textbf{E}culative 解码 (PEARL)。具体来说，PEARL 提出 \textit{pre-verify} 在起草阶段提前验证第一个草稿 token，并提出 \textit{post-verify} 在验证阶段生成更多草稿 token。PEARL 通过应用这两种策略将起草阶段和验证阶段并行化，并针对不同场景实现自适应的草稿长度，有效缓解相互等待问题。此外，我们从理论上证明了 PEARL 的平均接受 token 数量高于现有的 \textit{draft-then-verify} 工作。在各种文本生成基准上的实验证明了我们的 \name 的有效性，与自回归解码和 vanilla 推测解码相比，其加速性能分别高达 \textbf{3.79$\times$} 和 \textbf{1.52$\times$}。

Title: Fast Training Dataset Attribution via In-Context Learning

• Authors: Milad Fotouhi, Mohammad Taha Bahadori, Oluwaseyi Feyisetan, Payman Arabshahi, David Heckerman
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11852
• Pdf URL: https://arxiv.org/pdf/2408.11852
• Copy Paste: [[2408.11852]] Fast Training Dataset Attribution via In-Context Learning(https://arxiv.org/abs/2408.11852)
• Keywords: language model, llm, prompt
• Abstract: We investigate the use of in-context learning and prompt engineering to estimate the contributions of training data in the outputs of instruction-tuned large language models (LLMs). We propose two novel approaches: (1) a similarity-based approach that measures the difference between LLM outputs with and without provided context, and (2) a mixture distribution model approach that frames the problem of identifying contribution scores as a matrix factorization task. Our empirical comparison demonstrates that the mixture model approach is more robust to retrieval noise in in-context learning, providing a more reliable estimation of data contributions.
• 摘要：我们研究了使用上下文学习和提示工程来估计训练数据在指令调整大型语言模型 (LLM) 输出中的贡献。我们提出了两种新方法：(1) 一种基于相似性的方法，用于测量有和没有提供上下文的 LLM 输出之间的差异；(2) 一种混合分布模型方法，将识别贡献分数的问题框架化为矩阵分解任务。我们的实证比较表明，混合模型方法对上下文学习中的检索噪声更具鲁棒性，从而可以更可靠地估计数据贡献。

Title: When Raw Data Prevails: Are Large Language Model Embeddings Effective in Numerical Data Representation for Medical Machine Learning Applications?

• Authors: Yanjun Gao, Skatje Myers, Shan Chen, Dmitriy Dligach, Timothy A Miller, Danielle Bitterman, Matthew Churpek, Majid Afshar
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11854
• Pdf URL: https://arxiv.org/pdf/2408.11854
• Copy Paste: [[2408.11854]] When Raw Data Prevails: Are Large Language Model Embeddings Effective in Numerical Data Representation for Medical Machine Learning Applications?(https://arxiv.org/abs/2408.11854)
• Keywords: language model, llm, prompt
• Abstract: The introduction of Large Language Models (LLMs) has advanced data representation and analysis, bringing significant progress in their use for medical questions and answering. Despite these advancements, integrating tabular data, especially numerical data pivotal in clinical contexts, into LLM paradigms has not been thoroughly explored. In this study, we examine the effectiveness of vector representations from last hidden states of LLMs for medical diagnostics and prognostics using electronic health record (EHR) data. We compare the performance of these embeddings with that of raw numerical EHR data when used as feature inputs to traditional machine learning (ML) algorithms that excel at tabular data learning, such as eXtreme Gradient Boosting. We focus on instruction-tuned LLMs in a zero-shot setting to represent abnormal physiological data and evaluating their utilities as feature extractors to enhance ML classifiers for predicting diagnoses, length of stay, and mortality. Furthermore, we examine prompt engineering techniques on zero-shot and few-shot LLM embeddings to measure their impact comprehensively. Although findings suggest the raw data features still prevails in medical ML tasks, zero-shot LLM embeddings demonstrate competitive results, suggesting a promising avenue for future research in medical applications.
• 摘要：大型语言模型 (LLM) 的引入促进了数据表示和分析，为其在医学问答中的应用带来了重大进展。尽管取得了这些进步，但将表格数据（尤其是在临床环境中至关重要的数值数据）集成到 LLM 范式中尚未得到彻底探索。在本研究中，我们使用电子健康记录 (EHR) 数据检查了 LLM 最后隐藏状态的向量表示对医学诊断和预后的有效性。我们将这些嵌入与原始数值 EHR 数据的性能进行比较，当这些嵌入用作擅长表格数据学习的传统机器学习 (ML) 算法（例如 eXtreme Gradient Boosting）的特征输入时。我们专注于零样本设置中的指令调整 LLM 来表示异常生理数据，并评估它们作为特征提取器的效用，以增强 ML 分类器以预测诊断、住院时间和死亡率。此外，我们研究了零样本和少样本 LLM 嵌入的即时工程技术，以全面衡量它们的影响。尽管研究结果表明原始数据特征在医学 ML 任务中仍然占主导地位，但零样本 LLM 嵌入表现出了有竞争力的结果，为未来医学应用研究指明了一条有希望的途径。

Title: FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Models

• Authors: Zhongyu Zhao, Menghang Dong, Rongyu Zhang, Wenzhao Zheng, Yunpeng Zhang, Huanrui Yang, Dalong Du, Kurt Keutzer, Shanghang Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11855
• Pdf URL: https://arxiv.org/pdf/2408.11855
• Copy Paste: [[2408.11855]] FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Models(https://arxiv.org/abs/2408.11855)
• Keywords: language model, llm
• Abstract: Recent research has demonstrated that Feed-Forward Networks (FFNs) in Large Language Models (LLMs) play a pivotal role in storing diverse linguistic and factual knowledge. Conventional methods frequently face challenges due to knowledge confusion stemming from their monolithic and redundant architectures, which calls for more efficient solutions with minimal computational overhead, particularly for LLMs. In this paper, we explore the FFN computation paradigm in LLMs and introduce FactorLLM, a novel approach that decomposes well-trained dense FFNs into sparse sub-networks without requiring any further modifications, while maintaining the same level of performance. Furthermore, we embed a router from the Mixture-of-Experts (MoE), combined with our devised Prior-Approximate (PA) loss term that facilitates the dynamic activation of experts and knowledge adaptation, thereby accelerating computational processes and enhancing performance using minimal training data and fine-tuning steps. FactorLLM thus enables efficient knowledge factorization and activates select groups of experts specifically tailored to designated tasks, emulating the interactive functional segmentation of the human brain. Extensive experiments across various benchmarks demonstrate the effectiveness of our proposed FactorLLM which achieves comparable performance to the source model securing up to 85% model performance while obtaining over a 30% increase in inference speed. Code: this https URL.
• 摘要：最近的研究表明，大型语言模型 (LLM) 中的前馈网络 (FFN) 在存储各种语言和事实知识方面发挥着关键作用。传统方法经常面临挑战，因为它们的单片和冗余架构导致知识混乱，这需要更高效的解决方案和最小的计算开销，尤其是对于 LLM。在本文中，我们探索了 LLM 中的 FFN 计算范式，并引入了 FactorLLM，这是一种新颖的方法，它将训练有素的密集 FFN 分解为稀疏子网络，而无需任何进一步的修改，同时保持相同的性能水平。此外，我们嵌入了一个来自混合专家 (MoE) 的路由器，结合我们设计的先验近似 (PA) 损失项，以促进专家的动态激活和知识适应，从而加速计算过程并使用最少的训练数据和微调步骤提高性能。因此，FactorLLM 可以实现高效的知识分解，并激活专门针对指定任务的精选专家组，模拟人类大脑的交互式功能分割。在各种基准测试中开展的大量实验证明了我们提出的 FactorLLM 的有效性，其性能与源模型相当，模型性能最高可达 85%，推理速度提高 30% 以上。代码：此 https URL。

Title: Dynamic Adaptive Optimization for Effective Sentiment Analysis Fine-Tuning on Large Language Models

• Authors: Hongcheng Ding, Xuanze Zhao, Shamsul Nahar Abdullah, Deshinta Arrova Dewi, Zixiao Jiang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11856
• Pdf URL: https://arxiv.org/pdf/2408.11856
• Copy Paste: [[2408.11856]] Dynamic Adaptive Optimization for Effective Sentiment Analysis Fine-Tuning on Large Language Models(https://arxiv.org/abs/2408.11856)
• Keywords: language model, llm
• Abstract: Sentiment analysis plays a crucial role in various domains, such as business intelligence and financial forecasting. Large language models (LLMs) have become a popular paradigm for sentiment analysis, leveraging multi-task learning to address specific tasks concurrently. However, LLMs with fine-tuning for sentiment analysis often underperforms due to the inherent challenges in managing diverse task complexities. Moreover, constant-weight approaches in multi-task learning struggle to adapt to variations in data characteristics, further complicating model effectiveness. To address these issues, we propose a novel multi-task learning framework with a dynamic adaptive optimization (DAO) module. This module is designed as a plug-and-play component that can be seamlessly integrated into existing models, providing an effective and flexible solution for multi-task learning. The key component of the DAO module is dynamic adaptive loss, which dynamically adjusts the weights assigned to different tasks based on their relative importance and data characteristics during training. Sentiment analyses on a standard and customized financial text dataset demonstrate that the proposed framework achieves superior performance. Specifically, this work improves the Mean Squared Error (MSE) and Accuracy (ACC) by 15.58% and 1.24% respectively, compared with previous work.
• 摘要：情绪分析在商业智能和财务预测等各个领域都发挥着至关重要的作用。大型语言模型 (LLM) 已成为情绪分析的流行范例，利用多任务学习同时解决特定任务。然而，由于管理各种任务复杂性的固有挑战，具有情绪分析微调的 LLM 往往表现不佳。此外，多任务学习中的恒定权重方法难以适应数据特征的变化，进一步复杂化了模型的有效性。为了解决这些问题，我们提出了一种具有动态自适应优化 (DAO) 模块的新型多任务学习框架。该模块设计为即插即用组件，可以无缝集成到现有模型中，为多任务学习提供有效而灵活的解决方案。DAO 模块的关键组件是动态自适应损失，它根据训练期间不同任务的相对重要性和数据特征动态调整分配给不同任务的权重。对标准和定制财务文本数据集的情绪分析表明，所提出的框架实现了卓越的性能。具体来说，与之前的工作相比，本工作将均方误差 (MSE) 和准确度 (ACC) 分别提高了 15.58% 和 1.24%。

Title: Hermes 3 Technical Report

• Authors: Ryan Teknium, Jeffrey Quesnelle, Chen Guang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11857
• Pdf URL: https://arxiv.org/pdf/2408.11857
• Copy Paste: [[2408.11857]] Hermes 3 Technical Report(https://arxiv.org/abs/2408.11857)
• Keywords: language model, chat
• Abstract: Instruct (or "chat") tuned models have become the primary way in which most people interact with large language models. As opposed to "base" or "foundation" models, instruct-tuned models are optimized to respond to imperative statements. We present Hermes 3, a neutrally-aligned generalist instruct and tool use model with strong reasoning and creative abilities. Its largest version, Hermes 3 405B, achieves state of the art performance among open weight models on several public benchmarks.
• 摘要：指令（或“聊天”）调优模型已成为大多数人与大型语言模型交互的主要方式。与“基础”或“基础”模型不同，指令调优模型经过优化，可响应命令式语句。我们推出了 Hermes 3，这是一种中立的通用指令和工具使用模型，具有强大的推理和创造能力。其最大版本 Hermes 3 405B 在多个公共基准测试中实现了开放权重模型中最先进的性能。

Title: Risks and NLP Design: A Case Study on Procedural Document QA

• Authors: Nikita Haduong (1), Alice Gao (1), Noah A. Smith (1 and 2) ((1) Paul G. Allen School of Computer Science & Engineering, University of Washington, (2) Allen Institute for Artificial Intelligence)
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2408.11860
• Pdf URL: https://arxiv.org/pdf/2408.11860
• Copy Paste: [[2408.11860]] Risks and NLP Design: A Case Study on Procedural Document QA(https://arxiv.org/abs/2408.11860)
• Keywords: language model
• Abstract: As NLP systems are increasingly deployed at scale, concerns about their potential negative impacts have attracted the attention of the research community, yet discussions of risk have mostly been at an abstract level and focused on generic AI or NLP applications. We argue that clearer assessments of risks and harms to users--and concrete strategies to mitigate them--will be possible when we specialize the analysis to more concrete applications and their plausible users. As an illustration, this paper is grounded in cooking recipe procedural document question answering (ProcDocQA), where there are well-defined risks to users such as injuries or allergic reactions. Our case study shows that an existing language model, applied in "zero-shot" mode, quantitatively answers real-world questions about recipes as well or better than the humans who have answered the questions on the web. Using a novel questionnaire informed by theoretical work on AI risk, we conduct a risk-oriented error analysis that could then inform the design of a future system to be deployed with lower risk of harm and better performance.
• 摘要：随着 NLP 系统越来越大规模地部署，对其潜在负面影响的担忧引起了研究界的关注，但风险讨论大多处于抽象层面，并集中在通用 AI 或 NLP 应用程序上。我们认为，当我们将分析专门用于更具体的应用程序及其可能的用户时，对用户的风险和危害的更清晰评估以及减轻风险的具体策略将成为可能。作为一个例子，本文以烹饪食谱程序文档问答 (ProcDocQA) 为基础，其中存在明确定义的用户风险，例如受伤或过敏反应。我们的案例研究表明，在“零样本”模式下应用的现有语言模型可以定量回答有关食谱的真实问题，其回答效果与在网络上回答问题的人一样好甚至更好。使用基于 AI 风险理论工作的新型问卷，我们进行了面向风险的错误分析，然后可以为设计未来部署的系统提供信息，以降低危害风险并提高性能。

Title: Speaking the Same Language: Leveraging LLMs in Standardizing Clinical Data for AI

• Authors: Arindam Sett, Somaye Hashemifar, Mrunal Yadav, Yogesh Pandit, Mohsen Hejrati
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11861
• Pdf URL: https://arxiv.org/pdf/2408.11861
• Copy Paste: [[2408.11861]] Speaking the Same Language: Leveraging LLMs in Standardizing Clinical Data for AI(https://arxiv.org/abs/2408.11861)
• Keywords: language model, llm
• Abstract: The implementation of Artificial Intelligence (AI) in the healthcare industry has garnered considerable attention, attributable to its prospective enhancement of clinical outcomes, expansion of access to superior healthcare, cost reduction, and elevation of patient satisfaction. Nevertheless, the primary hurdle that persists is related to the quality of accessible multi-modal healthcare data in conjunction with the evolution of AI methodologies. This study delves into the adoption of large language models to address specific challenges, specifically, the standardization of healthcare data. We advocate the use of these models to identify and map clinical data schemas to established data standard attributes, such as the Fast Healthcare Interoperability Resources. Our results illustrate that employing large language models significantly diminishes the necessity for manual data curation and elevates the efficacy of the data standardization process. Consequently, the proposed methodology has the propensity to expedite the integration of AI in healthcare, ameliorate the quality of patient care, whilst minimizing the time and financial resources necessary for the preparation of data for AI.
• 摘要：人工智能 (AI) 在医疗保健行业的应用已引起广泛关注，这归因于它有望改善临床结果、扩大优质医疗保健的可及性、降低成本和提高患者满意度。然而，仍然存在的主要障碍与可访问的多模式医疗保健数据的质量以及 AI 方法的发展有关。本研究深入探讨了采用大型语言模型来应对特定挑战，特别是医疗保健数据的标准化。我们提倡使用这些模型来识别临床数据模式并将其映射到已建立的数据标准属性，例如快速医疗保健互操作性资源。我们的结果表明，采用大型语言模型可显著减少手动数据管理的必要性并提高数据标准化过程的效率。因此，所提出的方法倾向于加快 AI 在医疗保健中的整合，改善患者护理的质量，同时最大限度地减少为 AI 准备数据所需的时间和财务资源。

Title: Sentiment analysis of preservice teachers' reflections using a large language model

• Authors: Yunsoo Park, Younkyung Hong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11862
• Pdf URL: https://arxiv.org/pdf/2408.11862
• Copy Paste: [[2408.11862]] Sentiment analysis of preservice teachers' reflections using a large language model(https://arxiv.org/abs/2408.11862)
• Keywords: language model, gpt, llm
• Abstract: In this study, the emotion and tone of preservice teachers' reflections were analyzed using sentiment analysis with LLMs: GPT-4, Gemini, and BERT. We compared the results to understand how each tool categorizes and describes individual reflections and multiple reflections as a whole. This study aims to explore ways to bridge the gaps between qualitative, quantitative, and computational analyses of reflective practices in teacher education. This study finds that to effectively integrate LLM analysis into teacher education, developing an analysis method and result format that are both comprehensive and relevant for preservice teachers and teacher educators is crucial.
• 摘要：在本研究中，使用 LLM 情绪分析工具（GPT-4、Gemini 和 BERT）分析了在职前教师反思的情绪和语气。我们比较了结果，以了解每种工具如何对单个反思和多个反思进行分类和描述。本研究旨在探索弥合教师教育反思实践的定性、定量和计算分析之间差距的方法。本研究发现，要将 LLM 分析有效地融入教师教育，开发一种对在职前教师和教师教育者来说既全面又相关的分析方法和结果格式至关重要。

Title: How Susceptible are LLMs to Influence in Prompts?

• Authors: Sotiris Anagnostidis, Jannis Bulian
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11865
• Pdf URL: https://arxiv.org/pdf/2408.11865
• Copy Paste: [[2408.11865]] How Susceptible are LLMs to Influence in Prompts?(https://arxiv.org/abs/2408.11865)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) are highly sensitive to prompts, including additional context provided therein. As LLMs grow in capability, understanding their prompt-sensitivity becomes increasingly crucial for ensuring reliable and robust performance, particularly since evaluating these models becomes more challenging. In this work, we investigate how current models (Llama, Mixtral, Falcon) respond when presented with additional input from another model, mimicking a scenario where a more capable model -- or a system with access to more external information -- provides supplementary information to the target model. Across a diverse spectrum of question-answering tasks, we study how an LLM's response to multiple-choice questions changes when the prompt includes a prediction and explanation from another model. Specifically, we explore the influence of the presence of an explanation, the stated authoritativeness of the source, and the stated confidence of the supplementary input. Our findings reveal that models are strongly influenced, and when explanations are provided they are swayed irrespective of the quality of the explanation. The models are more likely to be swayed if the input is presented as being authoritative or confident, but the effect is small in size. This study underscores the significant prompt-sensitivity of LLMs and highlights the potential risks of incorporating outputs from external sources without thorough scrutiny and further validation. As LLMs continue to advance, understanding and mitigating such sensitivities will be crucial for their reliable and trustworthy deployment.
• 摘要：大型语言模型 (LLM) 对提示（包括其中提供的额外上下文）高度敏感。随着 LLM 能力的增强，了解它们的提示敏感性对于确保可靠和稳健的性能变得越来越重要，特别是因为评估这些模型变得越来越具有挑战性。在这项工作中，我们研究了当前模型（Llama、Mixtral、Falcon）在从另一个模型获得额外输入时如何响应，模拟一个更强大的模型（或可以访问更多外部信息的系统）向目标模型提供补充信息的场景。在各种各样的问答任务中，我们研究了当提示包含来自另一个模型的预测和解释时，LLM 对多项选择题的回答如何变化。具体来说，我们探讨了解释的存在、来源的权威性和补充输入的可信度的影响。我们的研究结果表明，模型受到了强烈的影响，当提供解释时，无论解释的质量如何，它们都会受到影响。如果输入内容具有权威性或可信度，模型更容易受到影响，但影响很小。这项研究强调了 LLM 的显著即时敏感性，并强调了在未经彻底审查和进一步验证的情况下纳入外部来源的输出的潜在风险。随着 LLM 的不断进步，理解和减轻这种敏感性对于其可靠和值得信赖的部署至关重要。

Title: Crossing New Frontiers: Knowledge-Augmented Large Language Model Prompting for Zero-Shot Text-Based De Novo Molecule Design

• Authors: Sakhinana Sagar Srinivas, Venkataramana Runkana
• Subjects: cs.CL, cs.AI, cs.LG, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2408.11866
• Pdf URL: https://arxiv.org/pdf/2408.11866
• Copy Paste: [[2408.11866]] Crossing New Frontiers: Knowledge-Augmented Large Language Model Prompting for Zero-Shot Text-Based De Novo Molecule Design(https://arxiv.org/abs/2408.11866)
• Keywords: language model, llm, prompt
• Abstract: Molecule design is a multifaceted approach that leverages computational methods and experiments to optimize molecular properties, fast-tracking new drug discoveries, innovative material development, and more efficient chemical processes. Recently, text-based molecule design has emerged, inspired by next-generation AI tasks analogous to foundational vision-language models. Our study explores the use of knowledge-augmented prompting of large language models (LLMs) for the zero-shot text-conditional de novo molecular generation task. Our approach uses task-specific instructions and a few demonstrations to address distributional shift challenges when constructing augmented prompts for querying LLMs to generate molecules consistent with technical descriptions. Our framework proves effective, outperforming state-of-the-art (SOTA) baseline models on benchmark datasets.
• 摘要：分子设计是一种多方面的方法，它利用计算方法和实验来优化分子特性、加快新药发现、创新材料开发和更高效的化学过程。最近，基于文本的分子设计应运而生，其灵感来自类似于基础视觉语言模型的下一代人工智能任务。我们的研究探索了知识增强提示在大型语言模型 (LLM) 中的应用，用于零样本文本条件从头分子生成任务。我们的方法使用特定于任务的指令和一些演示来解决在构建增强提示以查询 LLM 以生成与技术描述一致的分子时出现的分布偏移挑战。我们的框架被证明是有效的，在基准数据集上的表现优于最先进的 (SOTA) 基线模型。

Title: Enhance Lifelong Model Editing with Continuous Data-Adapter Association

• Authors: Jiaang Li, Quan Wang, Zhongnan Wang, Yongdong Zhang, Zhendong Mao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11869
• Pdf URL: https://arxiv.org/pdf/2408.11869
• Copy Paste: [[2408.11869]] Enhance Lifelong Model Editing with Continuous Data-Adapter Association(https://arxiv.org/abs/2408.11869)
• Keywords: language model, gpt, llm
• Abstract: Large language models (LLMs) require model editing to efficiently update specific knowledge within them and avoid factual errors. Most model editing methods are solely designed for single-time use and lead to a significant forgetting effect after sequential edits over time, referred to as lifelong editing. Current approaches manage sequential edits by freezing original parameters and allocating new adapters for each knowledge modification. However, these methods lack robustness to minor input variations. To address this challenge, we propose ELDER, \textbf{E}nhancing \textbf{L}ifelong mo\textbf{D}el \textbf{E}diting with mixtu\textbf{R}e of Low-Rank Adapter (LoRA). ELDER is an adaptive approach that integrates multiple LoRAs through a router network. It learns to create a continuous and smooth association between data and adapters, thereby enhancing robustness and generalization to semantically equivalent inputs. Additionally, we introduce a novel loss to help learn associations between adapter allocations and edit semantics. A deferral mechanism is also proposed to retain the original LLM capabilities post-edit. Extensive experiments on GPT-2 XL and LLaMA2-7B demonstrate that ELDER effectively edits models in the lifelong setting and exhibits strong scalability, while retaining LLM's general abilities on downstream tasks.
• 摘要：大型语言模型 (LLM) 需要模型编辑来有效更新其中的特定知识并避免事实错误。大多数模型编辑方法仅设计为一次性使用，并且在一段时间内进行连续编辑后会导致显着的遗忘效应，这称为终身编辑。当前的方法通过冻结原始参数并为每个知识修改分配新的适配器来管理连续编辑。但是，这些方法对微小输入变化缺乏鲁棒性。为了应对这一挑战，我们提出了 ELDER，\textbf{E} 增强 \textbf{L}ifelong mo\textbf{D}el \textbf{E} 编辑，并使用低秩适配器 (LoRA) 的 mixtu\textbf{R}e。ELDER 是一种通过路由器网络集成多个 LoRA 的自适应方法。它学习在数据和适配器之间建立连续而平滑的关联，从而增强对语义等效输入的鲁棒性和泛化。此外，我们引入了一种新颖的损失来帮助学习适配器分配和编辑语义之间的关联。还提出了一种延迟机制，以在编辑后保留原始的 LLM 功能。在 GPT-2 XL 和 LLaMA2-7B 上进行的大量实验表明，ELDER 可以在终身设置中有效地编辑模型并表现出强大的可扩展性，同时保留了 LLM 在下游任务上的一般能力。

Title: MegaFake: A Theory-Driven Dataset of Fake News Generated by Large Language Models

• Authors: Lionel Z. Wang, Yiming Ma, Renfei Gao, Beichen Guo, Zhuoran Li, Han Zhu, Wenqi Fan, Zexin Lu, Ka Chung Ng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.11871
• Pdf URL: https://arxiv.org/pdf/2408.11871
• Copy Paste: [[2408.11871]] MegaFake: A Theory-Driven Dataset of Fake News Generated by Large Language Models(https://arxiv.org/abs/2408.11871)
• Keywords: language model, llm
• Abstract: The advent of large language models (LLMs) has revolutionized online content creation, making it much easier to generate high-quality fake news. This misuse threatens the integrity of our digital environment and ethical standards. Therefore, understanding the motivations and mechanisms behind LLM-generated fake news is crucial. In this study, we analyze the creation of fake news from a social psychology perspective and develop a comprehensive LLM-based theoretical framework, LLM-Fake Theory. We introduce a novel pipeline that automates the generation of fake news using LLMs, thereby eliminating the need for manual annotation. Utilizing this pipeline, we create a theoretically informed Machine-generated Fake news dataset, MegaFake, derived from the GossipCop dataset. We conduct comprehensive analyses to evaluate our MegaFake dataset. We believe that our dataset and insights will provide valuable contributions to future research focused on the detection and governance of fake news in the era of LLMs.
• 摘要：大型语言模型 (LLM) 的出现彻底改变了在线内容创作，使生成高质量虚假新闻变得更加容易。这种滥用威胁到我们数字环境的完整性和道德标准。因此，了解 LLM 生成虚假新闻背后的动机和机制至关重要。在本研究中，我们从社会心理学的角度分析了虚假新闻的产生，并开发了一个基于 LLM 的综合理论框架，即 LLM-Fake 理论。我们引入了一种新颖的流程，可以使用 LLM 自动生成虚假新闻，从而无需手动注释。利用这个流程，我们创建了一个理论上合理的机器生成的虚假新闻数据集 MegaFake，它源自 GossipCop 数据集。我们进行了全面的分析以评估我们的 MegaFake 数据集。我们相信，我们的数据集和见解将为未来专注于 LLM 时代虚假新闻检测和治理的研究做出宝贵贡献。

Title: Hierarchical Retrieval-Augmented Generation Model with Rethink for Multi-hop Question Answering

• Authors: Xiaoming Zhang, Ming Wang, Xiaocui Yang, Daling Wang, Shi Feng, Yifei Zhang
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.11875
• Pdf URL: https://arxiv.org/pdf/2408.11875
• Copy Paste: [[2408.11875]] Hierarchical Retrieval-Augmented Generation Model with Rethink for Multi-hop Question Answering(https://arxiv.org/abs/2408.11875)
• Keywords: retrieval-augmented generation
• Abstract: Multi-hop Question Answering (QA) necessitates complex reasoning by integrating multiple pieces of information to resolve intricate questions. However, existing QA systems encounter challenges such as outdated information, context window length limitations, and an accuracy-quantity trade-off. To address these issues, we propose a novel framework, the Hierarchical Retrieval-Augmented Generation Model with Rethink (HiRAG), comprising Decomposer, Definer, Retriever, Filter, and Summarizer five key modules. We introduce a new hierarchical retrieval strategy that incorporates both sparse retrieval at the document level and dense retrieval at the chunk level, effectively integrating their strengths. Additionally, we propose a single-candidate retrieval method to mitigate the limitations of multi-candidate retrieval. We also construct two new corpora, Indexed Wikicorpus and Profile Wikicorpus, to address the issues of outdated and insufficient knowledge. Our experimental results on four datasets demonstrate that HiRAG outperforms state-of-the-art models across most metrics, and our Indexed Wikicorpus is effective. The code for HiRAG is available at this https URL
• 摘要：多跳问答 (QA) 需要通过整合多条信息来解决复杂问题，从而进行复杂的推理。然而，现有的 QA 系统面临着诸如信息过时、上下文窗口长度限制以及准确度与数量之间的权衡等挑战。为了解决这些问题，我们提出了一个新颖的框架，即带有 Rethink 的分层检索增强生成模型 (HiRAG)，包括分解器、定义器、检索器、过滤器和摘要器五个关键模块。我们引入了一种新的分层检索策略，该策略结合了文档级别的稀疏检索和块级别的密集检索，有效地整合了它们的优势。此外，我们提出了一种单候选检索方法来减轻多候选检索的局限性。我们还构建了两个新的语料库，即 Indexed Wikicorpus 和 Profile Wikicorpus，以解决知识过时和不足的问题。我们在四个数据集上的实验结果表明，HiRAG 在大多数指标上的表现都优于最先进的模型，而且我们的索引维基语料库是有效的。HiRAG 的代码可在此 https URL 上找到

Title: Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications

• Authors: Qianqian Xie, Dong Li, Mengxi Xiao, Zihao Jiang, Ruoyu Xiang, Xiao Zhang, Zhengyu Chen, Yueru He, Weiguang Han, Yuzhe Yang, Shunian Chen, Yifei Zhang, Lihang Shen, Daniel Kim, Zhiwei Liu, Zheheng Luo, Yangyang Yu, Yupeng Cao, Zhiyang Deng, Zhiyuan Yao, Haohang Li, Duanyu Feng, Yongfu Dai, VijayaSai Somasundaram, Peng Lu, Yilun Zhao, Yitao Long, Guojun Xiong, Kaleb Smith, Honghai Yu, Yanzhao Lai, Min Peng, Jianyun Nie, Jordan W. Suchow, Xiao-Yang Liu, Benyou Wang, Alejandro Lopez-Lira, Jimin Huang, Sophia Ananiadou
• Subjects: cs.CL, cs.CE, q-fin.CP
• Abstract URL: https://arxiv.org/abs/2408.11878
• Pdf URL: https://arxiv.org/pdf/2408.11878
• Copy Paste: [[2408.11878]] Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications(https://arxiv.org/abs/2408.11878)
• Keywords: language model, gpt, llm
• Abstract: Large language models (LLMs) have advanced financial applications, yet they often lack sufficient financial knowledge and struggle with tasks involving multi-modal inputs like tables and time series data. To address these limitations, we introduce \textit{Open-FinLLMs}, a series of Financial LLMs. We begin with FinLLaMA, pre-trained on a 52 billion token financial corpus, incorporating text, tables, and time-series data to embed comprehensive financial knowledge. FinLLaMA is then instruction fine-tuned with 573K financial instructions, resulting in FinLLaMA-instruct, which enhances task performance. Finally, we present FinLLaVA, a multimodal LLM trained with 1.43M image-text instructions to handle complex financial data types. Extensive evaluations demonstrate FinLLaMA's superior performance over LLaMA3-8B, LLaMA3.1-8B, and BloombergGPT in both zero-shot and few-shot settings across 19 and 4 datasets, respectively. FinLLaMA-instruct outperforms GPT-4 and other Financial LLMs on 15 datasets. FinLLaVA excels in understanding tables and charts across 4 multimodal tasks. Additionally, FinLLaMA achieves impressive Sharpe Ratios in trading simulations, highlighting its robust financial application capabilities. We will continually maintain and improve our models and benchmarks to support ongoing innovation in academia and industry.
• 摘要：大型语言模型 (LLM) 具有先进的金融应用，但它们通常缺乏足够的金融知识，并且在涉及表格和时间序列数据等多模态输入的任务中举步维艰。为了解决这些限制，我们引入了 \textit{Open-FinLLMs}，这是一系列金融 LLM。我们从 FinLLaMA 开始，它在 520 亿个 token 金融语料库上进行了预训练，结合了文本、表格和时间序列数据以嵌入全面的金融知识。然后使用 573K 条金融指令对 FinLLaMA 进行指令微调，从而生成 FinLLaMA-instruct，这提高了任务性能。最后，我们介绍了 FinLLaVA，这是一种多模态 LLM，使用 1.43M 图像文本指令进行训练，以处理复杂的金融数据类型。广泛的评估表明，FinLLaMA 在 19 个和 4 个数据集的零样本和小样本设置中均优于 LLaMA3-8B、LLaMA3.1-8B 和 BloombergGPT。 FinLLaMA-instruct 在 15 个数据集上的表现优于 GPT-4 和其他金融 LLM。FinLLaVA 在 4 个多模态任务中擅长理解表格和图表。此外，FinLLaMA 在交易模拟中实现了令人印象深刻的夏普比率，凸显了其强大的金融应用能力。我们将不断维护和改进我们的模型和基准，以支持学术界和行业的持续创新。

Title: Beyond Labels: Aligning Large Language Models with Human-like Reasoning

• Authors: Muhammad Rafsan Kabir, Rafeed Mohammad Sultan, Ihsanul Haque Asif, Jawad Ibn Ahad, Fuad Rahman, Mohammad Ruhul Amin, Nabeel Mohammed, Shafin Rahman
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.11879
• Pdf URL: https://arxiv.org/pdf/2408.11879
• Copy Paste: [[2408.11879]] Beyond Labels: Aligning Large Language Models with Human-like Reasoning(https://arxiv.org/abs/2408.11879)
• Keywords: language model, llm
• Abstract: Aligning large language models (LLMs) with a human reasoning approach ensures that LLMs produce morally correct and human-like decisions. Ethical concerns are raised because current models are prone to generating false positives and providing malicious responses. To contribute to this issue, we have curated an ethics dataset named Dataset for Aligning Reasons (DFAR), designed to aid in aligning language models to generate human-like reasons. The dataset comprises statements with ethical-unethical labels and their corresponding reasons. In this study, we employed a unique and novel fine-tuning approach that utilizes ethics labels and their corresponding reasons (L+R), in contrast to the existing fine-tuning approach that only uses labels (L). The original pre-trained versions, the existing fine-tuned versions, and our proposed fine-tuned versions of LLMs were then evaluated on an ethical-unethical classification task and a reason-generation task. Our proposed fine-tuning strategy notably outperforms the others in both tasks, achieving significantly higher accuracy scores in the classification task and lower misalignment rates in the reason-generation task. The increase in classification accuracies and decrease in misalignment rates indicate that the L+R fine-tuned models align more with human ethics. Hence, this study illustrates that injecting reasons has substantially improved the alignment of LLMs, resulting in more human-like responses. We have made the DFAR dataset and corresponding codes publicly available at this https URL.
• 摘要：将大型语言模型 (LLM) 与人类推理方法对齐可确保 LLM 做出道德上正确且类似人类的决策。由于当前模型容易产生误报并提供恶意响应，因此引发了道德问题。为了解决此问题，我们整理了一个名为“对齐原因数据集”(DFAR) 的道德数据集，旨在帮助对齐语言模型以生成类似人类的原因。该数据集包含带有道德-不道德标签的语句及其相应的原因。在本研究中，我们采用了一种独特而新颖的微调方法，该方法利用道德标签及其相应的原因 (L+R)，而不是现有的仅使用标签 (L) 的微调方法。然后在道德-不道德分类任务和原因生成任务上评估原始预训练版本、现有微调版本和我们提出的 LLM 微调版本。我们提出的微调策略在两个任务中都明显优于其他策略，在分类任务中实现了更高的准确度分数，在原因生成任务中实现了更低的错位率。分类准确率的提高和错位率的下降表明 L+R 微调模型更符合人类道德观。因此，这项研究表明，注入原因大大改善了 LLM 的对齐，从而产生了更像人类的响应。我们已在此 https URL 上公开了 DFAR 数据集和相应的代码。

Title: Ancient Wisdom, Modern Tools: Exploring Retrieval-Augmented LLMs for Ancient Indian Philosophy

• Authors: Priyanka Mandikal
• Subjects: cs.CL, cs.CY, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.11903
• Pdf URL: https://arxiv.org/pdf/2408.11903
• Copy Paste: [[2408.11903]] Ancient Wisdom, Modern Tools: Exploring Retrieval-Augmented LLMs for Ancient Indian Philosophy(https://arxiv.org/abs/2408.11903)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: LLMs have revolutionized the landscape of information retrieval and knowledge dissemination. However, their application in specialized areas is often hindered by factual inaccuracies and hallucinations, especially in long-tail knowledge distributions. We explore the potential of retrieval-augmented generation (RAG) models for long-form question answering (LFQA) in a specialized knowledge domain. We present VedantaNY-10M, a dataset curated from extensive public discourses on the ancient Indian philosophy of Advaita Vedanta. We develop and benchmark a RAG model against a standard, non-RAG LLM, focusing on transcription, retrieval, and generation performance. Human evaluations by computational linguists and domain experts show that the RAG model significantly outperforms the standard model in producing factual and comprehensive responses having fewer hallucinations. In addition, a keyword-based hybrid retriever that emphasizes unique low-frequency terms further improves results. Our study provides insights into effectively integrating modern large language models with ancient knowledge systems. Project page with dataset and code: this https URL
• 摘要：LLM 彻底改变了信息检索和知识传播的格局。然而，它们在专业领域的应用往往受到事实不准确和幻觉的阻碍，尤其是在长尾知识分布中。我们探索检索增强生成 (RAG) 模型在专业知识领域中用于长篇问答 (LFQA) 的潜力。我们展示了 VedantaNY-10M，这是一个从大量关于古印度哲学 Advaita Vedanta 的公开讨论中精选出来的数据集。我们开发了一个 RAG 模型，并将其与标准的非 RAG LLM 进行对比，重点关注转录、检索和生成性能。计算语言学家和领域专家的人工评估表明，RAG 模型在产生事实和全面的反应方面明显优于标准模型，幻觉更少。此外，强调独特低频术语的基于关键字的混合检索器进一步改善了结果。我们的研究为有效整合现代大型语言模型与古代知识系统提供了见解。包含数据集和代码的项目页面：此 https URL

Title: Decoding SEC Actions: Enforcement Trends through Analyzing Blockchain litigation using LLM-based Thematic Factor Mapping

• Authors: Junliang Luo, Xihan Xiong, William Knottenbelt, Xue Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11961
• Pdf URL: https://arxiv.org/pdf/2408.11961
• Copy Paste: [[2408.11961]] Decoding SEC Actions: Enforcement Trends through Analyzing Blockchain litigation using LLM-based Thematic Factor Mapping(https://arxiv.org/abs/2408.11961)
• Keywords: language model, llm
• Abstract: The proliferation of blockchain entities (persons or enterprises) exposes them to potential regulatory actions (e.g., being litigated) by regulatory authorities. Regulatory frameworks for crypto assets are actively being developed and refined, increasing the likelihood of such actions. The lack of systematic analysis of the factors driving litigation against blockchain entities leaves companies in need of clarity to navigate compliance risks. This absence of insight also deprives investors of the information for informed decision-making. This study focuses on U.S. litigation against blockchain entities, particularly by the U.S. Securities and Exchange Commission (SEC) given its influence on global crypto regulation. Utilizing frontier pretrained language models and large language models, we systematically map all SEC complaints against blockchain companies from 2012 to 2024 to thematic factors conceptualized by our study to delineate the factors driving SEC actions. We quantify the thematic factors and assess their influence on specific legal Acts cited within the complaints on an annual basis, allowing us to discern the regulatory emphasis, patterns and conduct trend analysis.
• 摘要：区块链实体（个人或企业）的激增使它们面临监管机构的潜在监管行动（例如，被诉讼）。加密资产的监管框架正在积极开发和完善，增加了此类行动的可能性。缺乏对推动针对区块链实体的诉讼的因素的系统分析，使得公司需要明确应对合规风险。这种缺乏洞察力也使投资者无法获得做出明智决策的信息。本研究重点关注美国针对区块链实体的诉讼，特别是美国证券交易委员会 (SEC) 的诉讼，因为它对全球加密监管有影响。利用前沿预训练语言模型和大型语言模型，我们系统地将 2012 年至 2024 年针对区块链公司的所有 SEC 投诉映射到我们研究概念化的主题因素，以描述推动 SEC 行动的因素。我们量化主题因素并按年度评估它们对投诉中引用的特定法律行为的影响，使我们能够辨别监管重点、模式并进行趋势分析。

Title: Large Language Models for Page Stream Segmentation

• Authors: Hunter Heidenreich, Ratish Dalvi, Rohith Mukku, Nikhil Verma, Neven Pičuljan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.11981
• Pdf URL: https://arxiv.org/pdf/2408.11981
• Copy Paste: [[2408.11981]] Large Language Models for Page Stream Segmentation(https://arxiv.org/abs/2408.11981)
• Keywords: language model, llm
• Abstract: Page Stream Segmentation (PSS) is an essential prerequisite for automated document processing at scale. However, research progress has been limited by the absence of realistic public benchmarks. This paper works towards addressing this gap by introducing TABME++, an enhanced benchmark featuring commercial Optical Character Recognition (OCR) annotations. We evaluate the performance of large language models (LLMs) on PSS, focusing on decoder-based models fine-tuned with parameter-efficient methods. Our results show that decoder-based LLMs outperform smaller multimodal encoders. Through a review of existing PSS research and datasets, we identify key challenges and advancements in the field. Our findings highlight the key importance of robust OCR, providing valuable insights for the development of more effective document processing systems.
• 摘要：页面流分割 (PSS) 是大规模自动文档处理的重要先决条件。然而，由于缺乏现实的公共基准，研究进展受到限制。本文致力于通过引入 TABME++ 来解决这一差距，TABME++ 是一种增强型基准，具有商业光学字符识别 (OCR) 注释。我们评估了大型语言模型 (LLM) 在 PSS 上的性能，重点关注使用参数高效方法微调的基于解码器的模型。我们的结果表明，基于解码器的 LLM 优于较小的多模态编码器。通过回顾现有的 PSS 研究和数据集，我们确定了该领域的主要挑战和进步。我们的研究结果强调了强大的 OCR 的重要性，为开发更有效的文档处理系统提供了宝贵的见解。

Title: RAG-Optimized Tibetan Tourism LLMs: Enhancing Accuracy and Personalization

• Authors: Jinhu Qi, Shuai Yan, Yibo Zhang, Wentao Zhang, Rong Jin, Yuwei Hu, Ke Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12003
• Pdf URL: https://arxiv.org/pdf/2408.12003
• Copy Paste: [[2408.12003]] RAG-Optimized Tibetan Tourism LLMs: Enhancing Accuracy and Personalization(https://arxiv.org/abs/2408.12003)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: With the development of the modern social economy, tourism has become an important way to meet people's spiritual needs, bringing development opportunities to the tourism industry. However, existing large language models (LLMs) face challenges in personalized recommendation capabilities and the generation of content that can sometimes produce hallucinations. This study proposes an optimization scheme for Tibet tourism LLMs based on retrieval-augmented generation (RAG) technology. By constructing a database of tourist viewpoints and processing the data using vectorization techniques, we have significantly improved retrieval accuracy. The application of RAG technology effectively addresses the hallucination problem in content generation. The optimized model shows significant improvements in fluency, accuracy, and relevance of content generation. This research demonstrates the potential of RAG technology in the standardization of cultural tourism information and data analysis, providing theoretical and technical support for the development of intelligent cultural tourism service systems.
• 摘要：随着现代社会经济的发展，旅游成为满足人们精神需求的重要方式，为旅游行业带来了发展机遇。然而现有的大型语言模型（LLM）面临着个性化推荐能力和生成内容时产生幻觉的挑战。本研究提出了一种基于检索增强生成（RAG）技术的西藏旅游大型语言模型优化方案，通过构建游客视点数据库，并利用向量化技术对数据进行处理，检索准确率明显提升。RAG技术的应用有效解决了内容生成中的幻觉问题，优化后的模型在内容生成的流畅性、准确性和相关性方面有明显提升。本研究展示了RAG技术在文化旅游信息标准化与数据分析方面的潜力，为智能化文化旅游服务系统的发展提供了理论与技术支持。

Title: Understanding Epistemic Language with a Bayesian Theory of Mind

• Authors: Lance Ying, Tan Zhi-Xuan, Lionel Wong, Vikash Mansinghka, Joshua B. Tenenbaum
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12022
• Pdf URL: https://arxiv.org/pdf/2408.12022
• Copy Paste: [[2408.12022]] Understanding Epistemic Language with a Bayesian Theory of Mind(https://arxiv.org/abs/2408.12022)
• Keywords: gpt, llm, agent
• Abstract: How do people understand and evaluate claims about others' beliefs, even though these beliefs cannot be directly observed? In this paper, we introduce a cognitive model of epistemic language interpretation, grounded in Bayesian inferences about other agents' goals, beliefs, and intentions: a language-augmented Bayesian theory-of-mind (LaBToM). By translating natural language into an epistemic ``language-of-thought'', then evaluating these translations against the inferences produced by inverting a probabilistic generative model of rational action and perception, LaBToM captures graded plausibility judgments about epistemic claims. We validate our model in an experiment where participants watch an agent navigate a maze to find keys hidden in boxes needed to reach their goal, then rate sentences about the agent's beliefs. In contrast with multimodal LLMs (GPT-4o, Gemini Pro) and ablated models, our model correlates highly with human judgments for a wide range of expressions, including modal language, uncertainty expressions, knowledge claims, likelihood comparisons, and attributions of false belief.
• 摘要：即使无法直接观察到他人的信念，人们如何理解和评估这些信念？在本文中，我们引入了一种认知语言解释的认知模型，该模型基于对其他代理的目标、信念和意图的贝叶斯推理：语言增强贝叶斯心智理论 (LaBToM)。通过将自然语言翻译成认知的“思维语言”，然后根据通过反转理性行为和感知的概率生成模型产生的推论来评估这些翻译，LaBToM 可以捕获关于认知主张的分级可信度判断。我们通过一项实验验证了我们的模型，实验中参与者观察代理在迷宫中寻找隐藏在盒子中的钥匙，以达到目标，然后对有关代理信念的句子进行评分。与多模态 LLM（GPT-4o、Gemini Pro）和烧蚀模型相比，我们的模型与人类对各种表达的判断高度相关，包括模态语言、不确定性表达、知识主张、可能性比较和错误信念的归因。

Title: Aligning (Medical) LLMs for (Counterfactual) Fairness

• Authors: Raphael Poulain, Hamed Fayyaz, Rahmatollah Beheshti
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12055
• Pdf URL: https://arxiv.org/pdf/2408.12055
• Copy Paste: [[2408.12055]] Aligning (Medical) LLMs for (Counterfactual) Fairness(https://arxiv.org/abs/2408.12055)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have emerged as promising solutions for a variety of medical and clinical decision support applications. However, LLMs are often subject to different types of biases, which can lead to unfair treatment of individuals, worsening health disparities, and reducing trust in AI-augmented medical tools. Aiming to address this important issue, in this study, we present a new model alignment approach for aligning LLMs using a preference optimization method within a knowledge distillation framework. Prior to presenting our proposed method, we first use an evaluation framework to conduct a comprehensive (largest to our knowledge) empirical evaluation to reveal the type and nature of existing biases in LLMs used for medical applications. We then offer a bias mitigation technique to reduce the unfair patterns in LLM outputs across different subgroups identified by the protected attributes. We show that our mitigation method is effective in significantly reducing observed biased patterns. Our code is publicly available at \url{this https URL}.
• 摘要：大型语言模型 (LLM) 已成为各种医疗和临床决策支持应用的有前途的解决方案。然而，LLM 通常会受到不同类型的偏见的影响，这可能导致对个人的不公平待遇、加剧健康差距并降低对人工智能增强医疗工具的信任。为了解决这一重要问题，在本研究中，我们提出了一种新的模型对齐方法，用于在知识蒸馏框架内使用偏好优化方法来对齐 LLM。在介绍我们提出的方法之前，我们首先使用评估框架进行全面（据我们所知最大的）实证评估，以揭示用于医疗应用的 LLM 中现有偏见的类型和性质。然后，我们提供了一种偏见缓解技术，以减少受保护属性标识的不同子组中 LLM 输出中的不公平模式。我们表明，我们的缓解方法可以有效地显着减少观察到的偏见模式。我们的代码可在 \url{this https URL} 上公开获取。

Title: Evidence-backed Fact Checking using RAG and Few-Shot In-Context Learning with LLMs

• Authors: Ronit Singhal, Pransh Patwa, Parth Patwa, Aman Chadha, Amitava Das
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12060
• Pdf URL: https://arxiv.org/pdf/2408.12060
• Copy Paste: [[2408.12060]] Evidence-backed Fact Checking using RAG and Few-Shot In-Context Learning with LLMs(https://arxiv.org/abs/2408.12060)
• Keywords: language model, llm
• Abstract: Given the widespread dissemination of misinformation on social media, implementing fact-checking mechanisms for online claims is essential. Manually verifying every claim is highly challenging, underscoring the need for an automated fact-checking system. This paper presents our system designed to address this issue. We utilize the Averitec dataset to assess the veracity of claims. In addition to veracity prediction, our system provides supporting evidence, which is extracted from the dataset. We develop a Retrieve and Generate (RAG) pipeline to extract relevant evidence sentences from a knowledge base, which are then inputted along with the claim into a large language model (LLM) for classification. We also evaluate the few-shot In-Context Learning (ICL) capabilities of multiple LLMs. Our system achieves an 'Averitec' score of 0.33, which is a 22% absolute improvement over the baseline. All code will be made available on All code will be made available on this https URL.
• 摘要：鉴于社交媒体上虚假信息的广泛传播，实施在线声明的事实核查机制至关重要。手动验证每个声明极具挑战性，这凸显了对自动化事实核查系统的需求。本文介绍了我们为解决此问题而设计的系统。我们利用 Averitec 数据集来评估声明的真实性。除了真实性预测之外，我们的系统还提供从数据集中提取的支持证据。我们开发了一个检索和生成 (RAG) 管道，从知识库中提取相关证据句子，然后将其与声明一起输入到大型语言模型 (LLM) 中进行分类。我们还评估了多个 LLM 的少样本上下文学习 (ICL) 能力。我们的系统获得了 0.33 的“Averitec”分数，比基线绝对提高了 22%。所有代码都将在此 https URL 上提供。

Title: ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLM

• Authors: Zhaochen Su, Jun Zhang, Xiaoye Qu, Tong Zhu, Yanshu Li, Jiashuo Sun, Juntao Li, Min Zhang, Yu Cheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12076
• Pdf URL: https://arxiv.org/pdf/2408.12076
• Copy Paste: [[2408.12076]] ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLM(https://arxiv.org/abs/2408.12076)
• Keywords: language model, llm, hallucination
• Abstract: Large language models (LLMs) have achieved impressive advancements across numerous disciplines, yet the critical issue of knowledge conflicts, a major source of hallucinations, has rarely been studied. Only a few research explored the conflicts between the inherent knowledge of LLMs and the retrieved contextual knowledge. However, a thorough assessment of knowledge conflict in LLMs is still missing. Motivated by this research gap, we present ConflictBank, the first comprehensive benchmark developed to systematically evaluate knowledge conflicts from three aspects: (i) conflicts encountered in retrieved knowledge, (ii) conflicts within the models' encoded knowledge, and (iii) the interplay between these conflict forms. Our investigation delves into four model families and twelve LLM instances, meticulously analyzing conflicts stemming from misinformation, temporal discrepancies, and semantic divergences. Based on our proposed novel construction framework, we create 7,453,853 claim-evidence pairs and 553,117 QA pairs. We present numerous findings on model scale, conflict causes, and conflict types. We hope our ConflictBank benchmark will help the community better understand model behavior in conflicts and develop more reliable LLMs.
• 摘要：大型语言模型 (LLM) 在许多学科中取得了令人瞩目的进步，但知识冲突这一关键问题（幻觉的主要来源）却很少得到研究。只有少数研究探讨了 LLM 固有知识与检索到的上下文知识之间的冲突。然而，对 LLM 中的知识冲突的全面评估仍然缺失。受此研究空白的启发，我们提出了 ConflictBank，这是第一个全面的基准，旨在从三个方面系统地评估知识冲突：(i) 检索到的知识中遇到的冲突，(ii) 模型编码知识中的冲突，以及 (iii) 这些冲突形式之间的相互作用。我们的调查深入研究了四个模型系列和十二个 LLM 实例，细致地分析了由错误信息、时间差异和语义分歧引起的冲突。基于我们提出的新构建框架，我们创建了 7,453,853 个声明-证据对和 553,117 个 QA 对。我们在模型规模、冲突原因和冲突类型方面提出了许多发现。我们希望我们的 ConflictBank 基准将帮助社区更好地理解冲突中的模型行为并开发更可靠的 LLM。

Title: uMedSum: A Unified Framework for Advancing Medical Abstractive Summarization

• Authors: Aishik Nagar, Yutong Liu, Andy T. Liu, Viktor Schlegel, Vijay Prakash Dwivedi, Arun-Kumar Kaliya-Perumal, Guna Pratheep Kalanchiam, Yili Tang, Robby T. Tan
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12095
• Pdf URL: https://arxiv.org/pdf/2408.12095
• Copy Paste: [[2408.12095]] uMedSum: A Unified Framework for Advancing Medical Abstractive Summarization(https://arxiv.org/abs/2408.12095)
• Keywords: gpt
• Abstract: Medical abstractive summarization faces the challenge of balancing faithfulness and informativeness. Current methods often sacrifice key information for faithfulness or introduce confabulations when prioritizing informativeness. While recent advancements in techniques like in-context learning (ICL) and fine-tuning have improved medical summarization, they often overlook crucial aspects such as faithfulness and informativeness without considering advanced methods like model reasoning and self-improvement. Moreover, the field lacks a unified benchmark, hindering systematic evaluation due to varied metrics and datasets. This paper addresses these gaps by presenting a comprehensive benchmark of six advanced abstractive summarization methods across three diverse datasets using five standardized metrics. Building on these findings, we propose uMedSum, a modular hybrid summarization framework that introduces novel approaches for sequential confabulation removal followed by key missing information addition, ensuring both faithfulness and informativeness. Our work improves upon previous GPT-4-based state-of-the-art (SOTA) medical summarization methods, significantly outperforming them in both quantitative metrics and qualitative domain expert evaluations. Notably, we achieve an average relative performance improvement of 11.8% in reference-free metrics over the previous SOTA. Doctors prefer uMedSum's summaries 6 times more than previous SOTA in difficult cases where there are chances of confabulations or missing information. These results highlight uMedSum's effectiveness and generalizability across various datasets and metrics, marking a significant advancement in medical summarization.
• 摘要：医学抽象概括面临着平衡忠实性和信息量的挑战。当前的方法通常会牺牲关键信息以求忠实，或者在优先考虑信息量时引入虚构内容。虽然近期在情境学习 (ICL) 和微调等技术方面的进步改善了医学概括，但它们往往忽视了忠实性和信息量等关键方面，而没有考虑模型推理和自我改进等高级方法。此外，该领域缺乏统一的基准，由于指标和数据集各异，阻碍了系统评估。本文通过使用五个标准化指标，在三个不同的数据集上对六种高级抽象概括方法进行了全面的基准测试，解决了这些差距。基于这些发现，我们提出了 uMedSum，这是一个模块化混合概括框架，它引入了新方法，用于顺序删除虚构内容，然后添加关键缺失信息，确保忠实性和信息量。我们的工作改进了之前基于 GPT-4 的最先进的 (SOTA) 医学摘要方法，在定量指标和定性领域专家评估方面均显著优于它们。值得注意的是，与之前的 SOTA 相比，我们在无参考指标中实现了 11.8% 的平均相对性能提升。在可能出现虚构或信息缺失的困难病例中，医生更喜欢 uMedSum 的摘要，而不是之前的 SOTA 的 6 倍。这些结果凸显了 uMedSum 在各种数据集和指标中的有效性和通用性，标志着医学摘要的重大进步。

Title: MDD-5k: A New Diagnostic Conversation Dataset for Mental Disorders Synthesized via Neuro-Symbolic LLM Agents

• Authors: Congchi Yin, Feng Li, Shu Zhang, Zike Wang, Jun Shao, Piji Li, Jianhua Chen, Xun Jiang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12142
• Pdf URL: https://arxiv.org/pdf/2408.12142
• Copy Paste: [[2408.12142]] MDD-5k: A New Diagnostic Conversation Dataset for Mental Disorders Synthesized via Neuro-Symbolic LLM Agents(https://arxiv.org/abs/2408.12142)
• Keywords: language model, llm, agent
• Abstract: The clinical diagnosis of most mental disorders primarily relies on the conversations between psychiatrist and patient. The creation of such diagnostic conversation datasets is promising to boost the AI mental healthcare community. However, directly collecting the conversations in real diagnosis scenarios is near impossible due to stringent privacy and ethical considerations. To address this issue, we seek to synthesize diagnostic conversation by exploiting anonymous patient cases that are easier to access. Specifically, we design a neuro-symbolic multi-agent framework for synthesizing the diagnostic conversation of mental disorders with large language models. It takes patient case as input and is capable of generating multiple diverse conversations with one single patient case. The framework basically involves the interaction between a doctor agent and a patient agent, and achieves text generation under symbolic control via a dynamic diagnosis tree from a tool agent. By applying the proposed framework, we develop the largest Chinese mental disorders diagnosis dataset MDD-5k, which is built upon 1000 cleaned real patient cases by cooperating with a pioneering psychiatric hospital, and contains 5000 high-quality long conversations with diagnosis results as labels. To the best of our knowledge, it's also the first labelled Chinese mental disorders diagnosis dataset. Human evaluation demonstrates the proposed MDD-5k dataset successfully simulates human-like diagnostic process of mental disorders. The dataset and code will become publicly accessible in this https URL.
• 摘要：大多数精神疾病的临床诊断主要依赖于精神科医生和患者之间的对话。此类诊断对话数据集的创建有望推动人工智能精神医疗社区的发展。然而，由于严格的隐私和道德考虑，直接收集真实诊断场景中的对话几乎是不可能的。为了解决这个问题，我们试图通过利用更容易访问的匿名患者案例来合成诊断对话。具体来说，我们设计了一个神经符号多智能体框架，用于使用大型语言模型合成精神疾病的诊断对话。它以患者案例为输入，能够用一个患者案例生成多个不同的对话。该框架基本上涉及医生代理和患者代理之间的交互，并通过工具代理的动态诊断树在符号控制下实现文本生成。通过应用所提出的框架，我们开发了最大的中国精神疾病诊断数据集 MDD-5k，该数据集建立在 1000 个经过清理的真实患者案例的基础上，与一家先驱精神病院合作，包含 5000 个高质量的长对话，并以诊断结果为标签。据我们所知，这也是第一个带标签的中国精神障碍诊断数据集。人工评估表明，拟议的 MDD-5k 数据集成功模拟了类似人类的精神障碍诊断过程。该数据集和代码将在此 https URL 中公开访问。

Title: Implicit Sentiment Analysis Based on Chain of Thought Prompting

• Authors: Zhihua Duan, Jialin Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12157
• Pdf URL: https://arxiv.org/pdf/2408.12157
• Copy Paste: [[2408.12157]] Implicit Sentiment Analysis Based on Chain of Thought Prompting(https://arxiv.org/abs/2408.12157)
• Keywords: language model, prompt
• Abstract: Implicit Sentiment Analysis (ISA) is a crucial research area in natural language processing. Inspired by the idea of large language model Chain of Thought (CoT), this paper introduces a Sentiment Analysis of Thinking (SAoT) framework. The framework first analyzes the implicit aspects and opinions in the text using common sense and thinking chain capabilities. Then, it reflects on the process of implicit sentiment analysis and finally deduces the polarity of sentiment. The model is evaluated on the SemEval 2014 dataset, consisting of 1120 restaurant reviews and 638 laptop reviews. The experimental results demonstrate that the utilization of the ERNIE-Bot-4+SAoT model yields a notable performance improvement. Specifically, on the restaurant dataset, the F1 score reaches 75.27, accompanied by an ISA score of 66.29. Similarly, on the computer dataset, the F1 score achieves 76.50, while the ISA score amounts to 73.46. Comparatively, the ERNIE-Bot-4+SAoT model surpasses the BERTAsp + SCAPt baseline by an average margin of 47.99%.
• 摘要：隐性情感分析（ISA）是自然语言处理中的一个重要研究领域。受大型语言模型思维链（CoT）思想的启发，本文提出了一种思维情感分析（SAoT）框架。该框架首先利用常识和思维链能力分析文本中隐含的方面和观点，然后反思隐性情感分析的过程，最终推断出情感的极性。该模型在SemEval 2014数据集上进行了评估，该数据集包含1120条餐厅评论和638条笔记本电脑评论。实验结果表明，使用ERNIE-Bot-4+SAoT模型可以获得明显的性能提升。具体而言，在餐厅数据集上，F1得分达到75.27，ISA得分为66.29。同样，在计算机数据集上，F1得分达到76.50，而ISA得分为73.46。相比之下，ERNIE-Bot-4+SAoT 模型平均比 BERTAsp + SCAPt 基线高出 47.99%。

Title: Preference-Guided Reflective Sampling for Aligning Language Models

• Authors: Hai Ye, Hwee Tou Ng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12163
• Pdf URL: https://arxiv.org/pdf/2408.12163
• Copy Paste: [[2408.12163]] Preference-Guided Reflective Sampling for Aligning Language Models(https://arxiv.org/abs/2408.12163)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are aligned with human preferences by reinforcement learning from human feedback (RLHF). Effective data sampling is crucial for RLHF, as it determines the efficiency of model training, ensuring that models learn from the informative samples. To achieve better data generation, we propose a new sampling method called Preference-Guided Reflective Sampling (PRS). PRS frames the response generation as an optimization process to the explicitly specified user preference described in natural language. It employs a tree-based generation framework to enable an efficient sampling process, which guides the direction of generation through preference and better explores the sampling space with adaptive self-refinement. Notably, PRS can align LLMs to diverse preferences. We study preference-controlled text generation for instruction following and keyword-focused document summarization. Our findings indicate that PRS, across different LLM policies, generates training data with much higher rewards than strong baselines. PRS also excels in post-RL training.
• 摘要：通过从人类反馈中强化学习 (RLHF)，大型语言模型 (LLM) 与人类偏好保持一致。有效的数据采样对于 RLHF 至关重要，因为它决定了模型训练的效率，确保模型从信息样本中学习。为了实现更好的数据生成，我们提出了一种称为偏好引导反射采样 (PRS) 的新采样方法。PRS 将响应生成框定为对自然语言中描述的明确指定的用户偏好的优化过程。它采用基于树的生成框架来实现高效的采样过程，该过程通过偏好引导生成方向，并通过自适应自我细化更好地探索采样空间。值得注意的是，PRS 可以将 LLM 与不同的偏好保持一致。我们研究了偏好控制的文本生成，以实现指令遵循和以关键字为中心的文档摘要。我们的研究结果表明，在不同的 LLM 策略中，PRS 生成的训练数据的奖励远高于强基线。PRS 在后强化学习训练中也表现出色。

Title: FIRST: Teach A Reliable Large Language Model Through Efficient Trustworthy Distillation

• Authors: KaShun Shum, Minrui Xu, Jianshu Zhang, Zixin Chen, Shizhe Diao, Hanze Dong, Jipeng Zhang, Muhammad Omer Raza
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12168
• Pdf URL: https://arxiv.org/pdf/2408.12168
• Copy Paste: [[2408.12168]] FIRST: Teach A Reliable Large Language Model Through Efficient Trustworthy Distillation(https://arxiv.org/abs/2408.12168)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have become increasingly prevalent in our daily lives, leading to an expectation for LLMs to be trustworthy -- - both accurate and well-calibrated (the prediction confidence should align with its ground truth correctness likelihood). Nowadays, fine-tuning has become the most popular method for adapting a model to practical usage by significantly increasing accuracy on downstream tasks. Despite the great accuracy it achieves, we found fine-tuning is still far away from satisfactory trustworthiness due to "tuning-induced mis-calibration". In this paper, we delve deeply into why and how mis-calibration exists in fine-tuned models, and how distillation can alleviate the issue. Then we further propose a brand new method named Efficient Trustworthy Distillation (FIRST), which utilizes a small portion of teacher's knowledge to obtain a reliable language model in a cost-efficient way. Specifically, we identify the "concentrated knowledge" phenomenon during distillation, which can significantly reduce the computational burden. Then we apply a "trustworthy maximization" process to optimize the utilization of this small portion of concentrated knowledge before transferring it to the student. Experimental results demonstrate the effectiveness of our method, where better accuracy (+2.3%) and less mis-calibration (-10%) are achieved on average across both in-domain and out-of-domain scenarios, indicating better trustworthiness.
• 摘要：大型语言模型 (LLM) 在我们的日常生活中越来越普遍，因此人们期望 LLM 是值得信赖的——既准确又经过良好校准（预测置信度应与其基本事实正确性可能性相一致）。如今，微调已成为最流行的方法，通过显著提高下游任务的准确性，使模型适应实际使用。尽管它达到了很高的准确性，但我们发现由于“调整引起的错误校准”，微调仍然远远达不到令人满意的可信度。在本文中，我们深入研究了微调模型中存在错误校准的原因和方式，以及蒸馏如何缓解这一问题。然后，我们进一步提出了一种名为高效可信蒸馏 (FIRST) 的全新方法，该方法利用教师的一小部分知识以经济高效的方式获得可靠的语言模型。具体而言，我们在蒸馏过程中识别了“集中知识”现象，这可以显著减轻计算负担。然后，我们应用“可信度最大化”流程来优化这一小部分集中知识的利用率，然后再将其传授给学生。实验结果证明了我们方法的有效性，在域内和域外场景中，平均准确率更高（+2.3%），校准误差更少（-10%），这表明可信度更高。

Title: Reasoning Factual Knowledge in Structured Data with Large Language Models

• Authors: Sirui Huang, Yanggan Gu, Xuming Hu, Zhonghao Li, Qing Li, Guandong Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12188
• Pdf URL: https://arxiv.org/pdf/2408.12188
• Copy Paste: [[2408.12188]] Reasoning Factual Knowledge in Structured Data with Large Language Models(https://arxiv.org/abs/2408.12188)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have made remarkable progress in various natural language processing tasks as a benefit of their capability to comprehend and reason with factual knowledge. However, a significant amount of factual knowledge is stored in structured data, which possesses unique characteristics that differ from the unstructured texts used for pretraining. This difference can introduce imperceptible inference parameter deviations, posing challenges for LLMs in effectively utilizing and reasoning with structured data to accurately infer factual knowledge. To this end, we propose a benchmark named StructFact, to evaluate the structural reasoning capabilities of LLMs in inferring factual knowledge. StructFact comprises 8,340 factual questions encompassing various tasks, domains, timelines, and regions. This benchmark allows us to investigate the capability of LLMs across five factual tasks derived from the unique characteristics of structural facts. Extensive experiments on a set of LLMs with different training strategies reveal the limitations of current LLMs in inferring factual knowledge from structured data. We present this benchmark as a compass to navigate the strengths and weaknesses of LLMs in reasoning with structured data for knowledge-sensitive tasks, and to encourage advancements in related real-world applications. Please find our code at this https URL.
• 摘要：大型语言模型 (LLM) 凭借其理解和推理事实知识的能力，在各种自然语言处理任务中取得了显著进展。然而，大量事实知识存储在结构化数据中，这些数据具有与用于预训练的非结构化文本不同的独特特征。这种差异可能会引入难以察觉的推理参数偏差，对 LLM 有效利用和推理结构化数据以准确推断事实知识提出了挑战。为此，我们提出了一个名为 StructFact 的基准，以评估 LLM 在推断事实知识方面的结构化推理能力。StructFact 包含 8,340 个事实问题，涵盖各种任务、领域、时间线和区域。该基准使我们能够从结构事实的独特特征中研究 LLM 在五个事实任务中的能力。对一组具有不同训练策略的 LLM 进行的大量实验揭示了当前 LLM 在从结构化数据推断事实知识方面的局限性。我们将此基准作为指南针，以了解 LLM 在知识敏感任务中使用结构化数据进行推理的优势和劣势，并促进相关实际应用的进步。请在此 https URL 上找到我们的代码。

Title: Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment

• Authors: Kun Luo, Minghao Qin, Zheng Liu, Shitao Xiao, Jun Zhao, Kang Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12194
• Pdf URL: https://arxiv.org/pdf/2408.12194
• Copy Paste: [[2408.12194]] Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment(https://arxiv.org/abs/2408.12194)
• Keywords: language model, llm
• Abstract: Pretrained language models like BERT and T5 serve as crucial backbone encoders for dense retrieval. However, these models often exhibit limited generalization capabilities and face challenges in improving in domain accuracy. Recent research has explored using large language models (LLMs) as retrievers, achieving SOTA performance across various tasks. Despite these advancements, the specific benefits of LLMs over traditional retrievers and the impact of different LLM configurations, such as parameter sizes, pretraining duration, and alignment processes on retrieval tasks remain unclear. In this work, we conduct a comprehensive empirical study on a wide range of retrieval tasks, including in domain accuracy, data efficiency, zero shot generalization, lengthy retrieval, instruction based retrieval, and multi task learning. We evaluate over 15 different backbone LLMs and non LLMs. Our findings reveal that larger models and extensive pretraining consistently enhance in domain accuracy and data efficiency. Additionally, larger models demonstrate significant potential in zero shot generalization, lengthy retrieval, instruction based retrieval, and multi task learning. These results underscore the advantages of LLMs as versatile and effective backbone encoders in dense retrieval, providing valuable insights for future research and development in this field.
• 摘要：预训练语言模型（如 BERT 和 T5）是密集检索的关键骨干编码器。然而，这些模型通常表现出有限的泛化能力，并且在提高领域准确性方面面临挑战。最近的研究探索了使用大型语言模型 (LLM) 作为检索器，在各种任务中实现 SOTA 性能。尽管取得了这些进步，但 LLM 相对于传统检索器的具体优势以及不同 LLM 配置（例如参数大小、预训练持续时间和对齐过程）对检索任务的影响仍不清楚。在这项工作中，我们对广泛的检索任务进行了全面的实证研究，包括领域准确性、数据效率、零样本泛化、长时间检索、基于指令的检索和多任务学习。我们评估了 15 多种不同的骨干 LLM 和非 LLM。我们的研究结果表明，更大的模型和广泛的预训练可以持续提高领域准确性和数据效率。此外，更大的模型在零样本泛化、长时间检索、基于指令的检索和多任务学习方面表现出巨大的潜力。这些结果强调了 LLM 作为密集检索中多功能和有效的主干编码器的优势，为该领域未来的研究和发展提供了宝贵的见解。

Title: EvalYaks: Instruction Tuning Datasets and LoRA Fine-tuned Models for Automated Scoring of CEFR B2 Speaking Assessment Transcripts

• Authors: Nicy Scaria, Silvester John Joseph Kennedy, Thomas Latinovich, Deepak Subramani
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12226
• Pdf URL: https://arxiv.org/pdf/2408.12226
• Copy Paste: [[2408.12226]] EvalYaks: Instruction Tuning Datasets and LoRA Fine-tuned Models for Automated Scoring of CEFR B2 Speaking Assessment Transcripts(https://arxiv.org/abs/2408.12226)
• Keywords: language model, llm
• Abstract: Relying on human experts to evaluate CEFR speaking assessments in an e-learning environment creates scalability challenges, as it limits how quickly and widely assessments can be conducted. We aim to automate the evaluation of CEFR B2 English speaking assessments in e-learning environments from conversation transcripts. First, we evaluate the capability of leading open source and commercial Large Language Models (LLMs) to score a candidate's performance across various criteria in the CEFR B2 speaking exam in both global and India-specific contexts. Next, we create a new expert-validated, CEFR-aligned synthetic conversational dataset with transcripts that are rated at different assessment scores. In addition, new instruction-tuned datasets are developed from the English Vocabulary Profile (up to CEFR B2 level) and the CEFR-SP WikiAuto datasets. Finally, using these new datasets, we perform parameter efficient instruction tuning of Mistral Instruct 7B v0.2 to develop a family of models called EvalYaks. Four models in this family are for assessing the four sections of the CEFR B2 speaking exam, one for identifying the CEFR level of vocabulary and generating level-specific vocabulary, and another for detecting the CEFR level of text and generating level-specific text. EvalYaks achieved an average acceptable accuracy of 96%, a degree of variation of 0.35 levels, and performed 3 times better than the next best model. This demonstrates that a 7B parameter LLM instruction tuned with high-quality CEFR-aligned assessment data can effectively evaluate and score CEFR B2 English speaking assessments, offering a promising solution for scalable, automated language proficiency evaluation.
• 摘要：依靠人类专家在电子学习环境中评估 CEFR 口语评估会带来可扩展性挑战，因为这限制了评估的速度和范围。我们的目标是通过对话记录自动评估电子学习环境中的 CEFR B2 英语口语评估。首先，我们评估领先的开源和商业大型语言模型 (LLM) 在全球和印度特定环境中根据 CEFR B2 口语考试的各种标准对考生的表现进行评分的能力。接下来，我们创建一个新的经过专家验证的、与 CEFR 一致的合成对话数据集，其中包含按不同评估分数评分的记录。此外，我们还从英语词汇档案（最高到 CEFR B2 级别）和 CEFR-SP WikiAuto 数据集开发了新的指令调整数据集。最后，使用这些新数据集，我们对 Mistral Instruct 7B v0.2 执行参数有效的指令调整，以开发名为 EvalYaks 的模型系列。该系列中的四个模型用于评估 CEFR B2 口语考试的四个部分，一个用于识别 CEFR 级别的词汇并生成特定级别的词汇，另一个用于检测 CEFR 级别的文本并生成特定级别的文本。EvalYaks 的平均准确度为 96%，变异度为 0.35 级，性能比下一个最佳模型高出 3 倍。这表明，使用高质量的 CEFR 对齐评估数据调整的 7B 参数 LLM 指令可以有效地评估和评分 CEFR B2 英语口语评估，为可扩展的自动化语言能力评估提供了一种有前途的解决方案。

Title: LLMs are not Zero-Shot Reasoners for Biomedical Information Extraction

• Authors: Aishik Nagar, Viktor Schlegel, Thanh-Tung Nguyen, Hao Li, Yuping Wu, Kuluhan Binici, Stefan Winkler
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12249
• Pdf URL: https://arxiv.org/pdf/2408.12249
• Copy Paste: [[2408.12249]] LLMs are not Zero-Shot Reasoners for Biomedical Information Extraction(https://arxiv.org/abs/2408.12249)
• Keywords: language model, llm, prompt, retrieval-augmented generation, chain-of-thought
• Abstract: Large Language Models (LLMs) are increasingly adopted for applications in healthcare, reaching the performance of domain experts on tasks such as question answering and document summarisation. Despite their success on these tasks, it is unclear how well LLMs perform on tasks that are traditionally pursued in the biomedical domain, such as structured information extration. To breach this gap, in this paper, we systematically benchmark LLM performance in Medical Classification and Named Entity Recognition (NER) tasks. We aim to disentangle the contribution of different factors to the performance, particularly the impact of LLMs' task knowledge and reasoning capabilities, their (parametric) domain knowledge, and addition of external knowledge. To this end we evaluate various open LLMs -- including BioMistral and Llama-2 models -- on a diverse set of biomedical datasets, using standard prompting, Chain-of-Thought (CoT) and Self-Consistency based reasoning as well as Retrieval-Augmented Generation (RAG) with PubMed and Wikipedia corpora. Counter-intuitively, our results reveal that standard prompting consistently outperforms more complex techniques across both tasks, laying bare the limitations in the current application of CoT, self-consistency and RAG in the biomedical domain. Our findings suggest that advanced prompting methods developed for knowledge- or reasoning-intensive tasks, such as CoT or RAG, are not easily portable to biomedical tasks where precise structured outputs are required. This highlights the need for more effective integration of external knowledge and reasoning mechanisms in LLMs to enhance their performance in real-world biomedical applications.
• 摘要：大型语言模型 (LLM) 越来越多地被应用于医疗保健领域，在问答和文档摘要等任务上达到了领域专家的表现。尽管 LLM 在这些任务上取得了成功，但尚不清楚它们在传统上在生物医学领域进行的任务（例如结构化信息提取）上的表现如何。为了弥补这一差距，在本文中，我们系统地对 LLM 在医学分类和命名实体识别 (NER) 任务中的表现进行了基准测试。我们的目标是理清不同因素对性能的贡献，特别是 LLM 的任务知识和推理能力、它们的（参数）领域知识以及外部知识的增加的影响。为此，我们在一组不同的生物医学数据集上评估了各种开放的 LLM（包括 BioMistral 和 Llama-2 模型），使用标准提示、基于思路链 (CoT) 和自洽的推理以及使用 PubMed 和 Wikipedia 语料库的检索增强生成 (RAG)。与直觉相反的是，我们的结果表明，标准提示在两项任务中的表现始终优于更复杂的技术，这暴露了 CoT、自洽性和 RAG 在生物医学领域的当前应用的局限性。我们的研究结果表明，为知识或推理密集型任务开发的高级提示方法（如 CoT 或 RAG）不易移植到需要精确结构化输出的生物医学任务中。这凸显了 LLM 需要更有效地整合外部知识和推理机制，以提高其在现实世界生物医学应用中的表现。

Title: Toward the Evaluation of Large Language Models Considering Score Variance across Instruction Templates

• Authors: Yusuke Sakai, Adam Nohejl, Jiangnan Hang, Hidetaka Kamigaito, Taro Watanabe
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12263
• Pdf URL: https://arxiv.org/pdf/2408.12263
• Copy Paste: [[2408.12263]] Toward the Evaluation of Large Language Models Considering Score Variance across Instruction Templates(https://arxiv.org/abs/2408.12263)
• Keywords: language model, llm, prompt
• Abstract: The natural language understanding (NLU) performance of large language models (LLMs) has been evaluated across various tasks and datasets. The existing evaluation methods, however, do not take into account the variance in scores due to differences in prompts, which leads to unfair evaluation and comparison of NLU performance. Moreover, evaluation designed for specific prompts is inappropriate for instruction tuning, which aims to perform well with any prompt. It is therefore necessary to find a way to measure NLU performance in a fair manner, considering score variance between different instruction templates. In this study, we provide English and Japanese cross-lingual datasets for evaluating the NLU performance of LLMs, which include multiple instruction templates for fair evaluation of each task, along with regular expressions to constrain the output format. Furthermore, we propose the Sharpe score as an evaluation metric that takes into account the variance in scores between templates. Comprehensive analysis of English and Japanese LLMs reveals that the high variance among templates has a significant impact on the fair evaluation of LLMs.
• 摘要：大型语言模型 (LLM) 的自然语言理解 (NLU) 性能已在各种任务和数据集中得到评估。然而，现有的评估方法没有考虑到由于提示不同而导致的分数差异，这导致对 NLU 性能的评估和比较不公平。此外，针对特定提示设计的评估不适合指令调整，指令调整旨在在任何提示下表现良好。因此，有必要找到一种公平地衡量 NLU 性能的方法，考虑到不同指令模板之间的分数差异。在本研究中，我们提供了英语和日语跨语言数据集来评估 LLM 的 NLU 性能，其中包括多个指令模板以公平评估每个任务，以及用于约束输出格式的正则表达式。此外，我们提出了 Sharpe 分数作为评估指标，该指标考虑了模板之间的分数差异。对英语和日语 LLM 的综合分析表明，模板之间的高差异对 LLM 的公平评估有显著影响。

Title: MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model

• Authors: Chaoya Jiang, Jia Hongrui, Haiyang Xu, Wei Ye, Mengfan Dong, Ming Yan, Ji Zhang, Fei Huang, Shikun Zhang
• Subjects: cs.CL, cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.12321
• Pdf URL: https://arxiv.org/pdf/2408.12321
• Copy Paste: [[2408.12321]] MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model(https://arxiv.org/abs/2408.12321)
• Keywords: language model, llm
• Abstract: This paper presents MaVEn, an innovative Multi-granularity Visual Encoding framework designed to enhance the capabilities of Multimodal Large Language Models (MLLMs) in multi-image reasoning. Current MLLMs primarily focus on single-image visual understanding, limiting their ability to interpret and integrate information across multiple images. MaVEn addresses this limitation by combining discrete visual symbol sequences, which abstract coarse-grained semantic concepts, with traditional continuous representation sequences that model fine-grained features. This dual approach bridges the semantic gap between visual and textual data, thereby improving the model's ability to process and interpret information from multiple images effectively. Additionally, we design a dynamic reduction mechanism by for long-sequence continuous features to enhance multi-image processing efficiency. Experimental results demonstrate that MaVEn significantly enhances MLLMs' understanding in complex multi-image scenarios, while also improving performance in single-image contexts.
• 摘要：本文介绍了一种创新的多粒度视觉编码框架 MaVEn，旨在增强多模态大型语言模型 (MLLM) 在多图像推理中的能力。当前的 MLLM 主要专注于单图像视觉理解，这限制了它们解释和整合多图像信息的能力。MaVEn 通过将抽象粗粒度语义概念的离散视觉符号序列与模拟细粒度特征的传统连续表示序列相结合来解决这一限制。这种双重方法弥合了视觉和文本数据之间的语义鸿沟，从而提高了模型有效处理和解释多图像信息的能力。此外，我们设计了一种针对长序列连续特征的动态缩减机制，以提高多图像处理效率。实验结果表明，MaVEn 显著增强了 MLLM 在复杂多图像场景中的理解能力，同时也提高了单图像环境中的性能。

Title: Improving Factuality in Large Language Models via Decoding-Time Hallucinatory and Truthful Comparators

• Authors: Dingkang Yang, Dongling Xiao, Jinjie Wei, Mingcheng Li, Zhaoyu Chen, Ke Li, Lihua Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12325
• Pdf URL: https://arxiv.org/pdf/2408.12325
• Copy Paste: [[2408.12325]] Improving Factuality in Large Language Models via Decoding-Time Hallucinatory and Truthful Comparators(https://arxiv.org/abs/2408.12325)
• Keywords: language model, llm, hallucination
• Abstract: Despite their remarkable capabilities, Large Language Models (LLMs) are prone to generate responses that contradict verifiable facts, i.e., unfaithful hallucination content. Existing efforts generally focus on optimizing model parameters or editing semantic representations, which compromise the internal factual knowledge of target LLMs. In addition, hallucinations typically exhibit multifaceted patterns in downstream tasks, limiting the model's holistic performance across tasks. In this paper, we propose a Comparator-driven Decoding-Time (CDT) framework to alleviate the response hallucination. Firstly, we construct hallucinatory and truthful comparators with multi-task fine-tuning samples. In this case, we present an instruction prototype-guided mixture of experts strategy to enhance the ability of the corresponding comparators to capture different hallucination or truthfulness patterns in distinct task instructions. CDT constrains next-token predictions to factuality-robust distributions by contrasting the logit differences between the target LLMs and these comparators. Systematic experiments on multiple downstream tasks show that our framework can significantly improve the model performance and response factuality.
• 摘要：尽管大型语言模型 (LLM) 具有出色的能力，但它们很容易产生与可验证事实相矛盾的反应，即不真实的幻觉内容。现有的努力通常侧重于优化模型参数或编辑语义表示，这会损害目标 LLM 的内部事实知识。此外，幻觉通常在下游任务中表现出多方面的模式，限制了模型在各个任务中的整体性能。在本文中，我们提出了一个比较器驱动的解码时间 (CDT) 框架来缓解反应幻觉。首先，我们使用多任务微调样本构建幻觉和真实比较器。在这种情况下，我们提出了一种指令原型引导的专家混合策略，以增强相应比较器在不同任务指令中捕获不同幻觉或真实模式的能力。CDT 通过对比目标 LLM 和这些比较器之间的逻辑差异，将下一个标记预测限制为事实稳健分布。在多个下游任务上的系统实验表明，我们的框架可以显著提高模型性能和响应真实性。

Title: Interactive DualChecker for Mitigating Hallucinations in Distilling Large Language Models

• Authors: Meiyun Wang, Masahiro Suzuki, Hiroki Sakaji, Kiyoshi Izumi
• Subjects: cs.CL, cs.AI, cs.CE, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.12326
• Pdf URL: https://arxiv.org/pdf/2408.12326
• Copy Paste: [[2408.12326]] Interactive DualChecker for Mitigating Hallucinations in Distilling Large Language Models(https://arxiv.org/abs/2408.12326)
• Keywords: language model, llm, hallucination, prompt
• Abstract: Large Language Models (LLMs) have demonstrated exceptional capabilities across various machine learning (ML) tasks. Given the high costs of creating annotated datasets for supervised learning, LLMs offer a valuable alternative by enabling effective few-shot in-context learning. However, these models can produce hallucinations, particularly in domains with incomplete knowledge. Additionally, current methods for knowledge distillation using LLMs often struggle to enhance the effectiveness of both teacher and student models. To address these challenges, we introduce DualChecker, an innovative framework designed to mitigate hallucinations and improve the performance of both teacher and student models during knowledge distillation. DualChecker employs ContextAligner to ensure that the context provided by teacher models aligns with human labeling standards. It also features a dynamic checker system that enhances model interaction: one component re-prompts teacher models with more detailed content when they show low confidence, and another identifies borderline cases from student models to refine the teaching templates. This interactive process promotes continuous improvement and effective knowledge transfer between the models. We evaluate DualChecker using a green innovation textual dataset that includes binary, multiclass, and token classification tasks. The experimental results show that DualChecker significantly outperforms existing state-of-the-art methods, achieving up to a 17% improvement in F1 score for teacher models and 10% for student models. Notably, student models fine-tuned with LLM predictions perform comparably to those fine-tuned with actual data, even in a challenging domain. We make all datasets, models, and code from this research publicly available.
• 摘要：大型语言模型 (LLM) 在各种机器学习 (ML) 任务中都表现出了卓越的能力。鉴于为监督学习创建带注释的数据集的成本很高，LLM 通过实现有效的少样本上下文学习提供了一种有价值的替代方案。然而，这些模型可能会产生幻觉，尤其是在知识不完整的领域。此外，当前使用 LLM 进行知识提炼的方法通常难以提高教师和学生模型的有效性。为了应对这些挑战，我们推出了 DualChecker，这是一个创新框架，旨在减轻幻觉并提高教师和学生模型在知识提炼过程中的性能。DualChecker 使用 ContextAligner 来确保教师模型提供的上下文符合人类标记标准。它还具有一个动态检查器系统，可增强模型交互：当教师模型显示低置信度时，一个组件会用更详细的内容重新提示教师模型，另一个组件会从学生模型中识别边缘案例以改​​进教学模板。这个互动过程促进了模型之间的持续改进和有效的知识转移。我们使用包含二分类、多分类和 token 分类任务的绿色创新文本数据集对 DualChecker 进行了评估。实验结果表明，DualChecker 的表现明显优于现有的最先进方法，教师模型的 F1 得分提高了 17%，学生模型提高了 10%。值得注意的是，即使在具有挑战性的领域，使用 LLM 预测进行微调的学生模型的表现也与使用实际数据进行微调的模型相当。我们将本研究的所有数据集、模型和代码公开。

Title: Fine-tuning Smaller Language Models for Question Answering over Financial Documents

• Authors: Karmvir Singh Phogat, Sai Akhil Puranam, Sridhar Dasaratha, Chetan Harsha, Shashishekar Ramakrishna
• Subjects: cs.CL, cs.AI, cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2408.12337
• Pdf URL: https://arxiv.org/pdf/2408.12337
• Copy Paste: [[2408.12337]] Fine-tuning Smaller Language Models for Question Answering over Financial Documents(https://arxiv.org/abs/2408.12337)
• Keywords: language model
• Abstract: Recent research has shown that smaller language models can acquire substantial reasoning abilities when fine-tuned with reasoning exemplars crafted by a significantly larger teacher model. We explore this paradigm for the financial domain, focusing on the challenge of answering questions that require multi-hop numerical reasoning over financial texts. We assess the performance of several smaller models that have been fine-tuned to generate programs that encode the required financial reasoning and calculations. Our findings demonstrate that these fine-tuned smaller models approach the performance of the teacher model. To provide a granular analysis of model performance, we propose an approach to investigate the specific student model capabilities that are enhanced by fine-tuning. Our empirical analysis indicates that fine-tuning refines the student models ability to express and apply the required financial concepts along with adapting the entity extraction for the specific data format. In addition, we hypothesize and demonstrate that comparable financial reasoning capability can be induced using relatively smaller datasets.
• 摘要：最近的研究表明，较小的语言模型在使用由较大的教师模型制作的推理样本进行微调时，可以获得强大的推理能力。我们在金融领域探索了这种范式，重点关注回答需要对金融文本进行多跳数字推理的问题的挑战。我们评估了几个较小的模型的性能，这些模型已经过微调，可以生成编码所需金融推理和计算的程序。我们的研究结果表明，这些经过微调的较小模型接近教师模型的性能。为了对模型性能进行细粒度分析，我们提出了一种方法来研究通过微调增强的特定学生模型功能。我们的实证分析表明，微调可以改进学生模型表达和应用所需金融概念的能力，同时调整实体提取以适应特定数据格式。此外，我们假设并证明，使用相对较小的数据集可以诱导出相当的金融推理能力。

Title: Positional Description for Numerical Normalization

• Authors: Deepanshu Gupta, Javier Latorre
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12430
• Pdf URL: https://arxiv.org/pdf/2408.12430
• Copy Paste: [[2408.12430]] Positional Description for Numerical Normalization(https://arxiv.org/abs/2408.12430)
• Keywords: language model
• Abstract: We present a Positional Description Scheme (PDS) tailored for digit sequences, integrating placeholder value information for each digit. Given the structural limitations of subword tokenization algorithms, language models encounter critical Text Normalization (TN) challenges when handling numerical tasks. Our schema addresses this challenge through straightforward pre-processing, preserving the model architecture while significantly simplifying number normalization, rendering the problem tractable. This simplifies the task and facilitates more compact production-ready models capable of learning from smaller datasets. Furthermore, our investigations reveal that PDS enhances the arithmetic processing capabilities of language models, resulting in a relative accuracy improvement of 23% to 51% on complex arithmetic tasks. We demonstrate that PDS effectively mitigates fatal numerical normalization errors in neural models, requiring only a modest amount of training data without rule-based Finite State Transducers (FST). We demonstrate that PDS is essential for both the Text-To-Speech and Speech Recognition text processing, enabling effective TN under production constraints.
• 摘要：我们提出了一种针对数字序列量身定制的位置描述方案 (PDS)，集成了每个数字的占位符值信息。鉴于子词标记化算法的结构限制，语言模型在处理数字任务时会遇到关键的文本规范化 (TN) 挑战。我们的方案通过直接的预处理解决了这一挑战，保留了模型架构，同时显著简化了数字规范化，使问题变得易于处理。这简化了任务并促进了更紧凑的生产就绪模型能够从较小的数据集中学习。此外，我们的调查显示，PDS 增强了语言模型的算术处理能力，导致复杂算术任务的相对准确率提高了 23% 至 51%。我们证明 PDS 有效地减轻了神经模型中致命的数值规范化错误，只需要少量的训练数据，而无需基于规则的有限状态转换器 (FST)。我们证明 PDS 对于文本转语音和语音识别文本处理都至关重要，可以在生产约束下实现有效的 TN。

Title: Enhancing Multi-hop Reasoning through Knowledge Erasure in Large Language Model Editing

• Authors: Mengqi Zhang, Bowen Fang, Qiang Liu, Pengjie Ren, Shu Wu, Zhumin Chen, Liang Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12456
• Pdf URL: https://arxiv.org/pdf/2408.12456
• Copy Paste: [[2408.12456]] Enhancing Multi-hop Reasoning through Knowledge Erasure in Large Language Model Editing(https://arxiv.org/abs/2408.12456)
• Keywords: language model, gpt, llm
• Abstract: Large language models (LLMs) face challenges with internal knowledge inaccuracies and outdated information. Knowledge editing has emerged as a pivotal approach to mitigate these issues. Although current knowledge editing techniques exhibit promising performance in single-hop reasoning tasks, they show limitations when applied to multi-hop reasoning. Drawing on cognitive neuroscience and the operational mechanisms of LLMs, we hypothesize that the residual single-hop knowledge after editing causes edited models to revert to their original answers when processing multi-hop questions, thereby undermining their performance in multihop reasoning tasks. To validate this hypothesis, we conduct a series of experiments that empirically confirm our assumptions. Building on the validated hypothesis, we propose a novel knowledge editing method that incorporates a Knowledge Erasure mechanism for Large language model Editing (KELE). Specifically, we design an erasure function for residual knowledge and an injection function for new knowledge. Through joint optimization, we derive the optimal recall vector, which is subsequently utilized within a rank-one editing framework to update the parameters of targeted model layers. Extensive experiments on GPT-J and GPT-2 XL demonstrate that KELE substantially enhances the multi-hop reasoning capability of edited LLMs.
• 摘要：大型语言模型 (LLM) 面临着内部知识不准确和信息过时的挑战。知识编辑已成为缓解这些问题的关键方法。尽管当前的知识编辑技术在单跳推理任务中表现出良好的性能，但它们在应用于多跳推理时显示出局限性。借鉴认知神经科学和 LLM 的运行机制，我们假设编辑后的残留单跳知识会导致编辑后的模型在处理多跳问题时恢复到其原始答案，从而损害其在多跳推理任务中的表现。为了验证这一假设，我们进行了一系列实验，以经验证实我们的假设。基于已验证的假设，我们提出了一种新颖的知识编辑方法，该方法结合了大型语言模型编辑 (KELE) 的知识擦除机制。具体而言，我们设计了一个用于残留知识的擦除函数和一个用于新知识的注入函数。通过联合优化，我们得出了最佳召回向量，随后将其用于秩一编辑框架中以更新目标模型层的参数。在 GPT-J 和 GPT-2 XL 上进行的大量实验表明，KELE 显著增强了编辑后的 ​​LLM 的多跳推理能力。

Title: GenderCARE: A Comprehensive Framework for Assessing and Reducing Gender Bias in Large Language Models

• Authors: Kunsheng Tang, Wenbo Zhou, Jie Zhang, Aishan Liu, Gelei Deng, Shuai Li, Peigui Qi, Weiming Zhang, Tianwei Zhang, Nenghai Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12494
• Pdf URL: https://arxiv.org/pdf/2408.12494
• Copy Paste: [[2408.12494]] GenderCARE: A Comprehensive Framework for Assessing and Reducing Gender Bias in Large Language Models(https://arxiv.org/abs/2408.12494)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have exhibited remarkable capabilities in natural language generation, but they have also been observed to magnify societal biases, particularly those related to gender. In response to this issue, several benchmarks have been proposed to assess gender bias in LLMs. However, these benchmarks often lack practical flexibility or inadvertently introduce biases. To address these shortcomings, we introduce GenderCARE, a comprehensive framework that encompasses innovative Criteria, bias Assessment, Reduction techniques, and Evaluation metrics for quantifying and mitigating gender bias in LLMs. To begin, we establish pioneering criteria for gender equality benchmarks, spanning dimensions such as inclusivity, diversity, explainability, objectivity, robustness, and realisticity. Guided by these criteria, we construct GenderPair, a novel pair-based benchmark designed to assess gender bias in LLMs comprehensively. Our benchmark provides standardized and realistic evaluations, including previously overlooked gender groups such as transgender and non-binary individuals. Furthermore, we develop effective debiasing techniques that incorporate counterfactual data augmentation and specialized fine-tuning strategies to reduce gender bias in LLMs without compromising their overall performance. Extensive experiments demonstrate a significant reduction in various gender bias benchmarks, with reductions peaking at over 90% and averaging above 35% across 17 different LLMs. Importantly, these reductions come with minimal variability in mainstream language tasks, remaining below 2%. By offering a realistic assessment and tailored reduction of gender biases, we hope that our GenderCARE can represent a significant step towards achieving fairness and equity in LLMs. More details are available at this https URL.
• 摘要：大型语言模型 (LLM) 在自然语言生成方面表现出了卓越的能力，但也观察到它们会放大社会偏见，尤其是与性别有关的偏见。针对这一问题，已经提出了几个基准来评估 LLM 中的性别偏见。然而，这些基准往往缺乏实际灵活性或无意中引入偏见。为了解决这些缺点，我们引入了 GenderCARE，这是一个全面的框架，涵盖创新的标准、偏见评估、减少技术和评估指标，用于量化和减轻 LLM 中的性别偏见。首先，我们为性别平等基准建立了开创性的标准，涵盖了包容性、多样性、可解释性、客观性、稳健性和现实性等维度。在这些标准的指导下，我们构建了 GenderPair，这是一个新颖的基于配对的基准，旨在全面评估 LLM 中的性别偏见。我们的基准提供了标准化和现实的评估，包括以前被忽视的性别群体，例如跨性别者和非二元性别者。此外，我们开发了有效的去偏技术，结合反事实数据增强和专门的微调策略，以减少法学硕士中的性别偏见，而不会影响其整体表现。大量实验表明，各种性别偏见基准都有显著减少，在 17 个不同的法学硕士中，减少幅度最高达到 90% 以上，平均超过 35%。重要的是，这些减少在主流语言任务中几乎没有变化，保持在 2% 以下。通过提供现实的评估和量身定制的性别偏见减少，我们希望我们的 GenderCARE 能够代表实现法学硕士公平和公正的重要一步。更多详细信息请访问此 https URL。

Title: Towards Evaluating and Building Versatile Large Language Models for Medicine

• Authors: Chaoyi Wu, Pengcheng Qiu, Jinxin Liu, Hongfei Gu, Na Li, Ya Zhang, Yanfeng Wang, Weidi Xie
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12547
• Pdf URL: https://arxiv.org/pdf/2408.12547
• Copy Paste: [[2408.12547]] Towards Evaluating and Building Versatile Large Language Models for Medicine(https://arxiv.org/abs/2408.12547)
• Keywords: language model, gpt, llm, prompt
• Abstract: In this study, we present MedS-Bench, a comprehensive benchmark designed to evaluate the performance of large language models (LLMs) in clinical contexts. Unlike existing benchmarks that focus on multiple-choice question answering, MedS-Bench spans 11 high-level clinical tasks, including clinical report summarization, treatment recommendations, diagnosis, named entity recognition, and medical concept explanation, among others. We evaluated six leading LLMs, e.g., MEDITRON, Mistral, InternLM 2, Llama 3, GPT-4, and Claude-3.5 using few-shot prompting, and found that even the most sophisticated models struggle with these complex tasks. To address these limitations, we developed MedS-Ins, a large-scale instruction tuning dataset for medicine. MedS-Ins comprises 58 medically oriented language corpora, totaling 13.5 million samples across 122 tasks. To demonstrate the dataset's utility, we conducted a proof-of-concept experiment by performing instruction tuning on a lightweight, open-source medical language model. The resulting model, MMedIns-Llama 3, significantly outperformed existing models across nearly all clinical tasks. To promote further advancements in the application of LLMs to clinical challenges, we have made the MedS-Ins dataset fully accessible and invite the research community to contribute to its expansion.Additionally, we have launched a dynamic leaderboard for MedS-Bench, which we plan to regularly update the test set to track progress and enhance the adaptation of general LLMs to the medical domain. Leaderboard: this https URL. Github: this https URL.
• 摘要：在本研究中，我们提出了 MedS-Bench，这是一个全面的基准，旨在评估大型语言模型 (LLM) 在临床环境中的性能。与专注于多项选择题回答的现有基准不同，MedS-Bench 涵盖 11 项高级临床任务，包括临床报告摘要、治疗建议、诊断、命名实体识别和医学概念解释等。我们使用少样本提示评估了六个领先的 LLM，例如 MEDITRON、Mistral、InternLM 2、Llama 3、GPT-4 和 Claude-3.5，发现即使是最复杂的模型也难以完成这些复杂的任务。为了解决这些限制，我们开发了 MedS-Ins，这是一个用于医学的大规模指令调整数据集。MedS-Ins 包含 58 个面向医学的语言语料库，共计 122 个任务中的 1350 万个样本。为了证明数据集的实用性，我们通过对轻量级开源医学语言模型执行指令调整进行了概念验证实验。由此产生的模型 MMedIns-Llama 3 在几乎所有临床任务中的表现都显著优于现有模型。为了推动 LLM 在临床挑战中的应用取得进一步进展，我们已将 MedS-Ins 数据集完全开放，并邀请研究界为其扩展做出贡献。此外，我们还为 MedS-Bench 推出了一个动态排行榜，我们计划定期更新测试集以跟踪进度并增强通用 LLM 在医学领域的适应性。排行榜：此 https URL。Github：此 https URL。

Title: Jamba-1.5: Hybrid Transformer-Mamba Models at Scale

• Authors: Jamba Team: Barak Lenz, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, Daniel Gissin, Daniel Jannai, Dor Muhlgay, Dor Zimberg, Edden M Gerber, Elad Dolev, Eran Krakovsky, Erez Safahi, Erez Schwartz, Gal Cohen, Gal Shachaf, Haim Rozenblum, Hofit Bata, Ido Blass, Inbal Magar, Itay Dalmedigos, Jhonathan Osin, Julie Fadlon, Maria Rozman, Matan Danos, Michael Gokhman, Mor Zusman, Naama Gidron, Nir Ratner, Noam Gat, Noam Rozen, Oded Fried, Ohad Leshno, Omer Antverg, Omri Abend, Opher Lieber, Or Dagan, Orit Cohavi, Raz Alon, Ro'i Belson, Roi Cohen, Rom Gilad, Roman Glozman, Shahar Lev, Shaked Meirom, Tal Delbari, Tal Ness, Tomer Asida, Tom Ben Gal, Tom Braude, Uriya Pumerantz, Yehoshua Cohen, Yonatan Belinkov, Yuval Globerson, Yuval Peleg Levy, Yoav Shoham
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12570
• Pdf URL: https://arxiv.org/pdf/2408.12570
• Copy Paste: [[2408.12570]] Jamba-1.5: Hybrid Transformer-Mamba Models at Scale(https://arxiv.org/abs/2408.12570)
• Keywords: language model, chat
• Abstract: We present Jamba-1.5, new instruction-tuned large language models based on our Jamba architecture. Jamba is a hybrid Transformer-Mamba mixture of experts architecture, providing high throughput and low memory usage across context lengths, while retaining the same or better quality as Transformer models. We release two model sizes: Jamba-1.5-Large, with 94B active parameters, and Jamba-1.5-Mini, with 12B active parameters. Both models are fine-tuned for a variety of conversational and instruction-following capabilties, and have an effective context length of 256K tokens, the largest amongst open-weight models. To support cost-effective inference, we introduce ExpertsInt8, a novel quantization technique that allows fitting Jamba-1.5-Large on a machine with 8 80GB GPUs when processing 256K-token contexts without loss of quality. When evaluated on a battery of academic and chatbot benchmarks, Jamba-1.5 models achieve excellent results while providing high throughput and outperforming other open-weight models on long-context benchmarks. The model weights for both sizes are publicly available under the Jamba Open Model License and we release ExpertsInt8 as open source.
• 摘要：我们推出了基于 Jamba 架构的新型指令调整大型语言模型 Jamba-1.5。Jamba 是 Transformer-Mamba 混合专家架构，可在各种上下文长度上提供高吞吐量和低内存使用率，同时保持与 Transformer 模型相同或更好的质量。我们发布了两种模型大小：Jamba-1.5-Large，具有 94B 活动参数，以及 Jamba-1.5-Mini，具有 12B 活动参数。这两种模型都针对各种对话和指令跟踪功能进行了微调，并且有效上下文长度为 256K 个 token，是开放权重模型中最大的。为了支持经济高效的推理，我们引入了 ExpertsInt8，这是一种新颖的量化技术，允许在处理 256K 个 token 上下文时在具有 8 个 80GB GPU 的机器上安装 Jamba-1.5-Large 而不会损失质量。在一系列学术和聊天机器人基准测试中，Jamba-1.5 模型取得了出色的结果，同时提供了高吞吐量，并且在长上下文基准测试中优于其他开放权重模型。两种尺寸的模型权重均根据 Jamba 开放模型许可证公开提供，我们将 ExpertsInt8 作为开源发布。

Title: RuleAlign: Making Large Language Models Better Physicians with Diagnostic Rule Alignment

• Authors: Xiaohan Wang, Xiaoyan Yang, Yuqi Zhu, Yue Shen, Jian Wang, Peng Wei, Lei Liang, Jinjie Gu, Huajun Chen, Ningyu Zhang
• Subjects: cs.CL, cs.AI, cs.HC, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12579
• Pdf URL: https://arxiv.org/pdf/2408.12579
• Copy Paste: [[2408.12579]] RuleAlign: Making Large Language Models Better Physicians with Diagnostic Rule Alignment(https://arxiv.org/abs/2408.12579)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) like GPT-4, MedPaLM-2, and Med-Gemini achieve performance competitively with human experts across various medical benchmarks. However, they still face challenges in making professional diagnoses akin to physicians, particularly in efficiently gathering patient information and reasoning the final diagnosis. To this end, we introduce the RuleAlign framework, designed to align LLMs with specific diagnostic rules. We develop a medical dialogue dataset comprising rule-based communications between patients and physicians and design an alignment learning approach through preference learning. Experimental results demonstrate the effectiveness of the proposed approach. We hope that our work can serve as an inspiration for exploring the potential of LLMs as AI physicians.
• 摘要：GPT-4、MedPaLM-2 和 Med-Gemini 等大型语言模型 (LLM) 在各种医学基准上的表现都堪比人类专家。然而，它们在做出类似于医生的专业诊断方面仍然面临挑战，特别是在高效收集患者信息和推理最终诊断方面。为此，我们引入了 RuleAlign 框架，旨在将 LLM 与特定诊断规则对齐。我们开发了一个包含患者和医生之间基于规则的通信的医学对话数据集，并通过偏好学习设计了一种对齐学习方法。实验结果证明了所提方法的有效性。我们希望我们的工作能够为探索 LLM 作为 AI 医生的潜力提供灵感。

Title: Controllable Text Generation for Large Language Models: A Survey

• Authors: Xun Liang, Hanyu Wang, Yezhaohui Wang, Shichao Song, Jiawei Yang, Simin Niu, Jie Hu, Dan Liu, Shunyu Yao, Feiyu Xiong, Zhiyu Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12599
• Pdf URL: https://arxiv.org/pdf/2408.12599
• Copy Paste: [[2408.12599]] Controllable Text Generation for Large Language Models: A Survey(https://arxiv.org/abs/2408.12599)
• Keywords: language model, llm, prompt
• Abstract: In Natural Language Processing (NLP), Large Language Models (LLMs) have demonstrated high text generation quality. However, in real-world applications, LLMs must meet increasingly complex requirements. Beyond avoiding misleading or inappropriate content, LLMs are also expected to cater to specific user needs, such as imitating particular writing styles or generating text with poetic richness. These varied demands have driven the development of Controllable Text Generation (CTG) techniques, which ensure that outputs adhere to predefined control conditions--such as safety, sentiment, thematic consistency, and linguistic style--while maintaining high standards of helpfulness, fluency, and diversity. This paper systematically reviews the latest advancements in CTG for LLMs, offering a comprehensive definition of its core concepts and clarifying the requirements for control conditions and text quality. We categorize CTG tasks into two primary types: content control and attribute control. The key methods are discussed, including model retraining, fine-tuning, reinforcement learning, prompt engineering, latent space manipulation, and decoding-time intervention. We analyze each method's characteristics, advantages, and limitations, providing nuanced insights for achieving generation control. Additionally, we review CTG evaluation methods, summarize its applications across domains, and address key challenges in current research, including reduced fluency and practicality. We also propose several appeals, such as placing greater emphasis on real-world applications in future research. This paper aims to offer valuable guidance to researchers and developers in the field. Our reference list and Chinese version are open-sourced at this https URL.
• 摘要：在自然语言处理 (NLP) 中，大型语言模型 (LLM) 已展现出较高的文本生成质量。然而，在实际应用中，LLM 必须满足日益复杂的要求。除了避免误导或不适当的内容外，LLM 还应满足特定的用户需求，例如模仿特定的写作风格或生成具有诗意丰富性的文本。这些不同的需求推动了可控文本生成 (CTG) 技术的发展，这些技术确保输出符合预定义的控制条件（例如安全性、情感、主题一致性和语言风格），同时保持高标准的有用性、流畅性和多样性。本文系统地回顾了 LLM 的 CTG 的最新进展，全面定义了其核心概念，并阐明了对控制条件和文本质量的要求。我们将 CTG 任务分为两种主要类型：内容控制和属性控制。讨论了关键方法，包括模型再训练、微调、强化学习、提示工程、潜在空间操纵和解码时间干预。我们分析了每种方法的特点、优势和局限性，为实现发电控制提供了细致入微的见解。此外，我们回顾了 CTG 评估方法，总结了其在各个领域的应用，并解决了当前研究中的关键挑战，包括流畅性和实用性降低。我们还提出了一些呼吁，例如在未来的研究中更加重视现实世界的应用。本文旨在为该领域的研究人员和开发人员提供宝贵的指导。我们的参考文献列表和中文版在此 https URL 上开源。