2024-11-07

Title: A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness

• Authors: Fali Wang, Zhiwei Zhang, Xianren Zhang, Zongyu Wu, Tzuhao Mo, Qiuhao Lu, Wanjing Wang, Rui Li, Junjie Xu, Xianfeng Tang, Qi He, Yao Ma, Ming Huang, Suhang Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03350
• Pdf URL: https://arxiv.org/pdf/2411.03350
• Copy Paste: [[2411.03350]] A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness(https://arxiv.org/abs/2411.03350)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLM) have demonstrated emergent abilities in text generation, question answering, and reasoning, facilitating various tasks and domains. Despite their proficiency in various tasks, LLMs like LaPM 540B and Llama-3.1 405B face limitations due to large parameter sizes and computational demands, often requiring cloud API use which raises privacy concerns, limits real-time applications on edge devices, and increases fine-tuning costs. Additionally, LLMs often underperform in specialized domains such as healthcare and law due to insufficient domain-specific knowledge, necessitating specialized models. Therefore, Small Language Models (SLMs) are increasingly favored for their low inference latency, cost-effectiveness, efficient development, and easy customization and adaptability. These models are particularly well-suited for resource-limited environments and domain knowledge acquisition, addressing LLMs' challenges and proving ideal for applications that require localized data handling for privacy, minimal inference latency for efficiency, and domain knowledge acquisition through lightweight fine-tuning. The rising demand for SLMs has spurred extensive research and development. However, a comprehensive survey investigating issues related to the definition, acquisition, application, enhancement, and reliability of SLM remains lacking, prompting us to conduct a detailed survey on these topics. The definition of SLMs varies widely, thus to standardize, we propose defining SLMs by their capability to perform specialized tasks and suitability for resource-constrained settings, setting boundaries based on the minimal size for emergent abilities and the maximum size sustainable under resource constraints. For other aspects, we provide a taxonomy of relevant models/methods and develop general frameworks for each category to enhance and utilize SLMs effectively.
• 摘要：大型语言模型 (LLM) 已展现出在文本生成、问答和推理方面的新兴能力，有助于完成各种任务和领域。尽管它们在各种任务中都表现出色，但 LaPM 540B 和 Llama-3.1 405B 等 LLM 仍因参数规模大和计算需求大而面临限制，通常需要使用云 API，这引发了隐私问题，限制了边缘设备上的实时应用程序，并增加了微调成本。此外，由于缺乏领域特定知识，LLM 在医疗保健和法律等专业领域往往表现不佳，需要专门的模型。因此，小型语言模型 (SLM) 因其低推理延迟、成本效益高、开发高效、易于定制和适应性而越来越受到青睐。这些模型特别适合资源有限的环境和领域知识获取，解决了 LLM 的挑战，并被证明是需要本地化数据处理以保护隐私、最小推理延迟以提高效率以及通过轻量级微调获取领域知识的应用程序的理想选择。对 SLM 的需求不断增长，这刺激了广泛的研究和开发。然而，对 SLM 的定义、获取、应用、增强和可靠性相关问题的全面调查仍然缺乏，这促使我们对这些主题进行详细调查。SLM 的定义千差万别，因此为了标准化，我们建议根据 SLM 执行专门任务的能力和对资源受限环境的适用性来定义 SLM，根据新兴能力的最小规模和资源受限下可持续的最大规模设定边界。对于其他方面，我们提供了相关模型/方法的分类，并为每个类别开发了通用框架，以有效地增强和利用 SLM。

Title: SAUCE: Synchronous and Asynchronous User-Customizable Environment for Multi-Agent LLM Interaction

• Authors: Shlomo Neuberger, Niv Eckhaus, Uri Berger, Amir Taubenfeld, Gabriel Stanovsky, Ariel Goldstein
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.03397
• Pdf URL: https://arxiv.org/pdf/2411.03397
• Copy Paste: [[2411.03397]] SAUCE: Synchronous and Asynchronous User-Customizable Environment for Multi-Agent LLM Interaction(https://arxiv.org/abs/2411.03397)
• Keywords: llm, agent
• Abstract: Many human interactions, such as political debates, are carried out in group settings, where there are arbitrarily many participants, each with different views and agendas. To explore such complex social settings, we present SAUCE: a customizable Python platform, allowing researchers to plug-and-play various LLMs participating in discussions on any topic chosen by the user. Our platform takes care of instantiating the models, scheduling their responses, managing the discussion history, and producing a comprehensive output log, all customizable through configuration files, requiring little to no coding skills. A novel feature of SAUCE is our asynchronous communication feature, where models decide when to speak in addition to what to say, thus modeling an important facet of human communication. We show SAUCE's attractiveness in two initial experiments, and invite the community to use it in simulating various group simulations.
• 摘要：许多人际交往，例如政治辩论，都是在群体环境中进行的，其中有任意多的参与者，每个参与者都有不同的观点和议程。为了探索这种复杂的社交环境，我们提出了 SAUCE：一个可定制的 Python 平台，允许研究人员即插即用各种 LLM 参与用户选择的任何主题的讨论。我们的平台负责实例化模型、安排他们的响应、管理讨论历史以及生成全面的输出日志，所有这些都可以通过配置文件进行定制，几乎不需要任何编码技能。SAUCE 的一个新功能是我们的异步通信功能，其中模型除了决定说什么之外还决定何时说话，从而模拟人类交流的一个重要方面。我们在两个初步实验中展示了 SAUCE 的吸引力，并邀请社区使用它来模拟各种群体模拟。

Title: Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment

• Authors: Alexander Goldberg, Ihsan Ullah, Thanh Gia Hieu Khuong, Benedictus Kent Rachmat, Zhen Xu, Isabelle Guyon, Nihar B. Shah
• Subjects: cs.CL, cs.DL, cs.HC
• Abstract URL: https://arxiv.org/abs/2411.03417
• Pdf URL: https://arxiv.org/pdf/2411.03417
• Copy Paste: [[2411.03417]] Usefulness of LLMs as an Author Checklist Assistant for Scientific Papers: NeurIPS'24 Experiment(https://arxiv.org/abs/2411.03417)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) represent a promising, but controversial, tool in aiding scientific peer review. This study evaluates the usefulness of LLMs in a conference setting as a tool for vetting paper submissions against submission standards. We conduct an experiment at the 2024 Neural Information Processing Systems (NeurIPS) conference, where 234 papers were voluntarily submitted to an "LLM-based Checklist Assistant." This assistant validates whether papers adhere to the author checklist used by NeurIPS, which includes questions to ensure compliance with research and manuscript preparation standards. Evaluation of the assistant by NeurIPS paper authors suggests that the LLM-based assistant was generally helpful in verifying checklist completion. In post-usage surveys, over 70% of authors found the assistant useful, and 70% indicate that they would revise their papers or checklist responses based on its feedback. While causal attribution to the assistant is not definitive, qualitative evidence suggests that the LLM contributed to improving some submissions. Survey responses and analysis of re-submissions indicate that authors made substantive revisions to their submissions in response to specific feedback from the LLM. The experiment also highlights common issues with LLMs: inaccuracy (20/52) and excessive strictness (14/52) were the most frequent issues flagged by authors. We also conduct experiments to understand potential gaming of the system, which reveal that the assistant could be manipulated to enhance scores through fabricated justifications, highlighting potential vulnerabilities of automated review tools.
• 摘要：大型语言模型 (LLM) 是一种有前途但有争议的科学同行评审辅助工具。本研究评估了 LLM 在会议环境中作为根据提交标准审查论文提交的工具的实用性。我们在 2024 年神经信息处理系统 (NeurIPS) 会议上进行了一项实验，其中 234 篇论文自愿提交给“基于 LLM 的检查表助手”。该助手验证论文是否符合 NeurIPS 使用的作者检查表，其中包括确保符合研究和手稿准备标准的问题。NeurIPS 论文作者对助手的评估表明，基于 LLM 的助手通常有助于验证检查表的完成情况。在使用后调查中，超过 70% 的作者认为该助手很有用，70% 的作者表示他们会根据其反馈修改论文或检查表回复。虽然将因果关系归因于助手尚不确定，但定性证据表明 LLM 有助于改进一些提交内容。调查回复和重新提交的分析表明，作者根据法学硕士的具体反馈对其提交的内容进行了实质性修改。实验还强调了法学硕士的常见问题：不准确（20/52）和过于严格（14/52）是作者最常提到的问题。我们还进行了实验以了解系统的潜在游戏，结果表明，可以通过捏造的理由操纵助手来提高分数，这凸显了自动审查工具的潜在漏洞。

Title: Automatic Generation of Question Hints for Mathematics Problems using Large Language Models in Educational Technology

• Authors: Junior Cedric Tonga, Benjamin Clement, Pierre-Yves Oudeyer
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03495
• Pdf URL: https://arxiv.org/pdf/2411.03495
• Copy Paste: [[2411.03495]] Automatic Generation of Question Hints for Mathematics Problems using Large Language Models in Educational Technology(https://arxiv.org/abs/2411.03495)
• Keywords: language model, gpt, llm, prompt
• Abstract: The automatic generation of hints by Large Language Models (LLMs) within Intelligent Tutoring Systems (ITSs) has shown potential to enhance student learning. However, generating pedagogically sound hints that address student misconceptions and adhere to specific educational objectives remains challenging. This work explores using LLMs (GPT-4o and Llama-3-8B-instruct) as teachers to generate effective hints for students simulated through LLMs (GPT-3.5-turbo, Llama-3-8B-Instruct, or Mistral-7B-instruct-v0.3) tackling math exercises designed for human high-school students, and designed using cognitive science principles. We present here the study of several dimensions: 1) identifying error patterns made by simulated students on secondary-level math exercises; 2) developing various prompts for GPT-4o as a teacher and evaluating their effectiveness in generating hints that enable simulated students to self-correct; and 3) testing the best-performing prompts, based on their ability to produce relevant hints and facilitate error correction, with Llama-3-8B-Instruct as the teacher, allowing for a performance comparison with GPT-4o. The results show that model errors increase with higher temperature settings. Notably, when hints are generated by GPT-4o, the most effective prompts include prompts tailored to specific errors as well as prompts providing general hints based on common mathematical errors. Interestingly, Llama-3-8B-Instruct as a teacher showed better overall performance than GPT-4o. Also the problem-solving and response revision capabilities of the LLMs as students, particularly GPT-3.5-turbo, improved significantly after receiving hints, especially at lower temperature settings. However, models like Mistral-7B-Instruct demonstrated a decline in performance as the temperature increased.
• 摘要：智能辅导系统 (ITS) 中的大型语言模型 (LLM) 自动生成提示已显示出提高学生学习的潜力。然而，生成合理的教学提示以解决学生的误解并遵循特定的教育目标仍然具有挑战性。这项工作探索了使用 LLM（GPT-4o 和 Llama-3-8B-instruct）作为教师，为通过 LLM（GPT-3.5-turbo、Llama-3-8B-Instruct 或 Mistral-7B-instruct-v0.3）模拟的学生生成有效的提示，这些练习是使用认知科学原理为人类高中生设计的数学练习。我们在这里介绍了几个维度的研究：1）识别模拟学生在中学数学练习中的错误模式；2）为 GPT-4o 作为老师开发各种提示，并评估它们在生成使模拟学生能够自我纠正的提示方面的有效性； 3) 以 Llama-3-8B-Instruct 为老师，根据提示产生相关提示和促进错误纠正的能力，测试表现最佳的提示，以便与 GPT-4o 进行性能比较。结果表明，模型错误随温度升高而增加。值得注意的是，当 GPT-4o 生成提示时，最有效的提示包括针对特定错误定制的提示以及基于常见数学错误提供一般提示的提示。有趣的是，作为老师的 Llama-3-8B-Instruct 表现出比 GPT-4o 更好的整体表现。此外，作为学生的 LLM，尤其是 GPT-3.5-turbo，在收到提示后，解决问题和响应修改能力显著提高，尤其是在较低温度设置下。然而，像 Mistral-7B-Instruct 这样的模型表现出随着温度升高而下降的性能。

Title: Uncertainty Quantification for Clinical Outcome Predictions with (Large) Language Models

• Authors: Zizhang Chen, Peizhao Li, Xiaomeng Dong, Pengyu Hong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03497
• Pdf URL: https://arxiv.org/pdf/2411.03497
• Copy Paste: [[2411.03497]] Uncertainty Quantification for Clinical Outcome Predictions with (Large) Language Models(https://arxiv.org/abs/2411.03497)
• Keywords: language model, gpt, prompt
• Abstract: To facilitate healthcare delivery, language models (LMs) have significant potential for clinical prediction tasks using electronic health records (EHRs). However, in these high-stakes applications, unreliable decisions can result in high costs due to compromised patient safety and ethical concerns, thus increasing the need for good uncertainty modeling of automated clinical predictions. To address this, we consider the uncertainty quantification of LMs for EHR tasks in white- and black-box settings. We first quantify uncertainty in white-box models, where we can access model parameters and output logits. We show that an effective reduction of model uncertainty can be achieved by using the proposed multi-tasking and ensemble methods in EHRs. Continuing with this idea, we extend our approach to black-box settings, including popular proprietary LMs such as GPT-4. We validate our framework using longitudinal clinical data from more than 6,000 patients in ten clinical prediction tasks. Results show that ensembling methods and multi-task prediction prompts reduce uncertainty across different scenarios. These findings increase the transparency of the model in white-box and black-box settings, thus advancing reliable AI healthcare.
• 摘要：为了促进医疗保健服务，语言模型 (LM) 在使用电子健康记录 (EHR) 的临床预测任务中具有巨大潜力。然而，在这些高风险应用中，不可靠的决策可能会因患者安全和道德问题而导致高昂的成本，从而增加了对自动临床预测的良好不确定性建模的需求。为了解决这个问题，我们考虑在白盒和黑盒设置中量化 EHR 任务的 LM 的不确定性。我们首先量化白盒模型中的不确定性，我们可以在其中访问模型参数和输出对数。我们表明，通过在 EHR 中使用所提出的多任务和集成方法可以有效减少模型不确定性。继续这个想法，我们将我们的方法扩展到黑盒设置，包括流行的专有 LM，例如 GPT-4。我们使用来自十个临床预测任务中 6,000 多名患者的纵向临床数据验证了我们的框架。结果表明，集成方法和多任务预测提示可以减少不同场景中的不确定性。这些发现提高了模型在白盒和黑盒设置中的透明度，从而推动了可靠的人工智能医疗保健。

Title: Change Is the Only Constant: Dynamic LLM Slicing based on Layer Redundancy

• Authors: Razvan-Gabriel Dumitru, Paul-Ioan Clotan, Vikas Yadav, Darius Peteleaza, Mihai Surdeanu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03513
• Pdf URL: https://arxiv.org/pdf/2411.03513
• Copy Paste: [[2411.03513]] Change Is the Only Constant: Dynamic LLM Slicing based on Layer Redundancy(https://arxiv.org/abs/2411.03513)
• Keywords: language model, gpt, llm
• Abstract: This paper introduces a novel model compression approach through dynamic layer-specific pruning in Large Language Models (LLMs), enhancing the traditional methodology established by SliceGPT. By transitioning from constant to dynamic slicing, our method leverages the newly proposed Layer Redundancy (LR) score, which assesses how much change each layer changes its input by measuring the cosine similarity of the input to the output of the layer. We use this score to prune parts of individual layers based on redundancy in such a way that the average pruned percentage for all layers is a fixed value. We conducted extensive experiments using models like Llama3-8B and Mistral-7B on multiple datasets, evaluating different slicing bases and percentages to determine optimal configurations that balance efficiency and performance. Our findings show that our dynamic slicing approach not only maintains but, in many cases, enhances model performance compared to the baseline established by constant slicing methods. For instance, in several settings, we see performance improvements of up to 5% over the SliceGPT baseline. Additionally, a perplexity decrease by as much as 7% was observed across multiple benchmarks, validating the effectiveness of our method. The code, model weights, and datasets are open-sourced at this https URL.
• 摘要：本文介绍了一种通过大型语言模型 (LLM) 中的动态层特定修剪实现的新型模型压缩方法，增强了 SliceGPT 建立的传统方法。通过从恒定切片过渡到动态切片，我们的方法利用了新提出的层冗余 (LR) 分数，该分数通过测量输入与层输出的余弦相似度来评估每层对其输入的改变程度。我们使用此分数根据冗余度修剪各个层的部分，以使所有层的平均修剪百分比为固定值。我们在多个数据集上使用 Llama3-8B 和 Mistral-7B 等模型进行了广泛的实验，评估了不同的切片基础和百分比，以确定平衡效率和性能的最佳配置。我们的研究结果表明，与恒定切片方法建立的基线相比，我们的动态切片方法不仅保持了模型性能，而且在许多情况下还提高了模型性能。例如，在几种设置中，我们看到性能比 SliceGPT 基线提高了 5%。此外，在多个基准测试中，困惑度降低了 7%，证明了我们方法的有效性。代码、模型权重和数据集在此 https URL 上开源。

Title: Exploring the Benefits of Domain-Pretraining of Generative Large Language Models for Chemistry

• Authors: Anurag Acharya, Shivam Sharma, Robin Cosbey, Megha Subramanian, Scott Howland, Maria Glenski
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03542
• Pdf URL: https://arxiv.org/pdf/2411.03542
• Copy Paste: [[2411.03542]] Exploring the Benefits of Domain-Pretraining of Generative Large Language Models for Chemistry(https://arxiv.org/abs/2411.03542)
• Keywords: language model, gpt, hallucination, prompt
• Abstract: A proliferation of Large Language Models (the GPT series, BLOOM, LLaMA, and more) are driving forward novel development of multipurpose AI for a variety of tasks, particularly natural language processing (NLP) tasks. These models demonstrate strong performance on a range of tasks; however, there has been evidence of brittleness when applied to more niche or narrow domains where hallucinations or fluent but incorrect responses reduce performance. Given the complex nature of scientific domains, it is prudent to investigate the trade-offs of leveraging off-the-shelf versus more targeted foundation models for scientific domains. In this work, we examine the benefits of in-domain pre-training for a given scientific domain, chemistry, and compare these to open-source, off-the-shelf models with zero-shot and few-shot prompting. Our results show that not only do in-domain base models perform reasonably well on in-domain tasks in a zero-shot setting but that further adaptation using instruction fine-tuning yields impressive performance on chemistry-specific tasks such as named entity recognition and molecular formula generation.
• 摘要：大型语言模型（GPT 系列、BLOOM、LLaMA 等）的激增正在推动多用途 AI 的新发展，用于各种任务，特别是自然语言处理 (NLP) 任务。这些模型在一系列任务上表现出色；然而，有证据表明，当应用于更小众或狭窄的领域时，它们会变得脆弱，在这些领域中，幻觉或流畅但不正确的反应会降低性能。鉴于科学领域的复杂性，明智的做法是研究利用现成的模型与更有针对性的基础模型在科学领域的权衡。在这项工作中，我们研究了域内预训练对给定科学领域（化学）的好处，并将其与具有零样本和少量样本提示的开源现成模型进行比较。我们的结果表明，域内基础模型不仅在零样本设置中的域内任务上表现相当好，而且使用指令微调进一步调整可以在化学特定任务（例如命名实体识别和分子式生成）上产生令人印象深刻的性能。

Title: The American Sign Language Knowledge Graph: Infusing ASL Models with Linguistic Knowledge

• Authors: Lee Kezar, Nidhi Munikote, Zian Zeng, Zed Sehyr, Naomi Caselli, Jesse Thomason
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.03568
• Pdf URL: https://arxiv.org/pdf/2411.03568
• Copy Paste: [[2411.03568]] The American Sign Language Knowledge Graph: Infusing ASL Models with Linguistic Knowledge(https://arxiv.org/abs/2411.03568)
• Keywords: language model
• Abstract: Language models for American Sign Language (ASL) could make language technologies substantially more accessible to those who sign. To train models on tasks such as isolated sign recognition (ISR) and ASL-to-English translation, datasets provide annotated video examples of ASL signs. To facilitate the generalizability and explainability of these models, we introduce the American Sign Language Knowledge Graph (ASLKG), compiled from twelve sources of expert linguistic knowledge. We use the ASLKG to train neuro-symbolic models for 3 ASL understanding tasks, achieving accuracies of 91% on ISR, 14% for predicting the semantic features of unseen signs, and 36% for classifying the topic of Youtube-ASL videos.
• 摘要：美国手语 (ASL) 的语言模型可以让语言技术更容易被手语使用者所接受。为了训练模型完成孤立手势识别 (ISR) 和 ASL 到英语翻译等任务，数据集提供了带注释的 ASL 手势视频示例。为了提高这些模型的通用性和可解释性，我们引入了美国手语知识图谱 (ASLKG)，它由 12 个专家语言知识来源汇编而成。我们使用 ASLKG 训练神经符号模型来完成 3 项 ASL 理解任务，在 ISR 上实现了 91% 的准确率，在预测未见过手势的语义特征上实现了 14% 的准确率，在对 Youtube-ASL 视频主题进行分类上实现了 36% 的准确率。

Title: From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyond

• Authors: Harsha Nori, Naoto Usuyama, Nicholas King, Scott Mayer McKinney, Xavier Fernandes, Sheng Zhang, Eric Horvitz
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03590
• Pdf URL: https://arxiv.org/pdf/2411.03590
• Copy Paste: [[2411.03590]] From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyond(https://arxiv.org/abs/2411.03590)
• Keywords: language model, gpt, llm, prompt
• Abstract: Run-time steering strategies like Medprompt are valuable for guiding large language models (LLMs) to top performance on challenging tasks. Medprompt demonstrates that a general LLM can be focused to deliver state-of-the-art performance on specialized domains like medicine by using a prompt to elicit a run-time strategy involving chain of thought reasoning and ensembling. OpenAI's o1-preview model represents a new paradigm, where a model is designed to do run-time reasoning before generating final responses. We seek to understand the behavior of o1-preview on a diverse set of medical challenge problem benchmarks. Following on the Medprompt study with GPT-4, we systematically evaluate the o1-preview model across various medical benchmarks. Notably, even without prompting techniques, o1-preview largely outperforms the GPT-4 series with Medprompt. We further systematically study the efficacy of classic prompt engineering strategies, as represented by Medprompt, within the new paradigm of reasoning models. We found that few-shot prompting hinders o1's performance, suggesting that in-context learning may no longer be an effective steering approach for reasoning-native models. While ensembling remains viable, it is resource-intensive and requires careful cost-performance optimization. Our cost and accuracy analysis across run-time strategies reveals a Pareto frontier, with GPT-4o representing a more affordable option and o1-preview achieving state-of-the-art performance at higher cost. Although o1-preview offers top performance, GPT-4o with steering strategies like Medprompt retains value in specific contexts. Moreover, we note that the o1-preview model has reached near-saturation on many existing medical benchmarks, underscoring the need for new, challenging benchmarks. We close with reflections on general directions for inference-time computation with LLMs.
• 摘要：像 Medprompt 这样的运行时指导策略对于引导大型语言模型 (LLM) 在具有挑战性的任务上取得最佳性能非常有用。Medprompt 表明，通过使用提示来引出涉及思路推理和集成的运行时策略，通用 LLM 可以专注于在医学等专业领域提供最先进的性能。OpenAI 的 o1-preview 模型代表了一种新范式，其中模型旨在在生成最终响应之前进行运行时推理。我们试图了解 o1-preview 在一系列不同的医学挑战问题基准上的行为。继使用 GPT-4 进行 Medprompt 研究之后，我们在各种医学基准上系统地评估了 o1-preview 模型。值得注意的是，即使没有提示技术，o1-preview 的表现也远远优于使用 Medprompt 的 GPT-4 系列。我们进一步系统地研究了以 Medprompt 为代表的经典提示工程策略在新推理模型范式中的有效性。我们发现，少量提示会阻碍 o1 的表现，这表明情境学习可能不再是推理原生模型的有效指导方法。虽然集成仍然可行，但它是资源密集型的，需要仔细的性价比优化。我们对运行时策略的成本和准确性分析揭示了帕累托前沿，GPT-4o 代表了更实惠的选择，而 o1-preview 以更高的成本实现了最先进的性能。虽然 o1-preview 提供了顶级性能，但具有 Medprompt 等指导策略的 GPT-4o 在特定情况下仍具有价值。此外，我们注意到 o1-preview 模型在许多现有的医学基准上已经达到接近饱和，这凸显了对新的、具有挑战性的基准的需求。最后，我们反思了使用 LLM 进行推理时间计算的一​​般方向。

Title: Deploying Multi-task Online Server with Large Language Model

• Authors: Yincen Qu, Chao Ma, Yiting Wu, Xiangying Dai, Hui Zhou, Hengyue Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03644
• Pdf URL: https://arxiv.org/pdf/2411.03644
• Copy Paste: [[2411.03644]] Deploying Multi-task Online Server with Large Language Model(https://arxiv.org/abs/2411.03644)
• Keywords: language model
• Abstract: In the industry, numerous tasks are deployed online. Traditional approaches often tackle each task separately by its own network, which leads to excessive costs for developing and scaling models, especially in the context of large language models. Although multi-task methods can save costs through parameter sharing, they often struggle to outperform single-task methods in real-world applications. To tackle these challenges, we present a three-stage multi-task learning framework for large language models. It involves task filtering, followed by fine-tuning on high-resource tasks, and finally fine-tuning on all tasks. We conducted comprehensive experiments in single-task and multi-task settings. Our approach, exemplified on different benchmarks, demonstrates that it is able to achieve performance comparable to the single-task method while reducing up to 90.9\% of its overhead.
• 摘要：在业界，许多任务都部署在线。传统方法通常通过自己的网络单独处理每个任务，这导致开发和扩展模型的成本过高，尤其是在大型语言模型的背景下。尽管多任务方法可以通过参数共享来节省成本，但它们在实际应用中往往难以超越单任务方法。为了应对这些挑战，我们为大型语言模型提出了一个三阶段多任务学习框架。它包括任务过滤，然后对高资源任务进行微调，最后对所有任务进行微调。我们在单任务和多任务设置中进行了全面的实验。我们的方法在不同的基准上进行了示例，表明它能够实现与单任务方法相当的性能，同时将其开销降低高达 90.9% 。

Title: Evaluating Moral Beliefs across LLMs through a Pluralistic Framework

• Authors: Xuelin Liu, Yanfei Zhu, Shucheng Zhu, Pengyuan Liu, Ying Liu, Dong Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03665
• Pdf URL: https://arxiv.org/pdf/2411.03665
• Copy Paste: [[2411.03665]] Evaluating Moral Beliefs across LLMs through a Pluralistic Framework(https://arxiv.org/abs/2411.03665)
• Keywords: language model, gpt, llm, chat
• Abstract: Proper moral beliefs are fundamental for language models, yet assessing these beliefs poses a significant challenge. This study introduces a novel three-module framework to evaluate the moral beliefs of four prominent large language models. Initially, we constructed a dataset containing 472 moral choice scenarios in Chinese, derived from moral words. The decision-making process of the models in these scenarios reveals their moral principle preferences. By ranking these moral choices, we discern the varying moral beliefs held by different language models. Additionally, through moral debates, we investigate the firmness of these models to their moral choices. Our findings indicate that English language models, namely ChatGPT and Gemini, closely mirror moral decisions of the sample of Chinese university students, demonstrating strong adherence to their choices and a preference for individualistic moral beliefs. In contrast, Chinese models such as Ernie and ChatGLM lean towards collectivist moral beliefs, exhibiting ambiguity in their moral choices and debates. This study also uncovers gender bias embedded within the moral beliefs of all examined language models. Our methodology offers an innovative means to assess moral beliefs in both artificial and human intelligence, facilitating a comparison of moral values across different cultures.
• 摘要：正确的道德信念是语言模型的基础，但评估这些信念是一项重大挑战。本研究引入了一个新颖的三模块框架来评估四个著名大型语言模型的道德信念。首先，我们构建了一个包含 472 个中文道德选择场景的数据集，这些场景源自道德词汇。这些场景中的模型的决策过程揭示了它们的道德原则偏好。通过对这些道德选择进行排序，我们辨别出不同语言模型持有的不同道德信念。此外，通过道德辩论，我们调查了这些模型对其道德选择的坚定性。我们的研究结果表明，英语语言模型 ChatGPT 和 Gemini 与中国大学生样本的道德决策非常相似，表现出对他们选择的强烈坚持和对个人主义道德信念的偏好。相比之下，Ernie 和 ChatGLM 等中文模型倾向于集体主义道德信念，在道德选择和辩论中表现出模糊性。本研究还揭示了所有被研究语言模型的道德信念中都存在性别偏见。我们的方法提供了一种创新的方法来评估人工智能和人类智能的道德信仰，有助于比较不同文化之间的道德价值观。

Title: QUILL: Quotation Generation Enhancement of Large Language Models

• Authors: Jin Xiao, Bowei Zhang, Qianyu He, Jiaqing Liang, Feng Wei, Jinglei Chen, Zujie Liang, Deqing Yang, Yanghua Xiao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03675
• Pdf URL: https://arxiv.org/pdf/2411.03675
• Copy Paste: [[2411.03675]] QUILL: Quotation Generation Enhancement of Large Language Models(https://arxiv.org/abs/2411.03675)
• Keywords: language model, llm
• Abstract: While Large language models (LLMs) have become excellent writing assistants, they still struggle with quotation generation. This is because they either hallucinate when providing factual quotations or fail to provide quotes that exceed human expectations. To bridge the gap, we systematically study how to evaluate and improve LLMs' performance in quotation generation tasks. We first establish a holistic and automatic evaluation system for quotation generation task, which consists of five criteria each with corresponding automatic metric. To improve the LLMs' quotation generation abilities, we construct a bilingual knowledge base that is broad in scope and rich in dimensions, containing up to 32,022 quotes. Moreover, guided by our critiria, we further design a quotation-specific metric to rerank the retrieved quotations from the knowledge base. Extensive experiments show that our metrics strongly correlate with human preferences. Existing LLMs struggle to generate desired quotes, but our quotation knowledge base and reranking metric help narrow this gap. Our dataset and code are publicly available at this https URL.
• 摘要：虽然大型语言模型 (LLM) 已成为出色的写作助手，但它们在引语生成方面仍然举步维艰。这是因为它们要么在提供事实引语时产生幻觉，要么无法提供超出人类期望的引语。为了弥补这一差距，我们系统地研究了如何评估和改进 LLM 在引语生成任务中的表现。我们首先为引语生成任务建立了一个整体的自动评估系统，该系统由五个标准组成，每个标准都有相应的自动指标。为了提高 LLM 的引语生成能力，我们构建了一个范围广泛、维度丰富的双语知识库，包含多达 32,022 条引语。此外，在我们的标准的指导下，我们进一步设计了一个引语特定的指标来对从知识库中检索到的引语进行重新排序。大量实验表明，我们的指标与人类的偏好高度相关。现有的 LLM 很难生成所需的引语，但我们的引语知识库和重新排序指标有助于缩小这一差距。我们的数据集和代码可在此 https URL 上公开获取。

Title: The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models

• Authors: Anaelia Ovalle, Krunoslav Lehman Pavasovic, Louis Martin, Luke Zettlemoyer, Eric Michael Smith, Adina Williams, Levent Sagun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03700
• Pdf URL: https://arxiv.org/pdf/2411.03700
• Copy Paste: [[2411.03700]] The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models(https://arxiv.org/abs/2411.03700)
• Keywords: language model, llm
• Abstract: Natural-language assistants are designed to provide users with helpful responses while avoiding harmful outputs, largely achieved through alignment to human preferences. Yet there is limited understanding of whether alignment techniques may inadvertently perpetuate or even amplify harmful biases inherited from their pre-aligned base models. This issue is compounded by the choice of bias evaluation benchmarks in popular preference-finetuned models, which predominantly focus on dominant social categories, such as binary gender, thereby limiting insights into biases affecting underrepresented groups. Towards addressing this gap, we center transgender, nonbinary, and other gender-diverse identities to investigate how alignment procedures interact with pre-existing gender-diverse bias in LLMs. Our key contributions include: 1) a comprehensive survey of bias evaluation modalities across leading preference-finetuned LLMs, highlighting critical gaps in gender-diverse representation, 2) systematic evaluation of gender-diverse biases across 12 models spanning Direct Preference Optimization (DPO) stages, uncovering harms popular bias benchmarks fail to detect, and 3) a flexible framework for measuring harmful biases in implicit reward signals applicable to other social contexts. Our findings reveal that DPO-aligned models are particularly sensitive to supervised finetuning (SFT), and can amplify two forms of real-world gender-diverse harms from their base models: stigmatization and gender non-affirmative language. We conclude with recommendations tailored to DPO and broader alignment practices, advocating for the adoption of community-informed bias evaluation frameworks to more effectively identify and address underrepresented harms in LLMs.
• 摘要：自然语言助手旨在为用户提供有用的响应，同时避免有害的输出，这主要通过与人类偏好保持一致来实现。然而，对于对齐技术是否会无意中延续甚至放大从其预先对齐的基础模型中继承的有害偏见，人们的理解有限。这个问题因流行的偏好微调模型中选择的偏见评估基准而加剧，这些模型主要关注二元性别等主要社会类别，从而限制了对影响代表性不足群体的偏见的洞察。为了解决这一差距，我们以跨性别、非二元性别和其他性别多元化身份为中心，研究对齐程序如何与法学硕士中预先存在的性别多元化偏见相互作用。我们的主要贡献包括：1）全面调查领先的偏好微调法学硕士的偏见评估模式，强调性别多样性代表性方面的关键差距；2）系统评估 12 个模型中的性别多样性偏见，涵盖直接偏好优化 (DPO) 阶段，揭示流行偏见基准无法检测到的危害；3）灵活的框架，用于衡量适用于其他社会背景的隐性奖励信号中的有害偏见。我们的研究结果表明，与 DPO 对齐的模型对监督微调 (SFT) 特别敏感，并且可以从其基础模型中放大两种形式的现实世界性别多样性危害：污名化和性别非肯定语言。我们最后提出了针对 DPO 和更广泛的对齐实践的建议，倡导采用社区知情的偏见评估框架，以更有效地识别和解决法学硕士中代表性不足的危害。

Title: Number Cookbook: Number Understanding of Language Models and How to Improve It

• Authors: Haotong Yang, Yi Hu, Shijia Kang, Zhouchen Lin, Muhan Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03766
• Pdf URL: https://arxiv.org/pdf/2411.03766
• Copy Paste: [[2411.03766]] Number Cookbook: Number Understanding of Language Models and How to Improve It(https://arxiv.org/abs/2411.03766)
• Keywords: language model, llm, chain-of-thought
• Abstract: Large language models (LLMs) can solve an increasing number of complex reasoning tasks while making surprising mistakes in basic numerical understanding and processing (such as 9.11 > 9.9). The latter ability is essential for tackling complex arithmetic and mathematical problems and serves as a foundation for most reasoning tasks, but previous work paid little attention to it or only discussed several restricted tasks (like integer addition). In this paper, we comprehensively investigate the numerical understanding and processing ability (NUPA) of LLMs. Firstly, we introduce a benchmark covering four common numerical representations and 17 distinct numerical tasks in four major categories, resulting in 41 meaningful combinations in total. These tasks are derived from primary and secondary education curricula, encompassing nearly all everyday numerical understanding and processing scenarios, and the rules of these tasks are very simple and clear. Through the benchmark, we find that current LLMs fail frequently in many of the tasks. To study the problem, we train small models with existing and potential techniques for enhancing NUPA (such as special tokenizers, PEs, and number formats), comprehensively evaluating their effectiveness using our testbed. We also finetune practical-scale LLMs on our proposed NUPA tasks and find that 1) naive finetuning can improve NUPA a lot on many but not all tasks, and 2) surprisingly, techniques designed to enhance NUPA prove ineffective for finetuning pretrained models. We further explore the impact of chain-of-thought techniques on NUPA. Our work takes a preliminary step towards understanding and improving NUPA of LLMs. Our benchmark and code are released at this https URL.
• 摘要：大型语言模型（LLM）可以解决越来越多复杂的推理任务，但在基本的数值理解和处理方面却犯下令人惊讶的错误（例如 9.11 > 9.9）。后一种能力对于解决复杂的算术和数学问题至关重要，是大多数推理任务的基础，但之前的工作很少关注它或仅讨论几个受限制的任务（如整数加法）。在本文中，我们全面研究了LLM的数值理解和处理能力（NUPA）。首先，我们引入了一个基准，涵盖四种常见的数值表示和四大类17个不同的数值任务，总共得到41个有意义的组合。这些任务来自中小学教育课程，涵盖了几乎所有日常的数值理解和处理场景，这些任务的规则非常简单明了。通过基准测试，我们发现当前的LLM在许多任务中经常失败。为了研究这个问题，我们使用现有和潜在的增强 NUPA 的技术（例如特殊的标记器、PE 和数字格式）训练小型模型，并使用我们的测试平台全面评估它们的有效性。我们还在我们提出的 NUPA 任务上对实际规模的 LLM 进行了微调，发现 1）简单的微调可以在许多但不是所有任务上大大改善 NUPA，2）令人惊讶的是，旨在增强 NUPA 的技术对于微调预训练模型无效。我们进一步探讨了思路链技术对 NUPA 的影响。我们的工作朝着理解和改进 LLM 的 NUPA 迈出了初步的一步。我们的基准和代码发布在此 https URL 上。

Title: A Comparative Study of Recent Large Language Models on Generating Hospital Discharge Summaries for Lung Cancer Patients

• Authors: Yiming Li, Fang Li, Kirk Roberts, Licong Cui, Cui Tao, Hua Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03805
• Pdf URL: https://arxiv.org/pdf/2411.03805
• Copy Paste: [[2411.03805]] A Comparative Study of Recent Large Language Models on Generating Hospital Discharge Summaries for Lung Cancer Patients(https://arxiv.org/abs/2411.03805)
• Keywords: language model, gpt, llm
• Abstract: Generating discharge summaries is a crucial yet time-consuming task in clinical practice, essential for conveying pertinent patient information and facilitating continuity of care. Recent advancements in large language models (LLMs) have significantly enhanced their capability in understanding and summarizing complex medical texts. This research aims to explore how LLMs can alleviate the burden of manual summarization, streamline workflow efficiencies, and support informed decision-making in healthcare settings. Clinical notes from a cohort of 1,099 lung cancer patients were utilized, with a subset of 50 patients for testing purposes, and 102 patients used for model fine-tuning. This study evaluates the performance of multiple LLMs, including GPT-3.5, GPT-4, GPT-4o, and LLaMA 3 8b, in generating discharge summaries. Evaluation metrics included token-level analysis (BLEU, ROUGE-1, ROUGE-2, ROUGE-L) and semantic similarity scores between model-generated summaries and physician-written gold standards. LLaMA 3 8b was further tested on clinical notes of varying lengths to examine the stability of its performance. The study found notable variations in summarization capabilities among LLMs. GPT-4o and fine-tuned LLaMA 3 demonstrated superior token-level evaluation metrics, while LLaMA 3 consistently produced concise summaries across different input lengths. Semantic similarity scores indicated GPT-4o and LLaMA 3 as leading models in capturing clinical relevance. This study contributes insights into the efficacy of LLMs for generating discharge summaries, highlighting LLaMA 3's robust performance in maintaining clarity and relevance across varying clinical contexts. These findings underscore the potential of automated summarization tools to enhance documentation precision and efficiency, ultimately improving patient care and operational capability in healthcare settings.
• 摘要：生成出院总结是临床实践中一项重要但耗时的任务，对于传达相关的患者信息和促进护理的连续性至关重要。大型语言模型 (LLM) 的最新进展显著增强了它们理解和总结复杂医学文本的能力。本研究旨在探索 LLM 如何减轻手动总结的负担、简化工作流程效率并支持医疗保健环境中的明智决策。研究使用了 1,099 名肺癌患者的临床记录，其中 50 名患者用于测试目的，102 名患者用于模型微调。本研究评估了多个 LLM（包括 GPT-3.5、GPT-4、GPT-4o 和 LLaMA 3 8b）在生成出院总结方面的表现。评估指标包括 token 级分析（BLEU、ROUGE-1、ROUGE-2、ROUGE-L）以及模型生成的总结与医生编写的黄金标准之间的语义相似性得分。 LLaMA 3 8b 还在不同长度的临床记录上进行了测试，以检查其性能的稳定性。研究发现，不同 LLM 的总结能力存在显著差异。GPT-4o 和经过微调的 LLaMA 3 表现出了卓越的标记级评估指标，而 LLaMA 3 在不同输入长度下始终能生成简洁的总结。语义相似度得分表明 GPT-4o 和 LLaMA 3 是捕捉临床相关性的领先模型。这项研究深入了解了 LLM 生成出院总结的有效性，突出了 LLaMA 3 在保持不同临床环境中的清晰度和相关性方面的强大性能。这些发现强调了自动总结工具在提高文档准确性和效率方面的潜力，最终改善医疗环境中的患者护理和运营能力。

Title: Understanding the Effects of Human-written Paraphrases in LLM-generated Text Detection

• Authors: Hiu Ting Lau, Arkaitz Zubiaga
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03806
• Pdf URL: https://arxiv.org/pdf/2411.03806
• Copy Paste: [[2411.03806]] Understanding the Effects of Human-written Paraphrases in LLM-generated Text Detection(https://arxiv.org/abs/2411.03806)
• Keywords: language model, gpt, llm
• Abstract: Natural Language Generation has been rapidly developing with the advent of large language models (LLMs). While their usage has sparked significant attention from the general public, it is important for readers to be aware when a piece of text is LLM-generated. This has brought about the need for building models that enable automated LLM-generated text detection, with the aim of mitigating potential negative outcomes of such content. Existing LLM-generated detectors show competitive performances in telling apart LLM-generated and human-written text, but this performance is likely to deteriorate when paraphrased texts are considered. In this study, we devise a new data collection strategy to collect Human & LLM Paraphrase Collection (HLPC), a first-of-its-kind dataset that incorporates human-written texts and paraphrases, as well as LLM-generated texts and paraphrases. With the aim of understanding the effects of human-written paraphrases on the performance of state-of-the-art LLM-generated text detectors OpenAI RoBERTa and watermark detectors, we perform classification experiments that incorporate human-written paraphrases, watermarked and non-watermarked LLM-generated documents from GPT and OPT, and LLM-generated paraphrases from DIPPER and BART. The results show that the inclusion of human-written paraphrases has a significant impact of LLM-generated detector performance, promoting TPR@1%FPR with a possible trade-off of AUROC and accuracy.
• 摘要：随着大型语言模型 (LLM) 的出现，自然语言生成得到了迅速发展。虽然它们的使用引起了公众的极大关注，但读者有必要了解一段文本是否由 LLM 生成。这带来了构建模型的需求，这些模型能够自动检测 LLM 生成的文本，以减轻此类内容可能带来的负面影响。现有的 LLM 生成的检测器在区分 LLM 生成的文本和人工编写的文本方面表现出色，但当考虑释义文本时，这种性能可能会下降。在本研究中，我们设计了一种新的数据收集策略来收集人类和 LLM 释义集合 (HLPC)，这是一种首创的数据集，它结合了人工编写的文本和释义，以及 LLM 生成的文本和释义。为了了解人工编写的释义对最先进的 LLM 生成文本检测器 OpenAI RoBERTa 和水印检测器性能的影响，我们进行了分类实验，结合了人工编写的释义、来自 GPT 和 OPT 的带水印和不带水印的 LLM 生成文档以及来自 DIPPER 和 BART 的 LLM 生成的释义。结果表明，加入人工编写的释义对 LLM 生成的检测器性能有显著影响，提升了 TPR@1%FPR，但可能会牺牲 AUROC 和准确性。

Title: Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forward

• Authors: Shashi Kumar, Iuliia Thorbecke, Sergio Burdisso, Esaú Villatoro-Tello, Manjunath K E, Kadri Hacioğlu, Pradeep Rangappa, Petr Motlicek, Aravind Ganapathiraju, Andreas Stolcke
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2411.03866
• Pdf URL: https://arxiv.org/pdf/2411.03866
• Copy Paste: [[2411.03866]] Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forward(https://arxiv.org/abs/2411.03866)
• Keywords: language model, llm
• Abstract: Recent research has demonstrated that training a linear connector between speech foundation encoders and large language models (LLMs) enables this architecture to achieve strong ASR capabilities. Despite the impressive results, it remains unclear whether these simple approaches are robust enough across different scenarios and speech conditions, such as domain shifts and different speech perturbations. In this paper, we address these questions by conducting various ablation experiments using a recent and widely adopted approach called SLAM-ASR. We present novel empirical findings that offer insights on how to effectively utilize the SLAM-ASR architecture across a wide range of settings. Our main findings indicate that the SLAM-ASR exhibits poor performance in cross-domain evaluation settings. Additionally, speech perturbations within in-domain data, such as changes in speed or the presence of additive noise, can significantly impact performance. Our findings offer critical insights for fine-tuning and configuring robust LLM-based ASR models, tailored to different data characteristics and computational resources.
• 摘要：最近的研究表明，在语音基础编码器和大型语言模型 (LLM) 之间训练线性连接器可使该架构实现强大的 ASR 功能。尽管结果令人印象深刻，但仍不清楚这些简单的方法是否在不同场景和语音条件（例如域转换和不同的语音扰动）中具有足够的鲁棒性。在本文中，我们通过使用一种最近被广泛采用的方法（称为 SLAM-ASR）进行各种消融实验来解决这些问题。我们提出了新颖的经验发现，为如何在广泛的环境中有效利用 SLAM-ASR 架构提供了见解。我们的主要发现表明，SLAM-ASR 在跨域评估设置中表现不佳。此外，域内数据中的语音扰动（例如速度变化或存在附加噪声）会显著影响性能。我们的研究结果为微调和配置基于 LLM 的鲁棒 ASR 模型提供了关键见解，这些模型可针对不同的数据特​​征和计算资源进行定制。

Title: MEG: Medical Knowledge-Augmented Large Language Models for Question Answering

• Authors: Laura Cabello, Carmen Martin-Turrero, Uchenna Akujuobi, Anders Søgaard, Carlos Bobed
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03883
• Pdf URL: https://arxiv.org/pdf/2411.03883
• Copy Paste: [[2411.03883]] MEG: Medical Knowledge-Augmented Large Language Models for Question Answering(https://arxiv.org/abs/2411.03883)
• Keywords: language model, llm
• Abstract: Question answering is a natural language understanding task that involves reasoning over both explicit context and unstated, relevant domain knowledge. Large language models (LLMs), which underpin most contemporary question answering systems, struggle to induce how concepts relate in specialized domains such as medicine. Existing medical LLMs are also costly to train. In this work, we present MEG, a parameter-efficient approach for medical knowledge-augmented LLMs. MEG uses a lightweight mapping network to integrate graph embeddings into the LLM, enabling it to leverage external knowledge in a cost-effective way. We evaluate our method on four popular medical multiple-choice datasets and show that LLMs greatly benefit from the factual grounding provided by knowledge graph embeddings. MEG attains an average of +10.2% accuracy over the Mistral-Instruct baseline, and +6.7% over specialized models like BioMistral. We also show results based on Llama-3. Finally, we show that MEG's performance remains robust to the choice of graph encoder.
• 摘要：问答是一种自然语言理解任务，涉及对显式上下文和未说明的相关领域知识进行推理。大型语言模型 (LLM) 是大多数当代问答系统的基础，很难推断出医学等专业领域中概念之间的关系。现有的医学 LLM 的训练成本也很高。在这项工作中，我们提出了 MEG，这是一种用于医学知识增强 LLM 的参数高效方法。MEG 使用轻量级映射网络将图嵌入集成到 LLM 中，使其能够以经济高效的方式利用外部知识。我们在四个流行的医学多项选择数据集上评估了我们的方法，并表明 LLM 极大地受益于知识图嵌入提供的事实基础。MEG 在 Mistral-Instruct 基线上的平均准确率达到 +10.2%，在 BioMistral 等专业模型上的平均准确率达到 +6.7%。我们还展示了基于 Llama-3 的结果。最后，我们表明 MEG 的性能对于图编码器的选择仍然具有鲁棒性。

Title: Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models

• Authors: Zhijian Zhuo, Ya Wang, Yutao Zeng, Xiaoqing Li, Xun Zhou, Jinwen Ma
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.03884
• Pdf URL: https://arxiv.org/pdf/2411.03884
• Copy Paste: [[2411.03884]] Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models(https://arxiv.org/abs/2411.03884)
• Keywords: language model, llm
• Abstract: Transformers have found extensive applications across various domains due to the powerful fitting capabilities. This success can be partially attributed to their inherent nonlinearity. Thus, in addition to the ReLU function employed in the original transformer architecture, researchers have explored alternative modules such as GeLU and SwishGLU to enhance nonlinearity and thereby augment representational capacity. In this paper, we propose a novel category of polynomial composition activations (PolyCom), designed to optimize the dynamics of transformers. Theoretically, we provide a comprehensive mathematical analysis of PolyCom, highlighting its enhanced expressivity and efficacy relative to other activation functions. Notably, we demonstrate that networks incorporating PolyCom achieve the $\textbf{optimal approximation rate}$, indicating that PolyCom networks require minimal parameters to approximate general smooth functions in Sobolev spaces. We conduct empirical experiments on the pre-training configurations of large language models (LLMs), including both dense and sparse architectures. By substituting conventional activation functions with PolyCom, we enable LLMs to capture higher-order interactions within the data, thus improving performance metrics in terms of accuracy and convergence rates. Extensive experimental results demonstrate the effectiveness of our method, showing substantial improvements over other activation functions. Code is available at this https URL.
• 摘要：由于强大的拟合能力，Transformer 在各个领域得到了广泛的应用。这一成功可以部分归因于其固有的非线性。因此，除了原始 Transformer 架构中使用的 ReLU 函数外，研究人员还探索了 GeLU 和 SwishGLU 等替代模块，以增强非线性，从而增强表示能力。在本文中，我们提出了一种新型多项式组合激活 (PolyCom)，旨在优化 Transformer 的动态。从理论上讲，我们对 PolyCom 进行了全面的数学分析，强调了它相对于其他激活函数增强的表达能力和有效性。值得注意的是，我们证明包含 PolyCom 的网络实现了 $\textbf{最佳近似率}$，这表明 PolyCom 网络需要最少的参数来近似 Sobolev 空间中的一般平滑函数。我们对大型语言模型 (LLM) 的预训练配置进行了实证实验，包括密集和稀疏架构。通过用 PolyCom 替代传统的激活函数，我们使 LLM 能够捕获数据中的高阶交互，从而提高准确度和收敛速度方面的性能指标。大量实验结果证明了我们方法的有效性，与其他激活函数相比有显著的改进。代码可在此 https URL 上找到。

Title: Multi3Hate: Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision-Language Models

• Authors: Minh Duc Bui, Katharina von der Wense, Anne Lauscher
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03888
• Pdf URL: https://arxiv.org/pdf/2411.03888
• Copy Paste: [[2411.03888]] Multi3Hate: Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision-Language Models(https://arxiv.org/abs/2411.03888)
• Keywords: language model, prompt
• Abstract: Warning: this paper contains content that may be offensive or upsetting Hate speech moderation on global platforms poses unique challenges due to the multimodal and multilingual nature of content, along with the varying cultural perceptions. How well do current vision-language models (VLMs) navigate these nuances? To investigate this, we create the first multimodal and multilingual parallel hate speech dataset, annotated by a multicultural set of annotators, called Multi3Hate. It contains 300 parallel meme samples across 5 languages: English, German, Spanish, Hindi, and Mandarin. We demonstrate that cultural background significantly affects multimodal hate speech annotation in our dataset. The average pairwise agreement among countries is just 74%, significantly lower than that of randomly selected annotator groups. Our qualitative analysis indicates that the lowest pairwise label agreement-only 67% between the USA and India-can be attributed to cultural factors. We then conduct experiments with 5 large VLMs in a zero-shot setting, finding that these models align more closely with annotations from the US than with those from other cultures, even when the memes and prompts are presented in the dominant language of the other culture. Code and dataset are available at this https URL.
• 摘要：警告：本文包含可能令人反感或不安的内容 由于内容的多模态和多语言性质，以及不同的文化观念，在全球平台上审核仇恨言论带来了独特的挑战。当前的视觉语言模型 (VLM) 如何很好地处理这些细微差别？为了研究这个问题，我们创建了第一个多模态和多语言并行仇恨言论数据集，由一组多元文化注释者进行注释，称为 Multi3Hate。它包含 5 种语言的 300 个并行模因样本：英语、德语、西班牙语、印地语和普通话。我们证明，文化背景显著影响我们数据集中的多模态仇恨言论注释。各国之间的平均成对一致性仅为 74%，明显低于随机选择的注释者组的一致性。我们的定性分析表明，最低的成对标签一致性——美国和印度之间仅为 67%——可以归因于文化因素。然后，我们在零样本设置中使用 5 个大型 VLM 进行实验，发现这些模型与来自美国的注释的匹配度比与来自其他文化的注释的匹配度更高，即使模因和提示以其他文化的主导语言呈现。代码和数据集可在此 https URL 上找到。

Title: RAGulator: Lightweight Out-of-Context Detectors for Grounded Text Generation

• Authors: Ian Poey, Jiajun Liu, Qishuai Zhong, Adrien Chenailler
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03920
• Pdf URL: https://arxiv.org/pdf/2411.03920
• Copy Paste: [[2411.03920]] RAGulator: Lightweight Out-of-Context Detectors for Grounded Text Generation(https://arxiv.org/abs/2411.03920)
• Keywords: llm
• Abstract: Real-time detection of out-of-context LLM outputs is crucial for enterprises looking to safely adopt RAG applications. In this work, we train lightweight models to discriminate LLM-generated text that is semantically out-of-context from retrieved text documents. We preprocess a combination of summarisation and semantic textual similarity datasets to construct training data using minimal resources. We find that DeBERTa is not only the best-performing model under this pipeline, but it is also fast and does not require additional text preprocessing or feature engineering. While emerging work demonstrates that generative LLMs can also be fine-tuned and used in complex data pipelines to achieve state-of-the-art performance, we note that speed and resource limits are important considerations for on-premise deployment.
• 摘要：对于希望安全采用 RAG 应用程序的企业来说，实时检测脱离上下文的 LLM 输出至关重要。在这项工作中，我们训练轻量级模型来区分从检索到的文本文档中语义上脱离上下文的 LLM 生成文本。我们对摘要和语义文本相似性数据集的组合进行预处理，以使用最少的资源构建训练数据。我们发现 DeBERTa 不仅是此管道下性能最佳的模型，而且速度很快，不需要额外的文本预处理或特征工程。虽然新兴的研究表明生成式 LLM 也可以进行微调并用于复杂的数据管道以实现最先进的性能，但我们注意到速度和资源限制是本地部署的重要考虑因素。

Title: Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?

• Authors: Aaditya K. Singh, Muhammed Yusuf Kocyigit, Andrew Poulton, David Esiobu, Maria Lomeli, Gergely Szilvasy, Dieuwke Hupkes
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03923
• Pdf URL: https://arxiv.org/pdf/2411.03923
• Copy Paste: [[2411.03923]] Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?(https://arxiv.org/abs/2411.03923)
• Keywords: llm
• Abstract: Hampering the interpretation of benchmark scores, evaluation data contamination has become a growing concern in the evaluation of LLMs, and an active area of research studies its effects. While evaluation data contamination is easily understood intuitively, it is surprisingly difficult to define precisely which samples should be considered contaminated and, consequently, how it impacts benchmark scores. We propose that these questions should be addressed together and that contamination metrics can be assessed based on whether models benefit from the examples they mark contaminated. We propose a novel analysis method called ConTAM, and show with a large scale survey of existing and novel n-gram based contamination metrics across 13 benchmarks and 7 models from 2 different families that ConTAM can be used to better understand evaluation data contamination and its effects. We find that contamination may have a much larger effect than reported in recent LLM releases and benefits models differently at different scales. We also find that considering only the longest contaminated substring provides a better signal than considering a union of all contaminated substrings, and that doing model and benchmark specific threshold analysis greatly increases the specificity of the results. Lastly, we investigate the impact of hyperparameter choices, finding that, among other things, both using larger values of n and disregarding matches that are infrequent in the pre-training data lead to many false negatives. With ConTAM, we provide a method to empirically ground evaluation data contamination metrics in downstream effects. With our exploration, we shed light on how evaluation data contamination can impact LLMs and provide insight into the considerations important when doing contamination analysis. We end our paper by discussing these in more detail and providing concrete suggestions for future work.
• 摘要：评估数据污染妨碍了对基准分数的解释，已成为 LLM 评估中日益关注的问题，并且一个活跃的研究领域正在研究其影响。虽然评估数据污染很容易直观地理解，但要准确定义哪些样本应被视为受污染，以及它如何影响基准分数却出奇地困难。我们建议应该一起解决这些问题，并可以根据模型是否从它们标记为受污染的示例中受益来评估污染指标。我们提出了一种名为 ConTAM 的新型分析方法，并通过对 13 个基准和来自 2 个不同系列的 7 个模型的现有和新型 n-gram 污染指标进行大规模调查，表明 ConTAM 可用于更好地理解评估数据污染及其影响。我们发现污染的影响可能比最近发布的 LLM 报告的要大得多，并且在不同的规模上对模型有不同的好处。我们还发现，仅考虑最长的受污染子字符串比考虑所有受污染子字符串的并集能提供更好的信号，并且进行模型和基准特定阈值分析会大大提高结果的特异性。最后，我们研究了超参数选择的影响，发现除其他因素外，使用较大的 n 值和忽略预训练数据中不频繁的匹配都会导致许多假阴性。借助 ConTAM，我们提供了一种方法，可以在下游效应中实证地确定评估数据污染指标。通过我们的探索，我们阐明了评估数据污染如何影响 LLM，并深入了解了进行污染分析时需要考虑的重要事项。我们在论文的最后更详细地讨论了这些问题，并为未来的工作提供了具体的建议。

Title: How Does A Text Preprocessing Pipeline Affect Ontology Syntactic Matching?

• Authors: Zhangcheng Qiang, Kerry Taylor, Weiqing Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.03962
• Pdf URL: https://arxiv.org/pdf/2411.03962
• Copy Paste: [[2411.03962]] How Does A Text Preprocessing Pipeline Affect Ontology Syntactic Matching?(https://arxiv.org/abs/2411.03962)
• Keywords: language model, llm
• Abstract: The generic text preprocessing pipeline, comprising Tokenisation, Normalisation, Stop Words Removal, and Stemming/Lemmatisation, has been implemented in many ontology matching (OM) systems. However, the lack of standardisation in text preprocessing creates diversity in mapping results. In this paper, we investigate the effect of the text preprocessing pipeline on OM tasks at syntactic levels. Our experiments on 8 Ontology Alignment Evaluation Initiative (OAEI) track repositories with 49 distinct alignments indicate: (1) Tokenisation and Normalisation are currently more effective than Stop Words Removal and Stemming/Lemmatisation; and (2) The selection of Lemmatisation and Stemming is task-specific. We recommend standalone Lemmatisation or Stemming with post-hoc corrections. We find that (3) Porter Stemmer and Snowball Stemmer perform better than Lancaster Stemmer; and that (4) Part-of-Speech (POS) Tagging does not help Lemmatisation. To repair less effective Stop Words Removal and Stemming/Lemmatisation used in OM tasks, we propose a novel context-based pipeline repair approach that significantly improves matching correctness and overall matching performance. We also discuss the use of text preprocessing pipeline in the new era of large language models (LLMs).
• 摘要：通用文本预处理流程包括标记化、规范化、停用词删除和词干提取/词形还原，已在许多本体匹配 (OM) 系统中实现。然而，文本预处理缺乏标准化，导致映射结果存在多样性。在本文中，我们研究了文本预处理流程在句法层面对 OM 任务的影响。我们在 8 个本体对齐评估倡议 (OAEI) 轨道存储库上进行的具有 49 种不同对齐的实验表明：(1) 标记化和规范化目前比停用词删除和词干提取/词形还原更有效；(2) 词形还原和词干提取的选择是特定于任务的。我们建议使用独立的词形还原或事后更正的词干提取。我们发现 (3) Porter 词干提取器和 Snowball 词干提取器比 Lancaster 词干提取器表现更好；(4) 词性 (POS) 标记无助于词形还原。为了修复 OM 任务中使用的效果较差的停用词删除和词干提取/词形还原，我们提出了一种新颖的基于上下文的管道修复方法，可显著提高匹配正确性和整体匹配性能。我们还讨论了在大型语言模型 (LLM) 新时代使用文本预处理管道。

Title: What Really is Commonsense Knowledge?

• Authors: Quyet V. Do, Junze Li, Tung-Duong Vuong, Zhaowei Wang, Yangqiu Song, Xiaojuan Ma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.03964
• Pdf URL: https://arxiv.org/pdf/2411.03964
• Copy Paste: [[2411.03964]] What Really is Commonsense Knowledge?(https://arxiv.org/abs/2411.03964)
• Keywords: language model, llm
• Abstract: Commonsense datasets have been well developed in Natural Language Processing, mainly through crowdsource human annotation. However, there are debates on the genuineness of commonsense reasoning benchmarks. In specific, a significant portion of instances in some commonsense benchmarks do not concern commonsense knowledge. That problem would undermine the measurement of the true commonsense reasoning ability of evaluated models. It is also suggested that the problem originated from a blurry concept of commonsense knowledge, as distinguished from other types of knowledge. To demystify all of the above claims, in this study, we survey existing definitions of commonsense knowledge, ground into the three frameworks for defining concepts, and consolidate them into a multi-framework unified definition of commonsense knowledge (so-called consolidated definition). We then use the consolidated definition for annotations and experiments on the CommonsenseQA and CommonsenseQA 2.0 datasets to examine the above claims. Our study shows that there exists a large portion of non-commonsense-knowledge instances in the two datasets, and a large performance gap on these two subsets where Large Language Models (LLMs) perform worse on commonsense-knowledge instances.
• 摘要：常识数据集在自然语言处理中已经得到了很好的发展，主要通过众包人工注释实现。然而，关于常识推理基准的真实性存在争议。具体而言，一些常识基准中相当一部分实例并不涉及常识知识。这个问题会破坏对被评估模型真正的常识推理能力的衡量。也有人认为，问题的根源在于常识知识的概念模糊，与其他类型的知识区分开来。为了揭开上述所有说法的神秘面纱，在本研究中，我们调查了现有的常识知识定义，扎根于三个定义概念的框架，并将它们合并为一个多框架的统一常识知识定义（所谓的合并定义）。然后，我们使用合并定义进行注释，并在 CommonsenseQA 和 CommonsenseQA 2.0 数据集上进行实验，以检验上述说法。我们的研究表明，这两个数据集中都存在大量非常识知识实例，且在这两个子集上存在很大的性能差距，其中大型语言模型 (LLM) 在常识知识实例上的表现较差。

Title: Prompt Engineering Using GPT for Word-Level Code-Mixed Language Identification in Low-Resource Dravidian Languages

• Authors: Aniket Deroy, Subhankar Maity
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04025
• Pdf URL: https://arxiv.org/pdf/2411.04025
• Copy Paste: [[2411.04025]] Prompt Engineering Using GPT for Word-Level Code-Mixed Language Identification in Low-Resource Dravidian Languages(https://arxiv.org/abs/2411.04025)
• Keywords: language model, gpt, prompt
• Abstract: Language Identification (LI) is crucial for various natural language processing tasks, serving as a foundational step in applications such as sentiment analysis, machine translation, and information retrieval. In multilingual societies like India, particularly among the youth engaging on social media, text often exhibits code-mixing, blending local languages with English at different linguistic levels. This phenomenon presents formidable challenges for LI systems, especially when languages intermingle within single words. Dravidian languages, prevalent in southern India, possess rich morphological structures yet suffer from under-representation in digital platforms, leading to the adoption of Roman or hybrid scripts for communication. This paper introduces a prompt based method for a shared task aimed at addressing word-level LI challenges in Dravidian languages. In this work, we leveraged GPT-3.5 Turbo to understand whether the large language models is able to correctly classify words into correct categories. Our findings show that the Kannada model consistently outperformed the Tamil model across most metrics, indicating a higher accuracy and reliability in identifying and categorizing Kannada language instances. In contrast, the Tamil model showed moderate performance, particularly needing improvement in precision and recall.
• 摘要：语言识别 (LI) 对于各种自然语言处理任务至关重要，是情感分析、机器翻译和信息检索等应用的基础步骤。在印度这样的多语言社会中，尤其是在社交媒体上的年轻人中，文本经常表现出代码混合，在不同的语言层面上将当地语言与英语混合在一起。这种现象给 LI 系统带来了巨大的挑战，尤其是当语言在单个单词中混合时。在印度南部流行的达罗毗荼语拥有丰富的形态结构，但在数字平台上却缺乏代表性，导致人们采用罗马或混合文字进行交流。本文介绍了一种基于提示的共享任务方法，旨在解决达罗毗荼语中单词级的 LI 挑战。在这项工作中，我们利用 GPT-3.5 Turbo 来了解大型语言模型是否能够正确地将单词归类到正确的类别中。我们的研究结果表明，卡纳达语模型在大多数指标上始终优于泰米尔语模型，表明在识别和分类卡纳达语实例方面具有更高的准确性和可靠性。相比之下，泰米尔语模型表现一般，尤其需要提高准确率和召回率。

Title: Beemo: Benchmark of Expert-edited Machine-generated Outputs

• Authors: Ekaterina Artemova, Jason Lucas, Saranya Venkatraman, Jooyoung Lee, Sergei Tilga, Adaku Uchendu, Vladislav Mikhailov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04032
• Pdf URL: https://arxiv.org/pdf/2411.04032
• Copy Paste: [[2411.04032]] Beemo: Benchmark of Expert-edited Machine-generated Outputs(https://arxiv.org/abs/2411.04032)
• Keywords: language model, llm
• Abstract: The rapid proliferation of large language models (LLMs) has increased the volume of machine-generated texts (MGTs) and blurred text authorship in various domains. However, most existing MGT benchmarks include single-author texts (human-written and machine-generated). This conventional design fails to capture more practical multi-author scenarios, where the user refines the LLM response for natural flow, coherence, and factual correctness. Our paper introduces the Benchmark of Expert-edited Machine-generated Outputs (Beemo), which includes 6.5k texts written by humans, generated by ten instruction-finetuned LLMs, and edited by experts for various use cases, ranging from creative writing to summarization. Beemo additionally comprises 13.1k machine-generated and LLM-edited texts, allowing for diverse MGT detection evaluation across various edit types. We document Beemo's creation protocol and present the results of benchmarking 33 configurations of MGT detectors in different experimental setups. We find that expert-based editing evades MGT detection, while LLM-edited texts are unlikely to be recognized as human-written. Beemo and all materials are publicly available.
• 摘要：大型语言模型 (LLM) 的快速普及增加了机器生成文本 (MGT) 的数量，并模糊了各个领域的文本作者身份。然而，大多数现有的 MGT 基准都包括单一作者文本（人工编写和机器生成）。这种传统设计无法捕捉更实际的多作者场景，在这些场景中，用户会优化 LLM 响应以实现自然流畅、连贯性和事实正确性。我们的论文介绍了专家编辑的机器生成输出基准 (Beemo)，其中包括 6.5k 篇由人类编写的文本，由十个指令微调的 LLM 生成，并由专家针对从创意写作到总结的各种用例进行编辑。Beemo 还包括 13.1k 篇机器生成和 LLM 编辑的文本，允许在各种编辑类型中进行不同的 MGT 检测评估。我们记录了 Beemo 的创建协议，并展示了在不同实验设置中对 33 种 MGT 检测器配置进行基准测试的结果。我们发现，专家编辑可以逃避 MGT 检测，而 LLM 编辑的文本不太可能被识别为人工编写。Beemo 和所有材料均可供公众使用。

Title: Summarization of Opinionated Political Documents with Varied Perspectives

• Authors: Nicholas Deas, Kathleen McKeown
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.04093
• Pdf URL: https://arxiv.org/pdf/2411.04093
• Copy Paste: [[2411.04093]] Summarization of Opinionated Political Documents with Varied Perspectives(https://arxiv.org/abs/2411.04093)
• Keywords: gpt
• Abstract: Global partisan hostility and polarization has increased, and this polarization is heightened around presidential elections. Models capable of generating accurate summaries of diverse perspectives can help reduce such polarization by exposing users to alternative perspectives. In this work, we introduce a novel dataset and task for independently summarizing each political perspective in a set of passages from opinionated news articles. For this task, we propose a framework for evaluating different dimensions of perspective summary performance. We benchmark 10 models of varying sizes and architectures through both automatic and human evaluation. While recent models like GPT-4o perform well on this task, we find that all models struggle to generate summaries faithful to the intended perspective. Our analysis of summaries focuses on how extraction behavior depends on the features of the input documents.
• 摘要：全球党派敌意和两极分化加剧，这种两极分化在总统选举期间尤为严重。能够生成不同观点准确摘要的模型可以通过向用户展示不同的观点来帮助减少这种两极分化。在这项工作中，我们引入了一个新颖的数据集和任务，用于独立总结一组观点性新闻文章段落中的每个政治观点。对于这项任务，我们提出了一个框架来评估观点摘要性能的不同维度。我们通过自动和人工评估对 10 个不同大小和架构的模型进行了基准测试。虽然最近的模型（如 GPT-4o）在此任务上表现良好，但我们发现所有模型都难以生成忠实于预期观点的摘要。我们对摘要的分析重点在于提取行为如何依赖于输入文档的特征。

Title: Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?

• Authors: Daniel P. Jeong, Saurabh Garg, Zachary C. Lipton, Michael Oberst
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.04118
• Pdf URL: https://arxiv.org/pdf/2411.04118
• Copy Paste: [[2411.04118]] Medical Adaptation of Large Language and Vision-Language Models: Are We Making Progress?(https://arxiv.org/abs/2411.04118)
• Keywords: language model, llm, prompt
• Abstract: Several recent works seek to develop foundation models specifically for medical applications, adapting general-purpose large language models (LLMs) and vision-language models (VLMs) via continued pretraining on publicly available biomedical corpora. These works typically claim that such domain-adaptive pretraining (DAPT) improves performance on downstream medical tasks, such as answering medical licensing exam questions. In this paper, we compare seven public "medical" LLMs and two VLMs against their corresponding base models, arriving at a different conclusion: all medical VLMs and nearly all medical LLMs fail to consistently improve over their base models in the zero-/few-shot prompting regime for medical question-answering (QA) tasks. For instance, across the tasks and model pairs we consider in the 3-shot setting, medical LLMs only outperform their base models in 12.1% of cases, reach a (statistical) tie in 49.8% of cases, and are significantly worse than their base models in the remaining 38.2% of cases. Our conclusions are based on (i) comparing each medical model head-to-head, directly against the corresponding base model; (ii) optimizing the prompts for each model separately; and (iii) accounting for statistical uncertainty in comparisons. While these basic practices are not consistently adopted in the literature, our ablations show that they substantially impact conclusions. Our findings suggest that state-of-the-art general-domain models may already exhibit strong medical knowledge and reasoning capabilities, and offer recommendations to strengthen the conclusions of future studies.
• 摘要：最近有几篇论文试图开发专门用于医学应用的基础模型，通过在公开的生物医学语料库上进行持续预训练来调整通用大型语言模型 (LLM) 和视觉语言模型 (VLM)。这些论文通常声称，这种领域自适应预训练 (DAPT) 可以提高下游医学任务（例如回答医学执照考试问题）的性能。在本文中，我们将七个公共“医学”LLM 和两个 VLM 与它们相应的基础模型进行比较，得出了一个不同的结论：在医学问答 (QA) 任务的零/少样本提示机制中，所有医学 VLM 和几乎所有医学 LLM 都无法持续改进其基础模型。例如，在我们考虑的 3 次设置中的任务和模型对中，医学 LLM 仅在 12.1% 的情况下优于其基础模型，在 49.8% 的情况下达到（统计）平局，并且在其余 38.2% 的情况下明显比其基础模型差。我们的结论基于以下几点：(i) 将每个医学模型与相应的基础模型直接进行比较；(ii) 分别优化每个模型的提示；(iii) 考虑比较中的统计不确定性。虽然这些基本做法在文献中并未得到一致采用，但我们的修正表明它们对结论有重大影响。我们的研究结果表明，最先进的通用领域模型可能已经展现出强大的医学知识和推理能力，并提出了加强未来研究结论的建议。