2024-09-30

Title: Evaluation of Large Language Models for Summarization Tasks in the Medical Domain: A Narrative Review

• Authors: Emma Croxford, Yanjun Gao, Nicholas Pellegrino, Karen K. Wong, Graham Wills, Elliot First, Frank J. Liao, Cherodeep Goswami, Brian Patterson, Majid Afshar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18170
• Pdf URL: https://arxiv.org/pdf/2409.18170
• Copy Paste: [[2409.18170]] Evaluation of Large Language Models for Summarization Tasks in the Medical Domain: A Narrative Review(https://arxiv.org/abs/2409.18170)
• Keywords: language model
• Abstract: Large Language Models have advanced clinical Natural Language Generation, creating opportunities to manage the volume of medical text. However, the high-stakes nature of medicine requires reliable evaluation, which remains a challenge. In this narrative review, we assess the current evaluation state for clinical summarization tasks and propose future directions to address the resource constraints of expert human evaluation.
• 摘要：大型语言模型推动了临床自然语言生成的发展，为管理大量医学文本创造了机会。然而，医学的高风险性质要求可靠的评估，这仍然是一个挑战。在这篇叙述性评论中，我们评估了临床总结任务的当前评估状态，并提出了解决专家人工评估资源限制的未来方向。

Title: LowREm: A Repository of Word Embeddings for 87 Low-Resource Languages Enhanced with Multilingual Graph Knowledge

• Authors: Daniil Gurgurov, Rishu Kumar, Simon Ostermann
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18193
• Pdf URL: https://arxiv.org/pdf/2409.18193
• Copy Paste: [[2409.18193]] LowREm: A Repository of Word Embeddings for 87 Low-Resource Languages Enhanced with Multilingual Graph Knowledge(https://arxiv.org/abs/2409.18193)
• Keywords: language model, llm
• Abstract: Contextualized embeddings based on large language models (LLMs) are available for various languages, but their coverage is often limited for lower resourced languages. Training LLMs for such languages is often difficult due to insufficient data and high computational cost. Especially for very low resource languages, static word embeddings thus still offer a viable alternative. There is, however, a notable lack of comprehensive repositories with such embeddings for diverse languages. To address this, we present LowREm, a centralized repository of static embeddings for 87 low-resource languages. We also propose a novel method to enhance GloVe-based embeddings by integrating multilingual graph knowledge, utilizing another source of knowledge. We demonstrate the superior performance of our enhanced embeddings as compared to contextualized embeddings extracted from XLM-R on sentiment analysis. Our code and data are publicly available under this https URL.
• 摘要：基于大型语言模型 (LLM) 的语境化嵌入可用于各种语言，但它们的覆盖范围通常仅限于资源较少的语言。由于数据不足和计算成本高，训练此类语言的 LLM 通常很困难。因此，特别是对于资源非常少的语言，静态词嵌入仍然是一种可行的替代方案。然而，对于各种语言，缺乏具有此类嵌入的综合存储库。为了解决这个问题，我们提出了 LowREm，这是一个集中式的 87 种资源较少语言的静态嵌入存储库。我们还提出了一种新方法，通过集成多语言图知识，利用另一种知识来源来增强基于 GloVe 的嵌入。我们展示了与从 XLM-R 中提取的语境化嵌入相比，我们的增强嵌入在情感分析方面具有更优越的性能。我们的代码和数据在此 https URL 下公开提供。

Title: LangSAMP: Language-Script Aware Multilingual Pretraining

• Authors: Yihong Liu, Haotian Ye, Chunlan Ma, Mingyang Wang, Hinrich Schütze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18199
• Pdf URL: https://arxiv.org/pdf/2409.18199
• Copy Paste: [[2409.18199]] LangSAMP: Language-Script Aware Multilingual Pretraining(https://arxiv.org/abs/2409.18199)
• Keywords: language model
• Abstract: Recent multilingual pretrained language models (mPLMs) often avoid using language embeddings -- learnable vectors assigned to different languages. These embeddings are discarded for two main reasons: (1) mPLMs are expected to have a single, unified parameter set across all languages, and (2) they need to function seamlessly as universal text encoders without requiring language IDs as input. However, this removal increases the burden on token embeddings to encode all language-specific information, which may hinder the model's ability to produce more language-neutral representations. To address this challenge, we propose Language-Script Aware Multilingual Pretraining (LangSAMP), a method that incorporates both language and script embeddings to enhance representation learning while maintaining a simple architecture. Specifically, we integrate these embeddings into the output of the transformer blocks before passing the final representations to the language modeling head for prediction. We apply LangSAMP to the continual pretraining of XLM-R on a highly multilingual corpus covering more than 500 languages. The resulting model consistently outperforms the baseline. Extensive analysis further shows that language/script embeddings encode language/script-specific information, which improves the selection of source languages for crosslingual transfer. We make our code and models publicly available at \url{this https URL}.
• 摘要：最近的多语言预训练语言模型 (mPLM) 通常避免使用语言嵌入——分配给不同语言的可学习向量。这些嵌入被丢弃有两个主要原因：(1) mPLM 应该在所有语言中具有一个统一的参数集，(2) 它们需要无缝地用作通用文本编码器，而无需语言 ID 作为输入。然而，这种删除增加了标记嵌入对所有语言特定信息的编码负担，这可能会妨碍模型生成更多语言中立表示的能力。为了应对这一挑战，我们提出了语言脚本感知多语言预训练 (LangSAMP)，这种方法结合了语言和脚本嵌入来增强表示学习，同时保持简单的架构。具体来说，我们将这些嵌入集成到转换器块的输出中，然后将最终表示传递给语言建模头进行预测。我们将 LangSAMP 应用于在涵盖 500 多种语言的高度多语言语料库上对 XLM-R 进行持续预训练。最终的模型始终优于基线。广泛的分析进一步表明，语言/脚本嵌入编码了特定于语言/脚本的信息，从而改善了跨语言传输的源语言选择。我们将我们的代码和模型公开发布在 \url{此 https URL} 上。

Title: DisGeM: Distractor Generation for Multiple Choice Questions with Span Masking

• Authors: Devrim Cavusoglu, Secil Sen, Ulas Sert
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18263
• Pdf URL: https://arxiv.org/pdf/2409.18263
• Copy Paste: [[2409.18263]] DisGeM: Distractor Generation for Multiple Choice Questions with Span Masking(https://arxiv.org/abs/2409.18263)
• Keywords: language model
• Abstract: Recent advancements in Natural Language Processing (NLP) have impacted numerous sub-fields such as natural language generation, natural language inference, question answering, and more. However, in the field of question generation, the creation of distractors for multiple-choice questions (MCQ) remains a challenging task. In this work, we present a simple, generic framework for distractor generation using readily available Pre-trained Language Models (PLMs). Unlike previous methods, our framework relies solely on pre-trained language models and does not require additional training on specific datasets. Building upon previous research, we introduce a two-stage framework consisting of candidate generation and candidate selection. Our proposed distractor generation framework outperforms previous methods without the need for training or fine-tuning. Human evaluations confirm that our approach produces more effective and engaging distractors. The related codebase is publicly available at this https URL.
• 摘要：自然语言处理 (NLP) 的最新进展影响了许多子领域，例如自然语言生成、自然语言推理、问答等。然而，在问题生成领域，为多项选择题 (MCQ) 创建干扰项仍然是一项艰巨的任务。在这项工作中，我们使用现成的预训练语言模型 (PLM) 提出了一个简单、通用的干扰项生成框架。与以前的方法不同，我们的框架完全依赖于预训练的语言模型，不需要对特定数据集进行额外训练。在以前研究的基础上，我们引入了一个由候选生成和候选选择组成的两阶段框架。我们提出的干扰项生成框架优于以前的方法，无需训练或微调。人工评估证实，我们的方法可以产生更有效、更吸引人的干扰项。相关代码库可在此 https URL 上公开获取。

Title: AER-LLM: Ambiguity-aware Emotion Recognition Leveraging Large Language Models

• Authors: Xin Hong, Yuan Gong, Vidhyasaharan Sethu, Ting Dang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18339
• Pdf URL: https://arxiv.org/pdf/2409.18339
• Copy Paste: [[2409.18339]] AER-LLM: Ambiguity-aware Emotion Recognition Leveraging Large Language Models(https://arxiv.org/abs/2409.18339)
• Keywords: language model, llm, prompt
• Abstract: Recent advancements in Large Language Models (LLMs) have demonstrated great success in many Natural Language Processing (NLP) tasks. In addition to their cognitive intelligence, exploring their capabilities in emotional intelligence is also crucial, as it enables more natural and empathetic conversational AI. Recent studies have shown LLMs' capability in recognizing emotions, but they often focus on single emotion labels and overlook the complex and ambiguous nature of human emotions. This study is the first to address this gap by exploring the potential of LLMs in recognizing ambiguous emotions, leveraging their strong generalization capabilities and in-context learning. We design zero-shot and few-shot prompting and incorporate past dialogue as context information for ambiguous emotion recognition. Experiments conducted using three datasets indicate significant potential for LLMs in recognizing ambiguous emotions, and highlight the substantial benefits of including context information. Furthermore, our findings indicate that LLMs demonstrate a high degree of effectiveness in recognizing less ambiguous emotions and exhibit potential for identifying more ambiguous emotions, paralleling human perceptual capabilities.
• 摘要：大型语言模型 (LLM) 的最新进展已在许多自然语言处理 (NLP) 任务中取得了巨大成功。除了认知智能之外，探索其情商能力也至关重要，因为它可以实现更自然、更具同理心的对话式 AI。最近的研究表明 LLM 具有识别情绪的能力，但它们通常专注于单一情绪标签，而忽略了人类情绪的复杂和模糊性。这项研究首次通过探索 LLM 在识别模糊情绪方面的潜力来解决这一差距，利用其强大的泛化能力和情境学习。我们设计了零样本和少样本提示，并将过去的对话作为模糊情绪识别的情境信息。使用三个数据集进行的实验表明 LLM 在识别模糊情绪方面具有巨大的潜力，并强调了包含情境信息的巨大好处。此外，我们的研究结果表明，LLM 在识别不太模糊的情绪方面表现出很高的有效性，并表现出识别更模糊情绪的潜力，与人类的感知能力相似。

Title: A Generalized LLM-Augmented BIM Framework: Application to a Speech-to-BIM system

• Authors: Ghang Lee, Suhyung Jang, Seokho Hyun
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.18345
• Pdf URL: https://arxiv.org/pdf/2409.18345
• Copy Paste: [[2409.18345]] A Generalized LLM-Augmented BIM Framework: Application to a Speech-to-BIM system(https://arxiv.org/abs/2409.18345)
• Keywords: language model, llm
• Abstract: Performing building information modeling (BIM) tasks is a complex process that imposes a steep learning curve and a heavy cognitive load due to the necessity of remembering sequences of numerous commands. With the rapid advancement of large language models (LLMs), it is foreseeable that BIM tasks, including querying and managing BIM data, 4D and 5D BIM, design compliance checking, or authoring a design, using written or spoken natural language (i.e., text-to-BIM or speech-to-BIM), will soon supplant traditional graphical user interfaces. This paper proposes a generalized LLM-augmented BIM framework to expedite the development of LLM-enhanced BIM applications by providing a step-by-step development process. The proposed framework consists of six steps: interpret-fill-match-structure-execute-check. The paper demonstrates the applicability of the proposed framework through implementing a speech-to-BIM application, NADIA-S (Natural-language-based Architectural Detailing through Interaction with Artificial Intelligence via Speech), using exterior wall detailing as an example.
• 摘要：执行建筑信息模型 (BIM) 任务是一个复杂的过程，由于需要记住大量命令序列，因此学习难度高，认知负担重。随着大型语言模型 (LLM) 的快速发展，可以预见，BIM 任务（包括查询和管理 BIM 数据、4D 和 5D BIM、设计合规性检查或使用书面或口头自然语言（即文本到 BIM 或语音到 BIM）创作设计）将很快取代传统的图形用户界面。本文提出了一种通用的 LLM 增强型 BIM 框架，通过提供分步开发流程来加快 LLM 增强型 BIM 应用程序的开发。提出的框架包括六个步骤：解释-填充-匹配-结构-执行-检查。本文通过实现语音到 BIM 应用程序 NADIA-S（通过语音与人工智能交互的基于自然语言的建筑细节），以外墙细节为例，证明了所提出的框架的适用性。

Title: MultiClimate: Multimodal Stance Detection on Climate Change Videos

• Authors: Jiawen Wang, Longfei Zuo, Siyao Peng, Barbara Plank
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18346
• Pdf URL: https://arxiv.org/pdf/2409.18346
• Copy Paste: [[2409.18346]] MultiClimate: Multimodal Stance Detection on Climate Change Videos(https://arxiv.org/abs/2409.18346)
• Keywords: language model
• Abstract: Climate change (CC) has attracted increasing attention in NLP in recent years. However, detecting the stance on CC in multimodal data is understudied and remains challenging due to a lack of reliable datasets. To improve the understanding of public opinions and communication strategies, this paper presents MultiClimate, the first open-source manually-annotated stance detection dataset with $100$ CC-related YouTube videos and $4,209$ frame-transcript pairs. We deploy state-of-the-art vision and language models, as well as multimodal models for MultiClimate stance detection. Results show that text-only BERT significantly outperforms image-only ResNet50 and ViT. Combining both modalities achieves state-of-the-art, $0.747$/$0.749$ in accuracy/F1. Our 100M-sized fusion models also beat CLIP and BLIP, as well as the much larger 9B-sized multimodal IDEFICS and text-only Llama3 and Gemma2, indicating that multimodal stance detection remains challenging for large language models. Our code, dataset, as well as supplementary materials, are available at this https URL.
• 摘要：近年来，气候变化 (CC) 在 NLP 领域引起了越来越多的关注。然而，由于缺乏可靠的数据集，检测多模态数据中对 CC 的立场研究不足且仍然具有挑战性。为了提高对公众观点和沟通策略的理解，本文介绍了 MultiClimate，这是第一个开源手动注释的立场检测数据集，其中包含 100 个与 CC 相关的 YouTube 视频和 4,209 个帧转录对。我们部署了最先进的视觉和语言模型，以及用于 MultiClimate 立场检测的多模态模型。结果表明，纯文本 BERT 明显优于纯图像的 ResNet50 和 ViT。结合两种模态可实现最先进的准确度/F1，分别为 0.747/0.749。我们的 100M 大小的融合模型也击败了 CLIP 和 BLIP，以及更大的 9B 大小的多模态 IDEFICS 和纯文本 Llama3 和 Gemma2，这表明多模态立场检测对于大型语言模型来说仍然具有挑战性。我们的代码、数据集以及补充材料可在此 https URL 上找到。

Title: SciDFM: A Large Language Model with Mixture-of-Experts for Science

• Authors: Liangtai Sun, Danyu Luo, Da Ma, Zihan Zhao, Baocai Chen, Zhennan Shen, Su Zhu, Lu Chen, Xin Chen, Kai Yu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18412
• Pdf URL: https://arxiv.org/pdf/2409.18412
• Copy Paste: [[2409.18412]] SciDFM: A Large Language Model with Mixture-of-Experts for Science(https://arxiv.org/abs/2409.18412)
• Keywords: language model, llm
• Abstract: Recently, there has been a significant upsurge of interest in leveraging large language models (LLMs) to assist scientific discovery. However, most LLMs only focus on general science, while they lack domain-specific knowledge, such as chemical molecules and amino acid sequences. To bridge these gaps, we introduce SciDFM, a mixture-of-experts LLM, which is trained from scratch and is able to conduct college-level scientific reasoning and understand molecules and amino acid sequences. We collect a large-scale training corpus containing numerous scientific papers and books from different disciplines as well as data from domain-specific databases. We further fine-tune the pre-trained model on lots of instruction data to improve performances on downstream benchmarks. From experiment results, we show that SciDFM achieves strong performance on general scientific benchmarks such as SciEval and SciQ, and it reaches a SOTA performance on domain-specific benchmarks among models of similar size. We further analyze the expert layers and show that the results of expert selection vary with data from different disciplines. To benefit the broader research community, we open-source SciDFM at this https URL.
• 摘要：最近，人们对利用大型语言模型 (LLM) 来协助科学发现的兴趣大增。然而，大多数 LLM 只关注一般科学，而缺乏特定领域的知识，例如化学分子和氨基酸序列。为了弥补这些差距，我们引入了 SciDFM，这是一种混合专家的 LLM，它从头开始训练，能够进行大学水平的科学推理并理解分子和氨基酸序列。我们收集了一个大规模的训练语料库，其中包含来自不同学科的大量科学论文和书籍以及来自领域特定数据库的数据。我们进一步在大量指令数据上微调预训练模型，以提高下游基准测试的性能。从实验结果来看，我们表明 SciDFM 在 SciEval 和 SciQ 等通用科学基准测试上取得了强劲的表现，并且在类似规模的模型中，它在领域特定基准测试上达到了 SOTA 性能。我们进一步分析了专家层，并表明专家选择的结果因来自不同学科的数据而异。为了造福更广泛的研究界，我们在此 https URL 上开源了 SciDFM。

Title: Improving Multilingual ASR in the Wild Using Simple N-best Re-ranking

• Authors: Brian Yan, Vineel Pratap, Shinji Watanabe, Michael Auli
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.18428
• Pdf URL: https://arxiv.org/pdf/2409.18428
• Copy Paste: [[2409.18428]] Improving Multilingual ASR in the Wild Using Simple N-best Re-ranking(https://arxiv.org/abs/2409.18428)
• Keywords: language model
• Abstract: Multilingual Automatic Speech Recognition (ASR) models are typically evaluated in a setting where the ground-truth language of the speech utterance is known, however, this is often not the case for most practical settings. Automatic Spoken Language Identification (SLID) models are not perfect and misclassifications have a substantial impact on the final ASR accuracy. In this paper, we present a simple and effective N-best re-ranking approach to improve multilingual ASR accuracy for several prominent acoustic models by employing external features such as language models and text-based language identification models. Our results on FLEURS using the MMS and Whisper models show spoken language identification accuracy improvements of 8.7% and 6.1%, respectively and word error rates which are 3.3% and 2.0% lower on these benchmarks.
• 摘要：多语言自动语音识别 (ASR) 模型通常在已知语音真实语言的环境中进行评估，然而，在大多数实际环境中，情况往往并非如此。自动口语语言识别 (SLID) 模型并不完美，错误分类会对最终的 ASR 准确率产生重大影响。在本文中，我们提出了一种简单有效的 N-best 重新排序方法，通过采用语言模型和基于文本的语言识别模型等外部特征来提高几种著名声学模型的多语言 ASR 准确率。我们在 FLEURS 上使用 MMS 和 Whisper 模型的结果显示，口语语言识别准确率分别提高了 8.7% 和 6.1%，而这些基准上的单词错误率分别降低了 3.3% 和 2.0%。

Title: Exploring Language Model Generalization in Low-Resource Extractive QA

• Authors: Saptarshi Sengupta, Wenpeng Yin, Preslav Nakov, Shreya Ghosh, Suhang Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18446
• Pdf URL: https://arxiv.org/pdf/2409.18446
• Copy Paste: [[2409.18446]] Exploring Language Model Generalization in Low-Resource Extractive QA(https://arxiv.org/abs/2409.18446)
• Keywords: language model, llm
• Abstract: In this paper, we investigate Extractive Question Answering (EQA) with Large Language Models (LLMs) under domain drift, i.e., can LLMs generalize well to closed-domains that require specific knowledge such as medicine and law in a zero-shot fashion without additional in-domain training? To this end, we devise a series of experiments to empirically explain the performance gap. Our findings suggest that: a) LLMs struggle with dataset demands of closed-domains such as retrieving long answer-spans; b) Certain LLMs, despite showing strong overall performance, display weaknesses in meeting basic requirements as discriminating between domain-specific senses of words which we link to pre-processing decisions; c) Scaling model parameters is not always effective for cross-domain generalization; and d) Closed-domain datasets are quantitatively much different than open-domain EQA datasets and current LLMs struggle to deal with them. Our findings point out important directions for improving existing LLMs.
• 摘要：在本文中，我们研究了领域漂移下使用大型语言模型 (LLM) 的抽取式问答 (EQA)，即 LLM 能否以零样本方式很好地推广到需要特定知识的封闭领域（例如医学和法律），而无需额外的领域内训练？为此，我们设计了一系列实验来实证解释性能差距。我们的研究结果表明：a) LLM 难以满足封闭领域的数据集要求，例如检索较长的答案跨度；b) 某些 LLM 虽然表现出强劲的整体性能，但在满足基本要求方面表现出弱点，例如区分我们与预处理决策相关的领域特定词义；c) 缩放模型参数对于跨域泛化并不总是有效的；d) 封闭域数据集在数量上与开放域 EQA 数据集有很大不同，当前的 LLM 难以处理它们。我们的研究结果为改进现有的 LLM 指出了重要方向。

Title: Leveraging Long-Context Large Language Models for Multi-Document Understanding and Summarization in Enterprise Applications

• Authors: Aditi Godbole, Jabin Geevarghese George, Smita Shandilya
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18454
• Pdf URL: https://arxiv.org/pdf/2409.18454
• Copy Paste: [[2409.18454]] Leveraging Long-Context Large Language Models for Multi-Document Understanding and Summarization in Enterprise Applications(https://arxiv.org/abs/2409.18454)
• Keywords: language model, llm
• Abstract: The rapid increase in unstructured data across various fields has made multi-document comprehension and summarization a critical task. Traditional approaches often fail to capture relevant context, maintain logical consistency, and extract essential information from lengthy documents. This paper explores the use of Long-context Large Language Models (LLMs) for multi-document summarization, demonstrating their exceptional capacity to grasp extensive connections, provide cohesive summaries, and adapt to various industry domains and integration with enterprise applications/systems. The paper discusses the workflow of multi-document summarization for effectively deploying long-context LLMs, supported by case studies in legal applications, enterprise functions such as HR, finance, and sourcing, as well as in the medical and news domains. These case studies show notable enhancements in both efficiency and accuracy. Technical obstacles, such as dataset diversity, model scalability, and ethical considerations like bias mitigation and factual accuracy, are carefully analyzed. Prospective research avenues are suggested to augment the functionalities and applications of long-context LLMs, establishing them as pivotal tools for transforming information processing across diverse sectors and enterprise applications.
• 摘要：各个领域非结构化数据的快速增长使得多文档理解和总结成为一项关键任务。传统方法通常无法捕捉相关上下文、保持逻辑一致性以及从冗长的文档中提取重要信息。本文探讨了长上下文大型语言模型 (LLM) 在多文档总结中的应用，展示了其在掌握广泛联系、提供连贯总结、适应各种行业领域和与企业应用程序/系统集成方面的卓越能力。本文讨论了有效部署长上下文 LLM 的多文档总结工作流程，并提供了法律应用、人力资源、财务和采购等企业职能以及医疗和新闻领域的案例研究。这些案例研究显示效率和准确性都有显著提高。本文仔细分析了技术障碍，例如数据集多样性、模型可扩展性以及减轻偏见和事实准确性等道德考虑因素。建议通过前瞻性的研究途径来增强长期 LLM 的功能和应用，使其成为转变跨不同行业和企业应用的信息处理的关键工具。

Title: Evaluation of OpenAI o1: Opportunities and Challenges of AGI

• Authors: Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, Chao Cao, Hanqi Jiang, Hanxu Chen, Yiwei Li, Junhao Chen, Huawen Hu, Yihen Liu, Huaqin Zhao, Shaochen Xu, Haixing Dai, Lin Zhao, Ruidong Zhang, Wei Zhao, Zhenyuan Yang, Jingyuan Chen, Peilong Wang, Wei Ruan, Hui Wang, Huan Zhao, Jing Zhang, Yiming Ren, Shihuan Qin, Tong Chen, Jiaxi Li, Arif Hassan Zidan, Afrar Jahin, Minheng Chen, Sichen Xia, Jason Holmes, Yan Zhuang, Jiaqi Wang, Bochen Xu, Weiran Xia, Jichao Yu, Kaibo Tang, Yaxuan Yang, Bolun Sun, Tao Yang, Guoyu Lu, Xianqiao Wang, Lilong Chai, He Li, Jin Lu, Lichao Sun, Xin Zhang, Bao Ge, Xintao Hu, Lian Zhang, Hua Zhou, Lu Zhang, Shu Zhang, Ninghao Liu, Bei Jiang, Linglong Kong, Zhen Xiang, Yudan Ren, Jun Liu, Xi Jiang, Yu Bao, Wei Zhang, Xiang Li, Gang Li, Wei Liu, Dinggang Shen, Andrea Sikora, Xiaoming Zhai, Dajiang Zhu, Tianming Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18486
• Pdf URL: https://arxiv.org/pdf/2409.18486
• Copy Paste: [[2409.18486]] Evaluation of OpenAI o1: Opportunities and Challenges of AGI(https://arxiv.org/abs/2409.18486)
• Keywords: language model
• Abstract: This comprehensive study evaluates the performance of OpenAI's o1-preview large language model across a diverse array of complex reasoning tasks, spanning multiple domains, including computer science, mathematics, natural sciences, medicine, linguistics, and social sciences. Through rigorous testing, o1-preview demonstrated remarkable capabilities, often achieving human-level or superior performance in areas ranging from coding challenges to scientific reasoning and from language processing to creative problem-solving. Key findings include: -83.3% success rate in solving complex competitive programming problems, surpassing many human experts. -Superior ability in generating coherent and accurate radiology reports, outperforming other evaluated models. -100% accuracy in high school-level mathematical reasoning tasks, providing detailed step-by-step solutions. -Advanced natural language inference capabilities across general and specialized domains like medicine. -Impressive performance in chip design tasks, outperforming specialized models in areas such as EDA script generation and bug analysis. -Remarkable proficiency in anthropology and geology, demonstrating deep understanding and reasoning in these specialized fields. -Strong capabilities in quantitative investing. O1 has comprehensive financial knowledge and statistical modeling skills. -Effective performance in social media analysis, including sentiment analysis and emotion recognition. The model excelled particularly in tasks requiring intricate reasoning and knowledge integration across various fields. While some limitations were observed, including occasional errors on simpler problems and challenges with certain highly specialized concepts, the overall results indicate significant progress towards artificial general intelligence.
• 摘要：这项综合研究评估了 OpenAI 的 o1-preview 大型语言模型在多种复杂推理任务中的表现，涵盖多个领域，包括计算机科学、数学、自然科学、医学、语言学和社会科学。通过严格的测试，o1-preview 展示了非凡的能力，在从编码挑战到科学推理、从语言处理到创造性解决问题等领域往往达到人类水平或更优异的表现。主要发现包括：-解决复杂的竞争性编程问题的成功率为 83.3%，超过了许多人类专家。-在生成连贯和准确的放射学报告方面具有卓越的能力，优于其他评估模型。-高中水平的数学推理任务准确率为 100%，提供详细的分步解决方案。-在医学等一般和专业领域具有先进的自然语言推理能力。-在芯片设计任务中表现出色，超越了 EDA 脚本生成和错误分析等领域的专门模型。 - 人类学和地质学的卓越能力，展现出对这些专业领域的深刻理解和推理能力。 - 强大的量化投资能力。O1 拥有全面的金融知识和统计建模技能。 - 在社交媒体分析方面表现优异，包括情绪分析和情绪识别。该模型在需要复杂推理和跨领域知识整合的任务中表现尤为出色。虽然观察到一些局限性，包括对简单问题偶尔出现错误以及某些高度专业化概念的挑战，但总体结果表明人工智能取得了重大进展。

Title: Do We Need Domain-Specific Embedding Models? An Empirical Investigation

• Authors: Yixuan Tang, Yi Yang
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.18511
• Pdf URL: https://arxiv.org/pdf/2409.18511
• Copy Paste: [[2409.18511]] Do We Need Domain-Specific Embedding Models? An Empirical Investigation(https://arxiv.org/abs/2409.18511)
• Keywords: language model, llm
• Abstract: Embedding models play a crucial role in representing and retrieving information across various NLP applications. Recent advancements in Large Language Models (LLMs) have further enhanced the performance of embedding models, which are trained on massive amounts of text covering almost every domain. These models are often benchmarked on general-purpose datasets like Massive Text Embedding Benchmark (MTEB), where they demonstrate superior performance. However, a critical question arises: Is the development of domain-specific embedding models necessary when general-purpose models are trained on vast corpora that already include specialized domain texts? In this paper, we empirically investigate this question, choosing the finance domain as an example. We introduce the Finance Massive Text Embedding Benchmark (FinMTEB), a counterpart to MTEB that consists of financial domain-specific text datasets. We evaluate the performance of seven state-of-the-art embedding models on FinMTEB and observe a significant performance drop compared to their performance on MTEB. To account for the possibility that this drop is driven by FinMTEB's higher complexity, we propose four measures to quantify dataset complexity and control for this factor in our analysis. Our analysis provides compelling evidence that state-of-the-art embedding models struggle to capture domain-specific linguistic and semantic patterns, even when trained on large general-purpose corpora. This study sheds light on the necessity of developing domain-specific embedding models in the LLM era, offering valuable insights for researchers and practitioners.
• 摘要：嵌入模型在各种 NLP 应用程序中表示和检索信息方面起着至关重要的作用。大型语言模型 (LLM) 的最新进展进一步增强了嵌入模型的性能，这些模型在几乎涵盖所有领域的大量文本上进行训练。这些模型通常在通用数据集（如海量文本嵌入基准 (MTEB)）上进行基准测试，在这些数据集上它们表现出卓越的性能。然而，一个关键问题出现了：当通用模型在已经包含专业领域文本的大量语料库上进行训练时，是否有必要开发特定领域的嵌入模型？在本文中，我们以金融领域为例，对这个问题进行了实证研究。我们介绍了金融海量文本嵌入基准 (FinMTEB)，它是 MTEB 的对应物，由金融领域特定的文本数据集组成。我们评估了七种最先进的嵌入模型在 FinMTEB 上的性能，并观察到与它们在 MTEB 上的性能相比，性能显著下降。为了解释这种下降可能是由 FinMTEB 的更高复杂性导致的，我们提出了四种措施来量化数据集的复杂性并控制我们的分析中的这一因素。我们的分析提供了令人信服的证据，表明最先进的嵌入模型很难捕捉特定领域的语言和语义模式，即使在大型通用语料库上进行训练也是如此。这项研究揭示了在 LLM 时代开发特定领域嵌入模型的必要性，为研究人员和从业者提供了宝贵的见解。

Title: A Survey on Complex Tasks for Goal-Directed Interactive Agents

• Authors: Mareike Hartmann, Alexander Koller
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18538
• Pdf URL: https://arxiv.org/pdf/2409.18538
• Copy Paste: [[2409.18538]] A Survey on Complex Tasks for Goal-Directed Interactive Agents(https://arxiv.org/abs/2409.18538)
• Keywords: language model, llm, agent
• Abstract: Goal-directed interactive agents, which autonomously complete tasks through interactions with their environment, can assist humans in various domains of their daily lives. Recent advances in large language models (LLMs) led to a surge of new, more and more challenging tasks to evaluate such agents. To properly contextualize performance across these tasks, it is imperative to understand the different challenges they pose to agents. To this end, this survey compiles relevant tasks and environments for evaluating goal-directed interactive agents, structuring them along dimensions relevant for understanding current obstacles. An up-to-date compilation of relevant resources can be found on our project website: this https URL.
• 摘要：目标导向型交互式代理通过与环境的交互自主完成任务，可以帮助人类处理日常生活的各个领域。大型语言模型 (LLM) 的最新进展导致评估此类代理的新任务激增，这些任务越来越具有挑战性。为了正确地将这些任务的性能置于上下文中，必须了解它们给代理带来的不同挑战。为此，本调查汇编了用于评估目标导向型交互式代理的相关任务和环境，并根据与理解当前障碍相关的维度对其进行了结构化。可以在我们的项目网站上找到相关资源的最新汇编：此 https URL。

Title: Research on Predicting Public Opinion Event Heat Levels Based on Large Language Models

• Authors: Yi Ren, Tianyi Zhang, Weibin Li, DuoMu Zhou, Chenhao Qin, FangCheng Dong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18548
• Pdf URL: https://arxiv.org/pdf/2409.18548
• Copy Paste: [[2409.18548]] Research on Predicting Public Opinion Event Heat Levels Based on Large Language Models(https://arxiv.org/abs/2409.18548)
• Keywords: language model, gpt
• Abstract: In recent years, with the rapid development of large language models, serval models such as GPT-4o have demonstrated extraordinary capabilities, surpassing human performance in various language tasks. As a result, many researchers have begun exploring their potential applications in the field of public opinion analysis. This study proposes a novel large-language-models-based method for public opinion event heat level prediction. First, we preprocessed and classified 62,836 Chinese hot event data collected between July 2022 and December 2023. Then, based on each event's online dissemination heat index, we used the MiniBatchKMeans algorithm to automatically cluster the events and categorize them into four heat levels (ranging from low heat to very high heat). Next, we randomly selected 250 events from each heat level, totalling 1,000 events, to build the evaluation dataset. During the evaluation process, we employed various large language models to assess their accuracy in predicting event heat levels in two scenarios: without reference cases and with similar case references. The results showed that GPT-4o and DeepseekV2 performed the best in the latter case, achieving prediction accuracies of 41.4% and 41.5%, respectively. Although the overall prediction accuracy remains relatively low, it is worth noting that for low-heat (Level 1) events, the prediction accuracies of these two models reached 73.6% and 70.4%, respectively. Additionally, the prediction accuracy showed a downward trend from Level 1 to Level 4, which correlates with the uneven distribution of data across the heat levels in the actual dataset. This suggests that with the more robust dataset, public opinion event heat level prediction based on large language models will have significant research potential for the future.
• 摘要：近年来，随着大型语言模型的快速发展，GPT-4o 等多种模型展现出非凡能力，在多项语言任务上超越人类表现，不少研究者开始探索其在舆情分析领域的潜在应用。本研究提出一种基于大型语言模型的舆情事件热度预测新方法。首先，我们对 2022 年 7 月至 2023 年 12 月期间收集的 62,836 条中文热点事件数据进行预处理和分类；然后，根据事件的线上传播热度指数，使用 MiniBatchKMeans 算法对事件进行自动聚类，将其分为四个热度等级（从低热度到非常高热度）；接下来，从每个热度等级中随机选取 250 个事件，共计 1,000 个事件，构建评估数据集。在评估过程中，我们采用各种大型语言模型在无参考案例和有类似案例参考两种场景下评估其对事件热度预测的准确率。结果显示，在后一种情况下，GPT-4o 和 DeepseekV2 表现最佳，分别达到了 41.4% 和 41.5% 的预测准确率。虽然整体预测准确率仍然较低，但值得注意的是，对于低热度（Level 1）事件，这两个模型的预测准确率分别达到了 73.6% 和 70.4%。此外，预测准确率从 Level 1 到 Level 4 呈现下降趋势，这与实际数据集中数据在热度等级间分布不均有关。这意味着在更加稳健的数据集下，基于大型语言模型的舆情事件热度预测将在未来具有巨大的研究潜力。

Title: Hit the Sweet Spot! Span-Level Ensemble for Large Language Models

• Authors: Yangyifan Xu, Jianghao Chen, Junhong Wu, Jiajun Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18583
• Pdf URL: https://arxiv.org/pdf/2409.18583
• Copy Paste: [[2409.18583]] Hit the Sweet Spot! Span-Level Ensemble for Large Language Models(https://arxiv.org/abs/2409.18583)
• Keywords: language model, llm
• Abstract: Ensembling various LLMs to unlock their complementary potential and leverage their individual strengths is highly valuable. Previous studies typically focus on two main paradigms: sample-level and token-level ensembles. Sample-level ensemble methods either select or blend fully generated outputs, which hinders dynamic correction and enhancement of outputs during the generation process. On the other hand, token-level ensemble methods enable real-time correction through fine-grained ensemble at each generation step. However, the information carried by an individual token is quite limited, leading to suboptimal decisions at each step. To address these issues, we propose SweetSpan, a span-level ensemble method that effectively balances the need for real-time adjustments and the information required for accurate ensemble decisions. Our approach involves two key steps: First, we have each candidate model independently generate candidate spans based on the shared prefix. Second, we calculate perplexity scores to facilitate mutual evaluation among the candidate models and achieve robust span selection by filtering out unfaithful scores. To comprehensively evaluate ensemble methods, we propose a new challenging setting (ensemble models with significant performance gaps) in addition to the standard setting (ensemble the best-performing models) to assess the performance of model ensembles in more realistic scenarios. Experimental results in both standard and challenging settings across various language generation tasks demonstrate the effectiveness, robustness, and versatility of our approach compared with previous ensemble methods.
• 摘要：将各种 LLM 组合在一起以释放其互补潜力并发挥各自的优势是非常有价值的。以前的研究通常关注两个主要范例：样本级和 token 级组合。样本级组合方法要么选择要么混合完全生成的输出，这阻碍了生成过程中输出的动态校正和增强。另一方面，token 级组合方法通过在每个生成步骤中进行细粒度组合实现实时校正。然而，单个 token 所携带的信息非常有限，导致每个步骤的决策都不理想。为了解决这些问题，我们提出了 SweetSpan，这是一种跨度级组合方法，可以有效地平衡实时调整的需求和准确组合决策所需的信息。我们的方法包括两个关键步骤：首先，我们让每个候选模型根据共享前缀独立生成候选跨度。其次，我们计算困惑度分数以促进候选模型之间的相互评估，并通过过滤掉不真实的分数来实现稳健的跨度选择。为了全面评估集成方法，我们在标准设置（集成表现最佳的模型）之外提出了一种新的挑战性设置（具有显著性能差距的集成模型），以评估模型集成在更现实场景中的表现。在各种语言生成任务的标准和挑战性设置中的实验结果证明了与以前的集成方法相比，我们的方法的有效性、稳健性和多功能性。

Title: Do LLMs suffer from Multi-Party Hangover? A Diagnostic Approach to Addressee Recognition and Response Selection in Conversations

• Authors: Nicolò Penzo, Maryam Sajedinia, Bruno Lepri, Sara Tonelli, Marco Guerini
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18602
• Pdf URL: https://arxiv.org/pdf/2409.18602
• Copy Paste: [[2409.18602]] Do LLMs suffer from Multi-Party Hangover? A Diagnostic Approach to Addressee Recognition and Response Selection in Conversations(https://arxiv.org/abs/2409.18602)
• Keywords: llm, prompt
• Abstract: Assessing the performance of systems to classify Multi-Party Conversations (MPC) is challenging due to the interconnection between linguistic and structural characteristics of conversations. Conventional evaluation methods often overlook variances in model behavior across different levels of structural complexity on interaction graphs. In this work, we propose a methodological pipeline to investigate model performance across specific structural attributes of conversations. As a proof of concept we focus on Response Selection and Addressee Recognition tasks, to diagnose model weaknesses. To this end, we extract representative diagnostic subdatasets with a fixed number of users and a good structural variety from a large and open corpus of online MPCs. We further frame our work in terms of data minimization, avoiding the use of original usernames to preserve privacy, and propose alternatives to using original text messages. Results show that response selection relies more on the textual content of conversations, while addressee recognition requires capturing their structural dimension. Using an LLM in a zero-shot setting, we further highlight how sensitivity to prompt variations is task-dependent.
• 摘要：由于对话的语言特征和结构特征相互关联，评估系统对多方对话 (MPC) 进行分类的性能具有挑战性。传统的评估方法通常会忽略交互图上不同结构复杂度级别的模型行为差异。在这项工作中，我们提出了一种方法论流程，用于研究对话特定结构属性的模型性能。作为概念验证，我们专注于响应选择和收件人识别任务，以诊断模型的弱点。为此，我们从大量开放的在线 MPC 语料库中提取了具有固定用户数量和良好结构多样性的代表性诊断子数据集。我们进一步从数据最小化的角度来构建我们的工作，避免使用原始用户名来保护隐私，并提出了使用原始文本消息的替代方案。结果表明，响应选择更多地依赖于对话的文本内容，而收件人识别需要捕获其结构维度。在零样本设置中使用 LLM，我们进一步强调了对提示变化的敏感性如何依赖于任务。

Title: Model-based Preference Optimization in Abstractive Summarization without Human Feedback

• Authors: Jaepill Choi, Kyubyung Chae, Jiwoo Song, Yohan Jo, Taesup Kim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18618
• Pdf URL: https://arxiv.org/pdf/2409.18618
• Copy Paste: [[2409.18618]] Model-based Preference Optimization in Abstractive Summarization without Human Feedback(https://arxiv.org/abs/2409.18618)
• Keywords: language model, llm
• Abstract: In abstractive summarization, the challenge of producing concise and accurate summaries arises from the vast amount of information contained in the source document. Consequently, although Large Language Models (LLMs) can generate fluent text, they often introduce inaccuracies by hallucinating content not found in the original source. While supervised fine-tuning methods that maximize likelihood contribute to this issue, they do not consistently enhance the faithfulness of the summaries. Preference-based optimization methods, such as Direct Preference Optimization (DPO), can further refine the model to align with human preferences. However, these methods still heavily depend on costly human feedback. In this work, we introduce a novel and straightforward approach called Model-based Preference Optimization (MPO) to fine-tune LLMs for improved summarization abilities without any human feedback. By leveraging the model's inherent summarization capabilities, we create a preference dataset that is fully generated by the model using different decoding strategies. Our experiments on standard summarization datasets and various metrics demonstrate that our proposed MPO significantly enhances the quality of generated summaries without relying on human feedback.
• 摘要：在抽象摘要中，生成简洁准确的摘要的挑战来自于源文档中包含的大量信息。因此，尽管大型语言模型 (LLM) 可以生成流畅的文本，但它们通常会通过幻化原始源中不存在的内容而引入不准确性。虽然最大化可能性的监督微调方法有助于解决此问题，但它们并不能持续提高摘要的忠实度。基于偏好的优化方法，例如直接偏好优化 (DPO)，可以进一步完善模型以符合人类偏好。然而，这些方法仍然严重依赖昂贵的人工反馈。在这项工作中，我们引入了一种新颖而直接的方法，称为基于模型的偏好优化 (MPO)，以微调 LLM 以提高摘要能力，而无需任何人工反馈。通过利用模型固有的摘要功能，我们创建了一个完全由模型使用不同解码策略生成的偏好数据集。我们在标准摘要数据集和各种指标上进行的实验表明，我们提出的 MPO 显著提高了生成的摘要的质量，而无需依赖人工反馈。

Title: Rehearsing Answers to Probable Questions with Perspective-Taking

• Authors: Yung-Yu Shih, Ziwei Xu, Hiroya Takamura, Yun-Nung Chen, Chung-Chi Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18678
• Pdf URL: https://arxiv.org/pdf/2409.18678
• Copy Paste: [[2409.18678]] Rehearsing Answers to Probable Questions with Perspective-Taking(https://arxiv.org/abs/2409.18678)
• Keywords: language model, llm
• Abstract: Question answering (QA) has been a long-standing focus in the NLP field, predominantly addressing reading comprehension and common sense QA. However, scenarios involving the preparation of answers to probable questions during professional oral presentations remain underexplored. In this paper, we pioneer the examination of this crucial yet overlooked topic by utilizing real-world QA conversation transcripts between company managers and professional analysts. We explore the proposed task using three causal knowledge graphs (KGs) and three large language models (LLMs). This work provides foundational insights into the application of LLMs in professional QA scenarios, highlighting the importance of causal KGs and perspective-taking in generating effective responses.
• 摘要：问答 (QA) 一直是 NLP 领域的研究重点，主要解决阅读理解和常识性问答。然而，在专业口头陈述期间准备可能出现的问题的答案的场景仍未得到充分探索。在本文中，我们利用公司经理和专业分析师之间真实的问答对话记录，率先研究了这一至关重要但被忽视的主题。我们使用三个因果知识图谱 (KG) 和三个大型语言模型 (LLM) 探索了所提出的任务。这项工作为 LLM 在专业问答场景中的应用提供了基础见解，强调了因果知识图谱和换位思考在生成有效响应方面的重要性。

Title: "Why" Has the Least Side Effect on Model Editing

• Authors: Tsung-Hsuan Pan, Chung-Chi Chen, Hen-Hsen Huang, Hsin-Hsi Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18679
• Pdf URL: https://arxiv.org/pdf/2409.18679
• Copy Paste: [[2409.18679]] "Why" Has the Least Side Effect on Model Editing(https://arxiv.org/abs/2409.18679)
• Keywords: language model, llm
• Abstract: Training large language models (LLMs) from scratch is an expensive endeavor, particularly as world knowledge continually evolves. To maintain relevance and accuracy of LLMs, model editing has emerged as a pivotal research area. While these methods hold promise, they can also produce unintended side effects. Their underlying factors and causes remain largely unexplored. This paper delves into a critical factor-question type-by categorizing model editing questions. Our findings reveal that the extent of performance degradation varies significantly across different question types, providing new insights for experimental design in knowledge editing. Furthermore, we investigate whether insights from smaller models can be extrapolated to larger models. Our results indicate discrepancies in findings between models of different sizes, suggesting that insights from smaller models may not necessarily apply to larger models. Additionally, we examine the impact of batch size on side effects, discovering that increasing the batch size can mitigate performance drops.
• 摘要：从头开始训练大型语言模型 (LLM) 是一项昂贵的工作，尤其是在世界知识不断发展的情况下。为了保持 LLM 的相关性和准确性，模型编辑已成为一个关键的研究领域。虽然这些方法很有前景，但它们也可能产生意想不到的副作用。其潜在因素和原因在很大程度上仍未被探索。本文通过对模型编辑问题进行分类，深入研究了一个关键因素——问题类型。我们的研究结果表明，不同类型的问题性能下降程度差异很大，为知识编辑的实验设计提供了新的见解。此外，我们研究了较小模型的见解是否可以推广到较大的模型。我们的结果表明，不同大小的模型之间的研究结果存在差异，这表明较小模型的见解不一定适用于较大的模型。此外，我们研究了批量大小对副作用的影响，发现增加批量大小可以减轻性能下降。

Title: Read Over the Lines: Attacking LLMs and Toxicity Detection Systems with ASCII Art to Mask Profanity

• Authors: Sergey Berezin, Reza Farahbakhsh, Noel Crespi
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.18708
• Pdf URL: https://arxiv.org/pdf/2409.18708
• Copy Paste: [[2409.18708]] Read Over the Lines: Attacking LLMs and Toxicity Detection Systems with ASCII Art to Mask Profanity(https://arxiv.org/abs/2409.18708)
• Keywords: language model, llm
• Abstract: We introduce a novel family of adversarial attacks that exploit the inability of language models to interpret ASCII art. To evaluate these attacks, we propose the ToxASCII benchmark and develop two custom ASCII art fonts: one leveraging special tokens and another using text-filled letter shapes. Our attacks achieve a perfect 1.0 Attack Success Rate across ten models, including OpenAI's o1-preview and LLaMA 3.1. Warning: this paper contains examples of toxic language used for research purposes.
• 摘要：我们引入了一系列新型对抗性攻击，利用语言模型无法解释 ASCII 艺术这一特点。为了评估这些攻击，我们提出了 ToxASCII 基准并开发了两种自定义 ASCII 艺术字体：一种利用特殊标记，另一种使用文本填充的字母形状。我们的攻击在十个模型中实现了完美的 1.0 攻击成功率，包括 OpenAI 的 o1-preview 和 LLaMA 3.1。警告：本文包含用于研究目的的恶意语言示例。

Title: A Survey on the Honesty of Large Language Models

• Authors: Siheng Li, Cheng Yang, Taiqiang Wu, Chufan Shi, Yuji Zhang, Xinyu Zhu, Zesen Cheng, Deng Cai, Mo Yu, Lemao Liu, Jie Zhou, Yujiu Yang, Ngai Wong, Xixin Wu, Wai Lam
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18786
• Pdf URL: https://arxiv.org/pdf/2409.18786
• Copy Paste: [[2409.18786]] A Survey on the Honesty of Large Language Models(https://arxiv.org/abs/2409.18786)
• Keywords: language model, llm
• Abstract: Honesty is a fundamental principle for aligning large language models (LLMs) with human values, requiring these models to recognize what they know and don't know and be able to faithfully express their knowledge. Despite promising, current LLMs still exhibit significant dishonest behaviors, such as confidently presenting wrong answers or failing to express what they know. In addition, research on the honesty of LLMs also faces challenges, including varying definitions of honesty, difficulties in distinguishing between known and unknown knowledge, and a lack of comprehensive understanding of related research. To address these issues, we provide a survey on the honesty of LLMs, covering its clarification, evaluation approaches, and strategies for improvement. Moreover, we offer insights for future research, aiming to inspire further exploration in this important area.
• 摘要：诚实是大型语言模型 (LLM) 与人类价值观相一致的一项基本原则，要求这些模型能够识别它们知道什么和不知道什么，并能够忠实地表达它们的知识。尽管前景广阔，但目前的 LLM 仍然表现出明显的不诚实行为，例如自信地给出错误答案或无法表达它们所知道的内容。此外，对 LLM 诚实性的研究也面临着挑战，包括对诚实的不同定义、难以区分已知和未知知识以及缺乏对相关研究的全面了解。针对这些问题，我们对 LLM 的诚实性进行了调查，涵盖了其澄清、评估方法和改进策略。此外，我们还为未来的研究提供了见解，旨在激发对这一重要领域的进一步探索。

Title: LLMs4Synthesis: Leveraging Large Language Models for Scientific Synthesis

• Authors: Hamed Babaei Giglou, Jennifer D'Souza, Sören Auer
• Subjects: cs.CL, cs.AI, cs.DL
• Abstract URL: https://arxiv.org/abs/2409.18812
• Pdf URL: https://arxiv.org/pdf/2409.18812
• Copy Paste: [[2409.18812]] LLMs4Synthesis: Leveraging Large Language Models for Scientific Synthesis(https://arxiv.org/abs/2409.18812)
• Keywords: language model, llm
• Abstract: In response to the growing complexity and volume of scientific literature, this paper introduces the LLMs4Synthesis framework, designed to enhance the capabilities of Large Language Models (LLMs) in generating high-quality scientific syntheses. This framework addresses the need for rapid, coherent, and contextually rich integration of scientific insights, leveraging both open-source and proprietary LLMs. It also examines the effectiveness of LLMs in evaluating the integrity and reliability of these syntheses, alleviating inadequacies in current quantitative metrics. Our study contributes to this field by developing a novel methodology for processing scientific papers, defining new synthesis types, and establishing nine detailed quality criteria for evaluating syntheses. The integration of LLMs with reinforcement learning and AI feedback is proposed to optimize synthesis quality, ensuring alignment with established criteria. The LLMs4Synthesis framework and its components are made available, promising to enhance both the generation and evaluation processes in scientific research synthesis.
• 摘要：为应对日益复杂的科学文献和日益庞大的科学文献数量，本文介绍了 LLMs4Synthesis 框架，旨在增强大型语言模型 (LLM) 生成高质量科学综合的能力。该框架利用开源和专有 LLM，满足快速、连贯且语境丰富的科学见解整合需求。它还研究了 LLM 在评估这些综合的完整性和可靠性方面的有效性，从而缓解了当前定量指标的不足。我们的研究通过开发一种处理科学论文的新方法、定义新的综合类型以及建立九个评估综合的详细质量标准，为该领域做出了贡献。我们提出了将 LLM 与强化学习和 AI 反馈相结合以优化综合质量，确保与既定标准保持一致。LLMs4Synthesis 框架及其组件现已推出，有望增强科学研究综合的生成和评估过程。

Title: Suicide Phenotyping from Clinical Notes in Safety-Net Psychiatric Hospital Using Multi-Label Classification with Pre-Trained Language Models

• Authors: Zehan Li, Yan Hu, Scott Lane, Salih Selek, Lokesh Shahani, Rodrigo Machado-Vieira, Jair Soares, Hua Xu, Hongfang Liu, Ming Huang
• Subjects: cs.CL, cs.AI, cs.CY, cs.IR
• Abstract URL: https://arxiv.org/abs/2409.18878
• Pdf URL: https://arxiv.org/pdf/2409.18878
• Copy Paste: [[2409.18878]] Suicide Phenotyping from Clinical Notes in Safety-Net Psychiatric Hospital Using Multi-Label Classification with Pre-Trained Language Models(https://arxiv.org/abs/2409.18878)
• Keywords: language model
• Abstract: Accurate identification and categorization of suicidal events can yield better suicide precautions, reducing operational burden, and improving care quality in high-acuity psychiatric settings. Pre-trained language models offer promise for identifying suicidality from unstructured clinical narratives. We evaluated the performance of four BERT-based models using two fine-tuning strategies (multiple single-label and single multi-label) for detecting coexisting suicidal events from 500 annotated psychiatric evaluation notes. The notes were labeled for suicidal ideation (SI), suicide attempts (SA), exposure to suicide (ES), and non-suicidal self-injury (NSSI). RoBERTa outperformed other models using binary relevance (acc=0.86, F1=0.78). MentalBERT (F1=0.74) also exceeded BioClinicalBERT (F1=0.72). RoBERTa fine-tuned with a single multi-label classifier further improved performance (acc=0.88, F1=0.81), highlighting that models pre-trained on domain-relevant data and the single multi-label classification strategy enhance efficiency and performance. Keywords: EHR-based Phynotyping; Natural Language Processing; Secondary Use of EHR Data; Suicide Classification; BERT-based Model; Psychiatry; Mental Health
• 摘要：准确识别和分类自杀事件可以更好地预防自杀，减轻运营负担，并提高高危精神病环境中的护理质量。预先训练的语言模型有望从非结构化的临床叙述中识别自杀倾向。我们使用两种微调策略（多个单标签和单个多标签）评估了四个基于 BERT 的模型的性能，以从 500 份带注释的精神病评估笔记中检测共存的自杀事件。这些笔记被标记为自杀意念 (SI)、自杀未遂 (SA)、自杀暴露 (ES) 和非自杀性自残 (NSSI)。RoBERTa 的表现优于使用二元相关性的其他模型（acc=0.86，F1=0.78）。MentalBERT（F1=0.74）也超过了 BioClinicalBERT（F1=0.72）。使用单个多标签分类器进行微调的 RoBERTa 进一步提高了性能（acc=0.88，F1=0.81），突出了在领域相关数据上进行预训练的模型和单个多标签分类策略提高了效率和性能。关键词：基于 EHR 的表型分析；自然语言处理；EHR 数据的二次利用；自杀分类；基于 BERT 的模型；精神病学；心理健康

Title: IDGen: Item Discrimination Induced Prompt Generation for LLM Evaluation

• Authors: Fan Lin, Shuyi Xie, Yong Dai, Wenlin Yao, Tianjiao Lang, Zishan Xu, Zhichao Hu, Xiao Xiao, Yuhong Liu, Yu Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18892
• Pdf URL: https://arxiv.org/pdf/2409.18892
• Copy Paste: [[2409.18892]] IDGen: Item Discrimination Induced Prompt Generation for LLM Evaluation(https://arxiv.org/abs/2409.18892)
• Keywords: language model, llm, prompt
• Abstract: As Large Language Models (LLMs) grow increasingly adept at managing complex tasks, the evaluation set must keep pace with these advancements to ensure it remains sufficiently discriminative. Item Discrimination (ID) theory, which is widely used in educational assessment, measures the ability of individual test items to differentiate between high and low performers. Inspired by this theory, we propose an ID-induced prompt synthesis framework for evaluating LLMs to ensure the evaluation set can continually update and refine according to model abilities. Our data synthesis framework prioritizes both breadth and specificity. It can generate prompts that comprehensively evaluate the capabilities of LLMs while revealing meaningful performance differences between models, allowing for effective discrimination of their relative strengths and weaknesses across various tasks and domains. To produce high-quality data, we incorporate a self-correct mechanism into our generalization framework, and develop two models to predict prompt discrimination and difficulty score to facilitate our data synthesis framework, contributing valuable tools to evaluation data synthesis research. We apply our generated data to evaluate five SOTA models. Our data achieves an average score of 51.92, accompanied by a variance of 10.06. By contrast, previous works (i.e., SELF-INSTRUCT and WizardLM) obtain an average score exceeding 67, with a variance below 3.2. The results demonstrate that the data generated by our framework is more challenging and discriminative compared to previous works. We will release a dataset of over 3,000 carefully crafted prompts to facilitate evaluation research of LLMs.
• 摘要：随着大型语言模型 (LLM) 越来越擅长处理复杂任务，评估集必须跟上这些进步的步伐，以确保其保持足够的辨别力。项目辨别 (ID) 理论广泛应用于教育评估，它衡量单个测试项目区分高绩效和低绩效的能力。受此理论的启发，我们提出了一个 ID 诱导的快速合成框架来评估 LLM，以确保评估集能够根据模型能力不断更新和改进。我们的数据合成框架优先考虑广度和特异性。它可以生成全面评估 LLM 能力的提示，同时揭示模型之间有意义的性能差异，从而有效地区分它们在不同任务和领域的相对优势和劣势。为了生成高质量的数据，我们在泛化框架中加入了自我纠正机制，并开发了两个模型来预测快速辨别和难度分数，以促进我们的数据合成框架，为评估数据合成研究贡献宝贵的工具。我们应用生成的数据来评估五个 SOTA 模型。我们的数据平均得分为 51.92，方差为 10.06。相比之下，以前的研究（即 SELF-INSTRUCT 和 WizardLM）平均得分超过 67，方差低于 3.2。结果表明，与以前的研究相比，我们的框架生成的数据更具挑战性和区分性。我们将发布一个包含 3,000 多个精心设计的提示的数据集，以促进 LLM 的评估研究。

Title: Soft Measures for Extracting Causal Collective Intelligence

• Authors: Maryam Berijanian, Spencer Dork, Kuldeep Singh, Michael Riley Millikan, Ashlin Riggs, Aadarsh Swaminathan, Sarah L. Gibbs, Scott E. Friedman, Nathan Brugnone
• Subjects: cs.CL, cs.AI, cs.CY, cs.SI
• Abstract URL: https://arxiv.org/abs/2409.18911
• Pdf URL: https://arxiv.org/pdf/2409.18911
• Copy Paste: [[2409.18911]] Soft Measures for Extracting Causal Collective Intelligence(https://arxiv.org/abs/2409.18911)
• Keywords: language model, llm
• Abstract: Understanding and modeling collective intelligence is essential for addressing complex social systems. Directed graphs called fuzzy cognitive maps (FCMs) offer a powerful tool for encoding causal mental models, but extracting high-integrity FCMs from text is challenging. This study presents an approach using large language models (LLMs) to automate FCM extraction. We introduce novel graph-based similarity measures and evaluate them by correlating their outputs with human judgments through the Elo rating system. Results show positive correlations with human evaluations, but even the best-performing measure exhibits limitations in capturing FCM nuances. Fine-tuning LLMs improves performance, but existing measures still fall short. This study highlights the need for soft similarity measures tailored to FCM extraction, advancing collective intelligence modeling with NLP.
• 摘要：理解和建模集体智慧对于解决复杂的社会系统至关重要。有向图称为模糊认知图 (FCM)，它为编码因果心理模型提供了强大的工具，但从文本中提取高完整性的 FCM 具有挑战性。本研究提出了一种使用大型语言模型 (LLM) 自动提取 FCM 的方法。我们引入了新的基于图的相似性度量，并通过 Elo 评分系统将其输出与人类判断相关联来评估它们。结果显示与人类评估呈正相关，但即使是表现最佳的度量在捕捉 FCM 细微差别方面也表现出局限性。微调 LLM 可以提高性能，但现有度量仍然存在不足。本研究强调需要针对 FCM 提取量身定制的软相似性度量，从而通过 NLP 推进集体智慧建模。

Title: AIPatient: Simulating Patients with EHRs and LLM Powered Agentic Workflow

• Authors: Huizi Yu, Jiayan Zhou, Lingyao Li, Shan Chen, Jack Gallifant, Anye Shi, Xiang Li, Wenyue Hua, Mingyu Jin, Guang Chen, Yang Zhou, Zhao Li, Trisha Gupte, Ming-Li Chen, Zahra Azizi, Yongfeng Zhang, Themistocles L. Assimes, Xin Ma, Danielle S. Bitterman, Lin Lu, Lizhou Fan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18924
• Pdf URL: https://arxiv.org/pdf/2409.18924
• Copy Paste: [[2409.18924]] AIPatient: Simulating Patients with EHRs and LLM Powered Agentic Workflow(https://arxiv.org/abs/2409.18924)
• Keywords: language model, llm, retrieval-augmented generation, agent
• Abstract: Simulated patient systems play a crucial role in modern medical education and research, providing safe, integrative learning environments and enabling clinical decision-making simulations. Large Language Models (LLM) could advance simulated patient systems by replicating medical conditions and patient-doctor interactions with high fidelity and low cost. However, ensuring the effectiveness and trustworthiness of these systems remains a challenge, as they require a large, diverse, and precise patient knowledgebase, along with a robust and stable knowledge diffusion to users. Here, we developed AIPatient, an advanced simulated patient system with AIPatient Knowledge Graph (AIPatient KG) as the input and the Reasoning Retrieval-Augmented Generation (Reasoning RAG) agentic workflow as the generation backbone. AIPatient KG samples data from Electronic Health Records (EHRs) in the Medical Information Mart for Intensive Care (MIMIC)-III database, producing a clinically diverse and relevant cohort of 1,495 patients with high knowledgebase validity (F1 0.89). Reasoning RAG leverages six LLM powered agents spanning tasks including retrieval, KG query generation, abstraction, checker, rewrite, and summarization. This agentic framework reaches an overall accuracy of 94.15% in EHR-based medical Question Answering (QA), outperforming benchmarks that use either no agent or only partial agent integration. Our system also presents high readability (median Flesch Reading Ease 77.23; median Flesch Kincaid Grade 5.6), robustness (ANOVA F-value 0.6126, p<0.1), and stability (ANOVA F-value 0.782, p<0.1). The promising performance of the AIPatient system highlights its potential to support a wide range of applications, including medical education, model evaluation, and system integration.
• 摘要：模拟病人系统在现代医学教育和研究中发挥着至关重要的作用，它提供了安全、综合的学习环境，并支持临床决策模拟。大型语言模型 (LLM) 可以通过高保真度和低成本复制医疗状况和医患互动来推进模拟病人系统。然而，确保这些系统的有效性和可信度仍然是一个挑战，因为它们需要庞大、多样化和精确的患者知识库，以及向用户提供强大而稳定的知识传播。在这里，我们开发了 AIPatient，这是一种先进的模拟病人系统，以 AIPatient 知识图谱 (AIPatient KG) 作为输入，以推理检索增强生成 (Reasoning RAG) 代理工作流作为生成主干。AIPatient KG 从重症监护医疗信息集市 (MIMIC)-III 数据库中的电子健康记录 (EHR) 中抽样数据，生成一个具有高知识库有效性 (F1 0.89) 的 1,495 名临床多样化且相关的患者队列。 Reasoning RAG 利用六个 LLM 驱动的代理，涵盖检索、KG 查询生成、抽象、检查器、重写和摘要等任务。该代理框架在基于 EHR 的医疗问答 (QA) 中达到了 94.15% 的总体准确率，优于不使用代理或仅使用部分代理集成的基准。我们的系统还具有很高的可读性（Flesch 阅读难易度中位数 77.23；Flesch Kincaid 等级中位数 5.6）、稳健性（ANOVA F 值 0.6126，p<0.1）和稳定性（ANOVA F 值 0.782，p<0.1）。AIPatient 系统的出色性能凸显了其支持广泛应用的潜力，包括医学教育、模型评估和系统集成。

Title: Ruler: A Model-Agnostic Method to Control Generated Length for Large Language Models

• Authors: Jiaming Li, Lei Zhang, Yunshui Li, Ziqiang Liu, yuelin bai, Run Luo, Longze Chen, Min Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.18943
• Pdf URL: https://arxiv.org/pdf/2409.18943
• Copy Paste: [[2409.18943]] Ruler: A Model-Agnostic Method to Control Generated Length for Large Language Models(https://arxiv.org/abs/2409.18943)
• Keywords: language model, llm, agent
• Abstract: The instruction-following ability of large language models enables humans to interact with AI agents in a natural way. However, when required to generate responses of a specific length, large language models often struggle to meet users' needs due to their inherent difficulty in accurately perceiving numerical constraints. To explore the ability of large language models to control the length of generated responses, we propose the Target Length Generation Task (TLG) and design two metrics, Precise Match (PM) and Flexible Match (FM) to evaluate the model's performance in adhering to specified response lengths. Furthermore, we introduce a novel, model-agnostic approach called Ruler, which employs Meta Length Tokens (MLTs) to enhance the instruction-following ability of large language models under length-constrained instructions. Specifically, Ruler equips LLMs with the ability to generate responses of a specified length based on length constraints within the instructions. Moreover, Ruler can automatically generate appropriate MLT when length constraints are not explicitly provided, demonstrating excellent versatility and generalization. Comprehensive experiments show the effectiveness of Ruler across different LLMs on Target Length Generation Task, e.g., at All Level 27.97 average gain on PM, 29.57 average gain on FM. In addition, we conduct extensive ablation experiments to further substantiate the efficacy and generalization of Ruler. Our code and data is available at this https URL.
• 摘要：大型语言模型的指令跟随能力使人类能够以自然的方式与 AI 代理进行交互。然而，当需要生成特定长度的响应时，大型语言模型往往难以满足用户的需求，因为它们本身难以准确感知数值约束。为了探索大型语言模型控制生成响应长度的能力，我们提出了目标长度生成任务 (TLG)，并设计了两个指标，精确匹配 (PM) 和灵活匹配 (FM)，以评估模型在遵守指定响应长度方面的表现。此外，我们引入了一种新颖的、与模型无关的方法，称为 Ruler，它采用元长度标记 (MLT) 来增强大型语言模型在长度受限指令下的指令跟随能力。具体来说，Ruler 使 LLM 能够根据指令中的长度约束生成指定长度的响应。此外，当未明确提供长度约束时，Ruler 可以自动生成适当的 MLT，表现出出色的多功能性和泛化能力。综合实验表明，Ruler 在目标长度生成任务中对不同 LLM 均有效，例如，在所有级别上，PM 平均增益为 27.97，FM 平均增益为 29.57。此外，我们还进行了广泛的消融实验，以进一步证实 Ruler 的有效性和泛化能力。我们的代码和数据可在此 https URL 上找到。

Title: LML: Language Model Learning a Dataset for Data-Augmented Prediction

• Authors: Praneeth Vadlapati
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18957
• Pdf URL: https://arxiv.org/pdf/2409.18957
• Copy Paste: [[2409.18957]] LML: Language Model Learning a Dataset for Data-Augmented Prediction(https://arxiv.org/abs/2409.18957)
• Keywords: language model, llm, prompt
• Abstract: This paper introduces a new approach to using Large Language Models (LLMs) for classification tasks, which are typically handled using Machine Learning (ML) models. Unlike ML models that rely heavily on data cleaning and feature engineering, this method streamlines the process using LLMs. This paper proposes a new concept called "Language Model Learning (LML)" powered by a new method called "Data-Augmented Prediction (DAP)". The classification is performed by LLMs using a method similar to humans manually exploring and understanding the data and deciding classifications using data as a reference. Training data is summarized and evaluated to determine the features that lead to the classification of each label the most. In the process of DAP, the system uses the data summary to automatically create a query, which is used to retrieve relevant rows from the dataset. A classification is generated by the LLM using data summary and relevant rows, ensuring satisfactory accuracy even with complex data. Usage of data summary and similar data in DAP ensures context-aware decision-making. The proposed method uses the words "Act as an Explainable Machine Learning Model" in the prompt to enhance the interpretability of the predictions by allowing users to review the logic behind each prediction. In some test cases, the system scored an accuracy above 90%, proving the effectiveness of the system and its potential to outperform conventional ML models in various scenarios. The code is available at this https URL
• 摘要：本文介绍了一种使用大型语言模型 (LLM) 进行分类任务的新方法，这些任务通常使用机器学习 (ML) 模型来处理。与严重依赖数据清理和特征工程的 ML 模型不同，此方法使用 LLM 简化了流程。本文提出了一种名为“语言模型学习 (LML)”的新概念，由一种名为“数据增强预测 (DAP)”的新方法提供支持。LLM 使用类似于人类手动探索和理解数据并使用数据作为参考来决定分类的方法执行分类。对训练数据进行汇总和评估，以确定最能导致每个标签分类的特征。在 DAP 过程中，系统使用数据摘要自动创建查询，该查询用于从数据集中检索相关行。LLM 使用数据摘要和相关行生成分类，即使数据复杂也能确保令人满意的准确性。在 DAP 中使用数据摘要和类似数据可确保上下文感知决策。所提出的方法在提示中使用了“充当可解释的机器学习模型”一词，通过允许用户查看每个预测背后的逻辑来增强预测的可解释性。在某些测试案例中，该系统的准确率超过 90%，证明了该系统的有效性及其在各种场景中超越传统 ML 模型的潜力。代码可从此 https URL 获取