2024-05-15

Title: Many-Shot Regurgitation (MSR) Prompting

• Authors: Shashank Sonkar, Richard G. Baraniuk
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.08134
• Pdf URL: https://arxiv.org/pdf/2405.08134
• Copy Paste: [[2405.08134]] Many-Shot Regurgitation (MSR) Prompting(https://arxiv.org/abs/2405.08134)
• Keywords: language model, gpt, llm, prompt
• Abstract: We introduce Many-Shot Regurgitation (MSR) prompting, a new black-box membership inference attack framework for examining verbatim content reproduction in large language models (LLMs). MSR prompting involves dividing the input text into multiple segments and creating a single prompt that includes a series of faux conversation rounds between a user and a language model to elicit verbatim regurgitation. We apply MSR prompting to diverse text sources, including Wikipedia articles and open educational resources (OER) textbooks, which provide high-quality, factual content and are continuously updated over time. For each source, we curate two dataset types: one that LLMs were likely exposed to during training ($D_{\rm pre}$) and another consisting of documents published after the models' training cutoff dates ($D_{\rm post}$). To quantify the occurrence of verbatim matches, we employ the Longest Common Substring algorithm and count the frequency of matches at different length thresholds. We then use statistical measures such as Cliff's delta, Kolmogorov-Smirnov (KS) distance, and Kruskal-Wallis H test to determine whether the distribution of verbatim matches differs significantly between $D_{\rm pre}$ and $D_{\rm post}$. Our findings reveal a striking difference in the distribution of verbatim matches between $D_{\rm pre}$ and $D_{\rm post}$, with the frequency of verbatim reproduction being significantly higher when LLMs (e.g. GPT models and LLaMAs) are prompted with text from datasets they were likely trained on. For instance, when using GPT-3.5 on Wikipedia articles, we observe a substantial effect size (Cliff's delta $= -0.984$) and a large KS distance ($0.875$) between the distributions of $D_{\rm pre}$ and $D_{\rm post}$. Our results provide compelling evidence that LLMs are more prone to reproducing verbatim content when the input text is likely sourced from their training data.
• 摘要：我们引入了Many-Shot Regurgitation (MSR)提示，这是一种新的黑盒成员推理攻击框架，用于检查大型语言模型(LLM)中的逐字内容再现。 MSR 提示涉及将输入文本分成多个片段并创建单个提示，其中包括用户和语言模型之间的一系列虚假对话，以引发逐字反省。我们将 MSR 提示应用于不同的文本源，包括维基百科文章和开放教育资源 (OER) 教科书，它们提供高质量、真实的内容，并随着时间的推移不断更新。对于每个来源，我们策划了两种数据集类型：一种是法学硕士在训练期间可能接触到的数据集 ($D_{\rm pre}$)，另一种由模型训练截止日期之后发布的文档组成 ($D_{\rm post} $）。为了量化逐字匹配的出现，我们采用最长公共子串算法并计算不同长度阈值下的匹配频率。然后，我们使用 Cliff δ、Kolmogorov-Smirnov (KS) 距离和 Kruskal-Wallis H 检验等统计测量来确定 $D_{\rm pre}$ 和 $D_{\rm post} 之间逐字匹配的分布是否存在显着差异}$。我们的研究结果揭示了 $D_{\rm pre}$ 和 $D_{\rm post}$ 之间逐字匹配的分布存在显着差异，当 LLM（例如 GPT 模型和 LLaMA）被用他们可能接受过训练的数据集中的文本进行提示。例如，当在维基百科文章上使用 GPT-3.5 时，我们观察到 $D_{\rm pre}$ 和 $ 的分布之间存在显着的效应大小（Cliff 的增量 $= -0.984$）和较大的 KS 距离（$0.875$） D_{\rm 帖子}$。我们的结果提供了令人信服的证据，表明当输入文本可能来自其训练数据时，法学硕士更容易逐字复制内容。

Title: Benchmarking Retrieval-Augmented Large Language Models in Biomedical NLP: Application, Robustness, and Self-Awareness

• Authors: Mingchen Li, Zaifu Zhan, Han Yang, Yongkang Xiao, Jiatan Huang, Rui Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.08151
• Pdf URL: https://arxiv.org/pdf/2405.08151
• Copy Paste: [[2405.08151]] Benchmarking Retrieval-Augmented Large Language Models in Biomedical NLP: Application, Robustness, and Self-Awareness(https://arxiv.org/abs/2405.08151)
• Keywords: language model, llm, hallucination
• Abstract: Large language models (LLM) have demonstrated remarkable capabilities in various biomedical natural language processing (NLP) tasks, leveraging the demonstration within the input context to adapt to new tasks. However, LLM is sensitive to the selection of demonstrations. To address the hallucination issue inherent in LLM, retrieval-augmented LLM (RAL) offers a solution by retrieving pertinent information from an established database. Nonetheless, existing research work lacks rigorous evaluation of the impact of retrieval-augmented large language models on different biomedical NLP tasks. This deficiency makes it challenging to ascertain the capabilities of RAL within the biomedical domain. Moreover, the outputs from RAL are affected by retrieving the unlabeled, counterfactual, or diverse knowledge that is not well studied in the biomedical domain. However, such knowledge is common in the real world. Finally, exploring the self-awareness ability is also crucial for the RAL system. So, in this paper, we systematically investigate the impact of RALs on 5 different biomedical tasks (triple extraction, link prediction, classification, question answering, and natural language inference). We analyze the performance of RALs in four fundamental abilities, including unlabeled robustness, counterfactual robustness, diverse robustness, and negative awareness. To this end, we proposed an evaluation framework to assess the RALs' performance on different biomedical NLP tasks and establish four different testbeds based on the aforementioned fundamental abilities. Then, we evaluate 3 representative LLMs with 3 different retrievers on 5 tasks over 9 datasets.
• 摘要：大语言模型 (LLM) 在各种生物医学自然语言处理 (NLP) 任务中展示了卓越的能力，利用输入上下文中的演示来适应新任务。然而，LLM对示范的选择很敏感。为了解决法学硕士固有的幻觉问题，检索增强法学硕士（RAL）通过从已建立的数据库中检索相关信息来提供解决方案。尽管如此，现有的研究工作缺乏对检索增强大型语言模型对不同生物医学 NLP 任务的影响的严格评估。这一缺陷使得确定 RAL 在生物医学领域的能力变得具有挑战性。此外，RAL 的输出受到检索未标记的、反事实的或在生物医学领域尚未得到充分研究的多样化知识的影响。然而，这样的知识在现实世界中很常见。最后，探索自我意识能力对于RAL系统也至关重要。因此，在本文中，我们系统地研究了 RAL 对 5 种不同生物医学任务（三元组提取、链接预测、分类、问答和自然语言推理）的影响。我们分析了 RAL 在四种基本能力方面的表现，包括无标签稳健性、反事实稳健性、多样性稳健性和负面意识。为此，我们提出了一个评估框架来评估 RAL 在不同生物医学 NLP 任务上的表现，并根据上述基本能力建立了四个不同的测试平台。然后，我们使用 3 个不同的检索器在 9 个数据集的 5 项任务上评估 3 个代表性的法学硕士。

Title: Interpreting Latent Student Knowledge Representations in Programming Assignments

• Authors: Nigel Fernandez, Andrew Lan
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.08213
• Pdf URL: https://arxiv.org/pdf/2405.08213
• Copy Paste: [[2405.08213]] Interpreting Latent Student Knowledge Representations in Programming Assignments(https://arxiv.org/abs/2405.08213)
• Keywords: language model
• Abstract: Recent advances in artificial intelligence for education leverage generative large language models, including using them to predict open-ended student responses rather than their correctness only. However, the black-box nature of these models limits the interpretability of the learned student knowledge representations. In this paper, we conduct a first exploration into interpreting latent student knowledge representations by presenting InfoOIRT, an Information regularized Open-ended Item Response Theory model, which encourages the latent student knowledge states to be interpretable while being able to generate student-written code for open-ended programming questions. InfoOIRT maximizes the mutual information between a fixed subset of latent knowledge states enforced with simple prior distributions and generated student code, which encourages the model to learn disentangled representations of salient syntactic and semantic code features including syntactic styles, mastery of programming skills, and code structures. Through experiments on a real-world programming education dataset, we show that InfoOIRT can both accurately generate student code and lead to interpretable student knowledge representations.
• 摘要：教育人工智能的最新进展利用了生成式大型语言模型，包括使用它们来预测开放式学生的反应，而不仅仅是预测其正确性。然而，这些模型的黑盒性质限制了所学学生知识表示的可解释性。在本文中，我们通过提出 InfoOIRT（一种信息正则化开放式项目响应理论模型）对解释潜在学生知识表示进行了首次探索，该模型鼓励潜在学生知识状态可解释，同时能够生成学生编写的代码开放式编程问题。 InfoOIRT 最大化了使用简单先验分布强制执行的潜在知识状态的固定子集与生成的学生代码之间的互信息，这鼓励模型学习显着句法和语义代码特征的解开表示，包括句法风格、编程技能的掌握和代码结构。通过对现实世界编程教育数据集的实验，我们表明 InfoOIRT 既可以准确生成学生代码，又可以生成可解释的学生知识表示。

Title: SpeechVerse: A Large-scale Generalizable Audio Language Model

• Authors: Nilaksh Das, Saket Dingliwal, Srikanth Ronanki, Rohit Paturi, David Huang, Prashant Mathur, Jie Yuan, Dhanush Bekal, Xing Niu, Sai Muralidhar Jayanthi, Xilai Li, Karel Mundnich, Monica Sunkara, Sundararajan Srinivasan, Kyu J Han, Katrin Kirchhoff
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2405.08295
• Pdf URL: https://arxiv.org/pdf/2405.08295
• Copy Paste: [[2405.08295]] SpeechVerse: A Large-scale Generalizable Audio Language Model(https://arxiv.org/abs/2405.08295)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have shown incredible proficiency in performing tasks that require semantic understanding of natural language instructions. Recently, many works have further expanded this capability to perceive multimodal audio and text inputs, but their capabilities are often limited to specific fine-tuned tasks such as automatic speech recognition and translation. We therefore develop SpeechVerse, a robust multi-task training and curriculum learning framework that combines pre-trained speech and text foundation models via a small set of learnable parameters, while keeping the pre-trained models frozen during training. The models are instruction finetuned using continuous latent representations extracted from the speech foundation model to achieve optimal zero-shot performance on a diverse range of speech processing tasks using natural language instructions. We perform extensive benchmarking that includes comparing our model performance against traditional baselines across several datasets and tasks. Furthermore, we evaluate the model's capability for generalized instruction following by testing on out-of-domain datasets, novel prompts, and unseen tasks. Our empirical experiments reveal that our multi-task SpeechVerse model is even superior to conventional task-specific baselines on 9 out of the 11 tasks.
• 摘要：大型语言模型 (LLM) 在执行需要对自然语言指令进行语义理解的任务方面表现出了令人难以置信的熟练程度。最近，许多作品进一步扩展了这种能力来感知多模态音频和文本输入，但它们的能力通常仅限于特定的微调任务，例如自动语音识别和翻译。因此，我们开发了 SpeechVerse，这是一个强大的多任务训练和课程学习框架，它通过一小组可学习参数将预训练的语音和文本基础模型结合起来，同时在训练期间保持预训练模型的冻结。这些模型使用从语音基础模型中提取的连续潜在表示进行指令微调，以使用自然语言指令在各种语音处理任务上实现最佳的零样本性能。我们执行广泛的基准测试，包括将我们的模型性能与跨多个数据集和任务的传统基线进行比较。此外，我们通过对域外数据集、新颖的提示和未见过的任务进行测试来评估模型的通用指令能力。我们的实证实验表明，我们的多任务 SpeechVerse 模型在 11 项任务中的 9 项上甚至优于传统的特定任务基线。

Title: Computational Thought Experiments for a More Rigorous Philosophy and Science of the Mind

• Authors: Iris Over, Nikhil Krishnaswamy, James Pustejovsky, Joshua Hartshorne
• Subjects: cs.CL, cs.AI, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2405.08304
• Pdf URL: https://arxiv.org/pdf/2405.08304
• Copy Paste: [[2405.08304]] Computational Thought Experiments for a More Rigorous Philosophy and Science of the Mind(https://arxiv.org/abs/2405.08304)
• Keywords: agent
• Abstract: We offer philosophical motivations for a method we call Virtual World Cognitive Science (VW CogSci), in which researchers use virtual embodied agents that are embedded in virtual worlds to explore questions in the field of Cognitive Science. We focus on questions about mental and linguistic representation and the ways that such computational modeling can add rigor to philosophical thought experiments, as well as the terminology used in the scientific study of such representations. We find that this method forces researchers to take a god's-eye view when describing dynamical relationships between entities in minds and entities in an environment in a way that eliminates the need for problematic talk of belief and concept types, such as the belief that cats are silly, and the concept CAT, while preserving belief and concept tokens in individual cognizers' minds. We conclude with some further key advantages of VW CogSci for the scientific study of mental and linguistic representation and for Cognitive Science more broadly.
• 摘要：我们为一种称为虚拟世界认知科学（VW CogSci）的方法提供了哲学动机，其中研究人员使用嵌入虚拟世界的虚拟实体来探索认知科学领域的问题。我们关注有关心理和语言表征的问题，以及这种计算模型如何为哲学思想实验增加严谨性，以及在此类表征的科学研究中使用的术语。我们发现，这种方法迫使研究人员在描述思想中的实体与环境中的实体之间的动态关系时采取上帝视角的观点，从而消除了对信念和概念类型有问题的谈论的需要，例如猫是的信念愚蠢的，以及概念 CAT，同时在个体认知者的头脑中保留信念和概念标记。我们总结了 VW CogSci 对于心理和语言表征科学研究以及更广泛的认知科学的一些进一步的关键优势。

Title: SpeechGuard: Exploring the Adversarial Robustness of Multimodal Large Language Models

• Authors: Raghuveer Peri, Sai Muralidhar Jayanthi, Srikanth Ronanki, Anshu Bhatia, Karel Mundnich, Saket Dingliwal, Nilaksh Das, Zejiang Hou, Goeric Huybrechts, Srikanth Vishnubhotla, Daniel Garcia-Romero, Sundararajan Srinivasan, Kyu J Han, Katrin Kirchhoff
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2405.08317
• Pdf URL: https://arxiv.org/pdf/2405.08317
• Copy Paste: [[2405.08317]] SpeechGuard: Exploring the Adversarial Robustness of Multimodal Large Language Models(https://arxiv.org/abs/2405.08317)
• Keywords: language model
• Abstract: Integrated Speech and Large Language Models (SLMs) that can follow speech instructions and generate relevant text responses have gained popularity lately. However, the safety and robustness of these models remains largely unclear. In this work, we investigate the potential vulnerabilities of such instruction-following speech-language models to adversarial attacks and jailbreaking. Specifically, we design algorithms that can generate adversarial examples to jailbreak SLMs in both white-box and black-box attack settings without human involvement. Additionally, we propose countermeasures to thwart such jailbreaking attacks. Our models, trained on dialog data with speech instructions, achieve state-of-the-art performance on spoken question-answering task, scoring over 80% on both safety and helpfulness metrics. Despite safety guardrails, experiments on jailbreaking demonstrate the vulnerability of SLMs to adversarial perturbations and transfer attacks, with average attack success rates of 90% and 10% respectively when evaluated on a dataset of carefully designed harmful questions spanning 12 different toxic categories. However, we demonstrate that our proposed countermeasures reduce the attack success significantly.
• 摘要：可以遵循语音指令并生成相关文本响应的集成语音和大型语言模型 (SLM) 最近很受欢迎。然而，这些模型的安全性和稳健性在很大程度上仍不清楚。在这项工作中，我们研究了这种遵循指令的语音语言模型在对抗性攻击和越狱方面的潜在漏洞。具体来说，我们设计的算法可以生成对抗性示例，以便在白盒和黑盒攻击设置中越狱 SLM，而无需人工参与。此外，我们还提出了阻止此类越狱攻击的对策。我们的模型经过语音指令的对话数据训练，在语音问答任务中实现了最先进的性能，在安全性和有用性指标上得分超过 80%。尽管有安全护栏，越狱实验证明了 SLM 容易受到对抗性扰动和转移攻击的影响，在精心设计的涵盖 12 个不同有毒类别的有害问题数据集上进行评估时，平均攻击​​成功率分别为 90% 和 10%。然而，我们证明我们提出的对策会显着降低攻击成功率。

Title: Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark

• Authors: Mengsong Wu, Tong Zhu, Han Han, Chuanyuan Tan, Xiang Zhang, Wenliang Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.08355
• Pdf URL: https://arxiv.org/pdf/2405.08355
• Copy Paste: [[2405.08355]] Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark(https://arxiv.org/abs/2405.08355)
• Keywords: llm, agent
• Abstract: This paper presents a new tool learning dataset Seal-Tools, which contains self-instruct API-like tools. Seal-Tools not only offers a large number of tools, but also includes instances which demonstrate the practical application of tools. Seeking to generate data on a large scale while ensuring reliability, we propose a self-instruct method to generate tools and instances, allowing precise control over the process. Moreover, our Seal-Tools contains hard instances that call multiple tools to complete the job, among which some are nested tool callings. For precise and comprehensive evaluation, we use strict format control and design three metrics from different dimensions. Therefore, Seal-Tools can serve as a new benchmark to evaluate the tool-calling ability of LLMs. Finally, we evaluate several prevalent LLMs and our finetuned model on Seal-Tools. The results show that current systems are far from perfect. The code, data and experiment results are available at this https URL .
• 摘要：本文提出了一个新的工具学习数据集 Seal-Tools，其中包含类似 API 的自指导工具。 Seal-Tools不仅提供了大量的工具，还包含展示工具实际应用的实例。为了在确保可靠性的同时大规模生成数据，我们提出了一种自指导方法来生成工具和实例，从而可以精确控制过程。此外，我们的 Seal-Tools 包含调用多个工具来完成工作的硬实例，其中一些是嵌套的工具调用。为了精准、全面的评价，我们采用严格的格式控制，从不同维度设计了三个指标。因此，Seal-Tools可以作为评价LLM工具调用能力的新标杆。最后，我们评估了几种流行的法学硕士和我们在 Seal-Tools 上的微调模型。结果表明，当前的系统远非完美。代码、数据和实验结果可在此 https URL 获取。

Title: PromptMind Team at MEDIQA-CORR 2024: Improving Clinical Text Correction with Error Categorization and LLM Ensembles

• Authors: Satya Kesav Gundabathula, Sriram R Kolar
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.08373
• Pdf URL: https://arxiv.org/pdf/2405.08373
• Copy Paste: [[2405.08373]] PromptMind Team at MEDIQA-CORR 2024: Improving Clinical Text Correction with Error Categorization and LLM Ensembles(https://arxiv.org/abs/2405.08373)
• Keywords: language model, llm, prompt
• Abstract: This paper describes our approach to the MEDIQA-CORR shared task, which involves error detection and correction in clinical notes curated by medical professionals. This task involves handling three subtasks: detecting the presence of errors, identifying the specific sentence containing the error, and correcting it. Through our work, we aim to assess the capabilities of Large Language Models (LLMs) trained on a vast corpora of internet data that contain both factual and unreliable information. We propose to comprehensively address all subtasks together, and suggest employing a unique prompt-based in-context learning strategy. We will evaluate its efficacy in this specialized task demanding a combination of general reasoning and medical knowledge. In medical systems where prediction errors can have grave consequences, we propose leveraging self-consistency and ensemble methods to enhance error correction and error detection performance.
• 摘要：本文描述了我们执行 MEDIQA-CORR 共享任务的方法，其中涉及医疗专业人员整理的临床记录中的错误检测和纠正。该任务涉及处理三个子任务：检测错误的存在、识别包含错误的特定句子并纠正它。通过我们的工作，我们的目标是评估大型语言模型 (LLM) 的能力，这些模型是在包含事实信息和不可靠信息的大量互联网数据上训练的。我们建议综合解决所有子任务，并建议采用独特的基于提示的情境学习策略。我们将评估其在这项需要结合一般推理和医学知识的专门任务中的功效。在预测错误可能产生严重后果的医疗系统中，我们建议利用自洽和集成方法来增强纠错和错误检测性能。

Title: Stylometric Watermarks for Large Language Models

• Authors: Georg Niess, Roman Kern
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.08400
• Pdf URL: https://arxiv.org/pdf/2405.08400
• Copy Paste: [[2405.08400]] Stylometric Watermarks for Large Language Models(https://arxiv.org/abs/2405.08400)
• Keywords: language model, llm
• Abstract: The rapid advancement of large language models (LLMs) has made it increasingly difficult to distinguish between text written by humans and machines. Addressing this, we propose a novel method for generating watermarks that strategically alters token probabilities during generation. Unlike previous works, this method uniquely employs linguistic features such as stylometry. Concretely, we introduce acrostica and sensorimotor norms to LLMs. Further, these features are parameterized by a key, which is updated every sentence. To compute this key, we use semantic zero shot classification, which enhances resilience. In our evaluation, we find that for three or more sentences, our method achieves a false positive and false negative rate of 0.02. For the case of a cyclic translation attack, we observe similar results for seven or more sentences. This research is of particular of interest for proprietary LLMs to facilitate accountability and prevent societal harm.
• 摘要：大型语言模型（LLM）的快速发展使得区分人类和机器编写的文本变得越来越困难。为了解决这个问题，我们提出了一种生成水印的新方法，该方法可以在生成过程中策略性地改变令牌概率。与以前的作品不同，该方法独特地采用了语言特征，例如文体测量学。具体来说，我们向法学硕士介绍了离合诗和感觉运动规范。此外，这些特征由一个键参数化，每个句子都会更新该键。为了计算这个密钥，我们使用语义零样本分类，这增强了弹性。在我们的评估中，我们发现对于三个或更多句子，我们的方法实现了 0.02 的假阳性率和假阴性率。对于循环翻译攻击的情况，我们观察到七个或更多句子的类似结果。这项研究对于专有法学硕士特别感兴趣，可以促进问责制并防止社会危害。

Title: Impact of Stickers on Multimodal Chat Sentiment Analysis and Intent Recognition: A New Task, Dataset and Baseline

• Authors: Yuanchen Shi, Biao Ma, Fang Kong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.08427
• Pdf URL: https://arxiv.org/pdf/2405.08427
• Copy Paste: [[2405.08427]] Impact of Stickers on Multimodal Chat Sentiment Analysis and Intent Recognition: A New Task, Dataset and Baseline(https://arxiv.org/abs/2405.08427)
• Keywords: chat
• Abstract: Stickers are increasingly used in social media to express sentiment and intent. When finding typing troublesome, people often use a sticker instead. Despite the significant impact of stickers on sentiment analysis and intent recognition, little research has been conducted. To address this gap, we propose a new task: Multimodal chat Sentiment Analysis and Intent Recognition involving Stickers (MSAIRS). Additionally, we introduce a novel multimodal dataset containing Chinese chat records and stickers excerpted from several mainstream social media platforms. Our dataset includes paired data with the same text but different stickers, and various stickers consisting of the same images with different texts, allowing us to better understand the impact of stickers on chat sentiment and intent. We also propose an effective multimodal joint model, MMSAIR, for our task, which is validated on our datasets and indicates that visual information of stickers counts. Our dataset and code will be publicly available.
• 摘要：社交媒体中越来越多地使用贴纸来表达情感和意图。当发现打字麻烦时，人们通常会使用贴纸来代替。尽管贴纸对情感分析和意图识别有重大影响，但很少有研究进行。为了解决这一差距，我们提出了一项新任务：涉及贴纸的多模式聊天情感分析和意图识别（MSAIRS）。此外，我们还引入了一个新颖的多模态数据集，其中包含从几个主流社交媒体平台摘录的中文聊天记录和贴纸。我们的数据集包括具有相同文本但不同贴纸的配对数据，以及由相同图像和不同文本组成的各种贴纸，使我们能够更好地了解贴纸对聊天情绪和意图的影响。我们还为我们的任务提出了一种有效的多模态联合模型 MMSAIR，该模型在我们的数据集上进行了验证，并表明贴纸的视觉信息很重要。我们的数据集和代码将公开。

Title: Evaluating LLMs at Evaluating Temporal Generalization

• Authors: Chenghao Zhu, Nuo Chen, Yufei Gao, Benyou Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Evaluating LLMs at Evaluating Temporal Generalization(https://arxiv.org/abs/)
• Keywords: language model, llm
• Abstract: The rapid advancement of Large Language Models (LLMs) highlights the urgent need for evolving evaluation methodologies that keep pace with improvements in language comprehension and information processing. However, traditional benchmarks, which are often static, fail to capture the continually changing information landscape, leading to a disparity between the perceived and actual effectiveness of LLMs in ever-changing real-world scenarios. Furthermore, these benchmarks do not adequately measure the models' capabilities over a broader temporal range or their adaptability over time. We examine current LLMs in terms of temporal generalization and bias, revealing that various temporal biases emerge in both language likelihood and prognostic prediction. This serves as a caution for LLM practitioners to pay closer attention to mitigating temporal biases. Also, we propose an evaluation framework Freshbench for dynamically generating benchmarks from the most recent real-world prognostication prediction. Our code is available at this https URL. The dataset will be released soon.
• 摘要：大型语言模型（LLM）的快速发展凸显了对不断发展的评估方法的迫切需要，以跟上语言理解和信息处理的进步。然而，传统的基准通常是静态的，无法捕捉不断变化的信息环境，导致法学硕士在不断变化的现实场景中的感知效果和实际效果之间存在差异。此外，这些基准测试并不能充分衡量模型在更广泛的时间范围内的能力或其随时间的适应性。我们从时间泛化和偏差方面检查了当前的法学硕士，揭示了语言可能性和预后预测中都出现了各种时间偏差。这提醒法学硕士从业者要更加关注减轻时间偏差。此外，我们提出了一个评估框架 Freshbench，用于根据最新的现实世界预测动态生成基准。我们的代码可以在这个 https URL 上找到。该数据集即将发布。

Title: Challenges and Opportunities in Text Generation Explainability

• Authors: Kenza Amara, Rita Sevastjanova, Mennatallah El-Assady
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.08468
• Pdf URL: https://arxiv.org/pdf/2405.08468
• Copy Paste: [[2405.08468]] Challenges and Opportunities in Text Generation Explainability(https://arxiv.org/abs/2405.08468)
• Keywords: language model
• Abstract: The necessity for interpretability in natural language processing (NLP) has risen alongside the growing prominence of large language models. Among the myriad tasks within NLP, text generation stands out as a primary objective of autoregressive models. The NLP community has begun to take a keen interest in gaining a deeper understanding of text generation, leading to the development of model-agnostic explainable artificial intelligence (xAI) methods tailored to this task. The design and evaluation of explainability methods are non-trivial since they depend on many factors involved in the text generation process, e.g., the autoregressive model and its stochastic nature. This paper outlines 17 challenges categorized into three groups that arise during the development and assessment of attribution-based explainability methods. These challenges encompass issues concerning tokenization, defining explanation similarity, determining token importance and prediction change metrics, the level of human intervention required, and the creation of suitable test datasets. The paper illustrates how these challenges can be intertwined, showcasing new opportunities for the community. These include developing probabilistic word-level explainability methods and engaging humans in the explainability pipeline, from the data design to the final evaluation, to draw robust conclusions on xAI methods.
• 摘要：随着大型语言模型的日益重要，自然语言处理 (NLP) 中可解释性的必要性也随之增加。在 NLP 的众多任务中，文本生成是自回归模型的主要目标。 NLP 社区已经开始对更深入地理解文本生成产生了浓厚的兴趣，从而导致了针对此任务量身定制的与模型无关的可解释人工智能 (xAI) 方法的开发。可解释性方法的设计和评估并非易事，因为它们取决于文本生成过程中涉及的许多因素，例如自回归模型及其随机性质。本文概述了在开发和评估基于归因的可解释性方法期间出现的 17 个挑战，分为三组。这些挑战包括有关标记化、定义解释相似性、确定标记重要性和预测变化指标、所需的人工干预水平以及创建合适的测试数据集的问题。本文阐述了这些挑战如何交织在一起，为社区展示了新的机遇。其中包括开发概率词级可解释性方法，并让人类参与从数据设计到最终评估的可解释性流程，以得出关于 xAI 方法的可靠结论。

Title: GPT-3.5 for Grammatical Error Correction

• Authors: Anisia Katinskaia, Roman Yangarber
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.08469
• Pdf URL: https://arxiv.org/pdf/2405.08469
• Copy Paste: [[2405.08469]] GPT-3.5 for Grammatical Error Correction(https://arxiv.org/abs/2405.08469)
• Keywords: language model, gpt
• Abstract: This paper investigates the application of GPT-3.5 for Grammatical Error Correction (GEC) in multiple languages in several settings: zero-shot GEC, fine-tuning for GEC, and using GPT-3.5 to re-rank correction hypotheses generated by other GEC models. In the zero-shot setting, we conduct automatic evaluations of the corrections proposed by GPT-3.5 using several methods: estimating grammaticality with language models (LMs), the Scribendi test, and comparing the semantic embeddings of sentences. GPT-3.5 has a known tendency to over-correct erroneous sentences and propose alternative corrections. For several languages, such as Czech, German, Russian, Spanish, and Ukrainian, GPT-3.5 substantially alters the source sentences, including their semantics, which presents significant challenges for evaluation with reference-based metrics. For English, GPT-3.5 demonstrates high recall, generates fluent corrections, and generally preserves sentence semantics. However, human evaluation for both English and Russian reveals that, despite its strong error-detection capabilities, GPT-3.5 struggles with several error types, including punctuation mistakes, tense errors, syntactic dependencies between words, and lexical compatibility at the sentence level.
• 摘要：本文研究了 GPT-3.5 在多种语言中在多种设置中的语法错误校正 (GEC) 的应用：零样本 GEC、GEC 微调以及使用 GPT-3.5 对其他 GEC 模型生成的校正假设进行重新排序。在零样本设置中，我们使用多种方法对 GPT-3.5 提出的修正进行自动评估：使用语言模型（LM）估计语法性、Scribendi 测试以及比较句子的语义嵌入。众所周知，GPT-3.5 倾向于过度纠正错误句子并提出替代纠正措施。对于捷克语、德语、俄语、西班牙语和乌克兰语等多种语言，GPT-3.5 极大地改变了源句子，包括其语义，这对使用基于参考的指标进行评估提出了重大挑战。对于英语，GPT-3.5 表现出高召回率，生成流畅的更正，并且通常保留句子语义。然而，对英语和俄语的人工评估表明，尽管 GPT-3.5 具有强大的错误检测能力，但它仍难以解决多种错误类型，包括标点符号错误、时态错误、单词之间的句法依赖性以及句子级别的词汇兼容性。

Title: Enhancing Gender-Inclusive Machine Translation with Neomorphemes and Large Language Models

• Authors: Andrea Piergentili, Beatrice Savoldi, Matteo Negri, Luisa Bentivogli
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.08477
• Pdf URL: https://arxiv.org/pdf/2405.08477
• Copy Paste: [[2405.08477]] Enhancing Gender-Inclusive Machine Translation with Neomorphemes and Large Language Models(https://arxiv.org/abs/2405.08477)
• Keywords: language model, llm, prompt
• Abstract: Machine translation (MT) models are known to suffer from gender bias, especially when translating into languages with extensive gendered morphology. Accordingly, they still fall short in using gender-inclusive language, also representative of non-binary identities. In this paper, we look at gender-inclusive neomorphemes, neologistic elements that avoid binary gender markings as an approach towards fairer MT. In this direction, we explore prompting techniques with large language models (LLMs) to translate from English into Italian using neomorphemes. So far, this area has been under-explored due to its novelty and the lack of publicly available evaluation resources. We fill this gap by releasing Neo-GATE, a resource designed to evaluate gender-inclusive en-it translation with neomorphemes. With Neo-GATE, we assess four LLMs of different families and sizes and different prompt formats, identifying strengths and weaknesses of each on this novel task for MT.
• 摘要：众所周知，机器翻译（MT）模型存在性别偏见，尤其是在翻译成具有广泛性别形态的语言时。因此，他们在使用性别包容性语言（也代表非二元身份）方面仍然存在不足。在本文中，我们研究了性别包容的新词素，即避免二元性别标记的新元素，作为实现更公平的机器翻译的方法。在这个方向上，我们探索使用大语言模型（LLM）的提示技术，使用新词素将英语翻译成意大利语。到目前为止，由于其新颖性和缺乏公开可用的评估资源，该领域尚未得到充分探索。我们通过发布 Neo-GATE 来填补这一空白，该资源旨在评估具有新词素的性别包容 en-it 翻译。通过 Neo-GATE，我们评估了四位不同家族、规模和不同提示格式的法学硕士，确定了每个人在这项新的 MT 任务中的优势和劣势。

Title: Is Less More? Quality, Quantity and Context in Idiom Processing with Natural Language Models

• Authors: Agne Knietaite, Adam Allsebrook, Anton Minkov, Adam Tomaszewski, Norbert Slinko, Richard Johnson, Thomas Pickard, Dylan Phelps, Aline Villavicencio
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.08497
• Pdf URL: https://arxiv.org/pdf/2405.08497
• Copy Paste: [[2405.08497]] Is Less More? Quality, Quantity and Context in Idiom Processing with Natural Language Models(https://arxiv.org/abs/2405.08497)
• Keywords: language model
• Abstract: Compositionality in language models presents a problem when processing idiomatic expressions, as their meaning often cannot be directly derived from their individual parts. Although fine-tuning and other optimization strategies can be used to improve representations of idiomatic expressions, this depends on the availability of relevant data. We present the Noun Compound Synonym Substitution in Books - NCSSB - datasets, which are created by substitution of synonyms of potentially idiomatic English noun compounds in public domain book texts. We explore the trade-off between data quantity and quality when training models for idiomaticity detection, in conjunction with contextual information obtained locally (from the surrounding sentences) or externally (through language resources). Performance on an idiomaticity detection task indicates that dataset quality is a stronger factor for context-enriched models, but that quantity also plays a role in models without context inclusion strategies.
• 摘要：语言模型中的组合性在处理惯用表达时会出现问题，因为它们的含义通常无法直接从其各个部分得出。尽管可以使用微调和其他优化策略来改进惯用表达的表示，但这取决于相关数据的可用性。我们提出了书籍中的名词复合同义词替换 - NCSSB - 数据集，这些数据集是通过替换公共领域书籍文本中潜在惯用的英语名词复合词的同义词而创建的。我们在训练惯用性检测模型时，结合本地（从周围句子）或外部（通过语言资源）获得的上下文信息，探索数据数量和质量之间的权衡。惯用性检测任务的表现表明，数据集质量对于上下文丰富的模型来说是一个更重要的因素，但该数量在没有上下文包含策略的模型中也发挥着作用。

Title: Archimedes-AUEB at SemEval-2024 Task 5: LLM explains Civil Procedure

• Authors: Odysseas S. Chlapanis, Ion Androutsopoulos, Dimitrios Galanis
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.08502
• Pdf URL: https://arxiv.org/pdf/2405.08502
• Copy Paste: [[2405.08502]] Archimedes-AUEB at SemEval-2024 Task 5: LLM explains Civil Procedure(https://arxiv.org/abs/2405.08502)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: The SemEval task on Argument Reasoning in Civil Procedure is challenging in that it requires understanding legal concepts and inferring complex arguments. Currently, most Large Language Models (LLM) excelling in the legal realm are principally purposed for classification tasks, hence their reasoning rationale is subject to contention. The approach we advocate involves using a powerful teacher-LLM (ChatGPT) to extend the training dataset with explanations and generate synthetic data. The resulting data are then leveraged to fine-tune a small student-LLM. Contrary to previous work, our explanations are not directly derived from the teacher's internal knowledge. Instead they are grounded in authentic human analyses, therefore delivering a superior reasoning signal. Additionally, a new `mutation' method generates artificial data instances inspired from existing ones. We are publicly releasing the explanations as an extension to the original dataset, along with the synthetic dataset and the prompts that were used to generate both. Our system ranked 15th in the SemEval competition. It outperforms its own teacher and can produce explanations aligned with the original human analyses, as verified by legal experts.
• 摘要：民事诉讼中论证推理的 SemEval 任务具有挑战性，因为它需要理解法律概念并推断复杂的论证。目前，大多数在法律领域表现出色的大型语言模型（LLM）主要用于分类任务，因此它们的推理原理存在争议。我们提倡的方法涉及使用强大的法学硕士教师（ChatGPT）来扩展训练数据集并提供解释并生成合成数据。然后利用所得数据对小型法学硕士学生进行微调。与以前的工作相反，我们的解释并不是直接来自教师的内部知识。相反，它们以真实的人类分析为基础，因此可以提供卓越的推理信号。此外，一种新的“变异”方法会根据现有数据实例生成人工数据实例。我们公开发布这些解释作为原始数据集的扩展，以及合成数据集和用于生成这两个数据集的提示。我们的系统在 SemEval 竞赛中排名第 15。它的表现优于自己的老师，并且可以产生与原始人类分析一致的解释，并得到法律专家的验证。

Title: A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine

• Authors: Hanguang Xiao, Feizhong Zhou, Xingyue Liu, Tianqi Liu, Zhipeng Li, Xin Liu, Xiaoxuan Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.08603
• Pdf URL: https://arxiv.org/pdf/2405.08603
• Copy Paste: [[2405.08603]] A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine(https://arxiv.org/abs/2405.08603)
• Keywords: language model, gpt, llm, chat
• Abstract: Since the release of ChatGPT and GPT-4, large language models (LLMs) and multimodal large language models (MLLMs) have garnered significant attention due to their powerful and general capabilities in understanding, reasoning, and generation, thereby offering new paradigms for the integration of artificial intelligence with medicine. This survey comprehensively overviews the development background and principles of LLMs and MLLMs, as well as explores their application scenarios, challenges, and future directions in medicine. Specifically, this survey begins by focusing on the paradigm shift, tracing the evolution from traditional models to LLMs and MLLMs, summarizing the model structures to provide detailed foundational knowledge. Subsequently, the survey details the entire process from constructing and evaluating to using LLMs and MLLMs with a clear logic. Following this, to emphasize the significant value of LLMs and MLLMs in healthcare, we survey and summarize 6 promising applications in healthcare. Finally, the survey discusses the challenges faced by medical LLMs and MLLMs and proposes a feasible approach and direction for the subsequent integration of artificial intelligence with medicine. Thus, this survey aims to provide researchers with a valuable and comprehensive reference guide from the perspectives of the background, principles, and clinical applications of LLMs and MLLMs.
• 摘要：自 ChatGPT 和 GPT-4 发布以来，大语言模型（LLM）和多模态大语言模型（MLLM）因其在理解、推理和生成方面强大而通用的能力而受到了极大的关注，从而为集成提供了新的范例人工智能与医学的结合。本次调查全面概述了LLM和MLLM的发展背景和原理，并探讨了它们在医学领域的应用场景、挑战和未来方向。具体来说，本次调查首先关注范式转变，追踪从传统模型到法学硕士和 MLLM 的演变，总结模型结构以提供详细的基础知识。随后，调查详细介绍了从构建、评估到使用LLM和MLLM的整个过程，逻辑清晰。接下来，为了强调 LLM 和 MLLM 在医疗保健领域的重要价值，我们调查并总结了 6 个有前景的医疗保健应用。最后，调查讨论了医学LLM和MLLM面临的挑战，并为后续人工智能与医学的融合提出了可行的方法和方向。因此，本次调查旨在从LLM和MLLM的背景、原理和临床应用等角度为研究人员提供有价值且全面的参考指南。

Title: ALMol: Aligned Language-Molecule Translation LLMs through Offline Preference Contrastive Optimisation

• Authors: Dimitris Gkoumas
• Subjects: cs.CL, cs.MM
• Abstract URL: https://arxiv.org/abs/2405.08619
• Pdf URL: https://arxiv.org/pdf/2405.08619
• Copy Paste: [[2405.08619]] ALMol: Aligned Language-Molecule Translation LLMs through Offline Preference Contrastive Optimisation(https://arxiv.org/abs/2405.08619)
• Keywords: language model, llm
• Abstract: The field of chemistry and Artificial Intelligence (AI) intersection is an area of active research that aims to accelerate scientific discovery. The integration of large language models (LLMs) with scientific modalities has shown significant promise in this endeavour. However, challenges persist in effectively addressing training efficacy and the out-of-distribution problem, particularly as existing approaches rely on larger models and datasets. In this context, we focus on machine language-molecule translation and deploy a novel training approach called contrastive preference optimisation, which avoids generating translations that are merely adequate but not perfect. To ensure generalisability and mitigate memorisation effects, we conduct experiments using only 10\% of the data. Our results demonstrate that our models achieve up to a 32\% improvement compared to counterpart models. We also introduce a scalable fine-grained evaluation methodology that accommodates responsibility.
• 摘要：化学和人工智能 (AI) 交叉领域是一个活跃的研究领域，旨在加速科学发现。大语言模型（LLM）与科学模式的整合在这一努力中显示出了巨大的前景。然而，有效解决训练效率和分布外问题仍然存在挑战，特别是当现有方法依赖于更大的模型和数据集时。在这种背景下，我们专注于机器语言分子翻译，并部署了一种称为对比偏好优化的新颖训练方法，该方法避免生成仅仅足够但不完美的翻译。为了确保通用性并减轻记忆效应，我们仅使用 10% 的数据进行实验。我们的结果表明，与对应模型相比，我们的模型实现了高达 32% 的改进。我们还引入了一种可扩展的细粒度评估方法来适应责任。

Title: Thinking Tokens for Language Modeling

• Authors: David Herel, Tomas Mikolov
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.08644
• Pdf URL: https://arxiv.org/pdf/2405.08644
• Copy Paste: [[2405.08644]] Thinking Tokens for Language Modeling(https://arxiv.org/abs/2405.08644)
• Keywords: language model
• Abstract: How much is 56 times 37? Language models often make mistakes in these types of difficult calculations. This is usually explained by their inability to perform complex reasoning. Since language models rely on large training sets and great memorization capability, naturally they are not equipped to run complex calculations. However, one can argue that humans also cannot perform this calculation immediately and require a considerable amount of time to construct the solution. In order to enhance the generalization capability of language models, and as a parallel to human behavior, we propose to use special 'thinking tokens' which allow the model to perform much more calculations whenever a complex problem is encountered.
• 摘要：56 乘以 37 等于多少？语言模型在这些类型的困难计算中经常会出错。这通常是因为他们无法进行复杂的推理。由于语言模型依赖于大量的训练集和强大的记忆能力，因此它们自然不具备运行复杂计算的能力。然而，有人可能会说，人类也无法立即执行这一计算，并且需要大量时间来构建解决方案。为了增强语言模型的泛化能力，并与人类行为平行，我们建议使用特殊的“思维标记”，使模型在遇到复杂问题时能够执行更多计算。

Title: Is the Pope Catholic? Yes, the Pope is Catholic. Generative Evaluation of Intent Resolution in LLMs

• Authors: Akhila Yerukola, Saujas Vaduguru, Daniel Fried, Maarten Sap
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Is the Pope Catholic? Yes, the Pope is Catholic. Generative Evaluation of Intent Resolution in LLMs(https://arxiv.org/abs/)
• Keywords: language model, llm, chain-of-thought
• Abstract: Humans often express their communicative intents indirectly or non-literally, which requires their interlocutors -- human or AI -- to understand beyond the literal meaning of words. While most existing work has focused on discriminative evaluations, we present a new approach to generatively evaluate large language models' (LLMs') intention understanding by examining their responses to non-literal utterances. Ideally, an LLM should respond in line with the true intention of a non-literal utterance, not its literal interpretation. Our findings show that LLMs struggle to generate pragmatically relevant responses to non-literal language, achieving only 50-55% accuracy on average. While explicitly providing oracle intentions significantly improves performance (e.g., 75% for Mistral-Instruct), this still indicates challenges in leveraging given intentions to produce appropriate responses. Using chain-of-thought to make models spell out intentions yields much smaller gains (60% for Mistral-Instruct). These findings suggest that LLMs are not yet effective pragmatic interlocutors, highlighting the need for better approaches for modeling intentions and utilizing them for pragmatic generation.
• 摘要：人类经常间接或非字面地表达他们的交流意图，这要求他们的对话者——人类或人工智能——理解单词的字面意义之外的含义。虽然大多数现有工作都集中在判别性评估上，但我们提出了一种新方法，通过检查大型语言模型（LLM）对非文字话语的反应来生成评估其意图理解。理想情况下，法学硕士应该根据非字面话语的真实意图做出回应，而不是其字面解释。我们的研究结果表明，法学硕士很难对非文字语言做出务实相关的回答，平均准确率仅为 50-55%。虽然明确提供预言机意图可以显着提高性能（例如，Mistral-Instruct 可以提高 75%），但这仍然表明在利用给定意图产生适当响应方面存在挑战。使用思维链让模型阐明意图产生的收益要小得多（Mistral-Instruct 为 60%）。这些发现表明法学硕士还不是有效的务实对话者，强调需要更好的方法来建模意图并利用它们进行务实生成。

Title: Refinement of an Epilepsy Dictionary through Human Annotation of Health-related posts on Instagram

• Authors: Aehong Min, Xuan Wang, Rion Brattig Correia, Jordan Rozum, Wendy R. Miller, Luis M. Rocha
• Subjects: cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2405.08784
• Pdf URL: https://arxiv.org/pdf/2405.08784
• Copy Paste: [[2405.08784]] Refinement of an Epilepsy Dictionary through Human Annotation of Health-related posts on Instagram(https://arxiv.org/abs/2405.08784)
• Keywords: gpt
• Abstract: We used a dictionary built from biomedical terminology extracted from various sources such as DrugBank, MedDRA, MedlinePlus, TCMGeneDIT, to tag more than 8 million Instagram posts by users who have mentioned an epilepsy-relevant drug at least once, between 2010 and early 2016. A random sample of 1,771 posts with 2,947 term matches was evaluated by human annotators to identify false-positives. OpenAI's GPT series models were compared against human annotation. Frequent terms with a high false-positive rate were removed from the dictionary. Analysis of the estimated false-positive rates of the annotated terms revealed 8 ambiguous terms (plus synonyms) used in Instagram posts, which were removed from the original dictionary. To study the effect of removing those terms, we constructed knowledge networks using the refined and the original dictionaries and performed an eigenvector-centrality analysis on both networks. We show that the refined dictionary thus produced leads to a significantly different rank of important terms, as measured by their eigenvector-centrality of the knowledge networks. Furthermore, the most important terms obtained after refinement are of greater medical relevance. In addition, we show that OpenAI's GPT series models fare worse than human annotators in this task.
• 摘要：我们使用从 DrugBank、MedDRA、MedlinePlus、TCMGeneDIT 等各种来源提取的生物医学术语构建的词典，对 2010 年至 2016 年初至少一次提到癫痫相关药物的用户发布的超过 800 万条 Instagram 帖子进行了标记。人类注释者对 1,771 个帖子（其中有 2,947 个术语匹配）的随机样本进行了评估，以识别误报。 OpenAI 的 GPT 系列模型与人工注释进行了比较。误报率高的常用术语已从词典中删除。对带注释术语的估计误报率进行分析，发现 Instagram 帖子中使用了 8 个不明确的术语（以及同义词），这些术语已从原始词典中删除。为了研究删除这些术语的效果，我们使用精炼词典和原始词典构建了知识网络，并对两个网络进行了特征向量中心性分析。我们表明，由此产生的精炼词典导致重要术语的排名显着不同，这是通过知识网络的特征向量中心性来衡量的。此外，细化后获得的最重要术语具有更大的医学相关性。此外，我们还表明 OpenAI 的 GPT 系列模型在此任务中的表现比人类注释者差。