2024-05-21

Title: The Unappreciated Role of Intent in Algorithmic Moderation of Social Media Content

• Authors: Xinyu Wang, Sai Koneru, Pranav Narayanan Venkit, Brett Frischmann, Sarah Rajtmajer
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.11030
• Pdf URL: https://arxiv.org/pdf/2405.11030
• Copy Paste: [[2405.11030]] The Unappreciated Role of Intent in Algorithmic Moderation of Social Media Content(https://arxiv.org/abs/2405.11030)
• Keywords: language model
• Abstract: As social media has become a predominant mode of communication globally, the rise of abusive content threatens to undermine civil discourse. Recognizing the critical nature of this issue, a significant body of research has been dedicated to developing language models that can detect various types of online abuse, e.g., hate speech, cyberbullying. However, there exists a notable disconnect between platform policies, which often consider the author's intention as a criterion for content moderation, and the current capabilities of detection models, which typically lack efforts to capture intent. This paper examines the role of intent in content moderation systems. We review state of the art detection models and benchmark training datasets for online abuse to assess their awareness and ability to capture intent. We propose strategic changes to the design and development of automated detection and moderation systems to improve alignment with ethical and policy conceptualizations of abuse.
• 摘要：随着社交媒体已成为全球主要的交流方式，辱骂内容的兴起可能会破坏公民话语。认识到这个问题的严重性，大量研究致力于开发可以检测各种类型的在线滥用行为的语言模型，例如仇恨言论、网络欺凌。然而，平台政策（通常将作者的意图作为内容审核的标准）与检测模型的当前功能（通常缺乏捕获意图的努力）之间存在明显的脱节。本文探讨了意图在内容审核系统中的作用。我们审查最先进的在线滥用检测模型和基准训练数据集，以评估他们捕获意图的意识和能力。我们建议对自动检测和审核系统的设计和开发进行战略性改变，以提高与滥用行为的道德和政策概念的一致性。

Title: CC-GPX: Extracting High-Quality Annotated Geospatial Data from Common Crawl

• Authors: Ilya Ilyankou, James Haworth, Stefano Cavazzi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.11039
• Pdf URL: https://arxiv.org/pdf/2405.11039
• Copy Paste: [[2405.11039]] CC-GPX: Extracting High-Quality Annotated Geospatial Data from Common Crawl(https://arxiv.org/abs/2405.11039)
• Keywords: language model
• Abstract: The Common Crawl (CC) corpus is the largest open web crawl dataset containing 9.5+ petabytes of data captured since 2008. The dataset is instrumental in training large language models, and as such it has been studied for (un)desirable content, and distilled for smaller, domain-specific datasets. However, to our knowledge, no research has been dedicated to using CC as a source of annotated geospatial data. In this paper, we introduce an efficient pipeline to extract annotated user-generated tracks from GPX files found in CC, and the resulting multimodal dataset with 1,416 pairings of human-written descriptions and MultiLineString vector data. The dataset can be used to study people's outdoor activity patterns, the way people talk about their outdoor experiences, and for developing trajectory generation or track annotation models.
• 摘要：Common Crawl (CC) 语料库是最大的开放网络爬行数据集，包含自 2008 年以来捕获的超过 9.5 PB 的数据。该数据集在训练大型语言模型方面发挥了重要作用，因此已针对（不）需要的内容进行了研究，并进行了提炼对于较小的、特定领域的数据集。然而，据我们所知，还没有研究致力于使用 CC 作为带注释的地理空间数据的来源。在本文中，我们介绍了一种高效的管道，用于从 CC 中找到的 GPX 文件中提取带注释的用户生成的轨迹，以及生成的包含 1,416 对人工编写的描述和 MultiLineString 矢量数据的多模态数据集。该数据集可用于研究人们的户外活动模式、人们谈论户外体验的方式，以及开发轨迹生成或轨迹注释模型。

Title: From Generalist to Specialist: Improving Large Language Models for Medical Physics Using ARCoT

• Authors: Jace Grandinetti, Rafe McBeth
• Subjects: cs.CL, physics.med-ph
• Abstract URL: https://arxiv.org/abs/2405.11040
• Pdf URL: https://arxiv.org/pdf/2405.11040
• Copy Paste: [[2405.11040]] From Generalist to Specialist: Improving Large Language Models for Medical Physics Using ARCoT(https://arxiv.org/abs/2405.11040)
• Keywords: language model, llm, hallucination, prompt, chain-of-thought
• Abstract: Large Language Models (LLMs) have achieved remarkable progress, yet their application in specialized fields, such as medical physics, remains challenging due to the need for domain-specific knowledge. This study introduces ARCoT (Adaptable Retrieval-based Chain of Thought), a framework designed to enhance the domain-specific accuracy of LLMs without requiring fine-tuning or extensive retraining. ARCoT integrates a retrieval mechanism to access relevant domain-specific information and employs step-back and chain-of-thought prompting techniques to guide the LLM's reasoning process, ensuring more accurate and context-aware responses. Benchmarking on a medical physics multiple-choice exam, our model outperformed standard LLMs and reported average human performance, demonstrating improvements of up to 68% and achieving a high score of 90%. This method reduces hallucinations and increases domain-specific performance. The versatility and model-agnostic nature of ARCoT make it easily adaptable to various domains, showcasing its significant potential for enhancing the accuracy and reliability of LLMs in specialized fields.
• 摘要：大型语言模型（LLM）已经取得了显着的进步，但由于需要特定领域的知识，它们在医学物理等专业领域的应用仍然具有挑战性。本研究引入了 ARCoT（基于自适应检索的思想链），这是一个旨在提高法学硕士特定领域准确性的框架，无需进行微调或广泛的再培训。 ARCoT 集成了检索机制来访问相关的特定领域信息，并采用后退和思维链提示技术来指导法学硕士的推理过程，确保更准确和上下文感知的响应。以医学物理多项选择题考试为基准，我们的模型优于标准法学硕士，并报告了人类的平均表现，表现出高达 68% 的改进，并取得了 90% 的高分。这种方法可以减少幻觉并提高特定领域的性能。 ARCoT 的多功能性和与模型无关的性质使其能够轻松适应各种领域，展示了其在提高专业领域法学硕士准确性和可靠性方面的巨大潜力。

Title: Prompt Exploration with Prompt Regression

• Authors: Michael Feffer, Ronald Xu, Yuekai Sun, Mikhail Yurochkin
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.11083
• Pdf URL: https://arxiv.org/pdf/2405.11083
• Copy Paste: [[2405.11083]] Prompt Exploration with Prompt Regression(https://arxiv.org/abs/2405.11083)
• Keywords: language model, llm, prompt
• Abstract: In the advent of democratized usage of large language models (LLMs), there is a growing desire to systematize LLM prompt creation and selection processes beyond iterative trial-and-error. Prior works majorly focus on searching the space of prompts without accounting for relations between prompt variations. Here we propose a framework, Prompt Exploration with Prompt Regression (PEPR), to predict the effect of prompt combinations given results for individual prompt elements as well as a simple method to select an effective prompt for a given use-case. We evaluate our approach with open-source LLMs of different sizes on several different tasks.
• 摘要：随着大型语言模型 (LLM) 民主化使用的出现，人们越来越希望将 LLM 提示创建和选择过程系统化，而不仅仅是迭代试错。先前的工作主要集中于搜索提示空间，而不考虑提示变化之间的关系。在这里，我们提出了一个框架，即提示回归的提示探索（PEPR），用于预测给定单个提示元素结果的提示组合的效果，以及为给定用例选择有效提示的简单方法。我们使用不同规模的开源法学硕士在几个不同的任务上评估我们的方法。

Title: Multilingual Substitution-based Word Sense Induction

• Authors: Denis Kokosinskii, Nikolay Arefyev
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.11086
• Pdf URL: https://arxiv.org/pdf/2405.11086
• Copy Paste: [[2405.11086]] Multilingual Substitution-based Word Sense Induction(https://arxiv.org/abs/2405.11086)
• Keywords: language model
• Abstract: Word Sense Induction (WSI) is the task of discovering senses of an ambiguous word by grouping usages of this word into clusters corresponding to these senses. Many approaches were proposed to solve WSI in English and a few other languages, but these approaches are not easily adaptable to new languages. We present multilingual substitution-based WSI methods that support any of 100 languages covered by the underlying multilingual language model with minimal to no adaptation required. Despite the multilingual capabilities, our methods perform on par with the existing monolingual approaches on popular English WSI datasets. At the same time, they will be most useful for lower-resourced languages which miss lexical resources available for English, thus, have higher demand for unsupervised methods like WSI.
• 摘要：词义归纳 (WSI) 是通过将单词的用法分组为与这些含义相对应的簇来发现歧义单词的含义的任务。人们提出了许多方法来解决英语和其他几种语言的 WSI，但这些方法不容易适应新语言。我们提出了基于多语言替换的 WSI 方法，该方法支持底层多语言语言模型涵盖的 100 种语言中的任何一种，并且几乎不需要任何调整。尽管具有多语言功能，我们的方法在流行的英语 WSI 数据集上的表现与现有的单语言方法相当。同时，它们对于资源匮乏的语言最有用，因为这些语言缺少英语可用的词汇资源，因此对 WSI 这样的无监督方法有更高的需求。

Title: Dynamic Embeddings with Task-Oriented prompting

• Authors: Allmin Balloccu, Jack Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.11117
• Pdf URL: https://arxiv.org/pdf/2405.11117
• Copy Paste: [[2405.11117]] Dynamic Embeddings with Task-Oriented prompting(https://arxiv.org/abs/2405.11117)
• Keywords: prompt
• Abstract: This paper introduces Dynamic Embeddings with Task-Oriented prompting (DETOT), a novel approach aimed at improving the adaptability and efficiency of machine learning models by implementing a flexible embedding layer. Unlike traditional static embeddings [14], DETOT dynamically adjusts embeddings based on task-specific requirements and performance feedback, optimizing input data representation for individual tasks [4]. This method enhances both accuracy and computational performance by tailoring the representation layer to meet the unique needs of each task. The structure of DETOT is detailed, highlighting its task-specific adaptation, continuous feedback loop, and mechanisms for preventing overfitting. Empirical evaluations demonstrate its superiority over existing methods.
• 摘要：本文介绍了具有面向任务提示的动态嵌入（DETOT），这是一种旨在通过实现灵活的嵌入层来提高机器学习模型的适应性和效率的新方法。与传统的静态嵌入[14]不同，DETOT根据特定任务的要求和性能反馈动态调整嵌入，优化各个任务的输入数据表示[4]。该方法通过定制表示层来满足每个任务的独特需求，从而提高了准确性和计算性能。 DETOT 的结构很详细，突出了其针对特定任务的适应、持续的反馈循环以及防止过度拟合的机制。实证评估表明其优于现有方法。

Title: LG AI Research & KAIST at EHRSQL 2024: Self-Training Large Language Models with Pseudo-Labeled Unanswerable Questions for a Reliable Text-to-SQL System on EHRs

• Authors: Yongrae Jo, Seongyun Lee, Minju Seo, Sung Ju Hwang, Moontae Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.11162
• Pdf URL: https://arxiv.org/pdf/2405.11162
• Copy Paste: [[2405.11162]] LG AI Research & KAIST at EHRSQL 2024: Self-Training Large Language Models with Pseudo-Labeled Unanswerable Questions for a Reliable Text-to-SQL System on EHRs(https://arxiv.org/abs/2405.11162)
• Keywords: language model
• Abstract: Text-to-SQL models are pivotal for making Electronic Health Records (EHRs) accessible to healthcare professionals without SQL knowledge. With the advancements in large language models, these systems have become more adept at translating complex questions into SQL queries. Nonetheless, the critical need for reliability in healthcare necessitates these models to accurately identify unanswerable questions or uncertain predictions, preventing misinformation. To address this problem, we present a self-training strategy using pseudo-labeled unanswerable questions to enhance the reliability of text-to-SQL models for EHRs. This approach includes a two-stage training process followed by a filtering method based on the token entropy and query execution. Our methodology's effectiveness is validated by our top performance in the EHRSQL 2024 shared task, showcasing the potential to improve healthcare decision-making through more reliable text-to-SQL systems.
• 摘要：文本到 SQL 模型对于让没有 SQL 知识的医疗保健专业人员访问电子健康记录 (EHR) 至关重要。随着大型语言模型的进步，这些系统在将复杂问题转换为 SQL 查询方面变得更加熟练。尽管如此，医疗保健对可靠性的迫切需求要求这些模型能够准确识别无法回答的问题或不确定的预测，从而防止错误信息。为了解决这个问题，我们提出了一种使用伪标记无法回答的问题的自训练策略来增强 EHR 文本到 SQL 模型的可靠性。这种方法包括一个两阶段的训练过程，然后是基于标记熵和查询执行的过滤方法。我们在 EHRSQL 2024 共享任务中的最佳表现验证了我们方法的有效性，展示了通过更可靠的文本到 SQL 系统改善医疗保健决策的潜力。

Title: Automating PTSD Diagnostics in Clinical Interviews: Leveraging Large Language Models for Trauma Assessments

• Authors: Sichang Tu, Abigail Powers, Natalie Merrill, Negar Fani, Sierra Carter, Stephen Doogan, Jinho D. Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.11178
• Pdf URL: https://arxiv.org/pdf/2405.11178
• Copy Paste: [[2405.11178]] Automating PTSD Diagnostics in Clinical Interviews: Leveraging Large Language Models for Trauma Assessments(https://arxiv.org/abs/2405.11178)
• Keywords: language model, gpt, llm
• Abstract: The shortage of clinical workforce presents significant challenges in mental healthcare, limiting access to formal diagnostics and services. We aim to tackle this shortage by integrating a customized large language model (LLM) into the workflow, thus promoting equity in mental healthcare for the general population. Although LLMs have showcased their capability in clinical decision-making, their adaptation to severe conditions like Post-traumatic Stress Disorder (PTSD) remains largely unexplored. Therefore, we collect 411 clinician-administered diagnostic interviews and devise a novel approach to obtain high-quality data. Moreover, we build a comprehensive framework to automate PTSD diagnostic assessments based on interview contents by leveraging two state-of-the-art LLMs, GPT-4 and Llama-2, with potential for broader clinical diagnoses. Our results illustrate strong promise for LLMs, tested on our dataset, to aid clinicians in diagnostic validation. To the best of our knowledge, this is the first AI system that fully automates assessments for mental illness based on clinician-administered interviews.
• 摘要：临床劳动力的短缺给精神卫生保健带来了重大挑战，限制了获得正式诊断和服务的机会。我们的目标是通过将定制的大语言模型（LLM）集成到工作流程中来解决这一短缺问题，从而促进普通大众心理保健的公平性。尽管法学硕士已经展示了他们在临床决策方面的能力，但他们对创伤后应激障碍（PTSD）等严重情况的适应能力在很大程度上仍未得到探索。因此，我们收集了 411 名临床医生进行的诊断访谈，并设计了一种新方法来获取高质量数据。此外，我们利用两个最先进的法学硕士（GPT-4 和 Llama-2）构建了一个全面的框架，根据访谈内容自动进行 PTSD 诊断评估，具有更广泛的临床诊断潜力。我们的结果表明，在我们的数据集上进行测试的法学硕士在帮助临床医生进行诊断验证方面有着强大的前景。据我们所知，这是第一个基于临床医生进行的访谈完全自动化精神疾病评估的人工智能系统。

Title: BrainStorm @ iREL at SMM4H 2024: Leveraging Translation and Topical Embeddings for Annotation Detection in Tweets

• Authors: Manav Chaudhary, Harshit Gupta, Vasudeva Varma
• Subjects: cs.CL, cs.SI
• Abstract URL: https://arxiv.org/abs/2405.11192
• Pdf URL: https://arxiv.org/pdf/2405.11192
• Copy Paste: [[2405.11192]] BrainStorm @ iREL at SMM4H 2024: Leveraging Translation and Topical Embeddings for Annotation Detection in Tweets(https://arxiv.org/abs/2405.11192)
• Keywords: llm, hallucination
• Abstract: The proliferation of LLMs in various NLP tasks has sparked debates regarding their reliability, particularly in annotation tasks where biases and hallucinations may arise. In this shared task, we address the challenge of distinguishing annotations made by LLMs from those made by human domain experts in the context of COVID-19 symptom detection from tweets in Latin American Spanish. This paper presents BrainStorm @ iREL's approach to the SMM4H 2024 Shared Task, leveraging the inherent topical information in tweets, we propose a novel approach to identify and classify annotations, aiming to enhance the trustworthiness of annotated data.
• 摘要：LLM 在各种 NLP 任务中的激增引发了关于其可靠性的争论，特别是在可能出现偏见和幻觉的注释任务中。在这个共享任务中，我们解决了在从拉丁美洲西班牙语推文中检测 COVID-19 症状的背景下区分 LLM 所做的注释与人类领域专家所做的注释的挑战。本文介绍了 BrainStorm @ iREL 对 SMM4H 2024 共享任务的方法，利用推文中固有的主题信息，我们提出了一种识别和分类注释的新方法，旨在提高注释数据的可信度。

Title: MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencing

• Authors: Siddhant Agarwal, Shivam Sharma, Preslav Nakov, Tanmoy Chakraborty
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2405.11215
• Pdf URL: https://arxiv.org/pdf/2405.11215
• Copy Paste: [[2405.11215]] MemeMQA: Multimodal Question Answering for Memes via Rationale-Based Inferencing(https://arxiv.org/abs/2405.11215)
• Keywords: llm
• Abstract: Memes have evolved as a prevalent medium for diverse communication, ranging from humour to propaganda. With the rising popularity of image-focused content, there is a growing need to explore its potential harm from different aspects. Previous studies have analyzed memes in closed settings - detecting harm, applying semantic labels, and offering natural language explanations. To extend this research, we introduce MemeMQA, a multimodal question-answering framework aiming to solicit accurate responses to structured questions while providing coherent explanations. We curate MemeMQACorpus, a new dataset featuring 1,880 questions related to 1,122 memes with corresponding answer-explanation pairs. We further propose ARSENAL, a novel two-stage multimodal framework that leverages the reasoning capabilities of LLMs to address MemeMQA. We benchmark MemeMQA using competitive baselines and demonstrate its superiority - ~18% enhanced answer prediction accuracy and distinct text generation lead across various metrics measuring lexical and semantic alignment over the best baseline. We analyze ARSENAL's robustness through diversification of question-set, confounder-based evaluation regarding MemeMQA's generalizability, and modality-specific assessment, enhancing our understanding of meme interpretation in the multimodal communication landscape.
• 摘要：模因已发展成为从幽默到宣传等多种交流的流行媒介。随着以图像为中心的内容越来越受欢迎，越来越需要从不同方面探讨其潜在危害。之前的研究分析了封闭环境中的模因——检测伤害、应用语义标签并提供自然语言解释。为了扩展这项研究，我们引入了 MemeMQA，这是一个多模式问答框架，旨在征求对结构化问题的准确回答，同时提供连贯的解释。我们策划了 MemeMQACorpus，这是一个新数据集，包含与 1,122 个模因相关的 1,880 个问题以及相应的答案-解释对。我们进一步提出了 ARSENAL，这是一种新颖的两阶段多模式框架，利用 LLM 的推理能力来解决 MemeMQA。我们使用竞争性基线对 MemeMQA 进行基准测试，并证明其优越性 - 答案预测准确性提高约 18%，并且在衡量词汇和语义对齐的各种指标中，不同的文本生成领先于最佳基线。我们通过问题集的多样化、关于 MemeMQA 的普遍性的基于混杂因素的评估以及特定于模态的评估来分析 ARSENAL 的稳健性，从而增强我们对多模态通信领域中模因解释的理解。

Title: Transformer based neural networks for emotion recognition in conversations

• Authors: Claudiu Creanga, Liviu P. Dinu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.11222
• Pdf URL: https://arxiv.org/pdf/2405.11222
• Copy Paste: [[2405.11222]] Transformer based neural networks for emotion recognition in conversations(https://arxiv.org/abs/2405.11222)
• Keywords: language model, prompt
• Abstract: This paper outlines the approach of the ISDS-NLP team in the SemEval 2024 Task 10: Emotion Discovery and Reasoning its Flip in Conversation (EDiReF). For Subtask 1 we obtained a weighted F1 score of 0.43 and placed 12 in the leaderboard. We investigate two distinct approaches: Masked Language Modeling (MLM) and Causal Language Modeling (CLM). For MLM, we employ pre-trained BERT-like models in a multilingual setting, fine-tuning them with a classifier to predict emotions. Experiments with varying input lengths, classifier architectures, and fine-tuning strategies demonstrate the effectiveness of this approach. Additionally, we utilize Mistral 7B Instruct V0.2, a state-of-the-art model, applying zero-shot and few-shot prompting techniques. Our findings indicate that while Mistral shows promise, MLMs currently outperform them in sentence-level emotion classification.
• 摘要：本文概述了 ISDS-NLP 团队在 SemEval 2024 任务 10：对话中的情绪发现和推理翻转 (EDiReF) 中的方法。对于子任务 1，我们获得了 0.43 的加权 F1 分数，并在排行榜上排名第 12。我们研究了两种不同的方法：掩码语言建模（MLM）和因果语言建模（CLM）。对于传销，我们在多语言环境中采用预先训练的类似 BERT 的模型，并使用分类器对其进行微调以预测情绪。不同输入长度、分类器架构和微调策略的实验证明了这种方法的有效性。此外，我们还利用最先进的模型 Mistral 7B Instruct V0.2，应用零样本和少样本提示技术。我们的研究结果表明，虽然 Mistral 显示出前景，但 MLM 目前在句子级情感分类方面优于它们。

Title: WisPerMed at "Discharge Me!": Advancing Text Generation in Healthcare with Large Language Models, Dynamic Expert Selection, and Priming Techniques on MIMIC-IV

• Authors: Hendrik Damm, Tabea M. G. Pakull, Bahadır Eryılmaz, Helmut Becker, Ahmad Idrissi-Yaghir, Henning Schäfer, Sergej Schultenkämper, Christoph M. Friedrich
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] WisPerMed at "Discharge Me!": Advancing Text Generation in Healthcare with Large Language Models, Dynamic Expert Selection, and Priming Techniques on MIMIC-IV(https://arxiv.org/abs/)
• Keywords: language model
• Abstract: This study aims to leverage state of the art language models to automate generating the "Brief Hospital Course" and "Discharge Instructions" sections of Discharge Summaries from the MIMIC-IV dataset, reducing clinicians' administrative workload. We investigate how automation can improve documentation accuracy, alleviate clinician burnout, and enhance operational efficacy in healthcare facilities. This research was conducted within our participation in the Shared Task Discharge Me! at BioNLP @ ACL 2024. Various strategies were employed, including few-shot learning, instruction tuning, and Dynamic Expert Selection (DES), to develop models capable of generating the required text sections. Notably, utilizing an additional clinical domain-specific dataset demonstrated substantial potential to enhance clinical language processing. The DES method, which optimizes the selection of text outputs from multiple predictions, proved to be especially effective. It achieved the highest overall score of 0.332 in the competition, surpassing single-model outputs. This finding suggests that advanced deep learning methods in combination with DES can effectively automate parts of electronic health record documentation. These advancements could enhance patient care by freeing clinician time for patient interactions. The integration of text selection strategies represents a promising avenue for further research.
• 摘要：本研究旨在利用最先进的语言模型从 MIMIC-IV 数据集中自动生成出院摘要的“医院简介”和“出院说明”部分，从而减少临床医生的管理工作量。我们研究自动化如何提高文档准确性、减轻临床医生的倦怠并提高医疗机构的运营效率。这项研究是在我们参与共享任务“Discharge Me！”的过程中进行的。 BioNLP @ ACL 2024。采用了各种策略，包括小样本学习、指令调整和动态专家选择 (DES)，来开发能够生成所需文本部分的模型。值得注意的是，利用额外的临床领域特定数据集证明了增强临床语言处理的巨大潜力。 DES 方法优化了多个预测中文本输出的选择，事实证明特别有效。它取得了比赛中最高的总分0.332，超越了单一模型的输出。这一发现表明，先进的深度学习方法与 DES 相结合可以有效地自动化部分电子健康记录文档。这些进步可以通过释放临床医生与患者互动的时间来增强患者护理。文本选择策略的整合代表了进一步研究的有希望的途径。

Title: Cross-Language Assessment of Mathematical Capability of ChatGPT

• Authors: Gargi Sathe, Aneesh Shamraj, Aditya Surve, Nahush Patil, Kumkum Saxena
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.11264
• Pdf URL: https://arxiv.org/pdf/2405.11264
• Copy Paste: [[2405.11264]] Cross-Language Assessment of Mathematical Capability of ChatGPT(https://arxiv.org/abs/2405.11264)
• Keywords: gpt, prompt, chat, chain-of-thought
• Abstract: This paper presents an evaluation of the mathematical capability of ChatGPT across diverse languages like Hindi, Gujarati, and Marathi. ChatGPT, based on GPT-3.5 by OpenAI, has garnered significant attention for its natural language understanding and generation abilities. However, its performance in solving mathematical problems across multiple natural languages remains a comparatively unexplored area, especially in regional Indian languages. In this paper, we explore those capabilities as well as using chain-of-thought prompting to figure out if it increases the accuracy of responses as much as it does in the English language and provide insights into the current limitations.
• 摘要：本文对 ChatGPT 跨印地语、古吉拉特语和马拉地语等多种语言的数学能力进行了评估。 ChatGPT 基于 OpenAI 的 GPT-3.5，因其自然语言理解和生成能力而受到广泛关注。然而，它在解决跨多种自然语言的数学问题方面的表现仍然是一个相对未经探索的领域，特别是在印度地方语言中。在本文中，我们探讨了这些功能，并使用思维链提示来弄清楚它是否能像英语一样提高回答的准确性，并提供对当前局限性的见解。

Title: EnviroExam: Benchmarking Environmental Science Knowledge of Large Language Models

• Authors: Yu Huang, Liang Guo, Wanqian Guo, Zhe Tao, Yang Lv, Zhihao Sun, Dongfang Zhao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.11265
• Pdf URL: https://arxiv.org/pdf/2405.11265
• Copy Paste: [[2405.11265]] EnviroExam: Benchmarking Environmental Science Knowledge of Large Language Models(https://arxiv.org/abs/2405.11265)
• Keywords: language model
• Abstract: In the field of environmental science, it is crucial to have robust evaluation metrics for large language models to ensure their efficacy and accuracy. We propose EnviroExam, a comprehensive evaluation method designed to assess the knowledge of large language models in the field of environmental science. EnviroExam is based on the curricula of top international universities, covering undergraduate, master's, and doctoral courses, and includes 936 questions across 42 core courses. By conducting 0-shot and 5-shot tests on 31 open-source large language models, EnviroExam reveals the performance differences among these models in the domain of environmental science and provides detailed evaluation standards. The results show that 61.3% of the models passed the 5-shot tests, while 48.39% passed the 0-shot tests. By introducing the coefficient of variation as an indicator, we evaluate the performance of mainstream open-source large language models in environmental science from multiple perspectives, providing effective criteria for selecting and fine-tuning language models in this field. Future research will involve constructing more domain-specific test sets using specialized environmental science textbooks to further enhance the accuracy and specificity of the evaluation.
• 摘要：在环境科学领域，为大型语言模型提供稳健的评估指标以确保其有效性和准确性至关重要。我们提出了EnviroExam，一种旨在评估环境科学领域大语言模型知识的综合评估方法。 EnviroExam基于国际顶尖大学课程，涵盖本科、硕士、博士课程，包含42门核心课程936道题。 EnviroExam通过对31个开源大语言模型进行0-shot和5-shot测试，揭示了这些模型在环境科学领域的性能差异，并提供了详细的评估标准。结果显示，61.3%的模型通过了5次测试，48.39%的模型通过了0次测试。通过引入变异系数作为指标，我们从多个角度评估环境科学中主流开源大语言模型的性能，为该领域的语言模型选择和微调提供有效标准。未来的研究将涉及使用专门的环境科学教科书构建更多特定领域的测试集，以进一步提高评估的准确性和特异性。

Title: MBIAS: Mitigating Bias in Large Language Models While Retaining Context

• Authors: Shaina Raza, Ananya Raval, Veronica Chatrath
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.11290
• Pdf URL: https://arxiv.org/pdf/2405.11290
• Copy Paste: [[2405.11290]] MBIAS: Mitigating Bias in Large Language Models While Retaining Context(https://arxiv.org/abs/2405.11290)
• Keywords: language model, llm
• Abstract: In addressing the critical need for safety in Large Language Models (LLMs), it is crucial to ensure that the outputs are not only safe but also retain their contextual accuracy. Many existing LLMs are safe fine-tuned either with safety demonstrations, or rely only on adversarial testing. While able to get safe outputs, they often risk losing contextual meaning as they mitigate bias and toxicity. In response, we present MBIAS, a LLM framework instruction fine-tuned on a custom dataset specifically designed for safety interventions. MBIAS aims to address the significant issues of bias and toxicity in LLMs generations that typically manifest as underrepresentation or negative portrayals across various demographics, including inappropriate linguistic mentions and biased content in social media. We experiment on MBIAS for safety interventions using various configurations, and demonstrate more than a 30\% reduction in overall bias and toxicity while successfully retaining key information. Additionally, a demographic analysis on an out-of-distribution test set confirms the robustness of our approach, with reductions in bias and toxicity exceeding 90\% across various demographics. The dataset and instruction fine-tuned MBIAS are made available to the research community at this https URL.
• 摘要：在满足大型语言模型 (LLM) 安全的关键需求时，确保输出不仅安全而且保持上下文准确性至关重要。许多现有的法学硕士要么通过安全演示进行安全微调，要么仅依赖对抗性测试。虽然能够获得安全的输出，但它们常常面临失去上下文意义的风险，因为它们减轻了偏见和毒性。作为回应，我们提出了 MBIAS，这是一种 LLM 框架指令，在专门为安全干预而设计的自定义数据集上进行了微调。 MBIAS 旨在解决法学硕士世代中存在的重大偏见和毒性问题，这些问题通常表现为不同人群中代表性不足或负面描述，包括不恰当的语言提及和社交媒体中的偏见内容。我们使用各种配置对 MBIAS 进行安全干预实验，并证明总体偏差和毒性降低了 30% 以上，同时成功保留了关键信息。此外，对分布外测试集的人口统计分析证实了我们方法的稳健性，在不同人口统计数据中，偏差和毒性的减少超过 90%。经过微调的 MBIAS 数据集和指令可通过此 https URL 向研究社区提供。

Title: Unveiling Key Aspects of Fine-Tuning in Sentence Embeddings: A Representation Rank Analysis

• Authors: Euna Jung, Jaeill Kim, Jungmin Ko, Jinwoo Park, Wonjong Rhee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.11297
• Pdf URL: https://arxiv.org/pdf/2405.11297
• Copy Paste: [[2405.11297]] Unveiling Key Aspects of Fine-Tuning in Sentence Embeddings: A Representation Rank Analysis(https://arxiv.org/abs/2405.11297)
• Keywords: language model
• Abstract: The latest advancements in unsupervised learning of sentence embeddings predominantly involve employing contrastive learning-based (CL-based) fine-tuning over pre-trained language models. In this study, we analyze the latest sentence embedding methods by adopting representation rank as the primary tool of analysis. We first define Phase 1 and Phase 2 of fine-tuning based on when representation rank peaks. Utilizing these phases, we conduct a thorough analysis and obtain essential findings across key aspects, including alignment and uniformity, linguistic abilities, and correlation between performance and rank. For instance, we find that the dynamics of the key aspects can undergo significant changes as fine-tuning transitions from Phase 1 to Phase 2. Based on these findings, we experiment with a rank reduction (RR) strategy that facilitates rapid and stable fine-tuning of the latest CL-based methods. Through empirical investigations, we showcase the efficacy of RR in enhancing the performance and stability of five state-of-the-art sentence embedding methods.
• 摘要：句子嵌入无监督学习的最新进展主要涉及对预训练语言模型采用基于对比学习（基于 CL）的微调。在本研究中，我们采用表示排序作为主要分析工具来分析最新的句子嵌入方法。我们首先根据表示排名达到峰值的时间定义微调的阶段 1 和阶段 2。利用这些阶段，我们进行彻底的分析，并获得关键方面的重要发现，包括一致性和一致性、语言能力以及表现和排名之间的相关性。例如，我们发现，随着从第一阶段到第二阶段的微调过渡，关键方面的动态可能会发生重大变化。基于这些发现，我们尝试了一种降级（RR）策略，该策略有助于快速稳定的微调调整最新的基于 CL 的方法。通过实证研究，我们展示了 RR 在提高五种最先进的句子嵌入方法的性能和稳定性方面的功效。

Title: Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Models

• Authors: Canshi Wei
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2405.11301
• Pdf URL: https://arxiv.org/pdf/2405.11301
• Copy Paste: [[2405.11301]] Enhancing Fine-Grained Image Classifications via Cascaded Vision Language Models(https://arxiv.org/abs/2405.11301)
• Keywords: language model
• Abstract: Fine-grained image classification, particularly in zero/few-shot scenarios, presents a significant challenge for vision-language models (VLMs), such as CLIP. These models often struggle with the nuanced task of distinguishing between semantically similar classes due to limitations in their pre-trained recipe, which lacks supervision signals for fine-grained categorization. This paper introduces CascadeVLM, an innovative framework that overcomes the constraints of previous CLIP-based methods by effectively leveraging the granular knowledge encapsulated within large vision-language models (LVLMs). Experiments across various fine-grained image datasets demonstrate that CascadeVLM significantly outperforms existing models, specifically on the Stanford Cars dataset, achieving an impressive 85.6% zero-shot accuracy. Performance gain analysis validates that LVLMs produce more accurate predictions for challenging images that CLIPs are uncertain about, bringing the overall accuracy boost. Our framework sheds light on a holistic integration of VLMs and LVLMs for effective and efficient fine-grained image classification.
• 摘要：细粒度图像分类，特别是在零/少样本场景中，对视觉语言模型 (VLM)（例如 CLIP）提出了重大挑战。由于预训练方法的限制，这些模型经常难以区分语义相似的类别，而这些方法缺乏细粒度分类的监督信号。本文介绍了 CascadeVLM，这是一种创新框架，它通过有效利用大型视觉语言模型 (LVLM) 中封装的粒度知识来克服先前基于 CLIP 的方法的限制。在各种细粒度图像数据集上进行的实验表明，CascadeVLM 的性能显着优于现有模型，特别是在斯坦福汽车数据集上，实现了令人印象深刻的 85.6% 的零样本精度。性能增益分析验证了 LVLM 可以对 CLIP 不确定的具有挑战性的图像产生更准确的预测，从而提高整体准确性。我们的框架揭示了 VLM 和 LVLM 的整体集成，以实现有效且高效的细粒度图像分类。

Title: Large Language Models Lack Understanding of Character Composition of Words

• Authors: Andrew Shin, Kunitake Kaneko
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.11357
• Pdf URL: https://arxiv.org/pdf/2405.11357
• Copy Paste: [[2405.11357]] Large Language Models Lack Understanding of Character Composition of Words(https://arxiv.org/abs/2405.11357)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have demonstrated remarkable performances on a wide range of natural language tasks. Yet, LLMs' successes have been largely restricted to tasks concerning words, sentences, or documents, and it remains questionable how much they understand the minimal units of text, namely characters. In this paper, we examine contemporary LLMs regarding their ability to understand character composition of words, and show that most of them fail to reliably carry out even the simple tasks that can be handled by humans with perfection. We analyze their behaviors with comparison to token level performances, and discuss the potential directions for future research.
• 摘要：大型语言模型（LLM）在广泛的自然语言任务中表现出了卓越的性能。然而，法学硕士的成功在很大程度上仅限于有关单词、句子或文档的任务，并且他们对文本的最小单位（即字符）的理解程度仍然存在疑问。在本文中，我们研究了当代法学硕士理解单词字符构成的能力，结果表明，他们中的大多数人甚至无法可靠地执行人类可以完美处理的简单任务。我们通过与代币级别的表现进行比较来分析他们的行为，并讨论未来研究的潜在方向。

Title: MapCoder: Multi-Agent Code Generation for Competitive Problem Solving

• Authors: Md. Ashraful Islam, Mohammed Eunus Ali, Md Rizwan Parvez
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.11403
• Pdf URL: https://arxiv.org/pdf/2405.11403
• Copy Paste: [[2405.11403]] MapCoder: Multi-Agent Code Generation for Competitive Problem Solving(https://arxiv.org/abs/2405.11403)
• Keywords: language model, llm, prompt, agent
• Abstract: Code synthesis, which requires a deep understanding of complex natural language problem descriptions, generation of code instructions for complex algorithms and data structures, and the successful execution of comprehensive unit tests, presents a significant challenge. While large language models (LLMs) demonstrate impressive proficiency in natural language processing, their performance in code generation tasks remains limited. In this paper, we introduce a new approach to code generation tasks leveraging multi-agent prompting that uniquely replicates the full cycle of program synthesis as observed in human developers. Our framework, MapCoder, consists of four LLM agents specifically designed to emulate the stages of this cycle: recalling relevant examples, planning, code generation, and debugging. After conducting thorough experiments, with multiple LLM ablations and analyses across eight challenging competitive problem-solving and program synthesis benchmarks, MapCoder showcases remarkable code generation capabilities, achieving new state-of-the-art results (pass@1) on HumanEval (93.9%), MBPP (83.1%), APPS (22.0%), CodeContests (28.5%), and xCodeEval (45.3%). Moreover, our method consistently delivers superior performance across various programming languages and varying problem difficulties. We open-source our framework at this https URL.
• 摘要：代码合成需要深入理解复杂的自然语言问题描述、生成复杂算法和数据结构的代码指令以及成功执行全面的单元测试，这提出了重大挑战。虽然大型语言模型 (LLM) 在自然语言处理方面表现出令人印象深刻的熟练程度，但它们在代码生成任务中的性能仍然有限。在本文中，我们介绍了一种利用多代理提示来执行代码生成任务的新方法，该方法独特地复制了人类开发人员观察到的程序合成的完整周期。我们的框架 MapCoder 由四个 LLM 代理组成，专门用于模拟该周期的各个阶段：回忆相关示例、规划、代码生成和调试。经过彻底的实验，通过对八个具有挑战性的竞争性问题解决和程序综合基准的多个 LLM 消融和分析，MapCoder 展示了卓越的代码生成能力，在 HumanEval 上取得了新的最先进的结果 (pass@1) (93.9%) ）、MBPP (83.1%)、APPS (22.0%)、CodeContests (28.5%) 和 xCodeEval (45.3%)。此外，我们的方法在各种编程语言和不同的问题难度上始终如一地提供卓越的性能。我们在此 https URL 开源我们的框架。

Title: Can Public LLMs be used for Self-Diagnosis of Medical Conditions ?

• Authors: Nikil Sharan Prabahar Balasubramanian, Sagnik Dakshit
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Can Public LLMs be used for Self-Diagnosis of Medical Conditions ?(https://arxiv.org/abs/)
• Keywords: language model, gpt, llm, prompt, retrieval augmented generation
• Abstract: The advancements in the development of Large Language Models have evolved as a transformative paradigm in conversational tasks which has led to its integration in the critical domain of healthcare. With LLMs becoming widely popular and their public access through open-source models, there is a need to investigate their potential and limitations. One such critical task where LLMs are applied but require a deeper understanding is that of self-diagnosis of medical conditions in the interest of public health. The widespread integration of Gemini with Google search, GPT-4.0 with Bing search, has led to shift in trend of self-diagnosis from search engine LLMs. In this paper, we prepare a prompt engineered dataset of 10000 samples and test the performance on the general task of self-diagnosis. We compare the performance of GPT-4.0 and Gemini model on the task of self-diagnosis and record accuracies of 63.07% and 6.01% respectively. We also discuss the challenges, limitations, and potential of both Gemini and GPT-4.0 for the task of self-diagnosis to facilitate future research and towards the broader impact of general public knowledge. Furthermore, we demonstrate the potential and improvement in performance for the task of self-diagnosis using Retrieval Augmented Generation.
• 摘要：大型语言模型开发的进步已经发展成为对话任务中的变革范式，这导致其集成到医疗保健的关键领域。随着法学硕士越来越受欢迎，并且通过开源模型向公众开放，有必要研究它们的潜力和局限性。应用法学硕士但需要更深入理解的一项关键任务是为了公共卫生利益而对医疗状况进行自我诊断。 Gemini 与 Google 搜索、GPT-4.0 与 Bing 搜索的广泛集成，导致了搜索引擎法学硕士自我诊断趋势的转变。在本文中，我们准备了 10000 个样本的即时工程数据集，并测试了自诊断一般任务的性能。我们比较了 GPT-4.0 和 Gemini 模型在自诊断任务上的表现，分别记录了 63.07% 和 6.01% 的准确率。我们还讨论了 Gemini 和 GPT-4.0 在自我诊断任务方面的挑战、局限性和潜力，以促进未来的研究并扩大公众知识的影响。此外，我们展示了使用检索增强生成进行自我诊断任务的潜力和性能改进。

Title: Large Language Models are Biased Reinforcement Learners

• Authors: William M. Hayes, Nicolas Yax, Stefano Palminteri
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.11422
• Pdf URL: https://arxiv.org/pdf/2405.11422
• Copy Paste: [[2405.11422]] Large Language Models are Biased Reinforcement Learners(https://arxiv.org/abs/2405.11422)
• Keywords: language model, llm, prompt, agent
• Abstract: In-context learning enables large language models (LLMs) to perform a variety of tasks, including learning to make reward-maximizing choices in simple bandit tasks. Given their potential use as (autonomous) decision-making agents, it is important to understand how these models perform such reinforcement learning (RL) tasks and the extent to which they are susceptible to biases. Motivated by the fact that, in humans, it has been widely documented that the value of an outcome depends on how it compares to other local outcomes, the present study focuses on whether similar value encoding biases apply to how LLMs encode rewarding outcomes. Results from experiments with multiple bandit tasks and models show that LLMs exhibit behavioral signatures of a relative value bias. Adding explicit outcome comparisons to the prompt produces opposing effects on performance, enhancing maximization in trained choice sets but impairing generalization to new choice sets. Computational cognitive modeling reveals that LLM behavior is well-described by a simple RL algorithm that incorporates relative values at the outcome encoding stage. Lastly, we present preliminary evidence that the observed biases are not limited to fine-tuned LLMs, and that relative value processing is detectable in the final hidden layer activations of a raw, pretrained model. These findings have important implications for the use of LLMs in decision-making applications.
• 摘要：上下文学习使大型语言模型 (LLM) 能够执行各种任务，包括学习在简单的老虎机任务中做出奖励最大化的选择。考虑到它们作为（自主）决策代理的潜在用途，了解这些模型如何执行此类强化学习（RL）任务以及它们容易受到偏差的程度非常重要。受人类广泛记录的事实的启发，结果的价值取决于它与其他本地结果的比较，本研究的重点是类似的价值编码偏差是否适用于法学硕士如何编码奖励结果。多个老虎机任务和模型的实验结果表明，法学硕士表现出相对价值偏差的行为特征。在提示中添加明确的结果比较会对性能产生相反的影响，增强训练选择集的最大化，但会损害对新选择集的泛化。计算认知模型表明，LLM 行为可以通过简单的 RL 算法很好地描述，该算法在结果编码阶段结合了相对值。最后，我们提出了初步证据，表明观察到的偏差并不限于微调的 LLM，并且在原始预训练模型的最终隐藏层激活中可以检测到相对值处理。这些发现对于法学硕士在决策应用中的使用具有重要意义。

Title: MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation

• Authors: Jianbo Dai, Jianqiao Lu, Yunlong Feng, Rongju Ruan, Ming Cheng, Haochen Tan, Zhijiang Guo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.11430
• Pdf URL: https://arxiv.org/pdf/2405.11430
• Copy Paste: [[2405.11430]] MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation(https://arxiv.org/abs/2405.11430)
• Keywords: language model, gpt, llm
• Abstract: Recent advancements in large language models (LLMs) have greatly improved code generation, specifically at the function level. For instance, GPT-4 has achieved an 88.4% pass rate on HumanEval. However, this draws into question the adequacy of existing benchmarks in thoroughly assessing function-level code generation capabilities. Our study analyzed two common benchmarks, HumanEval and MBPP, and found that these might not thoroughly evaluate LLMs' code generation capacities due to limitations in quality, difficulty, and granularity. To resolve this, we introduce the Mostly Hard Python Problems (MHPP) dataset, consisting of 140 unique human-curated problems. By focusing on the combination of natural language and code reasoning, MHPP gauges LLMs' abilities to comprehend specifications and restrictions, engage in multi-step reasoning, and apply coding knowledge effectively. Initial evaluations of 22 LLMs using MHPP showed many high-performing models on HumanEval failed to achieve similar success on MHPP. Moreover, MHPP highlighted various previously undiscovered limitations within various LLMs, leading us to believe that it could pave the way for a better understanding of LLMs' capabilities and limitations. Dataset and code are available at this https URL.
• 摘要：大型语言模型 (LLM) 的最新进展极大地改进了代码生成，特别是在函数级别。例如，GPT-4 在 HumanEval 上的通过率达到了 88.4%。然而，这引发了对现有基准在彻底评估函数级代码生成能力方面的充分性的质疑。我们的研究分析了两个常见的基准测试：HumanEval 和 MBPP，发现由于质量、难度和粒度的限制，这些基准测试可能无法彻底评估 LLM 的代码生成能力。为了解决这个问题，我们引入了 Mostly Hard Python Problems (MHPP) 数据集，其中包含 140 个独特的人工问题。 MHPP 注重自然语言和代码推理的结合，衡量法学硕士理解规范和限制、进行多步推理以及有效应用编码知识的能力。使用 MHPP 对 22 个法学硕士进行的初步评估表明，HumanEval 上的许多高性能模型未能在 MHPP 上取得类似的成功。此外，MHPP 强调了各种法学硕士之前未发现的局限性，使我们相信它可以为更好地理解法学硕士的能力和局限性铺平道路。数据集和代码可从此 https URL 获取。

Title: MAML-en-LLM: Model Agnostic Meta-Training of LLMs for Improved In-Context Learning

• Authors: Sanchit Sinha, Yuguang Yue, Victor Soto, Mayank Kulkarni, Jianhua Lu, Aidong Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.11446
• Pdf URL: https://arxiv.org/pdf/2405.11446
• Copy Paste: [[2405.11446]] MAML-en-LLM: Model Agnostic Meta-Training of LLMs for Improved In-Context Learning(https://arxiv.org/abs/2405.11446)
• Keywords: language model, llm
• Abstract: Adapting large language models (LLMs) to unseen tasks with in-context training samples without fine-tuning remains an important research problem. To learn a robust LLM that adapts well to unseen tasks, multiple meta-training approaches have been proposed such as MetaICL and MetaICT, which involve meta-training pre-trained LLMs on a wide variety of diverse tasks. These meta-training approaches essentially perform in-context multi-task fine-tuning and evaluate on a disjointed test set of tasks. Even though they achieve impressive performance, their goal is never to compute a truly general set of parameters. In this paper, we propose MAML-en-LLM, a novel method for meta-training LLMs, which can learn truly generalizable parameters that not only perform well on disjointed tasks but also adapts to unseen tasks. We see an average increase of 2% on unseen domains in the performance while a massive 4% improvement on adaptation performance. Furthermore, we demonstrate that MAML-en-LLM outperforms baselines in settings with limited amount of training data on both seen and unseen domains by an average of 2%. Finally, we discuss the effects of type of tasks, optimizers and task complexity, an avenue barely explored in meta-training literature. Exhaustive experiments across 7 task settings along with two data settings demonstrate that models trained with MAML-en-LLM outperform SOTA meta-training approaches.
• 摘要：在不进行微调的情况下，使用上下文训练样本使大型语言模型（LLM）适应看不见的任务仍然是一个重要的研究问题。为了学习一个能够很好地适应未见过的任务的强大的法学硕士，人们提出了多种元训练方法，例如 MetaICL 和 MetaICT，其中涉及对各种不同任务的预训练法学硕士进行元训练。这些元训练方法本质上是执行上下文中的多任务微调，并对脱节的测试任务集进行评估。尽管他们取得了令人印象深刻的性能，但他们的目标从来不是计算一组真正通用的参数。在本文中，我们提出了 MAML-en-LLM，这是一种用于元训练 LLM 的新方法，它可以学习真正可泛化的参数，这些参数不仅在脱节任务上表现良好，而且还能适应未见过的任务。我们发现未见过的领域的性能平均提高了 2%，而适应性能则大幅提高了 4%。此外，我们证明，在可见领域和未见领域的训练数据量有限的情况下，MAML-en-LLM 的性能平均优于基线 2%。最后，我们讨论任务类型、优化器和任务复杂性的影响，这是元训练文献中很少探索的途径。跨 7 种任务设置和两种数据设置的详尽实验表明，使用 MAML-en-LLM 训练的模型优于 SOTA 元训练方法。

Title: Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion

• Authors: Pengxiang Lan, Enneng Yang, Yuting Liu, Guibing Guo, Linying Jiang, Jianzhe Zhao, Xingwei Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.11464
• Pdf URL: https://arxiv.org/pdf/2405.11464
• Copy Paste: [[2405.11464]] Efficient Prompt Tuning by Multi-Space Projection and Prompt Fusion(https://arxiv.org/abs/2405.11464)
• Keywords: language model, prompt
• Abstract: Prompt tuning is a promising method to fine-tune a pre-trained language model without retraining its large-scale parameters. Instead, it attaches a soft prompt to the input text, whereby downstream tasks can be well adapted by merely learning the embeddings of prompt tokens. Nevertheless, existing methods still suffer from two challenges: (i) they are hard to balance accuracy and efficiency. A longer (shorter) soft prompt generally leads to a better (worse) accuracy but at the cost of more (less) training time. (ii) The performance may not be consistent when adapting to different downstream tasks. We attribute it to the same embedding space but responsible for different requirements of downstream tasks. To address these issues, we propose an Efficient Prompt Tuning method (EPT) by multi-space projection and prompt fusion. Specifically, it decomposes a given soft prompt into a shorter prompt and two low-rank matrices, whereby the number of parameters is greatly reduced as well as the training time. The accuracy is also enhanced by leveraging low-rank matrices and the short prompt as additional knowledge sources to enrich the semantics of the original short prompt. In addition, we project the soft prompt into multiple subspaces to improve the performance consistency, and then adaptively learn the combination weights of different spaces through a gating network. Experimental experiments on 13 natural language processing downstream tasks show that our method significantly and consistently outperforms 11 comparison methods with the relative percentage of improvements up to 28.8%, and training time decreased by 14%.
• 摘要：快速调优是一种很有前途的方法，可以微调预训练的语言模型，而无需重新训练其大规模参数。相反，它将软提示附加到输入文本中，从而仅通过学习提示标记的嵌入就可以很好地适应下游任务。然而，现有的方法仍然面临两个挑战：（i）它们很难平衡准确性和效率。较长（较短）的软提示通常会带来更好（更差）的准确性，但代价是更多（更少）的训练时间。 (ii) 适应不同的下游任务时，性能可能不一致。我们将其归因于相同的嵌入空间，但负责下游任务的不同要求。为了解决这些问题，我们提出了一种通过多空间投影和提示融合的高效提示调整方法（EPT）。具体来说，它将给定的软提示分解为较短的提示和两个低秩矩阵，从而大大减少了参数数量和训练时间。通过利用低秩矩阵和简短提示作为附加知识源来丰富原始简短提示的语义，还可以提高准确性。此外，我们将软提示投影到多个子空间中以提高性能一致性，然后通过门网络自适应地学习不同空间的组合权重。对 13 个自然语言处理下游任务的实验表明，我们的方法明显且一致地优于 11 个比较方法，相对改进百分比高达 28.8%，训练时间减少了 14%。

Title: Effective In-Context Example Selection through Data Compression

• Authors: Zhongxiang Sun, Kepu Zhang, Haoyu Wang, Xiao Zhang, Jun Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.11465
• Pdf URL: https://arxiv.org/pdf/2405.11465
• Copy Paste: [[2405.11465]] Effective In-Context Example Selection through Data Compression(https://arxiv.org/abs/2405.11465)
• Keywords: language model
• Abstract: In-context learning has been extensively validated in large language models. However, the mechanism and selection strategy for in-context example selection, which is a crucial ingredient in this approach, lacks systematic and in-depth research. In this paper, we propose a data compression approach to the selection of in-context examples. We introduce a two-stage method that can effectively choose relevant examples and retain sufficient information about the training dataset within the in-context examples. Our method shows a significant improvement of an average of 5.90% across five different real-world datasets using four language models.
• 摘要：上下文学习已在大型语言模型中得到广泛验证。然而，作为该方法的关键组成部分的上下文实例选择的机制和选择策略缺乏系统和深入的研究。在本文中，我们提出了一种数据压缩方法来选择上下文中的示例。我们引入了一种两阶段方法，可以有效地选择相关示例并在上下文示例中保留有关训练数据集的足够信息。我们的方法显示，使用四种语言模型在五个不同的现实世界数据集中平均显着提高了 5.90%。

Title: Simple-Sampling and Hard-Mixup with Prototypes to Rebalance Contrastive Learning for Text Classification

• Authors: Mengyu Li, Yonghao Liu, Fausto Giunchiglia, Xiaoyue Feng, Renchu Guan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.11524
• Pdf URL: https://arxiv.org/pdf/2405.11524
• Copy Paste: [[2405.11524]] Simple-Sampling and Hard-Mixup with Prototypes to Rebalance Contrastive Learning for Text Classification(https://arxiv.org/abs/2405.11524)
• Keywords: language model
• Abstract: Text classification is a crucial and fundamental task in natural language processing. Compared with the previous learning paradigm of pre-training and fine-tuning by cross entropy loss, the recently proposed supervised contrastive learning approach has received tremendous attention due to its powerful feature learning capability and robustness. Although several studies have incorporated this technique for text classification, some limitations remain. First, many text datasets are imbalanced, and the learning mechanism of supervised contrastive learning is sensitive to data imbalance, which may harm the model performance. Moreover, these models leverage separate classification branch with cross entropy and supervised contrastive learning branch without explicit mutual guidance. To this end, we propose a novel model named SharpReCL for imbalanced text classification tasks. First, we obtain the prototype vector of each class in the balanced classification branch to act as a representation of each class. Then, by further explicitly leveraging the prototype vectors, we construct a proper and sufficient target sample set with the same size for each class to perform the supervised contrastive learning procedure. The empirical results show the effectiveness of our model, which even outperforms popular large language models across several datasets.
• 摘要：文本分类是自然语言处理中至关重要的基础任务。与之前通过交叉熵损失进行预训练和微调的学习范式相比，最近提出的监督对比学习方法由于其强大的特征学习能力和鲁棒性而受到了极大的关注。尽管一些研究已将这种技术纳入文本分类，但仍然存在一些局限性。首先，许多文本数据集是不平衡的，监督对比学习的学习机制对数据不平衡很敏感，这可能会损害模型性能。此外，这些模型利用具有交叉熵的单独分类分支和有监督对比学习分支，而没有明确的相互指导。为此，我们提出了一种名为 SharpReCL 的新颖模型，用于不平衡的文本分类任务。首先，我们获得平衡分类分支中每个类的原型向量，作为每个类的表示。然后，通过进一步明确地利用原型向量，我们为每个类别构建适当且足够的具有相同大小的目标样本集来执行监督对比学习过程。实证结果表明了我们的模型的有效性，它甚至在多个数据集上优于流行的大型语言模型。

Title: DaVinci at SemEval-2024 Task 9: Few-shot prompting GPT-3.5 for Unconventional Reasoning

• Authors: Suyash Vardhan Mathur, Akshett Rai Jindal, Manish Shrivastava
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.11559
• Pdf URL: https://arxiv.org/pdf/2405.11559
• Copy Paste: [[2405.11559]] DaVinci at SemEval-2024 Task 9: Few-shot prompting GPT-3.5 for Unconventional Reasoning(https://arxiv.org/abs/2405.11559)
• Keywords: gpt, prompt
• Abstract: While significant work has been done in the field of NLP on vertical thinking, which involves primarily logical thinking, little work has been done towards lateral thinking, which involves looking at problems from an unconventional perspective and defying existing conceptions and notions. Towards this direction, SemEval 2024 introduces the task of BRAINTEASER, which involves two types of questions -- Sentence Puzzles and Word Puzzles that defy conventional common-sense reasoning and constraints. In this paper, we tackle both types of questions using few-shot prompting on GPT-3.5 and gain insights regarding the difference in the nature of the two types. Our prompting strategy placed us 26th on the leaderboard for the Sentence Puzzle and 15th on the Word Puzzle task.
• 摘要：虽然 NLP 领域在垂直思维（主要涉及逻辑思维）方面已经做了重要的工作，但在横向思维（涉及从非常规的角度看待问题并挑战现有的观念和观念）方面却做得很少。朝着这个方向，SemEval 2024 引入了 BRAINTEASER 任务，其中涉及两种类型的问题——句子谜题和单词谜题，这些问题挑战了传统的常识推理和约束。在本文中，我们使用 GPT-3.5 上的小样本提示来解决这两种类型的问题，并深入了解这两种类型的本质差异。我们的提示策略使我们在句子谜题排行榜上排名第 26 位，在单词谜题排行榜上排名第 15 位。

Title: A Multi-Perspective Analysis of Memorization in Large Language Models

• Authors: Bowen Chen, Namgi Han, Yusuke Miyao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.11577
• Pdf URL: https://arxiv.org/pdf/2405.11577
• Copy Paste: [[2405.11577]] A Multi-Perspective Analysis of Memorization in Large Language Models(https://arxiv.org/abs/2405.11577)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs), trained on massive corpora with billions of parameters, show unprecedented performance in various fields. Though surprised by their excellent performances, researchers also noticed some special behaviors of those LLMs. One of those behaviors is memorization, in which LLMs can generate the same content used to train them. Though previous research has discussed memorization, the memorization of LLMs still lacks explanation, especially the cause of memorization and the dynamics of generating them. In this research, we comprehensively discussed memorization from various perspectives and extended the discussion scope to not only just the memorized content but also less and unmemorized content. Through various studies, we found that: (1) Through experiments, we revealed the relation of memorization between model size, continuation size, and context size. Further, we showed how unmemorized sentences transition to memorized sentences. (2) Through embedding analysis, we showed the distribution and decoding dynamics across model size in embedding space for sentences with different memorization scores. The n-gram statistics analysis presents d (3) An analysis over n-gram and entropy decoding dynamics discovered a boundary effect when the model starts to generate memorized sentences or unmemorized sentences. (4)We trained a Transformer model to predict the memorization of different models, showing that it is possible to predict memorizations by context.
• 摘要：大型语言模型（LLM）在具有数十亿参数的海量语料库上进行训练，在各个领域表现出了前所未有的性能。尽管对他们的出色表现感到惊讶，研究人员也注意到这些法学硕士的一些特殊行为。其中一种行为是记忆，法学硕士可以生成用于训练他们的相同内容。尽管之前的研究已经讨论了记忆，但法学硕士的记忆仍然缺乏解释，特别是记忆的原因和产生记忆的动力。在本研究中，我们从各个角度对记忆进行了全面的讨论，并将讨论范围不仅限于记忆的内容，还包括较少的和未记忆的内容。通过各种研究，我们发现：（1）通过实验，我们揭示了模型大小、延续大小和上下文大小之间的记忆关系。此外，我们还展示了未记忆的句子如何转变为记忆的句子。 （2）通过嵌入分析，我们显示了具有不同记忆分数的句子在嵌入空间中跨模型大小的分布和解码动态。 n-gram 统计分析提出 d (3) 对 n-gram 和熵解码动力学的分析发现，当模型开始生成已记忆句子或未记忆句子时，存在边界效应。 (4)我们训练了一个 Transformer 模型来预测不同模型的记忆，表明通过上下文预测记忆是可能的。

Title: Exploring the Capabilities of Prompted Large Language Models in Educational and Assessment Applications

• Authors: Subhankar Maity, Aniket Deroy, Sudeshna Sarkar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.11579
• Pdf URL: https://arxiv.org/pdf/2405.11579
• Copy Paste: [[2405.11579]] Exploring the Capabilities of Prompted Large Language Models in Educational and Assessment Applications(https://arxiv.org/abs/2405.11579)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: In the era of generative artificial intelligence (AI), the fusion of large language models (LLMs) offers unprecedented opportunities for innovation in the field of modern education. We embark on an exploration of prompted LLMs within the context of educational and assessment applications to uncover their potential. Through a series of carefully crafted research questions, we investigate the effectiveness of prompt-based techniques in generating open-ended questions from school-level textbooks, assess their efficiency in generating open-ended questions from undergraduate-level technical textbooks, and explore the feasibility of employing a chain-of-thought inspired multi-stage prompting approach for language-agnostic multiple-choice question (MCQ) generation. Additionally, we evaluate the ability of prompted LLMs for language learning, exemplified through a case study in the low-resource Indian language Bengali, to explain Bengali grammatical errors. We also evaluate the potential of prompted LLMs to assess human resource (HR) spoken interview transcripts. By juxtaposing the capabilities of LLMs with those of human experts across various educational tasks and domains, our aim is to shed light on the potential and limitations of LLMs in reshaping educational practices.
• 摘要：在生成人工智能（AI）时代，大语言模型（LLM）的融合为现代教育领域的创新提供了前所未有的机遇。我们着手在教育和评估应用的背景下探索提示法学硕士，以发掘其潜力。通过一系列精心设计的研究问题，我们研究了基于提示的技术在从校本教科书中生成开放式问题的有效性，评估其从本科级技术教科书中生成开放式问题的效率，并探讨了可行性采用思想链启发的多阶段提示方法来生成与语言无关的多项选择问题（MCQ）。此外，我们还评估了法学硕士的语言学习能力，以资源匮乏的印度孟加拉语为例，解释孟加拉语语法错误。我们还评估了法学硕士评估人力资源 (HR) 口语面试笔录的潜力。通过将法学硕士的能力与跨不同教育任务和领域的人类专家的能力进行比较，我们的目标是揭示法学硕士在重塑教育实践方面的潜力和局限性。

Title: Language Reconstruction with Brain Predictive Coding from fMRI Data

• Authors: Congchi Yin, Ziyi Ye, Piji Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.11597
• Pdf URL: https://arxiv.org/pdf/2405.11597
• Copy Paste: [[2405.11597]] Language Reconstruction with Brain Predictive Coding from fMRI Data(https://arxiv.org/abs/2405.11597)
• Keywords: language model
• Abstract: Many recent studies have shown that the perception of speech can be decoded from brain signals and subsequently reconstructed as continuous language. However, there is a lack of neurological basis for how the semantic information embedded within brain signals can be used more effectively to guide language reconstruction. The theory of predictive coding suggests that human brain naturally engages in continuously predicting future word representations that span multiple timescales. This implies that the decoding of brain signals could potentially be associated with a predictable future. To explore the predictive coding theory within the context of language reconstruction, this paper proposes a novel model \textsc{PredFT} for jointly modeling neural decoding and brain prediction. It consists of a main decoding network for language reconstruction and a side network for predictive coding. The side network obtains brain predictive coding representation from related brain regions of interest with a multi-head self-attention module. This representation is fused into the main decoding network with cross-attention to facilitate the language models' generation process. Experiments are conducted on the largest naturalistic language comprehension fMRI dataset Narratives. \textsc{PredFT} achieves current state-of-the-art decoding performance with a maximum BLEU-1 score of $27.8\%$.
• 摘要：最近的许多研究表明，语音感知可以从大脑信号中解码，然后重建为连续语言。然而，如何更有效地利用大脑信号中嵌入的语义信息来指导语言重建，缺乏神经学基础。预测编码理论表明，人类大脑自然地持续预测跨越多个时间尺度的未来单词表示。这意味着大脑信号的解码可能与可预测的未来有关。为了探索语言重建背景下的预测编码理论，本文提出了一种新颖的模型 \textsc{PredFT} 来联合建模神经解码和大脑预测。它由用于语言重建的主解码网络和用于预测编码的侧网络组成。侧面网络通过多头自注意力模块从相关的大脑感兴趣区域获取大脑预测编码表示。这种表示形式通过交叉注意力融合到主解码网络中，以促进语言模型的生成过程。实验是在最大的自然语言理解 fMRI 数据集 Narratives 上进行的。 \textsc{PredFT} 实现了当前最先进的解码性能，最大 BLEU-1 得分为 $27.8\%$。

Title: Decoding by Contrasting Knowledge: Enhancing LLMs' Confidence on Edited Facts

• Authors: Baolong Bi, Shenghua Liu, Lingrui Mei, Yiwei Wang, Pengliang Ji, Xueqi Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.11613
• Pdf URL: https://arxiv.org/pdf/2405.11613
• Copy Paste: [[2405.11613]] Decoding by Contrasting Knowledge: Enhancing LLMs' Confidence on Edited Facts(https://arxiv.org/abs/2405.11613)
• Keywords: language model, llm
• Abstract: The knowledge within large language models (LLMs) may become outdated quickly. While in-context editing (ICE) is currently the most effective method for knowledge editing (KE), it is constrained by the black-box modeling of LLMs and thus lacks interpretability. Our work aims to elucidate the superior performance of ICE on the KE by analyzing the impacts of in-context new knowledge on token-wise distributions. We observe that despite a significant boost in logits of the new knowledge, the performance of is still hindered by stubborn knowledge. Stubborn knowledge refers to as facts that have gained excessive confidence during pretraining, making it hard to edit effectively. To address this issue and further enhance the performance of ICE, we propose a novel approach termed $\textbf{De}$coding by $\textbf{C}$ontrasting $\textbf{K}$nowledge (DeCK). DeCK derives the distribution of the next token by contrasting the logits obtained from the newly edited knowledge guided by ICE with those from the unedited parametric knowledge. Our experiments consistently demonstrate that DeCK enhances the confidence of LLMs in edited facts. For instance, it improves the performance of LLaMA3-8B-instruct on MQuAKE by up to 219%, demonstrating its capability to strengthen ICE in the editing of stubborn knowledge. Our work paves the way to develop the both effective and accountable KE methods for LLMs. (The source code is available at: $\href{this https URL}{\text{this https URL.}}$ )
• 摘要：大型语言模型 (LLM) 中的知识可能很快就会过时。虽然上下文编辑（ICE）是目前最有效的知识编辑（KE）方法，但它受到法学硕士黑盒建模的限制，缺乏可解释性。我们的工作旨在通过分析上下文新知识对 token-wise 分布的影响来阐明 ICE 在 KE 上的卓越性能。我们观察到，尽管新知识的 logits 显着提升，但顽固知识的表现仍然受到阻碍。顽固知识是指在预训练过程中获得过度自信而难以有效编辑的事实。为了解决这个问题并进一步提高 ICE 的性能，我们提出了一种新方法，称为 $\textbf{De}$coding by $\textbf{C}$ontrasting $\textbf{K}$nowledge (DeCK)。 DeCK 通过对比从 ICE 引导的新编辑的知识中获得的 logits 与未编辑的参数化知识中获得的 logits，得出下一个 token 的分布。我们的实验一致表明，DeCK 增强了法学硕士对编辑事实的信心。例如，它将 MQuAKE 上的 LLaMA3-8B-instruct 的性能提高了高达 219%，展示了其在顽固知识编辑方面增强 ICE 的能力。我们的工作为法学硕士开发有效且负责任的 KE 方法铺平了道路。 （源代码位于：$\href{此 https URL}{\text{此 https URL。}}$ ）

Title: Zero-Shot Stance Detection using Contextual Data Generation with LLMs

• Authors: Ghazaleh Mahmoudi, Babak Behkamkia, Sauleh Eetemadi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.11637
• Pdf URL: https://arxiv.org/pdf/2405.11637
• Copy Paste: [[2405.11637]] Zero-Shot Stance Detection using Contextual Data Generation with LLMs(https://arxiv.org/abs/2405.11637)
• Keywords: language model, gpt, llm
• Abstract: Stance detection, the classification of attitudes expressed in a text towards a specific topic, is vital for applications like fake news detection and opinion mining. However, the scarcity of labeled data remains a challenge for this task. To address this problem, we propose Dynamic Model Adaptation with Contextual Data Generation (DyMoAdapt) that combines Few-Shot Learning and Large Language Models. In this approach, we aim to fine-tune an existing model at test time. We achieve this by generating new topic-specific data using GPT-3. This method could enhance performance by allowing the adaptation of the model to new topics. However, the results did not increase as we expected. Furthermore, we introduce the Multi Generated Topic VAST (MGT-VAST) dataset, which extends VAST using GPT-3. In this dataset, each context is associated with multiple topics, allowing the model to understand the relationship between contexts and various potential topics
• 摘要：立场检测，即文本中对特定主题表达的态度的分类，对于假新闻检测和意见挖掘等应用至关重要。然而，标记数据的稀缺仍然是这项任务的一个挑战。为了解决这个问题，我们提出了结合了少样本学习和大型语言模型的动态模型适应与上下文数据生成（DyMoAdapt）。在这种方法中，我们的目标是在测试时微调现有模型。我们通过使用 GPT-3 生成新的特定于主题的数据来实现这一目标。该方法可以通过允许模型适应新主题来提高性能。然而，结果并没有像我们预期的那样增加。此外，我们还引入了多生成主题 VAST (MGT-VAST) 数据集，它使用 GPT-3 扩展了 VAST。在此数据集中，每个上下文都与多个主题相关联，使模型能够理解上下文与各种潜在主题之间的关系

Title: Cyber Risks of Machine Translation Critical Errors : Arabic Mental Health Tweets as a Case Study

• Authors: Hadeel Saadany, Ashraf Tantawy, Constantin Orasan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.11668
• Pdf URL: https://arxiv.org/pdf/2405.11668
• Copy Paste: [[2405.11668]] Cyber Risks of Machine Translation Critical Errors : Arabic Mental Health Tweets as a Case Study(https://arxiv.org/abs/2405.11668)
• Keywords: hallucination
• Abstract: With the advent of Neural Machine Translation (NMT) systems, the MT output has reached unprecedented accuracy levels which resulted in the ubiquity of MT tools on almost all online platforms with multilingual content. However, NMT systems, like other state-of-the-art AI generative systems, are prone to errors that are deemed machine hallucinations. The problem with NMT hallucinations is that they are remarkably \textit{fluent} hallucinations. Since they are trained to produce grammatically correct utterances, NMT systems are capable of producing mistranslations that are too fluent to be recognised by both users of the MT tool, as well as by automatic quality metrics that are used to gauge their performance. In this paper, we introduce an authentic dataset of machine translation critical errors to point to the ethical and safety issues involved in the common use of MT. The dataset comprises mistranslations of Arabic mental health postings manually annotated with critical error types. We also show how the commonly used quality metrics do not penalise critical errors and highlight this as a critical issue that merits further attention from researchers.
• 摘要：随着神经机器翻译 (NMT) 系统的出现，机器翻译输出达到了前所未有的准确性水平，导致机器翻译工具在几乎所有多语言内容的在线平台上无处不在。然而，NMT 系统与其他最先进的人工智能生成系统一样，很容易出现被视为机器幻觉的错误。 NMT 幻觉的问题在于它们是非常\textit{流畅}的幻觉。由于 NMT 系统经过训练可以生成语法正确的话语，因此能够生成过于流利的误译，以致机器翻译工具的用户以及用于衡量其性能的自动质量指标都无法识别。在本文中，我们引入了机器翻译关键错误的真实数据集，以指出机器翻译常见使用中涉及的道德和安全问题。该数据集包含阿拉伯语心理健康帖子的误译，并手动注释了严重错误类型。我们还展示了常用的质量指标如何不惩罚关键错误，并强调这是一个值得研究人员进一步关注的关键问题。

Title: Token-wise Influential Training Data Retrieval for Large Language Models

• Authors: Huawei Lin, Jikai Long, Zhaozhuo Xu, Weijie Zhao
• Subjects: cs.CL, cs.AI, cs.CR, cs.IR
• Abstract URL: https://arxiv.org/abs/2405.11724
• Pdf URL: https://arxiv.org/pdf/2405.11724
• Copy Paste: [[2405.11724]] Token-wise Influential Training Data Retrieval for Large Language Models(https://arxiv.org/abs/2405.11724)
• Keywords: language model, llm
• Abstract: Given a Large Language Model (LLM) generation, how can we identify which training data led to this generation? In this paper, we proposed RapidIn, a scalable framework adapting to LLMs for estimating the influence of each training data. The proposed framework consists of two stages: caching and retrieval. First, we compress the gradient vectors by over 200,000x, allowing them to be cached on disk or in GPU/CPU memory. Then, given a generation, RapidIn efficiently traverses the cached gradients to estimate the influence within minutes, achieving over a 6,326x speedup. Moreover, RapidIn supports multi-GPU parallelization to substantially accelerate caching and retrieval. Our empirical result confirms the efficiency and effectiveness of RapidIn.
• 摘要：给定大型语言模型 (LLM) 的生成，我们如何识别哪些训练数据导致了这一生成？在本文中，我们提出了RapidIn，一个适用于法学硕士的可扩展框架，用于估计每个训练数据的影响。所提出的框架由两个阶段组成：缓存和检索。首先，我们将梯度向量压缩超过 200,000 倍，允许它们缓存在磁盘或 GPU/CPU 内存中。然后，对于给定的一代，RapidIn 有效地遍历缓存的梯度以在几分钟内估计影响，实现超过 6,326 倍的加速。此外，RapidIn 支持多 GPU 并行化，可大幅加速缓存和检索。我们的实证结果证实了 RapidIn 的效率和有效性。

Title: Exploring Ordinality in Text Classification: A Comparative Study of Explicit and Implicit Techniques

• Authors: Siva Rajesh Kasa, Aniket Goel, Karan Gupta, Sumegh Roychowdhury, Anish Bhanushali, Nikhil Pattisapu, Prasanna Srinivasa Murthy
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.11775
• Pdf URL: https://arxiv.org/pdf/2405.11775
• Copy Paste: [[2405.11775]] Exploring Ordinality in Text Classification: A Comparative Study of Explicit and Implicit Techniques(https://arxiv.org/abs/2405.11775)
• Keywords: language model
• Abstract: Ordinal Classification (OC) is a widely encountered challenge in Natural Language Processing (NLP), with applications in various domains such as sentiment analysis, rating prediction, and more. Previous approaches to tackle OC have primarily focused on modifying existing or creating novel loss functions that \textbf{explicitly} account for the ordinal nature of labels. However, with the advent of Pretrained Language Models (PLMs), it became possible to tackle ordinality through the \textbf{implicit} semantics of the labels as well. This paper provides a comprehensive theoretical and empirical examination of both these approaches. Furthermore, we also offer strategic recommendations regarding the most effective approach to adopt based on specific settings.
• 摘要：序数分类 (OC) 是自然语言处理 (NLP) 中广泛遇到的挑战，在情感分析、评级预测等各个领域都有应用。以前解决 OC 的方法主要集中于修改现有的或创建新颖的损失函数，这些函数 \textbf{显式} 地解释了标签的序数性质。然而，随着预训练语言模型（PLM）的出现，通过标签的 \textbf{implicit} 语义来处理序数性也成为可能。本文对这两种方法进行了全面的理论和实证检验。此外，我们还根据具体情况提供有关采用最有效方法的战略建议。

Title: (Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts

• Authors: Minghao Wu, Yulin Yuan, Gholamreza Haffari, Longyue Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.11804
• Pdf URL: https://arxiv.org/pdf/2405.11804
• Copy Paste: [[2405.11804]] (Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts(https://arxiv.org/abs/2405.11804)
• Keywords: language model, llm, agent
• Abstract: Recent advancements in machine translation (MT) have significantly enhanced translation quality across various domains. However, the translation of literary texts remains a formidable challenge due to their complex language, figurative expressions, and cultural nuances. In this work, we introduce a novel multi-agent framework based on large language models (LLMs) for literary translation, implemented as a company called TransAgents, which mirrors traditional translation publication process by leveraging the collective capabilities of multiple agents, to address the intricate demands of translating literary works. To evaluate the effectiveness of our system, we propose two innovative evaluation strategies: Monolingual Human Preference (MHP) and Bilingual LLM Preference (BLP). MHP assesses translations from the perspective of monolingual readers of the target language, while BLP uses advanced LLMs to compare translations directly with the original texts. Empirical findings indicate that despite lower d-BLEU scores, translations from TransAgents are preferred by both human evaluators and LLMs over human-written references, particularly in genres requiring domain-specific knowledge. We also highlight the strengths and limitations of TransAgents through case studies and suggests directions for future research.
• 摘要：机器翻译 (MT) 的最新进展显着提高了各个领域的翻译质量。然而，由于其复杂的语言、比喻表达和文化差异，文学文本的翻译仍然是一个艰巨的挑战。在这项工作中，我们引入了一种基于大语言模型 (LLM) 的新颖的文学翻译多代理框架，由一家名为 TransAgents 的公司实施，该框架通过利用多个代理的集体能力来反映传统的翻译出版流程，以解决复杂的翻译问题。文学作品翻译的要求为了评估我们系统的有效性，我们提出了两种创新的评估策略：单语人类偏好（MHP）和双语法学硕士偏好（BLP）。 MHP 从目标语言的单语读者的角度评估翻译，而 BLP 使用高级法学硕士直接将翻译与原文进行比较。实证研究结果表明，尽管 d-BLEU 分数较低，但人类评估者和法学硕士都更喜欢来自 TransAgents 的翻译，而不是人类撰写的参考文献，特别是在需要特定领域知识的类型中。我们还通过案例研究强调 TransAgents 的优势和局限性，并提出未来研究的方向。

Title: Intuitive Fine-Tuning: Towards Unifying SFT and RLHF into a Single Process

• Authors: Ermo Hua, Biqing Qi, Kaiyan Zhang, Yue Yu, Ning Ding, Xingtai Lv, Kai Tian, Bowen Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] Intuitive Fine-Tuning: Towards Unifying SFT and RLHF into a Single Process(https://arxiv.org/abs/)
• Keywords: language model
• Abstract: Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) are two fundamental processes for enhancing the capabilities of Language Models (LMs) post pre-training, aligning them better with human preferences. Although SFT advances in training efficiency, RLHF delivers better alignment, thus they are often combined. However, common practices simply apply them sequentially without unifying their optimization targets, resulting in a trade-off between fitting different objectives, and ignoring the opportunities to bridge the paradigm gap and take the strength from both. To obtain a unified understanding, we interpret SFT and RLHF using two sub-processes -- Preference Estimation and Transition Optimization -- defined at token level within the Markov Decision Process (MDP) framework. This modeling shows that SFT is only a specialized case of RLHF with inferior estimation and optimization. RLHF evaluates the quality of model's entire generated answer, whereas SFT only scores predicted tokens based on preceding tokens from target answers. Therefore, SFT overestimates the ability of model, leading to inferior optimization. Building on this view, we introduce Intuitive Fine-tuning (IFT) to integrate SFT and RLHF into a single process. IFT captures LMs' intuitive sense of the entire answers through a temporal residual connection, while using a single policy and the same volume of non-preference-labeled data as SFT. Our experiments show that IFT performs comparably or even superiorly to sequential recipes of SFT and some typical alignment methods across several tasks, particularly those requires generation, reasoning, and fact-following abilities. An explainable Frozen Lake game further validates the effectiveness of IFT.
• 摘要：监督微调 (SFT) 和强化学习人类反馈 (RLHF) 是增强语言模型 (LM) 预训练后能力的两个基本过程，使其更好地与人类偏好保持一致。尽管 SFT 在训练效率方面有所提高，但 RLHF 提供了更好的一致性，因此它们经常结合在一起。然而，常见的做法只是按顺序应用它们，而没有统一它们的优化目标，导致在适应不同目标之间进行权衡，并忽略了弥合范式差距并利用两者优势的机会。为了获得统一的理解，我们使用马尔可夫决策过程 (MDP) 框架内以 token 级别定义的两个子过程——偏好估计和转换优化——来解释 SFT 和 RLHF。该建模表明，SFT 只是 RLHF 的一个特殊情况，其估计和优化较差。RLHF 评估模型整个生成答案的质量，而 SFT 仅根据目标答案中的前一个 token 对预测 token 进行评分。因此，SFT 高估了模型的能力，导致优化较差。基于此观点，我们引入了直观微调 (IFT)，将 SFT 和 RLHF 集成到一个流程中。IFT 通过时间残差连接捕获 LM 对整个答案的直观感受，同时使用单一策略和与 SFT 相同数量的非偏好标记数据。我们的实验表明，IFT 在多个任务中的表现与 SFT 的顺序配方和一些典型的对齐方法相当甚至更好，特别是那些需要生成、推理和事实跟踪能力的任务。可解释的《冰冻湖》游戏进一步验证了 IFT 的有效性。

Title: xFinder: Robust and Pinpoint Answer Extraction for Large Language Models

• Authors: Qingchen Yu, Zifan Zheng, Shichao Song, Zhiyu Li, Feiyu Xiong, Bo Tang, Ding Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.11874
• Pdf URL: https://arxiv.org/pdf/2405.11874
• Copy Paste: [[2405.11874]] xFinder: Robust and Pinpoint Answer Extraction for Large Language Models(https://arxiv.org/abs/2405.11874)
• Keywords: language model, llm, prompt
• Abstract: The continuous advancement of large language models (LLMs) has brought increasing attention to the critical issue of developing fair and reliable methods for evaluating their performance. Particularly, the emergence of subjective or non-subjective cheating phenomena, such as test set leakage and prompt format overfitting, poses significant challenges to the reliable evaluation of LLMs. Since evaluation frameworks often utilize Regular Expression (RegEx) for answer extraction, some models may adjust their responses to comply with specific formats that are easily extractable by RegEx. Nevertheless, the key answer extraction module based on RegEx frequently suffers from extraction errors. This paper conducts a comprehensive analysis of the entire LLM evaluation chain, demonstrating that optimizing the key answer extraction module can improve extraction accuracy, reduce LLMs' reliance on specific answer formats, and enhance the reliability of LLM evaluation. To address these issues, we propose xFinder, a model specifically designed for key answer extraction. As part of this process, we create a specialized dataset, the Key Answer Finder (KAF) dataset, to ensure effective model training and evaluation. Through generalization testing and evaluation in real-world scenarios, the results demonstrate that the smallest xFinder model with only 500 million parameters achieves an average answer extraction accuracy of 93.42%. In contrast, RegEx accuracy in the best evaluation framework is 74.38%. xFinder exhibits stronger robustness and higher accuracy compared to existing evaluation frameworks. All resources for xFinder are available at \url{this https URL}.
• 摘要：大型语言模型（LLM）的不断进步使人们越来越关注开发公平可靠的方法来评估其性能的关键问题。特别是测试集泄漏、提示格式过拟合等主观或非主观作弊现象的出现，给法学硕士的可靠评估带来了重大挑战。由于评估框架通常利用正则表达式 (RegEx) 进行答案提取，因此某些模型可能会调整其响应以符合 RegEx 可以轻松提取的特定格式。然而，基于正则表达式的关键答案提取模块经常出现提取错误。本文对整个LLM评估链进行了全面分析，证明优化关键答案提取模块可以提高提取准确性，减少LLM对特定答案格式的依赖，增强LLM评估的可靠性。为了解决这些问题，我们提出了 xFinder，一个专门为关键答案提取而设计的模型。作为此过程的一部分，我们创建了一个专门的数据集，即关键答案查找器 (KAF) 数据集，以确保有效的模型训练和评估。通过真实场景的泛化测试和评估，结果表明，只有 5 亿个参数的最小 xFinder 模型的平均答案提取准确率达到 93.42%。相比之下，最佳评估框架中的 RegEx 准确率为 74.38%。与现有评估框架相比，xFinder 表现出更强的鲁棒性和更高的准确性。 xFinder 的所有资源均可从 \url{此 https URL} 获取。

Title: A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpus

• Authors: Eduard Poesina, Cornelia Caragea, Radu Tudor Ionescu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.11877
• Pdf URL: https://arxiv.org/pdf/2405.11877
• Copy Paste: [[2405.11877]] A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpus(https://arxiv.org/abs/2405.11877)
• Keywords: agent
• Abstract: Natural language inference (NLI), the task of recognizing the entailment relationship in sentence pairs, is an actively studied topic serving as a proxy for natural language understanding. Despite the relevance of the task in building conversational agents and improving text classification, machine translation and other NLP tasks, to the best of our knowledge, there is no publicly available NLI corpus for the Romanian language. To this end, we introduce the first Romanian NLI corpus (RoNLI) comprising 58K training sentence pairs, which are obtained via distant supervision, and 6K validation and test sentence pairs, which are manually annotated with the correct labels. We conduct experiments with multiple machine learning methods based on distant learning, ranging from shallow models based on word embeddings to transformer-based neural networks, to establish a set of competitive baselines. Furthermore, we improve on the best model by employing a new curriculum learning strategy based on data cartography. Our dataset and code to reproduce the baselines are available this https URL.
• 摘要：自然语言推理（NLI）是识别句子对中蕴含关系的任务，是一个积极研究的主题，作为自然语言理解的代理。尽管该任务与构建会话代理和改进文本分类、机器翻译和其他 NLP 任务相关，但据我们所知，罗马尼亚语还没有公开可用的 NLI 语料库。为此，我们引入了第一个罗马尼亚语 NLI 语料库（RoNLI），其中包含通过远程监督获得的 58K 训练句子对，以及使用正确标签手动注释的 6K 验证和测试句子对。我们对基于远程学习的多种机器学习方法进行了实验，从基于词嵌入的浅层模型到基于变压器的神经网络，以建立一组有竞争力的基线。此外，我们通过采用基于数据制图的新课程学习策略来改进最佳模型。我们的数据集和重现基线的代码可通过此 https URL 获取。

Title: Unveiling and Manipulating Prompt Influence in Large Language Models

• Authors: Zijian Feng, Hanzhang Zhou, Zixiao Zhu, Junlang Qian, Kezhi Mao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.11891
• Pdf URL: https://arxiv.org/pdf/2405.11891
• Copy Paste: [[2405.11891]] Unveiling and Manipulating Prompt Influence in Large Language Models(https://arxiv.org/abs/2405.11891)
• Keywords: language model, llm, prompt
• Abstract: Prompts play a crucial role in guiding the responses of Large Language Models (LLMs). However, the intricate role of individual tokens in prompts, known as input saliency, in shaping the responses remains largely underexplored. Existing saliency methods either misalign with LLM generation objectives or rely heavily on linearity assumptions, leading to potential inaccuracies. To address this, we propose Token Distribution Dynamics (TDD), a \textcolor{black}{simple yet effective} approach to unveil and manipulate the role of prompts in generating LLM outputs. TDD leverages the robust interpreting capabilities of the language model head (LM head) to assess input saliency. It projects input tokens into the embedding space and then estimates their significance based on distribution dynamics over the vocabulary. We introduce three TDD variants: forward, backward, and bidirectional, each offering unique insights into token relevance. Extensive experiments reveal that the TDD surpasses state-of-the-art baselines with a big margin in elucidating the causal relationships between prompts and LLM outputs. Beyond mere interpretation, we apply TDD to two prompt manipulation tasks for controlled text generation: zero-shot toxic language suppression and sentiment steering. Empirical results underscore TDD's proficiency in identifying both toxic and sentimental cues in prompts, subsequently mitigating toxicity or modulating sentiment in the generated content.
• 摘要：提示在指导大型​​语言模型（LLM）的反应中发挥着至关重要的作用。然而，提示中的各个标记（称为输入显着性）在塑造响应方面的复杂作用仍然很大程度上未被充分探索。现有的显着性方法要么与 LLM 生成目标不一致，要么严重依赖线性假设，从而导致潜在的不准确性。为了解决这个问题，我们提出了令牌分布动态（TDD），这是一种 \textcolor{black}{简单而有效} 的方法，用于揭示和操纵提示在生成 LLM 输出中的作用。 TDD 利用语言模型头（LM head）强大的解释能力来评估输入显着性。它将输入标记投影到嵌入空间中，然后根据词汇表上的分布动态来估计它们的重要性。我们引入了三种 TDD 变体：前向、后向和双向，每种都提供了对代币相关性的独特见解。大量实验表明，TDD 在阐明提示与 LLM 输出之间的因果关系方面大幅超越了最先进的基线。除了单纯的解释之外，我们还将 TDD 应用于两个提示操作任务以控制文本生成：零样本有毒语言抑制和情绪引导。实证结果强调了 TDD 能够熟练地识别提示中的有毒和情感线索，从而减轻生成内容中的毒性或调节情感。

Title: A Constraint-Enforcing Reward for Adversarial Attacks on Text Classifiers

• Authors: Tom Roth, Inigo Jauregi Unanue, Alsharif Abuadbba, Massimo Piccardi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.11904
• Pdf URL: https://arxiv.org/pdf/2405.11904
• Copy Paste: [[2405.11904]] A Constraint-Enforcing Reward for Adversarial Attacks on Text Classifiers(https://arxiv.org/abs/2405.11904)
• Keywords: language model
• Abstract: Text classifiers are vulnerable to adversarial examples -- correctly-classified examples that are deliberately transformed to be misclassified while satisfying acceptability constraints. The conventional approach to finding adversarial examples is to define and solve a combinatorial optimisation problem over a space of allowable transformations. While effective, this approach is slow and limited by the choice of transformations. An alternate approach is to directly generate adversarial examples by fine-tuning a pre-trained language model, as is commonly done for other text-to-text tasks. This approach promises to be much quicker and more expressive, but is relatively unexplored. For this reason, in this work we train an encoder-decoder paraphrase model to generate a diverse range of adversarial examples. For training, we adopt a reinforcement learning algorithm and propose a constraint-enforcing reward that promotes the generation of valid adversarial examples. Experimental results over two text classification datasets show that our model has achieved a higher success rate than the original paraphrase model, and overall has proved more effective than other competitive attacks. Finally, we show how key design choices impact the generated examples and discuss the strengths and weaknesses of the proposed approach.
• 摘要：文本分类器很容易受到对抗性示例的影响——正确分类的示例在满足可接受性约束的同时被故意转换为错误分类。寻找对抗性例子的传统方法是在允许的变换空间上定义并解决组合优化问题。虽然有效，但这种方法很慢并且受到转换选择的限制。另一种方法是通过微调预先训练的语言模型来直接生成对抗性示例，就像其他文本到文本任务通常所做的那样。这种方法有望更快、更具表现力，但相对而言尚未得到探索。因此，在这项工作中，我们训练了一个编码器-解码器释义模型来生成各种对抗性示例。对于训练，我们采用强化学习算法并提出约束执行奖励，以促进有效对抗性示例的生成。两个文本分类数据集的实验结果表明，我们的模型比原始释义模型取得了更高的成功率，并且总体上证明比其他竞争性攻击更有效。最后，我们展示了关键的设计选择如何影响生成的示例，并讨论了所提出方法的优点和缺点。

Title: WisPerMed at BioLaySumm: Adapting Autoregressive Large Language Models for Lay Summarization of Scientific Articles

• Authors: Tabea M. G. Pakull, Hendrik Damm, Ahmad Idrissi-Yaghir, Henning Schäfer, Peter A. Horn, Christoph M. Friedrich
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.11950
• Pdf URL: https://arxiv.org/pdf/2405.11950
• Copy Paste: [[2405.11950]] WisPerMed at BioLaySumm: Adapting Autoregressive Large Language Models for Lay Summarization of Scientific Articles(https://arxiv.org/abs/2405.11950)
• Keywords: language model, llm, prompt
• Abstract: This paper details the efforts of the WisPerMed team in the BioLaySumm2024 Shared Task on automatic lay summarization in the biomedical domain, aimed at making scientific publications accessible to non-specialists. Large language models (LLMs), specifically the BioMistral and Llama3 models, were fine-tuned and employed to create lay summaries from complex scientific texts. The summarization performance was enhanced through various approaches, including instruction tuning, few-shot learning, and prompt variations tailored to incorporate specific context information. The experiments demonstrated that fine-tuning generally led to the best performance across most evaluated metrics. Few-shot learning notably improved the models' ability to generate relevant and factually accurate texts, particularly when using a well-crafted prompt. Additionally, a Dynamic Expert Selection (DES) mechanism to optimize the selection of text outputs based on readability and factuality metrics was developed. Out of 54 participants, the WisPerMed team reached the 4th place, measured by readability, factuality, and relevance. Determined by the overall score, our approach improved upon the baseline by approx. 5.5 percentage points and was only approx 1.5 percentage points behind the first place.
• 摘要：本文详细介绍了 WisPerMed 团队在 BioLaySumm2024 共享任务中关于生物医学领域自动外行总结的工作，旨在使非专业人士也能访问科学出版物。大型语言模型 (LLM)，特别是 BioMistral 和 Llama3 模型，经过微调并用于根据复杂的科学文本创建简单摘要。摘要性能通过各种方法得到了增强，包括指令调整、小样本学习以及为合并特定上下文信息而定制的提示变化。实验表明，微调通常会在大多数评估指标中带来最佳性能。少样本学习显着提高了模型生成相关且准确的文本的能力，特别是在使用精心设计的提示时。此外，还开发了动态专家选择（DES）机制，用于根据可读性和事实性指标优化文本输出的选择。根据可读性、真实性和相关性衡量，WisPerMed 团队在 54 名参与者中排名第四。根据总分，我们的方法比基线提高了大约 10%。 5.5个百分点，仅落后第一名约1.5个百分点。

Title: Multiple-Choice Questions are Efficient and Robust LLM Evaluators

• Authors: Ziyin Zhang, Lizhen Xu, Zhaokun Jiang, Hongkun Hao, Rui Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.11966
• Pdf URL: https://arxiv.org/pdf/2405.11966
• Copy Paste: [[2405.11966]] Multiple-Choice Questions are Efficient and Robust LLM Evaluators(https://arxiv.org/abs/2405.11966)
• Keywords: llm
• Abstract: We present GSM-MC and MATH-MC, two multiple-choice (MC) datasets constructed by collecting answers and incorrect predictions on GSM8K and MATH from over 50 open-source models. Through extensive experiments, we show that LLMs' performance on the MC versions of these two popular benchmarks is strongly correlated with their performance on the original versions, and is quite robust to distractor choices and option orders, while the evaluation time is reduced by a factor of up to 30. Following a similar procedure, we also introduce PythonIO, a new program output prediction MC dataset constructed from two other popular LLM evaluation benchmarks HumanEval and MBPP. Our data and code are available at this https URL.
• 摘要：我们提出了 GSM-MC 和 MATH-MC，这两个多项选择 (MC) 数据集是通过从 50 多个开源模型收集 GSM8K 和 MATH 的答案和错误预测而构建的。通过大量的实验，我们表明，LLM 在这两个流行基准的 MC 版本上的表现与其在原始版本上的表现密切相关，并且对于干扰项选择和期权订单具有相当的鲁棒性，同时评估时间减少了一个因子最多 30 个。按照类似的过程，我们还引入了 PythonIO，这是一个新的程序输出预测 MC 数据集，由另外两个流行的 LLM 评估基准 HumanEval 和 MBPP 构建。我们的数据和代码可通过此 https URL 获取。

Title: A review on the use of large language models as virtual tutors

• Authors: Silvia García-Méndez, Francisco de Arriba-Pérez, María del Carmen Somoza-López
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.11983
• Pdf URL: https://arxiv.org/pdf/2405.11983
• Copy Paste: [[2405.11983]] A review on the use of large language models as virtual tutors(https://arxiv.org/abs/2405.11983)
• Keywords: language model, llm
• Abstract: Transformer architectures contribute to managing long-term dependencies for Natural Language Processing, representing one of the most recent changes in the field. These architectures are the basis of the innovative, cutting-edge Large Language Models (LLMs) that have produced a huge buzz in several fields and industrial sectors, among the ones education stands out. Accordingly, these generative Artificial Intelligence-based solutions have directed the change in techniques and the evolution in educational methods and contents, along with network infrastructure, towards high-quality learning. Given the popularity of LLMs, this review seeks to provide a comprehensive overview of those solutions designed specifically to generate and evaluate educational materials and which involve students and teachers in their design or experimental plan. To the best of our knowledge, this is the first review of educational applications (e.g., student assessment) of LLMs. As expected, the most common role of these systems is as virtual tutors for automatic question generation. Moreover, the most popular models are GTP-3 and BERT. However, due to the continuous launch of new generative models, new works are expected to be published shortly.
• 摘要：Transformer 架构有助于管理自然语言处理的长期依赖关系，代表了该领域的最新变化之一。这些架构是创新、尖端的大型语言模型 (LLM) 的基础，这些模型在多个领域和工业部门引起了巨大的轰动，其中教育领域尤为突出。因此，这些基于生成人工智能的解决方案引导了技术的变革以及教育方法和内容的演变，以及网络基础设施，以实现高质量的学习。鉴于法学硕士的受欢迎程度，本次审查旨在全面概述专门为生成和评估教育材料而设计的解决方案，这些解决方案让学生和教师参与其设计或实验计划。据我们所知，这是对法学硕士教育申请（例如学生评估）的首次审查。正如预期的那样，这些系统最常见的作用是作为自动问题生成的虚拟导师。此外，最流行的模型是 GTP-3 和 BERT。不过，由于新的生成模型不断推出，新作品预计很快就会面世。

Title: Can AI Relate: Testing Large Language Model Response for Mental Health Support

• Authors: Saadia Gabriel, Isha Puri, Xuhai Xu, Matteo Malgaroli, Marzyeh Ghassemi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.12021
• Pdf URL: https://arxiv.org/pdf/2405.12021
• Copy Paste: [[2405.12021]] Can AI Relate: Testing Large Language Model Response for Mental Health Support(https://arxiv.org/abs/2405.12021)
• Keywords: language model, gpt, llm, chat
• Abstract: Large language models (LLMs) are already being piloted for clinical use in hospital systems like NYU Langone, Dana-Farber and the NHS. A proposed deployment use case is psychotherapy, where a LLM-powered chatbot can treat a patient undergoing a mental health crisis. Deployment of LLMs for mental health response could hypothetically broaden access to psychotherapy and provide new possibilities for personalizing care. However, recent high-profile failures, like damaging dieting advice offered by the Tessa chatbot to patients with eating disorders, have led to doubt about their reliability in high-stakes and safety-critical settings. In this work, we develop an evaluation framework for determining whether LLM response is a viable and ethical path forward for the automation of mental health treatment. Using human evaluation with trained clinicians and automatic quality-of-care metrics grounded in psychology research, we compare the responses provided by peer-to-peer responders to those provided by a state-of-the-art LLM. We show that LLMs like GPT-4 use implicit and explicit cues to infer patient demographics like race. We then show that there are statistically significant discrepancies between patient subgroups: Responses to Black posters consistently have lower empathy than for any other demographic group (2%-13% lower than the control group). Promisingly, we do find that the manner in which responses are generated significantly impacts the quality of the response. We conclude by proposing safety guidelines for the potential deployment of LLMs for mental health response.
• 摘要：大语言模型 (LLM) 已在纽约大学朗格尼分校、丹娜法伯癌症研究所和 NHS 等医院系统中进行临床试验。拟议的部署用例是心理治疗，其中由法学硕士驱动的聊天机器人可以治疗经历心理健康危机的患者。部署法学硕士进行心理健康反应理论上可以扩大心理治疗的范围，并为个性化护理提供新的可能性。然而，最近备受瞩目的失败事件，例如 Tessa 聊天机器人向饮食失调患者提供的破坏性节食建议，导致人们对其在高风险和安全关键环境中的可靠性产生怀疑。在这项工作中，我们开发了一个评估框架，用于确定法学硕士的反应是否是实现心理健康治疗自动化的可行且合乎道德的道路。我们使用训练有素的临床医生进行的人工评估以及基于心理学研究的自动护理质量指标，将同行响应者提供的响应与最先进的法学硕士提供的响应进行比较。我们表明，像 GPT-4 这样的法学硕士使用隐式和显式线索来推断患者的人口统计数据，例如种族。然后我们发现，患者亚组之间存在统计上显着的差异：对黑人海报的反应始终比任何其他人口群体的同理心要低（比对照组低 2%-13%）。令人欣喜的是，我们确实发现生成响应的方式会显着影响响应的质量。最后，我们提出了可能部署法学硕士进行心理健康应对的安全指南。

Title: STYLE: Improving Domain Transferability of Asking Clarification Questions in Large Language Model Powered Conversational Agents

• Authors: Yue Chen, Chen Huang, Yang Deng, Wenqiang Lei, Dingnan Jin, Jia Liu, Tat-Seng Chua
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.12059
• Pdf URL: https://arxiv.org/pdf/2405.12059
• Copy Paste: [[2405.12059]] STYLE: Improving Domain Transferability of Asking Clarification Questions in Large Language Model Powered Conversational Agents(https://arxiv.org/abs/2405.12059)
• Keywords: language model, llm, agent
• Abstract: Equipping a conversational search engine with strategies regarding when to ask clarification questions is becoming increasingly important across various domains. Attributing to the context understanding capability of LLMs and their access to domain-specific sources of knowledge, LLM-based clarification strategies feature rapid transfer to various domains in a post-hoc manner. However, they still struggle to deliver promising performance on unseen domains, struggling to achieve effective domain transferability. We take the first step to investigate this issue and existing methods tend to produce one-size-fits-all strategies across diverse domains, limiting their search effectiveness. In response, we introduce a novel method, called Style, to achieve effective domain transferability. Our experimental results indicate that Style bears strong domain transferability, resulting in an average search performance improvement of ~10% on four unseen domains.
• 摘要：为会话式搜索引擎配备有关何时提出澄清问题的策略在各个领域变得越来越重要。由于法学硕士的背景理解能力及其对特定领域知识来源的访问，基于法学硕士的澄清策略具有以事后方式快速转移到各个领域的特点。然而，他们仍然难以在未知的领域提供有希望的性能，努力实现有效的领域可转移性。我们迈出了调查这个问题的第一步，现有的方法往往会在不同的领域产生一刀切的策略，从而限制了它们的搜索效率。为此，我们引入了一种称为 Style 的新颖方法来实现有效的域可转移性。我们的实验结果表明，Style 具有很强的域可转移性，在四个未见过的域上平均搜索性能提高了约 10%。

Title: CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models

• Authors: Tong Zhang, Peixin Qin, Yang Deng, Chen Huang, Wenqiang Lei, Junhong Liu, Dingnan Jin, Hongru Liang, Tat-Seng Chua
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.12063
• Pdf URL: https://arxiv.org/pdf/2405.12063
• Copy Paste: [[2405.12063]] CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models(https://arxiv.org/abs/2405.12063)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Large language models (LLMs) are increasingly used to meet user information needs, but their effectiveness in dealing with user queries that contain various types of ambiguity remains unknown, ultimately risking user trust and satisfaction. To this end, we introduce CLAMBER, a benchmark for evaluating LLMs using a well-organized taxonomy. Building upon the taxonomy, we construct ~12K high-quality data to assess the strengths, weaknesses, and potential risks of various off-the-shelf LLMs. Our findings indicate the limited practical utility of current LLMs in identifying and clarifying ambiguous user queries, even enhanced by chain-of-thought (CoT) and few-shot prompting. These techniques may result in overconfidence in LLMs and yield only marginal enhancements in identifying ambiguity. Furthermore, current LLMs fall short in generating high-quality clarifying questions due to a lack of conflict resolution and inaccurate utilization of inherent knowledge. In this paper, CLAMBER presents a guidance and promotes further research on proactive and trustworthy LLMs. Our dataset is available at this https URL
• 摘要：大型语言模型 (LLM) 越来越多地用于满足用户信息需求，但其处理包含各种类型歧义的用户查询的有效性仍然未知，最终会危及用户的信任和满意度。为此，我们引入了 CLAMBER，这是一个使用组织良好的分类法评估法学硕士的基准。基于分类法，我们构建了约 12K 高质量数据来评估各种现成法学硕士的优势、劣势和潜在风险。我们的研究结果表明，当前的法学硕士在识别和澄清模糊的用户查询方面的实用性有限，即使通过思维链（CoT）和几次提示来增强也是如此。这些技术可能会导致法学硕士过度自信，并且在识别歧义方面只能产生边际增强。此外，由于缺乏冲突解决和固有知识的不准确利用，目前的法学硕士在提出高质量的澄清问题方面存在不足。在本文中，CLAMBER 提出了指导意见，并促进了对积极主动且值得信赖的法学硕士的进一步研究。我们的数据集可通过此 https URL 获取

Title: Distributional Semantics, Holism, and the Instability of Meaning

• Authors: Jumbly Grindrod, J.D. Porter, Nat Hansen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.12084
• Pdf URL: https://arxiv.org/pdf/2405.12084
• Copy Paste: [[2405.12084]] Distributional Semantics, Holism, and the Instability of Meaning(https://arxiv.org/abs/2405.12084)
• Keywords: language model
• Abstract: Current language models are built on the so-called distributional semantic approach to linguistic meaning that has the distributional hypothesis at its core. The distributional hypothesis involves a holistic conception of word meaning: the meaning of a word depends upon its relations to other words in the model. A standard objection to meaning holism is the charge of instability: any change in the meaning properties of a linguistic system (a human speaker, for example) would lead to many changes or possibly a complete change in the entire system. When the systems in question are trying to communicate with each other, it has been argued that instability of this kind makes communication impossible (Fodor and Lepore 1992, 1996, 1999). In this article, we examine whether the instability objection poses a problem for distributional models of meaning. First, we distinguish between distinct forms of instability that these models could exhibit, and we argue that only one such form is relevant for understanding the relation between instability and communication: what we call differential instability. Differential instability is variation in the relative distances between points in a space, rather than variation in the absolute position of those points. We distinguish differential and absolute instability by constructing two of our own models, a toy model constructed from the text of two novels, and a more sophisticated model constructed using the Word2vec algorithm from a combination of Wikipedia and SEP articles. We demonstrate the two forms of instability by showing how these models change as the corpora they are constructed from increase in size.
• 摘要：当前的语言模型建立在所谓的语言意义分布语义方法的基础上，该方法以分布假设为核心。分布假设涉及单词含义的整体概念：单词的含义取决于它与模型中其他单词的关系。对意义整体论的标准反对意见是对不稳定性的指控：语言系统（例如，人类说话者）的意义属性的任何变化都会导致整个系统的许多变化，甚至可能导致整个系统的彻底变化。当相关系统试图相互通信时，有人认为这种不稳定性使得通信变得不可能（Fodor and Lepore 1992、1996、1999）。在本文中，我们研究了不稳定性反对是否给意义分配模型带来了问题。首先，我们区分这些模型可能表现出的不同形式的不稳定性，并且我们认为只有一种形式与理解不稳定性和通信之间的关系相关：我们称之为差异不稳定性。微分不稳定性是空间中点之间相对距离的变化，而不是这些点绝对位置的变化。我们通过构建两个自己的模型来区分微分不稳定性和绝对不稳定性，一个是根据两本小说的文本构建的玩具模型，另一个是使用结合维基百科和 SEP 文章的 Word2vec 算法构建的更复杂的模型。我们通过展示这些模型如何随着它们构建的语料库大小的增加而变化，从而证明了两种形式的不稳定性。

Title: DOP: Diagnostic-Oriented Prompting for Large Language Models in Mathematical Correction

• Authors: Hao Chen, Biaojie Zeng, Xin Lin, Liang He, Aimin Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.12100
• Pdf URL: https://arxiv.org/pdf/2405.12100
• Copy Paste: [[2405.12100]] DOP: Diagnostic-Oriented Prompting for Large Language Models in Mathematical Correction(https://arxiv.org/abs/2405.12100)
• Keywords: language model, llm, prompt
• Abstract: Math world problems correction(MWPC) is a novel task dedicated to rectifying reasoning errors in the process of solving mathematical problems. In this paper, leveraging the advancements in large language models (LLMs), we address two key objectives:(1) Distinguishing between mathematical reasoning and error correction; (2) Exploring strategies to enhance the error correction capabilities of LLMs in mathematics to solve MWPC task. We noticed that, in real-time education,assisting students in recognizing their mistakes is more crucial than simply providing correct answers. However, current research tends to prioritize obtaining accurate solutions to math problems rather than correcting potentially incorrect ones. Therefore, we modify the research paradigm, demonstrating that improving mathematical reasoning abilities does not equate to mastery in error correction. Meanwhile, we propose a novel method called diagnostic-oriented promping(DOP) aimed at facilitating LLMs to excel in error correction. In experiments, DOP has shown outstanding performance, highlighting its significant impact. We argue that in mathematical education, the demand for outstanding correctors surpasses that for proficient reasoners. Codes and data are available on this https URL.
• 摘要：数学世界问题纠正（MWPC）是一项致力于纠正解决数学问题过程中的推理错误的新任务。在本文中，我们利用大型语言模型（LLM）的进步，解决了两个关键目标：（1）区分数学推理和错误纠正；（2）探索增强数学 LLM 纠错能力的策略以解决 MWPC 任务。我们注意到，在实时教育中，帮助学生认识到自己的错误比简单地提供正确答案更为重要。然而，当前的研究倾向于优先获得数学问题的准确解决方案，而不是纠正可能不正确的解决方案。因此，我们修改了研究范式，表明提高数学推理能力并不等同于掌握纠错能力。同时，我们提出了一种称为诊断导向提示（DOP）的新方法，旨在促进 LLM 在纠错方面表现出色。在实验中，DOP 表现出色，凸显了其重要影响。我们认为，在数学教育中，对优秀纠错者的需求超过了对熟练推理者的需求。代码和数据可在此 https URL 上获取。

Title: MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning

• Authors: Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/
• Pdf URL: https://arxiv.org/pdf/
• Copy Paste: [[]] MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning(https://arxiv.org/abs/)
• Keywords: language model, llm
• Abstract: Low-rank adaptation is a popular parameter-efficient fine-tuning method for large language models. In this paper, we analyze the impact of low-rank updating, as implemented in LoRA. Our findings suggest that the low-rank updating mechanism may limit the ability of LLMs to effectively learn and memorize new knowledge. Inspired by this observation, we propose a new method called MoRA, which employs a square matrix to achieve high-rank updating while maintaining the same number of trainable parameters. To achieve it, we introduce the corresponding non-parameter operators to reduce the input dimension and increase the output dimension for the square matrix. Furthermore, these operators ensure that the weight can be merged back into LLMs, which makes our method can be deployed like LoRA. We perform a comprehensive evaluation of our method across five tasks: instruction tuning, mathematical reasoning, continual pretraining, memory and pretraining. Our method outperforms LoRA on memory-intensive tasks and achieves comparable performance on other tasks.
• 摘要：低秩适应是一种流行的大型语言模型参数高效微调方法。在本文中，我们分析了 LoRA 中实现的低秩更新的影响。我们的研究结果表明，低等级更新机制可能会限制法学硕士有效学习和记忆新知识的能力。受这一观察的启发，我们提出了一种名为 MoRA 的新方法，它采用方阵来实现高秩更新，同时保持相同数量的可训练参数。为了实现这一点，我们引入相应的非参数运算符来减少方阵的输入维度并增加输出维度。此外，这些算子确保权重可以合并回 LLM，这使得我们的方法可以像 LoRA 一样部署。我们对五项任务的方法进行了全面评估：指令调整、数学推理、持续预训练、记忆和预训练。我们的方法在内存密集型任务上优于 LoRA，并在其他任务上实现了相当的性能。

Title: Fennec: Fine-grained Language Model Evaluation and Correction Extended through Branching and Bridging

• Authors: Xiaobo Liang, Haoke Zhang, Helan hu, Juntao Li, Jun Xu, Min Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.12163
• Pdf URL: https://arxiv.org/pdf/2405.12163
• Copy Paste: [[2405.12163]] Fennec: Fine-grained Language Model Evaluation and Correction Extended through Branching and Bridging(https://arxiv.org/abs/2405.12163)
• Keywords: language model, gpt
• Abstract: The rapid advancement of large language models has given rise to a plethora of applications across a myriad of real-world tasks, mainly centered on aligning with human intent. However, the complexities inherent in human intent necessitate a dependence on labor-intensive and time-consuming human evaluation. To alleviate this constraint, we delve into the paradigm of employing open-source large language models as evaluators, aligning with the prevailing trend of utilizing GPT-4. Particularly, we present a step-by-step evaluation framework: \textbf{Fennec}, capable of \textbf{F}ine-grained \textbf{E}valuatio\textbf{N} and correctio\textbf{N} \textbf{E}xtended through bran\textbf{C}hing and bridging. Specifically, the branching operation dissects the evaluation task into various dimensions and granularities, thereby alleviating the challenges associated with evaluation. Concurrently, the bridging operation amalgamates diverse training datasets, augmenting the variety of evaluation tasks. In experimental trials, our 7B model consistently outperforms open-source larger-scale evaluation models across various widely adopted benchmarks in terms of both \textit{Agreement} and \textit{Consistency}, closely approaching the capabilities of GPT-4. We employ the fine-grained correction capabilities induced by the evaluation model to refine multiple model responses, and the results show that the refinement elevates the quality of responses, leading to an improvement of 1-2 points on the MT-Bench. Our code is available at Github\footnote{\url{this https URL}}.
• 摘要：大型语言模型的快速发展催生了无数现实世界任务中的大量应用程序，主要集中在与人类意图保持一致。然而，人类意图固有的复杂性使得必须依赖劳动密集型且耗时的人类评估。为了缓解这一限制，我们深入研究了使用开源大型语言模型作为评估器的范式，这与利用 GPT-4 的流行趋势保持一致。特别是，我们提出了一个逐步评估框架：\textbf{Fennec}，能够进行\textbf{F}细粒度\textbf{E}valuatio\textbf{N}和校正\textbf{N}\textbf{ E}通过bran\textbf{C}hing和桥接进行扩展。具体来说，分支操作将评估任务分解为不同的维度和粒度，从而减轻与评估相关的挑战。同时，桥接操作合并了不同的训练数据集，增加了评估任务的多样性。在实验试验中，我们的 7B 模型在各种广泛采用的基准中，在 \textit{Agreement} 和 \textit{Consistency} 方面始终优于开源的大规模评估模型，非常接近 GPT-4 的功能。我们利用评估模型带来的细粒度校正能力来细化多个模型响应，结果表明，细化提高了响应的质量，导致 MT-Bench 提高了 1-2 个点。我们的代码可在 Github\footnote{\url{this https URL}} 获取。

Title: CT-Eval: Benchmarking Chinese Text-to-Table Performance in Large Language Models

• Authors: Haoxiang Shi, Jiaan Wang, Jiarong Xu, Cen Wang, Tetsuya Sakai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.12174
• Pdf URL: https://arxiv.org/pdf/2405.12174
• Copy Paste: [[2405.12174]] CT-Eval: Benchmarking Chinese Text-to-Table Performance in Large Language Models(https://arxiv.org/abs/2405.12174)
• Keywords: language model, gpt, llm, hallucination, chat
• Abstract: Text-to-Table aims to generate structured tables to convey the key information from unstructured documents. Existing text-to-table datasets are typically oriented English, limiting the research in non-English languages. Meanwhile, the emergence of large language models (LLMs) has shown great success as general task solvers in multi-lingual settings (e.g., ChatGPT), theoretically enabling text-to-table in other languages. In this paper, we propose a Chinese text-to-table dataset, CT-Eval, to benchmark LLMs on this task. Our preliminary analysis of English text-to-table datasets highlights two key factors for dataset construction: data diversity and data hallucination. Inspired by this, the CT-Eval dataset selects a popular Chinese multidisciplinary online encyclopedia as the source and covers 28 domains to ensure data diversity. To minimize data hallucination, we first train an LLM to judge and filter out the task samples with hallucination, then employ human annotators to clean the hallucinations in the validation and testing sets. After this process, CT-Eval contains 88.6K task samples. Using CT-Eval, we evaluate the performance of open-source and closed-source LLMs. Our results reveal that zero-shot LLMs (including GPT-4) still have a significant performance gap compared with human judgment. Furthermore, after fine-tuning, open-source LLMs can significantly improve their text-to-table ability, outperforming GPT-4 by a large margin. In short, CT-Eval not only helps researchers evaluate and quickly understand the Chinese text-to-table ability of existing LLMs but also serves as a valuable resource to significantly improve the text-to-table performance of LLMs.
• 摘要：文本到表格的目的是生成结构化表格来传达非结构化文档中的关键信息。现有的文本到表格数据集通常是面向英语的，限制了非英语语言的研究。与此同时，大型语言模型（LLM）的出现作为多语言环境中的通用任务解决器（例如 ChatGPT）取得了巨大成功，理论上可以实现其他语言的文本到表格。在本文中，我们提出了一个中文文本到表格数据集 CT-Eval，以在该任务上对法学硕士进行基准测试。我们对英语文本到表格数据集的初步分析强调了数据集构建的两个关键因素：数据多样性和数据幻觉。受此启发，CT-Eval数据集选择中国流行的多学科在线百科全书作为来源，涵盖28个领域，以确保数据多样性。为了最大限度地减少数据幻觉，我们首先训练法学硕士来判断和过滤掉具有幻觉的任务样本，然后使用人类注释者来清理验证和测试集中的幻觉。经过此过程，CT-Eval 包含 88.6K 任务样本。使用 CT-Eval，我们评估开源和闭源法学硕士的表现。我们的结果表明，零样本 LLM（包括 GPT-4）与人类判断相比仍然存在显着的性能差距。此外，经过微调，开源LLM可以显着提高其文本到表格的能力，大幅优于GPT-4。总之，CT-Eval不仅可以帮助研究人员评估和快速了解现有法学硕士的中文文本到表能力，而且可以作为显着提高法学硕士文本到表性能的宝贵资源。

Title: MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark

• Authors: Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou, Wenwei Zhang, Songyang Zhang, Dahua Lin, Kai Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.12209
• Pdf URL: https://arxiv.org/pdf/2405.12209
• Copy Paste: [[2405.12209]] MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark(https://arxiv.org/abs/2405.12209)
• Keywords: language model, llm
• Abstract: Recent advancements in large language models (LLMs) have showcased significant improvements in mathematics. However, traditional math benchmarks like GSM8k offer a unidimensional perspective, falling short in providing a holistic assessment of the LLMs' math capabilities. To address this gap, we introduce MathBench, a new benchmark that rigorously assesses the mathematical capabilities of large language models. MathBench spans a wide range of mathematical disciplines, offering a detailed evaluation of both theoretical understanding and practical problem-solving skills. The benchmark progresses through five distinct stages, from basic arithmetic to college mathematics, and is structured to evaluate models at various depths of knowledge. Each stage includes theoretical questions and application problems, allowing us to measure a model's mathematical proficiency and its ability to apply concepts in practical scenarios. MathBench aims to enhance the evaluation of LLMs' mathematical abilities, providing a nuanced view of their knowledge understanding levels and problem solving skills in a bilingual context. The project is released at this https URL .
• 摘要：大型语言模型 (LLM) 的最新进展展示了数学方面的显着进步。然而，像 GSM8k 这样的传统数学基准提供了单一维度的视角，无法对法学硕士的数学能力进行全面评估。为了解决这一差距，我们引入了 MathBench，这是一个新的基准，可以严格评估大型语言模型的数学能力。 MathBench 涵盖广泛的数学学科，提供对理论理解和实际解决问题技能的详细评估。该基准测试经历了从基础算术到大学数学的五个不同阶段，并且旨在评估不同知识深度的模型。每个阶段都包括理论问题和应用问题，使我们能够衡量模型的数学熟练程度及其在实际场景中应用概念的能力。 MathBench 旨在加强对法学硕士数学能力的评估，提供双语背景下他们的知识理解水平和问题解决能力的细致入微的视角。该项目在此 https URL 发布。