2024-03-19

Title: Neural Erosion: Emulating Controlled Neurodegeneration and Aging in AI Systems

• Authors: Antonios Alexos, Yu-Dai Tsai, Ian Domingo, Maryam Pishgar, Pierre Baldi
• Subjects: cs.CL, cs.AI, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2403.10596
• Pdf URL: https://arxiv.org/pdf/2403.10596
• Copy Paste: [[2403.10596]] Neural Erosion: Emulating Controlled Neurodegeneration and Aging in AI Systems(https://arxiv.org/abs/2403.10596)
• Keywords: language model, llm, prompt
• Abstract: Creating controlled methods to simulate neurodegeneration in artificial intelligence (AI) is crucial for applications that emulate brain function decline and cognitive disorders. We use IQ tests performed by Large Language Models (LLMs) and, more specifically, the LLaMA 2 to introduce the concept of ``neural erosion." This deliberate erosion involves ablating synapses or neurons, or adding Gaussian noise during or after training, resulting in a controlled progressive decline in the LLMs' performance. We are able to describe the neurodegeneration in the IQ tests and show that the LLM first loses its mathematical abilities and then its linguistic abilities, while further losing its ability to understand the questions. To the best of our knowledge, this is the first work that models neurodegeneration with text data, compared to other works that operate in the computer vision domain. Finally, we draw similarities between our study and cognitive decline clinical studies involving test subjects. We find that with the application of neurodegenerative methods, LLMs lose abstract thinking abilities, followed by mathematical degradation, and ultimately, a loss in linguistic ability, responding to prompts incoherently. These findings are in accordance with human studies.
• 摘要：创建受控方法来模拟人工智能 (AI) 中的神经退行性疾病对于模拟大脑功能衰退和认知障碍的应用至关重要。我们使用大型语言模型 (LLM) 执行的 IQ 测试，更具体地说，LLaMA 2 来引入“神经侵蚀”的概念。这种故意侵蚀涉及消融突触或神经元，或者在训练期间或之后添加高斯噪声，从而导致法学硕士的表现有控制地进行性下降。我们能够描述智商测试中的神经变性，并表明法学硕士首先失去了数学能力，然后失去了语言能力，同时进一步失去了理解问题的能力。据我们所知，与计算机视觉领域的其他工作相比，这是第一个用文本数据模拟神经退行性变的工作。最后，我们在我们的研究和涉及测试对象的认知衰退临床研究之间得出了相似之处。我们发现，在神经退行性方法的应用中，法学硕士失去了抽象思维能力，随后是数学退化，最后是语言能力丧失，对提示的反应不连贯。这些发现与人类研究一致。

Title: MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling

• Authors: Tomasz Limisiewicz, Terra Blevins, Hila Gonen, Orevaoghene Ahia, Luke Zettlemoyer
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.10691
• Pdf URL: https://arxiv.org/pdf/2403.10691
• Copy Paste: [[2403.10691]] MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling(https://arxiv.org/abs/2403.10691)
• Keywords: language model
• Abstract: A major consideration in multilingual language modeling is how to best represent languages with diverse vocabularies and scripts. Although contemporary text encoding methods cover most of the world's writing systems, they exhibit bias towards the high-resource languages of the Global West. As a result, texts of underrepresented languages tend to be segmented into long sequences of linguistically meaningless units. To address the disparities, we introduce a new paradigm that encodes the same information with segments of consistent size across diverse languages. Our encoding convention (MYTE) is based on morphemes, as their inventories are more balanced across languages than characters, which are used in previous methods. We show that MYTE produces shorter encodings for all 99 analyzed languages, with the most notable improvements for non-European languages and non-Latin scripts. This, in turn, improves multilingual LM performance and diminishes the perplexity gap throughout diverse languages.
• 摘要：多语言语言建模的一个主要考虑因素是如何最好地表示具有不同词汇和脚本的语言。尽管当代文本编码方法涵盖了世界上大多数书写系统，但它们对全球西方的高资源语言表现出偏见。因此，代表性不足的语言的文本往往被分割成语言上无意义的单元的长序列。为了解决这种差异，我们引入了一种新的范例，该范例可以在不同的语言中使用大小一致的片段对相同的信息进行编码。我们的编码约定（MYTE）基于语素，因为它们的库存在不同语言之间比以前方法中使用的字符更加平衡。我们表明，MYTE 为所有 99 种分析语言生成了更短的编码，其中对于非欧洲语言和非拉丁文字的改进最为显着。这反过来又提高了多语言 LM 的性能，并缩小了不同语言之间的困惑差距。

Title: EXPLORER: Exploration-guided Reasoning for Textual Reinforcement Learning

• Authors: Kinjal Basu, Keerthiram Murugesan, Subhajit Chaudhury, Murray Campbell, Kartik Talamadupula, Tim Klinger
• Subjects: cs.CL, cs.AI, cs.LO
• Abstract URL: https://arxiv.org/abs/2403.10692
• Pdf URL: https://arxiv.org/pdf/2403.10692
• Copy Paste: [[2403.10692]] EXPLORER: Exploration-guided Reasoning for Textual Reinforcement Learning(https://arxiv.org/abs/2403.10692)
• Keywords: agent
• Abstract: Text-based games (TBGs) have emerged as an important collection of NLP tasks, requiring reinforcement learning (RL) agents to combine natural language understanding with reasoning. A key challenge for agents attempting to solve such tasks is to generalize across multiple games and demonstrate good performance on both seen and unseen objects. Purely deep-RL-based approaches may perform well on seen objects; however, they fail to showcase the same performance on unseen objects. Commonsense-infused deep-RL agents may work better on unseen data; unfortunately, their policies are often not interpretable or easily transferable. To tackle these issues, in this paper, we present EXPLORER which is an exploration-guided reasoning agent for textual reinforcement learning. EXPLORER is neurosymbolic in nature, as it relies on a neural module for exploration and a symbolic module for exploitation. It can also learn generalized symbolic policies and perform well over unseen data. Our experiments show that EXPLORER outperforms the baseline agents on Text-World cooking (TW-Cooking) and Text-World Commonsense (TWC) games.
• 摘要：基于文本的游戏（TBG）已成为 NLP 任务的重要集合，要求强化学习（RL）代理将自然语言理解与推理结合起来。对于尝试解决此类任务的代理来说，一个关键挑战是在多个游戏中进行泛化，并在可见和不可见的对象上展示良好的性能。纯粹基于深度强化学习的方法可能在所见物体上表现良好；然而，他们未能在看不见的物体上展示出相同的性能。注入常​​识的深度强化学习代理可能在看不见的数据上表现得更好；不幸的是，他们的政策往往难以解释或轻易转移。为了解决这些问题，在本文中，我们提出了 EXPLORER，它是一种用于文本强化学习的探索引导推理代理。 EXPLORER 本质上是神经符号的，因为它依赖于神经模块进行探索和符号模块进行利用。它还可以学习广义的符号策略，并在未见过的数据上表现良好。我们的实验表明，EXPLORER 在文本世界烹饪 (TW-Cooking) 和文本世界常识 (TWC) 游戏上的表现优于基线代理。

Title: A Multilingual Perspective on Probing Gender Bias

• Authors: Karolina Stańczak
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.10699
• Pdf URL: https://arxiv.org/pdf/2403.10699
• Copy Paste: [[2403.10699]] A Multilingual Perspective on Probing Gender Bias(https://arxiv.org/abs/2403.10699)
• Keywords: language model
• Abstract: Gender bias represents a form of systematic negative treatment that targets individuals based on their gender. This discrimination can range from subtle sexist remarks and gendered stereotypes to outright hate speech. Prior research has revealed that ignoring online abuse not only affects the individuals targeted but also has broader societal implications. These consequences extend to the discouragement of women's engagement and visibility within public spheres, thereby reinforcing gender inequality. This thesis investigates the nuances of how gender bias is expressed through language and within language technologies. Significantly, this thesis expands research on gender bias to multilingual contexts, emphasising the importance of a multilingual and multicultural perspective in understanding societal biases. In this thesis, I adopt an interdisciplinary approach, bridging natural language processing with other disciplines such as political science and history, to probe gender bias in natural language and language models.
• 摘要：性别偏见是一种系统性的负面对待，根据性别来针对个人。这种歧视的范围可以从微妙的性别歧视言论和性别刻板印象到公然的仇恨言论。先前的研究表明，忽视网络虐待不仅会影响目标个人，还会产生更广泛的社会影响。这些后果阻碍了妇女在公共领域的参与和知名度，从而加剧了性别不平等。本论文研究了性别偏见如何通过语言和语言技术表达的细微差别。值得注意的是，本文将性别偏见的研究扩展到多语言背景，强调多语言和多文化视角在理解社会偏见方面的重要性。在这篇论文中，我采用跨学科的方法，将自然语言处理与政治学和历史学等其他学科联系起来，探讨自然语言和语言模型中的性别偏见。

Title: Uncovering Latent Themes of Messaging on Social Media by Integrating LLMs: A Case Study on Climate Campaigns

• Authors: Tunazzina Islam, Dan Goldwasser
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2403.10707
• Pdf URL: https://arxiv.org/pdf/2403.10707
• Copy Paste: [[2403.10707]] Uncovering Latent Themes of Messaging on Social Media by Integrating LLMs: A Case Study on Climate Campaigns(https://arxiv.org/abs/2403.10707)
• Keywords: language model, llm
• Abstract: This paper introduces a novel approach to uncovering and analyzing themes in social media messaging. Recognizing the limitations of traditional topic-level analysis, which tends to capture only the overarching patterns, this study emphasizes the need for a finer-grained, theme-focused exploration. Conventional methods of theme discovery, involving manual processes and a human-in-the-loop approach, are valuable but face challenges in scalability, consistency, and resource intensity in terms of time and cost. To address these challenges, we propose a machine-in-the-loop approach that leverages the advanced capabilities of Large Language Models (LLMs). This approach allows for a deeper investigation into the thematic aspects of social media discourse, enabling us to uncover a diverse array of themes, each with unique characteristics and relevance, thereby offering a comprehensive understanding of the nuances present within broader topics. Furthermore, this method efficiently maps the text and the newly discovered themes, enhancing our understanding of the thematic nuances in social media messaging. We employ climate campaigns as a case study and demonstrate that our methodology yields more accurate and interpretable results compared to traditional topic models. Our results not only demonstrate the effectiveness of our approach in uncovering latent themes but also illuminate how these themes are tailored for demographic targeting in social media contexts. Additionally, our work sheds light on the dynamic nature of social media, revealing the shifts in the thematic focus of messaging in response to real-world events.
• 摘要：本文介绍了一种发现和分析社交媒体消息传递主题的新颖方法。认识到传统主题级分析往往仅捕获总体模式的局限性，本研究强调需要进行更细粒度、以主题为中心的探索。传统的主题发现方法（涉及手动流程和人机交互方法）很有价值，但在时间和成本方面面临可扩展性、一致性和资源强度方面的挑战。为了应对这些挑战，我们提出了一种机器在环方法，该方法利用大型语言模型 (LLM) 的高级功能。这种方法可以更深入地调查社交媒体话语的主题方面，使我们能够发现各种不同的主题，每个主题都具有独特的特征和相关性，从而全面了解更广泛主题中存在的细微差别。此外，这种方法有效地映射了文本和新发现的主题，增强了我们对社交媒体消息传递中主题细微差别的理解。我们采用气候活动作为案例研究，并证明与传统主题模型相比，我们的方法可以产生更准确和可解释的结果。我们的结果不仅证明了我们的方法在发现潜在主题方面的有效性，而且还阐明了如何针对社交媒体环境中的人口目标定制这些主题。此外，我们的工作揭示了社交媒体的动态本质，揭示了针对现实世界事件的消息传递主题焦点的变化。

Title: Depression Detection on Social Media with Large Language Models

• Authors: Xiaochong Lan, Yiming Cheng, Li Sheng, Chen Gao, Yong Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.10750
• Pdf URL: https://arxiv.org/pdf/2403.10750
• Copy Paste: [[2403.10750]] Depression Detection on Social Media with Large Language Models(https://arxiv.org/abs/2403.10750)
• Keywords: language model, llm
• Abstract: Depression harms. However, due to a lack of mental health awareness and fear of stigma, many patients do not actively seek diagnosis and treatment, leading to detrimental outcomes. Depression detection aims to determine whether an individual suffers from depression by analyzing their history of posts on social media, which can significantly aid in early detection and intervention. It mainly faces two key challenges: 1) it requires professional medical knowledge, and 2) it necessitates both high accuracy and explainability. To address it, we propose a novel depression detection system called DORIS, combining medical knowledge and the recent advances in large language models (LLMs). Specifically, to tackle the first challenge, we proposed an LLM-based solution to first annotate whether high-risk texts meet medical diagnostic criteria. Further, we retrieve texts with high emotional intensity and summarize critical information from the historical mood records of users, so-called mood courses. To tackle the second challenge, we combine LLM and traditional classifiers to integrate medical knowledge-guided features, for which the model can also explain its prediction results, achieving both high accuracy and explainability. Extensive experimental results on benchmarking datasets show that, compared to the current best baseline, our approach improves by 0.036 in AUPRC, which can be considered significant, demonstrating the effectiveness of our approach and its high value as an NLP application.
• 摘要：抑郁症有害。然而，由于缺乏心理健康意识和害怕耻辱，许多患者不积极寻求诊断和治疗，导致不良后果。抑郁症检测旨在通过分析一个人在社交媒体上的帖子历史来确定一个人是否患有抑郁症，这对早期发现和干预有很大帮助。它主要面临两个关键挑战：1）需要专业的医学知识，2）需要高精度和可解释性。为了解决这个问题，我们提出了一种名为 DORIS 的新型抑郁症检测系统，该系统结合了医学知识和大语言模型 (LLM) 的最新进展。具体来说，为了解决第一个挑战，我们提出了一种基于LLM的解决方案，首先注释高风险文本是否符合医学诊断标准。此外，我们检索情绪强度高的文本，并从用户的历史情绪记录中总结关键信息，即所谓的情绪课程。为了解决第二个挑战，我们将LLM和传统分类器结合起来，整合医学知识引导的特征，模型也可以解释其预测结果，实现高精度和可解释性。在基准数据集上的大量实验结果表明，与当前最佳基线相比，我们的方法在 AUPRC 中提高了 0.036，这可以认为是显着的，证明了我们方法的有效性及其作为 NLP 应用的高价值。

Title: Detecting Bias in Large Language Models: Fine-tuned KcBERT

• Authors: J. K. Lee, T. M. Chung
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.10774
• Pdf URL: https://arxiv.org/pdf/2403.10774
• Copy Paste: [[2403.10774]] Detecting Bias in Large Language Models: Fine-tuned KcBERT(https://arxiv.org/abs/2403.10774)
• Keywords: language model, llm
• Abstract: The rapid advancement of large language models (LLMs) has enabled natural language processing capabilities similar to those of humans, and LLMs are being widely utilized across various societal domains such as education and healthcare. While the versatility of these models has increased, they have the potential to generate subjective and normative language, leading to discriminatory treatment or outcomes among social groups, especially due to online offensive language. In this paper, we define such harm as societal bias and assess ethnic, gender, and racial biases in a model fine-tuned with Korean comments using Bidirectional Encoder Representations from Transformers (KcBERT) and KOLD data through template-based Masked Language Modeling (MLM). To quantitatively evaluate biases, we employ LPBS and CBS metrics. Compared to KcBERT, the fine-tuned model shows a reduction in ethnic bias but demonstrates significant changes in gender and racial biases. Based on these results, we propose two methods to mitigate societal bias. Firstly, a data balancing approach during the pre-training phase adjusts the uniformity of data by aligning the distribution of the occurrences of specific words and converting surrounding harmful words into non-harmful words. Secondly, during the in-training phase, we apply Debiasing Regularization by adjusting dropout and regularization, confirming a decrease in training loss. Our contribution lies in demonstrating that societal bias exists in Korean language models due to language-dependent characteristics.
• 摘要：大语言模型（LLM）的快速发展使得自然语言处理能力与人类相似，并且LLM正在教育、医疗等各个社会领域得到广泛应用。虽然这些模型的多功能性有所增加，但它们有可能产生主观和规范性语言，导致社会群体之间的歧视性待遇或结果，特别是由于网络攻击性语言。在本文中，我们将此类危害定义为社会偏见，并在一个模型中评估种族、性别和种族偏见，该模型使用 Transformers 双向编码器表示 (KcBERT) 和 KOLD 数据通过基于模板的掩码语言模型 (MLM) 与韩国评论进行微调）。为了定量评估偏差，我们采用 LPBS 和 CBS 指标。与 KcBERT 相比，微调模型显示种族偏见有所减少，但性别和种族偏见发生了显着变化。根据这些结果，我们提出了两种减轻社会偏见的方法。首先，预训练阶段的数据平衡方法通过对齐特定单词出现的分布并将周围的有害单词转换为无害单词来调整数据的均匀性。其次，在训练阶段，我们通过调整 dropout 和正则化来应用去偏正则化，确认训练损失的减少。我们的贡献在于证明由于语言依赖的特征，韩语语言模型中存在社会偏见。

Title: LLM-based Conversational AI Therapist for Daily Functioning Screening and Psychotherapeutic Intervention via Everyday Smart Devices

• Authors: Jingping Nie, Hanya Shao, Yuang Fan, Qijia Shao, Haoxuan You, Matthias Preindl, Xiaofan Jiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.10779
• Pdf URL: https://arxiv.org/pdf/2403.10779
• Copy Paste: [[2403.10779]] LLM-based Conversational AI Therapist for Daily Functioning Screening and Psychotherapeutic Intervention via Everyday Smart Devices(https://arxiv.org/abs/2403.10779)
• Keywords: language model, llm
• Abstract: Despite the global mental health crisis, access to screenings, professionals, and treatments remains high. In collaboration with licensed psychotherapists, we propose a Conversational AI Therapist with psychotherapeutic Interventions (CaiTI), a platform that leverages large language models (LLM)s and smart devices to enable better mental health self-care. CaiTI can screen the day-to-day functioning using natural and psychotherapeutic conversations. CaiTI leverages reinforcement learning to provide personalized conversation flow. CaiTI can accurately understand and interpret user responses. When the user needs further attention during the conversation, CaiTI can provide conversational psychotherapeutic interventions, including cognitive behavioral therapy (CBT) and motivational interviewing (MI). Leveraging the datasets prepared by the licensed psychotherapists, we experiment and microbenchmark various LLMs' performance in tasks along CaiTI's conversation flow and discuss their strengths and weaknesses. With the psychotherapists, we implement CaiTI and conduct 14-day and 24-week studies. The study results, validated by therapists, demonstrate that CaiTI can converse with users naturally, accurately understand and interpret user responses, and provide psychotherapeutic interventions appropriately and effectively. We showcase the potential of CaiTI LLMs to assist the mental therapy diagnosis and treatment and improve day-to-day functioning screening and precautionary psychotherapeutic intervention systems.
• 摘要：尽管存在全球精神健康危机，但获得筛查、专业人士和治疗的机会仍然很高。我们与有执照的心理治疗师合作，提出了一种具有心理治疗干预功能的对话式人工智能治疗师 (CaiTI)，这是一个利用大语言模型 (LLM) 和智能设备来实现更好的心理健康自我护理的平台。 CaiTI 可以使用自然和心理治疗对话来筛选日常功能。 CaiTI 利用强化学习来提供个性化的对话流程。 CaiTI能够准确理解和解读用户的反应。当用户在对话过程中需要进一步关注时，CaiTI可以提供对话心理治疗干预，包括认知行为治疗（CBT）和动机访谈（MI）。利用持证心理治疗师准备的数据集，我们对各种法学硕士在 CaiTI 对话流程中的任务表现进行实验和微基准测试，并讨论他们的优点和缺点。我们与心理治疗师一起实施 CaiTI 并进行为期 14 天和 24 周的研究。研究结果经治疗师验证，表明CaiTI可以与用户自然地交谈，准确理解和解释用户的反应，并提供适当有效的心理治疗干预。我们展示了 CaiTI 法学硕士在协助心理治疗诊断和治疗以及改善日常功能筛查和预防性心理治疗干预系统方面的潜力。

Title: Exploring Chinese Humor Generation: A Study on Two-Part Allegorical Sayings

• Authors: Rongwu Xu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.10781
• Pdf URL: https://arxiv.org/pdf/2403.10781
• Copy Paste: [[2403.10781]] Exploring Chinese Humor Generation: A Study on Two-Part Allegorical Sayings(https://arxiv.org/abs/2403.10781)
• Keywords: language model, prompt
• Abstract: Humor, a culturally nuanced aspect of human language, poses challenges for computational understanding and generation, especially in Chinese humor, which remains relatively unexplored in the NLP community. This paper investigates the capability of state-of-the-art language models to comprehend and generate Chinese humor, specifically focusing on training them to create allegorical sayings. We employ two prominent training methods: fine-tuning a medium-sized language model and prompting a large one. Our novel fine-tuning approach incorporates fused Pinyin embeddings to consider homophones and employs contrastive learning with synthetic hard negatives to distinguish humor elements. Human-annotated results show that these models can generate humorous allegorical sayings, with prompting proving to be a practical and effective method. However, there is still room for improvement in generating allegorical sayings that match human creativity.
• 摘要：幽默是人类语言中文化上微妙的一个方面，对计算理解和生成提出了挑战，特别是在中国幽默中，这在 NLP 社区中仍然相对未经探索。本文研究了最先进的语言模型理解和生成中国幽默的能力，特别侧重于训练它们创造寓言谚语。我们采用两种著名的训练方法：微调中型语言模型和提示大型语言模型。我们新颖的微调方法结合了融合拼音嵌入来考虑同音词，并采用对比学习和合成硬底片来区分幽默元素。人工注释的结果表明，这些模型可以生成幽默的寓言谚语，并提示被证明是一种实用且有效的方法。然而，在生成与人类创造力相匹配的寓言谚语方面仍有改进的空间。

Title: From Words to Routes: Applying Large Language Models to Vehicle Routing

• Authors: Zhehui Huang, Guangyao Shi, Gaurav S. Sukhatme
• Subjects: cs.CL, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.10795
• Pdf URL: https://arxiv.org/pdf/2403.10795
• Copy Paste: [[2403.10795]] From Words to Routes: Applying Large Language Models to Vehicle Routing(https://arxiv.org/abs/2403.10795)
• Keywords: language model, gpt, llm, prompt
• Abstract: LLMs have shown impressive progress in robotics (e.g., manipulation and navigation) with natural language task descriptions. The success of LLMs in these tasks leads us to wonder: What is the ability of LLMs to solve vehicle routing problems (VRPs) with natural language task descriptions? In this work, we study this question in three steps. First, we construct a dataset with 21 types of single- or multi-vehicle routing problems. Second, we evaluate the performance of LLMs across four basic prompt paradigms of text-to-code generation, each involving different types of text input. We find that the basic prompt paradigm, which generates code directly from natural language task descriptions, performs the best for GPT-4, achieving 56% feasibility, 40% optimality, and 53% efficiency. Third, based on the observation that LLMs may not be able to provide correct solutions at the initial attempt, we propose a framework that enables LLMs to refine solutions through self-reflection, including self-debugging and self-verification. With GPT-4, our proposed framework achieves a 16% increase in feasibility, a 7% increase in optimality, and a 15% increase in efficiency. Moreover, we examine the sensitivity of GPT-4 to task descriptions, specifically focusing on how its performance changes when certain details are omitted from the task descriptions, yet the core meaning is preserved. Our findings reveal that such omissions lead to a notable decrease in performance: 4% in feasibility, 4% in optimality, and 5% in efficiency. Website: https://sites.google.com/view/words-to-routes/
• 摘要：法学硕士在自然语言任务描述的机器人技术（例如操纵和导航）方面取得了令人瞩目的进展。法学硕士在这些任务中的成功让我们想知道：法学硕士用自然语言任务描述解决车辆路径问题（VRP）的能力如何？在这项工作中，我们分三个步骤研究这个问题。首先，我们构建了一个包含 21 种单车或多车路径问题的数据集。其次，我们评估了法学硕士在文本到代码生成的四种基本提示范例中的表现，每种范例都涉及不同类型的文本输入。我们发现直接从自然语言任务描述生成代码的基本提示范式在 GPT-4 中表现最佳，实现了 56% 的可行性、40% 的最优性和 53% 的效率。第三，基于LLMs可能无法在初次尝试时提供正确解决方案的观察，我们提出了一个框架，使LLMs能够通过自我反思（包括自我调试和自我验证）来完善解决方案。通过 GPT-4，我们提出的框架实现了可行性提高 16%、优化性提高 7%、效率提高 15%。此外，我们检查了 GPT-4 对任务描述的敏感性，特别关注当任务描述中省略某些细节但保留核心含义时，其性能如何变化。我们的研究结果表明，此类遗漏会导致性能显着下降：可行性下降 4%，最优性下降 4%，效率下降 5%。网站：https://sites.google.com/view/words-to-routes/

Title: Efficient Pruning of Large Language Model with Adaptive Estimation Fusion

• Authors: Jun Liu, Chao Wu, Changdi Yang, Hao Tang, Haoye Dong, Zhenglun Kong, Geng Yuan, Wei Niu, Dong Huang, Yanzhi Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.10799
• Pdf URL: https://arxiv.org/pdf/2403.10799
• Copy Paste: [[2403.10799]] Efficient Pruning of Large Language Model with Adaptive Estimation Fusion(https://arxiv.org/abs/2403.10799)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have become crucial for many generative downstream tasks, leading to an inevitable trend and significant challenge to deploy them efficiently on resource-constrained devices. Structured pruning is a widely used method to address this challenge. However, when dealing with the complex structure of the multiple decoder layers, general methods often employ common estimation approaches for pruning. These approaches lead to a decline in accuracy for specific downstream tasks. In this paper, we introduce a simple yet efficient method that adaptively models the importance of each substructure. Meanwhile, it can adaptively fuse coarse-grained and finegrained estimations based on the results from complex and multilayer structures. All aspects of our design seamlessly integrate into the endto-end pruning framework. Our experimental results, compared with state-of-the-art methods on mainstream datasets, demonstrate average accuracy improvements of 1.1%, 1.02%, 2.0%, and 1.2% for LLaMa-7B,Vicuna-7B, Baichuan-7B, and Bloom-7b1, respectively.
• 摘要：大型语言模型（LLM）对于许多生成性下游任务至关重要，这导致了不可避免的趋势和在资源有限的设备上有效部署它们的重大挑战。结构化修剪是解决这一挑战的广泛使用的方法。然而，当处理多个解码器层的复杂结构时，一般方法通常采用常见的估计方法进行剪枝。这些方法会导致特定下游任务的准确性下降。在本文中，我们介绍了一种简单而有效的方法，可以自适应地对每个子结构的重要性进行建模。同时，它可以根据复杂和多层结构的结果自适应地融合粗粒度和细粒度估计。我们设计的所有方面都无缝集成到端到端修剪框架中。与主流数据集上最先进的方法相比，我们的实验结果表明 LLaMa-7B、Vicuna-7B、Baichuan-7B 和 Bloom 的平均准确度提高了 1.1%、1.02%、2.0% 和 1.2%分别为-7b1。

Title: Do Large Language Models understand Medical Codes?

• Authors: Simon A. Lee, Timothy Lindsey
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.10822
• Pdf URL: https://arxiv.org/pdf/2403.10822
• Copy Paste: [[2403.10822]] Do Large Language Models understand Medical Codes?(https://arxiv.org/abs/2403.10822)
• Keywords: language model, gpt, llm, hallucination, prompt
• Abstract: The overarching goal of recent AI research has been to make steady progress towards achieving Artificial General Intelligence (AGI), prompting the evaluation of Large Language Models (LLMs) across a variety of tasks and domains. One such domain is healthcare, where LLMs can greatly benefit clinical practice by assisting with a wide range of tasks. However, these models are also prone to producing "hallucinations" or incorrect responses when faced with queries they cannot adequately address, raising concerns and skepticism, especially within the healthcare community. Therefore, in this work, we investigate whether LLMs understand the inherent meaning of medical codes, which are widely used in healthcare practice. We evaluate various off-the-shelf LLMs (e.g., GPT, LLaMA, etc.) and LLMs specifically designed for biomedical applications to assess their awareness and understanding of these domain-specific terminologies. Our results indicate that these models do not comprehend the meaning of the medical codes, highlighting the need for better representation of these alphanumeric codes extensively used in healthcare. We call for improved strategies to effectively capture and represent the nuances of medical codes and terminologies within LLMs, enabling them to become more reliable and trustworthy tools for healthcare professionals.
• 摘要：最近人工智能研究的总体目标是在实现通用人工智能 (AGI) 方面取得稳步进展，促进跨各种任务和领域对大型语言模型 (LLM) 的评估。其中一个领域是医疗保健，法学硕士可以通过协助完成广泛的任务来极大地有益于临床实践。然而，当面对它们无法充分解决的问题时，这些模型也容易产生“幻觉”或不正确的反应，从而引起担忧和怀疑，尤其是在医疗保健界。因此，在这项工作中，我们调查法学硕士是否理解医疗保健实践中广泛使用的医疗代码的内在含义。我们评估各种现成的法学硕士（例如 GPT、LLaMA 等）和专门为生物医学应用设计的法学硕士，以评估他们对这些特定领域术语的认识和理解。我们的结果表明，这些模型不理解医疗代码的含义，强调需要更好地表示这些在医疗保健中广泛使用的字母数字代码。我们呼吁改进策略，以有效捕捉和表达法学硕士内医学规范和术语的细微差别，使它们成为医疗保健专业人员更可靠和值得信赖的工具。

Title: Multi-party Response Generation with Relation Disentanglement

• Authors: Tianhao Dai, Chengyu Huang, Lizi Liao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.10827
• Pdf URL: https://arxiv.org/pdf/2403.10827
• Copy Paste: [[2403.10827]] Multi-party Response Generation with Relation Disentanglement(https://arxiv.org/abs/2403.10827)
• Keywords: chat
• Abstract: Existing neural response generation models have achieved impressive improvements for two-party conversations, which assume that utterances are sequentially organized. However, many real-world dialogues involve multiple interlocutors and the structure of conversational context is much more complex, e.g. utterances from different interlocutors can occur "in parallel". Facing this challenge, there are works trying to model the relations among utterances or interlocutors to facilitate response generation with clearer context. Nonetheless, these methods rely heavily on such relations and all assume that these are given beforehand, which is impractical and hinders the generality of such methods. In this work, we propose to automatically infer the relations via relational thinking on subtle clues inside the conversation context without any human label, and leverage these relations to guide the neural response generation. Specifically, we first apply a deep graph random process to fully consider all possible relations among utterances in the conversational context. Then the inferred relation graphs are integrated with a variational auto-encoder framework to train a GAN for structure-aware response generation. Experimental results on the Ubuntu Internet Relay Chat (IRC) channel benchmark and the most recent Movie Dialogues show that our method outperforms various baseline models for multi-party response generation.
• 摘要：现有的神经响应生成模型在两方对话方面取得了令人印象深刻的改进，该模型假设话语是按顺序组织的。然而，许多现实世界的对话涉及多个对话者，并且对话上下文的结构要复杂得多，例如不同对话者的话语可以“并行”发生。面对这一挑战，有些工作试图对话语或对话者之间的关系进行建模，以促进在更清晰的上下文中生成响应。尽管如此，这些方法严重依赖于这种关系，并且都假设这些关系是预先给定的，这是不切实际的并且阻碍了此类方法的通用性。在这项工作中，我们建议在没有任何人类标签的情况下，通过对话上下文中微妙线索的关系思维来自动推断关系，并利用这些关系来指导神经响应的生成。具体来说，我们首先应用深度图随机过程来充分考虑对话上下文中话语之间的所有可能关系。然后将推断的关系图与变分自动编码器框架集成，以训练 GAN 来生成结构感知响应。 Ubuntu Internet Relay Chat (IRC) 通道基准测试和最新的 Movie Dialogues 的实验结果表明，我们的方法优于多方响应生成的各种基线模型。

Title: Two-step Automated Cybercrime Coded Word Detection using Multi-level Representation Learning

• Authors: Yongyeon Kim, Byung-Won On, Ingyu Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.10838
• Pdf URL: https://arxiv.org/pdf/2403.10838
• Copy Paste: [[2403.10838]] Two-step Automated Cybercrime Coded Word Detection using Multi-level Representation Learning(https://arxiv.org/abs/2403.10838)
• Keywords: language model
• Abstract: In social network service platforms, crime suspects are likely to use cybercrime coded words for communication by adding criminal meanings to existing words or replacing them with similar words. For instance, the word 'ice' is often used to mean methamphetamine in drug crimes. To analyze the nature of cybercrime and the behavior of criminals, quickly detecting such words and further understanding their meaning are critical. In the automated cybercrime coded word detection problem, it is difficult to collect a sufficient amount of training data for supervised learning and to directly apply language models that utilize context information to better understand natural language. To overcome these limitations, we propose a new two-step approach, in which a mean latent vector is constructed for each cybercrime through one of five different AutoEncoder models in the first step, and cybercrime coded words are detected based on multi-level latent representations in the second step. Moreover, to deeply understand cybercrime coded words detected through the two-step approach, we propose three novel methods: (1) Detection of new words recently coined, (2) Detection of words frequently appeared in both drug and sex crimes, and (3) Automatic generation of word taxonomy. According to our experimental results, among various AutoEncoder models, the stacked AutoEncoder model shows the best performance. Additionally, the F1-score of the two-step approach is 0.991, which is higher than 0.987 and 0.903 of the existing dark-GloVe and dark-BERT models. By analyzing the experimental results of the three proposed methods, we can gain a deeper understanding of drug and sex crimes.
• 摘要：在社交网络服务平台中，犯罪嫌疑人很可能通过在现有词语中添加犯罪含义或用相似词语替换等方式，使用网络犯罪编码词语进行交流。例如，“冰毒”一词在毒品犯罪中经常被用来指甲基苯丙胺。为了分析网络犯罪的本质和犯罪分子的行为，快速检测此类词语并进一步理解其含义至关重要。在自动化网络犯罪编码单词检测问题中，很难收集足够量的训练数据用于监督学习，并直接应用利用上下文信息的语言模型来更好地理解自然语言。为了克服这些限制，我们提出了一种新的两步方法，其中第一步通过五种不同的自动编码器模型之一为每个网络犯罪构建平均潜在向量，并根据多级潜在表示检测网络犯罪编码词在第二步中。此外，为了深入了解通过两步法检测到的网络犯罪编码单词，我们提出了三种新颖的方法：（1）检测最近创造的新单词，（2）检测在毒品和性犯罪中频繁出现的单词，以及（3） ）自动生成单词分类。根据我们的实验结果，在各种 AutoEncoder 模型中，堆叠式 AutoEncoder 模型表现出最好的性能。此外，两步法的 F1 分数为 0.991，高于现有 dark-GloVe 和 dark-BERT 模型的 0.987 和 0.903。通过分析这三种方法的实验结果，我们可以对毒品和性犯罪有更深入的了解。

Title: Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean

• Authors: ChangSu Choi, Yongbin Jeong, Seoyoon Park, InHo Won, HyeonSeok Lim, SangMin Kim, Yejee Kang, Chanhyuk Yoon, Jaewan Park, Yiseul Lee, HyeJin Lee, Younggyun Hahm, Hansaem Kim, KyungTae Lim
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.10882
• Pdf URL: https://arxiv.org/pdf/2403.10882
• Copy Paste: [[2403.10882]] Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean(https://arxiv.org/abs/2403.10882)
• Keywords: language model, gpt, llm
• Abstract: Large language models (LLMs) use pretraining to predict the subsequent word; however, their expansion requires significant computing resources. Numerous big tech companies and research institutes have developed multilingual LLMs (MLLMs) to meet current demands, overlooking less-resourced languages (LRLs). This study proposed three strategies to enhance the performance of LRLs based on the publicly available MLLMs. First, the MLLM vocabularies of LRLs were expanded to enhance expressiveness. Second, bilingual data were used for pretraining to align the high- and less-resourced languages. Third, a high-quality small-scale instruction dataset was constructed and instruction-tuning was performed to augment the LRL. The experiments employed the Llama2 model and Korean was used as the LRL, which was quantitatively evaluated against other developed LLMs across eight tasks. Furthermore, a qualitative assessment was performed based on human evaluation and GPT4. Experimental results showed that our proposed Bllossom model exhibited superior performance in qualitative analyses compared to previously proposed Korean monolingual models.
• 摘要：大型语言模型（LLM）使用预训练来预测后续单词；然而，它们的扩展需要大量的计算资源。许多大型科技公司和研究机构都开发了多语言法学硕士（MLLM）来满足当前的需求，而忽略了资源较少的语言（LRL）。本研究基于公开的 MLLM 提出了三种提高 LRL 性能的策略。首先，扩展了 LRL 的 MLLM 词汇表以增强表达能力。其次，使用双语数据进行预训练，以调整资源丰富的语言和资源匮乏的语言。第三，构建了高质量的小规模指令数据集，并进行了指令调整以增强 LRL。实验采用了 Llama2 模型，并使用韩语作为 LRL，在八项任务中与其他开发的法学硕士进行了定量评估。此外，基于人类评估和 GPT4 进行了定性评估。实验结果表明，与之前提出的韩语单语模型相比，我们提出的 Blossom 模型在定性分析中表现出优越的性能。

Title: Towards Robustness and Diversity: Continual Learning in Dialog Generation with Text-Mixup and Batch Nuclear-Norm Maximization

• Authors: Zihan Wang, Jiayu Xiao, Mengxiang Li, Zhongjiang He, Yongxiang Li, Chao Wang, Shuangyong Song
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.10894
• Pdf URL: https://arxiv.org/pdf/2403.10894
• Copy Paste: [[2403.10894]] Towards Robustness and Diversity: Continual Learning in Dialog Generation with Text-Mixup and Batch Nuclear-Norm Maximization(https://arxiv.org/abs/2403.10894)
• Keywords: language model, chat
• Abstract: In our dynamic world where data arrives in a continuous stream, continual learning enables us to incrementally add new tasks/domains without the need to retrain from scratch. A major challenge in continual learning of language model is catastrophic forgetting, the tendency of models to forget knowledge from previously trained tasks/domains when training on new ones. This paper studies dialog generation under the continual learning setting. We propose a novel method that 1) uses \textit{Text-Mixup} as data augmentation to avoid model overfitting on replay memory and 2) leverages Batch-Nuclear Norm Maximization (BNNM) to alleviate the problem of mode collapse. Experiments on a $37$-domain task-oriented dialog dataset and DailyDialog (a $10$-domain chitchat dataset) demonstrate that our proposed approach outperforms the state-of-the-art in continual learning.
• 摘要：在数据连续不断到达的动态世界中，持续学习使我们能够逐步添加新任务/领域，而无需从头开始重新训练。语言模型持续学习的一个主要挑战是灾难性遗忘，即模型在训练新任务/领域时容易忘记之前训练过的任务/领域的知识。本文研究了持续学习环境下的对话生成。我们提出了一种新颖的方法，1）使用 \textit{Text-Mixup} 作为数据增强，以避免模型在重播内存上过度拟合，2）利用批量核范数最大化（BNNM）来缓解模式崩溃问题。在 37 美元域的面向任务的对话数据集和 DailyDialog（10 美元域的闲聊数据集）上进行的实验表明，我们提出的方法在持续学习方面优于最先进的方法。

Title: BEnQA: A Question Answering and Reasoning Benchmark for Bengali and English

• Authors: Sheikh Shafayat, H M Quamran Hasan, Minhajur Rahman Chowdhury Mahim, Rifki Afina Putri, James Thorne, Alice Oh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.10900
• Pdf URL: https://arxiv.org/pdf/2403.10900
• Copy Paste: [[2403.10900]] BEnQA: A Question Answering and Reasoning Benchmark for Bengali and English(https://arxiv.org/abs/2403.10900)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: In this study, we introduce BEnQA, a dataset comprising parallel Bengali and English exam questions for middle and high school levels in Bangladesh. Our dataset consists of approximately 5K questions covering several subjects in science with different types of questions, including factual, application, and reasoning-based questions. We benchmark several Large Language Models (LLMs) with our parallel dataset and observe a notable performance disparity between the models in Bengali and English. We also investigate some prompting methods, and find that Chain-of-Thought prompting is beneficial mostly on reasoning questions, but not so much on factual ones. We also find that appending English translation helps to answer questions in Bengali. Our findings point to promising future research directions for improving the performance of LLMs in Bengali and more generally in low-resource languages.
• 摘要：在这项研究中，我们引入了 BEnQA，这是一个包含孟加拉国初中和高中水平的平行孟加拉语和英语考试问题的数据集。我们的数据集包含大约 5000 个问题，涵盖多个科学主题，具有不同类型的问题，包括基于事实、应用和推理的问题。我们使用并行数据集对多个大型语言模型 (LLM) 进行基准测试，并观察到孟加拉语和英语模型之间存在显着的性能差异。我们还研究了一些提示方法，发现思路链提示主要对推理题有帮助，但对事实题则不太有效。我们还发现附加英语翻译有助于回答孟加拉语问题。我们的研究结果指出了未来有希望的研究方向，以提高孟加拉语和更广泛的低资源语言法学硕士的表现。

Title: SelfIE: Self-Interpretation of Large Language Model Embeddings

• Authors: Haozhe Chen, Carl Vondrick, Chengzhi Mao
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.10949
• Pdf URL: https://arxiv.org/pdf/2403.10949
• Copy Paste: [[2403.10949]] SelfIE: Self-Interpretation of Large Language Model Embeddings(https://arxiv.org/abs/2403.10949)
• Keywords: language model, llm, prompt
• Abstract: How do large language models (LLMs) obtain their answers? The ability to explain and control an LLM's reasoning process is key for reliability, transparency, and future model developments. We propose SelfIE (Self-Interpretation of Embeddings), a framework that enables LLMs to interpret their own embeddings in natural language by leveraging their ability to respond inquiry about a given passage. Capable of interpreting open-world concepts in the hidden embeddings, SelfIE reveals LLM internal reasoning in cases such as making ethical decisions, internalizing prompt injection, and recalling harmful knowledge. SelfIE's text descriptions on hidden embeddings also open up new avenues to control LLM reasoning. We propose Supervised Control, which allows editing open-ended concepts while only requiring gradient computation of individual layer. We extend RLHF to hidden embeddings and propose Reinforcement Control that erases harmful knowledge in LLM without supervision targets.
• 摘要：大型语言模型（LLM）如何获得答案？解释和控制法学硕士推理过程的能力是可靠性、透明度和未来模型开发的关键。我们提出了 SelfIE（嵌入的自我解释），这是一个框架，使法学硕士能够利用他们对给定段落的查询做出回应的能力来解释自己在自然语言中的嵌入。 SelfIE 能够解释隐藏嵌入中的开放世界概念，在做出道德决策、内化提示注入和回忆有害知识等情况下揭示 LLM 内部推理。 SelfIE 对隐藏嵌入的文本描述也为控制 LLM 推理开辟了新途径。我们提出了监督控制，它允许编辑开放式概念，同时只需要单个层的梯度计算。我们将 RLHF 扩展到隐藏嵌入，并提出强化控制，在没有监督目标的情况下消除 LLM 中的有害知识。

Title: Pre-Trained Language Models Represent Some Geographic Populations Better Than Others

• Authors: Jonathan Dunn, Benjamin Adams, Harish Tayyar Madabushi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.11025
• Pdf URL: https://arxiv.org/pdf/2403.11025
• Copy Paste: [[2403.11025]] Pre-Trained Language Models Represent Some Geographic Populations Better Than Others(https://arxiv.org/abs/2403.11025)
• Keywords: language model, llm
• Abstract: This paper measures the skew in how well two families of LLMs represent diverse geographic populations. A spatial probing task is used with geo-referenced corpora to measure the degree to which pre-trained language models from the OPT and BLOOM series represent diverse populations around the world. Results show that these models perform much better for some populations than others. In particular, populations across the US and the UK are represented quite well while those in South and Southeast Asia are poorly represented. Analysis shows that both families of models largely share the same skew across populations. At the same time, this skew cannot be fully explained by sociolinguistic factors, economic factors, or geographic factors. The basic conclusion from this analysis is that pre-trained models do not equally represent the world's population: there is a strong skew towards specific geographic populations. This finding challenges the idea that a single model can be used for all populations.
• 摘要：本文衡量了两个法学硕士家族代表不同地理人群的程度的偏差。空间探测任务与地理参考语料库一起使用，以衡量 OPT 和 BLOOM 系列中的预训练语言模型代表世界各地不同人群的程度。结果表明，这些模型对于某些人群的表现比其他人群要好得多。特别是，美国和英国的人口代表性相当高，而南亚和东南亚的人口代表性较差。分析表明，这两个模型系列在人群中基本上具有相同的偏差。同时，这种偏差不能完全用社会语言因素、经济因素或地理因素来解释。该分析的基本结论是，预先训练的模型并不能平等地代表世界人口：对特定地理人口有很大的偏向。这一发现挑战了单一模型可以用于所有人群的想法。

Title: RobustSentEmbed: Robust Sentence Embeddings Using Adversarial Self-Supervised Contrastive Learning

• Authors: Javad Rafiei Asl, Prajwal Panzade, Eduardo Blanco, Daniel Takabi, Zhipeng Cai
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.11082
• Pdf URL: https://arxiv.org/pdf/2403.11082
• Copy Paste: [[2403.11082]] RobustSentEmbed: Robust Sentence Embeddings Using Adversarial Self-Supervised Contrastive Learning(https://arxiv.org/abs/2403.11082)
• Keywords: language model
• Abstract: Pre-trained language models (PLMs) have consistently demonstrated outstanding performance across a diverse spectrum of natural language processing tasks. Nevertheless, despite their success with unseen data, current PLM-based representations often exhibit poor robustness in adversarial settings. In this paper, we introduce RobustSentEmbed, a self-supervised sentence embedding framework designed to improve both generalization and robustness in diverse text representation tasks and against a diverse set of adversarial attacks. Through the generation of high-risk adversarial perturbations and their utilization in a novel objective function, RobustSentEmbed adeptly learns high-quality and robust sentence embeddings. Our experiments confirm the superiority of RobustSentEmbed over state-of-the-art representations. Specifically, Our framework achieves a significant reduction in the success rate of various adversarial attacks, notably reducing the BERTAttack success rate by almost half (from 75.51\% to 38.81\%). The framework also yields improvements of 1.59\% and 0.23\% in semantic textual similarity tasks and various transfer tasks, respectively.
• 摘要：预训练语言模型 (PLM) 在各种自然语言处理任务中始终表现出出色的性能。然而，尽管当前基于 PLM 的表示在看不见的数据上取得了成功，但在对抗性环境中往往表现出较差的鲁棒性。在本文中，我们介绍了 RobustSentEmbed，这是一种自监督句子嵌入框架，旨在提高各种文本表示任务的泛化性和鲁棒性，并抵御各种对抗性攻击。通过生成高风险的对抗性扰动并在新颖的目标函数中利用它们，RobustSentEmbed 熟练地学习了高质量且稳健的句子嵌入。我们的实验证实了 RobustSentEmbed 相对于最先进的表示的优越性。具体来说，我们的框架显着降低了各种对抗性攻击的成功率，特别是将 BERTattack 成功率降低了近一半（从 75.51\% 到 38.81\%）。该框架在语义文本相似性任务和各种传输任务方面也分别提高了 1.59% 和 0.23%。

Title: Lost in Translation? Translation Errors and Challenges for Fair Assessment of Text-to-Image Models on Multilingual Concepts

• Authors: Michael Saxon, Yiran Luo, Sharon Levy, Chitta Baral, Yezhou Yang, William Yang Wang
• Subjects: cs.CL, cs.AI, cs.CV, cs.CY, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.11092
• Pdf URL: https://arxiv.org/pdf/2403.11092
• Copy Paste: [[2403.11092]] Lost in Translation? Translation Errors and Challenges for Fair Assessment of Text-to-Image Models on Multilingual Concepts(https://arxiv.org/abs/2403.11092)
• Keywords: prompt
• Abstract: Benchmarks of the multilingual capabilities of text-to-image (T2I) models compare generated images prompted in a test language to an expected image distribution over a concept set. One such benchmark, "Conceptual Coverage Across Languages" (CoCo-CroLa), assesses the tangible noun inventory of T2I models by prompting them to generate pictures from a concept list translated to seven languages and comparing the output image populations. Unfortunately, we find that this benchmark contains translation errors of varying severity in Spanish, Japanese, and Chinese. We provide corrections for these errors and analyze how impactful they are on the utility and validity of CoCo-CroLa as a benchmark. We reassess multiple baseline T2I models with the revisions, compare the outputs elicited under the new translations to those conditioned on the old, and show that a correction's impactfulness on the image-domain benchmark results can be predicted in the text domain with similarity scores. Our findings will guide the future development of T2I multilinguality metrics by providing analytical tools for practical translation decisions.
• 摘要：文本到图像 (T2I) 模型的多语言功能基准将测试语言提示的生成图像与概念集上的预期图像分布进行比较。其中一个基准是“跨语言的概念覆盖率”(CoCo-CroLa)，它通过提示 T2I 模型从翻译成七种语言的概念列表中生成图片并比较输出图像群体来评估 T2I 模型的有形名词库存。不幸的是，我们发现该基准测试包含西班牙语、日语和中文的不同严重程度的翻译错误。我们对这些错误进行了更正，并分析了它们对 CoCo-CroLa 作为基准的实用性和有效性的影响。我们重新评估了经过修订的多个基线 T2I 模型，将新翻译下产生的输出与旧翻译下的输出进行比较，并表明可以通过相似性分数在文本域中预测校正对图像域基准结果的影响。我们的研究结果将通过为实际翻译决策提供分析工具来指导 T2I 多语言指标的未来发展。

Title: ProgGen: Generating Named Entity Recognition Datasets Step-by-step with Self-Reflexive Large Language Models

• Authors: Yuzhao Heng, Chunyuan Deng, Yitong Li, Yue Yu, Yinghao Li, Rongzhi Zhang, Chao Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.11103
• Pdf URL: https://arxiv.org/pdf/2403.11103
• Copy Paste: [[2403.11103]] ProgGen: Generating Named Entity Recognition Datasets Step-by-step with Self-Reflexive Large Language Models(https://arxiv.org/abs/2403.11103)
• Keywords: language model, llm, prompt
• Abstract: Although Large Language Models (LLMs) exhibit remarkable adaptability across domains, these models often fall short in structured knowledge extraction tasks such as named entity recognition (NER). This paper explores an innovative, cost-efficient strategy to harness LLMs with modest NER capabilities for producing superior NER datasets. Our approach diverges from the basic class-conditional prompts by instructing LLMs to self-reflect on the specific domain, thereby generating domain-relevant attributes (such as category and emotions for movie reviews), which are utilized for creating attribute-rich training data. Furthermore, we preemptively generate entity terms and then develop NER context data around these entities, effectively bypassing the LLMs' challenges with complex structures. Our experiments across both general and niche domains reveal significant performance enhancements over conventional data generation methods while being more cost-effective than existing alternatives.
• 摘要：尽管大型语言模型 (LLM) 表现出卓越的跨领域适应性，但这些模型在命名实体识别 (NER) 等结构化知识提取任务中往往存在不足。本文探讨了一种创新、经济高效的策略，利用具有适度 NER 功能的法学硕士来生成卓越的 NER 数据集。我们的方法与基本的类别条件提示不同，通过指示法学硕士对特定领域进行自我反思，从而生成与领域相关的属性（例如电影评论的类别和情感），这些属性用于创建属性丰富的训练数据。此外，我们预先生成实体术语，然后围绕这些实体开发 NER 上下文数据，有效地绕过了法学硕士复杂结构的挑战。我们在通用和利基领域的实验表明，与传统数据生成方法相比，性能显着增强，同时比现有替代方法更具成本效益。

Title: Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment

• Authors: Feifan Song, Bowen Yu, Hao Lang, Haiyang Yu, Fei Huang, Houfeng Wang, Yongbin Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.11124
• Pdf URL: https://arxiv.org/pdf/2403.11124
• Copy Paste: [[2403.11124]] Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment(https://arxiv.org/abs/2403.11124)
• Keywords: language model, llm, prompt
• Abstract: Alignment with human preference prevents large language models (LLMs) from generating misleading or toxic content while requiring high-cost human feedback. Assuming resources of human annotation are limited, there are two different ways of allocating considered: more diverse PROMPTS or more diverse RESPONSES to be labeled. Nonetheless, a straightforward comparison between their impact is absent. In this work, we first control the diversity of both sides according to the number of samples for fine-tuning, which can directly reflect their influence. We find that instead of numerous prompts, more responses but fewer prompts better trigger LLMs for human alignment. Additionally, the concept of diversity for prompts can be more complex than responses that are typically quantified by single digits. Consequently, a new formulation of prompt diversity is proposed, further implying a linear correlation with the final performance of LLMs after fine-tuning. We also leverage it on data augmentation and conduct experiments to show its effect on different algorithms.
• 摘要：与人类偏好保持一致可以防止大型语言模型 (LLM) 生成误导性或有毒内容，同时需要高成本的人类反馈。假设人类注释资源有限，有两种不同的分配方式可供考虑：更多样化的提示或更多样化的待标记响应。尽管如此，它们的影响之间还没有直接的比较。在这项工作中，我们首先根据样本数量控制双方的多样性进行微调，这可以直接反映他们的影响力。我们发现，更多的反应和更少的提示可以更好地触发法学硕士的人类对齐，而不是大量的提示。此外，提示的多样性概念可能比通常用个位数量化的响应更复杂。因此，提出了一种新的即时多样性表述，进一步暗示了微调后与法学硕士最终表现的线性相关性。我们还将其用于数据增强并进行实验以展示其对不同算法的影响。

Title: Beyond Static Evaluation: A Dynamic Approach to Assessing AI Assistants' API Invocation Capabilities

• Authors: Honglin Mu, Yang Xu, Yunlong Feng, Xiaofeng Han, Yitong Li, Yutai Hou, Wanxiang Che
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.11128
• Pdf URL: https://arxiv.org/pdf/2403.11128
• Copy Paste: [[2403.11128]] Beyond Static Evaluation: A Dynamic Approach to Assessing AI Assistants' API Invocation Capabilities(https://arxiv.org/abs/2403.11128)
• Keywords: language model, llm, agent
• Abstract: With the rise of Large Language Models (LLMs), AI assistants' ability to utilize tools, especially through API calls, has advanced notably. This progress has necessitated more accurate evaluation methods. Many existing studies adopt static evaluation, where they assess AI assistants' API call based on pre-defined dialogue histories. However, such evaluation method can be misleading, as an AI assistant might fail in generating API calls from preceding human interaction in real cases. Instead of the resource-intensive method of direct human-machine interactions, we propose Automated Dynamic Evaluation (AutoDE) to assess an assistant's API call capability without human involvement. In our framework, we endeavor to closely mirror genuine human conversation patterns in human-machine interactions, using a LLM-based user agent, equipped with a user script to ensure human alignment. Experimental results highlight that AutoDE uncovers errors overlooked by static evaluations, aligning more closely with human assessment. Testing four AI assistants using our crafted benchmark, our method mirrored human evaluation with an correlation of 0.99, marking an 8% enhancement compared to conventional static evaluations.
• 摘要：随着大型语言模型 (LLM) 的兴起，人工智能助手利用工具的能力，特别是通过 API 调用的能力显着提高。这一进展需要更准确的评估方法。许多现有研究采用静态评估，根据预定义的对话历史来评估人工智能助手的 API 调用。然而，这种评估方法可能会产生误导，因为在实际情况下，人工智能助手可能无法从之前的人类交互中生成 API 调用。我们提出自动动态评估（AutoDE）来代替直接人机交互的资源密集型方法来评估助手的 API 调用能力，而无需人工参与。在我们的框架中，我们努力使用基于 LLM 的用户代理，并配备用户脚本来确保人机交互中真实的人类对话模式，以确保人类的一致性。实验结果强调，AutoDE 可以发现静态评估所忽略的错误，与人类评估更加一致。使用我们精心设计的基准测试四个人工智能助手，我们的方法反映了人类评估的相关性为 0.99，与传统静态评估相比提高了 8%。

Title: Enhancing Event Causality Identification with Rationale and Structure-Aware Causal Question Answering

• Authors: Baiyan Zhang, Qin Chen, Jie Zhou, Jian Jin, Liang He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.11129
• Pdf URL: https://arxiv.org/pdf/2403.11129
• Copy Paste: [[2403.11129]] Enhancing Event Causality Identification with Rationale and Structure-Aware Causal Question Answering(https://arxiv.org/abs/2403.11129)
• Keywords: language model
• Abstract: Document-level Event Causality Identification (DECI) aims to identify causal relations between two events in documents. Recent research tends to use pre-trained language models to generate the event causal relations. Whereas, these methods are prone to the errors of sequential generation due to multiple events in a document. Moreover, the potential structures such as event coreference and related causal chain are neglected. In this paper, we propose a multi-task learning framework to enhance event causality identification with rationale and structure-aware causal question answering. Specifically, the DECI task is transformed into multiple-choice question answering, and the causes and effects of the questioned event are generated with large language models. In addition, we generate the rationales to explain why these events have causal relations. Moreover, we construct an event structure graph, which models the multi-hop potential relations for causal reasoning of the current event. Experiments on two benchmark datasets show the great advantages of our proposed approach compared to the state-of-the-art methods. Moreover, we conduct both quantitative and qualitative analyses, which shed light on why each component of our approach can lead to great improvements.
• 摘要：文档级事件因果关系识别（DECI）旨在识别文档中两个事件之间的因果关系。最近的研究倾向于使用预先训练的语言模型来生成事件因果关系。然而，由于文档中存在多个事件，这些方法很容易出现顺序生成错误。此外，事件共指和相关因果链等潜在结构被忽略。在本文中，我们提出了一种多任务学习框架，通过基本原理和结构感知的因果问答来增强事件因果关系识别。具体来说，将DECI任务转化为多项选择题回答，并用大型语言模型生成提问事件的因果关系。此外，我们还提出了理由来解释为什么这些事件具有因果关系。此外，我们构建了一个事件结构图，它对当前事件的因果推理的多跳潜在关系进行建模。对两个基准数据集的实验表明，与最先进的方法相比，我们提出的方法具有巨大的优势。此外，我们还进行定量和定性分析，这揭示了为什么我们方法的每个组成部分都可以带来巨大的改进。

Title: Exploring Tokenization Strategies and Vocabulary Sizes for Enhanced Arabic Language Models

• Authors: Mohamed Taher Alrefaie, Nour Eldin Morsy, Nada Samir
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.11130
• Pdf URL: https://arxiv.org/pdf/2403.11130
• Copy Paste: [[2403.11130]] Exploring Tokenization Strategies and Vocabulary Sizes for Enhanced Arabic Language Models(https://arxiv.org/abs/2403.11130)
• Keywords: language model
• Abstract: This paper presents a comprehensive examination of the impact of tokenization strategies and vocabulary sizes on the performance of Arabic language models in downstream natural language processing tasks. Our investigation focused on the effectiveness of four tokenizers across various tasks, including News Classification, Hate Speech Detection, Sentiment Analysis, and Natural Language Inference. Leveraging a diverse set of vocabulary sizes, we scrutinize the intricate interplay between tokenization approaches and model performance. The results reveal that Byte Pair Encoding (BPE) with Farasa outperforms other strategies in multiple tasks, underscoring the significance of morphological analysis in capturing the nuances of the Arabic language. However, challenges arise in sentiment analysis, where dialect specific segmentation issues impact model efficiency. Computational efficiency analysis demonstrates the stability of BPE with Farasa, suggesting its practical viability. Our study uncovers limited impacts of vocabulary size on model performance while keeping the model size unchanged. This is challenging the established beliefs about the relationship between vocabulary, model size, and downstream tasks, emphasizing the need for the study of models' size and their corresponding vocabulary size to generalize across domains and mitigate biases, particularly in dialect based datasets. Paper's recommendations include refining tokenization strategies to address dialect challenges, enhancing model robustness across diverse linguistic contexts, and expanding datasets to encompass the rich dialect based Arabic. This work not only advances our understanding of Arabic language models but also lays the foundation for responsible and ethical developments in natural language processing technologies tailored to the intricacies of the Arabic language.
• 摘要：本文全面研究了标记化策略和词汇量对下游自然语言处理任务中阿拉伯语言模型性能的影响。我们的调查重点是四个标记器在各种任务中的有效性，包括新闻分类、仇恨言论检测、情感分析和自然语言推理。利用不同的词汇量，我们仔细研究了标记化方法和模型性能之间复杂的相互作用。结果表明，Farasa 的字节对编码 (BPE) 在多项任务中优于其他策略，强调了形态分析在捕捉阿拉伯语言细微差别方面的重要性。然而，情感分析中出现了挑战，方言特定的分割问题会影响模型的效率。计算效率分析证明了 Farasa 的 BPE 的稳定性，表明了其实际可行性。我们的研究揭示了在保持模型大小不变的情况下词汇量大小对模型性能的有限影响。这挑战了关于词汇量、模型大小和下游任务之间关系的既定信念，强调需要研究模型大小及其相应的词汇量，以跨领域泛化并减轻偏差，特别是在基于方言的数据集中。论文的建议包括完善标记化策略以应对方言挑战，增强跨不同语言环境的模型稳健性，以及扩展数据集以涵盖丰富的基于阿拉伯语的方言。这项工作不仅增进了我们对阿拉伯语言模型的理解，还为针对阿拉伯语言复杂性的自然语言处理技术的负责任和道德的发展奠定了基础。

Title: Evaluation Ethics of LLMs in Legal Domain

• Authors: Ruizhe Zhang, Haitao Li, Yueyue Wu, Qingyao Ai, Yiqun Liu, Min Zhang, Shaoping Ma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.11152
• Pdf URL: https://arxiv.org/pdf/2403.11152
• Copy Paste: [[2403.11152]] Evaluation Ethics of LLMs in Legal Domain(https://arxiv.org/abs/2403.11152)
• Keywords: language model, llm
• Abstract: In recent years, the utilization of large language models for natural language dialogue has gained momentum, leading to their widespread adoption across various domains. However, their universal competence in addressing challenges specific to specialized fields such as law remains a subject of scrutiny. The incorporation of legal ethics into the model has been overlooked by researchers. We asserts that rigorous ethic evaluation is essential to ensure the effective integration of large language models in legal domains, emphasizing the need to assess domain-specific proficiency and domain-specific ethic. To address this, we propose a novelty evaluation methodology, utilizing authentic legal cases to evaluate the fundamental language abilities, specialized legal knowledge and legal robustness of large language models (LLMs). The findings from our comprehensive evaluation contribute significantly to the academic discourse surrounding the suitability and performance of large language models in legal domains.
• 摘要：近年来，大型语言模型在自然语言对话中的应用势头强劲，导致其在各个领域得到广泛采用。然而，他们在应对法律等专业领域特有的挑战方面的普遍能力仍然受到密切关注。研究人员忽视了将法律伦理纳入模型的问题。我们认为，严格的道德评估对于确保大型语言模型在法律领域的有效整合至关重要，并强调需要评估特定领域的熟练程度和特定领域的道德。为此，我们提出了一种新颖性评估方法，利用真实的法律案例来评估大语言模型（LLM）的基本语言能力、专业法律知识和法律稳健性。我们的综合评估结果对围绕法律领域大型语言模型的适用性和性能的学术讨论做出了重大贡献。

Title: Correcting misinformation on social media with a large language model

• Authors: Xinyi Zhou, Ashish Sharma, Amy X. Zhang, Tim Althoff
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.11169
• Pdf URL: https://arxiv.org/pdf/2403.11169
• Copy Paste: [[2403.11169]] Correcting misinformation on social media with a large language model(https://arxiv.org/abs/2403.11169)
• Keywords: language model, gpt, llm, prompt
• Abstract: Misinformation undermines public trust in science and democracy, particularly on social media where inaccuracies can spread rapidly. Experts and laypeople have shown to be effective in correcting misinformation by manually identifying and explaining inaccuracies. Nevertheless, this approach is difficult to scale, a concern as technologies like large language models (LLMs) make misinformation easier to produce. LLMs also have versatile capabilities that could accelerate misinformation correction; however, they struggle due to a lack of recent information, a tendency to produce plausible but false content and references, and limitations in addressing multimodal information. To address these issues, we propose MUSE, an LLM augmented with access to and credibility evaluation of up-to-date information. By retrieving contextual evidence and refutations, MUSE can provide accurate and trustworthy explanations and references. It also describes visuals and conducts multimodal searches for correcting multimodal misinformation. We recruit fact-checking and journalism experts to evaluate corrections to real social media posts across 13 dimensions, ranging from the factuality of explanation to the relevance of references. The results demonstrate MUSE's ability to correct misinformation promptly after appearing on social media; overall, MUSE outperforms GPT-4 by 37% and even high-quality corrections from laypeople by 29%. This work underscores the potential of LLMs to combat real-world misinformation effectively and efficiently.
• 摘要：错误信息破坏了公众对科学和民主的信任，尤其是在社交媒体上，错误信息可能会迅速传播。事实证明，专家和外行人通过手动识别和解释不准确的内容可以有效地纠正错误信息。然而，这种方法很难扩展，这是一个令人担忧的问题，因为大型语言模型 (LLM) 等技术使错误信息更容易产生。法学硕士还具有多种能力，可以加速错误信息的纠正；然而，由于缺乏最新信息、倾向于产生看似合理但虚假的内容和参考文献以及处理多模式信息的局限性，它们陷入困境。为了解决这些问题，我们提出了 MUSE，这是一种法学硕士，增强了对最新信息的访问和可信度评估。通过检索上下文证据和反驳，MUSE 可以提供准确且值得信赖的解释和参考。它还描述视觉效果并进行多模态搜索以纠正多模态错误信息。我们聘请事实核查和新闻专家来评估对真实社交媒体帖子的更正，涵盖 13 个维度，从解释的真实性到参考文献的相关性。结果表明，MUSE 有能力在出现在社交媒体上后及时纠正错误信息；总体而言，MUSE 的性能比 GPT-4 高出 37%，甚至比外行人的高质量修正高出 29%。这项工作强调了法学硕士有效且高效地打击现实世界错误信息的潜力。

Title: TRELM: Towards Robust and Efficient Pre-training for Knowledge-Enhanced Language Models

• Authors: Junbing Yan, Chengyu Wang, Taolin Zhang, Xiaofeng He, Jun Huang, Longtao Huang, Hui Xue, Wei Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.11203
• Pdf URL: https://arxiv.org/pdf/2403.11203
• Copy Paste: [[2403.11203]] TRELM: Towards Robust and Efficient Pre-training for Knowledge-Enhanced Language Models(https://arxiv.org/abs/2403.11203)
• Keywords: language model
• Abstract: KEPLMs are pre-trained models that utilize external knowledge to enhance language understanding. Previous language models facilitated knowledge acquisition by incorporating knowledge-related pre-training tasks learned from relation triples in knowledge graphs. However, these models do not prioritize learning embeddings for entity-related tokens. Moreover, updating the entire set of parameters in KEPLMs is computationally demanding. This paper introduces TRELM, a Robust and Efficient Pre-training framework for Knowledge-Enhanced Language Models. We observe that entities in text corpora usually follow the long-tail distribution, where the representations of some entities are suboptimally optimized and hinder the pre-training process for KEPLMs. To tackle this, we employ a robust approach to inject knowledge triples and employ a knowledge-augmented memory bank to capture valuable information. Furthermore, updating a small subset of neurons in the feed-forward networks (FFNs) that store factual knowledge is both sufficient and efficient. Specifically, we utilize dynamic knowledge routing to identify knowledge paths in FFNs and selectively update parameters during pre-training. Experimental results show that TRELM reduces pre-training time by at least 50% and outperforms other KEPLMs in knowledge probing tasks and multiple knowledge-aware language understanding tasks.
• 摘要：KEPLM 是预先训练的模型，利用外部知识来增强语言理解。以前的语言模型通过结合从知识图中的关系三元组学习到的与知识相关的预训练任务来促进知识获取。然而，这些模型并没有优先考虑实体相关标记的学习嵌入。此外，更新 KEPLM 中的整套参数对计算要求很高。本文介绍了 TRELM，一种强大且高效的知识增强语言模型预训练框架。我们观察到文本语料库中的实体通常遵循长尾分布，其中一些实体的表示没有得到优化优化，并阻碍了 KEPLM 的预训练过程。为了解决这个问题，我们采用了一种强大的方法来注入知识三元组，并采用知识增强记忆库来捕获有价值的信息。此外，更新存储事实知识的前馈网络（FFN）中的一小部分神经元既足够又有效。具体来说，我们利用动态知识路由来识别 FFN 中的知识路径，并在预训练期间有选择地更新参数。实验结果表明，TRELM 减少了至少 50% 的预训练时间，并且在知识探测任务和多种知识感知语言理解任务中优于其他 KEPLM。

Title: Creating an African American-Sounding TTS: Guidelines, Technical Challenges,and Surprising Evaluations

• Authors: Claudio Pinhanez, Raul Fernandez, Marcelo Grave, Julio Nogima, Ron Hoory
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2403.11209
• Pdf URL: https://arxiv.org/pdf/2403.11209
• Copy Paste: [[2403.11209]] Creating an African American-Sounding TTS: Guidelines, Technical Challenges,and Surprising Evaluations(https://arxiv.org/abs/2403.11209)
• Keywords: agent
• Abstract: Representations of AI agents in user interfaces and robotics are predominantly White, not only in terms of facial and skin features, but also in the synthetic voices they use. In this paper we explore some unexpected challenges in the representation of race we found in the process of developing an U.S. English Text-to-Speech (TTS) system aimed to sound like an educated, professional, regional accent-free African American woman. The paper starts by presenting the results of focus groups with African American IT professionals where guidelines and challenges for the creation of a representative and appropriate TTS system were discussed and gathered, followed by a discussion about some of the technical difficulties faced by the TTS system developers. We then describe two studies with U.S. English speakers where the participants were not able to attribute the correct race to the African American TTS voice while overwhelmingly correctly recognizing the race of a White TTS system of similar quality. A focus group with African American IT workers not only confirmed the representativeness of the African American voice we built, but also suggested that the surprising recognition results may have been caused by the inability or the latent prejudice from non-African Americans to associate educated, non-vernacular, professionally-sounding voices to African American people.
• 摘要：用户界面和机器人中的人工智能代理主要是白人，不仅在面部和皮肤特征方面，而且在他们使用的合成声音方面也是如此。在本文中，我们探讨了在开发美国英语文本转语音 (TTS) 系统的过程中发现的种族表征方面的一些意想不到的挑战，该系统旨在使声音听起来像受过教育、专业、无地区口音的非裔美国女性。本文首先介绍了非裔美国 IT 专业人员焦点小组的结果，其中讨论和收集了创建具有代表性和适当的 TTS 系统的指导方针和挑战，然后讨论了 TTS 系统开发人员面临的一些技术困难。然后，我们描述了两项针对美国英语使用者的研究，其中参与者无法将正确的种族归因于非裔美国人的 TTS 声音，同时以压倒性的优势正确识别出类似质量的白人 TTS 系统的种族。一个由非裔美国 IT 工作者组成的焦点小组不仅证实了我们建立的非裔美国人声音的代表性，而且还表明，令人惊讶的识别结果可能是由于非非裔美国人无法或潜在的偏见将受过教育的、非非裔美国人与非裔美国人联系起来。 - 为非裔美国人提供专业的白话声音。

Title: Cheap Ways of Extracting Clinical Markers from Texts

• Authors: Anastasia Sandu, Teodor Mihailescu, Sergiu Nisioi
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.11227
• Pdf URL: https://arxiv.org/pdf/2403.11227
• Copy Paste: [[2403.11227]] Cheap Ways of Extracting Clinical Markers from Texts(https://arxiv.org/abs/2403.11227)
• Keywords: language model, llm, chain-of-thought
• Abstract: This paper describes the work of the UniBuc Archaeology team for CLPsych's 2024 Shared Task, which involved finding evidence within the text supporting the assigned suicide risk level. Two types of evidence were required: highlights (extracting relevant spans within the text) and summaries (aggregating evidence into a synthesis). Our work focuses on evaluating Large Language Models (LLM) as opposed to an alternative method that is much more memory and resource efficient. The first approach employs a good old-fashioned machine learning (GOML) pipeline consisting of a tf-idf vectorizer with a logistic regression classifier, whose representative features are used to extract relevant highlights. The second, more resource intensive, uses an LLM for generating the summaries and is guided by chain-of-thought to provide sequences of text indicating clinical markers.
• 摘要：本文介绍了 UniBuc 考古团队为 CLPsych 2024 年共享任务所做的工作，其中涉及在文本中寻找支持指定自杀风险级别的证据。需要两种类型的证据：亮点（提取文本中的相关跨度）和摘要（将证据聚合成综合）。我们的工作重点是评估大型语言模型 (LLM)，而不是内存和资源效率更高的替代方法。第一种方法采用了一个很好的老式机器学习（GOML）管道，由带有逻辑回归分类器的 tf-idf 矢量器组成，其代表性特征用于提取相关亮点。第二种是资源密集型，使用法学硕士来生成摘要，并在思想链的指导下提供指示临床标记的文本序列。

Title: ChartThinker: A Contextual Chain-of-Thought Approach to Optimized Chart Summarization

• Authors: Mengsha Liu, Daoyuan Chen, Yaliang Li, Guian Fang, Ying Shen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.11236
• Pdf URL: https://arxiv.org/pdf/2403.11236
• Copy Paste: [[2403.11236]] ChartThinker: A Contextual Chain-of-Thought Approach to Optimized Chart Summarization(https://arxiv.org/abs/2403.11236)
• Keywords: chain-of-thought
• Abstract: Data visualization serves as a critical means for presenting data and mining its valuable insights. The task of chart summarization, through natural language processing techniques, facilitates in-depth data analysis of charts. However, there still are notable deficiencies in terms of visual-language matching and reasoning ability for existing approaches. To address these limitations, this study constructs a large-scale dataset of comprehensive chart-caption pairs and fine-tuning instructions on each chart. Thanks to the broad coverage of various topics and visual styles within this dataset, better matching degree can be achieved from the view of training data. Moreover, we propose an innovative chart summarization method, ChartThinker, which synthesizes deep analysis based on chains of thought and strategies of context retrieval, aiming to improve the logical coherence and accuracy of the generated summaries. Built upon the curated datasets, our trained model consistently exhibits superior performance in chart summarization tasks, surpassing 8 state-of-the-art models over 7 evaluation metrics. Our dataset and codes are publicly accessible.
• 摘要：数据可视化是呈现数据和挖掘其有价值的见解的关键手段。图表汇总任务通过自然语言处理技术，有助于对图表进行深入的数据分析。然而，现有方法在视觉语言匹配和推理能力方面仍然存在明显的缺陷。为了解决这些限制，本研究构建了一个包含全面图表标题对的大型数据集，并在每个图表上微调说明。由于该数据集中各种主题和视觉风格的广泛覆盖，从训练数据的角度来看可以实现更好的匹配度。此外，我们提出了一种创新的图表摘要方法ChartThinker，该方法综合了基于思想链和上下文检索策略的深度分析，旨在提高生成的摘要的逻辑连贯性和准确性。基于精选的数据集，我们经过训练的模型在图表汇总任务中始终表现出卓越的性能，在 7 个评估指标上超越了 8 个最先进的模型。我们的数据集和代码可以公开访问。

Title: Mixture-of-Prompt-Experts for Multi-modal Semantic Understanding

• Authors: Zichen Wu, HsiuYuan Huang, Fanyi Qu, Yunfang Wu
• Subjects: cs.CL, cs.MM
• Abstract URL: https://arxiv.org/abs/2403.11311
• Pdf URL: https://arxiv.org/pdf/2403.11311
• Copy Paste: [[2403.11311]] Mixture-of-Prompt-Experts for Multi-modal Semantic Understanding(https://arxiv.org/abs/2403.11311)
• Keywords: language model, prompt
• Abstract: Deep multimodal semantic understanding that goes beyond the mere superficial content relation mining has received increasing attention in the realm of artificial intelligence. The challenges of collecting and annotating high-quality multi-modal data have underscored the significance of few-shot learning. In this paper, we focus on two critical tasks under this context: few-shot multi-modal sarcasm detection (MSD) and multi-modal sentiment analysis (MSA). To address them, we propose Mixture-of-Prompt-Experts with Block-Aware Prompt Fusion (MoPE-BAF), a novel multi-modal soft prompt framework based on the unified vision-language model (VLM). Specifically, we design three experts of soft prompts: a text prompt and an image prompt that extract modality-specific features to enrich the single-modal representation, and a unified prompt to assist multi-modal interaction. Additionally, we reorganize Transformer layers into several blocks and introduce cross-modal prompt attention between adjacent blocks, which smoothens the transition from single-modal representation to multi-modal fusion. On both MSD and MSA datasets in few-shot setting, our proposed model not only surpasses the 8.2B model InstructBLIP with merely 2% parameters (150M), but also significantly outperforms other widely-used prompt methods on VLMs or task-specific methods.
• 摘要：超越单纯表面内容关系挖掘的深层多模态语义理解在人工智能领域受到越来越多的关注。收集和注释高质量多模态数据的挑战凸显了小样本学习的重要性。在本文中，我们重点关注这种背景下的两个关键任务：少样本多模态讽刺检测（MSD）和多模态情感分析（MSA）。为了解决这些问题，我们提出了混合提示专家与块感知提示融合（MoPE-BAF），这是一种基于统一视觉语言模型（VLM）的新型多模式软提示框架。具体来说，我们设计了三个软提示专家：提取模态特定特征以丰富单模态表示的文本提示和图像提示，以及辅助多模态交互的统一提示。此外，我们将 Transformer 层重组为多个块，并在相邻块之间引入跨模态提示注意力，从而平滑从单模态表示到多模态融合的过渡。在少样本设置的 MSD 和 MSA 数据集上，我们提出的模型不仅超过了仅 2% 参数（150M）的 8.2B 模型 InstructBLIP，而且还显着优于 VLM 上其他广泛使用的提示方法或特定于任务的方法。

Title: Few-Shot VQA with Frozen LLMs: A Tale of Two Approaches

• Authors: Igor Sterner, Weizhe Lin, Jinghong Chen, Bill Byrne
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.11317
• Pdf URL: https://arxiv.org/pdf/2403.11317
• Copy Paste: [[2403.11317]] Few-Shot VQA with Frozen LLMs: A Tale of Two Approaches(https://arxiv.org/abs/2403.11317)
• Keywords: language model, llm
• Abstract: Two approaches have emerged to input images into large language models (LLMs). The first is to caption images into natural language. The second is to map image feature embeddings into the domain of the LLM and pass the mapped embeddings directly to the LLM. The majority of recent few-shot multimodal work reports performance using architectures that employ variations of one of these two approaches. But they overlook an important comparison between them. We design a controlled and focused experiment to compare these two approaches to few-shot visual question answering (VQA) with LLMs. Our findings indicate that for Flan-T5 XL, a 3B parameter LLM, connecting visual embeddings directly to the LLM embedding space does not guarantee improved performance over using image captions. In the zero-shot regime, we find using textual image captions is better. In the few-shot regimes, how the in-context examples are selected determines which is better.
• 摘要：已经出现了两种将图像输入大型语言模型（LLM）的方法。第一个是将图像添加为自然语言的字幕。第二种是将图像特征嵌入映射到 LLM 的域中，并将映射的嵌入直接传递到 LLM。最近的大多数少样本多模式工作都使用采用这两种方法之一的变体的架构来报告性能。但他们忽略了它们之间的一个重要比较。我们设计了一个受控且集中的实验，以比较这两种与法学硕士进行少镜头视觉问答（VQA）的方法。我们的研究结果表明，对于 3B 参数 LLM Flan-T5 XL，将视觉嵌入直接连接到 LLM 嵌入空间并不能保证比使用图像标题提高性能。在零样本情况下，我们发现使用文本图像标题更好。在少数情况下，如何选择上下文中的示例决定了哪个更好。

Title: StateFlow: Enhancing LLM Task-Solving through State-Driven Workflows

• Authors: Yiran Wu, Tianwei Yue, Shaokun Zhang, Chi Wang, Qingyun Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.11322
• Pdf URL: https://arxiv.org/pdf/2403.11322
• Copy Paste: [[2403.11322]] StateFlow: Enhancing LLM Task-Solving through State-Driven Workflows(https://arxiv.org/abs/2403.11322)
• Keywords: language model, llm, prompt
• Abstract: It is a notable trend to use Large Language Models (LLMs) to tackle complex tasks, e.g., tasks that require a sequence of actions and dynamic interaction with tools and environments. In this paper, we propose StateFlow, a novel LLM-based task-solving paradigm that conceptualizes complex task-solving processes backed by LLMs as state machines. With proper construction of states and definition of state transitions, StateFlow grounds the progress of task-solving, ensuring clear tracking and management of LLMs' responses throughout the task-solving process. Within each state, StateFlow allows execution of a series of actions, involving not only the generation of LLM's responses guided by a specific prompt, but also the utilization of external tools as needed. State transitions are controlled by specific rules or decisions made by the LLM, allowing for a dynamic and adaptive progression through the task's pre-defined StateFlow model. Evaluations on the InterCode SQL and Bash benchmarks show that StateFlow significantly enhances LLMs' efficiency.
• 摘要：使用大型语言模型（LLM）来处理复杂任务是一个显着的趋势，例如需要一系列操作以及与工具和环境进行动态交互的任务。在本文中，我们提出了 StateFlow，这是一种基于 LLM 的新型任务解决范例，它将由 LLM 支持的复杂任务解决过程概念化为状态机。通过正确构建状态和定义状态转换，StateFlow 为任务解决的进度奠定了基础，确保在整个任务解决过程中清晰跟踪和管理法学硕士的响应。在每个状态中，StateFlow 允许执行一系列操作，不仅涉及在特定提示的指导下生成 LLM 响应，还涉及根据需要利用外部工具。状态转换由法学硕士制定的特定规则或决策控制，允许通过任务的预定义 StateFlow 模型实现动态和自适应进展。对 InterCode SQL 和 Bash 基准的评估表明 StateFlow 显着提高了法学硕士的效率。

Title: Improving Dialogue Agents by Decomposing One Global Explicit Annotation with Local Implicit Multimodal Feedback

• Authors: Dong Won Lee, Hae Won Park, Yoon Kim, Cynthia Breazeal, Louis-Philippe Morency
• Subjects: cs.CL, cs.AI, cs.HC, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.11330
• Pdf URL: https://arxiv.org/pdf/2403.11330
• Copy Paste: [[2403.11330]] Improving Dialogue Agents by Decomposing One Global Explicit Annotation with Local Implicit Multimodal Feedback(https://arxiv.org/abs/2403.11330)
• Keywords: llm, agent
• Abstract: We describe an approach for aligning an LLM-based dialogue agent based on global (i.e., dialogue-level) rewards, while also taking into account naturally-occurring multimodal signals. At a high level, our approach (dubbed GELI) learns a local, turn-level reward model by decomposing the human-provided Global Explicit (GE) session-level reward, using Local Implicit (LI} multimodal reward signals to crossmodally shape the reward decomposition step. This decomposed reward model is then used as part of the standard RHLF pipeline improve an LLM-based dialog agent. We run quantitative and qualitative human studies to evaluate the performance of our GELI approach, and find that it shows consistent improvements across various conversational metrics compared to baseline methods.
• 摘要：我们描述了一种根据全局（即对话级别）奖励来调整基于 LLM 的对话代理的方法，同时还考虑到自然发生的多模态信号。在较高的层面上，我们的方法（称为 GELI）通过分解人类提供的全局显式（GE）会话级奖励，使用本地隐式（LI）多模态奖励信号来跨模态地塑造奖励，从而学习本地的回合级奖励模型分解步骤。然后将这种分解的奖励模型用作标准 RHLF 管道的一部分，改进基于 LLM 的对话代理。我们进行定量和定性人类研究来评估我们的 GELI 方法的性能，并发现它在各种方面表现出一致的改进对话指标与基线方法的比较。

Title: What Makes Math Word Problems Challenging for LLMs?

• Authors: KV Aditya Srivatsa, Ekaterina Kochmar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.11369
• Pdf URL: https://arxiv.org/pdf/2403.11369
• Copy Paste: [[2403.11369]] What Makes Math Word Problems Challenging for LLMs?(https://arxiv.org/abs/2403.11369)
• Keywords: language model, llm
• Abstract: This paper investigates the question of what makes math word problems (MWPs) challenging for large language models (LLMs). We conduct an in-depth analysis of the key linguistic and mathematical characteristics of MWPs. In addition, we train feature-based classifiers to better understand the impact of each feature on the overall difficulty of MWPs for prominent LLMs and investigate whether this helps predict how well LLMs fare against specific categories of MWPs.
• 摘要：本文研究了数学应用题 (MWP) 对大型语言模型 (LLM) 具有挑战性的问题。我们对 MWP 的关键语言和数学特征进行了深入分析。此外，我们还训练基于特征的分类器，以更好地了解每个特征对著名法学硕士 MWP 整体难度的影响，并研究这是否有助于预测法学硕士相对于特定类别 MWP 的表现。

Title: X-LLaVA: Optimizing Bilingual Large Vision-Language Alignment

• Authors: Dongjae Shin, Hyunseok Lim, Inho Won, Changsu Choi, Minjun Kim, Seungwoo Song, Hangyeol Yoo, Sangmin Kim, Kyungtae Lim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.11399
• Pdf URL: https://arxiv.org/pdf/2403.11399
• Copy Paste: [[2403.11399]] X-LLaVA: Optimizing Bilingual Large Vision-Language Alignment(https://arxiv.org/abs/2403.11399)
• Keywords: language model, gpt, llm
• Abstract: The impressive development of large language models (LLMs) is expanding into the realm of large multimodal models (LMMs), which incorporate multiple types of data beyond text. However, the nature of multimodal models leads to significant expenses in the creation of training data. Furthermore, constructing multilingual data for LMMs presents its own set of challenges due to language diversity and complexity. Therefore, in this study, we propose two cost-effective methods to solve this problem: (1) vocabulary expansion and pretraining of multilingual LLM for specific languages, and (2) automatic and elaborate construction of multimodal datasets using GPT4-V. Based on015 these methods, we constructed a 91K English-Korean-Chinese multilingual, multimodal training dataset. Additionally, we developed a bilingual multimodal model that exhibits excellent performance in both Korean and English, surpassing existing approaches.
• 摘要：大型语言模型 (LLM) 令人印象深刻的发展正在扩展到大型多模态模型 (LMM) 领域，其中包含文本以外的多种类型的数据。然而，多模态模型的性质导致创建训练数据的费用很高。此外，由于语言的多样性和复杂性，为 LMM 构建多语言数据也带来了一系列挑战。因此，在本研究中，我们提出了两种经济有效的方法来解决这个问题：（1）针对特定语言的多语言LLM的词汇扩展和预训练，以及（2）使用GPT4-V自动且精细地构建多模态数据集。基于这些方法，我们构建了 91K 英韩汉多语言、多模态训练数据集。此外，我们开发了一种双语多模式模型，该模型在韩语和英语方面均表现出出色的性能，超越了现有方法。

Title: Dynamic Contexts for Generating Suggestion Questions in RAG Based Conversational Systems

• Authors: Anuja Tayal, Aman Tyagi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.11413
• Pdf URL: https://arxiv.org/pdf/2403.11413
• Copy Paste: [[2403.11413]] Dynamic Contexts for Generating Suggestion Questions in RAG Based Conversational Systems(https://arxiv.org/abs/2403.11413)
• Keywords: prompt, retrieval-augmented generation, agent
• Abstract: When interacting with Retrieval-Augmented Generation (RAG)-based conversational agents, the users must carefully craft their queries to be understood correctly. Yet, understanding the system's capabilities can be challenging for the users, leading to ambiguous questions that necessitate further clarification. This work aims to bridge the gap by developing a suggestion question generator. To generate suggestion questions, our approach involves utilizing dynamic context, which includes both dynamic few-shot examples and dynamically retrieved contexts. Through experiments, we show that the dynamic contexts approach can generate better suggestion questions as compared to other prompting approaches.
• 摘要：当与基于检索增强生成 (RAG) 的对话代理交互时，用户必须仔细设计他们的查询，以便正确理解。然而，了解系统的功能对于用户来说可能具有挑战性，从而导致需要进一步澄清的模糊问题。这项工作旨在通过开发建议问题生成器来弥合差距。为了生成建议问题，我们的方法涉及利用动态上下文，其中包括动态的小样本示例和动态检索的上下文。通过实验，我们表明与其他提示方法相比，动态上下文方法可以生成更好的建议问题。

Title: A Novel Paradigm Boosting Translation Capabilities of Large Language Models

• Authors: Jiaxin Guo, Hao Yang, Zongyao Li, Daimeng Wei, Hengchao Shang, Xiaoyu Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.11430
• Pdf URL: https://arxiv.org/pdf/2403.11430
• Copy Paste: [[2403.11430]] A Novel Paradigm Boosting Translation Capabilities of Large Language Models(https://arxiv.org/abs/2403.11430)
• Keywords: language model, gpt, llm
• Abstract: This paper presents a study on strategies to enhance the translation capabilities of large language models (LLMs) in the context of machine translation (MT) tasks. The paper proposes a novel paradigm consisting of three stages: Secondary Pre-training using Extensive Monolingual Data, Continual Pre-training with Interlinear Text Format Documents, and Leveraging Source-Language Consistent Instruction for Supervised Fine-Tuning. Previous research on LLMs focused on various strategies for supervised fine-tuning (SFT), but their effectiveness has been limited. While traditional machine translation approaches rely on vast amounts of parallel bilingual data, our paradigm highlights the importance of using smaller sets of high-quality bilingual data. We argue that the focus should be on augmenting LLMs' cross-lingual alignment abilities during pre-training rather than solely relying on extensive bilingual data during SFT. Experimental results conducted using the Llama2 model, particularly on Chinese-Llama2 after monolingual augmentation, demonstrate the improved translation capabilities of LLMs. A significant contribution of our approach lies in Stage2: Continual Pre-training with Interlinear Text Format Documents, which requires less than 1B training data, making our method highly efficient. Additionally, in Stage3, we observed that setting instructions consistent with the source language benefits the supervised fine-tuning process. Experimental results demonstrate that our approach surpasses previous work and achieves superior performance compared to models such as NLLB-54B and GPT3.5-text-davinci-003, despite having a significantly smaller parameter count of only 7B or 13B. This achievement establishes our method as a pioneering strategy in the field of machine translation.
• 摘要：本文提出了一项关于在机器翻译（MT）任务背景下增强大型语言模型（LLM）翻译能力的策略的研究。该论文提出了一种由三个阶段组成的新颖范式：使用大量单语言数据的二次预训练、使用行间文本格式文档的持续预训练以及利用源语言一致指令进行监督微调。之前对法学硕士的研究主要集中在监督微调（SFT）的各种策略上，但其有效性有限。虽然传统的机器翻译方法依赖于大量并行双语数据，但我们的范例强调了使用较小的高质量双语数据集的重要性。我们认为，重点应该放在预训练期间增强法学硕士的跨语言对齐能力，而不是在 SFT 期间仅仅依赖大量双语数据。使用 Llama2 模型进行的实验结果，特别是单语增强后的 Chinese-Llama2，证明了法学硕士翻译能力的提高。我们的方法的一个重要贡献在于阶段2：使用行间文本格式文档进行持续预训练，它需要少于1B的训练数据，使得我们的方法非常高效。此外，在第三阶段，我们观察到设置与源语言一致的指令有利于监督微调过程。实验结果表明，尽管参数数量明显减少，仅为 7B 或 13B，但与 NLLB-54B 和 GPT3.5-text-davinci-003 等模型相比，我们的方法超越了之前的工作并实现了卓越的性能。这一成就确立了我们的方法作为机器翻译领域的开创性策略。

Title: InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructions

• Authors: Yifan Wang, Yafei Liu, Chufan Shi, Haoling Li, Chen Chen, Haonan Lu, Yujiu Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.11435
• Pdf URL: https://arxiv.org/pdf/2403.11435
• Copy Paste: [[2403.11435]] InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructions(https://arxiv.org/abs/2403.11435)
• Keywords: language model, llm
• Abstract: Instruction tuning effectively optimizes Large Language Models (LLMs) for downstream tasks. Due to the changing environment in real-life applications, LLMs necessitate continual task-specific adaptation without catastrophic forgetting. Considering the heavy computational cost, replay-based Continual Learning (CL) methods are the simplest and most widely used for LLMs to address the forgetting issue. However, traditional replay-based methods do not fully utilize instructions to customize the replay strategy. In this work, we propose a novel paradigm called Instruction-based Continual Learning (InsCL). InsCL dynamically replays previous data based on task similarity, calculated by Wasserstein Distance with instructions. Moreover, we further introduce an Instruction Information Metric (InsInfo) to quantify the complexity and diversity of instructions. According to InsInfo, InsCL guides the replay process more inclined to high-quality data. We conduct extensive experiments over 16 tasks with different training orders, observing consistent performance improvements of InsCL. When all tasks have been trained, InsCL achieves performance gains of 3.0 Relative Gain compared with Random Replay, and 27.96 Relative Gain compared with No Replay.
• 摘要：指令调优可有效优化下游任务的大型语言模型 (LLM)。由于现实生活中应用环境的不断变化，法学硕士需要不断针对特定任务进行适应，以免发生灾难性遗忘。考虑到巨大的计算成本，基于重播的持续学习（CL）方法是法学硕士解决遗忘问题最简单且最广泛使用的方法。然而，传统的基于重放的方法并没有充分利用指令来定制重放策略。在这项工作中，我们提出了一种称为基于指令的持续学习（InsCL）的新颖范式。 InsCL 根据任务相似性动态重播先前的数据，通过 Wasserstein Distance 和指令计算。此外，我们进一步引入指令信息度量（InsInfo）来量化指令的复杂性和多样性。据InsInfo介绍，InsCL引导重播过程更倾向于高质量数据。我们使用不同的训练顺序对 16 个任务进行了广泛的实验，观察到 InsCL 的性能持续提高。当所有任务都经过训练后，InsCL 与随机重放相比，实现了 3.0 相对增益的性能提升，与无重放相比，实现了 27.96 相对增益的性能提升。

Title: StyleChat: Learning Recitation-Augmented Memory in LLMs for Stylized Dialogue Generation

• Authors: Jinpeng Li, Zekai Zhang, Quan Tu, Xin Cheng, Dongyan Zhao, Rui Yan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.11439
• Pdf URL: https://arxiv.org/pdf/2403.11439
• Copy Paste: [[2403.11439]] StyleChat: Learning Recitation-Augmented Memory in LLMs for Stylized Dialogue Generation(https://arxiv.org/abs/2403.11439)
• Keywords: language model, llm, prompt, chat, agent
• Abstract: Large Language Models (LLMs) demonstrate superior performance in generative scenarios and have attracted widespread attention. Among them, stylized dialogue generation is essential in the context of LLMs for building intelligent and engaging dialogue agent. However the ability of LLMs is data-driven and limited by data bias, leading to poor performance on specific tasks. In particular, stylized dialogue generation suffers from a severe lack of supervised data. Furthermore, although many prompt-based methods have been proposed to accomplish specific tasks, their performance in complex real-world scenarios involving a wide variety of dialog styles further enhancement. In this work, we first introduce a stylized dialogue dataset StyleEval with 38 styles by leveraging the generative power of LLMs comprehensively, which has been carefully constructed with rigorous human-led quality control. Based on this, we propose the stylized dialogue framework StyleChat via recitation-augmented memory strategy and multi-task style learning strategy to promote generalization ability. To evaluate the effectiveness of our approach, we created a test benchmark that included both a generation task and a choice task to comprehensively evaluate trained models and assess whether styles and preferences are remembered and understood. Experimental results show that our proposed framework StyleChat outperforms all the baselines and helps to break the style boundary of LLMs.
• 摘要：大型语言模型（LLM）在生成场景中表现出卓越的性能，并引起了广泛的关注。其中，风格化对话生成对于法学硕士的背景下构建智能且引人入胜的对话代理至关重要。然而，法学硕士的能力是数据驱动的，并受到数据偏差的限制，导致在特定任务上表现不佳。特别是，程式化对话生成严重缺乏监督数据。此外，尽管已经提出了许多基于提示的方法来完成特定任务，但它们在涉及各种对话风格的复杂现实场景中的性能进一步增强。在这项工作中，我们首先综合利用法学硕士的生成能力，引入了具有 38 种风格的风格化对话数据集 StyleEval，该数据集是在严格的人工质量控制下精心构建的。基于此，我们提出了风格化对话框架StyleChat，通过背诵增强记忆策略和多任务风格学习策略来提升泛化能力。为了评估我们方法的有效性，我们创建了一个测试基准，其中包括生成任务和选择任务，以全面评估经过训练的模型并评估风格和偏好是否被记住和理解。实验结果表明，我们提出的框架 StyleChat 优于所有基线，有助于打破法学硕士的风格边界。

Title: HateCOT: An Explanation-Enhanced Dataset for Generalizable Offensive Speech Detection via Large Language Models

• Authors: Huy Nghiem, Hal Daumé III
• Subjects: cs.CL, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2403.11456
• Pdf URL: https://arxiv.org/pdf/2403.11456
• Copy Paste: [[2403.11456]] HateCOT: An Explanation-Enhanced Dataset for Generalizable Offensive Speech Detection via Large Language Models(https://arxiv.org/abs/2403.11456)
• Keywords: language model, gpt
• Abstract: The ubiquitousness of social media has led to the need for reliable and efficient detection of offensive content to limit harmful effects. This has led to a proliferation of datasets and models related to detecting offensive content. While sophisticated models have attained strong performance on individual datasets, these models often do not generalize due to differences between how "offensive content" is conceptualized, and the resulting differences in how these datasets are labeled. In this paper, we introduce HateCOT, a dataset of 52,000 samples drawn from diverse existing sources with explanations generated by GPT-3.5-Turbo and human-curated. We show that pre-training models for the detection of offensive content on HateCOT significantly boots open-sourced Language Models on three benchmark datasets in both zero and few-shot settings, despite differences in domain and task.} We further find that HateCOT enables effective K-shot fine-tuning in the low-resource settings.
• 摘要：社交媒体的无处不在导致需要可靠、有效地检测攻击性内容以限制有害影响。这导致与检测攻击性内容相关的数据集和模型激增。虽然复杂的模型在单个数据集上获得了强大的性能，但由于“攻击性内容”的概念化方式以及这些数据集的标记方式之间的差异，这些模型通常无法概括。在本文中，我们介绍了 HateCOT，这是一个包含 52,000 个样本的数据集，这些样本来自不同的现有来源，并由 GPT-3.5-Turbo 生成并由人工策划提供解释。我们表明，尽管领域和任务存在差异，但 HateCOT 上用于检测攻击性内容的预训练模型在零样本和少样本设置的三个基准数据集上显着启动了开源语言模型。}我们进一步发现，HateCOT 能够有效地启动开源语言模型。在低资源设置下进行 K 镜头微调。

Title: Word Order's Impacts: Insights from Reordering and Generation Analysis

• Authors: Qinghua Zhao, Jiaang Li, Lei Li, Zenghui Zhou, Junfeng Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.11473
• Pdf URL: https://arxiv.org/pdf/2403.11473
• Copy Paste: [[2403.11473]] Word Order's Impacts: Insights from Reordering and Generation Analysis(https://arxiv.org/abs/2403.11473)
• Keywords: gpt, chat
• Abstract: Existing works have studied the impacts of the order of words within natural text. They usually analyze it by destroying the original order of words to create a scrambled sequence, and then comparing the models' performance between the original and scrambled sequences. The experimental results demonstrate marginal drops. Considering this findings, different hypothesis about word order is proposed, including ``the order of words is redundant with lexical semantics'', and ``models do not rely on word order''. In this paper, we revisit the aforementioned hypotheses by adding a order reconstruction perspective, and selecting datasets of different spectrum. Specifically, we first select four different datasets, and then design order reconstruction and continuing generation tasks. Empirical findings support that ChatGPT relies on word order to infer, but cannot support or negate the redundancy relations between word order lexical semantics.
• 摘要：现有的作品研究了自然文本中词序的影响。他们通常通过破坏原始单词顺序来创建扰乱序列来进行分析，然后比较原始序列和扰乱序列之间的模型性能。实验结果表明边际下降。考虑到这一发现，提出了关于词序的不同假设，包括“词序与词汇语义是冗余的”和“模型不依赖于词序”。在本文中，我们通过添加顺序重建视角并选择不同频谱的数据集来重新审视上述假设。具体来说，我们首先选择四个不同的数据集，然后设计顺序重建和继续生成任务。实证结果支持ChatGPT依赖词序进行推理，但不能支持或否定词序词汇语义之间的冗余关系。

Title: DEE: Dual-stage Explainable Evaluation Method for Text Generation

• Authors: Shenyu Zhang, Yu Li, Rui Wu, Xiutian Huang, Yongrui Chen, Wenhao Xu, Guilin Qi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.11509
• Pdf URL: https://arxiv.org/pdf/2403.11509
• Copy Paste: [[2403.11509]] DEE: Dual-stage Explainable Evaluation Method for Text Generation(https://arxiv.org/abs/2403.11509)
• Keywords: language model, llm, hallucination
• Abstract: Automatic methods for evaluating machine-generated texts hold significant importance due to the expanding applications of generative systems. Conventional methods tend to grapple with a lack of explainability, issuing a solitary numerical score to signify the assessment outcome. Recent advancements have sought to mitigate this limitation by incorporating large language models (LLMs) to offer more detailed error analyses, yet their applicability remains constrained, particularly in industrial contexts where comprehensive error coverage and swift detection are paramount. To alleviate these challenges, we introduce DEE, a Dual-stage Explainable Evaluation method for estimating the quality of text generation. Built upon Llama 2, DEE follows a dual-stage principle guided by stage-specific instructions to perform efficient identification of errors in generated texts in the initial stage and subsequently delves into providing comprehensive diagnostic reports in the second stage. DEE is fine-tuned on our elaborately assembled dataset AntEval, which encompasses 15K examples from 4 real-world applications of Alipay that employ generative systems. The dataset concerns newly emerged issues like hallucination and toxicity, thereby broadening the scope of DEE's evaluation criteria. Experimental results affirm that DEE's superiority over existing evaluation methods, achieving significant improvements in both human correlation as well as efficiency.
• 摘要：由于生成系统应用的不断扩展，评估机器生成文本的自动方法变得非常重要。传统方法往往会解决缺乏可解释性的问题，通过发布单独的数字分数来表示评估结果。最近的进展试图通过合并大型语言模型（LLM）来提供更详细的错误分析来减轻这一限制，但它们的适用性仍然受到限制，特别是在全面错误覆盖和快速检测至关重要的工业环境中。为了缓解这些挑战，我们引入了 DEE，一种用于估计文本生成质量的双阶段可解释评估方法。 DEE 基于 Llama 2 构建，遵循双阶段原则，以阶段特定指令为指导，在初始阶段对生成文本中的错误进行有效识别，随后在第二阶段深入提供全面的诊断报告。 DEE 在我们精心组装的数据集 AntEval 上进行了微调，该数据集包含来自 4 个使用生成系统的支付宝实际应用程序的 15K 个示例。该数据集涉及幻觉和毒性等新出现的问题，从而扩大了 DEE 评估标准的范围。实验结果证实了DEE相对于现有评估方法的优越性，在人类相关性和效率方面均取得了显着提高。

Title: Reinforcement Learning with Token-level Feedback for Controllable Text Generation

• Authors: Wendi Li, Wei Wei, Kaihe Xu, Wenfeng Xie, Dangyang Chen, Yu Cheng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.11558
• Pdf URL: https://arxiv.org/pdf/2403.11558
• Copy Paste: [[2403.11558]] Reinforcement Learning with Token-level Feedback for Controllable Text Generation(https://arxiv.org/abs/2403.11558)
• Keywords: language model, llm
• Abstract: To meet the requirements of real-world applications, it is essential to control generations of large language models (LLMs). Prior research has tried to introduce reinforcement learning (RL) into controllable text generation while most existing methods suffer from overfitting issues (finetuning-based methods) or semantic collapse (post-processing methods). However, current RL methods are generally guided by coarse-grained (sentence/paragraph-level) feedback, which may lead to suboptimal performance owing to semantic twists or progressions within sentences. To tackle that, we propose a novel reinforcement learning algorithm named TOLE which formulates TOken-LEvel rewards for controllable text generation, and employs a "first-quantize-then-noise" paradigm to enhance the robustness of the RL algorithm.Furthermore, TOLE can be flexibly extended to multiple constraints with little computational expense. Experimental results show that our algorithm can achieve superior performance on both single-attribute and multi-attribute control tasks. We have released our codes at https://github.com/WindyLee0822/CTG
• 摘要：为了满足实际应用的要求，控制大语言模型（LLM）的生成至关重要。先前的研究试图将强化学习（RL）引入可控文本生成中，而大多数现有方法都存在过度拟合问题（基于微调的方法）或语义崩溃（后处理方法）。然而，当前的强化学习方法通​​常由粗粒度（句子/段落级）反馈引导，这可能会由于句子内的语义扭曲或进展而导致性能不佳。为了解决这个问题，我们提出了一种名为 TOLE 的新型强化学习算法，它为可控文本生成制定 TOken-LEvel 奖励，并采用“先量化然后噪声”范式来增强 RL 算法的鲁棒性。此外，TOLE 可以可以以很少的计算开销灵活地扩展到多个约束。实验结果表明，我们的算法在单属性和多属性控制任务上都能取得优异的性能。我们已经在 https://github.com/WindyLee0822/CTG 发布了我们的代码

Title: Let's Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Model

• Authors: Haoyun Xu, Runzhe Zhan, Derek F. Wong, Lidia S. Chao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.11621
• Pdf URL: https://arxiv.org/pdf/2403.11621
• Copy Paste: [[2403.11621]] Let's Focus on Neuron: Neuron-Level Supervised Fine-tuning for Large Language Model(https://arxiv.org/abs/2403.11621)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are composed of neurons that exhibit various behaviors and roles, which become increasingly diversified as models scale. Recent studies have revealed that not all neurons are active across different datasets, and this sparsity correlates positively with the task-specific ability, leading to advancements in model pruning and training efficiency. Traditional fine-tuning methods engage all parameters of LLMs, which is computationally expensive and may not be necessary. In contrast, Parameter-Efficient Fine-Tuning (PEFT) approaches aim to minimize the number of trainable parameters, yet they still operate at a relatively macro scale (e.g., layer-level). We introduce Neuron-Level Fine-Tuning (NeFT), a novel approach that refines the granularity of parameter training down to the individual neuron, enabling more precise and computationally efficient model updates. The experimental results show that NeFT not only exceeded the performance of full-parameter fine-tuning and PEFT but also provided insights into the analysis of neurons.
• 摘要：大型语言模型 (LLM) 由表现出各种行为和角色的神经元组成，随着模型规模的扩大，这些神经元变得越来越多样化。最近的研究表明，并非所有神经元在不同的数据集中都是活跃的，这种稀疏性与特定任务的能力呈正相关，从而导致模型剪枝和训练效率的进步。传统的微调方法涉及法学硕士的所有参数，这在计算上是昂贵的并且可能没有必要。相比之下，参数高效微调（PEFT）方法旨在最大限度地减少可训练参数的数量，但它们仍然在相对宏观的尺度（例如层级）上运行。我们引入了神经元级微调（NeFT），这是一种将参数训练的粒度细化到单个神经元的新颖方法，从而实现更精确和计算效率更高的模型更新。实验结果表明，NeFT不仅超越了全参数微调和PEFT的性能，而且为神经元的分析提供了见解。

Title: Embedded Named Entity Recognition using Probing Classifiers

• Authors: Nicholas Popovič, Michael Färber
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.11747
• Pdf URL: https://arxiv.org/pdf/2403.11747
• Copy Paste: [[2403.11747]] Embedded Named Entity Recognition using Probing Classifiers(https://arxiv.org/abs/2403.11747)
• Keywords: language model, gpt, retrieval augmented generation
• Abstract: Extracting semantic information from generated text is a useful tool for applications such as automated fact checking or retrieval augmented generation. Currently, this requires either separate models during inference, which increases computational cost, or destructive fine-tuning of the language model. Instead, we propose directly embedding information extraction capabilities into pre-trained language models using probing classifiers, enabling efficient simultaneous text generation and information extraction. For this, we introduce an approach called EMBER and show that it enables named entity recognition in decoder-only language models without fine-tuning them and while incurring minimal additional computational cost at inference time. Specifically, our experiments using GPT-2 show that EMBER maintains high token generation rates during streaming text generation, with only a negligible decrease in speed of around 1% compared to a 43.64% slowdown measured for a baseline using a separate NER model. Code and data are available at https://github.com/nicpopovic/EMBER.
• 摘要：从生成的文本中提取语义信息对于自动事实检查或检索增强生成等应用程序来说是一个有用的工具。目前，这需要在推理过程中使用单独的模型（这会增加计算成本），或者对语言模型进行破坏性微调。相反，我们建议使用探测分类器将信息提取功能直接嵌入到预先训练的语言模型中，从而实现高效的同步文本生成和信息提取。为此，我们引入了一种称为 EMBER 的方法，并表明它可以在仅解码器的语言模型中实现命名实体识别，而无需对其进行微调，同时在推理时产生最小的额外计算成本。具体来说，我们使用 GPT-2 的实验表明，EMBER 在流式文本生成过程中保持了较高的令牌生成率，与使用单独的 NER 模型测量的基线的 43.64% 的减速相比，速度仅下降了 1% 左右，可以忽略不计。代码和数据可在 https://github.com/nicpopovic/EMBER 获取。

Title: Revisiting The Classics: A Study on Identifying and Rectifying Gender Stereotypes in Rhymes and Poems

• Authors: Aditya Narayan Sankaran, Vigneshwaran Shankaran, Sampath Lonka, Rajesh Sharma
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.11752
• Pdf URL: https://arxiv.org/pdf/2403.11752
• Copy Paste: [[2403.11752]] Revisiting The Classics: A Study on Identifying and Rectifying Gender Stereotypes in Rhymes and Poems(https://arxiv.org/abs/2403.11752)
• Keywords: language model, llm
• Abstract: Rhymes and poems are a powerful medium for transmitting cultural norms and societal roles. However, the pervasive existence of gender stereotypes in these works perpetuates biased perceptions and limits the scope of individuals' identities. Past works have shown that stereotyping and prejudice emerge in early childhood, and developmental research on causal mechanisms is critical for understanding and controlling stereotyping and prejudice. This work contributes by gathering a dataset of rhymes and poems to identify gender stereotypes and propose a model with 97\% accuracy to identify gender bias. Gender stereotypes were rectified using a Large Language Model (LLM) and its effectiveness was evaluated in a comparative survey against human educator rectifications. To summarize, this work highlights the pervasive nature of gender stereotypes in literary works and reveals the potential of LLMs to rectify gender stereotypes. This study raises awareness and promotes inclusivity within artistic expressions, making a significant contribution to the discourse on gender equality.
• 摘要：韵律和诗歌是传播文化规范和社会角色的强大媒介。然而，这些作品中普遍存在的性别刻板印象使偏见观念长期存在，并限制了个人身份的范围。过去的研究表明，成见和偏见在儿童早期就出现了，因果机制的发展研究对于理解和控制成见和偏见至关重要。这项工作通过收集韵律和诗歌数据集来识别性别刻板印象，并提出一个准确率高达 97% 的模型来识别性别偏见。使用大语言模型（LLM）纠正性别刻板印象，并通过与人类教育者纠正的比较调查评估其有效性。总而言之，这项工作凸显了文学作品中普遍存在的性别刻板印象，并揭示了法学硕士纠正性别刻板印象的潜力。这项研究提高了人们的认识并促进了艺术表现形式的包容性，为性别平等的讨论做出了重大贡献。

Title: Construction of Hyper-Relational Knowledge Graphs Using Pre-Trained Large Language Models

• Authors: Preetha Datta, Fedor Vitiugin, Anastasiia Chizhikova, Nitin Sawhney
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.11786
• Pdf URL: https://arxiv.org/pdf/2403.11786
• Copy Paste: [[2403.11786]] Construction of Hyper-Relational Knowledge Graphs Using Pre-Trained Large Language Models(https://arxiv.org/abs/2403.11786)
• Keywords: language model, gpt, prompt
• Abstract: Extracting hyper-relations is crucial for constructing comprehensive knowledge graphs, but there are limited supervised methods available for this task. To address this gap, we introduce a zero-shot prompt-based method using OpenAI's GPT-3.5 model for extracting hyper-relational knowledge from text. Comparing our model with a baseline, we achieved promising results, with a recall of 0.77. Although our precision is currently lower, a detailed analysis of the model outputs has uncovered potential pathways for future research in this area.
• 摘要：提取超关系对于构建全面的知识图至关重要，但可用于此任务的监督方法有限。为了解决这一差距，我们引入了一种基于零样本提示的方法，使用 OpenAI 的 GPT-3.5 模型从文本中提取超关系知识。将我们的模型与基线进行比较，我们取得了可喜的结果，召回率为 0.77。尽管我们目前的精度较低，但对模型输出的详细分析揭示了该领域未来研究的潜在途径。

Title: Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus

• Authors: Seungpil Lee, Woochang Sim, Donghyeon Shin, Sanha Hwang, Wongyu Seo, Jiwon Park, Seokki Lee, Sejin Kim, Sundong Kim
• Subjects: cs.CL, cs.AI, cs.ET, cs.SC
• Abstract URL: https://arxiv.org/abs/2403.11793
• Pdf URL: https://arxiv.org/pdf/2403.11793
• Copy Paste: [[2403.11793]] Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus(https://arxiv.org/abs/2403.11793)
• Keywords: language model, llm
• Abstract: The existing methods for evaluating the inference abilities of Large Language Models (LLMs) have been results-centric, making it difficult to assess the inference process. We introduce a new approach using the Abstract and Reasoning Corpus (ARC) dataset to evaluate the inference and contextual understanding abilities of large language models in a process-centric manner. ARC demands rigorous logical structures for problem-solving, making it a benchmark that facilitates the comparison of model inference abilities with humans. Experimental results confirm that while large language models possess weak inference abilities, they still lag in terms of logical coherence, compositionality, and productivity. Our experiments highlight the reasoning capabilities of LLMs, proposing development paths for achieving human-level reasoning.
• 摘要：现有的评估大型语言模型（LLM）推理能力的方法一直以结果为中心，因此很难评估推理过程。我们引入了一种使用抽象和推理语料库（ARC）数据集以过程为中心的方式评估大型语言模型的推理和上下文理解能力的新方法。 ARC 需要严格的逻辑结构来解决问题，这使其成为有利于模型推理能力与人类进行比较的基准。实验结果证实，虽然大型语言模型的推理能力较弱，但在逻辑连贯性、组合性和生产力方面仍然落后。我们的实验突出了法学硕士的推理能力，提出了实现人类水平推理的发展路径。

Title: Counting-Stars: A Simple, Efficient, and Reasonable Strategy for Evaluating Long-Context Large Language Models

• Authors: Mingyang Song, Mao Zheng, Xuan Luo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.11802
• Pdf URL: https://arxiv.org/pdf/2403.11802
• Copy Paste: [[2403.11802]] Counting-Stars: A Simple, Efficient, and Reasonable Strategy for Evaluating Long-Context Large Language Models(https://arxiv.org/abs/2403.11802)
• Keywords: language model, gpt, llm, long context, chat
• Abstract: While recent research endeavors have concentrated on developing Large Language Models (LLMs) with robust long-context capabilities, due to the lack of appropriate evaluation strategies, relatively little is known about how well the long-context processing abilities and performance of leading LLMs (e.g., ChatGPT and KimiChat). To address this gap, we propose a simple, efficient, and reasonable strategy for evaluating long-context LLMs as a new benchmark, named Counting-Stars. The Counting-Stars is designed to require LLMs to fully understand and capture long dependencies in long contexts and be able to collect inter-dependency across multiple pieces of evidence spanning the entire context to finish the task. Based on the Counting-Stars, we conduct experiments to evaluate the two leading long-context LLMs, i.e., GPT-4 Turbo and Kimi Chat. The experimental results indicate that GPT-4 Turbo and Kimi Chat achieve significant performance in the long context from 4K to 128K. We further present two intriguing analyses regarding the behavior of LLMs processing long context.
• 摘要：虽然最近的研究工作集中在开发具有强大的长上下文能力的大型语言模型（LLM），但由于缺乏适当的评估策略，人们对领先的 LLM 的长上下文处理能力和性能知之甚少（例如，LLM）。 、ChatGPT 和 KimiChat）。为了解决这一差距，我们提出了一种简单、高效、合理的策略来评估长背景法学硕士作为新的基准，称为“计数之星”。计数之星旨在要求法学硕士充分理解和捕获长上下文中的长依赖关系，并能够收集跨越整个上下文的多个证据的相互依赖关系以完成任务。基于 Counting-Stars，我们进行了实验来评估两个领先的长上下文 LLM，即 GPT-4 Turbo 和 Kimi Chat。实验结果表明，GPT-4 Turbo 和 Kimi Chat 在 4K 到 128K 的长上下文中取得了显着的性能。我们进一步提出了关于法学硕士处理长上下文行为的两项有趣的分析。

Title: Metaphor Understanding Challenge Dataset for LLMs

• Authors: Xiaoyu Tong, Rochelle Choenni, Martha Lewis, Ekaterina Shutova
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.11810
• Pdf URL: https://arxiv.org/pdf/2403.11810
• Copy Paste: [[2403.11810]] Metaphor Understanding Challenge Dataset for LLMs(https://arxiv.org/abs/2403.11810)
• Keywords: language model, gpt, llm
• Abstract: Metaphors in natural language are a reflection of fundamental cognitive processes such as analogical reasoning and categorisation, and are deeply rooted in everyday communication. Metaphor understanding is therefore an essential task for large language models (LLMs). We release the Metaphor Understanding Challenge Dataset (MUNCH), designed to evaluate the metaphor understanding capabilities of LLMs. The dataset provides over 10k paraphrases for sentences containing metaphor use, as well as 1.5k instances containing inapt paraphrases. The inapt paraphrases were carefully selected to serve as control to determine whether the model indeed performs full metaphor interpretation or rather resorts to lexical similarity. All apt and inapt paraphrases were manually annotated. The metaphorical sentences cover natural metaphor uses across 4 genres (academic, news, fiction, and conversation), and they exhibit different levels of novelty. Experiments with LLaMA and GPT-3.5 demonstrate that MUNCH presents a challenging task for LLMs. The dataset is freely accessible at https://github.com/xiaoyuisrain/metaphor-understanding-challenge.
• 摘要：自然语言中的隐喻是类比推理和分类等基本认知过程的反映，并且深深植根于日常交流中。因此，隐喻理解是大型语言模型（LLM）的一项基本任务。我们发布了隐喻理解挑战数据集（MUNCH），旨在评估法学硕士的隐喻理解能力。该数据集提供了超过 10k 个包含隐喻使用的句子的释义，以及 1.5k 个包含不恰当释义的实例。精心选择不恰当的释义作为控制，以确定模型是否确实执行完整的隐喻解释，或者更确切地说诉诸词汇相似性。所有恰当和不恰当的释义均经过手动注释。这些隐喻句子涵盖了 4 个流派（学术、新闻、小说和对话）的自然隐喻用法，并且表现出不同程度的新颖性。 LLaMA 和 GPT-3.5 的实验表明，MUNCH 对法学硕士来说是一项具有挑战性的任务。该数据集可在 https://github.com/xiaoyuisrain/metaphor-understanding-challenge 上免费访问。

Title: SSCAE -- Semantic, Syntactic, and Context-aware natural language Adversarial Examples generator

• Authors: Javad Rafiei Asl, Mohammad H. Rafiei, Manar Alohaly, Daniel Takabi
• Subjects: cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.11833
• Pdf URL: https://arxiv.org/pdf/2403.11833
• Copy Paste: [[2403.11833]] SSCAE -- Semantic, Syntactic, and Context-aware natural language Adversarial Examples generator(https://arxiv.org/abs/2403.11833)
• Keywords: language model
• Abstract: Machine learning models are vulnerable to maliciously crafted Adversarial Examples (AEs). Training a machine learning model with AEs improves its robustness and stability against adversarial attacks. It is essential to develop models that produce high-quality AEs. Developing such models has been much slower in natural language processing (NLP) than in areas such as computer vision. This paper introduces a practical and efficient adversarial attack model called SSCAE for \textbf{S}emantic, \textbf{S}yntactic, and \textbf{C}ontext-aware natural language \textbf{AE}s generator. SSCAE identifies important words and uses a masked language model to generate an early set of substitutions. Next, two well-known language models are employed to evaluate the initial set in terms of semantic and syntactic characteristics. We introduce (1) a dynamic threshold to capture more efficient perturbations and (2) a local greedy search to generate high-quality AEs. As a black-box method, SSCAE generates humanly imperceptible and context-aware AEs that preserve semantic consistency and the source language's syntactical and grammatical requirements. The effectiveness and superiority of the proposed SSCAE model are illustrated with fifteen comparative experiments and extensive sensitivity analysis for parameter optimization. SSCAE outperforms the existing models in all experiments while maintaining a higher semantic consistency with a lower query number and a comparable perturbation rate.
• 摘要：机器学习模型容易受到恶意制作的对抗性示例 (AE) 的影响。使用 AE 训练机器学习模型可以提高其针对对抗性攻击的鲁棒性和稳定性。开发产生高质量 AE 的模型至关重要。自然语言处理（NLP）领域的此类模型的开发速度比计算机视觉等领域要慢得多。本文介绍了一种实用且高效的对抗性攻击模型，称为 SSCAE，用于 \textbf{S}emantic、\textbf{S}yntropic 和 \textbf{C}ontext-aware 自然语言 \textbf{AE}s 生成器。 SSCAE 识别重要的单词并使用掩码语言模型来生成一组早期的替换。接下来，采用两种著名的语言模型来评估初始集的语义和句法特征。我们引入（1）动态阈值来捕获更有效的扰动，以及（2）局部贪婪搜索来生成高质量的 AE。作为一种黑盒方法，SSCAE 生成人类难以察觉的上下文感知 AE，从而保持语义一致性以及源语言的句法和语法要求。通过十五个对比实验和参数优化的广泛敏感性分析，说明了所提出的 SSCAE 模型的有效性和优越性。 SSCAE 在所有实验中均优于现有模型，同时以较低的查询数和可比较的扰动率保持较高的语义一致性。

Title: Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models

• Authors: Yi Luo, Zhenghao Lin, Yuhao Zhang, Jiashuo Sun, Chen Lin, Chengjin Xu, Xiangdong Su, Yelong Shen, Jian Guo, Yeyun Gong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.11838
• Pdf URL: https://arxiv.org/pdf/2403.11838
• Copy Paste: [[2403.11838]] Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models(https://arxiv.org/abs/2403.11838)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) exhibit impressive capabilities but also present risks such as biased content generation and privacy issues. One of the current alignment techniques includes principle-driven integration, but it faces challenges arising from the imprecision of manually crafted rules and inadequate risk perception in models without safety training. To address these, we introduce Guide-Align, a two-stage approach. Initially, a safety-trained model identifies potential risks and formulates specific guidelines for various inputs, thereby establishing a comprehensive library of guidelines and models for input-guidelines retrieval. Subsequently, the retrieval model correlates new inputs with pertinent guidelines, guiding LLMs in response generation to ensure safe and high-quality outputs, thus aligning with human values. An additional optional stage involves fine-tuning a model with new well-aligned datasets generated through the process implemented in the second stage. Our method customizes guidelines to accommodate diverse inputs, thereby enhancing the fine-grainedness and comprehensiveness of the guideline library. Furthermore, it incorporates safety expertise from a safety-trained LLM through a lightweight retrieval model. We evaluated our approach on three benchmarks, demonstrating significant improvements in LLM security and quality. Notably, our fine-tuned model, Labrador, even at 13 billion parameters, outperforms GPT-3.5-turbo and surpasses GPT-4 in alignment capabilities.
• 摘要：大型语言模型 (LLM) 展现出令人印象深刻的功能，但也存在诸如有偏见的内容生成和隐私问题等风险。当前的对齐技术之一包括原则驱动的集成，但它面临着由于手工制定的规则不精确以及未经安全培训的模型中风险感知不足而带来的挑战。为了解决这些问题，我们引入了 Guide-Align，这是一种两阶段的方法。最初，经过安全训练的模型识别潜在风险并为各种输入制定具体指南，从而建立一个用于输入指南检索的综合指南和模型库。随后，检索模型将新的输入与相关指南相关联，指导法学硕士生成响应，以确保安全和高质量的输出，从而与人类价值观保持一致。另一个可选阶段涉及使用第二阶段实施的过程生成的新的对齐良好的数据集来微调模型。我们的方法定制指南以适应不同的输入，从而增强指南库的细粒度和全面性。此外，它通过轻量级检索模型融合了经过安全培训的法学硕士的安全专业知识。我们根据三个基准评估了我们的方法，证明了 LLM 安全性和质量的显着改进。值得注意的是，我们的微调模型 Labrador 即使在 130 亿个参数下，其性能也优于 GPT-3.5-turbo，并且在对齐能力方面也超过了 GPT-4。

Title: GPT-4 as Evaluator: Evaluating Large Language Models on Pest Management in Agriculture

• Authors: Shanglong Yang, Zhipeng Yuan, Shunbao Li, Ruoling Peng, Kang Liu, Po Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.11858
• Pdf URL: https://arxiv.org/pdf/2403.11858
• Copy Paste: [[2403.11858]] GPT-4 as Evaluator: Evaluating Large Language Models on Pest Management in Agriculture(https://arxiv.org/abs/2403.11858)
• Keywords: language model, gpt, llm, prompt
• Abstract: In the rapidly evolving field of artificial intelligence (AI), the application of large language models (LLMs) in agriculture, particularly in pest management, remains nascent. We aimed to prove the feasibility by evaluating the content of the pest management advice generated by LLMs, including the Generative Pre-trained Transformer (GPT) series from OpenAI and the FLAN series from Google. Considering the context-specific properties of agricultural advice, automatically measuring or quantifying the quality of text generated by LLMs becomes a significant challenge. We proposed an innovative approach, using GPT-4 as an evaluator, to score the generated content on Coherence, Logical Consistency, Fluency, Relevance, Comprehensibility, and Exhaustiveness. Additionally, we integrated an expert system based on crop threshold data as a baseline to obtain scores for Factual Accuracy on whether pests found in crop fields should take management action. Each model's score was weighted by percentage to obtain a final score. The results showed that GPT-3.4 and GPT-4 outperform the FLAN models in most evaluation categories. Furthermore, the use of instruction-based prompting containing domain-specific knowledge proved the feasibility of LLMs as an effective tool in agriculture, with an accuracy rate of 72%, demonstrating LLMs' effectiveness in providing pest management suggestions.
• 摘要：在快速发展的人工智能（AI）领域，大语言模型（LLM）在农业中的应用，特别是在害虫管理中，仍处于萌芽阶段。我们的目的是通过评估法学硕士生成的害虫管理建议的内容来证明可行性，包括 OpenAI 的生成式预训练变压器 (GPT) 系列和 Google 的 FLAN 系列。考虑到农业建议的特定背景属性，自动测量或量化法学硕士生成的文本质量成为一项重大挑战。我们提出了一种创新方法，使用 GPT-4 作为评估器，对生成的内容的连贯性、逻辑一致性、流畅性、相关性、可理解性和详尽性进行评分。此外，我们集成了一个基于作物阈值数据的专家系统作为基线，以获得关于农田中发现的害虫是否应该采取管理行动的事实准确性分数。每个模型的得分按百分比加权以获得最终得分。结果表明，GPT-3.4 和 GPT-4 在大多数评估类别中都优于 FLAN 模型。此外，使用包含特定领域知识的基于教学的提示证明了法学硕士作为农业有效工具的可行性，准确率达到72%，证明了法学硕士在提供害虫管理建议方面的有效性。

Title: QueryAgent: A Reliable and Efficient Reasoning Framework with Environmental Feedback based Self-Correction

• Authors: Xiang Huang, Sitao Cheng, Shanshan Huang, Jiayu Shen, Yong Xu, Chaoyun Zhang, Yuzhong Qu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.11886
• Pdf URL: https://arxiv.org/pdf/2403.11886
• Copy Paste: [[2403.11886]] QueryAgent: A Reliable and Efficient Reasoning Framework with Environmental Feedback based Self-Correction(https://arxiv.org/abs/2403.11886)
• Keywords: language model, llm, hallucination, agent
• Abstract: Employing Large Language Models (LLMs) for semantic parsing has achieved remarkable success. However, we find existing methods fall short in terms of reliability and efficiency when hallucinations are encountered. In this paper, we address these challenges with a framework called QueryAgent, which solves a question step-by-step and performs step-wise self-correction. We introduce an environmental feedback-based self-correction method called ERASER. Unlike traditional approaches, ERASER leverages rich environmental feedback in the intermediate steps to perform selective and differentiated self-correction only when necessary. Experimental results demonstrate that QueryAgent notably outperforms all previous few-shot methods using only one example on GrailQA and GraphQ by 7.0 and 15.0 F1. Moreover, our approach exhibits superiority in terms of efficiency, including runtime, query overhead, and API invocation costs. By leveraging ERASER, we further improve another baseline (i.e., AgentBench) by approximately 10 points, revealing the strong transferability of our approach.
• 摘要：采用大型语言模型（LLM）进行语义解析已经取得了显着的成功。然而，我们发现现有方法在遇到幻觉时在可靠性和效率方面存在不足。在本文中，我们使用名为 QueryAgent 的框架来应对这些挑战，该框架逐步解决问题并执行逐步自我纠正。我们引入了一种基于环境反馈的自我校正方法，称为 ERASER。与传统方法不同，ERASER 在中间步骤中利用丰富的环境反馈，仅在必要时执行选择性和差异化的自我校正。实验结果表明，仅在 GrailQA 和 GraphQ 上仅使用一个示例，QueryAgent 的 F1 性能就明显优于之前所有的少数几次方法，分别为 7.0 和 15.0。此外，我们的方法在效率方面表现出优势，包括运行时、查询开销和 API 调用成本。通过利用 ERASER，我们进一步将另一个基线（即 AgentBench）改进了大约 10 个点，揭示了我们方法的强大可移植性。

Title: From explainable to interpretable deep learning for natural language processing in healthcare: how far from reality?

• Authors: Guangming Huang, Yunfei Long, Yingya Li, Giorgos Papanastasiou
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.11894
• Pdf URL: https://arxiv.org/pdf/2403.11894
• Copy Paste: [[2403.11894]] From explainable to interpretable deep learning for natural language processing in healthcare: how far from reality?(https://arxiv.org/abs/2403.11894)
• Keywords: llm
• Abstract: Deep learning (DL) has substantially enhanced healthcare research by addressing various natural language processing (NLP) tasks. Yet, the increasing complexity of DL-based NLP methods necessitates transparent model interpretability, or at least explainability, for reliable decision-making. This work presents a thorough scoping review on explainable and interpretable DL in healthcare NLP. The term "XIAI" (eXplainable and Interpretable Artificial Intelligence) was introduced to distinguish XAI from IAI. Methods were further categorized based on their functionality (model-, input-, output-based) and scope (local, global). Our analysis shows that attention mechanisms were the most dominant emerging IAI. Moreover, IAI is increasingly used against XAI. The major challenges identified are that most XIAI do not explore "global" modeling processes, the lack of best practices, and the unmet need for systematic evaluation and benchmarks. Important opportunities were raised such as using "attention" to enhance multi-modal XIAI for personalized medicine and combine DL with causal reasoning. Our discussion encourages the integration of XIAI in LLMs and domain-specific smaller models. Our review can stimulate further research and benchmarks toward improving inherent IAI and engaging complex NLP in healthcare.
• 摘要：深度学习 (DL) 通过解决各种自然语言处理 (NLP) 任务，极大地增强了医疗保健研究。然而，基于深度学习的 NLP 方法日益复杂，需要透明的模型可解释性，或者至少是可解释性，才能做出可靠的决策。这项工作对医疗保健 NLP 中的可解释和可解释 DL 进行了全面的范围审查。引入术语“XIAI”（可解释和可解释的人工智能）来区分 XAI 和 IAI。方法根据其功能（基于模型、输入、输出）和范围（本地、全局）进一步分类。我们的分析表明，注意力机制是最主要的新兴 IAI。此外，IAI 越来越多地用于对抗 XAI。发现的主要挑战是大多数 XIAI 没有探索“全球”建模流程、缺乏最佳实践以及系统评估和基准的需求未得到满足。提出了重要的机会，例如使用“注意力”来增强个性化医疗的多模式 XIAI，并将深度学习与因果推理相结合。我们的讨论鼓励将 XIAI 集成到法学硕士和特定领域的较小模型中。我们的审查可以激发进一步的研究和基准，以改善固有的 IAI 并将复杂的 NLP 引入医疗保健领域。

Title: Investigating Markers and Drivers of Gender Bias in Machine Translations

• Authors: Peter J Barclay, Ashkan Sami (Edinburgh Napier University)
• Subjects: cs.CL, cs.CY, cs.SE
• Abstract URL: https://arxiv.org/abs/2403.11896
• Pdf URL: https://arxiv.org/pdf/2403.11896
• Copy Paste: [[2403.11896]] Investigating Markers and Drivers of Gender Bias in Machine Translations(https://arxiv.org/abs/2403.11896)
• Keywords: language model, llm
• Abstract: Implicit gender bias in Large Language Models (LLMs) is a well-documented problem, and implications of gender introduced into automatic translations can perpetuate real-world biases. However, some LLMs use heuristics or post-processing to mask such bias, making investigation difficult. Here, we examine bias in LLMss via back-translation, using the DeepL translation API to investigate the bias evinced when repeatedly translating a set of 56 Software Engineering tasks used in a previous study. Each statement starts with 'she', and is translated first into a 'genderless' intermediate language then back into English; we then examine pronoun- choice in the back-translated texts. We expand prior research in the following ways: (1) by comparing results across five intermediate languages, namely Finnish, Indonesian, Estonian, Turkish and Hungarian; (2) by proposing a novel metric for assessing the variation in gender implied in the repeated translations, avoiding the over-interpretation of individual pronouns, apparent in earlier work; (3) by investigating sentence features that drive bias; (4) and by comparing results from three time-lapsed datasets to establish the reproducibility of the approach. We found that some languages display similar patterns of pronoun use, falling into three loose groups, but that patterns vary between groups; this underlines the need to work with multiple languages. We also identify the main verb appearing in a sentence as a likely significant driver of implied gender in the translations. Moreover, we see a good level of replicability in the results, and establish that our variation metric proves robust despite an obvious change in the behaviour of the DeepL translation API during the course of the study. These results show that the back-translation method can provide further insights into bias in language models.
• 摘要：大型语言模型 (LLM) 中的隐性性别偏见是一个有据可查的问题，自动翻译中引入的性别影响可能会延续现实世界的偏见。然而，一些法学硕士使用启发法或后处理来掩盖这种偏见，使调查变得困难。在这里，我们通过反向翻译检查 LLMs 中的偏差，使用 DeepL 翻译 API 来调查在重复翻译先前研究中使用的一组 56 个软件工程任务时所表现出的偏差。每个语句都以“她”开头，首先被翻译成“无性别”中间语言，然后再翻译回英语；然后我们检查回译文本中的代词选择。我们通过以下方式扩展了先前的研究：（1）通过比较五种中间语言（即芬兰语、印度尼西亚语、爱沙尼亚语、土耳其语和匈牙利语）的结果； （2）提出一种新的衡量标准来评估重复翻译中隐含的性别差异，避免早期工作中明显的对单个代词的过度解释； (3) 通过调查导致偏见的句子特征； (4) 并通过比较三个延时数据集的结果来确定该方法的再现性。我们发现，一些语言显示出相似的代词使用模式，分为三个松散的组，但不同组之间的模式有所不同；这强调了使用多种语言的必要性。我们还发现句子中出现的主要动词可能是翻译中隐含性别的重要驱动因素。此外，我们在结果中看到了良好的可复制性，并确定尽管在研究过程中 DeepL 翻译 API 的行为发生了明显变化，但我们的变异指标仍然是稳健的。这些结果表明，反向翻译方法可以进一步洞察语言模型中的偏差。

Title: A Closer Look at Claim Decomposition

• Authors: Miriam Wanner, Seth Ebner, Zhengping Jiang, Mark Dredze, Benjamin Van Durme
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.11903
• Pdf URL: https://arxiv.org/pdf/2403.11903
• Copy Paste: [[2403.11903]] A Closer Look at Claim Decomposition(https://arxiv.org/abs/2403.11903)
• Keywords: llm
• Abstract: As generated text becomes more commonplace, it is increasingly important to evaluate how well-supported such text is by external knowledge sources. Many approaches for evaluating textual support rely on some method for decomposing text into its individual subclaims which are scored against a trusted reference. We investigate how various methods of claim decomposition -- especially LLM-based methods -- affect the result of an evaluation approach such as the recently proposed FActScore, finding that it is sensitive to the decomposition method used. This sensitivity arises because such metrics attribute overall textual support to the model that generated the text even though error can also come from the metric's decomposition step. To measure decomposition quality, we introduce an adaptation of FActScore, which we call DecompScore. We then propose an LLM-based approach to generating decompositions inspired by Bertrand Russell's theory of logical atomism and neo-Davidsonian semantics and demonstrate its improved decomposition quality over previous methods.
• 摘要：随着生成的文本变得越来越普遍，评估外部知识源对此类文本的支持程度变得越来越重要。许多评估文本支持的方法依赖于某种将文本分解为单独的子声明的方法，这些子声明是根据可信参考进行评分的。我们研究了各种权利要求分解方法（尤其是基于 LLM 的方法）如何影响评估方法（例如最近提出的 FActScore）的结果，发现它对所使用的分解方法很敏感。之所以出现这种敏感性，是因为此类度量将整体文本支持归因于生成文本的模型，即使错误也可能来自度量的分解步骤。为了衡量分解质量，我们引入了 FActScore 的改编版本，我们称之为 DecompScore。然后，我们提出了一种基于法学硕士的方法来生成分解，其灵感来自伯特兰·罗素的逻辑原子论和新戴维森语义学理论，并证明了其比以前的方法改进的分解质量。

Title: CICLe: Conformal In-Context Learning for Largescale Multi-Class Food Risk Classification

• Authors: Korbinian Randl, John Pavlopoulos, Aron Henriksson, Tony Lindgren
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.11904
• Pdf URL: https://arxiv.org/pdf/2403.11904
• Copy Paste: [[2403.11904]] CICLe: Conformal In-Context Learning for Largescale Multi-Class Food Risk Classification(https://arxiv.org/abs/2403.11904)
• Keywords: llm, prompt
• Abstract: Contaminated or adulterated food poses a substantial risk to human health. Given sets of labeled web texts for training, Machine Learning and Natural Language Processing can be applied to automatically detect such risks. We publish a dataset of 7,546 short texts describing public food recall announcements. Each text is manually labeled, on two granularity levels (coarse and fine), for food products and hazards that the recall corresponds to. We describe the dataset and benchmark naive, traditional, and Transformer models. Based on our analysis, Logistic Regression based on a tf-idf representation outperforms RoBERTa and XLM-R on classes with low support. Finally, we discuss different prompting strategies and present an LLM-in-the-loop framework, based on Conformal Prediction, which boosts the performance of the base classifier while reducing energy consumption compared to normal prompting.
• 摘要：受污染或掺假的食品对人类健康构成重大风险。给定用于训练的标记网络文本集，可以应用机器学习和自然语言处理来自动检测此类风险。我们发布了包含 7,546 条描述公共食品召回公告的短文本的数据集。每个文本都针对召回对应的食品和危害，在两个粒度级别（粗略和精细）上进行手动标记。我们描述了数据集和基准模型、传统模型和 Transformer 模型。根据我们的分析，基于 tf-idf 表示的 Logistic 回归在支持度较低的类上优于 RoBERTa 和 XLM-R。最后，我们讨论了不同的提示策略，并提出了一个基于保形预测的 LLM-in-the-loop 框架，与正常提示相比，它提高了基分类器的性能，同时减少了能耗。

Title: Language Evolution with Deep Learning

• Authors: Mathieu Rita, Paul Michel, Rahma Chaabouni, Olivier Pietquin, Emmanuel Dupoux, Florian Strub
• Subjects: cs.CL, cs.MA
• Abstract URL: https://arxiv.org/abs/2403.11958
• Pdf URL: https://arxiv.org/pdf/2403.11958
• Copy Paste: [[2403.11958]] Language Evolution with Deep Learning(https://arxiv.org/abs/2403.11958)
• Keywords: agent
• Abstract: Computational modeling plays an essential role in the study of language emergence. It aims to simulate the conditions and learning processes that could trigger the emergence of a structured language within a simulated controlled environment. Several methods have been used to investigate the origin of our language, including agent-based systems, Bayesian agents, genetic algorithms, and rule-based systems. This chapter explores another class of computational models that have recently revolutionized the field of machine learning: deep learning models. The chapter introduces the basic concepts of deep and reinforcement learning methods and summarizes their helpfulness for simulating language emergence. It also discusses the key findings, limitations, and recent attempts to build realistic simulations. This chapter targets linguists and cognitive scientists seeking an introduction to deep learning as a tool to investigate language evolution.
• 摘要：计算模型在语言出现的研究中起着至关重要的作用。它的目的是模拟可能在模拟受控环境中触发结构化语言出现的条件和学习过程。已经使用了多种方法来研究我们语言的起源，包括基于代理的系统、贝叶斯代理、遗传算法和基于规则的系统。本章探讨了最近彻底改变机器学习领域的另一类计算模型：深度学习模型。本章介绍了深度学习和强化学习方法的基本概念，并总结了它们对模拟语言涌现的帮助。它还讨论了主要发现、局限性以及最近构建真实模拟的尝试。本章的目标读者是寻求介绍深度学习作为研究语言进化工具的语言学家和认知科学家。

Title: Using Generative Text Models to Create Qualitative Codebooks for Student Evaluations of Teaching

• Authors: Andrew Katz, Mitchell Gerhardt, Michelle Soledad
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2403.11984
• Pdf URL: https://arxiv.org/pdf/2403.11984
• Copy Paste: [[2403.11984]] Using Generative Text Models to Create Qualitative Codebooks for Student Evaluations of Teaching(https://arxiv.org/abs/2403.11984)
• Keywords: language model, llm
• Abstract: Feedback is a critical aspect of improvement. Unfortunately, when there is a lot of feedback from multiple sources, it can be difficult to distill the information into actionable insights. Consider student evaluations of teaching (SETs), which are important sources of feedback for educators. They can give instructors insights into what worked during a semester. A collection of SETs can also be useful to administrators as signals for courses or entire programs. However, on a large scale as in high-enrollment courses or administrative records over several years, the volume of SETs can render them difficult to analyze. In this paper, we discuss a novel method for analyzing SETs using natural language processing (NLP) and large language models (LLMs). We demonstrate the method by applying it to a corpus of 5,000 SETs from a large public university. We show that the method can be used to extract, embed, cluster, and summarize the SETs to identify the themes they express. More generally, this work illustrates how to use the combination of NLP techniques and LLMs to generate a codebook for SETs. We conclude by discussing the implications of this method for analyzing SETs and other types of student writing in teaching and research settings.
• 摘要：反馈是改进的一个关键方面。不幸的是，当有来自多个来源的大量反馈时，可能很难将信息提炼成可操作的见解。考虑学生的教学评价 (SET)，这是教育工作者反馈的重要来源。他们可以让教师深入了解一个学期的工作情况。 SET 的集合对于管理员来说也很有用，可以作为课程或整个项目的信号。然而，在大规模的情况下，如高入学率课程或多年来的管理记录，SET 的数量可能会使它们难以分析。在本文中，我们讨论了一种使用自然语言处理（NLP）和大语言模型（LLM）来分析 SET 的新方法。我们通过将其应用于一所大型公立大学的 5,000 个 SET 语料库来演示该方法。我们证明该方法可用于提取、嵌入、聚类和总结 SET，以识别它们表达的主题。更一般地说，这项工作说明了如何结合使用 NLP 技术和 LLM 来生成 SET 的码本。最后，我们讨论了这种方法对分析 SET 和教学和研究环境中其他类型的学生写作的影响。

Title: EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents

• Authors: Abhay Zala, Jaemin Cho, Han Lin, Jaehong Yoon, Mohit Bansal
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12014
• Pdf URL: https://arxiv.org/pdf/2403.12014
• Copy Paste: [[2403.12014]] EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents(https://arxiv.org/abs/2403.12014)
• Keywords: language model, gpt, llm, prompt, agent
• Abstract: Recent SOTA approaches for embodied learning via interaction directly employ large language models (LLMs) as agents to determine the next steps in an environment. Due to their world knowledge and reasoning capabilities, LLM agents achieve stronger performance than previous smaller agents based on reinforcement learning (RL); however, frequently calling LLMs is slow and expensive. Instead of directly employing LLMs as agents, can we use LLMs' reasoning capabilities to adaptively create training environments to help smaller embodied RL agents learn useful skills that they are weak at? We propose EnvGen, a novel framework to address this question. First, we prompt an LLM to generate training environments that allow agents to quickly learn different tasks in parallel. Concretely, the LLM is given the task description and simulator objectives that the agents should learn and is then asked to generate a set of environment configurations (e.g., different terrains, items given to agents, etc.). Next, we train a small RL agent in a mixture of the original and LLM-generated environments. Then, we enable the LLM to continuously adapt the generated environments to progressively improve the skills that the agent is weak at, by providing feedback to the LLM in the form of the agent's performance. We demonstrate the usefulness of EnvGen with comprehensive experiments in Crafter and Heist environments. We find that a small RL agent trained with EnvGen can outperform SOTA methods, including a GPT-4 agent, and learns long-horizon tasks significantly faster. We show qualitatively how the LLM adapts training environments to help improve RL agents' weaker skills over time. Additionally, EnvGen is substantially more efficient as it only uses a small number of LLM calls (e.g., 4 in total), whereas LLM agents require thousands of LLM calls. Lastly, we present detailed ablation studies for our design choices.
• 摘要：最近通过交互进行体现学习的 SOTA 方法直接采用大型语言模型 (LLM) 作为代理来确定环境中的下一步。由于其世界知识和推理能力，LLM 智能体比之前基于强化学习（RL）的小型智能体取得了更强的性能；然而，频繁致电法学硕士既缓慢又昂贵。我们是否可以使用 LLM 的推理能力来自适应地创建训练环境，以帮助较小的具体 RL 代理学习他们不擅长的有用技能，而不是直接使用 LLM 作为代理？我们提出了 EnvGen，一个解决这个问题的新颖框架。首先，我们提示法学硕士生成训练环境，使代理能够快速并行地学习不同的任务。具体来说，LLM 被给予智能体应该学习的任务描述和模拟器目标，然后被要求生成一组环境配置（例如，不同的地形、给予智能体的物品等）。接下来，我们在原始环境和 LLM 生成的环境中训练一个小型 RL 代理。然后，我们通过以代理表现的形式向 LLM 提供反馈，使 LLM 能够不断适应生成的环境，以逐步提高代理薄弱的技能。我们通过 Crafter 和 Heist 环境中的综合实验展示了 EnvGen 的实用性。我们发现，使用 EnvGen 训练的小型 RL 代理可以优于 SOTA 方法（包括 GPT-4 代理），并且学习长视野任务的速度显着更快。我们定性地展示了 LLM 如何适应培训环境，以帮助提高 RL 智能体的较弱技能。此外，EnvGen 的效率要高得多，因为它只使用少量的 LLM 调用（例如总共 4 个），而 LLM 代理需要数千个 LLM 调用。最后，我们为我们的设计选择提供了详细的消融研究。

Title: Enhancing Hokkien Dual Translation by Exploring and Standardizing of Four Writing Systems

• Authors: Bo-Han Lu, Yi-Hsuan Lin, En-Shiun Annie Lee, Richard Tzong-Han Tsai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12024
• Pdf URL: https://arxiv.org/pdf/2403.12024
• Copy Paste: [[2403.12024]] Enhancing Hokkien Dual Translation by Exploring and Standardizing of Four Writing Systems(https://arxiv.org/abs/2403.12024)
• Keywords: gpt
• Abstract: Machine translation focuses mainly on high-resource languages (HRLs), while low-resource languages (LRLs) like Taiwanese Hokkien are relatively under-explored. This study aims to address this gap by developing a dual translation model between Taiwanese Hokkien and both Traditional Mandarin Chinese and English. We employ a pre-trained LLaMA2-7B model specialized in Traditional Mandarin Chinese to leverage the orthographic similarities between Taiwanese Hokkien Han and Traditional Mandarin Chinese. Our comprehensive experiments involve translation tasks across various writing systems of Taiwanese Hokkien and between Taiwanese Hokkien and other HRLs. We find that the use of a limited monolingual corpus also further improve the model's Taiwanese Hokkien capabilities. We then utilize our translation model to standardize all Taiwanese Hokkien writing systems into Hokkien Han, resulting in further performance improvements. Additionally, we introduce an evaluation method incorporating back-translation and GPT-4 to ensure reliable translation quality assessment even for LRLs. The study contributes to narrowing the resource gap for Taiwanese Hokkien and empirically investigates the advantages and limitations of pre-training and fine-tuning based on LLaMA 2.
• 摘要：机器翻译主要关注高资源语言（HRL），而像闽南语这样的低资源语言（LRL）则相对较少被开发。本研究旨在通过开发台湾闽南语与繁体中文和英语之间的双重翻译模型来解决这一差距。我们采用专门针对繁体中文的预训练 LLaMA2-7B 模型，以利用台湾闽南语和繁体中文之间的拼写相似性。我们的综合实验涉及跨台湾闽南语各种书写系统以及台湾闽南语和其他 HRL 之间的翻译任务。我们发现，使用有限的单语语料库也进一步提高了模型的台语闽南语能力。然后，我们利用我们的翻译模型将所有台湾闽南语书写系统标准化为闽南语汉字，从而进一步提高性能。此外，我们引入了一种结合反向翻译和 GPT-4 的评估方法，以确保即使对于 LRL 也能进行可靠的翻译质量评估。该研究有助于缩小台湾闽南语的资源差距，并实证研究了基于 LLaMA 2 的预训练和微调的优势和局限性。

Title: From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models

• Authors: Kung-Hsiang Huang, Hou Pong Chan, Yi R. Fung, Haoyi Qiu, Mingyang Zhou, Shafiq Joty, Shih-Fu Chang, Heng Ji
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12027
• Pdf URL: https://arxiv.org/pdf/2403.12027
• Copy Paste: [[2403.12027]] From Pixels to Insights: A Survey on Automatic Chart Understanding in the Era of Large Foundation Models(https://arxiv.org/abs/2403.12027)
• Keywords: language model, llm, agent
• Abstract: Data visualization in the form of charts plays a pivotal role in data analysis, offering critical insights and aiding in informed decision-making. Automatic chart understanding has witnessed significant advancements with the rise of large foundation models in recent years. Foundation models, such as large language models (LLMs), have revolutionized various natural language processing (NLP) tasks and are increasingly being applied to chart understanding tasks. This survey paper provides a comprehensive overview of the recent developments, challenges, and future directions in chart understanding within the context of these foundation models. The paper begins by defining chart understanding, outlining problem formulations, and discussing fundamental building blocks crucial for studying chart understanding tasks. In the section on tasks and datasets, we explore various tasks within chart understanding and discuss their evaluation metrics and sources of both charts and textual inputs. Modeling strategies are then examined, encompassing both classification-based and generation-based approaches, along with tool augmentation techniques that enhance chart understanding performance. Furthermore, we discuss the state-of-the-art performance of each task and discuss how we can improve the performance. Challenges and future directions are addressed in a dedicated section, highlighting issues such as domain-specific charts, lack of efforts in evaluation, and agent-oriented settings. This survey paper serves to provide valuable insights and directions for future research in chart understanding leveraging large foundation models. The studies mentioned in this paper, along with emerging new research, will be continually updated at: https://github.com/khuangaf/Awesome-Chart-Understanding.
• 摘要：图表形式的数据可视化在数据分析中发挥着关键作用，提供重要的见解并帮助做出明智的决策。近年来，随着大型基础模型的兴起，自动图表理解取得了显着进步。基础模型，例如大语言模型（LLM），已经彻底改变了各种自然语言处理（NLP）任务，并且越来越多地应用于图表理解任务。本调查论文全面概述了这些基础模型背景下图表理解的最新发展、挑战和未来方向。本文首先定义图表理解，概述问题表述，并讨论对于研究图表理解任务至关重要的基本构建块。在任务和数据集部分中，我们探讨了图表理解中的各种任务，并讨论了它们的评估指标以及图表和文本输入的来源。然后检查建模策略，包括基于分类和基于生成的方法，以及增强图表理解性能的工具增强技术。此外，我们还讨论了每项任务的最先进性能，并讨论了如何提高性能。挑战和未来的方向在专门的部分中讨论，突出显示特定领域的图表、缺乏评估工作以及面向代理的设置等问题。这篇调查论文旨在为利用大型基础模型进行图表理解的未来研究提供有价值的见解和方向。本文提到的研究以及不断出现的新研究将在以下位置不断更新：https://github.com/kuanaf/Awesome-Chart-Understanding。