2024-04-09

Title: A Novel BERT-based Classifier to Detect Political Leaning of YouTube Videos based on their Titles

• Authors: Nouar AlDahoul, Talal Rahwan, Yasir Zaki
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04261
• Pdf URL: https://arxiv.org/pdf/2404.04261
• Copy Paste: [[2404.04261]] A Novel BERT-based Classifier to Detect Political Leaning of YouTube Videos based on their Titles(https://arxiv.org/abs/2404.04261)
• Keywords: language model
• Abstract: A quarter of US adults regularly get their news from YouTube. Yet, despite the massive political content available on the platform, to date no classifier has been proposed to identify the political leaning of YouTube videos. To fill this gap, we propose a novel classifier based on Bert -- a language model from Google -- to classify YouTube videos merely based on their titles into six categories, namely: Far Left, Left, Center, Anti-Woke, Right, and Far Right. We used a public dataset of 10 million YouTube video titles (under various categories) to train and validate the proposed classifier. We compare the classifier against several alternatives that we trained on the same dataset, revealing that our classifier achieves the highest accuracy (75%) and the highest F1 score (77%). To further validate the classification performance, we collect videos from YouTube channels of numerous prominent news agencies, such as Fox News and New York Times, which have widely known political leanings, and apply our classifier to their video titles. For the vast majority of cases, the predicted political leaning matches that of the news agency.
• 摘要：四分之一的美国成年人定期从 YouTube 获取新闻。然而，尽管该平台上提供了大量政治内容，但迄今为止，尚未提出任何分类器来识别 YouTube 视频的政治倾向。为了填补这一空白，我们提出了一种基于 Bert（谷歌的语言模型）的新型分类器，仅根据标题将 YouTube 视频分为六类，即：远左、左、中、反唤醒、右、和极右翼。我们使用包含 1000 万个 YouTube 视频标题（不同类别）的公共数据集来训练和验证所提出的分类器。我们将分类器与在同一数据集上训练的几种替代方案进行比较，结果表明我们的分类器实现了最高的准确率 (75%) 和最高的 F1 分数 (77%)。为了进一步验证分类性能，我们从众多著名新闻机构（例如福克斯新闻和纽约时报）的 YouTube 频道收集视频，这些新闻机构的政治倾向众所周知，并将我们的分类器应用于他们的视频标题。在绝大多数情况下，预测的政治倾向与新闻机构的政治倾向相符。

Title: Similar Data Points Identification with LLM: A Human-in-the-loop Strategy Using Summarization and Hidden State Insights

• Authors: Xianlong Zeng, Fanghao Song, Ang Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04281
• Pdf URL: https://arxiv.org/pdf/2404.04281
• Copy Paste: [[2404.04281]] Similar Data Points Identification with LLM: A Human-in-the-loop Strategy Using Summarization and Hidden State Insights(https://arxiv.org/abs/2404.04281)
• Keywords: language model, llm
• Abstract: This study introduces a simple yet effective method for identifying similar data points across non-free text domains, such as tabular and image data, using Large Language Models (LLMs). Our two-step approach involves data point summarization and hidden state extraction. Initially, data is condensed via summarization using an LLM, reducing complexity and highlighting essential information in sentences. Subsequently, the summarization sentences are fed through another LLM to extract hidden states, serving as compact, feature-rich representations. This approach leverages the advanced comprehension and generative capabilities of LLMs, offering a scalable and efficient strategy for similarity identification across diverse datasets. We demonstrate the effectiveness of our method in identifying similar data points on multiple datasets. Additionally, our approach enables non-technical domain experts, such as fraud investigators or marketing operators, to quickly identify similar data points tailored to specific scenarios, demonstrating its utility in practical applications. In general, our results open new avenues for leveraging LLMs in data analysis across various domains.
• 摘要：本研究介绍了一种简单而有效的方法，用于使用大型语言模型 (LLM) 识别非自由文本域（例如表格和图像数据）中的相似数据点。我们的两步方法涉及数据点汇总和隐藏状态提取。最初，使用法学硕士通过摘要来压缩数据，降低复杂性并突出句子中的基本信息。随后，摘要句子通过另一个 LLM 来提取隐藏状态，作为紧凑、特征丰富的表示。这种方法利用了法学硕士的先进理解和生成能力，为跨不同数据集的相似性识别提供了可扩展且高效的策略。我们证明了我们的方法在识别多个数据集上的相似数据点方面的有效性。此外，我们的方法使非技术领域专家（例如欺诈调查人员或营销运营商）能够快速识别针对特定场景定制的类似数据点，展示其在实际应用中的实用性。总的来说，我们的结果为在各个领域利用法学硕士进行数据分析开辟了新途径。

Title: MIMIR: A Streamlined Platform for Personalized Agent Tuning in Domain Expertise

• Authors: Chunyuan Deng, Xiangru Tang, Yilun Zhao, Hanming Wang, Haoran Wang, Wangchunshu Zhou, Arman Cohan, Mark Gerstein
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04285
• Pdf URL: https://arxiv.org/pdf/2404.04285
• Copy Paste: [[2404.04285]] MIMIR: A Streamlined Platform for Personalized Agent Tuning in Domain Expertise(https://arxiv.org/abs/2404.04285)
• Keywords: language model, gpt, llm, agent
• Abstract: Recently, large language models (LLMs) have evolved into interactive agents, proficient in planning, tool use, and task execution across a wide variety of tasks. However, without specific agent tuning, open-source models like LLaMA currently struggle to match the efficiency of GPT- 4, particularly given the scarcity of agent-tuning datasets for fine-tuning. In response, we introduce \textsc{Mimir}: a streamlined platform offering a customizable pipeline that enables users to leverage both private knowledge and publicly available, legally compliant datasets at scale for \textbf{personalized agent tuning}. Additionally, \textsc{Mimir} supports the generation of general instruction-tuning datasets from the same input. This dual capability ensures that language agents developed through the platform possess both specific agent abilities and general competencies. \textsc{Mimir} integrates these features into a cohesive end-to-end platform, facilitating everything from the uploading of personalized files to one-click agent fine-tuning.
• 摘要：最近，大型语言模型 (LLM) 已发展成为交互式代理，精通各种任务的规划、工具使用和任务执行。然而，如果没有特定的代理调整，像 LLaMA 这样的开源模型目前很难与 GPT-4 的效率相匹配，特别是考虑到用于微调的代理调整数据集的稀缺。作为回应，我们推出了 \textsc{Mimir}：一个提供可定制管道的简化平台，使用户能够大规模利用私有知识和公开可用的、合法合规的数据集来进行 \textbf{个性化代理调整}。此外，\textsc{Mimir} 支持从同一输入生成通用指令调整数据集。这种双重能力确保了通过该平台开发的语言智能体同时拥有特定智能体能力和通用能力。 \textsc{Mimir} 将这些功能集成到一个有凝聚力的端到端平台中，方便从上传个性化文件到一键代理微调的一切。

Title: Language Model Evolution: An Iterated Learning Perspective

• Authors: Yi Ren, Shangmin Guo, Linlu Qiu, Bailin Wang, Danica J. Sutherland
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04286
• Pdf URL: https://arxiv.org/pdf/2404.04286
• Copy Paste: [[2404.04286]] Language Model Evolution: An Iterated Learning Perspective(https://arxiv.org/abs/2404.04286)
• Keywords: language model, llm, agent
• Abstract: With the widespread adoption of Large Language Models (LLMs), the prevalence of iterative interactions among these models is anticipated to increase. Notably, recent advancements in multi-round self-improving methods allow LLMs to generate new examples for training subsequent models. At the same time, multi-agent LLM systems, involving automated interactions among agents, are also increasing in prominence. Thus, in both short and long terms, LLMs may actively engage in an evolutionary process. We draw parallels between the behavior of LLMs and the evolution of human culture, as the latter has been extensively studied by cognitive scientists for decades. Our approach involves leveraging Iterated Learning (IL), a Bayesian framework that elucidates how subtle biases are magnified during human cultural evolution, to explain some behaviors of LLMs. This paper outlines key characteristics of agents' behavior in the Bayesian-IL framework, including predictions that are supported by experimental verification with various LLMs. This theoretical framework could help to more effectively predict and guide the evolution of LLMs in desired directions.
• 摘要：随着大型语言模型（LLM）的广泛采用，这些模型之间迭代交互的普遍性预计将会增加。值得注意的是，多轮自我改进方法的最新进展使法学硕士能够生成用于训练后续模型的新示例。与此同时，涉及代理之间自动交互的多代理法学硕士系统也越来越受到重视。因此，无论是短期还是长期，法学硕士都可以积极参与一个进化过程。我们将法学硕士的行为与人类文化的进化进行了比较，因为认知科学家几十年来已经对后者进行了广泛的研究。我们的方法涉及利用迭代学习（IL），这是一种贝叶斯框架，它阐明了人类文化进化过程中微妙的偏见是如何被放大的，来解释法学硕士的一些行为。本文概述了贝叶斯 IL 框架中代理行为的关键特征，包括由各种法学硕士实验验证支持的预测。这一理论框架可以帮助更有效地预测和指导法学硕士朝着期望的方向发展。

Title: CONFLARE: CONFormal LArge language model REtrieval

• Authors: Pouria Rouzrokh, Shahriar Faghani, Cooper U. Gamble, Moein Shariatnia, Bradley J. Erickson
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04287
• Pdf URL: https://arxiv.org/pdf/2404.04287
• Copy Paste: [[2404.04287]] CONFLARE: CONFormal LArge language model REtrieval(https://arxiv.org/abs/2404.04287)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: Retrieval-augmented generation (RAG) frameworks enable large language models (LLMs) to retrieve relevant information from a knowledge base and incorporate it into the context for generating responses. This mitigates hallucinations and allows for the updating of knowledge without retraining the LLM. However, RAG does not guarantee valid responses if retrieval fails to identify the necessary information as the context for response generation. Also, if there is contradictory content, the RAG response will likely reflect only one of the two possible responses. Therefore, quantifying uncertainty in the retrieval process is crucial for ensuring RAG trustworthiness. In this report, we introduce a four-step framework for applying conformal prediction to quantify retrieval uncertainty in RAG frameworks. First, a calibration set of questions answerable from the knowledge base is constructed. Each question's embedding is compared against document embeddings to identify the most relevant document chunks containing the answer and record their similarity scores. Given a user-specified error rate ({\alpha}), these similarity scores are then analyzed to determine a similarity score cutoff threshold. During inference, all chunks with similarity exceeding this threshold are retrieved to provide context to the LLM, ensuring the true answer is captured in the context with a (1-{\alpha}) confidence level. We provide a Python package that enables users to implement the entire workflow proposed in our work, only using LLMs and without human intervention.
• 摘要：检索增强生成 (RAG) 框架使大型语言模型 (LLM) 能够从知识库中检索相关信息，并将其合并到上下文中以生成响应。这可以减轻幻觉并允许更新知识而无需重新培训法学硕士。但是，如果检索无法将必要的信息识别为响应生成的上下文，RAG 不保证有效的响应。此外，如果存在矛盾的内容，RAG 响应可能仅反映两种可能响应之一。因此，量化检索过程中的不确定性对于确保 RAG 的可信度至关重要。在本报告中，我们介绍了一个四步框架，用于应用保形预测来量化 RAG 框架中的检索不确定性。首先，构建可从知识库回答的校准问题集。将每个问题的嵌入与文档嵌入进行比较，以识别包含答案的最相关的文档块并记录它们的相似性分数。给定用户指定的错误率（{\alpha}），然后分析这些相似性分数以确定相似性分数截止阈值。在推理过程中，检索所有相似度超过此阈值的块，为 LLM 提供上下文，确保在具有 (1-{\alpha}) 置信水平的上下文中捕获真实答案。我们提供了一个Python包，使用户能够实现我们工作中提出的整个工作流程，仅使用LLM并且无需人工干预。

Title: Conversational Disease Diagnosis via External Planner-Controlled Large Language Models

• Authors: Zhoujian Sun, Cheng Luo, Zhengxing Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04292
• Pdf URL: https://arxiv.org/pdf/2404.04292
• Copy Paste: [[2404.04292]] Conversational Disease Diagnosis via External Planner-Controlled Large Language Models(https://arxiv.org/abs/2404.04292)
• Keywords: language model, llm
• Abstract: The advancement of medical artificial intelligence (AI) has set the stage for the realization of conversational diagnosis, where AI systems mimic human doctors by engaging in dialogue with patients to deduce diagnoses. This study introduces an innovative approach using external planners augmented with large language models (LLMs) to develop a medical task-oriented dialogue system. This system comprises a policy module for information gathering, a LLM based module for natural language understanding and generation, addressing the limitations of previous AI systems in these areas. By emulating the two-phase decision-making process of doctors disease screening and differential diagnosis. we designed two distinct planners. The first focuses on collecting patient symptoms to identify potential diseases, while the second delves into specific inquiries to confirm or exclude these diseases. Utilizing reinforcement learning and active learning with LLMs, we trained these planners to navigate medical dialogues effectively. Our evaluation on the MIMIC-IV dataset demonstrated the system's capability to outperform existing models, indicating a significant step towards achieving automated conversational disease diagnostics and enhancing the precision and accessibility of medical diagnoses.
• 摘要：医疗人工智能（AI）的进步为对话式诊断的实现奠定了基础，人工智能系统模仿人类医生，通过与患者对话来推断诊断。本研究引入了一种创新方法，使用外部规划器和大型语言模型（LLM）增强来开发面向医疗任务的对话系统。该系统包括用于信息收集的政策模块、用于自然语言理解和生成的基于法学硕士的模块，解决了先前人工智能系统在这些领域的局限性。通过模拟医生疾病筛查和鉴别诊断的两阶段决策过程。我们设计了两个不同的规划器。第一个重点是收集患者症状以识别潜在的疾病，而第二个则深入研究具体询问以确认或排除这些疾病。利用法学硕士的强化学习和主动学习，我们培训这些规划人员有效地进行医学对话。我们对 MIMIC-IV 数据集的评估证明了该系统超越现有模型的能力，这表明朝着实现自动化对话疾病诊断和提高医疗诊断的准确性和可访问性迈出了重要一步。

Title: Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understanding

• Authors: Yanda Li, Dixuan Wang, Jiaqing Liang, Guochao Jiang, Qianyu He, Yanghua Xiao, Deqing Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04293
• Pdf URL: https://arxiv.org/pdf/2404.04293
• Copy Paste: [[2404.04293]] Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understanding(https://arxiv.org/abs/2404.04293)
• Keywords: language model, gpt, llm
• Abstract: Large Language Models (LLMs) have demonstrated good performance in many reasoning tasks, but they still struggle with some complicated reasoning tasks including logical reasoning. One non-negligible reason for LLMs' suboptimal performance on logical reasoning is their overlooking of understanding logical fallacies correctly. To evaluate LLMs' capability of logical fallacy understanding (LFU), we propose five concrete tasks from three cognitive dimensions of WHAT, WHY, and HOW in this paper. Towards these LFU tasks, we have successfully constructed a new dataset LFUD based on GPT-4 accompanied by a little human effort. Our extensive experiments justify that our LFUD can be used not only to evaluate LLMs' LFU capability, but also to fine-tune LLMs to obtain significantly enhanced performance on logical reasoning.
• 摘要：大型语言模型（LLM）在许多推理任务中表现出了良好的性能，但它们仍然难以处理包括逻辑推理在内的一些复杂的推理任务。法学硕士在逻辑推理方面表现欠佳的一个不可忽视的原因是他们忽视了正确理解逻辑谬误。为了评估法学硕士的逻辑谬误理解（LFU）能力，我们在本文中从“什么”、“为什么”和“如何”三个认知维度提出了五个具体任务。针对这些 LFU 任务，我们在 GPT-4 的基础上成功构建了一个新的数据集 LFUD，并付出了一点人力。我们大量的实验证明，我们的 LFUD 不仅可以用来评估 LLM 的 LFU 能力，还可以用来微调 LLM，以获得显着增强的逻辑推理性能。

Title: CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answering

• Authors: Nirmalie Wiratunga, Ramitha Abeyratne, Lasal Jayawardena, Kyle Martin, Stewart Massie, Ikechukwu Nkisi-Orji, Ruvan Weerasinghe, Anne Liret, Bruno Fleisch
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04302
• Pdf URL: https://arxiv.org/pdf/2404.04302
• Copy Paste: [[2404.04302]] CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answering(https://arxiv.org/abs/2404.04302)
• Keywords: language model, llm, prompt, retrieval augmented generation, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) enhances Large Language Model (LLM) output by providing prior knowledge as context to input. This is beneficial for knowledge-intensive and expert reliant tasks, including legal question-answering, which require evidence to validate generated text outputs. We highlight that Case-Based Reasoning (CBR) presents key opportunities to structure retrieval as part of the RAG process in an LLM. We introduce CBR-RAG, where CBR cycle's initial retrieval stage, its indexing vocabulary, and similarity knowledge containers are used to enhance LLM queries with contextually relevant cases. This integration augments the original LLM query, providing a richer prompt. We present an evaluation of CBR-RAG, and examine different representations (i.e. general and domain-specific embeddings) and methods of comparison (i.e. inter, intra and hybrid similarity) on the task of legal question-answering. Our results indicate that the context provided by CBR's case reuse enforces similarity between relevant components of the questions and the evidence base leading to significant improvements in the quality of generated answers.
• 摘要：检索增强生成 (RAG) 通过提供先验知识作为输入上下文来增强大型语言模型 (LLM) 输出。这对于知识密集型和依赖专家的任务是有益的，包括法律问答，这需要证据来验证生成的文本输出。我们强调，基于案例的推理 (CBR) 为法学硕士 RAG 流程的一部分提供了结构化检索的关键机会。我们引入了 CBR-RAG，其中 CBR 循环的初始检索阶段、其索引词汇和相似性知识容器用于增强具有上下文相关案例的 LLM 查询。此集成增强了原始 LLM 查询，提供更丰富的提示。我们提出了 CBR-RAG 的评估，并检查了法律问答任务的不同表示（即一般和特定领域的嵌入）和比较方法（即内部、内部和混合相似性）。我们的结果表明，CBR 案例重用提供的上下文增强了问题的相关组成部分与证据库之间的相似性，从而显着提高了生成答案的质量。

Title: Scope Ambiguities in Large Language Models

• Authors: Gaurav Kamath, Sebastian Schuster, Sowmya Vajjala, Siva Reddy
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04332
• Pdf URL: https://arxiv.org/pdf/2404.04332
• Copy Paste: [[2404.04332]] Scope Ambiguities in Large Language Models(https://arxiv.org/abs/2404.04332)
• Keywords: language model, gpt
• Abstract: Sentences containing multiple semantic operators with overlapping scope often create ambiguities in interpretation, known as scope ambiguities. These ambiguities offer rich insights into the interaction between semantic structure and world knowledge in language processing. Despite this, there has been little research into how modern large language models treat them. In this paper, we investigate how different versions of certain autoregressive language models -- GPT-2, GPT-3/3.5, Llama 2 and GPT-4 -- treat scope ambiguous sentences, and compare this with human judgments. We introduce novel datasets that contain a joint total of almost 1,000 unique scope-ambiguous sentences, containing interactions between a range of semantic operators, and annotated for human judgments. Using these datasets, we find evidence that several models (i) are sensitive to the meaning ambiguity in these sentences, in a way that patterns well with human judgments, and (ii) can successfully identify human-preferred readings at a high level of accuracy (over 90% in some cases).
• 摘要：包含具有重叠范围的多个语义运算符的句子通常会在解释中产生歧义，称为范围歧义。这些歧义为语言处理中语义结构和世界知识之间的相互作用提供了丰富的见解。尽管如此，关于现代大型语言模型如何处理它们的研究却很少。在本文中，我们研究了某些自回归语言模型的不同版本（GPT-2、GPT-3/3.5、Llama 2 和 GPT-4）如何处理范围歧义句子，并将其与人类判断进行比较。我们引入了新颖的数据集，其中总共包含近 1,000 个独特的范围模糊句子，包含一系列语义运算符之间的交互，并为人类判断进行了注释。使用这些数据集，我们发现有证据表明多个模型（i）对这些句子中的含义模糊性很敏感，其方式可以很好地与人类判断相匹配，并且（ii）可以成功地以高准确度识别人类偏好的阅读内容（在某些情况下超过 90%）。

Title: Assisting humans in complex comparisons: automated information comparison at scale

• Authors: Truman Yuen, Graham A. Watt, Yuri Lawryshyn
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04351
• Pdf URL: https://arxiv.org/pdf/2404.04351
• Copy Paste: [[2404.04351]] Assisting humans in complex comparisons: automated information comparison at scale(https://arxiv.org/abs/2404.04351)
• Keywords: language model, llm, prompt, retrieval augmented generation
• Abstract: Generative Large Language Models enable efficient analytics across knowledge domains, rivalling human experts in information comparisons. However, the applications of LLMs for information comparisons face scalability challenges due to the difficulties in maintaining information across large contexts and overcoming model token limitations. To address these challenges, we developed the novel Abstractive Summarization \& Criteria-driven Comparison Endpoint (ASC$^2$End) system to automate information comparison at scale. Our system employs Semantic Text Similarity comparisons for generating evidence-supported analyses. We utilize proven data-handling strategies such as abstractive summarization and retrieval augmented generation to overcome token limitations and retain relevant information during model inference. Prompts were designed using zero-shot strategies to contextualize information for improved model reasoning. We evaluated abstractive summarization using ROUGE scoring and assessed the generated comparison quality using survey responses. Models evaluated on the ASC$^2$End system show desirable results providing insights on the expected performance of the system. ASC$^2$End is a novel system and tool that enables accurate, automated information comparison at scale across knowledge domains, overcoming limitations in context length and retrieval.
• 摘要：生成大型语言模型可以跨知识领域进行高效分析，在信息比较方面可与人类专家相媲美。然而，由于在大型上下文中维护信息和克服模型令牌限制的困难，法学硕士在信息比较方面的应用面临着可扩展性的挑战。为了应对这些挑战，我们开发了新颖的抽象总结\和标准驱动的比较端点（ASC$^2$End）系统来大规模自动化信息比较。我们的系统采用语义文本相似性比较来生成证据支持的分析。我们利用经过验证的数据处理策略（例如抽象摘要和检索增强生成）来克服标记限制并在模型推理过程中保留相关信息。提示是使用零样本策略设计的，将信息置于上下文中以改进模型推理。我们使用 ROUGE 评分评估抽象摘要，并使用调查回复评估生成的比较质量。在 ASC$^2$End 系统上评估的模型显示出理想的结果，提供了对系统预期性能的见解。 ASC$^2$End 是一种新颖的系统和工具，可以跨知识领域进行大规模的准确、自动化的信息比较，克服上下文长度和检索的限制。

Title: Deciphering Political Entity Sentiment in News with Large Language Models: Zero-Shot and Few-Shot Strategies

• Authors: Alapan Kuila, Sudeshna Sarkar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04361
• Pdf URL: https://arxiv.org/pdf/2404.04361
• Copy Paste: [[2404.04361]] Deciphering Political Entity Sentiment in News with Large Language Models: Zero-Shot and Few-Shot Strategies(https://arxiv.org/abs/2404.04361)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Sentiment analysis plays a pivotal role in understanding public opinion, particularly in the political domain where the portrayal of entities in news articles influences public perception. In this paper, we investigate the effectiveness of Large Language Models (LLMs) in predicting entity-specific sentiment from political news articles. Leveraging zero-shot and few-shot strategies, we explore the capability of LLMs to discern sentiment towards political entities in news content. Employing a chain-of-thought (COT) approach augmented with rationale in few-shot in-context learning, we assess whether this method enhances sentiment prediction accuracy. Our evaluation on sentiment-labeled datasets demonstrates that LLMs, outperform fine-tuned BERT models in capturing entity-specific sentiment. We find that learning in-context significantly improves model performance, while the self-consistency mechanism enhances consistency in sentiment prediction. Despite the promising results, we observe inconsistencies in the effectiveness of the COT prompting method. Overall, our findings underscore the potential of LLMs in entity-centric sentiment analysis within the political news domain and highlight the importance of suitable prompting strategies and model architectures.
• 摘要：情感分析在理解公众舆论方面发挥着关键作用，特别是在政治领域，新闻文章中实体的描述会影响公众的看法。在本文中，我们研究了大型语言模型（LLM）在预测政治新闻文章中特定实体情绪方面的有效性。利用零样本和少样本策略，我们探索了法学硕士辨别新闻内容中对政治实体的情绪的能力。采用思想链（COT）方法，并在少镜头上下文学习中增强基本原理，我们评估该方法是否提高了情绪预测的准确性。我们对情感标记数据集的评估表明，LLM 在捕获特定实体情感方面优于微调的 BERT 模型。我们发现上下文学习显着提高了模型性能，而自我一致性机制增强了情绪预测的一致性。尽管结果令人鼓舞，但我们观察到 COT 提示方法的有效性不一致。总体而言，我们的研究结果强调了法学硕士在政治新闻领域以实体为中心的情感分析中的潜力，并强调了适当的提示策略和模型架构的重要性。

Title: KazQAD: Kazakh Open-Domain Question Answering Dataset

• Authors: Rustem Yeshpanov, Pavel Efimov, Leonid Boytsov, Ardak Shalkarbayuli, Pavel Braslavski
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04487
• Pdf URL: https://arxiv.org/pdf/2404.04487
• Copy Paste: [[2404.04487]] KazQAD: Kazakh Open-Domain Question Answering Dataset(https://arxiv.org/abs/2404.04487)
• Keywords: gpt, chat
• Abstract: We introduce KazQAD -- a Kazakh open-domain question answering (ODQA) dataset -- that can be used in both reading comprehension and full ODQA settings, as well as for information retrieval experiments. KazQAD contains just under 6,000 unique questions with extracted short answers and nearly 12,000 passage-level relevance judgements. We use a combination of machine translation, Wikipedia search, and in-house manual annotation to ensure annotation efficiency and data quality. The questions come from two sources: translated items from the Natural Questions (NQ) dataset (only for training) and the original Kazakh Unified National Testing (UNT) exam (for development and testing). The accompanying text corpus contains more than 800,000 passages from the Kazakh Wikipedia. As a supplementary dataset, we release around 61,000 question-passage-answer triples from the NQ dataset that have been machine-translated into Kazakh. We develop baseline retrievers and readers that achieve reasonable scores in retrieval (NDCG@10 = 0.389 MRR = 0.382), reading comprehension (EM = 38.5 F1 = 54.2), and full ODQA (EM = 17.8 F1 = 28.7) settings. Nevertheless, these results are substantially lower than state-of-the-art results for English QA collections, and we think that there should still be ample room for improvement. We also show that the current OpenAI's ChatGPTv3.5 is not able to answer KazQAD test questions in the closed-book setting with acceptable quality. The dataset is freely available under the Creative Commons licence (CC BY-SA) at https://github.com/IS2AI/KazQAD.
• 摘要：我们介绍 KazQAD——一个哈萨克开放域问答 (ODQA) 数据集——可用于阅读理解和完整的 ODQA 设置，以及信息检索实验。 KazQAD 包含近 6,000 个独特问题以及提取的简短答案和近 12,000 个段落级别的相关性判断。我们结合使用机器翻译、维基百科搜索和内部手动标注来确保标注效率和数据质量。这些问题来自两个来源：自然问题 (NQ) 数据集的翻译项目（仅用于训练）和原始的哈萨克斯坦统一国家测试 (UNT) 考试（用于开发和测试）。随附的文本语料库包含来自哈萨克语维基百科的 800,000 多个段落。作为补充数据集，我们从 NQ 数据集中发布了大约 61,000 个问题-段落-答案三元组，这些三元组已被机器翻译成哈萨克语。我们开发了在检索 (NDCG@10 = 0.389 MRR = 0.382)、阅读理解 (EM = 38.5 F1 = 54.2) 和完整 ODQA (EM = 17.8 F1 = 28.7) 设置方面取得合理分数的基线检索器和阅读器。尽管如此，这些结果远远低于英语 QA 集合的最新结果，我们认为仍然有足够的改进空间。我们还表明，当前 OpenAI 的 ChatGPTv3.5 无法在闭卷环境中以可接受的质量回答 KazQAD 测试问题。该数据集可根据知识共享许可 (CC BY-SA) 在 https://github.com/IS2AI/KazQAD 上免费获取。

Title: IITK at SemEval-2024 Task 2: Exploring the Capabilities of LLMs for Safe Biomedical Natural Language Inference for Clinical Trials

• Authors: Shreyasi Mandal, Ashutosh Modi
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04510
• Pdf URL: https://arxiv.org/pdf/2404.04510
• Copy Paste: [[2404.04510]] IITK at SemEval-2024 Task 2: Exploring the Capabilities of LLMs for Safe Biomedical Natural Language Inference for Clinical Trials(https://arxiv.org/abs/2404.04510)
• Keywords: language model, gpt, llm, retrieval-augmented generation
• Abstract: Large Language models (LLMs) have demonstrated state-of-the-art performance in various natural language processing (NLP) tasks across multiple domains, yet they are prone to shortcut learning and factual inconsistencies. This research investigates LLMs' robustness, consistency, and faithful reasoning when performing Natural Language Inference (NLI) on breast cancer Clinical Trial Reports (CTRs) in the context of SemEval 2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials. We examine the reasoning capabilities of LLMs and their adeptness at logical problem-solving. A comparative analysis is conducted on pre-trained language models (PLMs), GPT-3.5, and Gemini Pro under zero-shot settings using Retrieval-Augmented Generation (RAG) framework, integrating various reasoning chains. The evaluation yields an F1 score of 0.69, consistency of 0.71, and a faithfulness score of 0.90 on the test dataset.
• 摘要：大型语言模型 (LLM) 在跨多个领域的各种自然语言处理 (NLP) 任务中展示了最先进的性能，但它们很容易出现捷径学习和事实不一致的情况。这项研究调查了法学硕士在 SemEval 2024 任务 2：临床试验的安全生物医学自然语言推理的背景下对乳腺癌临床试验报告 (CTR) 进行自然语言推理 (NLI) 时的鲁棒性、一致性和忠实推理。我们考察法学硕士的推理能力及其解决逻辑问题的能力。使用检索增强生成（RAG）框架，集成各种推理链，在零样本设置下对预训练语言模型（PLM）、GPT-3.5 和 Gemini Pro 进行比较分析。评估在测试数据集上的 F1 得分为 0.69，一致性得分为 0.71，可信度得分为 0.90。

Title: Joint Visual and Text Prompting for Improved Object-Centric Perception with Multimodal Large Language Models

• Authors: Songtao Jiang, Yan Zhang, Chenyi Zhou, Yeying Jin, Yang Feng, Jian Wu, Zuozhu Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04514
• Pdf URL: https://arxiv.org/pdf/2404.04514
• Copy Paste: [[2404.04514]] Joint Visual and Text Prompting for Improved Object-Centric Perception with Multimodal Large Language Models(https://arxiv.org/abs/2404.04514)
• Keywords: language model, gpt, llm, hallucination, prompt
• Abstract: Multimodal Large Language Models (MLLMs) such as GPT-4V and Gemini Pro face challenges in achieving human-level perception in Visual Question Answering (VQA), particularly in object-oriented perception tasks which demand fine-grained understanding of object identities, locations or attributes, as indicated by empirical findings. This is mainly due to their limited capability to effectively integrate complex visual cues with textual information and potential object hallucinations. In this paper, we present a novel approach, Joint Visual and Text Prompting (VTPrompt), that employs fine-grained visual information to enhance the capability of MLLMs in VQA, especially for object-oriented perception. VTPrompt merges visual and text prompts to extract key concepts from textual questions and employs a detection model to highlight relevant objects as visual prompts in images. The processed images alongside text prompts are subsequently fed into MLLMs to produce more accurate answers. Our experiments with GPT-4V and Gemini Pro, on three benchmarks, i.e., MME , MMB and POPE, demonstrate significant improvements. Particularly, our method led to a score improvement of up to 183.5 for GPT-4V on MME and enhanced MMB performance by 8.17\% for GPT-4V and 15.69\% for Gemini Pro.
• 摘要：GPT-4V 和 Gemini Pro 等多模态大语言模型 (MLLM) 在视觉问答 (VQA) 中实现人类水平的感知方面面临挑战，特别是在面向对象的感知任务中，这些任务需要对对象身份、位置或对象进行细粒度的理解。属性，如实证结果所示。这主要是由于它们有效地将复杂的视觉线索与文本信息和潜在的物体幻觉结合起来的能力有限。在本文中，我们提出了一种新颖的方法，联合视觉和文本提示（VTPrompt），它利用细粒度的视觉信息来增强 MLLM 在 VQA 中的能力，特别是面向对象的感知。 VTPrompt 合并视觉和文本提示，从文本问题中提取关键概念，并采用检测模型将相关对象突出显示为图像中的视觉提示。处理后的图像和文本提示随后被输入 MLLM，以产生更准确的答案。我们使用 GPT-4V 和 Gemini Pro 在 MME、MMB 和 POPE 三个基准上进行的实验表明了显着的改进。特别是，我们的方法使 GPT-4V 在 MME 上的得分提高了 183.5，并使 GPT-4V 的 MMB 性能提高了 8.17%，Gemini Pro 的 MMB 性能提高了 15.69%。

Title: Q-PEFT: Query-dependent Parameter Efficient Fine-tuning for Text Reranking with Large Language Models

• Authors: Zhiyuan Peng, Xuyang Wu, Qifan Wang, Sravanthi Rajanala, Yi Fang
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04522
• Pdf URL: https://arxiv.org/pdf/2404.04522
• Copy Paste: [[2404.04522]] Q-PEFT: Query-dependent Parameter Efficient Fine-tuning for Text Reranking with Large Language Models(https://arxiv.org/abs/2404.04522)
• Keywords: language model, llm, prompt
• Abstract: Parameter Efficient Fine-Tuning (PEFT) methods have been extensively utilized in Large Language Models (LLMs) to improve the down-streaming tasks without the cost of fine-tuing the whole LLMs. Recent studies have shown how to effectively use PEFT for fine-tuning LLMs in ranking tasks with convincing performance; there are some limitations, including the learned prompt being fixed for different documents, overfitting to specific tasks, and low adaptation ability. In this paper, we introduce a query-dependent parameter efficient fine-tuning (Q-PEFT) approach for text reranking to leak the information of the true queries to LLMs and then make the generation of true queries from input documents much easier. Specifically, we utilize the query to extract the top-$k$ tokens from concatenated documents, serving as contextual clues. We further augment Q-PEFT by substituting the retrieval mechanism with a multi-head attention layer to achieve end-to-end training and cover all the tokens in the documents, guiding the LLMs to generate more document-specific synthetic queries, thereby further improving the reranking performance. Extensive experiments are conducted on four public datasets, demonstrating the effectiveness of our proposed approach.
• 摘要：参数高效微调（PEFT）方法已广泛应用于大型语言模型（LLM）中，以改进下游任务，而无需微调整个 LLM 的成本。最近的研究表明，如何有效地使用 PEFT 来微调 LLM 的排名任务，并取得令人信服的表现；存在一些局限性，包括学习的提示针对不同的文档是固定的、对特定任务的过度拟合以及适应能力较低。在本文中，我们引入了一种用于文本重新排名的查询相关参数高效微调（Q-PEFT）方法，将真实查询的信息泄露给LLM，然后使从输入文档生成真实查询变得更加容易。具体来说，我们利用查询从串联文档中提取 top-$k$ 标记，作为上下文线索。我们进一步增强了 Q-PEFT，用多头注意力层代替检索机制，以实现端到端训练并覆盖文档中的所有标记，引导 LLM 生成更多特定于文档的合成查询，从而进一步改进重新排名表现。在四个公共数据集上进行了广泛的实验，证明了我们提出的方法的有效性。

Title: A Morphology-Based Investigation of Positional Encodings

• Authors: Poulami Ghosh, Shikhar Vashishth, Raj Dabre, Pushpak Bhattacharyya
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04530
• Pdf URL: https://arxiv.org/pdf/2404.04530
• Copy Paste: [[2404.04530]] A Morphology-Based Investigation of Positional Encodings(https://arxiv.org/abs/2404.04530)
• Keywords: language model
• Abstract: How does the importance of positional encoding in pre-trained language models (PLMs) vary across languages with different morphological complexity? In this paper, we offer the first study addressing this question, encompassing 23 morphologically diverse languages and 5 different downstream tasks. We choose two categories of tasks: syntactic tasks (part-of-speech tagging, named entity recognition, dependency parsing) and semantic tasks (natural language inference, paraphrasing). We consider language-specific BERT models trained on monolingual corpus for our investigation. The main experiment consists of nullifying the effect of positional encoding during fine-tuning and investigating its impact across various tasks and languages. Our findings demonstrate that the significance of positional encoding diminishes as the morphological complexity of a language increases. Across all experiments, we observe clustering of languages according to their morphological typology - with analytic languages at one end and synthetic languages at the opposite end.
• 摘要：预训练语言模型 (PLM) 中位置编码的重要性在具有不同形态复杂性的语言中有何不同？在本文中，我们提供了第一个解决这个问题的研究，涵盖 23 种形态不同的语言和 5 种不同的下游任务。我们选择两类任务：句法任务（词性标注、命名实体识别、依存句法分析）和语义任务（自然语言推理、释义）。我们考虑在单语语料库上训练特定语言的 BERT 模型来进行我们的调查。主要实验包括在微调过程中消除位置编码的影响，并研究其对各种任务和语言的影响。我们的研究结果表明，位置编码的重要性随着语言形态复杂性的增加而减弱。在所有实验中，我们根据语言的形态类型观察到语言的聚类——一端是分析语言，另一端是合成语言。

Title: Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective

• Authors: Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, Wenqiang Lei
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04626
• Pdf URL: https://arxiv.org/pdf/2404.04626
• Copy Paste: [[2404.04626]] Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective(https://arxiv.org/abs/2404.04626)
• Keywords: language model, llm
• Abstract: Direct Preference Optimization (DPO), which derives reward signals directly from pairwise preference data, has shown its effectiveness on aligning Large Language Models (LLMs) with human preferences. Despite its widespread use across various tasks, DPO has been criticized for its sensitivity to the SFT's effectiveness and its hindrance to the learning capacity towards human-preferred responses, leading to less satisfactory performance. To overcome those limitations, the theoretical understanding of DPO are indispensable but still lacking. To this end, we take a step towards theoretically analyzing and understanding the limitations of DPO. Specifically, we provide an analytical framework using the field theory to analyze the optimization process of DPO. By analyzing the gradient vector field of the DPO loss function, we find that the DPO loss function decreases the probability of producing human dispreferred data at a faster rate than it increases the probability of producing preferred data. This provides theoretical insights for understanding the limitations of DPO discovered in the related research experiments, thereby setting the foundation for its improvement.
• 摘要：直接偏好优化 (DPO) 直接从成对偏好数据中得出奖励信号，已显示出其在使大型语言模型 (LLM) 与人类偏好保持一致方面的有效性。尽管 DPO 在各种任务中广泛使用，但因其对 SFT 有效性的敏感性以及对人类首选响应的学习能力的阻碍而受到批评，导致性能不太令人满意。为了克服这些限制，对 DPO 的理论理解是必不可少的，但仍然缺乏。为此，我们从理论上分析和理解DPO的局限性。具体来说，我们提供了一个使用场论的分析框架来分析DPO的优化过程。通过分析 DPO 损失函数的梯度向量场，我们发现 DPO 损失函数以更快的速度降低产生人类不偏好数据的概率，而不是增加产生偏好数据的概率。这为理解相关研究实验中发现的DPO的局限性提供了理论见解，从而为其改进奠定了基础。

Title: On the Limitations of Large Language Models (LLMs): False Attribution

• Authors: Tosin Adewumi, Nudrat Habib, Lama Alkhaled, Elisa Barney
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04631
• Pdf URL: https://arxiv.org/pdf/2404.04631
• Copy Paste: [[2404.04631]] On the Limitations of Large Language Models (LLMs): False Attribution(https://arxiv.org/abs/2404.04631)
• Keywords: language model, llm, hallucination
• Abstract: In this work, we provide insight into one important limitation of large language models (LLMs), i.e. false attribution, and introduce a new hallucination metric - Simple Hallucination Index (SHI). The task of automatic author attribution for relatively small chunks of text is an important NLP task but can be challenging. We empirically evaluate the power of 3 open SotA LLMs in zero-shot setting (LLaMA-2-13B, Mixtral 8x7B, and Gemma-7B), especially as human annotation can be costly. We collected the top 10 most popular books, according to Project Gutenberg, divided each one into equal chunks of 400 words, and asked each LLM to predict the author. We then randomly sampled 162 chunks for human evaluation from each of the annotated books, based on the error margin of 7% and a confidence level of 95% for the book with the most chunks (Great Expectations by Charles Dickens, having 922 chunks). The average results show that Mixtral 8x7B has the highest prediction accuracy, the lowest SHI, and a Pearson's correlation (r) of 0.737, 0.249, and -0.9996, respectively, followed by LLaMA-2-13B and Gemma-7B. However, Mixtral 8x7B suffers from high hallucinations for 3 books, rising as high as an SHI of 0.87 (in the range 0-1, where 1 is the worst). The strong negative correlation of accuracy and SHI, given by r, demonstrates the fidelity of the new hallucination metric, which is generalizable to other tasks. We publicly release the annotated chunks of data and our codes to aid the reproducibility and evaluation of other models.
• 摘要：在这项工作中，我们深入了解了大型语言模型（LLM）的一个重要限制，即错误归因，并引入了一种新的幻觉指标——简单幻觉指数（SHI）。对相对较小的文本块进行自动作者归因的任务是一项重要的 NLP 任务，但可能具有挑战性。我们凭经验评估了 3 个开放式 SotA LLM 在零样本设置（LLaMA-2-13B、Mixtral 8x7B 和 Gemma-7B）中的威力，特别是考虑到人工注释的成本可能很高。根据古腾堡计划，我们收集了排名前 10 的最受欢迎的书籍，将每一本书分成 400 个单词的等份块，并要求每位法学硕士预测作者。然后，我们根据包含最多块的书籍（查尔斯·狄更斯的《远大前程》，有 922 个块）的误差范围为 7% 和置信度为 95%，从每本带注释的书中随机抽取 162 个块进行人工评估。平均结果显示，Mixtral 8x7B 的预测精度最高，SHI 最低，Pearson 相关性 (r) 分别为 0.737、0.249 和 -0.9996，其次是 LLaMA-2-13B 和 Gemma-7B。然而，Mixtral 8x7B 在 3 本书中出现了很高的幻觉，SHI 高达 0.87（在 0-1 范围内，其中 1 是最差的）。由 r 给出的准确性和 SHI 的强负相关性证明了新幻觉度量的保真度，该度量可推广到其他任务。我们公开发布带注释的数据块和我们的代码，以帮助其他模型的再现和评估。

Title: Context versus Prior Knowledge in Language Models

• Authors: Kevin Du, Vésteinn Snæbjarnarson, Niklas Stoehr, Jennifer C. White, Aaron Schein, Ryan Cotterell
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04633
• Pdf URL: https://arxiv.org/pdf/2404.04633
• Copy Paste: [[2404.04633]] Context versus Prior Knowledge in Language Models(https://arxiv.org/abs/2404.04633)
• Keywords: language model
• Abstract: To answer a question, language models often need to integrate prior knowledge learned during pretraining and new information presented in context. We hypothesize that models perform this integration in a predictable way across different questions and contexts: models will rely more on prior knowledge for questions about entities (e.g., persons, places, etc.) that they are more familiar with due to higher exposure in the training corpus, and be more easily persuaded by some contexts than others. To formalize this problem, we propose two mutual information-based metrics to measure a model's dependency on a context and on its prior about an entity: first, the persuasion score of a given context represents how much a model depends on the context in its decision, and second, the susceptibility score of a given entity represents how much the model can be swayed away from its original answer distribution about an entity. Following well-established measurement modeling methods, we empirically test for the validity and reliability of these metrics. Finally, we explore and find a relationship between the scores and the model's expected familiarity with an entity, and provide two use cases to illustrate their benefits.
• 摘要：为了回答问题，语言模型通常需要整合预训练期间学到的先验知识和上下文中呈现的新信息。我们假设模型在不同的问题和上下文中以可预测的方式执行这种集成：模型将更多地依赖于有关实体（例如，人、地点等）的先验知识，这些实体由于在环境中的暴露程度较高而更熟悉。训练语料库，并且在某些情况下比其他情况更容易被说服。为了形式化这个问题，我们提出了两个基于相互信息的指标来衡量模型对上下文及其对实体的先验的依赖性：首先，给定上下文的说服分数表示模型在其决策中对上下文的依赖程度，其次，给定实体的敏感性分数表示模型可以偏离其关于实体的原始答案分布的程度。遵循完善的测量建模方法，我们凭经验测试这些指标的有效性和可靠性。最后，我们探索并找到分数与模型对实体的预期熟悉度之间的关系，并提供两个用例来说明它们的好处。

Title: Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly

• Authors: Changjiang Gao, Hongda Hu, Peng Hu, Jiajun Chen, Jixing Li, Shujian Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04659
• Pdf URL: https://arxiv.org/pdf/2404.04659
• Copy Paste: [[2404.04659]] Multilingual Pretraining and Instruction Tuning Improve Cross-Lingual Knowledge Alignment, But Only Shallowly(https://arxiv.org/abs/2404.04659)
• Keywords: language model, llm
• Abstract: Despite their strong ability to retrieve knowledge in English, current large language models show imbalance abilities in different languages. Two approaches are proposed to address this, i.e., multilingual pretraining and multilingual instruction tuning. However, whether and how do such methods contribute to the cross-lingual knowledge alignment inside the models is unknown. In this paper, we propose CLiKA, a systematic framework to assess the cross-lingual knowledge alignment of LLMs in the Performance, Consistency and Conductivity levels, and explored the effect of multilingual pretraining and instruction tuning on the degree of alignment. Results show that: while both multilingual pretraining and instruction tuning are beneficial for cross-lingual knowledge alignment, the training strategy needs to be carefully designed. Namely, continued pretraining improves the alignment of the target language at the cost of other languages, while mixed pretraining affect other languages less. Also, the overall cross-lingual knowledge alignment, especially in the conductivity level, is unsatisfactory for all tested LLMs, and neither multilingual pretraining nor instruction tuning can substantially improve the cross-lingual knowledge conductivity.
• 摘要：尽管它们检索英语知识的能力很强，但当前的大型语言模型在不同语言上表现出不平衡的能力。提出了两种方法来解决这个问题，即多语言预训练和多语言指令调整。然而，这些方法是否以及如何有助于模型内部的跨语言知识对齐尚不清楚。在本文中，我们提出了CLiKA，一个系统框架来评估法学硕士在绩效、一致性和传导性水平上的跨语言知识对齐，并探讨了多语言预训练和指令调整对对齐程度的影响。结果表明：虽然多语言预训练和指令调整都有利于跨语言知识对齐，但需要仔细设计训练策略。也就是说，持续预训练以牺牲其他语言为代价来提高目标语言的对齐，而混合预训练对其他语言的影响较小。此外，所有测试的法学硕士的整体跨语言知识一致性，特别是在传导性水平上都不尽如人意，多语言预训练和指令调整都无法实质性提高跨语言知识传导性。

Title: Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks

• Authors: Nicolas Yax, Pierre-Yves Oudeyer, Stefano Palminteri
• Subjects: cs.CL, cs.LG, q-bio.PE
• Abstract URL: https://arxiv.org/abs/2404.04671
• Pdf URL: https://arxiv.org/pdf/2404.04671
• Copy Paste: [[2404.04671]] Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks(https://arxiv.org/abs/2404.04671)
• Keywords: language model, llm
• Abstract: This paper introduces PhyloLM, a method applying phylogenetic algorithms to Large Language Models to explore their finetuning relationships, and predict their performance characteristics. By leveraging the phylogenetic distance metric, we construct dendrograms, which satisfactorily capture distinct LLM families (across a set of 77 open-source and 22 closed models). Furthermore, phylogenetic distance predicts performances in benchmarks (we test MMLU and ARC), thus enabling a time and cost-effective estimation of LLM capabilities. The approach translates genetic concepts to machine learning, offering tools to infer LLM development, relationships, and capabilities, even in the absence of transparent training information.
• 摘要：本文介绍了 PhyloLM，一种将系统发育算法应用于大型语言模型的方法，以探索它们的微调关系，并预测它们的性能特征。通过利用系统发育距离度量，我们构建了树状图，它令人满意地捕获了不同的 LLM 家族（涵盖 77 个开源模型和 22 个封闭模型）。此外，系统发育距离可以预测基准测试中的表现（我们测试 MMLU 和 ARC），从而能够对 LLM 能力进行时间和成本效益的估计。该方法将遗传概念转化为机器学习，即使在缺乏透明的培训信息的情况下，也提供了推断法学硕士发展、关系和能力的工具。

Title: Order-Based Pre-training Strategies for Procedural Text Understanding

• Authors: Abhilash Nandy, Yash Kulkarni, Pawan Goyal, Niloy Ganguly
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04676
• Pdf URL: https://arxiv.org/pdf/2404.04676
• Copy Paste: [[2404.04676]] Order-Based Pre-training Strategies for Procedural Text Understanding(https://arxiv.org/abs/2404.04676)
• Keywords: llm
• Abstract: In this paper, we propose sequence-based pretraining methods to enhance procedural understanding in natural language processing. Procedural text, containing sequential instructions to accomplish a task, is difficult to understand due to the changing attributes of entities in the context. We focus on recipes, which are commonly represented as ordered instructions, and use this order as a supervision signal. Our work is one of the first to compare several 'order as-supervision' transformer pre-training methods, including Permutation Classification, Embedding Regression, and Skip-Clip, and shows that these methods give improved results compared to the baselines and SoTA LLMs on two downstream Entity-Tracking datasets: NPN-Cooking dataset in recipe domain and ProPara dataset in open domain. Our proposed methods address the non-trivial Entity Tracking Task that requires prediction of entity states across procedure steps, which requires understanding the order of steps. These methods show an improvement over the best baseline by 1.6% and 7-9% on NPN-Cooking and ProPara Datasets respectively across metrics.
• 摘要：在本文中，我们提出了基于序列的预训练方法来增强自然语言处理中的程序理解。由于上下文中实体的属性不断变化，包含完成任务的顺序指令的程序文本很难理解。我们关注配方，它们通常表示为有序指令，并使用此顺序作为监督信号。我们的工作是第一个比较几种“顺序即监督”变压器预训练方法的工作之一，包括排列分类、嵌入回归和 Skip-Clip，并表明与基线和 SoTA LLM 相比，这些方法给出了改进的结果两个下游实体跟踪数据集：菜谱域中的 NPN-Cooking 数据集和开放域中的 ProPara 数据集。我们提出的方法解决了不平凡的实体跟踪任务，该任务需要跨过程步骤预测实体状态，这需要理解步骤的顺序。这些方法在 NPN-Cooking 和 ProPara 数据集上的各个指标上分别比最佳基线提高了 1.6% 和 7-9%。

Title: PoLLMgraph: Unraveling Hallucinations in Large Language Models via State Transition Dynamics

• Authors: Derui Zhu, Dingfan Chen, Qing Li, Zongxiong Chen, Lei Ma, Jens Grossklags, Mario Fritz
• Subjects: cs.CL, cs.CR, cs.SE
• Abstract URL: https://arxiv.org/abs/2404.04722
• Pdf URL: https://arxiv.org/pdf/2404.04722
• Copy Paste: [[2404.04722]] PoLLMgraph: Unraveling Hallucinations in Large Language Models via State Transition Dynamics(https://arxiv.org/abs/2404.04722)
• Keywords: language model, llm, hallucination
• Abstract: Despite tremendous advancements in large language models (LLMs) over recent years, a notably urgent challenge for their practical deployment is the phenomenon of hallucination, where the model fabricates facts and produces non-factual statements. In response, we propose PoLLMgraph, a Polygraph for LLMs, as an effective model-based white-box detection and forecasting approach. PoLLMgraph distinctly differs from the large body of existing research that concentrates on addressing such challenges through black-box evaluations. In particular, we demonstrate that hallucination can be effectively detected by analyzing the LLM's internal state transition dynamics during generation via tractable probabilistic models. Experimental results on various open-source LLMs confirm the efficacy of PoLLMgraph, outperforming state-of-the-art methods by a considerable margin, evidenced by over 20% improvement in AUC-ROC on common benchmarking datasets like TruthfulQA. Our work paves a new way for model-based white-box analysis of LLMs, motivating the research community to further explore, understand, and refine the intricate dynamics of LLM behaviors.
• 摘要：尽管近年来大型语言模型（LLM）取得了巨大进步，但其实际部署面临的一个特别紧迫的挑战是幻觉现象，即模型捏造事实并产生非事实陈述。作为回应，我们提出 PoLLMgraph，即法学硕士的测谎仪，作为一种有效的基于模型的白盒检测和预测方法。 PoLLMgraph 与现有的大量研究明显不同，后者专注于通过黑盒评估来应对此类挑战。特别是，我们证明，通过易处理的概率模型分析法学硕士在生成过程中的内部状态转换动态，可以有效地检测幻觉。各种开源 LLM 的实验结果证实了 PoLLMgraph 的有效性，其性能大大优于最先进的方法，在常见基准测试数据集（如 TruthfulQA）上 AUC-ROC 提高了 20% 以上就证明了这一点。我们的工作为基于模型的法学硕士白盒分析铺平了一条新途径，激励研究界进一步探索、理解和完善法学硕士行为的复杂动态。

Title: Navigating the Landscape of Hint Generation Research: From the Past to the Future

• Authors: Anubhav Jangra, Jamshid Mozafari, Adam Jatowt, Smaranda Muresan
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2404.04728
• Pdf URL: https://arxiv.org/pdf/2404.04728
• Copy Paste: [[2404.04728]] Navigating the Landscape of Hint Generation Research: From the Past to the Future(https://arxiv.org/abs/2404.04728)
• Keywords: language model
• Abstract: Digital education has gained popularity in the last decade, especially after the COVID-19 pandemic. With the improving capabilities of large language models to reason and communicate with users, envisioning intelligent tutoring systems (ITSs) that can facilitate self-learning is not very far-fetched. One integral component to fulfill this vision is the ability to give accurate and effective feedback via hints to scaffold the learning process. In this survey article, we present a comprehensive review of prior research on hint generation, aiming to bridge the gap between research in education and cognitive science, and research in AI and Natural Language Processing. Informed by our findings, we propose a formal definition of the hint generation task, and discuss the roadmap of building an effective hint generation system aligned with the formal definition, including open challenges, future directions and ethical considerations.
• 摘要：数字教育在过去十年中越来越受欢迎，尤其是在 COVID-19 大流行之后。随着大型语言模型推理和与用户交流的能力不断提高，设想能够促进自学的智能辅导系统（ITS）并不是很遥不可及。实现这一愿景的一个不可或缺的组成部分是能够通过提示提供准确有效的反馈来支撑学习过程。在这篇调查文章中，我们对提示生成的先前研究进行了全面回顾，旨在弥合教育和认知科学研究与人工智能和自然语言处理研究之间的差距。根据我们的发现，我们提出了提示生成任务的正式定义，并讨论了构建与正式定义一致的有效提示生成系统的路线图，包括开放挑战、未来方向和道德考虑。

Title: Multilingual Brain Surgeon: Large Language Models Can be Compressed Leaving No Language Behind

• Authors: Hongchuan Zeng, Hongshen Xu, Lu Chen, Kai Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04748
• Pdf URL: https://arxiv.org/pdf/2404.04748
• Copy Paste: [[2404.04748]] Multilingual Brain Surgeon: Large Language Models Can be Compressed Leaving No Language Behind(https://arxiv.org/abs/2404.04748)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have ushered in a new era in Natural Language Processing, but their massive size demands effective compression techniques for practicality. Although numerous model compression techniques have been investigated, they typically rely on a calibration set that overlooks the multilingual context and results in significant accuracy degradation for low-resource languages. This paper introduces Multilingual Brain Surgeon (MBS), a novel calibration data sampling method for multilingual LLMs compression. MBS overcomes the English-centric limitations of existing methods by sampling calibration data from various languages proportionally to the language distribution of the model training datasets. Our experiments, conducted on the BLOOM multilingual LLM, demonstrate that MBS improves the performance of existing English-centric compression methods, especially for low-resource languages. We also uncover the dynamics of language interaction during compression, revealing that the larger the proportion of a language in the training set and the more similar the language is to the calibration language, the better performance the language retains after compression. In conclusion, MBS presents an innovative approach to compressing multilingual LLMs, addressing the performance disparities and improving the language inclusivity of existing compression techniques.
• 摘要：大型语言模型 (LLM) 开创了自然语言处理的新时代，但其庞大的规模需要有效的压缩技术才能实现实用。尽管已经研究了许多模型压缩技术，但它们通常依赖于忽略多语言上下文的校准集，并导致低资源语言的准确性显着下降。本文介绍了多语言脑外科医生 (MBS)，这是一种用于多语言 LLM 压缩的新型校准数据采样方法。 MBS 通过与模型训练数据集的语言分布成比例地对各种语言的校准数据进行采样，克服了现有方法以英语为中心的局限性。我们在 BLOOM 多语言 LLM 上进行的实验表明，MBS 提高了现有以英语为中心的压缩方法的性能，特别是对于资源匮乏的语言。我们还揭示了压缩过程中语言交互的动态，揭示了训练集中语言所占的比例越大，并且该语言与校准语言越相似，该语言在压缩后保留的性能就越好。总之，MBS 提出了一种压缩多语言 LLM 的创新方法，解决了性能差异并提高了现有压缩技术的语言包容性。

Title: What Happens When Small Is Made Smaller? Exploring the Impact of Compression on Small Data Pretrained Language Models

• Authors: Busayo Awobade, Mardiyyah Oduwole, Steven Kolawole
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.04759
• Pdf URL: https://arxiv.org/pdf/2404.04759
• Copy Paste: [[2404.04759]] What Happens When Small Is Made Smaller? Exploring the Impact of Compression on Small Data Pretrained Language Models(https://arxiv.org/abs/2404.04759)
• Keywords: language model
• Abstract: Compression techniques have been crucial in advancing machine learning by enabling efficient training and deployment of large-scale language models. However, these techniques have received limited attention in the context of low-resource language models, which are trained on even smaller amounts of data and under computational constraints, a scenario known as the "low-resource double-bind." This paper investigates the effectiveness of pruning, knowledge distillation, and quantization on an exclusively low-resourced, small-data language model, AfriBERTa. Through a battery of experiments, we assess the effects of compression on performance across several metrics beyond accuracy. Our study provides evidence that compression techniques significantly improve the efficiency and effectiveness of small-data language models, confirming that the prevailing beliefs regarding the effects of compression on large, heavily parameterized models hold true for less-parameterized, small-data models.
• 摘要：压缩技术通过实现大规模语言模型的高效训练和部署，对于推进机器学习至关重要。然而，这些技术在低资源语言模型的背景下受到的关注有限，这些模型是在更少量的数据和计算限制下进行训练的，这种情况被称为“低资源双重绑定”。本文研究了在资源匮乏、小数据语言模型 AfriBERTa 上进行剪枝、知识蒸馏和量化的有效性。通过一系列实验，我们评估了压缩对准确性之外的多个指标的性能影响。我们的研究提供的证据表明，压缩技术显着提高了小数据语言模型的效率和有效性，证实了关于压缩对大型、重参数化模型的影响的普遍观点对于参数化程度较低的小数据模型也适用。

Title: Low-Resource Machine Translation through Retrieval-Augmented LLM Prompting: A Study on the Mambai Language

• Authors: Raphaël Merx, Aso Mahmudi, Katrina Langford, Leo Alberto de Araujo, Ekaterina Vylomova
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04809
• Pdf URL: https://arxiv.org/pdf/2404.04809
• Copy Paste: [[2404.04809]] Low-Resource Machine Translation through Retrieval-Augmented LLM Prompting: A Study on the Mambai Language(https://arxiv.org/abs/2404.04809)
• Keywords: language model, gpt, llm, prompt
• Abstract: This study explores the use of large language models (LLMs) for translating English into Mambai, a low-resource Austronesian language spoken in Timor-Leste, with approximately 200,000 native speakers. Leveraging a novel corpus derived from a Mambai language manual and additional sentences translated by a native speaker, we examine the efficacy of few-shot LLM prompting for machine translation (MT) in this low-resource context. Our methodology involves the strategic selection of parallel sentences and dictionary entries for prompting, aiming to enhance translation accuracy, using open-source and proprietary LLMs (LlaMa 2 70b, Mixtral 8x7B, GPT-4). We find that including dictionary entries in prompts and a mix of sentences retrieved through TF-IDF and semantic embeddings significantly improves translation quality. However, our findings reveal stark disparities in translation performance across test sets, with BLEU scores reaching as high as 21.2 on materials from the language manual, in contrast to a maximum of 4.4 on a test set provided by a native speaker. These results underscore the importance of diverse and representative corpora in assessing MT for low-resource languages. Our research provides insights into few-shot LLM prompting for low-resource MT, and makes available an initial corpus for the Mambai language.
• 摘要：本研究探讨了如何使用大型语言模型 (LLM) 将英语翻译成曼巴伊语，曼巴伊语是一种资源匮乏的南岛语，在东帝汶使用，约有 200,000 名母语使用者。利用源自曼巴伊语言手册的新颖语料库和由母语人士翻译的附加句子，我们研究了在这种资源匮乏的情况下，小样本法学硕士提示机器翻译（MT）的功效。我们的方法涉及战略性地选择平行句子和词典条目进行提示，旨在使用开源和专有的 LLM（LlaMa 2 70b、Mixtral 8x7B、GPT-4）提高翻译准确性。我们发现，在提示中包含字典条目以及通过 TF-IDF 和语义嵌入检索的句子混合可以显着提高翻译质量。然而，我们的研究结果表明，不同测试集的翻译性能存在明显差异，语言手册材料上的 BLEU 得分高达 21.2，而母语人士提供的测试集上的最高得分为 4.4。这些结果强调了多样化和代表性语料库在评估低资源语言机器翻译方面的重要性。我们的研究提供了对小样本法学硕士促进低资源机器翻译的见解，并提供了曼拜语言的初始语料库。

Title: FRACTAL: Fine-Grained Scoring from Aggregate Text Labels

• Authors: Yukti Makhija, Priyanka Agrawal, Rishi Saket, Aravindan Raghuveer
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04817
• Pdf URL: https://arxiv.org/pdf/2404.04817
• Copy Paste: [[2404.04817]] FRACTAL: Fine-Grained Scoring from Aggregate Text Labels(https://arxiv.org/abs/2404.04817)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are being increasingly tuned to power complex generation tasks such as writing, fact-seeking, querying and reasoning. Traditionally, human or model feedback for evaluating and further tuning LLM performance has been provided at the response level, enabling faster and more cost-effective assessments. However, recent works (Amplayo et al. [2022], Wu et al. [2023]) indicate that sentence-level labels may provide more accurate and interpretable feedback for LLM optimization. In this work, we introduce methods to disaggregate response-level labels into sentence-level (pseudo-)labels. Our approach leverages multiple instance learning (MIL) and learning from label proportions (LLP) techniques in conjunction with prior information (e.g., document-sentence cosine similarity) to train a specialized model for sentence-level scoring. We also employ techniques which use model predictions to pseudo-label the train-set at the sentence-level for model training to further improve performance. We conduct extensive evaluations of our methods across six datasets and four tasks: retrieval, question answering, summarization, and math reasoning. Our results demonstrate improved performance compared to multiple baselines across most of these tasks. Our work is the first to develop response-level feedback to sentence-level scoring techniques, leveraging sentence-level prior information, along with comprehensive evaluations on multiple tasks as well as end-to-end finetuning evaluation showing performance comparable to a model trained on fine-grained human annotated labels.
• 摘要：大型语言模型 (LLM) 越来越多地被调整为支持复杂的生成任务，例如写作、事实调查、查询和推理。传统上，用于评估和进一步调整 LLM 表现的人工或模型反馈是在响应级别提供的，从而实现更快、更具成本效益的评估。然而，最近的工作（Amplayo 等人[2022]、Wu 等人[2023]）表明句子级标签可以为 LLM 优化提供更准确和可解释的反馈。在这项工作中，我们介绍了将响应级别标签分解为句子级别（伪）标签的方法。我们的方法利用多实例学习（MIL）和从标签比例（LLP）技术中学习，结合先验信息（例如文档句子余弦相似度）来训练用于句子级评分的专门模型。我们还采用了使用模型预测在句子级别对训练集进行伪标记的技术，以进行模型训练，以进一步提高性能。我们在六个数据集和四个任务中对我们的方法进行了广泛的评估：检索、问答、总结和数学推理。我们的结果表明，与大多数这些任务的多个基线相比，性能有所提高。我们的工作是第一个开发句子级评分技术的响应级反馈，利用句子级先验信息，以及对多个任务的综合评估以及端到端微调评估，显示与训练的模型相当的性能细粒度的人工注释标签。

Title: Data Bias According to Bipol: Men are Naturally Right and It is the Role of Women to Follow Their Lead

• Authors: Irene Pagliai, Goya van Boven, Tosin Adewumi, Lama Alkhaled, Namrata Gurung, Isabella Södergren, Elisa Barney
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04838
• Pdf URL: https://arxiv.org/pdf/2404.04838
• Copy Paste: [[2404.04838]] Data Bias According to Bipol: Men are Naturally Right and It is the Role of Women to Follow Their Lead(https://arxiv.org/abs/2404.04838)
• Keywords: language model, llm
• Abstract: We introduce new large labeled datasets on bias in 3 languages and show in experiments that bias exists in all 10 datasets of 5 languages evaluated, including benchmark datasets on the English GLUE/SuperGLUE leaderboards. The 3 new languages give a total of almost 6 million labeled samples and we benchmark on these datasets using SotA multilingual pretrained models: mT5 and mBERT. The challenge of social bias, based on prejudice, is ubiquitous, as recent events with AI and large language models (LLMs) have shown. Motivated by this challenge, we set out to estimate bias in multiple datasets. We compare some recent bias metrics and use bipol, which has explainability in the metric. We also confirm the unverified assumption that bias exists in toxic comments by randomly sampling 200 samples from a toxic dataset population using the confidence level of 95% and error margin of 7%. Thirty gold samples were randomly distributed in the 200 samples to secure the quality of the annotation. Our findings confirm that many of the datasets have male bias (prejudice against women), besides other types of bias. We publicly release our new datasets, lexica, models, and codes.
• 摘要：我们引入了关于 3 种语言偏差的新大型标记数据集，并在实验中表明，所评估的 5 种语言的所有 10 个数据集中都存在偏差，包括英语 GLUE/SuperGLUE 排行榜上的基准数据集。这 3 种新语言总共提供了近 600 万个标记样本，我们使用 SotA 多语言预训练模型：mT5 和 mBERT 对这些数据集进行基准测试。正如最近有关人工智能和大型语言模型 (LLM) 的事件所表明的那样，基于偏见的社会偏见的挑战无处不在。受到这一挑战的激励，我们开始估计多个数据集中的偏差。我们比较了一些最近的偏差指标并使用 bipol，它在指标上具有可解释性。我们还通过使用 95% 的置信度和 7% 的误​​差范围从有毒数据集中随机抽取 200 个样本，证实了未经验证的假设，即有毒评论中存在偏见。 200个样本中随机分配了30个金样本，以保证注释的质量。我们的研究结果证实，除了其他类型的偏见之外，许多数据集还存在男性偏见（对女性的偏见）。我们公开发布新的数据集、词汇、模型和代码。

Title: SLPL SHROOM at SemEval\-2024 Task 06: A comprehensive study on models ability to detect hallucination

• Authors: Pouya Fallah, Soroush Gooran, Mohammad Jafarinasab, Pouya Sadeghi, Reza Farnia, Amirreza Tarabkhah, Zainab Sadat Taghavi, Hossein Sameti
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04845
• Pdf URL: https://arxiv.org/pdf/2404.04845
• Copy Paste: [[2404.04845]] SLPL SHROOM at SemEval\-2024 Task 06: A comprehensive study on models ability to detect hallucination(https://arxiv.org/abs/2404.04845)
• Keywords: language model, hallucination
• Abstract: Language models, particularly generative models, are susceptible to hallucinations, generating outputs that contradict factual knowledge or the source text. This study explores methods for detecting hallucinations in three SemEval-2024 Task 6 tasks: Machine Translation, Definition Modeling, and Paraphrase Generation. We evaluate two methods: semantic similarity between the generated text and factual references, and an ensemble of language models that judge each other's outputs. Our results show that semantic similarity achieves moderate accuracy and correlation scores in trial data, while the ensemble method offers insights into the complexities of hallucination detection but falls short of expectations. This work highlights the challenges of hallucination detection and underscores the need for further research in this critical area.
• 摘要：语言模型，特别是生成模型，很容易产生幻觉，生成与事实知识或源文本相矛盾的输出。本研究探讨了在三个 SemEval-2024 Task 6 任务中检测幻觉的方法：机器翻译、定义建模和释义生成。我们评估两种方法：生成的文本和事实参考之间的语义相似性，以及判断彼此输出的语言模型集合。我们的结果表明，语义相似性在试验数据中实现了中等的准确性和相关性分数，而集成方法提供了对幻觉检测复杂性的见解，但未达到预期。这项工作突出了幻觉检测的挑战，并强调了在这一关键领域进行进一步研究的必要性。

Title: Lucky 52: How Many Languages Are Needed to Instruction Fine-Tune Large Language Models?

• Authors: Shaoxiong Ji, Pinzhen Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04850
• Pdf URL: https://arxiv.org/pdf/2404.04850
• Copy Paste: [[2404.04850]] Lucky 52: How Many Languages Are Needed to Instruction Fine-Tune Large Language Models?(https://arxiv.org/abs/2404.04850)
• Keywords: language model
• Abstract: Fine-tuning large language models for multilingual downstream tasks requires a diverse set of languages to capture the nuances and structures of different linguistic contexts effectively. While the specific number varies depending on the desired scope and target languages, we argue that the number of languages, language exposure, and similarity that incorporate the selection of languages for fine-tuning are some important aspects to examine. By fine-tuning large multilingual models on 1 to 52 languages, this paper answers one question: How many languages are needed in instruction fine-tuning for multilingual tasks? We investigate how multilingual instruction fine-tuned models behave on multilingual benchmarks with an increasing number of languages and discuss our findings from the perspective of language exposure and similarity.
• 摘要：为多语言下游任务微调大型语言模型需要多种语言来有效捕获不同语言环境的细微差别和结构。虽然具体数量根据所需范围和目标语言而有所不同，但我们认为，语言数量、语言暴露以及包含微调语言选择的相似性是需要检查的一些重要方面。通过对 1 到 52 种语言的大型多语言模型进行微调，本文回答了一个问题：多语言任务的指令微调需要多少种语言？我们研究了多语言教学微调模型如何在越来越多的语言的多语言基准上表现，并从语言暴露和相似性的角度讨论我们的发现。

Title: Ethos and Pathos in Online Group Discussions: Corpora for Polarisation Issues in Social Media

• Authors: Ewelina Gajewska, Katarzyna Budzynska, Barbara Konat, Marcin Koszowy, Konrad Kiljan, Maciej Uberna, He Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04889
• Pdf URL: https://arxiv.org/pdf/2404.04889
• Copy Paste: [[2404.04889]] Ethos and Pathos in Online Group Discussions: Corpora for Polarisation Issues in Social Media(https://arxiv.org/abs/2404.04889)
• Keywords: language model
• Abstract: Growing polarisation in society caught the attention of the scientific community as well as news media, which devote special issues to this phenomenon. At the same time, digitalisation of social interactions requires to revise concepts from social science regarding establishment of trust, which is a key feature of all human interactions, and group polarisation, as well as new computational tools to process large quantities of available data. Existing methods seem insufficient to tackle the problem fully, thus, we propose to approach the problem by investigating rhetorical strategies employed by individuals in polarising discussions online. To this end, we develop multi-topic and multi-platform corpora with manual annotation of appeals to ethos and pathos, two modes of persuasion in Aristotelian rhetoric. It can be employed for training language models to advance the study of communication strategies online on a large scale. With the use of computational methods, our corpora allows an investigation of recurring patterns in polarising exchanges across topics of discussion and media platforms, and conduct both quantitative and qualitative analyses of language structures leading to and engaged in polarisation.
• 摘要：社会日益严重的两极分化引起了科学界和新闻媒体的关注，它们专门针对这一现象专门发行了专题。与此同时，社会互动的数字化需要修改社会科学中关于建立信任（这是所有人类互动的一个关键特征）和群体极化的概念，以及处理大量可用数据的新计算工具。现有的方法似乎不足以完全解决这个问题，因此，我们建议通过调查个人在网上两极分化讨论中使用的修辞策略来解决这个问题。为此，我们开发了多主题和多平台语料库，并手​​动注释了亚里士多德修辞中的两种说服模式：精神和情感。它可以用于训练语言模型，以大规模地推进在线交流策略的研究。通过使用计算方法，我们的语料库可以调查讨论主题和媒体平台上的两极分化交流中的重复模式，并对导致和参与两极分化的语言结构进行定量和定性分析。

Title: Radial Networks: Dynamic Layer Routing for High-Performance Large Language Models

• Authors: Jordan Dotzel, Yash Akhauri, Ahmed S. AbouElhamayed, Carly Jiang, Mohamed Abdelfattah, Zhiru Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04900
• Pdf URL: https://arxiv.org/pdf/2404.04900
• Copy Paste: [[2404.04900]] Radial Networks: Dynamic Layer Routing for High-Performance Large Language Models(https://arxiv.org/abs/2404.04900)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) often struggle with strict memory, latency, and power demands. To meet these demands, various forms of dynamic sparsity have been proposed that reduce compute on an input-by-input basis. These methods improve over static methods by exploiting the variance across individual inputs, which has steadily grown with the exponential increase in training data. Yet, the increasing depth within modern models, currently with hundreds of layers, has opened opportunities for dynamic layer sparsity, which skips the computation for entire layers. In this work, we explore the practicality of layer sparsity by profiling residual connections and establish the relationship between model depth and layer sparsity. For example, the residual blocks in the OPT-66B model have a median contribution of 5% to its output. We then take advantage of this dynamic sparsity and propose Radial Networks, which perform token-level routing between layers guided by a trained router module. These networks can be used in a post-training distillation from sequential networks or trained from scratch to co-learn the router and layer weights. They enable scaling to larger model sizes by decoupling the number of layers from the dynamic depth of the network, and their design allows for layer reuse. By varying the compute token by token, they reduce the overall resources needed for generating entire sequences. Overall, this leads to larger capacity networks with significantly lower compute and serving costs for large language models.
• 摘要：大型语言模型 (LLM) 常常面临严格的内存、延迟和功耗要求。为了满足这些需求，人们提出了各种形式的动态稀疏性，以减少逐个输入的计算量。这些方法通过利用各个输入之间的方差来改进静态方法，这些方差随着训练数据的指数增长而稳定增长。然而，现代模型的深度不断增加（目前有数百层），为动态层稀疏性提供了机会，从而跳过整个层的计算。在这项工作中，我们通过分析残差连接来探索层稀疏性的实用性，并建立模型深度和层稀疏性之间的关系。例如，OPT-66B 模型中的残差块对其输出的中值贡献为 5%。然后，我们利用这种动态稀疏性并提出径向网络，它在经过训练的路由器模块引导的层之间执行令牌级路由。这些网络可以用于序列网络的训练后蒸馏，或者从头开始训练以共同学习路由器和层权重。它们通过将层数与网络的动态深度解耦来实现扩展到更大的模型尺寸，并且它们的设计允许层重用。通过按令牌改变计算令牌，它们减少了生成整个序列所需的总体资源。总的来说，这会导致网络容量更大，大型语言模型的计算和服务成本显着降低。

Title: Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers

• Authors: Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, Philip S. Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04925
• Pdf URL: https://arxiv.org/pdf/2404.04925
• Copy Paste: [[2404.04925]] Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers(https://arxiv.org/abs/2404.04925)
• Keywords: language model, llm
• Abstract: Multilingual Large Language Models are capable of using powerful Large Language Models to handle and respond to queries in multiple languages, which achieves remarkable success in multilingual natural language processing tasks. Despite these breakthroughs, there still remains a lack of a comprehensive survey to summarize existing approaches and recent developments in this field. To this end, in this paper, we present a thorough review and provide a unified perspective to summarize the recent progress as well as emerging trends in multilingual large language models (MLLMs) literature. The contributions of this paper can be summarized: (1) First survey: to our knowledge, we take the first step and present a thorough review in MLLMs research field according to multi-lingual alignment; (2) New taxonomy: we offer a new and unified perspective to summarize the current progress of MLLMs; (3) New frontiers: we highlight several emerging frontiers and discuss the corresponding challenges; (4) Abundant resources: we collect abundant open-source resources, including relevant papers, data corpora, and leaderboards. We hope our work can provide the community with quick access and spur breakthrough research in MLLMs.
• 摘要：多语言大语言模型能够利用强大的大语言模型来处理和响应多种语言的查询，在多语言自然语言处理任务中取得了显着的成功。尽管取得了这些突破，但仍然缺乏全面的调查来总结该领域的现有方法和最新发展。为此，在本文中，我们对多语言大语言模型（MLLM）文献的最新进展和新兴趋势进行了全面的回顾并提供了统一的视角。本文的贡献可以概括为：（1）首次综述：据我们所知，我们迈出了第一步，根据多语言对齐对 MLLMs 研究领域进行了全面的回顾； （2）新的分类法：我们提供了一个新的、统一的视角来总结MLLM的当前进展； （3）新领域：我们重点介绍了几个新兴领域并讨论了相应的挑战； （4）丰富的资源：我们收集了丰富的开源资源，包括相关论文、数据语料、排行榜等。我们希望我们的工作能够为社区提供快速访问并促进 MLLM 的突破性研究。

Title: Towards Understanding the Influence of Reward Margin on Preference Model Performance

• Authors: Bowen Qin, Duanyu Feng, Xi Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04932
• Pdf URL: https://arxiv.org/pdf/2404.04932
• Copy Paste: [[2404.04932]] Towards Understanding the Influence of Reward Margin on Preference Model Performance(https://arxiv.org/abs/2404.04932)
• Keywords: language model
• Abstract: Reinforcement Learning from Human Feedback (RLHF) is a widely used framework for the training of language models. However, the process of using RLHF to develop a language model that is well-aligned presents challenges, especially when it comes to optimizing the reward model. Our research has found that existing reward models, when trained using the traditional ranking objective based on human preference data, often struggle to effectively distinguish between responses that are more or less favorable in real-world scenarios. To bridge this gap, our study introduces a novel method to estimate the preference differences without the need for detailed, exhaustive labels from human annotators. Our experimental results provide empirical evidence that incorporating margin values into the training process significantly improves the effectiveness of reward models. This comparative analysis not only demonstrates the superiority of our approach in terms of reward prediction accuracy but also highlights its effectiveness in practical applications.
• 摘要：来自人类反馈的强化学习（RLHF）是一种广泛使用的语言模型训练框架。然而，使用 RLHF 开发一致的语言模型的过程面临着挑战，特别是在优化奖励模型时。我们的研究发现，现有的奖励模型在使用基于人类偏好数据的传统排名目标进行训练时，通常很难有效区分在现实场景中或多或少有利的响应。为了弥补这一差距，我们的研究引入了一种新颖的方法来估计偏好差异，而不需要人类注释者提供详细、详尽的标签。我们的实验结果提供了经验证据，表明将保证金值纳入训练过程可以显着提高奖励模型的有效性。这种比较分析不仅证明了我们的方法在奖励预测准确性方面的优越性，而且凸显了其在实际应用中的有效性。

Title: Prompting Large Language Models for Zero-shot Essay Scoring via Multi-trait Specialization

• Authors: Sanwoo Lee, Yida Cai, Desong Meng, Ziyang Wang, Yunfang Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04941
• Pdf URL: https://arxiv.org/pdf/2404.04941
• Copy Paste: [[2404.04941]] Prompting Large Language Models for Zero-shot Essay Scoring via Multi-trait Specialization(https://arxiv.org/abs/2404.04941)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: Advances in automated essay scoring (AES) have traditionally relied on labeled essays, requiring tremendous cost and expertise for their acquisition. Recently, large language models (LLMs) have achieved great success in various tasks, but their potential is less explored in AES. In this paper, we propose Multi Trait Specialization (MTS), a zero-shot prompting framework to elicit essay scoring capabilities in LLMs. Specifically, we leverage ChatGPT to decompose writing proficiency into distinct traits and generate scoring criteria for each trait. Then, an LLM is prompted to extract trait scores from several conversational rounds, each round scoring one of the traits based on the scoring criteria. Finally, we derive the overall score via trait averaging and min-max scaling. Experimental results on two benchmark datasets demonstrate that MTS consistently outperforms straightforward prompting (Vanilla) in average QWK across all LLMs and datasets, with maximum gains of 0.437 on TOEFL11 and 0.355 on ASAP. Additionally, with the help of MTS, the small-sized Llama2-13b-chat substantially outperforms ChatGPT, facilitating an effective deployment in real applications.
• 摘要：自动论文评分 (AES) 的进步传统上依赖于带标签的论文，需要巨大的成本和专业知识来获取它们。最近，大型语言模型（LLM）在各种任务中取得了巨大成功，但它们的潜力在 AES 中却很少被开发。在本文中，我们提出了多特征专业化（MTS），这是一种零样本提示框架，用于激发法学硕士论文评分能力。具体来说，我们利用 ChatGPT 将写作能力分解为不同的特征，并为每个特征生成评分标准。然后，法学硕士被提示从几轮对话中提取特质分数，每一轮根据评分标准对其中一个特质进行评分。最后，我们通过特征平均和最小-最大缩放得出总体分数。两个基准数据集的实验结果表明，MTS 在所有 LLM 和数据集的平均 QWK 方面始终优于直接提示 (Vanilla)，TOEFL11 的最大增益为 0.437，ASAP 的最大增益为 0.355。此外，在MTS的帮助下，小型Llama2-13b-chat的性能大大优于ChatGPT，有助于在实际应用中有效部署。

Title: SilverSight: A Multi-Task Chinese Financial Large Language Model Based on Adaptive Semantic Space Learning

• Authors: Yuhang Zhou, Zeping Li, Siyu Tian, Yuchen Ni, Sen Liu, Guangnan Ye, Hongfeng Chai
• Subjects: cs.CL, cs.CE
• Abstract URL: https://arxiv.org/abs/2404.04949
• Pdf URL: https://arxiv.org/pdf/2404.04949
• Copy Paste: [[2404.04949]] SilverSight: A Multi-Task Chinese Financial Large Language Model Based on Adaptive Semantic Space Learning(https://arxiv.org/abs/2404.04949)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) are increasingly being applied across various specialized fields, leveraging their extensive knowledge to empower a multitude of scenarios within these domains. However, each field encompasses a variety of specific tasks that require learning, and the diverse, heterogeneous data across these domains can lead to conflicts during model task transfer. In response to this challenge, our study introduces an Adaptive Semantic Space Learning (ASSL) framework, which utilizes the adaptive reorganization of data distributions within the semantic space to enhance the performance and selection efficacy of multi-expert models. Utilizing this framework, we trained a financial multi-task LLM named "SilverSight". Our research findings demonstrate that our framework can achieve results close to those obtained with full data training using only 10% of the data, while also exhibiting strong generalization capabilities.
• 摘要：大型语言模型 (LLM) 越来越多地应用于各个专业领域，利用其丰富的知识来支持这些领域内的多种场景。然而，每个领域都包含各种需要学习的特定任务，而这些领域之间的多样化、异构数据可能会导致模型任务转移过程中的冲突。为了应对这一挑战，我们的研究引入了自适应语义空间学习（ASSL）框架，该框架利用语义空间内数据分布的自适应重组来增强多专家模型的性能和选择效率。利用这个框架，我们培训了一位名为“SilverSight”的金融多任务法学硕士。我们的研究结果表明，我们的框架仅使用 10% 的数据即可获得接近于完全数据训练所获得的结果，同时还表现出强大的泛化能力。

Title: SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials

• Authors: Mael Jullien, Marco Valentino, André Freitas
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.04963
• Pdf URL: https://arxiv.org/pdf/2404.04963
• Copy Paste: [[2404.04963]] SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials(https://arxiv.org/abs/2404.04963)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) are at the forefront of NLP achievements but fall short in dealing with shortcut learning, factual inconsistency, and vulnerability to adversarial inputs.These shortcomings are especially critical in medical contexts, where they can misrepresent actual model capabilities. Addressing this, we present SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for ClinicalTrials. Our contributions include the refined NLI4CT-P dataset (i.e., Natural Language Inference for Clinical Trials - Perturbed), designed to challenge LLMs with interventional and causal reasoning tasks, along with a comprehensive evaluation of methods and results for participant submissions. A total of 106 participants registered for the task contributing to over 1200 individual submissions and 25 system overview papers. This initiative aims to advance the robustness and applicability of NLI models in healthcare, ensuring safer and more dependable AI assistance in clinical decision-making. We anticipate that the dataset, models, and outcomes of this task can support future research in the field of biomedical NLI. The dataset, competition leaderboard, and website are publicly available.
• 摘要：大型语言模型 (LLM) 处于 NLP 成就的最前沿，但在处理快捷学习、事实不一致和对抗性输入的脆弱性方面存在不足。这些缺点在医学环境中尤其严重，它们可能会歪曲实际模型的功能。为了解决这个问题，我们提出了 SemEval-2024 任务 2：临床试验的安全生物医学自然语言推理。我们的贡献包括精炼的 NLI4CT-P 数据集（即临床试验的自然语言推理 - 扰动），旨在通过介入和因果推理任务挑战法学硕士，以及对参与者提交的方法和结果进行全面评估。共有 106 名参与者注册了该任务，贡献了 1200 多份个人提交内容和 25 篇系统概述论文。该举措旨在提高 NLI 模型在医疗保健领域的稳健性和适用性，确保在临床决策中提供更安全、更可靠的人工智能辅助。我们预计该任务的数据集、模型和结果可以支持生物医学 NLI 领域的未来研究。数据集、竞赛排行榜和网站都是公开的。

Title: MLaKE: Multilingual Knowledge Editing Benchmark for Large Language Models

• Authors: Zihao Wei, Jingcheng Deng, Liang Pang, Hanxing Ding, Huawei Shen, Xueqi Cheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.04990
• Pdf URL: https://arxiv.org/pdf/2404.04990
• Copy Paste: [[2404.04990]] MLaKE: Multilingual Knowledge Editing Benchmark for Large Language Models(https://arxiv.org/abs/2404.04990)
• Keywords: language model, llm
• Abstract: The extensive utilization of large language models (LLMs) underscores the crucial necessity for precise and contemporary knowledge embedded within their intrinsic parameters. Existing research on knowledge editing primarily concentrates on monolingual scenarios, neglecting the complexities presented by multilingual contexts and multi-hop reasoning. To address these challenges, our study introduces MLaKE (Multilingual Language Knowledge Editing), a novel benchmark comprising 4072 multi-hop and 5360 single-hop questions designed to evaluate the adaptability of knowledge editing methods across five languages: English, Chinese, Japanese, French, and German. MLaKE aggregates fact chains from Wikipedia across languages and utilizes LLMs to generate questions in both free-form and multiple-choice. We evaluate the multilingual knowledge editing generalization capabilities of existing methods on MLaKE. Existing knowledge editing methods demonstrate higher success rates in English samples compared to other languages. However, their generalization capabilities are limited in multi-language experiments. Notably, existing knowledge editing methods often show relatively high generalization for languages within the same language family compared to languages from different language families. These results underscore the imperative need for advancements in multilingual knowledge editing and we hope MLaKE can serve as a valuable resource for benchmarking and solution development.
• 摘要：大型语言模型（LLM）的广泛使用强调了将精确的当代知识嵌入其内在参数中的至关重要性。现有的知识编辑研究主要集中在单语言场景，忽略了多语言上下文和多跳推理所带来的复杂性。为了应对这些挑战，我们的研究引入了 MLaKE（多语言语言知识编辑），这是一种新颖的基准，包含 4072 个多跳和 5360 个单跳问题，旨在评估知识编辑方法跨五种语言的适应性：英语、中文、日语、法语和德语。 MLaKE 聚合来自维基百科的跨语言事实链，并利用法学硕士生成自由形式和多项选择的问题。我们评估 MLaKE 上现有方法的多语言知识编辑泛化能力。与其他语言相比，现有的知识编辑方法在英语样本中显示出更高的成功率。然而，它们的泛化能力在多语言实验中受到限制。值得注意的是，与不同语系的语言相比，现有的知识编辑方法通常对同一语系内的语言表现出相对较高的泛化性。这些结果强调了多语言知识编辑进步的迫切需要，我们希望 MLaKE 能够成为基准测试和解决方案开发的宝贵资源。

Title: How much reliable is ChatGPT's prediction on Information Extraction under Input Perturbations?

• Authors: Ishani Mondal, Abhilasha Sancheti
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05088
• Pdf URL: https://arxiv.org/pdf/2404.05088
• Copy Paste: [[2404.05088]] How much reliable is ChatGPT's prediction on Information Extraction under Input Perturbations?(https://arxiv.org/abs/2404.05088)
• Keywords: gpt, chat
• Abstract: In this paper, we assess the robustness (reliability) of ChatGPT under input perturbations for one of the most fundamental tasks of Information Extraction (IE) i.e. Named Entity Recognition (NER). Despite the hype, the majority of the researchers have vouched for its language understanding and generation capabilities; a little attention has been paid to understand its robustness: How the input-perturbations affect 1) the predictions, 2) the confidence of predictions and 3) the quality of rationale behind its prediction. We perform a systematic analysis of ChatGPT's robustness (under both zero-shot and few-shot setup) on two NER datasets using both automatic and human evaluation. Based on automatic evaluation metrics, we find that 1) ChatGPT is more brittle on Drug or Disease replacements (rare entities) compared to the perturbations on widely known Person or Location entities, 2) the quality of explanations for the same entity considerably differ under different types of "Entity-Specific" and "Context-Specific" perturbations and the quality can be significantly improved using in-context learning, and 3) it is overconfident for majority of the incorrect predictions, and hence it could lead to misguidance of the end-users.
• 摘要：在本文中，我们评估了 ChatGPT 在输入扰动下的鲁棒性（可靠性），以实现信息提取（IE）最基本的任务之一，即命名实体识别（NER）。尽管大肆宣传，但大多数研究人员都对其语言理解和生成能力表示认可。人们对理解其稳健性给予了一些关注：输入扰动如何影响 1) 预测，2) 预测的置信度和 3) 预测背后的基本原理的质量。我们使用自动和人工评估在两个 NER 数据集上对 ChatGPT 的鲁棒性（在零样本和少样本设置下）进行系统分析。基于自动评估指标，我们发现 1) 与广为人知的人员或位置实体上的扰动相比，ChatGPT 在药物或疾病替代品（罕见实体）上更脆弱，2) 在不同的情况下对同一实体的解释质量差异很大“特定于实体”和“特定于上下文”的扰动类型，并且使用上下文学习可以显着提高质量，3）对大多数不正确的预测过于自信，因此可能导致最终的误导-用户。

Title: Advancing Geometric Problem Solving: A Comprehensive Benchmark for Multimodal Model Evaluation

• Authors: Kai Sun, Yushi Bai, Nianyi Lin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05091
• Pdf URL: https://arxiv.org/pdf/2404.05091
• Copy Paste: [[2404.05091]] Advancing Geometric Problem Solving: A Comprehensive Benchmark for Multimodal Model Evaluation(https://arxiv.org/abs/2404.05091)
• Keywords: gpt
• Abstract: In this work, we present the MM-MATH dataset, a novel benchmark developed to rigorously evaluate the performance of advanced large language and multimodal models - including but not limited to GPT-4, GPT-4V, and Claude - within the domain of geometric computation. This dataset comprises 5,929 meticulously crafted geometric problems, each paired with a corresponding image, aimed at mirroring the complexity and requirements typical of ninth-grade mathematics. The motivation behind MM-MATH stems from the burgeoning interest and significant strides in multimodal technology, which necessitates a paradigm shift in assessment methodologies from mere outcome analysis to a more holistic evaluation encompassing reasoning and procedural correctness. Despite impressive gains in various benchmark performances, our analysis uncovers a persistent and notable deficiency in these models' ability to parse and interpret geometric information accurately from images, accounting for over 60% of observed errors. By deploying a dual-focused evaluation approach, examining both the end results and the underlying problem-solving processes, we unearthed a marked discrepancy between the capabilities of current multimodal models and human-level proficiency. The introduction of MM-MATH represents a tripartite contribution to the field: it not only serves as a comprehensive and challenging benchmark for assessing geometric problem-solving prowess but also illuminates critical gaps in textual and visual comprehension that current models exhibit. Through this endeavor, we aspire to catalyze further research and development aimed at bridging these gaps, thereby advancing the state of multimodal model capabilities to new heights.
• 摘要：在这项工作中，我们提出了 MM-MATH 数据集，这是一个新颖的基准，旨在严格评估几何领域内高级大语言和多模态模型（包括但不限于 GPT-4、GPT-4V 和 Claude）的性能计算。该数据集包含 5,929 个精心设计的几何问题，每个问题都配有相应的图像，旨在反映九年级数学的典型复杂性和要求。 MM-MATH 背后的动机源于多模式技术的新兴兴趣和重大进步，这需要评估方法的范式转变，从单纯的结果分析转向包含推理和程序正确性的更全面的评估。尽管在各种基准性能方面取得了令人印象深刻的进步，但我们的分析发现，这些模型在准确解析和解释图像中的几何信息的能力方面存在持续且显着的缺陷，占观察到的错误的 60% 以上。通过部署双重重点评估方法，检查最终结果和潜在的问题解决过程，我们发现当前多模式模型的能力与人类水平的熟练程度之间存在显着差异。 MM-MATH 的引入代表了对该领域的三方贡献：它不仅作为评估几何问题解决能力的全面且具有挑战性的基准，而且还阐明了当前模型在文本和视觉理解方面的关键差距。通过这一努力，我们渴望促进旨在弥合这些差距的进一步研究和开发，从而将多式联运模型能力的状态提升到新的高度。

Title: Plug and Play with Prompts: A Prompt Tuning Approach for Controlling Text Generation

• Authors: Rohan Deepak Ajwani, Zining Zhu, Jonathan Rose, Frank Rudzicz
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05143
• Pdf URL: https://arxiv.org/pdf/2404.05143
• Copy Paste: [[2404.05143]] Plug and Play with Prompts: A Prompt Tuning Approach for Controlling Text Generation(https://arxiv.org/abs/2404.05143)
• Keywords: language model, llm, prompt
• Abstract: Transformer-based Large Language Models (LLMs) have shown exceptional language generation capabilities in response to text-based prompts. However, controlling the direction of generation via textual prompts has been challenging, especially with smaller models. In this work, we explore the use of Prompt Tuning to achieve controlled language generation. Generated text is steered using prompt embeddings, which are trained using a small language model, used as a discriminator. Moreover, we demonstrate that these prompt embeddings can be trained with a very small dataset, with as low as a few hundred training examples. Our method thus offers a data and parameter efficient solution towards controlling language model outputs. We carry out extensive evaluation on four datasets: SST-5 and Yelp (sentiment analysis), GYAFC (formality) and JIGSAW (toxic language). Finally, we demonstrate the efficacy of our method towards mitigating harmful, toxic, and biased text generated by language models.
• 摘要：基于 Transformer 的大型语言模型 (LLM) 在响应基于文本的提示时表现出了卓越的语言生成能力。然而，通过文本提示控制生成方向一直具有挑战性，尤其是对于较小的模型。在这项工作中，我们探索使用 Prompt Tuning 来实现受控语言生成。生成的文本使用提示嵌入进行引导，提示嵌入使用小型语言模型进行训练，用作鉴别器。此外，我们证明这些提示嵌入可以使用非常小的数据集进行训练，训练样本少至几百个。因此，我们的方法为控制语言模型输出提供了数据和参数有效的解决方案。我们对四个数据集进行了广泛的评估：SST-5 和 Yelp（情感分析）、GYAFC（形式）和 JIGSAW（有毒语言）。最后，我们展示了我们的方法在减轻语言模型生成的有害、有毒和有偏见的文本方面的有效性。

Title: Enhancing Clinical Efficiency through LLM: Discharge Note Generation for Cardiac Patients

• Authors: HyoJe Jung, Yunha Kim, Heejung Choi, Hyeram Seo, Minkyoung Kim, JiYe Han, Gaeun Kee, Seohyun Park, Soyoung Ko, Byeolhee Kim, Suyeon Kim, Tae Joon Jun, Young-Hak Kim
• Subjects: cs.CL, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05144
• Pdf URL: https://arxiv.org/pdf/2404.05144
• Copy Paste: [[2404.05144]] Enhancing Clinical Efficiency through LLM: Discharge Note Generation for Cardiac Patients(https://arxiv.org/abs/2404.05144)
• Keywords: language model, llm
• Abstract: Medical documentation, including discharge notes, is crucial for ensuring patient care quality, continuity, and effective medical communication. However, the manual creation of these documents is not only time-consuming but also prone to inconsistencies and potential errors. The automation of this documentation process using artificial intelligence (AI) represents a promising area of innovation in healthcare. This study directly addresses the inefficiencies and inaccuracies in creating discharge notes manually, particularly for cardiac patients, by employing AI techniques, specifically large language model (LLM). Utilizing a substantial dataset from a cardiology center, encompassing wide-ranging medical records and physician assessments, our research evaluates the capability of LLM to enhance the documentation process. Among the various models assessed, Mistral-7B distinguished itself by accurately generating discharge notes that significantly improve both documentation efficiency and the continuity of care for patients. These notes underwent rigorous qualitative evaluation by medical expert, receiving high marks for their clinical relevance, completeness, readability, and contribution to informed decision-making and care planning. Coupled with quantitative analyses, these results confirm Mistral-7B's efficacy in distilling complex medical information into concise, coherent summaries. Overall, our findings illuminate the considerable promise of specialized LLM, such as Mistral-7B, in refining healthcare documentation workflows and advancing patient care. This study lays the groundwork for further integrating advanced AI technologies in healthcare, demonstrating their potential to revolutionize patient documentation and support better care outcomes.
• 摘要：包括出院记录在内的医疗文件对于确保患者护理质量、连续性和有效的医疗沟通至关重要。然而，手动创建这些文档不仅耗时，而且容易出现不一致和潜在错误。使用人工智能 (AI) 实现文档流程自动化代表了医疗保健领域一个充满希望的创新领域。这项研究通过采用人工智能技术，特别是大语言模型（LLM），直接解决了手动创建出院记录的低效和不准确问题，特别是对于心脏病患者。我们的研究利用心脏病学中心的大量数据集（包括广泛的医疗记录和医生评估）评估了法学硕士增强记录流程的能力。在评估的各种模型中，Mistral-7B 因准确生成出院记录而脱颖而出，显着提高了记录效率和患者护理的连续性。这些笔记经过了医学专家严格的定性评估，因其临床相关性、完整性、可读性以及对知情决策和护理计划的贡献而获得了高度评价。结合定量分析，这些结果证实了 Mistral-7B 在将复杂的医疗信息提炼成简洁、连贯的摘要方面的功效。总体而言，我们的研究结果阐明了 Mistral-7B 等专业法学硕士在完善医疗保健文档工作流程和推进患者护理方面的巨大前景。这项研究为进一步将先进的人工智能技术融入医疗保健奠定了基础，展示了它们彻底改变患者记录并支持更好的护理结果的潜力。

Title: Linguistic Changes in Spontaneous Speech for Detecting Parkinsons Disease Using Large Language Models

• Authors: Jonathan Crawford
• Subjects: cs.CL, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.05160
• Pdf URL: https://arxiv.org/pdf/2404.05160
• Copy Paste: [[2404.05160]] Linguistic Changes in Spontaneous Speech for Detecting Parkinsons Disease Using Large Language Models(https://arxiv.org/abs/2404.05160)
• Keywords: language model
• Abstract: Parkinsons disease is the second most prevalent neurodegenerative disorder with over ten million active cases worldwide and one million new diagnoses per year. Detecting and subsequently diagnosing the disease is challenging because of symptom heterogeneity with respect to complexity, as well as the type and timing of phenotypic manifestations. Typically, language impairment can present in the prodromal phase and precede motor symptoms suggesting that a linguistic-based approach could serve as a diagnostic method for incipient Parkinsons disease. Additionally, improved linguistic models may enhance other approaches through ensemble techniques. The field of large language models is advancing rapidly, presenting the opportunity to explore the use of these new models for detecting Parkinsons disease and to improve on current linguistic approaches with high-dimensional representations of linguistics. We evaluate the application of state-of-the-art large language models to detect Parkinsons disease automatically from spontaneous speech with up to 73% accuracy.
• 摘要：帕金森病是第二常见的神经退行性疾病，全球有超过一千万活跃病例，每年新诊断一百万例。由于症状在复杂性方面的异质性以及表型表现的类型和时间，检测和随后诊断该疾病具有挑战性。通常，语言障碍可能出现在前驱期并出现在运动症状之前，这表明基于语言的方法可以作为早期帕金森病的诊断方法。此外，改进的语言模型可以通过集成技术增强其他方法。大语言模型领域正在迅速发展，为探索使用这些新模型来检测帕金森病以及通过语言学的高维表示来改进当前的语言方法提供了机会。我们评估了最先进的大型语言模型从自发语音中自动检测帕金森病的应用，准确率高达 73%。

Title: LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models

• Authors: Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, Xinyuan Wang, Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, Zhen Wang, Zhiting Hu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05221
• Pdf URL: https://arxiv.org/pdf/2404.05221
• Copy Paste: [[2404.05221]] LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models(https://arxiv.org/abs/2404.05221)
• Keywords: language model, gpt, llm, prompt
• Abstract: Generating accurate step-by-step reasoning is essential for Large Language Models (LLMs) to address complex problems and enhance robustness and interpretability. Despite the flux of research on developing advanced reasoning approaches, systematically analyzing the diverse LLMs and reasoning strategies in generating reasoning chains remains a significant challenge. The difficulties stem from the lack of two key elements: (1) an automatic method for evaluating the generated reasoning chains on different tasks, and (2) a unified formalism and implementation of the diverse reasoning approaches for systematic comparison. This paper aims to close the gap: (1) We introduce AutoRace for fully automated reasoning chain evaluation. Existing metrics rely on expensive human annotations or pre-defined LLM prompts not adaptable to different tasks. In contrast, AutoRace automatically creates detailed evaluation criteria tailored for each task, and uses GPT-4 for accurate evaluation following the criteria. (2) We develop LLM Reasoners, a library for standardized modular implementation of existing and new reasoning algorithms, under a unified formulation of the search, reward, and world model components. With the new evaluation and library, (3) we conduct extensive study of different reasoning approaches (e.g., CoT, ToT, RAP). The analysis reveals interesting findings about different factors contributing to reasoning, including the reward-guidance, breadth-vs-depth in search, world model, and prompt formats, etc.
• 摘要：生成准确的逐步推理对于大型语言模型 (LLM) 解决复杂问题并增强鲁棒性和可解释性至关重要。尽管开发高级推理方法的研究不断涌现，但系统地分析各种法学硕士和生成推理链的推理策略仍然是一个重大挑战。困难源于缺乏两个关键要素：（1）评估不同任务上生成的推理链的自动方法，以及（2）统一的形式主义和实现系统比较的不同推理方法。本文旨在缩小差距：（1）我们引入 AutoRace 进行全自动推理链评估。现有指标依赖于昂贵的人工注释或预定义的 LLM 提示，无法适应不同的任务。相比之下，AutoRace会自动为每个任务创建详细的评估标准，并使用GPT-4按照标准进行准确评估。 (2) 我们开发了 LLM Reasoners，这是一个用于现有和新推理算法的标准化模块化实现的库，在搜索、奖励和世界模型组件的统一制定下。通过新的评估和库，(3)我们对不同的推理方法（例如，CoT、ToT、RAP）进行了广泛的研究。分析揭示了有关推理不同因素的有趣发现，包括奖励指导、搜索的广度与深度、世界模型和提示格式等。

Title: Product Description and QA Assisted Self-Supervised Opinion Summarization

• Authors: Tejpalsingh Siledar, Rupasai Rangaraju, Sankara Sri Raghava Ravindra Muddu, Suman Banerjee, Amey Patil, Sudhanshu Shekhar Singh, Muthusamy Chelliah, Nikesh Garera, Swaprava Nath, Pushpak Bhattacharyya
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05243
• Pdf URL: https://arxiv.org/pdf/2404.05243
• Copy Paste: [[2404.05243]] Product Description and QA Assisted Self-Supervised Opinion Summarization(https://arxiv.org/abs/2404.05243)
• Keywords: gpt, chat
• Abstract: In e-commerce, opinion summarization is the process of summarizing the consensus opinions found in product reviews. However, the potential of additional sources such as product description and question-answers (QA) has been considered less often. Moreover, the absence of any supervised training data makes this task challenging. To address this, we propose a novel synthetic dataset creation (SDC) strategy that leverages information from reviews as well as additional sources for selecting one of the reviews as a pseudo-summary to enable supervised training. Our Multi-Encoder Decoder framework for Opinion Summarization (MEDOS) employs a separate encoder for each source, enabling effective selection of information while generating the summary. For evaluation, due to the unavailability of test sets with additional sources, we extend the Amazon, Oposum+, and Flipkart test sets and leverage ChatGPT to annotate summaries. Experiments across nine test sets demonstrate that the combination of our SDC approach and MEDOS model achieves on average a 14.5% improvement in ROUGE-1 F1 over the SOTA. Moreover, comparative analysis underlines the significance of incorporating additional sources for generating more informative summaries. Human evaluations further indicate that MEDOS scores relatively higher in coherence and fluency with 0.41 and 0.5 (-1 to 1) respectively, compared to existing models. To the best of our knowledge, we are the first to generate opinion summaries leveraging additional sources in a self-supervised setting.
• 摘要：在电子商务中，意见总结是对产品评论中发现的共识意见进行总结的过程。然而，产品描述和问题解答 (QA) 等其他来源的潜力却很少被考虑。此外，缺乏任何监督训练数据使得这项任务具有挑战性。为了解决这个问题，我们提出了一种新颖的合成数据集创建（SDC）策略，该策略利用评论中的信息以及其他来源来选择其中一条评论作为伪摘要以实现监督训练。我们的意见摘要多编码器解码器框架 (MEDOS) 对每个源采用单独的编码器，从而在生成摘要时能够有效选择信息。为了进行评估，由于无法获得具有其他来源的测试集，我们扩展了 Amazon、Oposum+ 和 Flipkart 测试集，并利用 ChatGPT 来注释摘要。九个测试集的实验表明，我们的 SDC 方法和 MEDOS 模型的结合使 ROUGE-1 F1 比 SOTA 平均提高了 14.5%。此外，比较分析强调了纳入其他来源以生成更多信息摘要的重要性。人类评估进一步表明，与现有模型相比，MEDOS 在连贯性和流畅性方面得分相对较高，分别为 0.41 和 0.5（-1 到 1）。据我们所知，我们是第一个在自我监督的环境中利用其他来源生成意见摘要的人。

Title: PORTULAN ExtraGLUE Datasets and Models: Kick-starting a Benchmark for the Neural Processing of Portuguese

• Authors: Tomás Osório, Bernardo Leite, Henrique Lopes Cardoso, Luís Gomes, João Rodrigues, Rodrigo Santos, António Branco
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05333
• Pdf URL: https://arxiv.org/pdf/2404.05333
• Copy Paste: [[2404.05333]] PORTULAN ExtraGLUE Datasets and Models: Kick-starting a Benchmark for the Neural Processing of Portuguese(https://arxiv.org/abs/2404.05333)
• Keywords: language model
• Abstract: Leveraging research on the neural modelling of Portuguese, we contribute a collection of datasets for an array of language processing tasks and a corresponding collection of fine-tuned neural language models on these downstream tasks. To align with mainstream benchmarks in the literature, originally developed in English, and to kick start their Portuguese counterparts, the datasets were machine-translated from English with a state-of-the-art translation engine. The resulting PORTULAN ExtraGLUE benchmark is a basis for research on Portuguese whose improvement can be pursued in future work. Similarly, the respective fine-tuned neural language models, developed with a low-rank adaptation approach, are made available as baselines that can stimulate future work on the neural processing of Portuguese. All datasets and models have been developed and are made available for two variants of Portuguese: European and Brazilian.
• 摘要：利用对葡萄牙语神经建模的研究，我们为一系列语言处理任务提供了一组数据集，以及针对这些下游任务的相应微调神经语言模型集合。为了与最初用英语开发的文献中的主流基准保持一致，并启动葡萄牙语同行，这些数据集是使用最先进的翻译引擎从英语机器翻译的。由此产生的 PORTULAN ExtraGLUE 基准是葡萄牙语研究的基础，可以在未来的工作中进行改进。同样，采用低阶适应方法开发的相应微调神经语言模型可作为基线，可以刺激未来葡萄牙语神经处理的工作。所有数据集和模型均已开发完毕，并可用于葡萄牙语的两种变体：欧洲葡萄牙语和巴西葡萄牙语。

Title: Towards Objectively Benchmarking Social Intelligence for Language Agents at Action Level

• Authors: Chenxu Wang, Bin Dai, Huaping Liu, Baoyuan Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05337
• Pdf URL: https://arxiv.org/pdf/2404.05337
• Copy Paste: [[2404.05337]] Towards Objectively Benchmarking Social Intelligence for Language Agents at Action Level(https://arxiv.org/abs/2404.05337)
• Keywords: language model, agent
• Abstract: Prominent large language models have exhibited human-level performance in many domains, even enabling the derived agents to simulate human and social interactions. While practical works have substantiated the practicability of grounding language agents in sandbox simulation or embodied simulators, current social intelligence benchmarks either stay at the language level or use subjective metrics. In pursuit of a more realistic and objective evaluation, we introduce the Social Tasks in Sandbox Simulation (STSS) benchmark, which assesses language agents \textbf{objectively} at the \textbf{action level} by scrutinizing the goal achievements within the multi-agent simulation. Additionally, we sample conversation scenarios to build a language-level benchmark to provide an economically prudent preliminary evaluation and align with prevailing benchmarks. To gauge the significance of agent architecture, we implement a target-driven planning (TDP) module as an adjunct to the existing agent. Our evaluative findings highlight that the STSS benchmark is challenging for state-of-the-art language agents. Furthermore, it effectively discriminates between distinct language agents, suggesting its usefulness as a benchmark for evaluating both language models and agent architectures.
• 摘要：著名的大型语言模型在许多领域都表现出了人类水平的性能，甚至使派生的代理能够模拟人类和社会互动。虽然实际工作已经证实了将语言智能体置于沙箱模拟或实体模拟器中的实用性，但当前的社会智能基准要么停留在语言层面，要么使用主观指标。为了追求更现实和客观的评估，我们引入了沙盒模拟中的社交任务（STSS）基准，该基准通过审查多智能体中的目标实现情况，在 \textbf{action level} 上评估语言智能体 \textbf{objectively}模拟。此外，我们对对话场景进行采样，以构建语言级基准，以提供经济上审慎的初步评估，并与现行基准保持一致。为了衡量代理架构的重要性，我们实现了目标驱动规划（TDP）模块作为现有代理的辅助。我们的评估结果强调，STSS 基准对于最先进的语言代理来说具有挑战性。此外，它可以有效地区分不同的语言代理，这表明它可以作为评估语言模型和代理架构的基准。

Title: SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety

• Authors: Paul Röttger, Fabio Pernisi, Bertie Vidgen, Dirk Hovy
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05399
• Pdf URL: https://arxiv.org/pdf/2404.05399
• Copy Paste: [[2404.05399]] SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety(https://arxiv.org/abs/2404.05399)
• Keywords: language model, llm, prompt
• Abstract: The last two years have seen a rapid growth in concerns around the safety of large language models (LLMs). Researchers and practitioners have met these concerns by introducing an abundance of new datasets for evaluating and improving LLM safety. However, much of this work has happened in parallel, and with very different goals in mind, ranging from the mitigation of near-term risks around bias and toxic content generation to the assessment of longer-term catastrophic risk potential. This makes it difficult for researchers and practitioners to find the most relevant datasets for a given use case, and to identify gaps in dataset coverage that future work may fill. To remedy these issues, we conduct a first systematic review of open datasets for evaluating and improving LLM safety. We review 102 datasets, which we identified through an iterative and community-driven process over the course of several months. We highlight patterns and trends, such as a a trend towards fully synthetic datasets, as well as gaps in dataset coverage, such as a clear lack of non-English datasets. We also examine how LLM safety datasets are used in practice -- in LLM release publications and popular LLM benchmarks -- finding that current evaluation practices are highly idiosyncratic and make use of only a small fraction of available datasets. Our contributions are based on SafetyPrompts.com, a living catalogue of open datasets for LLM safety, which we commit to updating continuously as the field of LLM safety develops.
• 摘要：过去两年，人们对大型语言模型 (LLM) 安全性的担忧迅速增长。研究人员和从业者通过引入大量新数据集来评估和提高法学硕士的安全性，从而解决了这些问题。然而，这项工作的大部分都是并行进行的，并且有着截然不同的目标，从缓解偏见和有毒内容产生的近期风险到评估长期灾难性潜在风险。这使得研究人员和从业者很难找到与给定用例最相关的数据集，也很难确定未来工作可能填补的数据集覆盖范围的空白。为了解决这些问题，我们对开放数据集进行了首次系统审查，以评估和提高法学硕士的安全性。我们审查了 102 个数据集，这些数据集是我们在几个月的时间里通过社区驱动的迭代过程确定的。我们强调模式和趋势，例如完全合成数据集的趋势，以及数据集覆盖范围的差距，例如明显缺乏非英语数据集。我们还研究了 LLM 安全数据集在实践中的使用方式——在 LLM 发布出版物和流行的 LLM 基准中——发现当前的评估实践非常特殊，并且只使用了一小部分可用数据集。我们的贡献基于 SafetyPrompts.com，这是一个 LLM 安全开放数据集的实时目录，我们致力于随着 LLM 安全领域的发展不断更新。

Title: Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws

• Authors: Zeyuan Allen-Zhu, Yuanzhi Li
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05405
• Pdf URL: https://arxiv.org/pdf/2404.05405
• Copy Paste: [[2404.05405]] Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws(https://arxiv.org/abs/2404.05405)
• Keywords: language model, gpt
• Abstract: Scaling laws describe the relationship between the size of language models and their capabilities. Unlike prior studies that evaluate a model's capability via loss or benchmarks, we estimate the number of knowledge bits a model stores. We focus on factual knowledge represented as tuples, such as (USA, capital, Washington D.C.) from a Wikipedia page. Through multiple controlled datasets, we establish that language models can and only can store 2 bits of knowledge per parameter, even when quantized to int8, and such knowledge can be flexibly extracted for downstream applications. Consequently, a 7B model can store 14B bits of knowledge, surpassing the English Wikipedia and textbooks combined based on our estimation. More broadly, we present 12 results on how (1) training duration, (2) model architecture, (3) quantization, (4) sparsity constraints such as MoE, and (5) data signal-to-noise ratio affect a model's knowledge storage capacity. Notable insights include: * The GPT-2 architecture, with rotary embedding, matches or even surpasses LLaMA/Mistral architectures in knowledge storage, particularly over shorter training durations. This arises because LLaMA/Mistral uses GatedMLP, which is less stable and harder to train. * Prepending training data with domain names (e.g., wikipedia.org) significantly increases a model's knowledge capacity. Language models can autonomously identify and prioritize domains rich in knowledge, optimizing their storage capacity.
• 摘要：缩放定律描述了语言模型的大小与其能力之间的关系。与之前通过损失或基准评估模型能力的研究不同，我们估计模型存储的知识位数。我们专注于以元组表示的事实知识，例如维基百科页面中的（美国、首都、华盛顿特区）。通过多个受控数据集，我们确定语言模型每个参数只能存储 2 位知识，即使量化为 int8 也是如此，并且可以灵活地提取这些知识以供下游应用程序使用。因此，7B 模型可以存储 14B 位知识，超过了我们估计的英文维基百科和教科书的总和。更广泛地说，我们提出了 12 个结果，涉及 (1) 训练持续时间、(2) 模型架构、(3) 量化、(4) 稀疏约束（例如 MoE）以及 (5) 数据信噪比如何影响模型的知识存储容量。值得注意的见解包括： * 具有旋转嵌入功能的 GPT-2 架构在知识存储方面匹配甚至超越了 LLaMA/Mistral 架构，特别是在较短的训练持续时间内。出现这种情况是因为 LLaMA/Mistral 使用 GatedMLP，它不太稳定且难以训练。 * 在训练数据前添加域名（例如 wikipedia.org）可显着提高模型的知识容量。语言模型可以自主识别知识丰富的领域并对其进行优先级排序，从而优化其存储容量。

Title: PerkwE_COQA: enhance Persian Conversational Question Answering by combining contextual keyword extraction with Large Language Models

• Authors: Pardis Moradbeiki, Nasser Ghadiri
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05406
• Pdf URL: https://arxiv.org/pdf/2404.05406
• Copy Paste: [[2404.05406]] PerkwE_COQA: enhance Persian Conversational Question Answering by combining contextual keyword extraction with Large Language Models(https://arxiv.org/abs/2404.05406)
• Keywords: language model, llm
• Abstract: Smart cities need the involvement of their residents to enhance quality of life. Conversational query-answering is an emerging approach for user engagement. There is an increasing demand of an advanced conversational question-answering that goes beyond classic systems. Existing approaches have shown that LLMs offer promising capabilities for CQA, but may struggle to capture the nuances of conversational contexts. The new approach involves understanding the content and engaging in a multi-step conversation with the user to fulfill their needs. This paper presents a novel method to elevate the performance of Persian Conversational question-answering (CQA) systems. It combines the strengths of Large Language Models (LLMs) with contextual keyword extraction. Our method extracts keywords specific to the conversational flow, providing the LLM with additional context to understand the user's intent and generate more relevant and coherent responses. We evaluated the effectiveness of this combined approach through various metrics, demonstrating significant improvements in CQA performance compared to an LLM-only baseline. The proposed method effectively handles implicit questions, delivers contextually relevant answers, and tackles complex questions that rely heavily on conversational context. The findings indicate that our method outperformed the evaluation benchmarks up to 8% higher than existing methods and the LLM-only baseline.
• 摘要：智慧城市需要居民的参与来提高生活质量。对话式查询回答是一种新兴的用户参与方法。人们对超越经典系统的高级会话问答的需求日益增长。现有方法表明，法学硕士为 CQA 提供了很有前途的功能，但可能难以捕捉对话上下文的细微差别。新方法涉及理解内容并与用户进行多步骤对话以满足他们的需求。本文提出了一种提高波斯语会话问答（CQA）系统性能的新方法。它将大型语言模型 (LLM) 的优势与上下文关键字提取相结合。我们的方法提取特定于对话流的关键字，为法学硕士提供额外的上下文来理解用户的意图并生成更相关和连贯的响应。我们通过各种指标评估了这种组合方法的有效性，证明与仅法学硕士基线相比，CQA 性能有了显着改善。所提出的方法可以有效地处理隐含的问题，提供上下文相关的答案，并解决严重依赖于对话上下文的复杂问题。研究结果表明，我们的方法优于评估基准，比现有方法和仅 LLM 基线高出 8%。

Title: Know When To Stop: A Study of Semantic Drift in Text Generation

• Authors: Ava Spataru, Eric Hambro, Elena Voita, Nicola Cancedda
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05411
• Pdf URL: https://arxiv.org/pdf/2404.05411
• Copy Paste: [[2404.05411]] Know When To Stop: A Study of Semantic Drift in Text Generation(https://arxiv.org/abs/2404.05411)
• Keywords: llm
• Abstract: In this work, we explicitly show that modern LLMs tend to generate correct facts first, then "drift away" and generate incorrect facts later: this was occasionally observed but never properly measured. We develop a semantic drift score that measures the degree of separation between correct and incorrect facts in generated texts and confirm our hypothesis when generating Wikipedia-style biographies. This correct-then-incorrect generation pattern suggests that factual accuracy can be improved by knowing when to stop generation. Therefore, we explore the trade-off between information quantity and factual accuracy for several early stopping methods and manage to improve factuality by a large margin. We further show that reranking with semantic similarity can further improve these results, both compared to the baseline and when combined with early stopping. Finally, we try calling external API to bring the model back to the right generation path, but do not get positive results. Overall, our methods generalize and can be applied to any long-form text generation to produce more reliable information, by balancing trade-offs between factual accuracy, information quantity and computational cost.
• 摘要：在这项工作中，我们明确表明，现代法学硕士倾向于首先生成正确的事实，然后“偏离”并随后生成错误的事实：偶尔会观察到这种情况，但从未正确测量过。我们开发了一个语义漂移分数，用于衡量生成文本中正确和不正确事实之间的分离程度，并在生成维基百科风格的传记时证实我们的假设。这种先正确后错误的生成模式表明，通过了解何时停止生成可以提高事实准确性。因此，我们探索了几种早期停止方法的信息量和事实准确性之间的权衡，并设法大幅提高事实性。我们进一步表明，与基线相比以及与提前停止相结合时，具有语义相似性的重新排名可以进一步改善这些结果。最后，我们尝试调用外部 API 让模型回到正确的生成路径，但没有得到积极的结果。总的来说，我们的方法具有概括性，可以通过平衡事实准确性、信息量和计算成本之间的权衡，应用于任何长格式文本生成，以产生更可靠的信息。

Title: Relation Extraction Using Large Language Models: A Case Study on Acupuncture Point Locations

• Authors: Yiming Li, Xueqing Peng, Jianfu Li, Xu Zuo, Suyuan Peng, Donghong Pei, Cui Tao, Hua Xu, Na Hong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05415
• Pdf URL: https://arxiv.org/pdf/2404.05415
• Copy Paste: [[2404.05415]] Relation Extraction Using Large Language Models: A Case Study on Acupuncture Point Locations(https://arxiv.org/abs/2404.05415)
• Keywords: language model, gpt, llm
• Abstract: In acupuncture therapy, the accurate location of acupoints is essential for its effectiveness. The advanced language understanding capabilities of large language models (LLMs) like Generative Pre-trained Transformers (GPT) present a significant opportunity for extracting relations related to acupoint locations from textual knowledge sources. This study aims to compare the performance of GPT with traditional deep learning models (Long Short-Term Memory (LSTM) and Bidirectional Encoder Representations from Transformers for Biomedical Text Mining (BioBERT)) in extracting acupoint-related location relations and assess the impact of pretraining and fine-tuning on GPT's performance. We utilized the World Health Organization Standard Acupuncture Point Locations in the Western Pacific Region (WHO Standard) as our corpus, which consists of descriptions of 361 acupoints. Five types of relations ('direction_of,' 'distance_of,' 'part_of,' 'near_acupoint,' and 'located_near') (n= 3,174) between acupoints were annotated. Five models were compared: BioBERT, LSTM, pre-trained GPT-3.5, and fine-tuned GPT-3.5, as well as pre-trained GPT-4. Performance metrics included micro-average exact match precision, recall, and F1 scores. Our results demonstrate that fine-tuned GPT-3.5 consistently outperformed other models in F1 scores across all relation types. Overall, it achieved the highest micro-average F1 score of 0.92. This study underscores the effectiveness of LLMs like GPT in extracting relations related to acupoint locations, with implications for accurately modeling acupuncture knowledge and promoting standard implementation in acupuncture training and practice. The findings also contribute to advancing informatics applications in traditional and complementary medicine, showcasing the potential of LLMs in natural language processing.
• 摘要：在针灸治疗中，穴位的准确定位对于其疗效至关重要。生成式预训练变压器 (GPT) 等大型语言模型 (LLM) 的高级语言理解能力为从文本知识源中提取与穴位位置相关的关系提供了重要机会。本研究旨在比较 GPT 与传统深度学习模型（长短期记忆（LSTM）和用于生物医学文本挖掘的 Transformers 的双向编码器表示（BioBERT））在提取穴位相关位置关系方面的性能，并评估预训练的影响并对 GPT 的性能进行微调。我们使用世界卫生组织西太平洋地区标准穴位位置（WHO 标准）作为我们的语料库，其中包含 361 个穴位的描述。标注了穴位之间的五种关系（“direction_of”、“distance_of”、“part_of”、“near_acupoint”和“ located_near”）（n= 3,174）。比较了五种模型：BioBERT、LSTM、预训练的 GPT-3.5、微调的 GPT-3.5 以及预训练的 GPT-4。性能指标包括微平均精确匹配精度、召回率和 F1 分数。我们的结果表明，经过微调的 GPT-3.5 在所有关系类型的 F1 分数上始终优于其他模型。总体而言，它取得了最高的微平均 F1 分数 0.92。这项研究强调了 GPT 等法学硕士在提取与穴位位置相关的关系方面的有效性，这对于准确建模针灸知识和促进针灸培训和实践的标准实施具有重要意义。这些发现还有助于推进传统医学和补充医学中的信息学应用，展示了法学硕士在自然语言处理方面的潜力。

Title: Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretraining

• Authors: Nikola Ljubešić, Vít Suchomel, Peter Rupnik, Taja Kuzman, Rik van Noord
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05428
• Pdf URL: https://arxiv.org/pdf/2404.05428
• Copy Paste: [[2404.05428]] Language Models on a Diet: Cost-Efficient Development of Encoders for Closely-Related Languages via Additional Pretraining(https://arxiv.org/abs/2404.05428)
• Keywords: language model
• Abstract: The world of language models is going through turbulent times, better and ever larger models are coming out at an unprecedented speed. However, we argue that, especially for the scientific community, encoder models of up to 1 billion parameters are still very much needed, their primary usage being in enriching large collections of data with metadata necessary for downstream research. We investigate the best way to ensure the existence of such encoder models on the set of very closely related languages - Croatian, Serbian, Bosnian and Montenegrin, by setting up a diverse benchmark for these languages, and comparing the trained-from-scratch models with the new models constructed via additional pretraining of existing multilingual models. We show that comparable performance to dedicated from-scratch models can be obtained by additionally pretraining available multilingual models even with a limited amount of computation. We also show that neighboring languages, in our case Slovenian, can be included in the additional pretraining with little to no loss in the performance of the final model.
• 摘要：语言模型的世界正在经历动荡的时期，更好、更大的模型正在以前所未有的速度问世。然而，我们认为，特别是对于科学界来说，仍然非常需要多达 10 亿个参数的编码器模型，它们的主要用途是用下游研究所需的元数据来丰富大量数据集。我们通过为这些语言设置不同的基准，并将从头开始训练的模型与通过对现有多语言模型进行额外的预训练构建的新模型。我们表明，即使计算量有限，也可以通过额外预训练可用的多语言模型来获得与专用从头开始模型相当的性能。我们还表明，邻近语言（在我们的例子中是斯洛文尼亚语）可以包含在额外的预训练中，而最终模型的性能几乎没有损失。

Title: XL$^2$Bench: A Benchmark for Extremely Long Context Understanding with Long-range Dependencies

• Authors: Xuanfan Ni, Hengyi Cai, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Piji Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05446
• Pdf URL: https://arxiv.org/pdf/2404.05446
• Copy Paste: [[2404.05446]] XL$^2$Bench: A Benchmark for Extremely Long Context Understanding with Long-range Dependencies(https://arxiv.org/abs/2404.05446)
• Keywords: language model, llm, long context
• Abstract: Large Language Models (LLMs) have demonstrated remarkable performance across diverse tasks but are constrained by their small context window sizes. Various efforts have been proposed to expand the context window to accommodate even up to 200K input tokens. Meanwhile, building high-quality benchmarks with much longer text lengths and more demanding tasks to provide comprehensive evaluations is of immense practical interest to facilitate long context understanding research of LLMs. However, prior benchmarks create datasets that ostensibly cater to long-text comprehension by expanding the input of traditional tasks, which falls short to exhibit the unique characteristics of long-text understanding, including long dependency tasks and longer text length compatible with modern LLMs' context window size. In this paper, we introduce a benchmark for extremely long context understanding with long-range dependencies, XL$^2$Bench, which includes three scenarios: Fiction Reading, Paper Reading, and Law Reading, and four tasks of increasing complexity: Memory Retrieval, Detailed Understanding, Overall Understanding, and Open-ended Generation, covering 27 subtasks in English and Chinese. It has an average length of 100K+ words (English) and 200K+ characters (Chinese). Evaluating six leading LLMs on XL$^2$Bench, we find that their performance significantly lags behind human levels. Moreover, the observed decline in performance across both the original and enhanced datasets underscores the efficacy of our approach to mitigating data contamination.
• 摘要：大型语言模型 (LLM) 在不同的任务中表现出了卓越的性能，但受到其较小的上下文窗口大小的限制。人们已经提出了各种努力来扩展上下文窗口以容纳多达 200K 个输入标记。同时，建立具有更长文本长度和更高要求任务的高质量基准来提供全面评估对于促进法学硕士的长期上下文理解研究具有巨大的实际意义。然而，之前的基准测试创建的数据集表面上通过扩展传统任务的输入来迎合长文本理解，但未能展现长文本理解的独特特征，包括长依赖任务和与现代法学硕士上下文兼容的较长文本长度窗口大小。在本文中，我们介绍了一个具有长程依赖性的极长上下文理解基准，XL$^2$Bench，其中包括三个场景：小说阅读、论文阅读和法律阅读，以及四个日益复杂的任务：记忆检索、详细理解、整体理解、开放式生成，涵盖中英文27个子任务。它的平均长度为 100K+ 单词（英语）和 200K+ 字符（中文）。在 XL$^2$Bench 上评估六位领先的法学硕士，我们发现他们的表现明显落后于人类水平。此外，观察到的原始数据集和增强数据集的性能下降强调了我们减轻数据污染方法的有效性。

Title: RoT: Enhancing Large Language Models with Reflection on Search Trees

• Authors: Wenyang Hui, Yan Wang, Kewei Tu, Chengyue Jiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05449
• Pdf URL: https://arxiv.org/pdf/2404.05449
• Copy Paste: [[2404.05449]] RoT: Enhancing Large Language Models with Reflection on Search Trees(https://arxiv.org/abs/2404.05449)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Large language models (LLMs) have demonstrated impressive capability in reasoning and planning when integrated with tree-search-based prompting methods. However, since these methods ignore the previous search experiences, they often make the same mistakes in the search process. To address this issue, we introduce Reflection on search Trees (RoT), an LLM reflection framework designed to improve the performance of tree-search-based prompting methods. It uses a strong LLM to summarize guidelines from previous tree search experiences to enhance the ability of a weak LLM. The guidelines are instructions about solving this task through tree search which can prevent the weak LLMs from making similar mistakes in the past search process. In addition, we proposed a novel state selection method, which identifies the critical information from historical search processes to help RoT generate more specific and meaningful guidelines. In our extensive experiments, we find that RoT significantly improves the performance of LLMs in reasoning or planning tasks with various tree-search-based prompting methods (e.g., BFS and MCTS). Non-tree-search-based prompting methods such as Chain-of-Thought (CoT) can also benefit from RoT guidelines since RoT can provide task-specific knowledge collected from the search experience.
• 摘要：当与基于树搜索的提示方法集成时，大型语言模型（LLM）在推理和规划方面表现出了令人印象深刻的能力。然而，由于这些方法忽略了以前的搜索经验，因此在搜索过程中经常犯同样的错误。为了解决这个问题，我们引入了搜索树反射（RoT），这是一个LLM反射框架，旨在提高基于树搜索的提示方法的性能。它使用强LLM来总结以前树搜索经验的指导方针，以增强弱LLM的能力。该指南是关于通过树搜索解决此任务的说明，可以防止实力较弱的法学硕士在过去的搜索过程中犯类似的错误。此外，我们提出了一种新颖的状态选择方法，该方法可以从历史搜索过程中识别关键信息，以帮助 RoT 生成更具体和更有意义的指导方针。在我们广泛的实验中，我们发现 RoT 通过各种基于树搜索的提示方法（例如 BFS 和 MCTS）显着提高了法学硕士在推理或规划任务中的表现。基于非树搜索的提示方法（例如思想链 (CoT)）也可以从 RoT 指南中受益，因为 RoT 可以提供从搜索经验中收集的特定于任务的知识。

Title: PetKaz at SemEval-2024 Task 8: Can Linguistics Capture the Specifics of LLM-generated Text?

• Authors: Kseniia Petukhova, Roman Kazakov, Ekaterina Kochmar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05483
• Pdf URL: https://arxiv.org/pdf/2404.05483
• Copy Paste: [[2404.05483]] PetKaz at SemEval-2024 Task 8: Can Linguistics Capture the Specifics of LLM-generated Text?(https://arxiv.org/abs/2404.05483)
• Keywords: llm
• Abstract: In this paper, we present our submission to the SemEval-2024 Task 8 "Multigenerator, Multidomain, and Multilingual Black-Box Machine-Generated Text Detection", focusing on the detection of machine-generated texts (MGTs) in English. Specifically, our approach relies on combining embeddings from the RoBERTa-base with diversity features and uses a resampled training set. We score 12th from 124 in the ranking for Subtask A (monolingual track), and our results show that our approach is generalizable across unseen models and domains, achieving an accuracy of 0.91.
• 摘要：在本文中，我们提交了 SemEval-2024 任务 8“多生成器、多域和多语言黑盒机器生成文本检测”，重点关注英语机器生成文本 (MGT) 的检测。具体来说，我们的方法依赖于将 RoBERTa 库的嵌入与多样性特征相结合，并使用重新采样的训练集。我们在子任务 A（单语言轨道）的排名中从 124 分中获得第 12 名，我们的结果表明我们的方法可以在未见过的模型和领域中推广，达到 0.91 的准确度。

Title: PetKaz at SemEval-2024 Task 3: Advancing Emotion Classification with an LLM for Emotion-Cause Pair Extraction in Conversations

• Authors: Roman Kazakov, Kseniia Petukhova, Ekaterina Kochmar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05502
• Pdf URL: https://arxiv.org/pdf/2404.05502
• Copy Paste: [[2404.05502]] PetKaz at SemEval-2024 Task 3: Advancing Emotion Classification with an LLM for Emotion-Cause Pair Extraction in Conversations(https://arxiv.org/abs/2404.05502)
• Keywords: gpt, llm
• Abstract: In this paper, we present our submission to the SemEval-2023 Task~3 "The Competition of Multimodal Emotion Cause Analysis in Conversations", focusing on extracting emotion-cause pairs from dialogs. Specifically, our approach relies on combining fine-tuned GPT-3.5 for emotion classification and a BiLSTM-based neural network to detect causes. We score 2nd in the ranking for Subtask 1, demonstrating the effectiveness of our approach through one of the highest weighted-average proportional F1 scores recorded at 0.264.
• 摘要：在本文中，我们提交了 SemEval-2023 Task~3“对话中多模态情感原因分析的竞赛”，重点关注从对话中提取情感-原因对。具体来说，我们的方法依赖于结合用于情绪分类的微调 GPT-3.5 和基于 BiLSTM 的神经网络来检测原因。我们在子任务 1 的排名中获得第二名，通过记录的最高加权平均比例 F1 分数（0.264）证明了我们方法的有效性。

Title: Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data

• Authors: Tim Baumgärtner, Yang Gao, Dana Alon, Donald Metzler
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05530
• Pdf URL: https://arxiv.org/pdf/2404.05530
• Copy Paste: [[2404.05530]] Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data(https://arxiv.org/abs/2404.05530)
• Keywords: language model
• Abstract: Reinforcement Learning from Human Feedback (RLHF) is a popular method for aligning Language Models (LM) with human values and preferences. RLHF requires a large number of preference pairs as training data, which are often used in both the Supervised Fine-Tuning and Reward Model training, and therefore publicly available datasets are commonly used. In this work, we study to what extent a malicious actor can manipulate the LMs generations by poisoning the preferences, i.e., injecting poisonous preference pairs into these datasets and the RLHF training process. We propose strategies to build poisonous preference pairs and test their performance by poisoning two widely used preference datasets. Our results show that preference poisoning is highly effective: by injecting a small amount of poisonous data (1-5% of the original dataset), we can effectively manipulate the LM to generate a target entity in a target sentiment (positive or negative). The findings from our experiments also shed light on strategies to defend against the preference poisoning attack.
• 摘要：基于人类反馈的强化学习 (RLHF) 是一种流行的方法，用于使语言模型 (LM) 与人类价值观和偏好保持一致。 RLHF 需要大量的偏好对作为训练数据，这些数据常用于监督微调和奖励模型训练，因此通常使用公开的数据集。在这项工作中，我们研究了恶意行为者可以在多大程度上通过毒害偏好来操纵 LM 生成，即将有毒偏好对注入这些数据集和 RLHF 训练过程中。我们提出了构建有毒偏好对的策略，并通过毒害两个广泛使用的偏好数据集来测试其性能。我们的结果表明，偏好中毒是非常有效的：通过注入少量有毒数据（原始数据集的 1-5%），我们可以有效地操纵 LM 来生成目标情绪（积极或消极）的目标实体。我们的实验结果还揭示了防御偏好中毒攻击的策略。

Title: OPSD: an Offensive Persian Social media Dataset and its baseline evaluations

• Authors: Mehran Safayani, Amir Sartipi, Amir Hossein Ahmadi, Parniyan Jalali, Amir Hossein Mansouri, Mohammad Bisheh-Niasar, Zahra Pourbahman
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05540
• Pdf URL: https://arxiv.org/pdf/2404.05540
• Copy Paste: [[2404.05540]] OPSD: an Offensive Persian Social media Dataset and its baseline evaluations(https://arxiv.org/abs/2404.05540)
• Keywords: language model
• Abstract: The proliferation of hate speech and offensive comments on social media has become increasingly prevalent due to user activities. Such comments can have detrimental effects on individuals' psychological well-being and social behavior. While numerous datasets in the English language exist in this domain, few equivalent resources are available for Persian language. To address this gap, this paper introduces two offensive datasets. The first dataset comprises annotations provided by domain experts, while the second consists of a large collection of unlabeled data obtained through web crawling for unsupervised learning purposes. To ensure the quality of the former dataset, a meticulous three-stage labeling process was conducted, and kappa measures were computed to assess inter-annotator agreement. Furthermore, experiments were performed on the dataset using state-of-the-art language models, both with and without employing masked language modeling techniques, as well as machine learning algorithms, in order to establish the baselines for the dataset using contemporary cutting-edge approaches. The obtained F1-scores for the three-class and two-class versions of the dataset were 76.9% and 89.9% for XLM-RoBERTa, respectively.
• 摘要：由于用户活动，社交媒体上仇恨言论和攻击性评论的扩散变得越来越普遍。此类评论可能会对个人的心理健康和社会行为产生不利影响。虽然该领域存在大量英语数据集，但可用于波斯语的等效资源却很少。为了解决这一差距，本文引入了两个令人反感的数据集。第一个数据集包含领域专家提供的注释，而第二个数据集包含通过网络爬行获得的大量未标记数据，用于无监督学习目的。为了确保之前数据集的质量，进行了细致的三阶段标记过程，并计算了 kappa 度量来评估注释者间的一致性。此外，使用最先进的语言模型（使用或不使用掩码语言建模技术）以及机器学习算法对数据集进行了实验，以便使用当代尖端技术建立数据集的基线接近。 XLM-RoBERTa 数据集的三类和两类版本获得的 F1 分数分别为 76.9% 和 89.9%。

Title: Chinese Sequence Labeling with Semi-Supervised Boundary-Aware Language Model Pre-training

• Authors: Longhui Zhang, Dingkun Long, Meishan Zhang, Yanzhao Zhang, Pengjun Xie, Min Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05560
• Pdf URL: https://arxiv.org/pdf/2404.05560
• Copy Paste: [[2404.05560]] Chinese Sequence Labeling with Semi-Supervised Boundary-Aware Language Model Pre-training(https://arxiv.org/abs/2404.05560)
• Keywords: language model
• Abstract: Chinese sequence labeling tasks are heavily reliant on accurate word boundary demarcation. Although current pre-trained language models (PLMs) have achieved substantial gains on these tasks, they rarely explicitly incorporate boundary information into the modeling process. An exception to this is BABERT, which incorporates unsupervised statistical boundary information into Chinese BERT's pre-training objectives. Building upon this approach, we input supervised high-quality boundary information to enhance BABERT's learning, developing a semi-supervised boundary-aware PLM. To assess PLMs' ability to encode boundaries, we introduce a novel ``Boundary Information Metric'' that is both simple and effective. This metric allows comparison of different PLMs without task-specific fine-tuning. Experimental results on Chinese sequence labeling datasets demonstrate that the improved BABERT variant outperforms the vanilla version, not only on these tasks but also more broadly across a range of Chinese natural language understanding tasks. Additionally, our proposed metric offers a convenient and accurate means of evaluating PLMs' boundary awareness.
• 摘要：中文序列标注任务严重依赖于准确的词边界划分。尽管当前的预训练语言模型 (PLM) 在这些任务上取得了实质性进展，但它们很少将边界信息明确地纳入建模过程中。 BABERT 是一个例外，它将无监督统计边界信息纳入中国 BERT 的预训练目标中。在此方法的基础上，我们输入有监督的高质量边界信息来增强 BABERT 的学习能力，开发出半监督的边界感知 PLM。为了评估 PLM 编码边界的能力，我们引入了一种新颖的“边界信息度量”，​​它既简单又有效。该指标允许比较不同的 PLM，而无需针对特定任务进行微调。中文序列标记数据集上的实验结果表明，改进的 BABERT 变体不仅在这些任务上，而且在一系列中文自然语言理解任务上都优于普通版本。此外，我们提出的指标提供了一种方便而准确的方法来评估 PLM 的边界意识。

Title: Enhancing Software Related Information Extraction with Generative Language Models through Single-Choice Question Answering

• Authors: Wolfgang Otto, Sharmila Upadhyaya, Stefan Dietze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05587
• Pdf URL: https://arxiv.org/pdf/2404.05587
• Copy Paste: [[2404.05587]] Enhancing Software Related Information Extraction with Generative Language Models through Single-Choice Question Answering(https://arxiv.org/abs/2404.05587)
• Keywords: language model, retrieval-augmented generation
• Abstract: This paper describes our participation in the Shared Task on Software Mentions Disambiguation (SOMD), with a focus on improving relation extraction in scholarly texts through Generative Language Models (GLMs) using single-choice question-answering. The methodology prioritises the use of in-context learning capabilities of GLMs to extract software-related entities and their descriptive attributes, such as distributive information. Our approach uses Retrieval-Augmented Generation (RAG) techniques and GLMs for Named Entity Recognition (NER) and Attributive NER to identify relationships between extracted software entities, providing a structured solution for analysing software citations in academic literature. The paper provides a detailed description of our approach, demonstrating how using GLMs in a single-choice QA paradigm can greatly enhance IE methodologies. Our participation in the SOMD shared task highlights the importance of precise software citation practices and showcases our system's ability to overcome the challenges of disambiguating and extracting relationships between software mentions. This sets the groundwork for future research and development in this field.
• 摘要：本文描述了我们参与软件提及消歧共享任务 (SOMD) 的情况，重点是通过使用单选问答的生成语言模型 (GLM) 改进学术文本中的关系提取。该方法优先考虑使用 GLM 的上下文学习功能来提取与软件相关的实体及其描述属性，例如分布信息。我们的方法使用检索增强生成 (RAG) 技术和用于命名实体识别 (NER) 和属性 NER 的 GLM 来识别提取的软件实体之间的关系，为分析学术文献中的软件引用提供结构化解决方案。本文详细描述了我们的方法，展示了如何在单选 QA 范式中使用 GLM 可以极大地增强 IE 方法。我们对 SOMD 共享任务的参与凸显了精确软件引用实践的重要性，并展示了我们的系统克服消除歧义和提取软件提及之间关系的挑战的能力。这为该领域未来的研究和开发奠定了基础。

Title: MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering

• Authors: Iñigo Alonso, Maite Oronoz, Rodrigo Agerri
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05590
• Pdf URL: https://arxiv.org/pdf/2404.05590
• Copy Paste: [[2404.05590]] MedExpQA: Multilingual Benchmarking of Large Language Models for Medical Question Answering(https://arxiv.org/abs/2404.05590)
• Keywords: language model, llm, retrieval augmented generation
• Abstract: Large Language Models (LLMs) have the potential of facilitating the development of Artificial Intelligence technology to assist medical experts for interactive decision support, which has been demonstrated by their competitive performances in Medical QA. However, while impressive, the required quality bar for medical applications remains far from being achieved. Currently, LLMs remain challenged by outdated knowledge and by their tendency to generate hallucinated content. Furthermore, most benchmarks to assess medical knowledge lack reference gold explanations which means that it is not possible to evaluate the reasoning of LLMs predictions. Finally, the situation is particularly grim if we consider benchmarking LLMs for languages other than English which remains, as far as we know, a totally neglected topic. In order to address these shortcomings, in this paper we present MedExpQA, the first multilingual benchmark based on medical exams to evaluate LLMs in Medical Question Answering. To the best of our knowledge, MedExpQA includes for the first time reference gold explanations written by medical doctors which can be leveraged to establish various gold-based upper-bounds for comparison with LLMs performance. Comprehensive multilingual experimentation using both the gold reference explanations and Retrieval Augmented Generation (RAG) approaches show that performance of LLMs still has large room for improvement, especially for languages other than English. Furthermore, and despite using state-of-the-art RAG methods, our results also demonstrate the difficulty of obtaining and integrating readily available medical knowledge that may positively impact results on downstream evaluations for Medical Question Answering. So far the benchmark is available in four languages, but we hope that this work may encourage further development to other languages.
• 摘要：大语言模型（LLM）具有促进人工智能技术发展的潜力，可以协助医学专家进行交互式决策支持，这一点已经从其在医学质量保证方面的竞争表现得到了证明。然而，尽管令人印象深刻，但医疗应用所需的质量标准仍远未达到。目前，法学硕士仍然面临着过时知识和产生幻觉内容倾向的挑战。此外，大多数评估医学知识的基准缺乏参考黄金解释，这意味着无法评估法学硕士预测的推理。最后，如果我们考虑对英语以外的语言的法学硕士进行基准测试，情况就特别严峻，据我们所知，这仍然是一个完全被忽视的话题。为了解决这些缺点，在本文中，我们提出了 MedExpQA，这是第一个基于医学考试的多语言基准，用于评估医学问答领域的法学硕士。据我们所知，MedExpQA 首次包含了由医生撰写的参考黄金解释，这些解释可用于建立各种基于黄金的上限，以便与法学硕士的表现进行比较。使用黄金参考解释和检索增强生成（RAG）方法进行的综合多语言实验表明，法学硕士的表现仍有很大的改进空间，特别是对于英语以外的语言。此外，尽管使用了最先进的 RAG 方法，我们的结果也表明了获取和整合现成的医学知识的难度，这可能会对医学问答的下游评估结果产生积极影响。到目前为止，该基准测试有四种语言版本，但我们希望这项工作可以鼓励其他语言的进一步开发。

Title: SpeechAlign: Aligning Speech Generation to Human Preferences

• Authors: Dong Zhang, Zhaowei Li, Shimin Li, Xin Zhang, Pengyu Wang, Yaqian Zhou, Xipeng Qiu
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2404.05600
• Pdf URL: https://arxiv.org/pdf/2404.05600
• Copy Paste: [[2404.05600]] SpeechAlign: Aligning Speech Generation to Human Preferences(https://arxiv.org/abs/2404.05600)
• Keywords: language model, gpt
• Abstract: Speech language models have significantly advanced in generating realistic speech, with neural codec language models standing out. However, the integration of human feedback to align speech outputs to human preferences is often neglected. This paper addresses this gap by first analyzing the distribution gap in codec language models, highlighting how it leads to discrepancies between the training and inference phases, which negatively affects performance. Then we explore leveraging learning from human feedback to bridge the distribution gap. We introduce SpeechAlign, an iterative self-improvement strategy that aligns speech language models to human preferences. SpeechAlign involves constructing a preference codec dataset contrasting golden codec tokens against synthetic tokens, followed by preference optimization to improve the codec language model. This cycle of improvement is carried out iteratively to steadily convert weak models to strong ones. Through both subjective and objective evaluations, we show that SpeechAlign can bridge the distribution gap and facilitating continuous self-improvement of the speech language model. Moreover, SpeechAlign exhibits robust generalization capabilities and works for smaller models. Code and models will be available at https://github.com/0nutation/SpeechGPT.
• 摘要：语音语言模型在生成真实语音方面取得了显着进步，其中神经编解码器语言模型脱颖而出。然而，整合人类反馈以使语音输出与人类偏好保持一致常常被忽视。本文首先分析编解码器语言模型中的分布差距，强调它如何导致训练和推理阶段之间的差异，从而对性能产生负面影响，从而解决了这一差距。然后我们探索利用从人类反馈中学习来弥合分配差距。我们引入了 SpeechAlign，这是一种迭代的自我改进策略，可以使语音语言模型与人类偏好保持一致。 SpeechAlign 涉及构建偏好编解码器数据集，将黄金编解码器标记与合成标记进行对比，然后进行偏好优化以改进编解码器语言模型。这种改进循环迭代地进行，以稳定地将弱模型转换为强模型。通过主观和客观评估，我们表明 SpeechAlign 可以弥合分布差距并促进语音语言模型的不断自我改进。此外，SpeechAlign 具有强大的泛化能力，适用于较小的模型。代码和模型将在 https://github.com/0nutation/SpeechGPT 上提供。

Title: LTNER: Large Language Model Tagging for Named Entity Recognition with Contextualized Entity Marking

• Authors: Faren Yan, Peng Yu, Xin Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.05624
• Pdf URL: https://arxiv.org/pdf/2404.05624
• Copy Paste: [[2404.05624]] LTNER: Large Language Model Tagging for Named Entity Recognition with Contextualized Entity Marking(https://arxiv.org/abs/2404.05624)
• Keywords: language model, gpt, llm
• Abstract: The use of LLMs for natural language processing has become a popular trend in the past two years, driven by their formidable capacity for context comprehension and learning, which has inspired a wave of research from academics and industry professionals. However, for certain NLP tasks, such as NER, the performance of LLMs still falls short when compared to supervised learning methods. In our research, we developed a NER processing framework called LTNER that incorporates a revolutionary Contextualized Entity Marking Gen Method. By leveraging the cost-effective GPT-3.5 coupled with context learning that does not require additional training, we significantly improved the accuracy of LLMs in handling NER tasks. The F1 score on the CoNLL03 dataset increased from the initial 85.9% to 91.9%, approaching the performance of supervised fine-tuning. This outcome has led to a deeper understanding of the potential of LLMs.
• 摘要：由于法学硕士强大的语境理解和学习能力，利用法学硕士进行自然语言处理已成为过去两年的流行趋势，激发了学术界和行业专业人士的研究热潮。然而，对于某些 NLP 任务（例如 NER），与监督学习方法相比，LLM 的性能仍然存在不足。在我们的研究中，我们开发了一个名为 LTNER 的 NER 处理框架，它结合了革命性的上下文化实体标记生成方法。通过利用经济高效的 GPT-3.5 以及不需要额外训练的上下文学习，我们显着提高了法学硕士处理 NER 任务的准确性。 CoNLL03数据集上的F1分数从最初的85.9%增加到91.9%，接近监督微调的性能。这一结果使人们对法学硕士的潜力有了更深入的了解。

Title: Fighting crime with Transformers: Empirical analysis of address parsing methods in payment data

• Authors: Haitham Hammami, Louis Baligand, Bojan Petrovski
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05632
• Pdf URL: https://arxiv.org/pdf/2404.05632
• Copy Paste: [[2404.05632]] Fighting crime with Transformers: Empirical analysis of address parsing methods in payment data(https://arxiv.org/abs/2404.05632)
• Keywords: language model, llm
• Abstract: In the financial industry, identifying the location of parties involved in payments is a major challenge in the context of various regulatory requirements. For this purpose address parsing entails extracting fields such as street, postal code, or country from free text message attributes. While payment processing platforms are updating their standards with more structured formats such as SWIFT with ISO 20022, address parsing remains essential for a considerable volume of messages. With the emergence of Transformers and Generative Large Language Models (LLM), we explore the performance of state-of-the-art solutions given the constraint of processing a vast amount of daily data. This paper also aims to show the need for training robust models capable of dealing with real-world noisy transactional data. Our results suggest that a well fine-tuned Transformer model using early-stopping significantly outperforms other approaches. Nevertheless, generative LLMs demonstrate strong zero-shot performance and warrant further investigations.
• 摘要：在金融行业，在各种监管要求的背景下，确定参与支付的各方的所在地是一项重大挑战。为此，地址解析需要从自由文本消息属性中提取街道、邮政编码或国家/地区等字段。虽然支付处理平台正在使用更结构化的格式更新其标准，例如采用 ISO 20022 的 SWIFT，但地址解析对于大量消息仍然至关重要。随着 Transformer 和生成式大型语言模型 (LLM) 的出现，我们在处理大量日常数据的限制下探索最先进解决方案的性能。本文还旨在表明需要训练能够处理现实世界噪声交易数据的稳健模型。我们的结果表明，使用提前停止进行良好微调的 Transformer 模型的性能明显优于其他方法。尽管如此，生成式法学硕士表现出了强大的零样本性能，值得进一步研究。

Title: Evaluating Mathematical Reasoning Beyond Accuracy

• Authors: Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, Pengfei Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.05692
• Pdf URL: https://arxiv.org/pdf/2404.05692
• Copy Paste: [[2404.05692]] Evaluating Mathematical Reasoning Beyond Accuracy(https://arxiv.org/abs/2404.05692)
• Keywords: language model, llm
• Abstract: The leaderboard of Large Language Models (LLMs) in mathematical tasks has been continuously updated. However, the majority of evaluations focus solely on the final results, neglecting the quality of the intermediate steps. This oversight can mask underlying problems, such as logical errors or unnecessary steps in the reasoning process. To measure reasoning beyond final-answer accuracy, we introduce ReasonEval, a new methodology for evaluating the quality of reasoning steps. ReasonEval employs $\textit{validity}$ and $\textit{redundancy}$ to characterize the reasoning quality, as well as accompanying LLMs to assess them automatically. Instantiated by base models that possess strong mathematical knowledge and trained with high-quality labeled data, ReasonEval achieves state-of-the-art performance on human-labeled datasets and can accurately detect different types of errors generated by perturbation. When applied to evaluate LLMs specialized in math, we find that an increase in final-answer accuracy does not necessarily guarantee an improvement in the overall quality of the reasoning steps for challenging mathematical problems. Additionally, we observe that ReasonEval can play a significant role in data selection. We release the best-performing model, meta-evaluation script, and all evaluation results at https://github.com/GAIR-NLP/ReasonEval.
• 摘要：数学任务中的大型语言模型（LLM）排行榜不断更新。然而，大多数评估只关注最终结果，而忽略了中间步骤的质量。这种疏忽可能会掩盖潜在的问题，例如逻辑错误或推理过程中不必要的步骤。为了衡量最终答案准确性之外的推理，我们引入了 ReasonEval，这是一种评估推理步骤质量的新方法。 ReasonEval 使用 $\textit{validity}$ 和 $\textit{redundancy}$ 来表征推理质量，并使用随附的 LLM 来自动评估它们。 ReasonEval 由拥有强大数学知识的基础模型实例化，并使用高质量标记数据进行训练，在人类标记数据集上实现了最先进的性能，并且可以准确检测扰动产生的不同类型的错误。当应用于评估数学专业的法学硕士时，我们发现最终答案准确性的提高并不一定能保证挑战性数学问题的推理步骤的整体质量的提高。此外，我们观察到 ReasonEval 在数据选择中可以发挥重要作用。我们在 https://github.com/GAIR-NLP/ReasonEval 发布了性能最佳的模型、元评估脚本和所有评估结果。

Title: Comprehensive Study on German Language Models for Clinical and Biomedical Text Understanding

• Authors: Ahmad Idrissi-Yaghir, Amin Dada, Henning Schäfer, Kamyar Arzideh, Giulia Baldini, Jan Trienes, Max Hasin, Jeanette Bewersdorff, Cynthia S. Schmidt, Marie Bauer, Kaleb E. Smith, Jiang Bian, Yonghui Wu, Jörg Schlötterer, Torsten Zesch, Peter A. Horn, Christin Seifert, Felix Nensa, Jens Kleesiek, Christoph M. Friedrich
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.05694
• Pdf URL: https://arxiv.org/pdf/2404.05694
• Copy Paste: [[2404.05694]] Comprehensive Study on German Language Models for Clinical and Biomedical Text Understanding(https://arxiv.org/abs/2404.05694)
• Keywords: language model
• Abstract: Recent advances in natural language processing (NLP) can be largely attributed to the advent of pre-trained language models such as BERT and RoBERTa. While these models demonstrate remarkable performance on general datasets, they can struggle in specialized domains such as medicine, where unique domain-specific terminologies, domain-specific abbreviations, and varying document structures are common. This paper explores strategies for adapting these models to domain-specific requirements, primarily through continuous pre-training on domain-specific data. We pre-trained several German medical language models on 2.4B tokens derived from translated public English medical data and 3B tokens of German clinical data. The resulting models were evaluated on various German downstream tasks, including named entity recognition (NER), multi-label classification, and extractive question answering. Our results suggest that models augmented by clinical and translation-based pre-training typically outperform general domain models in medical contexts. We conclude that continuous pre-training has demonstrated the ability to match or even exceed the performance of clinical models trained from scratch. Furthermore, pre-training on clinical data or leveraging translated texts have proven to be reliable methods for domain adaptation in medical NLP tasks.
• 摘要：自然语言处理 (NLP) 的最新进展很大程度上归功于 BERT 和 RoBERTa 等预训练语言模型的出现。虽然这些模型在一般数据集上表现出卓越的性能，但它们在医学等专业领域可能会遇到困难，在这些领域，独特的特定领域术语、特定领域缩写和不同的文档结构很常见。本文探讨了使这些模型适应特定领域要求的策略，主要是通过对特定领域数据进行持续预训练。我们在源自翻译的公共英语医学数据的 2.4B 标记和德国临床数据的 3B 标记上预训练了多个德语医学语言模型。生成的模型在各种德国下游任务上进行了评估，包括命名实体识别 (NER)、多标签分类和提取式问答。我们的结果表明，通过临床和基于翻译的预训练增强的模型通常优于医学环境中的一般领域模型。我们的结论是，连续预训练已证明能够匹配甚至超过从头开始训练的临床模型的性能。此外，临床数据预训练或利用翻译文本已被证明是医学 NLP 任务中领域适应的可靠方法。