2024-05-16

Title: Large Language Models for Human-Machine Collaborative Particle Accelerator Tuning through Natural Language

• Authors: Jan Kaiser, Annika Eichler, Anne Lauscher
• Subjects: cs.CL, cs.AI, cs.LG, physics.acc-ph
• Abstract URL: https://arxiv.org/abs/2405.08888
• Pdf URL: https://arxiv.org/pdf/2405.08888
• Copy Paste: [[2405.08888]] Large Language Models for Human-Machine Collaborative Particle Accelerator Tuning through Natural Language(https://arxiv.org/abs/2405.08888)
• Keywords: language model, llm, prompt
• Abstract: Autonomous tuning of particle accelerators is an active and challenging field of research with the goal of enabling novel accelerator technologies cutting-edge high-impact applications, such as physics discovery, cancer research and material sciences. A key challenge with autonomous accelerator tuning remains that the most capable algorithms require an expert in optimisation, machine learning or a similar field to implement the algorithm for every new tuning task. In this work, we propose the use of large language models (LLMs) to tune particle accelerators. We demonstrate on a proof-of-principle example the ability of LLMs to successfully and autonomously tune a particle accelerator subsystem based on nothing more than a natural language prompt from the operator, and compare the performance of our LLM-based solution to state-of-the-art optimisation algorithms, such as Bayesian optimisation (BO) and reinforcement learning-trained optimisation (RLO). In doing so, we also show how LLMs can perform numerical optimisation of a highly non-linear real-world objective function. Ultimately, this work represents yet another complex task that LLMs are capable of solving and promises to help accelerate the deployment of autonomous tuning algorithms to the day-to-day operations of particle accelerators.
• 摘要：粒子加速器的自主调谐是一个活跃且具有挑战性的研究领域，其目标是使新型加速器技术能够应用于尖端的高影响力应用，例如物理发现、癌症研究和材料科学。自主加速器调整的一个关键挑战仍然是，最强大的算法需要优化、机器学习或类似领域的专家来为每个新的调整任务实施算法。在这项工作中，我们建议使用大型语言模型（LLM）来调整粒子加速器。我们在原理验证示例中展示了法学硕士仅基于操作员的自然语言提示即可成功自主调整粒子加速器子系统的能力，并将我们基于法学硕士的解决方案的性能与现状进行比较-最先进的优化算法，例如贝叶斯优化（BO）和强化学习训练优化（RLO）。在此过程中，我们还展示了法学硕士如何对高度非线性的现实世界目标函数进行数值优化。最终，这项工作代表了法学硕士能够解决的另一项复杂任务，并有望帮助加速​​将自主调整算法部署到粒子加速器的日常操作中。

Title: LLM-Assisted Rule Based Machine Translation for Low/No-Resource Languages

• Authors: Jared Coleman, Bhaskar Krishnamachari, Khalil Iskarous, Ruben Rosales
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.08997
• Pdf URL: https://arxiv.org/pdf/2405.08997
• Copy Paste: [[2405.08997]] LLM-Assisted Rule Based Machine Translation for Low/No-Resource Languages(https://arxiv.org/abs/2405.08997)
• Keywords: llm
• Abstract: We propose a new paradigm for machine translation that is particularly useful for no-resource languages (those without any publicly available bilingual or monolingual corpora): \acronym (LLM-Assisted Rule Based Machine Translation). Using the \acronym paradigm, we design the first language education/revitalization-oriented machine translator for Owens Valley Paiute (OVP), a critically endangered Indigenous American language for which there is virtually no publicly available data. We present a detailed evaluation of the translator's components: a rule-based sentence builder, an OVP to English translator, and an English to OVP translator. We also discuss the potential of the paradigm, its limitations, and the many avenues for future research that it opens up.
• 摘要：我们提出了一种新的机器翻译范例，对于无资源语言（没有任何公开可用的双语或单语语料库的语言）特别有用：\acronym（法学硕士辅助规则机器翻译）。使用\首字母缩略词范式，我们为欧文斯谷派尤特语（OVP）设计了第一个面向语言教育/振兴的机器翻译器，这是一种极度濒危的美洲土著语言，几乎没有公开可用的数据。我们对翻译器组件进行了详细评估：基于规则的句子构建器、OVP 到英语翻译器和英语到 OVP 翻译器。我们还讨论了该范式的潜力、其局限性以及它为未来研究开辟的许多途径。

Title: A Japanese-Chinese Parallel Corpus Using Crowdsourcing for Web Mining

• Authors: Masaaki Nagata, Makoto Morishita, Katsuki Chousa, Norihito Yasuda
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.09017
• Pdf URL: https://arxiv.org/pdf/2405.09017
• Copy Paste: [[2405.09017]] A Japanese-Chinese Parallel Corpus Using Crowdsourcing for Web Mining(https://arxiv.org/abs/2405.09017)
• Keywords: language model
• Abstract: Using crowdsourcing, we collected more than 10,000 URL pairs (parallel top page pairs) of bilingual websites that contain parallel documents and created a Japanese-Chinese parallel corpus of 4.6M sentence pairs from these websites. We used a Japanese-Chinese bilingual dictionary of 160K word pairs for document and sentence alignment. We then used high-quality 1.2M Japanese-Chinese sentence pairs to train a parallel corpus filter based on statistical language models and word translation probabilities. We compared the translation accuracy of the model trained on these 4.6M sentence pairs with that of the model trained on Japanese-Chinese sentence pairs from CCMatrix (12.4M), a parallel corpus from global web mining. Although our corpus is only one-third the size of CCMatrix, we found that the accuracy of the two models was comparable and confirmed that it is feasible to use crowdsourcing for web mining of parallel data.
• 摘要：通过众包，我们收集了超过 10,000 个包含平行文档的双语网站 URL 对（平行首页对），并从这些网站创建了包含 460 万个句子对的日汉平行语料库。我们使用包含 16 万个单词对的日汉双语词典来进行文档和句子对齐。然后，我们使用高质量的 120 万个日汉句子对来训练基于统计语言模型和单词翻译概率的并行语料库过滤器。我们将在这 460 万个句子对上训练的模型的翻译准确性与在来自 CCMatrix（12.4M）（来自全球网络挖掘的平行语料库）的日文-中文句子对上训练的模型的翻译准确性进行了比较。尽管我们的语料库只有 CCMatrix 的三分之一，但我们发现两个模型的准确性相当，并证实使用众包进行并行数据的 Web 挖掘是可行的。

Title: A safety realignment framework via subspace-oriented model fusion for large language models

• Authors: Xin Yi, Shunfan Zheng, Linlin Wang, Xiaoling Wang, Liang He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.09055
• Pdf URL: https://arxiv.org/pdf/2405.09055
• Copy Paste: [[2405.09055]] A safety realignment framework via subspace-oriented model fusion for large language models(https://arxiv.org/abs/2405.09055)
• Keywords: language model, llm
• Abstract: The current safeguard mechanisms for large language models (LLMs) are indeed susceptible to jailbreak attacks, making them inherently fragile. Even the process of fine-tuning on apparently benign data for downstream tasks can jeopardize safety. One potential solution is to conduct safety fine-tuning subsequent to downstream fine-tuning. However, there's a risk of catastrophic forgetting during safety fine-tuning, where LLMs may regain safety measures but lose the task-specific knowledge acquired during downstream fine-tuning. In this paper, we introduce a safety realignment framework through subspace-oriented model fusion (SOMF), aiming to combine the safeguard capabilities of initially aligned model and the current fine-tuned model into a realigned model. Our approach begins by disentangling all task vectors from the weights of each fine-tuned model. We then identify safety-related regions within these vectors by subspace masking techniques. Finally, we explore the fusion of the initial safely aligned LLM with all task vectors based on the identified safety subspace. We validate that our safety realignment framework satisfies the safety requirements of a single fine-tuned model as well as multiple models during their fusion. Our findings confirm that SOMF preserves safety without notably compromising performance on downstream tasks, including instruction following in Chinese, English, and Hindi, as well as problem-solving capabilities in Code and Math.
• 摘要：当前大型语言模型（LLM）的保护机制确实容易受到越狱攻击，这使得它们本质上很脆弱。即使是对下游任务看似良性的数据进行微调的过程也可能会危及安全。一种可能的解决方案是在下游微调之后进行安全微调。然而，在安全微调过程中存在灾难性遗忘的风险，法学硕士可能会重新获得安全措施，但会丢失在下游微调过程中获得的特定于任务的知识。在本文中，我们通过面向子空间的模型融合（SOMF）引入了一种安全重新调整框架，旨在将初始调整模型和当前微调模型的保障能力结合到重新调整的模型中。我们的方法首先将所有任务向量与每个微调模型的权重分开。然后，我们通过子空间掩蔽技术识别这些向量中的安全相关区域。最后，我们基于已识别的安全子空间探索初始安全对齐的 LLM 与所有任务向量的融合。我们验证我们的安全调整框架满足单个微调模型以及多个模型融合过程中的安全要求。我们的研究结果证实，SOMF 在不显着影响下游任务性能的情况下保持了安全性，包括遵循中文、英语和印地语的指令，以及解决代码和数学问题的能力。

Title: HumanRankEval: Automatic Evaluation of LMs as Conversational Assistants

• Authors: Milan Gritta, Gerasimos Lampouras, Ignacio Iacobacci
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.09186
• Pdf URL: https://arxiv.org/pdf/2405.09186
• Copy Paste: [[2405.09186]] HumanRankEval: Automatic Evaluation of LMs as Conversational Assistants(https://arxiv.org/abs/2405.09186)
• Keywords: language model
• Abstract: Language models (LMs) as conversational assistants recently became popular tools that help people accomplish a variety of tasks. These typically result from adapting LMs pretrained on general domain text sequences through further instruction-tuning and possibly preference optimisation methods. The evaluation of such LMs would ideally be performed using human judgement, however, this is not scalable. On the other hand, automatic evaluation featuring auxiliary LMs as judges and/or knowledge-based tasks is scalable but struggles with assessing conversational ability and adherence to instructions. To help accelerate the development of LMs as conversational assistants, we propose a novel automatic evaluation task: HumanRankEval (HRE). It consists of a large-scale, diverse and high-quality set of questions, each with several answers authored and scored by humans. To perform evaluation, HRE ranks these answers based on their log-likelihood under the LM's distribution, and subsequently calculates their correlation with the corresponding human rankings. We support HRE's efficacy by investigating how efficiently it separates pretrained and instruction-tuned LMs of various sizes. We show that HRE correlates well with human judgements and is particularly responsive to model changes following instruction-tuning.
• 摘要：作为对话助手的语言模型（LM）最近成为帮助人们完成各种任务的流行工具。这些通常是通过进一步的指令调整和可能的偏好优化方法来调整在通用域文本序列上预训练的 LM 的结果。理想情况下，此类语言模型的评估应使用人类判断来进行，然而，这是不可扩展的。另一方面，以辅助语言模型作为法官和/或基于知识的任务的自动评估是可扩展的，但在评估对话能力和对指令的遵守方面遇到了困难。为了帮助加速 LM 作为对话助理的发展，我们提出了一种新颖的自动评估任务：HumanRankEval (HRE)。它由一组大规模、多样化和高质量的问题组成，每个问题都有几个由人类编写和评分的答案。为了进行评估，HRE 根据 LM 分布下的对数似然对这些答案进行排名，然后计算它们与相应的人类排名的相关性。我们通过研究 HRE 分离不同大小的预训练和指令调整 LM 的效率来支持 HRE 的功效。我们表明，HRE 与人类判断有很好的相关性，并且对指令调整后的模型变化特别敏感。

Title: Word Alignment as Preference for Machine Translation

• Authors: Qiyu Wu, Masaaki Nagata, Zhongtao Miao, Yoshimasa Tsuruoka
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.09223
• Pdf URL: https://arxiv.org/pdf/2405.09223
• Copy Paste: [[2405.09223]] Word Alignment as Preference for Machine Translation(https://arxiv.org/abs/2405.09223)
• Keywords: language model, gpt, llm, hallucination
• Abstract: The problem of hallucination and omission, a long-standing problem in machine translation (MT), is more pronounced when a large language model (LLM) is used in MT because an LLM itself is susceptible to these phenomena. In this work, we mitigate the problem in an LLM-based MT model by guiding it to better word alignment. We first study the correlation between word alignment and the phenomena of hallucination and omission in MT. Then we propose to utilize word alignment as preference to optimize the LLM-based MT model. The preference data are constructed by selecting chosen and rejected translations from multiple MT tools. Subsequently, direct preference optimization is used to optimize the LLM-based model towards the preference signal. Given the absence of evaluators specifically designed for hallucination and omission in MT, we further propose selecting hard instances and utilizing GPT-4 to directly evaluate the performance of the models in mitigating these issues. We verify the rationality of these designed evaluation methods by experiments, followed by extensive results demonstrating the effectiveness of word alignment-based preference optimization to mitigate hallucination and omission.
• 摘要：幻觉和遗漏问题是机器翻译（MT）中长期存在的问题，当机器翻译中使用大型语言模型（LLM）时，这种现象会更加明显，因为LLM本身很容易受到这些现象的影响。在这项工作中，我们通过引导基于 LLM 的 MT 模型更好地进行单词对齐来缓解该模型中的问题。我们首先研究机器翻译中的词对齐与幻觉和遗漏现象之间的相关性。然后我们建议优先利用词对齐来优化基于 LLM 的 MT 模型。偏好数据是通过从多个 MT 工具中选择选定和拒绝的翻译来构建的。随后，使用直接偏好优化来针对偏好信号优化基于 LLM 的模型。鉴于缺乏专门针对 MT 中的幻觉和遗漏而设计的评估器，我们进一步建议选择硬实例并利用 GPT-4 来直接评估模型在缓解这些问题方面的性能。我们通过实验验证了这些设计的评估方法的合理性，随后的大量结果证明了基于单词对齐的偏好优化对减少幻觉和遗漏的有效性。

Title: New Textual Corpora for Serbian Language Modeling

• Authors: Mihailo Škorić, Nikola Janković
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.09250
• Pdf URL: https://arxiv.org/pdf/2405.09250
• Copy Paste: [[2405.09250]] New Textual Corpora for Serbian Language Modeling(https://arxiv.org/abs/2405.09250)
• Keywords: language model
• Abstract: This paper will present textual corpora for Serbian (and Serbo-Croatian), usable for the training of large language models and publicly available at one of the several notable online repositories. Each corpus will be classified using multiple methods and its characteristics will be detailed. Additionally, the paper will introduce three new corpora: a new umbrella web corpus of Serbo-Croatian, a new high-quality corpus based on the doctoral dissertations stored within National Repository of Doctoral Dissertations from all Universities in Serbia, and a parallel corpus of abstract translation from the same source. The uniqueness of both old and new corpora will be accessed via frequency-based stylometric methods, and the results will be briefly discussed.
• 摘要：本文将介绍塞尔维亚语（和塞尔维亚-克罗地亚语）文本语料库，可用于大型语言模型的训练，并可在几个著名的在线存储库之一公开获取。每个语料库将使用多种方法进行分类，并详细说明其特征。此外，本文还将引入三个新的语料库：一个新的塞尔维亚-克罗地亚语伞式网络语料库、一个基于存储在塞尔维亚所有大学国家博士论文存储库中的博士论文的新的高质量语料库，以及一个平行的摘要语料库。翻译自同一来源。新旧语料库的独特性将通过基于频率的文体测量方法来获取，并将简要讨论结果。

Title: Sign of the Times: Evaluating the use of Large Language Models for Idiomaticity Detection

• Authors: Dylan Phelps, Thomas Pickard, Maggie Mi, Edward Gow-Smith, Aline Villavicencio
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.09279
• Pdf URL: https://arxiv.org/pdf/2405.09279
• Copy Paste: [[2405.09279]] Sign of the Times: Evaluating the use of Large Language Models for Idiomaticity Detection(https://arxiv.org/abs/2405.09279)
• Keywords: language model, gpt, llm, prompt
• Abstract: Despite the recent ubiquity of large language models and their high zero-shot prompted performance across a wide range of tasks, it is still not known how well they perform on tasks which require processing of potentially idiomatic language. In particular, how well do such models perform in comparison to encoder-only models fine-tuned specifically for idiomaticity tasks? In this work, we attempt to answer this question by looking at the performance of a range of LLMs (both local and software-as-a-service models) on three idiomaticity datasets: SemEval 2022 Task 2a, FLUTE, and MAGPIE. Overall, we find that whilst these models do give competitive performance, they do not match the results of fine-tuned task-specific models, even at the largest scales (e.g. for GPT-4). Nevertheless, we do see consistent performance improvements across model scale. Additionally, we investigate prompting approaches to improve performance, and discuss the practicalities of using LLMs for these tasks.
• 摘要：尽管大型语言模型最近无处不在，并且它们的高零样本率促进了各种任务的性能，但仍然不知道它们在需要处理潜在惯用语言的任务上的表现如何。特别是，与专门针对惯用性任务进行微调的仅编码器模型相比，此类模型的表现如何？在这项工作中，我们试图通过观察一系列法学硕士（本地模型和软件即服务模型）在三个惯用数据集上的表现来回答这个问题：SemEval 2022 Task 2a、FLUTE 和 MAGPIE。总的来说，我们发现虽然这些模型确实提供了有竞争力的性能，但它们与微调的特定任务模型的结果不符，即使是在最大规模上（例如 GPT-4）。尽管如此，我们确实看到了跨模型规模的一致性能改进。此外，我们还研究了提高绩效的激励方法，并讨论了使用法学硕士来完成这些任务的实用性。

Title: Do language models capture implied discourse meanings? An investigation with exhaustivity implicatures of Korean morphology

• Authors: Hagyeong Shin, Sean Trott
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.09293
• Pdf URL: https://arxiv.org/pdf/2405.09293
• Copy Paste: [[2405.09293]] Do language models capture implied discourse meanings? An investigation with exhaustivity implicatures of Korean morphology(https://arxiv.org/abs/2405.09293)
• Keywords: language model
• Abstract: Markedness in natural language is often associated with non-literal meanings in discourse. Differential Object Marking (DOM) in Korean is one instance of this phenomenon, where post-positional markers are selected based on both the semantic features of the noun phrases and the discourse features that are orthogonal to the semantic features. Previous work has shown that distributional models of language recover certain semantic features of words -- do these models capture implied discourse-level meanings as well? We evaluate whether a set of large language models are capable of associating discourse meanings with different object markings in Korean. Results suggest that discourse meanings of a grammatical marker can be more challenging to encode than that of a discourse marker.
• 摘要：自然语言中的标记性通常与话语中的非字面意义相关。韩语中的差异对象标记（DOM）是这种现象的一个例子，其中后置标记是根据名词短语的语义特征和与语义特征正交的话语特征来选择的。先前的工作表明，语言的分布式模型恢复了单词的某些语义特征——这些模型是否也捕获了隐含的话语层面的含义？我们评估一组大型语言模型是否能够将韩语中的话语含义与不同的对象标记相关联。结果表明，语法标记的话语含义比话语标记的话语含义更难编码。

Title: Comparing the Efficacy of GPT-4 and Chat-GPT in Mental Health Care: A Blind Assessment of Large Language Models for Psychological Support

• Authors: Birger Moell
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2405.09300
• Pdf URL: https://arxiv.org/pdf/2405.09300
• Copy Paste: [[2405.09300]] Comparing the Efficacy of GPT-4 and Chat-GPT in Mental Health Care: A Blind Assessment of Large Language Models for Psychological Support(https://arxiv.org/abs/2405.09300)
• Keywords: language model, gpt, prompt, chat
• Abstract: Background: Rapid advancements in natural language processing have led to the development of large language models with the potential to revolutionize mental health care. These models have shown promise in assisting clinicians and providing support to individuals experiencing various psychological challenges. Objective: This study aims to compare the performance of two large language models, GPT-4 and Chat-GPT, in responding to a set of 18 psychological prompts, to assess their potential applicability in mental health care settings. Methods: A blind methodology was employed, with a clinical psychologist evaluating the models' responses without knowledge of their origins. The prompts encompassed a diverse range of mental health topics, including depression, anxiety, and trauma, to ensure a comprehensive assessment. Results: The results demonstrated a significant difference in performance between the two models (p > 0.05). GPT-4 achieved an average rating of 8.29 out of 10, while Chat-GPT received an average rating of 6.52. The clinical psychologist's evaluation suggested that GPT-4 was more effective at generating clinically relevant and empathetic responses, thereby providing better support and guidance to potential users. Conclusions: This study contributes to the growing body of literature on the applicability of large language models in mental health care settings. The findings underscore the importance of continued research and development in the field to optimize these models for clinical use. Further investigation is necessary to understand the specific factors underlying the performance differences between the two models and to explore their generalizability across various populations and mental health conditions.
• 摘要：背景：自然语言处理的快速进步导致了大型语言模型的发展，有可能彻底改变心理健康护理。这些模型在协助临床医生和为经历各种心理挑战的个人提供支持方面表现出了希望。目的：本研究旨在比较两种大型语言模型 GPT-4 和 Chat-GPT 在响应一组 18 种心理提示时的表现，以评估它们在心理健康护理环境中的潜在适用性。方法：采用盲法，由临床心理学家在不了解模型来源的情况下评估模型的反应。这些提示涵盖了各种心理健康主题，包括抑郁、焦虑和创伤，以确保进行全面的评估。结果：结果表明两个模型之间的性能存在显着差异 (p > 0.05)。 GPT-4 的平均评分为 8.29（满分 10），而 Chat-GPT 的平均评分为 6.52。临床心理学家的评估表明，GPT-4 在产生临床相关和同理心反应方面更有效，从而为潜在用户提供更好的支持和指导。结论：这项研究为越来越多的关于大语言模型在精神卫生保健环境中的适用性的文献做出了贡献。研究结果强调了该领域持续研究和开发以优化这些模型以供临床使用的重要性。有必要进行进一步的调查，以了解两种模型之间表现差异背后的具体因素，并探索它们在不同人群和心理健康状况下的普遍性。

Title: Prompting-based Synthetic Data Generation for Few-Shot Question Answering

• Authors: Maximilian Schmidt, Andrea Bartezzaghi, Ngoc Thang Vu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.09335
• Pdf URL: https://arxiv.org/pdf/2405.09335
• Copy Paste: [[2405.09335]] Prompting-based Synthetic Data Generation for Few-Shot Question Answering(https://arxiv.org/abs/2405.09335)
• Keywords: language model, prompt
• Abstract: Although language models (LMs) have boosted the performance of Question Answering, they still need plenty of data. Data annotation, in contrast, is a time-consuming process. This especially applies to Question Answering, where possibly large documents have to be parsed and annotated with questions and their corresponding answers. Furthermore, Question Answering models often only work well for the domain they were trained on. Since annotation is costly, we argue that domain-agnostic knowledge from LMs, such as linguistic understanding, is sufficient to create a well-curated dataset. With this motivation, we show that using large language models can improve Question Answering performance on various datasets in the few-shot setting compared to state-of-the-art approaches. For this, we perform data generation leveraging the Prompting framework, suggesting that language models contain valuable task-agnostic knowledge that can be used beyond the common pre-training/fine-tuning scheme. As a result, we consistently outperform previous approaches on few-shot Question Answering.
• 摘要：尽管语言模型（LM）提高了问答的性能，但它们仍然需要大量数据。相比之下，数据注释是一个耗时的过程。这尤其适用于问答，其中可能必须解析大型文档并用问题及其相应的答案进行注释。此外，问答模型通常只适用于它们所训练的领域。由于注释成本高昂，我们认为来自 LM 的领域不可知知识（例如语言理解）足以创建精心策划的数据集。出于这种动机，我们表明，与最先进的方法相比，使用大型语言模型可以提高少量镜头设置中各种数据集的问答性能。为此，我们利用提示框架执行数据生成，表明语言模型包含有价值的与任务无关的知识，可以在常见的预训练/微调方案之外使用。因此，我们在小样本问答方面始终优于以前的方法。

Title: Large Language Model Bias Mitigation from the Perspective of Knowledge Editing

• Authors: Ruizhe Chen, Yichen Li, Zikai Xiao, Zuozhu Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.09341
• Pdf URL: https://arxiv.org/pdf/2405.09341
• Copy Paste: [[2405.09341]] Large Language Model Bias Mitigation from the Perspective of Knowledge Editing(https://arxiv.org/abs/2405.09341)
• Keywords: language model, llm
• Abstract: Existing debiasing methods inevitably make unreasonable or undesired predictions as they are designated and evaluated to achieve parity across different social groups but leave aside individual facts, resulting in modified existing knowledge. In this paper, we first establish a new bias mitigation benchmark BiasKE leveraging existing and additional constructed datasets, which systematically assesses debiasing performance by complementary metrics on fairness, specificity, and generalization. Meanwhile, we propose a novel debiasing method, Fairness Stamp (FAST), which enables editable fairness through fine-grained calibration on individual biased knowledge. Comprehensive experiments demonstrate that FAST surpasses state-of-the-art baselines with remarkable debiasing performance while not hampering overall model capability for knowledge preservation, highlighting the prospect of fine-grained debiasing strategies for editable fairness in LLMs.
• 摘要：现有的去偏见方法不可避免地会做出不合理或不受欢迎的预测，因为它们被指定和评估以实现不同社会群体之间的平等，但忽略了个体事实，导致现有知识被修改。在本文中，我们首先利用现有和额外构建的数据集建立了一个新的偏差缓解基准 BiasKE，该数据集通过公平性、特异性和泛化性的补充指标系统地评估去偏差性能。同时，我们提出了一种新颖的去偏见方法——公平邮票（FAST），它通过对个体偏见知识的细粒度校准来实现可编辑的公平性。综合实验表明，FAST 凭借出色的去偏性能超越了最先进的基线，同时不妨碍知识保存的整体模型能力，凸显了细粒度去偏策略在法学硕士中实现可编辑公平性的前景。

Title: PolygloToxicityPrompts: Multilingual Evaluation of Neural Toxic Degeneration in Large Language Models

• Authors: Devansh Jain, Priyanshu Kumar, Samuel Gehman, Xuhui Zhou, Thomas Hartvigsen, Maarten Sap
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.09373
• Pdf URL: https://arxiv.org/pdf/2405.09373
• Copy Paste: [[2405.09373]] PolygloToxicityPrompts: Multilingual Evaluation of Neural Toxic Degeneration in Large Language Models(https://arxiv.org/abs/2405.09373)
• Keywords: language model, llm, prompt
• Abstract: Recent advances in large language models (LLMs) have led to their extensive global deployment, and ensuring their safety calls for comprehensive and multilingual toxicity evaluations. However, existing toxicity benchmarks are overwhelmingly focused on English, posing serious risks to deploying LLMs in other languages. We address this by introducing PolygloToxicityPrompts (PTP), the first large-scale multilingual toxicity evaluation benchmark of 425K naturally occurring prompts spanning 17 languages. We overcome the scarcity of naturally occurring toxicity in web-text and ensure coverage across languages with varying resources by automatically scraping over 100M web-text documents. Using PTP, we investigate research questions to study the impact of model size, prompt language, and instruction and preference-tuning methods on toxicity by benchmarking over 60 LLMs. Notably, we find that toxicity increases as language resources decrease or model size increases. Although instruction- and preference-tuning reduce toxicity, the choice of preference-tuning method does not have any significant impact. Our findings shed light on crucial shortcomings of LLM safeguarding and highlight areas for future research.
• 摘要：大语言模型 (LLM) 的最新进展导致其在全球范围内广泛部署，并确保其安全性需要进行全面的多语言毒性评估。然而，现有的毒性基准绝大多数集中在英语上，这给以其他语言部署法学硕士带来了严重风险。我们通过引入 PolygloToxicityPrompts (PTP) 来解决这个问题，这是第一个大规模多语言毒性评估基准，涵盖 17 种语言的 425K 自然出现的提示。我们克服了网络文本中自然产生的毒性的稀缺性，并通过自动抓取超过 1 亿个网络文本文档来确保覆盖不同资源的语言。使用 PTP，我们通过对 60 多个法学硕士进行基准测试，调查研究问题，以研究模型大小、提示语言以及指令和偏好调整方法对毒性的影响。值得注意的是，我们发现随着语言资源的减少或模型大小的增加，毒性会增加。尽管指令和偏好调整可以降低毒性，但偏好调整方法的选择不会产生任何显着影响。我们的研究结果揭示了法学硕士保护的关键缺陷，并突出了未来研究的领域。

Title: Facilitating Opinion Diversity through Hybrid NLP Approaches

• Authors: Michiel van der Meer
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.09439
• Pdf URL: https://arxiv.org/pdf/2405.09439
• Copy Paste: [[2405.09439]] Facilitating Opinion Diversity through Hybrid NLP Approaches(https://arxiv.org/abs/2405.09439)
• Keywords: language model
• Abstract: Modern democracies face a critical issue of declining citizen participation in decision-making. Online discussion forums are an important avenue for enhancing citizen participation. This thesis proposal 1) identifies the challenges involved in facilitating large-scale online discussions with Natural Language Processing (NLP), 2) suggests solutions to these challenges by incorporating hybrid human-AI technologies, and 3) investigates what these technologies can reveal about individual perspectives in online discussions. We propose a three-layered hierarchy for representing perspectives that can be obtained by a mixture of human intelligence and large language models. We illustrate how these representations can draw insights into the diversity of perspectives and allow us to investigate interactions in online discussions.
• 摘要：现代民主国家面临着公民决策参与度下降的关键问题。在线论坛是增强公民参与的重要途径。本论文提案 1) 确定了利用自然语言处理 (NLP) 促进大规模在线讨论所涉及的挑战，2) 通过结合混合人类人工智能技术提出了解决这些挑战的解决方案，以及 3) 调查了这些技术可以揭示个人的哪些信息在线讨论中的观点。我们提出了一个三层层次结构来表示可以通过人类智能和大型语言模型的混合获得的观点。我们说明了这些表征如何能够深入了解观点的多样性，并使我们能够调查在线讨论中的互动。

Title: Tell Me Why: Explainable Public Health Fact-Checking with Large Language Models

• Authors: Majid Zarharan, Pascal Wullschleger, Babak Behkam Kia, Mohammad Taher Pilehvar, Jennifer Foster
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.09454
• Pdf URL: https://arxiv.org/pdf/2405.09454
• Copy Paste: [[2405.09454]] Tell Me Why: Explainable Public Health Fact-Checking with Large Language Models(https://arxiv.org/abs/2405.09454)
• Keywords: language model, gpt, prompt
• Abstract: This paper presents a comprehensive analysis of explainable fact-checking through a series of experiments, focusing on the ability of large language models to verify public health claims and provide explanations or justifications for their veracity assessments. We examine the effectiveness of zero/few-shot prompting and parameter-efficient fine-tuning across various open and closed-source models, examining their performance in both isolated and joint tasks of veracity prediction and explanation generation. Importantly, we employ a dual evaluation approach comprising previously established automatic metrics and a novel set of criteria through human evaluation. Our automatic evaluation indicates that, within the zero-shot scenario, GPT-4 emerges as the standout performer, but in few-shot and parameter-efficient fine-tuning contexts, open-source models demonstrate their capacity to not only bridge the performance gap but, in some instances, surpass GPT-4. Human evaluation reveals yet more nuance as well as indicating potential problems with the gold explanations.
• 摘要：本文通过一系列实验对可解释的事实核查进行了全面分析，重点关注大型语言模型验证公共卫生声明并为其准确性评估提供解释或理由的能力。我们检查了各种开源和闭源模型中零/少样本提示和参数高效微调的有效性，检查它们在准确性预测和解释生成的孤立和联合任务中的性能。重要的是，我们采用了双重评估方法，包括先前建立的自动指标和通过人工评估得出的一套新颖的标准。我们的自动评估表明，在零样本场景中，GPT-4 表现出色，但在少样本和参数高效的微调环境中，开源模型展示了其不仅能够弥合性能差距的能力但在某些情况下，超越了 GPT-4。人类评估揭示了更多细微差别，并表明黄金解释的潜在问题。

Title: Beyond Flesch-Kincaid: Prompt-based Metrics Improve Difficulty Classification of Educational Texts

• Authors: Donya Rooein, Paul Rottger, Anastassia Shaitarova, Dirk Hovy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.09482
• Pdf URL: https://arxiv.org/pdf/2405.09482
• Copy Paste: [[2405.09482]] Beyond Flesch-Kincaid: Prompt-based Metrics Improve Difficulty Classification of Educational Texts(https://arxiv.org/abs/2405.09482)
• Keywords: language model, llm, prompt
• Abstract: Using large language models (LLMs) for educational applications like dialogue-based teaching is a hot topic. Effective teaching, however, requires teachers to adapt the difficulty of content and explanations to the education level of their students. Even the best LLMs today struggle to do this well. If we want to improve LLMs on this adaptation task, we need to be able to measure adaptation success reliably. However, current Static metrics for text difficulty, like the Flesch-Kincaid Reading Ease score, are known to be crude and brittle. We, therefore, introduce and evaluate a new set of Prompt-based metrics for text difficulty. Based on a user study, we create Prompt-based metrics as inputs for LLMs. They leverage LLM's general language understanding capabilities to capture more abstract and complex features than Static metrics. Regression experiments show that adding our Prompt-based metrics significantly improves text difficulty classification over Static metrics alone. Our results demonstrate the promise of using LLMs to evaluate text adaptation to different education levels.
• 摘要：将大型语言模型 (LLM) 用于基于对话的教学等教育应用是一个热门话题。然而，有效的教学需要教师根据学生的教育水平调整内容和解释的难度。即使是当今最好的法学硕士也很难做到这一点。如果我们想提高法学硕士在这项适应任务上的水平，我们需要能够可靠地衡量适应成功。然而，目前文本难度的静态指标（例如 Flesch-Kincaid 阅读轻松度分数）被认为是粗糙且脆弱的。因此，我们引入并评估了一组新的基于提示的文本难度指标。根据用户研究，我们创建基于提示的指标作为法学硕士的输入。他们利用LLM的通用语言理解能力来捕获比静态指标更抽象和更复杂的特征。回归实验表明，与单独的静态指标相比，添加基于提示的指标可显着提高文本难度分类。我们的结果证明了使用法学硕士来评估不同教育水平的文本适应性的前景。

Title: Modeling Bilingual Sentence Processing: Evaluating RNN and Transformer Architectures for Cross-Language Structural Priming

• Authors: Bushi Xiao, Chao Gao, Demi Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.09508
• Pdf URL: https://arxiv.org/pdf/2405.09508
• Copy Paste: [[2405.09508]] Modeling Bilingual Sentence Processing: Evaluating RNN and Transformer Architectures for Cross-Language Structural Priming(https://arxiv.org/abs/2405.09508)
• Keywords: language model, llm
• Abstract: This study evaluates the performance of Recurrent Neural Network (RNN) and Transformer in replicating cross-language structural priming: a key indicator of abstract grammatical representations in human language processing. Focusing on Chinese-English priming, which involves two typologically distinct languages, we examine how these models handle the robust phenomenon of structural priming, where exposure to a particular sentence structure increases the likelihood of selecting a similar structure subsequently. Additionally, we utilize large language models (LLM) to measure the cross-lingual structural priming effect. Our findings indicate that Transformer outperform RNN in generating primed sentence structures, challenging the conventional belief that human sentence processing primarily involves recurrent and immediate processing and suggesting a role for cue-based retrieval mechanisms. Overall, this work contributes to our understanding of how computational models may reflect human cognitive processes in multilingual contexts.
• 摘要：本研究评估了循环神经网络 (RNN) 和 Transformer 在复制跨语言结构启动方面的性能：人类语言处理中抽象语法表示的关键指标。我们重点关注汉英启动，其中涉及两种类型不同的语言，我们研究了这些模型如何处理结构启动的鲁棒现象，其中暴露于特定的句子结构会增加随后选择类似结构的可能性。此外，我们利用大型语言模型（LLM）来衡量跨语言结构启动效应。我们的研究结果表明，Transformer 在生成启动句子结构方面优于 RNN，挑战了人类句子处理主要涉及循环和立即处理的传统观念，并提出了基于线索的检索机制的作用。总的来说，这项工作有助于我们理解计算模型如何反映多语言环境中的人类认知过程。