2024-03-14

Title: Training Small Multimodal Models to Bridge Biomedical Competency Gap: A Case Study in Radiology Imaging

• Authors: Juan Manuel Zambrano Chaves, Shih-Cheng Huang, Yanbo Xu, Hanwen Xu, Naoto Usuyama, Sheng Zhang, Fei Wang, Yujia Xie, Mahmoud Khademi, Ziyi Yang, Hany Awadalla, Julia Gong, Houdong Hu, Jianwei Yang, Chunyuan Li, Jianfeng Gao, Yu Gu, Cliff Wong, Mu Wei, Tristan Naumann, Muhao Chen, Matthew P. Lungren, Serena Yeung-Levy, Curtis P. Langlotz, Sheng Wang, Hoifung Poon
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08002
• Pdf URL: https://arxiv.org/pdf/2403.08002
• Copy Paste: [[2403.08002]] Training Small Multimodal Models to Bridge Biomedical Competency Gap: A Case Study in Radiology Imaging(https://arxiv.org/abs/2403.08002)
• Keywords: gpt
• Abstract: The scaling laws and extraordinary performance of large foundation models motivate the development and utilization of such large models in biomedicine. However, despite early promising results on some biomedical benchmarks, there are still major challenges that need to be addressed before these models can be used in real-world applications. Frontier models such as GPT-4V still have major competency gaps in multimodal capabilities for biomedical applications. Moreover, pragmatic issues such as access, cost, latency, and compliance make it hard for clinicians to use privately-hosted state-of-the-art large models directly on private patient data. In this paper, we explore training open-source small multimodal models (SMMs) to bridge biomedical competency gaps for unmet clinical needs. To maximize data efficiency, we adopt a modular approach by incorporating state-of-the-art pre-trained models for image and text modalities, and focusing on training a lightweight adapter to ground each modality to the text embedding space. We conduct a comprehensive study of this approach on radiology imaging. For training, we assemble a large dataset with over 1 million image-text pairs. For evaluation, we propose a clinically driven novel approach using GPT-4 and demonstrate its parity with expert evaluation. We also study grounding qualitatively using attention. For best practice, we conduct a systematic ablation study on various choices in data engineering and multimodal training. The resulting LLaVA-Rad (7B) model attains state-of-the-art results on radiology tasks such as report generation and cross-modal retrieval, even outperforming much larger models such as GPT-4V and Med-PaLM M (84B). LLaVA-Rad is fast and can be run on a single V100 GPU in private settings, offering a promising state-of-the-art tool for real-world clinical applications.
• 摘要：大型基础模型的标度规律和非凡的性能激发了此类大型模型在生物医学中的开发和利用。然而，尽管一些生物医学基准的早期结果令人鼓舞，但在这些模型用于实际应用之前，仍然需要解决重大挑战。 GPT-4V 等前沿模型在生物医学应用的多模式功能方面仍然存在重大能力差距。此外，访问、成本、延迟和合规性等实用问题使得临床医生很难直接在私人患者数据上使用私人托管的最先进的大型模型。在本文中，我们探索训练开源小型多模式模型（SMM），以弥补未满足的临床需求的生物医学能力差距。为了最大限度地提高数据效率，我们采用模块化方法，结合最先进的图像和文本模态预训练模型，并专注于训练轻量级适配器，将每种模态植入文本嵌入空间。我们对这种放射成像方法进行了全面的研究。为了进行训练，我们组装了一个包含超过 100 万个图像文本对的大型数据集。为了进行评估，我们提出了一种使用 GPT-4 的临床驱动的新方法，并证明了其与专家评估的同等性。我们还利用注意力定性地研究接地。为了获得最佳实践，我们对数据工程和多模式训练中的各种选择进行了系统的消融研究。由此产生的 LLaVA-Rad (7B) 模型在报告生成和跨模式检索等放射学任务上获得了最先进的结果，甚至优于 GPT-4V 和 Med-PaLM M (84B) 等更大的模型。 LLaVA-Rad 速度很快，可以在私人环境中在单个 V100 GPU 上运行，为现实临床应用提供了一个有前途的最先进的工具。

Title: Pix2Pix-OnTheFly: Leveraging LLMs for Instruction-Guided Image Editing

• Authors: Rodrigo Santos, João Silva, António Branco
• Subjects: cs.CL, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08004
• Pdf URL: https://arxiv.org/pdf/2403.08004
• Copy Paste: [[2403.08004]] Pix2Pix-OnTheFly: Leveraging LLMs for Instruction-Guided Image Editing(https://arxiv.org/abs/2403.08004)
• Keywords: llm
• Abstract: The combination of language processing and image processing keeps attracting increased interest given recent impressive advances that leverage the combined strengths of both domains of research. Among these advances, the task of editing an image on the basis solely of a natural language instruction stands out as a most challenging endeavour. While recent approaches for this task resort, in one way or other, to some form of preliminary preparation, training or fine-tuning, this paper explores a novel approach: We propose a preparation-free method that permits instruction-guided image editing on the fly. This approach is organized along three steps properly orchestrated that resort to image captioning and DDIM inversion, followed by obtaining the edit direction embedding, followed by image editing proper. While dispensing with preliminary preparation, our approach demonstrates to be effective and competitive, outperforming recent, state of the art models for this task when evaluated on the MAGICBRUSH dataset.
• 摘要：鉴于最近利用两个研究领域的综合优势取得的令人印象深刻的进展，语言处理和图像处理的结合不断吸引着越来越多的兴趣。在这些进步中，仅基于自然语言指令编辑图像的任务是最具挑战性的工作。虽然这项任务的最新方法以某种方式诉诸于某种形式的初步准备、训练或微调，但本文探索了一种新颖的方法：我们提出了一种无需准备的方法，允许在飞。这种方法按照三个步骤进行组织，这些步骤经过适当的协调，首先使用图像字幕和 DDIM 反转，然后获得编辑方向嵌入，最后进行适当的图像编辑。虽然无需进行初步准备，但在 MAGICBRUSH 数据集上进行评估时，我们的方法证明是有效且有竞争力的，优于该任务的最新最先进模型。

Title: Debatrix: Multi-dimensinal Debate Judge with Iterative Chronological Analysis Based on LLM

• Authors: Jingcong Liang, Rong Ye, Meng Han, Ruofei Lai, Xinyu Zhang, Xuanjing Huang, Zhongyu Wei
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08010
• Pdf URL: https://arxiv.org/pdf/2403.08010
• Copy Paste: [[2403.08010]] Debatrix: Multi-dimensinal Debate Judge with Iterative Chronological Analysis Based on LLM(https://arxiv.org/abs/2403.08010)
• Keywords: language model, llm
• Abstract: How can we construct an automated debate judge to evaluate an extensive, vibrant, multi-turn debate? This task is challenging, as judging a debate involves grappling with lengthy texts, intricate argument relationships, and multi-dimensional assessments. At the same time, current research mainly focuses on short dialogues, rarely touching upon the evaluation of an entire debate. In this paper, by leveraging Large Language Models (LLMs), we propose Debatrix, which makes the analysis and assessment of multi-turn debates more aligned with majority preferences. Specifically, Debatrix features a vertical, iterative chronological analysis and a horizontal, multi-dimensional evaluation collaboration. To align with real-world debate scenarios, we introduced the PanelBench benchmark, comparing our system's performance to actual debate outcomes. The findings indicate a notable enhancement over directly using LLMs for debate evaluation. Source code and benchmark data are available online at https://github.com/ljcleo/Debatrix .
• 摘要：我们如何构建一个自动辩论法官来评估广泛、充满活力、多回合的辩论？这项任务具有挑战性，因为判断辩论涉及冗长的文本、错综复杂的论点关系和多维度的评估。同时，目前的研究主要集中在简短的对话上，很少涉及对整个辩论的评估。在本文中，通过利用大型语言模型（LLM），我们提出了 Debatrix，它使得多轮辩论的分析和评估更符合大多数人的偏好。具体来说，Debatrix 具有纵向、迭代的时序分析和横向、多维度的评估协作。为了与现实世界的辩论场景保持一致，我们引入了 PanelBench 基准，将我们的系统性能与实际辩论结果进行比较。研究结果表明，与直接使用法学硕士进行辩论评估相比，有显着的增强。源代码和基准数据可在线获取：https://github.com/ljcleo/Debatrix。

Title: Harnessing Artificial Intelligence to Combat Online Hate: Exploring the Challenges and Opportunities of Large Language Models in Hate Speech Detection

• Authors: Tharindu Kumarage, Amrita Bhattacharjee, Joshua Garland
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08035
• Pdf URL: https://arxiv.org/pdf/2403.08035
• Copy Paste: [[2403.08035]] Harnessing Artificial Intelligence to Combat Online Hate: Exploring the Challenges and Opportunities of Large Language Models in Hate Speech Detection(https://arxiv.org/abs/2403.08035)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) excel in many diverse applications beyond language generation, e.g., translation, summarization, and sentiment analysis. One intriguing application is in text classification. This becomes pertinent in the realm of identifying hateful or toxic speech -- a domain fraught with challenges and ethical dilemmas. In our study, we have two objectives: firstly, to offer a literature review revolving around LLMs as classifiers, emphasizing their role in detecting and classifying hateful or toxic content. Subsequently, we explore the efficacy of several LLMs in classifying hate speech: identifying which LLMs excel in this task as well as their underlying attributes and training. Providing insight into the factors that contribute to an LLM proficiency (or lack thereof) in discerning hateful content. By combining a comprehensive literature review with an empirical analysis, our paper strives to shed light on the capabilities and constraints of LLMs in the crucial domain of hate speech detection.
• 摘要：大型语言模型 (LLM) 在语言生成之外的许多不同应用中表现出色，例如翻译、摘要和情感分析。一种有趣的应用是文本分类。这在识别仇恨或有毒言论领域变得相关——这个领域充满了挑战和道德困境。在我们的研究中，我们有两个目标：首先，提供围绕法学硕士作为分类器的文献综述，强调他们在检测和分类仇恨或有毒内容方面的作用。随后，我们探讨了几位法学硕士在仇恨言论分类方面的功效：确定哪些法学硕士在这项任务中表现出色，以及他们的基本属性和培训。深入了解导致法学硕士熟练（或缺乏）识别仇恨内容的因素。通过将全面的文献综述与实证分析相结合，我们的论文致力于阐明法学硕士在仇恨言论检测这一关键领域的能力和限制。

Title: Big City Bias: Evaluating the Impact of Metropolitan Size on Computational Job Market Abilities of Language Models

• Authors: Charlie Campanella, Rob van der Goot
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08046
• Pdf URL: https://arxiv.org/pdf/2403.08046
• Copy Paste: [[2403.08046]] Big City Bias: Evaluating the Impact of Metropolitan Size on Computational Job Market Abilities of Language Models(https://arxiv.org/abs/2403.08046)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have emerged as a useful technology for job matching, for both candidates and employers. Job matching is often based on a particular geographic location, such as a city or region. However, LLMs have known biases, commonly derived from their training data. In this work, we aim to quantify the metropolitan size bias encoded within large language models, evaluating zero-shot salary, employer presence, and commute duration predictions in 384 of the United States' metropolitan regions. Across all benchmarks, we observe negative correlations between the metropolitan size and the performance of the LLMS, indicating that smaller regions are indeed underrepresented. More concretely, the smallest 10 metropolitan regions show upwards of 300% worse benchmark performance than the largest 10.
• 摘要：大语言模型 (LLM) 已成为一种对候选人和雇主而言有用的工作匹配技术。工作匹配通常基于特定的地理位置，例如城市或地区。然而，法学硕士存在已知的偏见，这些偏见通常源自他们的训练数据。在这项工作中，我们的目标是量化大型语言模型中编码的大都市规模偏差，评估美国 384 个大都市区的零样本工资、雇主存在和通勤时间预测。在所有基准中，我们观察到大都市规模与 LLMS 表现之间存在负相关性，这表明较小的地区确实代表性不足。更具体地说，最小的 10 个大都市地区的基准表现比最大的 10 个大都市地区差 300% 以上。

Title: Generating Clarification Questions for Disambiguating Contracts

• Authors: Anmol Singhal, Chirag Jain, Preethu Rose Anish, Arkajyoti Chakraborty, Smita Ghaisas
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08053
• Pdf URL: https://arxiv.org/pdf/2403.08053
• Copy Paste: [[2403.08053]] Generating Clarification Questions for Disambiguating Contracts(https://arxiv.org/abs/2403.08053)
• Keywords: gpt, prompt, chat
• Abstract: Enterprises frequently enter into commercial contracts that can serve as vital sources of project-specific requirements. Contractual clauses are obligatory, and the requirements derived from contracts can detail the downstream implementation activities that non-legal stakeholders, including requirement analysts, engineers, and delivery personnel, need to conduct. However, comprehending contracts is cognitively demanding and error-prone for such stakeholders due to the extensive use of Legalese and the inherent complexity of contract language. Furthermore, contracts often contain ambiguously worded clauses to ensure comprehensive coverage. In contrast, non-legal stakeholders require a detailed and unambiguous comprehension of contractual clauses to craft actionable requirements. In this work, we introduce a novel legal NLP task that involves generating clarification questions for contracts. These questions aim to identify contract ambiguities on a document level, thereby assisting non-legal stakeholders in obtaining the necessary details for eliciting requirements. This task is challenged by three core issues: (1) data availability, (2) the length and unstructured nature of contracts, and (3) the complexity of legal text. To address these issues, we propose ConRAP, a retrieval-augmented prompting framework for generating clarification questions to disambiguate contractual text. Experiments conducted on contracts sourced from the publicly available CUAD dataset show that ConRAP with ChatGPT can detect ambiguities with an F2 score of 0.87. 70% of the generated clarification questions are deemed useful by human evaluators.
• 摘要：企业经常签订商业合同，这些合同可以作为项目特定要求的重要来源。合同条款是强制性的，由合同衍生的需求可以详细说明非法律利益相关者（包括需求分析师、工程师和交付人员）需要进行的下游实施活动。然而，由于法律术语的广泛使用和合同语言固有的复杂性，理解合同对于这些利益相关者来说是认知上的要求很高并且容易出错。此外，合同通常包含措辞含糊的条款，以确保全面覆盖。相比之下，非法律利益相关者需要详细且明确地理解合同条款，以制定可操作的要求。在这项工作中，我们引入了一项新颖的法律 NLP 任务，其中涉及生成合同的澄清问题。这些问题旨在识别文件层面上的合同歧义，从而帮助非法律利益相关者获得提出要求所需的详细信息。这项任务面临三个核心问题的挑战：（1）数据可用性，（2）合同的长度和非结构化性质，以及（3）法律文本的复杂性。为了解决这些问题，我们提出了 ConRAP，这是一种检索增强的提示框架，用于生成澄清问题以消除合同文本的歧义。对来自公开可用的 CUAD 数据集的合同进行的实验表明，ConRAP 与 ChatGPT 可以检测歧义，F2 得分为 0.87。人类评估者认为 70% 生成的澄清问题有用。

Title: BAGEL: Bootstrapping Agents by Guiding Exploration with Language

• Authors: Shikhar Murty, Christopher Manning, Peter Shaw, Mandar Joshi, Kenton Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08140
• Pdf URL: https://arxiv.org/pdf/2403.08140
• Copy Paste: [[2403.08140]] BAGEL: Bootstrapping Agents by Guiding Exploration with Language(https://arxiv.org/abs/2403.08140)
• Keywords: language model, agent
• Abstract: Following natural language instructions by executing actions in digital environments (e.g. web-browsers and REST APIs) is a challenging task for language model (LM) agents. Unfortunately, LM agents often fail to generalize to new environments without human demonstrations. This work presents BAGEL, a method for bootstrapping LM agents without human supervision. BAGEL converts a seed set of randomly explored trajectories or synthetic instructions, into demonstrations, via round-trips between two noisy LM components: an LM labeler which converts a trajectory into a synthetic instruction, and a zero-shot LM agent which maps the synthetic instruction into a refined trajectory. By performing these round-trips iteratively, BAGEL quickly converts the initial distribution of trajectories towards those that are well-described by natural language. We use BAGEL demonstrations to adapt a zero shot LM agent at test time via in-context learning over retrieved demonstrations, and find improvements of over 2-13% absolute on ToolQA and MiniWob++, with up to 13x reduction in execution failures.
• 摘要：对于语言模型 (LM) 代理来说，通过在数字环境（例如网络浏览器和 REST API）中执行操作来遵循自然语言指令是一项具有挑战性的任务。不幸的是，如果没有人工演示，LM 代理通常无法推广到新环境。这项工作提出了 BAGEL，一种无需人工监督即可引导 LM 代理的方法。 BAGEL 通过两个噪声 LM 组件之间的往返，将随机探索的轨迹或合成指令的种子集转换为演示：LM 标记器将轨迹转换为合成指令，以及零样本 LM 代理将合成指令映射进入精细化轨迹。通过迭代执行这些往返，BAGEL 快速将轨迹的初始分布转换为自然语言可以很好描述的分布。我们使用 BAGEL 演示，通过对检索到的演示进行上下文学习，在测试时调整零样本 LM 代理，并发现 ToolQA 和 MiniWob++ 的绝对改进超过 2-13%，执行失败最多减少 13 倍。

Title: Embedded Translations for Low-resource Automated Glossing

• Authors: Changbing Yang, Garrett Nicolai, Miikka Silfverberg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08189
• Pdf URL: https://arxiv.org/pdf/2403.08189
• Copy Paste: [[2403.08189]] Embedded Translations for Low-resource Automated Glossing(https://arxiv.org/abs/2403.08189)
• Keywords: language model
• Abstract: We investigate automatic interlinear glossing in low-resource settings. We augment a hard-attentional neural model with embedded translation information extracted from interlinear glossed text. After encoding these translations using large language models, specifically BERT and T5, we introduce a character-level decoder for generating glossed output. Aided by these enhancements, our model demonstrates an average improvement of 3.97\%-points over the previous state of the art on datasets from the SIGMORPHON 2023 Shared Task on Interlinear Glossing. In a simulated ultra low-resource setting, trained on as few as 100 sentences, our system achieves an average 9.78\%-point improvement over the plain hard-attentional baseline. These results highlight the critical role of translation information in boosting the system's performance, especially in processing and interpreting modest data sources. Our findings suggest a promising avenue for the documentation and preservation of languages, with our experiments on shared task datasets indicating significant advancements over the existing state of the art.
• 摘要：我们研究资源匮乏环境中的自动行间光泽。我们使用从行间注释文本中提取的嵌入翻译信息来增强硬注意力神经模型。使用大型语言模型（特别是 BERT 和 T5）对这些翻译进行编码后，我们引入了字符级解码器来生成注释输出。在这些增强的帮助下，我们的模型在 SIGMORPHON 2023 线间光泽共享任务的数据集上表现出比之前最先进技术水平平均提高了 3.97% 点。在模拟的超低资源环境中，仅接受 100 个句子的训练，我们的系统比普通的硬注意力基线平均提高了 9.78% 点。这些结果凸显了翻译信息在提高系统性能方面的关键作用，特别是在处理和解释适度数据源方面。我们的发现为语言的记录和保存提供了一条有前途的途径，我们对共享任务数据集的实验表明比现有技术取得了重大进步。

Title: MoleculeQA: A Dataset to Evaluate Factual Accuracy in Molecular Comprehension

• Authors: Xingyu Lu, He Cao, Zijing Liu, Shengyuan Bai, Leqing Chen, Yuan Yao, Hai-Tao Zheng, Yu Li
• Subjects: cs.CL, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2403.08192
• Pdf URL: https://arxiv.org/pdf/2403.08192
• Copy Paste: [[2403.08192]] MoleculeQA: A Dataset to Evaluate Factual Accuracy in Molecular Comprehension(https://arxiv.org/abs/2403.08192)
• Keywords: language model, llm
• Abstract: Large language models are playing an increasingly significant role in molecular research, yet existing models often generate erroneous information, posing challenges to accurate molecular comprehension. Traditional evaluation metrics for generated content fail to assess a model's accuracy in molecular understanding. To rectify the absence of factual evaluation, we present MoleculeQA, a novel question answering (QA) dataset which possesses 62K QA pairs over 23K molecules. Each QA pair, composed of a manual question, a positive option and three negative options, has consistent semantics with a molecular description from authoritative molecular corpus. MoleculeQA is not only the first benchmark for molecular factual bias evaluation but also the largest QA dataset for molecular research. A comprehensive evaluation on MoleculeQA for existing molecular LLMs exposes their deficiencies in specific areas and pinpoints several particularly crucial factors for molecular understanding.
• 摘要：大型语言模型在分子研究中发挥着越来越重要的作用，但现有模型经常生成错误信息，对准确的分子理解提出了挑战。生成内容的传统评估指标无法评估模型在分子理解方面的准确性。为了纠正事实评估的缺失，我们提出了 MoleculeQA，这是一个新颖的问答 (QA) 数据集，拥有超过 23K 个分子的 62K 个 QA 对。每个问答对由一个手动问题、一个肯定选项和三个否定选项组成，与权威分子语料库中的分子描述具有一致的语义。 MoleculeQA 不仅是分子事实偏差评估的第一个基准，也是分子研究最大的 QA 数据集。对现有分子法学硕士的 MoleculeQA 进行的全面评估暴露了他们在特定领域的缺陷，并指出了分子理解的几个特别关键的因素。

Title: Large Language Models are Contrastive Reasoners

• Authors: Liang Yao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08211
• Pdf URL: https://arxiv.org/pdf/2403.08211
• Copy Paste: [[2403.08211]] Large Language Models are Contrastive Reasoners(https://arxiv.org/abs/2403.08211)
• Keywords: language model, gpt, llm, prompt
• Abstract: Prompting methods play a crucial role in enhancing the capabilities of pre-trained large language models (LLMs). We explore how contrastive prompting (CP) significantly improves the ability of large language models to perform complex reasoning. We demonstrate that LLMs are decent contrastive reasoners by simply adding "Let's give a correct and a wrong answer." before LLMs provide answers. Experiments on two large language models show that zero-shot contrastive prompting improves performance on a range of arithmetic, commonsense, and symbolic reasoning tasks without any hand-crafted few-shot examples, such as increasing the accuracy on GSM8K from 35.9% to 88.8% and AQUA-RAT from 41.3% to 62.2% with the state-of-the-art GPT-4 model. Our method not only surpasses zero-shot CoT and few-shot CoT in most arithmetic and commonsense reasoning tasks but also can seamlessly integrate with existing prompting methods, resulting in improved or comparable results when compared to state-of-the-art methods. Our code is available at https://github.com/yao8839836/cp
• 摘要：提示方法在增强预训练大语言模型（LLM）的能力方面发挥着至关重要的作用。我们探索对比提示（CP）如何显着提高大型语言模型执行复杂推理的能力。我们通过简单地添加“让我们给出正确和错误的答案”来证明法学硕士是不错的对比推理者。在法学硕士提供答案之前。在两个大型语言模型上的实验表明，零样本对比提示可以提高一系列算术、常识和符号推理任务的性能，而无需任何手工制作的少样本示例，例如将 GSM8K 上的准确率从 35.9% 提高到 88.8%使用最先进的 GPT-4 模型，AQUA-RAT 从 41.3% 提高到 62.2%。我们的方法不仅在大多数算术和常识推理任务中超越了零样本 CoT 和少样本 CoT，而且还可以与现有的提示方法无缝集成，从而与最先进的方法相比得到改进或可比的结果。我们的代码位于 https://github.com/yao8839836/cp

Title: Can Large Language Models Identify Authorship?

• Authors: Baixiang Huang, Canyu Chen, Kai Shu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08213
• Pdf URL: https://arxiv.org/pdf/2403.08213
• Copy Paste: [[2403.08213]] Can Large Language Models Identify Authorship?(https://arxiv.org/abs/2403.08213)
• Keywords: language model, llm
• Abstract: The ability to accurately identify authorship is crucial for verifying content authenticity and mitigating misinformation. Large Language Models (LLMs) have demonstrated exceptional capacity for reasoning and problem-solving. However, their potential in authorship analysis, encompassing authorship verification and attribution, remains underexplored. This paper conducts a comprehensive evaluation of LLMs in these critical tasks. Traditional studies have depended on hand-crafted stylistic features, whereas state-of-the-art approaches leverage text embeddings from pre-trained language models. These methods, which typically require fine-tuning on labeled data, often suffer from performance degradation in cross-domain applications and provide limited explainability. This work seeks to address three research questions: (1) Can LLMs perform zero-shot, end-to-end authorship verification effectively? (2) Are LLMs capable of accurately attributing authorship among multiple candidates authors (e.g., 10 and 20)? (3) How can LLMs provide explainability in authorship analysis, particularly through the role of linguistic features? Moreover, we investigate the integration of explicit linguistic features to guide LLMs in their reasoning processes. Our extensive assessment demonstrates LLMs' proficiency in both tasks without the need for domain-specific fine-tuning, providing insights into their decision-making via a detailed analysis of linguistic features. This establishes a new benchmark for future research on LLM-based authorship analysis. The code and data are available at https://github.com/baixianghuang/authorship-llm.
• 摘要：准确识别作者身份的能力对于验证内容真实性和减少错误信息至关重要。大型语言模型 (LLM) 已表现出卓越的推理和解决问题的能力。然而，它们在作者身份分析（包括作者身份验证和归因）方面的潜力仍未得到充分开发。本文对法学硕士在这些关键任务中的表现进行了全面评估。传统研究依赖于手工设计的风格特征，而最先进的方法则利用预先训练的语言模型中的文本嵌入。这些方法通常需要对标记数据进行微调，在跨域应用中常常会遇到性能下降的问题，并且可解释性有限。这项工作旨在解决三个研究问题：（1）法学硕士能否有效地进行零样本、端到端的作者身份验证？ (2) 法学硕士是否能够在多个候选作者（例如 10 和 20）之间准确归属作者身份？ (3) 法学硕士如何在作者身份分析中提供可解释性，特别是通过语言特征的作用？此外，我们研究了明确语言特征的整合，以指导法学硕士的推理过程。我们的广泛评估表明法学硕士对这两项任务的熟练程度，无需针对特定领域进行微调，通过对语言特征的详细分析提供对其决策的见解。这为未来基于法学硕士的作者身份分析研究建立了新的基准。代码和数据可在https://github.com/baifangang/authorship-llm获取。

Title: Boosting Disfluency Detection with Large Language Model as Disfluency Generator

• Authors: Zhenrong Cheng, Jiayan Guo, Hao Sun, Yan Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08229
• Pdf URL: https://arxiv.org/pdf/2403.08229
• Copy Paste: [[2403.08229]] Boosting Disfluency Detection with Large Language Model as Disfluency Generator(https://arxiv.org/abs/2403.08229)
• Keywords: language model, llm, prompt
• Abstract: Current disfluency detection methods heavily rely on costly and scarce human-annotated data. To tackle this issue, some approaches employ heuristic or statistical features to generate disfluent sentences, partially improving detection performance. However, these sentences often deviate from real-life scenarios, constraining overall model enhancement. In this study, we propose a lightweight data augmentation approach for disfluency detection, utilizing the superior generative and semantic understanding capabilities of large language model (LLM) to generate disfluent sentences as augmentation data. We leverage LLM to generate diverse and more realistic sentences guided by specific prompts, without the need for fine-tuning the LLM. Subsequently, we apply an uncertainty-aware data filtering approach to improve the quality of the generated sentences, utilized in training a small detection model for improved performance. Experiments using enhanced data yielded state-of-the-art results. The results showed that using a small amount of LLM-generated enhanced data can significantly improve performance, thereby further enhancing cost-effectiveness.
• 摘要：当前的不流畅检测方法严重依赖昂贵且稀缺的人工注释数据。为了解决这个问题，一些方法采用启发式或统计特征来生成不连贯的句子，部分提高了检测性能。然而，这些句子往往偏离现实生活场景，限制了整体模型的增强。在本研究中，我们提出了一种用于不流畅检测的轻量级数据增强方法，利用大语言模型（LLM）卓越的生成和语义理解能力来生成不流畅的句子作为增强数据。我们利用 LLM 在特定提示的指导下生成多样化且更真实的句子，而不需要对 LLM 进行微调。随后，我们应用不确定性感知数据过滤方法来提高生成句子的质量，用于训练小型检测模型以提高性能。使用增强数据的实验产生了最先进的结果。结果表明，使用少量LLM生成的增强数据可以显着提高性能，从而进一步增强成本效益。

Title: RECIPE4U: Student-ChatGPT Interaction Dataset in EFL Writing Education

• Authors: Jieun Han, Haneul Yoo, Junho Myung, Minsun Kim, Tak Yeon Lee, So-Yeon Ahn, Alice Oh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08272
• Pdf URL: https://arxiv.org/pdf/2403.08272
• Copy Paste: [[2403.08272]] RECIPE4U: Student-ChatGPT Interaction Dataset in EFL Writing Education(https://arxiv.org/abs/2403.08272)
• Keywords: gpt, llm, chat
• Abstract: The integration of generative AI in education is expanding, yet empirical analyses of large-scale and real-world interactions between students and AI systems still remain limited. Addressing this gap, we present RECIPE4U (RECIPE for University), a dataset sourced from a semester-long experiment with 212 college students in English as Foreign Language (EFL) writing courses. During the study, students engaged in dialogues with ChatGPT to revise their essays. RECIPE4U includes comprehensive records of these interactions, including conversation logs, students' intent, students' self-rated satisfaction, and students' essay edit histories. In particular, we annotate the students' utterances in RECIPE4U with 13 intention labels based on our coding schemes. We establish baseline results for two subtasks in task-oriented dialogue systems within educational contexts: intent detection and satisfaction estimation. As a foundational step, we explore student-ChatGPT interaction patterns through RECIPE4U and analyze them by focusing on students' dialogue, essay data statistics, and students' essay edits. We further illustrate potential applications of RECIPE4U dataset for enhancing the incorporation of LLMs in educational frameworks. RECIPE4U is publicly available at https://zeunie.github.io/RECIPE4U/.
• 摘要：生成式人工智能在教育中的整合正在扩大，但对学生与人工智能系统之间大规模和现实世界交互的实证分析仍然有限。为了解决这一差距，我们提出了 RECIPE4U（大学 RECIPE），这是一个数据集，来源于对 212 名英语作为外语 (EFL) 写作课程的大学生进行的为期一个学期的实验。在学习过程中，学生们与ChatGPT进行对话以修改他们的论文。 RECIPE4U 包含这些交互的全面记录，包括对话日志、学生的意图、学生的自评满意度以及学生的论文编辑历史。特别是，我们根据我们的编码方案用 13 个意图标签来注释 RECIPE4U 中学生的话语。我们为教育背景下面向任务的对话系统中的两个子任务建立了基线结果：意图检测和满意度估计。作为基础步骤，我们通过 RECIPE4U 探索学生与 ChatGPT 的交互模式，并通过关注学生的对话、论文数据统计和学生的论文编辑来分析它们。我们进一步说明了 RECIPE4U 数据集在加强法学硕士纳入教育框架方面的潜在应用。 RECIPE4U 可在 https://zeunie.github.io/RECIPE4U/ 上公开获取。

Title: Mastering Text, Code and Math Simultaneously via Fusing Highly Specialized Language Models

• Authors: Ning Ding, Yulin Chen, Ganqu Cui, Xingtai Lv, Ruobing Xie, Bowen Zhou, Zhiyuan Liu, Maosong Sun
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08281
• Pdf URL: https://arxiv.org/pdf/2403.08281
• Copy Paste: [[2403.08281]] Mastering Text, Code and Math Simultaneously via Fusing Highly Specialized Language Models(https://arxiv.org/abs/2403.08281)
• Keywords: language model, llm, chat
• Abstract: Underlying data distributions of natural language, programming code, and mathematical symbols vary vastly, presenting a complex challenge for large language models (LLMs) that strive to achieve high performance across all three domains simultaneously. Achieving a very high level of proficiency for an LLM within a specific domain often requires extensive training with relevant corpora, which is typically accompanied by a sacrifice in performance in other domains. In this paper, we propose to fuse models that are already highly-specialized directly. The proposed fusing framework, UltraFuser, consists of three distinct specialists that are already sufficiently trained on language, coding, and mathematics. A token-level gating mechanism is introduced to blend the specialists' outputs. A two-stage training strategy accompanied by balanced sampling is designed to ensure stability. To effectively train the fused model, we further construct a high-quality supervised instruction tuning dataset, UltraChat 2, which includes text, code, and mathematical content. This dataset comprises approximately 300,000 instructions and covers a wide range of topics in each domain. Experiments show that our model could simultaneously achieve mastery of the three crucial domains.
• 摘要：自然语言、编程代码和数学符号的底层数据分布差异很大，这对努力同时在所有三个领域实现高性能的大型语言模型 (LLM) 提出了复杂的挑战。在特定领域内实现法学硕士的非常高水平的熟练程度通常需要使用相关语料库进行广泛的培训，这通常伴随着其他领域表现的牺牲。在本文中，我们建议直接融合已经高度专业化的模型。所提出的融合框架 UltraFuser 由三位不同的专家组成，他们已经在语言、编码和数学方面接受过充分的培训。引入代币级别的门控机制来混合专家的输出。设计了两阶段训练策略并辅以平衡采样以确保稳定性。为了有效地训练融合模型，我们进一步构建了高质量的监督指令调优数据集 UltraChat 2，其中包括文本、代码和数学内容。该数据集包含大约 300,000 条指令，涵盖每个领域的广泛主题。实验表明，我们的模型可以同时掌握三个关键领域。

Title: Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale

• Authors: Xiang Hu, Pengyu Ji, Qingyang Zhu, Wei Wu, Kewei Tu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08293
• Pdf URL: https://arxiv.org/pdf/2403.08293
• Copy Paste: [[2403.08293]] Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale(https://arxiv.org/abs/2403.08293)
• Keywords: language model, gpt
• Abstract: A syntactic language model (SLM) incrementally generates a sentence with its syntactic tree in a left-to-right manner. We present Generative Pretrained Structured Transformers (GPST), an unsupervised SLM at scale capable of being pre-trained from scratch on raw texts with high parallelism. GPST circumvents the limitations of previous SLMs such as relying on gold trees and sequential training. It consists of two components, a usual SLM supervised by a uni-directional language modeling loss, and an additional composition model, which induces syntactic parse trees and computes constituent representations, supervised by a bi-directional language modeling loss. We propose a representation surrogate to enable joint parallel training of the two models in a hard-EM fashion. We pre-train GPST on OpenWebText, a corpus with $9$ billion tokens, and demonstrate the superiority of GPST over GPT-2 with a comparable size in numerous tasks covering both language understanding and language generation. Meanwhile, GPST also significantly outperforms existing unsupervised SLMs on left-to-right grammar induction, while holding a substantial acceleration on training.
• 摘要：句法语言模型（SLM）通过其句法树以从左到右的方式增量生成句子。我们提出了生成式预训练结构化变压器（GPST），这是一种大规模的无监督 SLM，能够在高度并行性的原始文本上从头开始进行预训练。 GPST 规避了之前 SLM 的局限性，例如依赖金树和顺序训练。它由两个组件组成，一个是由单向语言建模损失监督的常用 SLM，另一个是附加的组合模型，该模型诱导句法解析树并计算成分表示，由双向语言建模损失监督。我们提出了一种表示代理，以便能够以硬电磁方式联合并行训练两个模型。我们在 OpenWebText（一个拥有 9 亿美元代币的语料库）上预训练 GPST，并在涵盖语言理解和语言生成的众多任务中展示了 GPST 相对于 GPT-2 的优越性，且规模相当。同时，GPST 在从左到右的语法归纳方面也显着优于现有的无监督 SLM，同时在训练方面保持了显着的加速。

Title: Gemma: Open Models Based on Gemini Research and Technology

• Authors: Gemma Team: Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, et al. (59 additional authors not shown)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08295
• Pdf URL: https://arxiv.org/pdf/2403.08295
• Copy Paste: [[2403.08295]] Gemma: Open Models Based on Gemini Research and Technology(https://arxiv.org/abs/2403.08295)
• Keywords: llm
• Abstract: This work introduces Gemma, a family of lightweight, state-of-the art open models built from the research and technology used to create Gemini models. Gemma models demonstrate strong performance across academic benchmarks for language understanding, reasoning, and safety. We release two sizes of models (2 billion and 7 billion parameters), and provide both pretrained and fine-tuned checkpoints. Gemma outperforms similarly sized open models on 11 out of 18 text-based tasks, and we present comprehensive evaluations of safety and responsibility aspects of the models, alongside a detailed description of model development. We believe the responsible release of LLMs is critical for improving the safety of frontier models, and for enabling the next wave of LLM innovations.
• 摘要：这项工作介绍了 Gemma，这是一个轻量级、最先进的开放模型系列，由用于创建 Gemini 模型的研究和技术构建而成。 Gemma 模型在语言理解、推理和安全方面的学术基准上表现出了强劲的性能。我们发布了两种规模的模型（20 亿和 70 亿个参数），并提供预训练和微调的检查点。 Gemma 在 18 项基于文本的任务中的 11 项上优于类似规模的开放模型，我们对模型的安全和责任方面进行了全面评估，并详细描述了模型开发。我们相信，负责任地发布法学硕士对于提高前沿模型的安全性以及实现下一波法学硕士创新至关重要。

Title: Towards Personalized Evaluation of Large Language Models with An Anonymous Crowd-Sourcing Platform

• Authors: Mingyue Cheng, Hao Zhang, Jiqian Yang, Qi Liu, Li Li, Xin Huang, Liwei Song, Zhi Li, Zhenya Huang, Enhong Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08305
• Pdf URL: https://arxiv.org/pdf/2403.08305
• Copy Paste: [[2403.08305]] Towards Personalized Evaluation of Large Language Models with An Anonymous Crowd-Sourcing Platform(https://arxiv.org/abs/2403.08305)
• Keywords: language model
• Abstract: Large language model evaluation plays a pivotal role in the enhancement of its capacity. Previously, numerous methods for evaluating large language models have been proposed in this area. Despite their effectiveness, these existing works mainly focus on assessing objective questions, overlooking the capability to evaluate subjective questions which is extremely common for large language models. Additionally, these methods predominantly utilize centralized datasets for evaluation, with question banks concentrated within the evaluation platforms themselves. Moreover, the evaluation processes employed by these platforms often overlook personalized factors, neglecting to consider the individual characteristics of both the evaluators and the models being evaluated. To address these limitations, we propose a novel anonymous crowd-sourcing evaluation platform, BingJian, for large language models that employs a competitive scoring mechanism where users participate in ranking models based on their performance. This platform stands out not only for its support of centralized evaluations to assess the general capabilities of models but also for offering an open evaluation gateway. Through this gateway, users have the opportunity to submit their questions, testing the models on a personalized and potentially broader range of capabilities. Furthermore, our platform introduces personalized evaluation scenarios, leveraging various forms of human-computer interaction to assess large language models in a manner that accounts for individual user preferences and contexts. The demonstration of BingJian can be accessed at https://github.com/Mingyue-Cheng/Bingjian.
• 摘要：大语言模型评估对于其能力的提升起着举足轻重的作用。此前，该领域已经提出了许多评估大型语言模型的方法。尽管它们很有效，但这些现有的工作主要集中在评估客观问题，忽视了评估主观问题的能力，这对于大型语言模型来说是极其常见的。此外，这些方法主要利用集中式数据集进行评估，问题库集中在评估平台本身内。此外，这些平台采用的评估流程往往忽视个性化因素，忽略考虑评估者和被评估模型的个体特征。为了解决这些限制，我们提出了一种新颖的匿名众包评估平台冰剑，用于大型语言模型，该平台采用竞争性评分机制，用户根据自己的表现参与对模型的排名。该平台的突出之处不仅在于支持集中评估以评估模型的一般能力，还在于提供开放的评估网关。通过此网关，用户有机会提交问题，并根据个性化且可能更广泛的功能测试模型。此外，我们的平台引入了个性化评估场景，利用各种形式的人机交互，以考虑个人用户偏好和上下文的方式评估大型语言模型。 Bingjian的演示可以访问https://github.com/Mingyue-Cheng/Bingjian。

Title: StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Losses

• Authors: Jia-Nan Li, Quan Tu, Cunli Mao, Zhengtao Yu, Ji-Rong Wen, Rui Yan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08312
• Pdf URL: https://arxiv.org/pdf/2403.08312
• Copy Paste: [[2403.08312]] StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Losses(https://arxiv.org/abs/2403.08312)
• Keywords: language model, llm, long context
• Abstract: Standard Large Language Models (LLMs) struggle with handling dialogues with long contexts due to efficiency and consistency issues. According to our observation, dialogue contexts are highly structured, and the special token of \textit{End-of-Utterance} (EoU) in dialogues has the potential to aggregate information. We refer to the EoU tokens as ``conversational attention sinks'' (conv-attn sinks). Accordingly, we introduce StreamingDialogue, which compresses long dialogue history into conv-attn sinks with minimal losses, and thus reduces computational complexity quadratically with the number of sinks (i.e., the number of utterances). Current LLMs already demonstrate the ability to handle long context window, e.g., a window size of 200k or more. To this end, by compressing utterances into EoUs, our method has the potential to handle more than 200k of utterances, resulting in a prolonged dialogue learning. In order to minimize information losses from reconstruction after compression, we design two learning strategies of short-memory reconstruction (SMR) and long-memory reactivation (LMR). Our method outperforms strong baselines in dialogue tasks and achieves a 4 $\times$ speedup while reducing memory usage by 18 $\times$ compared to dense attention recomputation.
• 摘要：由于效率和一致性问题，标准大型语言模型 (LLM) 难以处理长上下文对话。根据我们的观察，对话上下文是高度结构化的，对话中的特殊标记 \textit{End-of-Utterance} (EoU) 具有聚合信息的潜力。我们将 EoU 代币称为“会话注意力接收器”（conv-attn 接收器）。因此，我们引入了 StreamingDialogue，它以最小的损失将长对话历史压缩到 conv-attn 接收器中，从而以接收器数量（即话语数量）的二次方降低计算复杂度。目前的法学硕士已经展示了处理长上下文窗口的能力，例如 200k 或更大的窗口大小。为此，通过将话语压缩为 EoU，我们的方法有可能处理超过 200k 的话语，从而实现长时间的对话学习。为了最大限度地减少压缩后重建造成的信息损失，我们设计了短记忆重建（SMR）和长记忆重新激活（LMR）两种学习策略。与密集注意力重新计算相比，我们的方法在对话任务中优于强大的基线，并实现了 4 $\times$ 的加速，同时减少了 18 $\times$ 的内存使用量。

Title: Is Context Helpful for Chat Translation Evaluation?

• Authors: Sweta Agrawal, Amin Farajian, Patrick Fernandes, Ricardo Rei, André F.T. Martins
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08314
• Pdf URL: https://arxiv.org/pdf/2403.08314
• Copy Paste: [[2403.08314]] Is Context Helpful for Chat Translation Evaluation?(https://arxiv.org/abs/2403.08314)
• Keywords: language model, llm, chat
• Abstract: Despite the recent success of automatic metrics for assessing translation quality, their application in evaluating the quality of machine-translated chats has been limited. Unlike more structured texts like news, chat conversations are often unstructured, short, and heavily reliant on contextual information. This poses questions about the reliability of existing sentence-level metrics in this domain as well as the role of context in assessing the translation quality. Motivated by this, we conduct a meta-evaluation of existing sentence-level automatic metrics, primarily designed for structured domains such as news, to assess the quality of machine-translated chats. We find that reference-free metrics lag behind reference-based ones, especially when evaluating translation quality in out-of-English settings. We then investigate how incorporating conversational contextual information in these metrics affects their performance. Our findings show that augmenting neural learned metrics with contextual information helps improve correlation with human judgments in the reference-free scenario and when evaluating translations in out-of-English settings. Finally, we propose a new evaluation metric, Context-MQM, that utilizes bilingual context with a large language model (LLM) and further validate that adding context helps even for LLM-based evaluation metrics.
• 摘要：尽管最近用于评估翻译质量的自动指标取得了成功，但它们在评估机器翻译聊天质量方面的应用仍然受到限制。与新闻等结构化文本不同，聊天对话通常是非结构化的、简短的，并且严重依赖上下文信息。这对该领域现有句子级指标的可靠性以及上下文在评估翻译质量中的作用提出了疑问。受此启发，我们对现有的句子级自动指标（主要为新闻等结构化领域设计）进行了元评估，以评估机器翻译聊天的质量。我们发现无参考指标落后于基于参考的指标，尤其是在非英语环境中评估翻译质量时。然后，我们研究将对话上下文信息纳入这些指标如何影响他们的表现。我们的研究结果表明，利用上下文信息增强神经学习指标有助于提高在无参考场景中以及在非英语环境中评估翻译时与人类判断的相关性。最后，我们提出了一种新的评估指标 Context-MQM，它利用带有大语言模型 (LLM) 的双语上下文，并进一步验证添加上下文甚至对于基于 LLM 的评估指标也有帮助。

Title: Knowledge Conflicts for LLMs: A Survey

• Authors: Rongwu Xu, Zehan Qi, Cunxiang Wang, Hongru Wang, Yue Zhang, Wei Xu
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.08319
• Pdf URL: https://arxiv.org/pdf/2403.08319
• Copy Paste: [[2403.08319]] Knowledge Conflicts for LLMs: A Survey(https://arxiv.org/abs/2403.08319)
• Keywords: language model, llm
• Abstract: This survey provides an in-depth analysis of knowledge conflicts for large language models (LLMs), highlighting the complex challenges they encounter when blending contextual and parametric knowledge. Our focus is on three categories of knowledge conflicts: context-memory, inter-context, and intra-memory conflict. These conflicts can significantly impact the trustworthiness and performance of LLMs, especially in real-world applications where noise and misinformation are common. By categorizing these conflicts, exploring the causes, examining the behaviors of LLMs under such conflicts, and reviewing available solutions, this survey aims to shed light on strategies for improving the robustness of LLMs, thereby serving as a valuable resource for advancing research in this evolving area.
• 摘要：这项调查对大型语言模型 (LLM) 的知识冲突进行了深入分析，强调了他们在混合上下文知识和参数知识时遇到的复杂挑战。我们的重点是三类知识冲突：上下文记忆冲突、上下文间冲突和内存内冲突。这些冲突可能会严重影响法学硕士的可信度和表现，尤其是在噪音和错误信息普遍存在的现实应用中。通过对这些冲突进行分类、探索原因、检查法学硕士在此类冲突下的行为以及审查可用的解决方案，本调查旨在阐明提高法学硕士稳健性的策略，从而为推进这一不断发展的研究提供宝贵的资源。区域。

Title: Autoregressive Score Generation for Multi-trait Essay Scoring

• Authors: Heejin Do, Yunsu Kim, Gary Geunbae Lee
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08332
• Pdf URL: https://arxiv.org/pdf/2403.08332
• Copy Paste: [[2403.08332]] Autoregressive Score Generation for Multi-trait Essay Scoring(https://arxiv.org/abs/2403.08332)
• Keywords: prompt
• Abstract: Recently, encoder-only pre-trained models such as BERT have been successfully applied in automated essay scoring (AES) to predict a single overall score. However, studies have yet to explore these models in multi-trait AES, possibly due to the inefficiency of replicating BERT-based models for each trait. Breaking away from the existing sole use of encoder, we propose an autoregressive prediction of multi-trait scores (ArTS), incorporating a decoding process by leveraging the pre-trained T5. Unlike prior regression or classification methods, we redefine AES as a score-generation task, allowing a single model to predict multiple scores. During decoding, the subsequent trait prediction can benefit by conditioning on the preceding trait scores. Experimental results proved the efficacy of ArTS, showing over 5% average improvements in both prompts and traits.
• 摘要：最近，仅编码器预训练的模型（例如 BERT）已成功应用于自动论文评分（AES）以预测单个总分。然而，研究尚未在多特征 AES 中探索这些模型，这可能是由于针对每个特征复制基于 BERT 的模型效率低下。摆脱现有编码器的单独使用，我们提出了多特征分数（ArTS）的自回归预测，通过利用预训练的 T5 结合解码过程。与之前的回归或分类方法不同，我们将 AES 重新定义为分数生成任务，允许单个模型预测多个分数。在解码过程中，后续的性状预测可以通过调节先前的性状得分而受益。实验结果证明了 ArTS 的有效性，在提示和特征方面平均提高了 5% 以上。

Title: From human experts to machines: An LLM supported approach to ontology and knowledge graph construction

• Authors: Vamsi Krishna Kommineni, Birgitta König-Ries, Sheeba Samuel
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08345
• Pdf URL: https://arxiv.org/pdf/2403.08345
• Copy Paste: [[2403.08345]] From human experts to machines: An LLM supported approach to ontology and knowledge graph construction(https://arxiv.org/abs/2403.08345)
• Keywords: language model, llm
• Abstract: The conventional process of building Ontologies and Knowledge Graphs (KGs) heavily relies on human domain experts to define entities and relationship types, establish hierarchies, maintain relevance to the domain, fill the ABox (or populate with instances), and ensure data quality (including amongst others accuracy and completeness). On the other hand, Large Language Models (LLMs) have recently gained popularity for their ability to understand and generate human-like natural language, offering promising ways to automate aspects of this process. This work explores the (semi-)automatic construction of KGs facilitated by open-source LLMs. Our pipeline involves formulating competency questions (CQs), developing an ontology (TBox) based on these CQs, constructing KGs using the developed ontology, and evaluating the resultant KG with minimal to no involvement of human experts. We showcase the feasibility of our semi-automated pipeline by creating a KG on deep learning methodologies by exploiting scholarly publications. To evaluate the answers generated via Retrieval-Augmented-Generation (RAG) as well as the KG concepts automatically extracted using LLMs, we design a judge LLM, which rates the generated content based on ground truth. Our findings suggest that employing LLMs could potentially reduce the human effort involved in the construction of KGs, although a human-in-the-loop approach is recommended to evaluate automatically generated KGs.
• 摘要：构建本体和知识图 (KG) 的传统过程严重依赖人类领域专家来定义实体和关系类型、建立层次结构、维护与领域的相关性、填充 ABox（或填充实例）并确保数据质量（包括其中包括准确性和完整性）。另一方面，大型语言模型（LLM）最近因其理解和生成类似人类的自然语言的能力而受到欢迎，为该过程的各个方面的自动化提供了有前途的方法。这项工作探索了由开源法学硕士促进的知识图谱的（半）自动构建。我们的流程包括制定能力问题（CQ）、基于这些 CQ 开发本体（TBox）、使用开发的本体构建知识图谱，并在人类专家极少参与或不参与的情况下评估生成的知识图谱。我们利用学术出版物创建深度学习方法的知识图谱，展示半自动化管道的可行性。为了评估通过检索增强生成（RAG）生成的答案以及使用 LLM 自动提取的 KG 概念，我们设计了一个法官 LLM，它根据基本事实对生成的内容进行评分。我们的研究结果表明，尽管建议使用人机交互方法来评估自动生成的知识图谱，但采用法学硕士可能会减少构建知识图谱所需的人力。

Title: SMART: Submodular Data Mixture Strategy for Instruction Tuning

• Authors: H S V N S Kowndinya Renduchintala, Sumit Bhatia, Ganesh Ramakrishnan
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.08370
• Pdf URL: https://arxiv.org/pdf/2403.08370
• Copy Paste: [[2403.08370]] SMART: Submodular Data Mixture Strategy for Instruction Tuning(https://arxiv.org/abs/2403.08370)
• Keywords: language model
• Abstract: Instruction Tuning involves finetuning a language model on a collection of instruction-formatted datasets in order to enhance the generalizability of the model to unseen tasks. Studies have shown the importance of balancing different task proportions during finetuning, but finding the right balance remains challenging. Unfortunately, there's currently no systematic method beyond manual tuning or relying on practitioners' intuition. In this paper, we introduce SMART (Submodular data Mixture strAtegy for instRuction Tuning) - a novel data mixture strategy which makes use of a submodular function to assign importance scores to tasks which are then used to determine the mixture weights. Given a fine-tuning budget, SMART redistributes the budget among tasks and selects non-redundant samples from each task. Experimental results demonstrate that SMART significantly outperforms traditional methods such as examples proportional mixing and equal mixing. Furthermore, SMART facilitates the creation of data mixtures based on a few representative subsets of tasks alone and through task pruning analysis, we reveal that in a limited budget setting, allocating budget among a subset of representative tasks yields superior performance compared to distributing the budget among all tasks. The code for reproducing our results is open-sourced at https://github.com/kowndinya-renduchintala/SMART.
• 摘要：指令调优涉及在指令格式数据集集合上微调语言模型，以增强模型对未见过的任务的通用性。研究表明在微调过程中平衡不同任务比例的重要性，但找到正确的平衡仍然具有挑战性。不幸的是，除了手动调整或依赖从业者的直觉之外，目前还没有系统的方法。在本文中，我们介绍了 SMART（用于指令调整的子模数据混合策略）——一种新颖的数据混合策略，它利用子模函数为任务分配重要性分数，然后用于确定混合权重。给定微调预算，SMART 在任务之间重新分配预算，并从每个任务中选择非冗余样本。实验结果表明，SMART显着优于示例比例混合、等量混合等传统方法。此外，SMART 有助于仅基于几个代表性任务子集创建数据混合物，并通过任务修剪分析，我们发现在有限的预算设置中，在代表性任务子集之间分配预算与在多个代表性任务之间分配预算相比，可以产生更优异的性能。所有任务。用于重现我们结果的代码是开源的，位于 https://github.com/kowndinya-renduchintala/SMART。

Title: Learning to Describe for Predicting Zero-shot Drug-Drug Interactions

• Authors: Fangqi Zhu, Yongqi Zhang, Lei Chen, Bing Qin, Ruifeng Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08377
• Pdf URL: https://arxiv.org/pdf/2403.08377
• Copy Paste: [[2403.08377]] Learning to Describe for Predicting Zero-shot Drug-Drug Interactions(https://arxiv.org/abs/2403.08377)
• Keywords: language model
• Abstract: Adverse drug-drug interactions~(DDIs) can compromise the effectiveness of concurrent drug administration, posing a significant challenge in healthcare. As the development of new drugs continues, the potential for unknown adverse effects resulting from DDIs becomes a growing concern. Traditional computational methods for DDI prediction may fail to capture interactions for new drugs due to the lack of knowledge. In this paper, we introduce a new problem setup as zero-shot DDI prediction that deals with the case of new drugs. Leveraging textual information from online databases like DrugBank and PubChem, we propose an innovative approach TextDDI with a language model-based DDI predictor and a reinforcement learning~(RL)-based information selector, enabling the selection of concise and pertinent text for accurate DDI prediction on new drugs. Empirical results show the benefits of the proposed approach on several settings including zero-shot and few-shot DDI prediction, and the selected texts are semantically relevant. Our code and data are available at \url{https://github.com/zhufq00/DDIs-Prediction}.
• 摘要：药物间不良相互作用（DDI）可能会损害同时给药的有效性，给医疗保健带来重大挑战。随着新药开发的不断进行，DDI 可能产生的未知不良反应日益引起人们的关注。由于缺乏知识，传统的 DDI 预测计算方法可能无法捕获新药的相互作用。在本文中，我们引入了一种新的问题设置，即处理新药情况的零样本 DDI 预测。利用来自 DrugBank 和 PubChem 等在线数据库的文本信息，我们提出了一种创新方法 TextDDI，具有基于语言模型的 DDI 预测器和基于强化学习 (RL) 的信息选择器，能够选择简洁且相关的文本以进行准确的 DDI 预测关于新药。实证结果显示了所提出的方法在包括零样本和少样本 DDI 预测在内的多种设置上的优点，并且所选文本在语义上是相关的。我们的代码和数据可在 \url{https://github.com/zhufq00/DDIs-Prediction} 获取。

Title: Authorship Verification based on the Likelihood Ratio of Grammar Models

• Authors: Andrea Nini, Oren Halvani, Lukas Graner, Valerio Gherardi, Shunichi Ishihara
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.08462
• Pdf URL: https://arxiv.org/pdf/2403.08462
• Copy Paste: [[2403.08462]] Authorship Verification based on the Likelihood Ratio of Grammar Models(https://arxiv.org/abs/2403.08462)
• Keywords: language model
• Abstract: Authorship Verification (AV) is the process of analyzing a set of documents to determine whether they were written by a specific author. This problem often arises in forensic scenarios, e.g., in cases where the documents in question constitute evidence for a crime. Existing state-of-the-art AV methods use computational solutions that are not supported by a plausible scientific explanation for their functioning and that are often difficult for analysts to interpret. To address this, we propose a method relying on calculating a quantity we call $\lambda_G$ (LambdaG): the ratio between the likelihood of a document given a model of the Grammar for the candidate author and the likelihood of the same document given a model of the Grammar for a reference population. These Grammar Models are estimated using $n$-gram language models that are trained solely on grammatical features. Despite not needing large amounts of data for training, LambdaG still outperforms other established AV methods with higher computational complexity, including a fine-tuned Siamese Transformer network. Our empirical evaluation based on four baseline methods applied to twelve datasets shows that LambdaG leads to better results in terms of both accuracy and AUC in eleven cases and in all twelve cases if considering only topic-agnostic methods. The algorithm is also highly robust to important variations in the genre of the reference population in many cross-genre comparisons. In addition to these properties, we demonstrate how LambdaG is easier to interpret than the current state-of-the-art. We argue that the advantage of LambdaG over other methods is due to fact that it is compatible with Cognitive Linguistic theories of language processing.
• 摘要：作者身份验证 (AV) 是分析一组文档以确定它们是否由特定作者撰写的过程。这个问题经常出现在法医场景中，例如，当相关文件构成犯罪证据的情况下。现有最先进的 AV 方法使用的计算解决方案并没有对其功能进行合理的科学解释支持，而且分析师通常很难解释。为了解决这个问题，我们提出了一种方法，依赖于计算我们称为 $\lambda_G$ (LambdaG) 的量：给定候选作者的语法模型的文档的可能性与给定 a 的同一文档的可能性之间的比率参考人群的语法模型。这些语法模型是使用仅根据语法特征进行训练的 $n$-gram 语言模型来估计的。尽管不需要大量数据进行训练，LambdaG 仍然优于其他具有更高计算复杂性的现有 AV 方法，包括微调的 Siamese Transformer 网络。我们基于应用于 12 个数据集的四种基线方法进行的实证评估表明，如果仅考虑与主题无关的方法，则 LambdaG 在 11 种情况下以及在所有 12 种情况下在准确性和 AUC 方面都能带来更好的结果。在许多跨类型比较中，该算法对于参考人群类型的重要变化也具有高度鲁棒性。除了这些属性之外，我们还演示了 LambdaG 如何比当前最先进的技术更容易解释。我们认为 LambdaG 相对于其他方法的优势在于它与语言处理的认知语言理论兼容。

Title: Data-oriented Dynamic Fine-tuning Parameter Selection Strategy for FISH Mask based Efficient Fine-tuning

• Authors: Ming Dong, Kang Xue, Bolong Zheng, Tingting He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08484
• Pdf URL: https://arxiv.org/pdf/2403.08484
• Copy Paste: [[2403.08484]] Data-oriented Dynamic Fine-tuning Parameter Selection Strategy for FISH Mask based Efficient Fine-tuning(https://arxiv.org/abs/2403.08484)
• Keywords: language model, llm
• Abstract: In view of the huge number of parameters of Large language models (LLMs) , tuning all parameters is very costly, and accordingly fine-tuning specific parameters is more sensible. Most of parameter efficient fine-tuning (PEFT) concentrate on parameter selection strategies, such as additive method, selective method and reparametrization-based method. However, there are few methods that consider the impact of data samples on parameter selecting, such as Fish Mask based method. Fish Mask randomly choose a part of data samples and treat them equally during parameter selection, which is unable to dynamically select optimal parameters for inconstant data distributions. In this work, we adopt a data-oriented perspective, then proposing an IRD ($\mathrm{\underline I}$terative sample-parameter $\mathrm{\underline R}$ange $\mathrm{\underline D}$ecreasing) algorithm to search the best setting of sample-parameter pair for FISH Mask. In each iteration, by searching the set of samples and parameters with larger Fish information, IRD can find better sample-parameter pair in most scale. We demonstrate the effectiveness and rationality of proposed strategy by conducting experiments on GLUE benchmark. Experimental results show our strategy optimizes the parameter selection and achieves preferable performance.
• 摘要：鉴于大型语言模型（LLM）的参数数量巨大，调整所有参数的成本非常高，因此对特定参数进行微调更为明智。大多数参数有效微调（PEFT）集中在参数选择策略上，例如加性方法、选择性方法和基于重参数化的方法。然而，很少有方法考虑数据样本对参数选择的影响，例如基于 Fish Mask 的方法。 Fish Mask随机选择一部分数据样本，并在参数选择时一视同仁，无法针对不稳定的数据分布动态选择最优参数。在这项工作中，我们采用面向数据的视角，然后提出IRD ($\mathrm{\underline I}$terative样本参数$\mathrm{\underline R}$ange $\mathrm{\underline D}$ecreasing ）算法来搜索 FISH Mask 样本参数对的最佳设置。在每次迭代中，通过搜索具有较大Fish信息的样本和参数集合，IRD可以在大多数尺度上找到更好的样本参数对。我们通过在 GLUE 基准上进行实验来证明所提出策略的有效性和合理性。实验结果表明我们的策略优化了参数选择并取得了更好的性能。

Title: Rich Semantic Knowledge Enhanced Large Language Models for Few-shot Chinese Spell Checking

• Authors: Ming Dong, Yujing Chen, Miao Zhang, Hao Sun, Tingting He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08492
• Pdf URL: https://arxiv.org/pdf/2403.08492
• Copy Paste: [[2403.08492]] Rich Semantic Knowledge Enhanced Large Language Models for Few-shot Chinese Spell Checking(https://arxiv.org/abs/2403.08492)
• Keywords: language model, llm
• Abstract: Chinese Spell Checking (CSC) is a widely used technology, which plays a vital role in speech to text (STT) and optical character recognition (OCR). Most of the existing CSC approaches relying on BERT architecture achieve excellent performance. However, limited by the scale of the foundation model, BERT-based method does not work well in few-shot scenarios, showing certain limitations in practical applications. In this paper, we explore using an in-context learning method named RS-LLM (Rich Semantic based LLMs) to introduce large language models (LLMs) as the foundation model. Besides, we study the impact of introducing various Chinese rich semantic information in our framework. We found that by introducing a small number of specific Chinese rich semantic structures, LLMs achieve better performance than the BERT-based model on few-shot CSC task. Furthermore, we conduct experiments on multiple datasets, and the experimental results verified the superiority of our proposed framework.
• 摘要：中文拼写检查（CSC）是一项广泛使用的技术，在语音转文本（STT）和光学字符识别（OCR）中发挥着至关重要的作用。大多数现有的 CSC 方法依赖于 BERT 架构，取得了优异的性能。然而，受基础模型规模的限制，基于BERT的方法在少镜头场景下效果不佳，在实际应用中表现出一定的局限性。在本文中，我们探索使用名为 RS-LLM（基于丰富语义的 LLM）的上下文学习方法来引入大语言模型（LLM）作为基础模型。此外，我们研究了在我们的框架中引入各种中文丰富语义信息的影响。我们发现，通过引入少量特定的中文丰富语义结构，LLM 在少样本 CSC 任务上比基于 BERT 的模型取得了更好的性能。此外，我们在多个数据集上进行了实验，实验结果验证了我们提出的框架的优越性。

Title: Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator

• Authors: Yusheng Liao, Yutong Meng, Yuhao Wang, Hongcheng Liu, Yanfeng Wang, Yu Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08495
• Pdf URL: https://arxiv.org/pdf/2403.08495
• Copy Paste: [[2403.08495]] Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator(https://arxiv.org/abs/2403.08495)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated remarkable proficiency in human interactions, yet their application within the medical field remains insufficiently explored. Previous works mainly focus on the performance of medical knowledge with examinations, which is far from the realistic scenarios, falling short in assessing the abilities of LLMs on clinical tasks. In the quest to enhance the application of Large Language Models (LLMs) in healthcare, this paper introduces the Automated Interactive Evaluation (AIE) framework and the State-Aware Patient Simulator (SAPS), targeting the gap between traditional LLM evaluations and the nuanced demands of clinical practice. Unlike prior methods that rely on static medical knowledge assessments, AIE and SAPS provide a dynamic, realistic platform for assessing LLMs through multi-turn doctor-patient simulations. This approach offers a closer approximation to real clinical scenarios and allows for a detailed analysis of LLM behaviors in response to complex patient interactions. Our extensive experimental validation demonstrates the effectiveness of the AIE framework, with outcomes that align well with human evaluations, underscoring its potential to revolutionize medical LLM testing for improved healthcare delivery.
• 摘要：大型语言模型（LLM）在人类交互方面表现出了卓越的熟练程度，但其在医学领域的应用仍未得到充分探索。以往的工作主要集中在医学知识的考试表现上，与现实场景相距甚远，在评估法学硕士临床任务能力方面存在不足。为了加强大型语言模型（LLM）在医疗保健中的应用，本文介绍了自动交互式评估（AIE）框架和状态感知患者模拟器（SAPS），针对传统LLM评估与细致入微的需求之间的差距的临床实践。与依赖静态医学知识评估的现有方法不同，AIE 和 SAPS 提供了一个动态、真实的平台，用于通过多轮医患模拟评估法学硕士。这种方法更接近真实的临床场景，并允许对法学硕士行为进行详细分析，以应对复杂的患者互动。我们广泛的实验验证证明了 AIE 框架的有效性，其结果与人类评估非常一致，强调了其彻底改变医学法学硕士测试以改善医疗保健服务的潜力。

Title: Language models scale reliably with over-training and on downstream tasks

• Authors: Samir Yitzhak Gadre, Georgios Smyrnis, Vaishaal Shankar, Suchin Gururangan, Mitchell Wortsman, Rulin Shao, Jean Mercat, Alex Fang, Jeffrey Li, Sedrick Keh, Rui Xin, Marianna Nezhurina, Igor Vasiljevic, Jenia Jitsev, Alexandros G. Dimakis, Gabriel Ilharco, Shuran Song, Thomas Kollar, Yair Carmon, Achal Dave, Reinhard Heckel, Niklas Muennighoff, Ludwig Schmidt
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.08540
• Pdf URL: https://arxiv.org/pdf/2403.08540
• Copy Paste: [[2403.08540]] Language models scale reliably with over-training and on downstream tasks(https://arxiv.org/abs/2403.08540)
• Keywords: language model
• Abstract: Scaling laws are useful guides for developing language models, but there are still gaps between current scaling studies and how language models are ultimately trained and evaluated. For instance, scaling is usually studied in the compute-optimal training regime (i.e., "Chinchilla optimal" regime); however, in practice, models are often over-trained to reduce inference costs. Moreover, scaling laws mostly predict loss on next-token prediction, but ultimately models are compared based on downstream task performance. In this paper, we address both shortcomings. To do so, we create a testbed of 104 models with 0.011B to 6.9B parameters trained with various numbers of tokens on three data distributions. First, we investigate scaling in the over-trained regime. We fit scaling laws that extrapolate in both the number of model parameters and the ratio of training tokens to parameters. This enables us to predict the validation loss of a 1.4B parameter, 900B token run (i.e., 32$\times$ over-trained) and a 6.9B parameter, 138B token run$\unicode{x2014}$each from experiments that take 300$\times$ less compute. Second, we relate the perplexity of a language model to its downstream task performance via a power law. We use this law to predict top-1 error averaged over downstream tasks for the two aforementioned models using experiments that take 20$\times$ less compute. Our experiments are available at https://github.com/mlfoundations/scaling.
• 摘要：标度法则对于开发语言模型来说是有用的指南，但当前的标度研究与语言模型的最终训练和评估方式之间仍然存在差距。例如，通常在计算最优训练体系（即“龙猫最优”体系）中研究缩放；然而，在实践中，模型经常被过度训练以降低推理成本。此外，缩放法则主要预测下一个令牌预测的损失，但最终模型是根据下游任务性能进行比较的。在本文中，我们解决了这两个缺点。为此，我们创建了一个包含 104 个模型的测试床，参数为 0.011B 到 6.9B，并在三个数据分布上使用不同数量的标记进行训练。首先，我们研究过度训练体系中的扩展。我们拟合缩放法则，推断模型参数的数量和训练标记与参数的比率。这使我们能够预测 1.4B 参数、900B 令牌运行（即 32$\times$ 过度训练）和 6.9B 参数、138B 令牌运行$\unicode{x2014}$的验证损失，每个参数都来自于以下实验：计算量减少 300$\times$。其次，我们通过幂律将语言模型的复杂性与其下游任务性能联系起来。我们使用该定律来预测上述两个模型的下游任务的平均 top-1 误差，使用的实验所需的计算量减少了 20 倍。我们的实验可以在 https://github.com/mlfoundations/scaling 上找到。

Title: Non-discrimination Criteria for Generative Language Models

• Authors: Sara Sterlie, Nina Weng, Aasa Feragen
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2403.08564
• Pdf URL: https://arxiv.org/pdf/2403.08564
• Copy Paste: [[2403.08564]] Non-discrimination Criteria for Generative Language Models(https://arxiv.org/abs/2403.08564)
• Keywords: language model, prompt
• Abstract: Within recent years, generative AI, such as large language models, has undergone rapid development. As these models become increasingly available to the public, concerns arise about perpetuating and amplifying harmful biases in applications. Gender stereotypes can be harmful and limiting for the individuals they target, whether they consist of misrepresentation or discrimination. Recognizing gender bias as a pervasive societal construct, this paper studies how to uncover and quantify the presence of gender biases in generative language models. In particular, we derive generative AI analogues of three well-known non-discrimination criteria from classification, namely independence, separation and sufficiency. To demonstrate these criteria in action, we design prompts for each of the criteria with a focus on occupational gender stereotype, specifically utilizing the medical test to introduce the ground truth in the generative AI context. Our results address the presence of occupational gender bias within such conversational language models.
• 摘要：近年来，大型语言模型等生成式人工智能得到了快速发展。随着这些模型越来越多地向公众开放，人们开始担心应用程序中有害偏见的存在和扩大。性别刻板印象可能对他们所针对的个人有害并具有限制性，无论其是否包含歪曲或歧视。认识到性别偏见是一种普遍存在的社会结构，本文研究了如何发现和量化生成语言模型中性别偏见的存在。特别是，我们从分类中推导出三个著名的非歧视标准的生成人工智能类似物，即独立性、分离性和充分性。为了展示这些标准的实际应用，我们为每个标准设计了提示，重点关注职业性别刻板印象，特别是利用医学测试来介绍生成人工智能背景下的基本事实。我们的结果解决了此类会话语言模型中存在的职业性别偏见。

Title: Call Me When Necessary: LLMs can Efficiently and Faithfully Reason over Structured Environments

• Authors: Sitao Cheng, Ziyuan Zhuang, Yong Xu, Fangkai Yang, Chaoyun Zhang, Xiaoting Qin, Xiang Huang, Ling Chen, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, Qi Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08593
• Pdf URL: https://arxiv.org/pdf/2403.08593
• Copy Paste: [[2403.08593]] Call Me When Necessary: LLMs can Efficiently and Faithfully Reason over Structured Environments(https://arxiv.org/abs/2403.08593)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have shown potential in reasoning over structured environments, e.g., knowledge graph and table. Such tasks typically require multi-hop reasoning, i.e., match natural language utterance with instances in the environment. Previous methods leverage LLMs to incrementally build a reasoning path, where the LLMs either invoke tools or pick up schemas by step-by-step interacting with the environment. We propose Reasoning-Path-Editing (Readi), a novel framework where LLMs can efficiently and faithfully reason over structured environments. In Readi, LLMs initially generate a reasoning path given a query, and edit the path only when necessary. We instantiate the path on structured environments and provide feedback to edit the path if anything goes wrong. Experimental results on three KGQA datasets and two TableQA datasets show the effectiveness of Readi, significantly surpassing all LLM-based methods (by 9.1% on WebQSP, 12.4% on MQA-3H and 10.9% on WTQ), comparable with state-of-the-art fine-tuned methods (67% on CWQ and 74.7% on WebQSP) and substantially boosting the vanilla LLMs (by 14.9% on CWQ). Our code will be available upon publication.
• 摘要：大型语言模型（LLM）已显示出在结构化环境（例如知识图和表格）上进行推理的潜力。此类任务通常需要多跳推理，即将自然语言话语与环境中的实例进行匹配。以前的方法利用 LLM 逐步构建推理路径，其中 LLM 要么调用工具，要么通过与环境逐步交互来获取模式。我们提出推理路径编辑（Readi），这是一个新颖的框架，法学硕士可以在结构化环境中高效、忠实地进行推理。在 Readi 中，法学硕士最初根据查询生成推理路径，并仅在必要时编辑该路径。我们在结构化环境中实例化路径，并在出现任何问题时提供反馈以编辑路径。三个 KGQA 数据集和两个 TableQA 数据集上的实验结果显示了 Readi 的有效性，显着超过了所有基于 LLM 的方法（WebQSP 上 9.1%、MQA-3H 上 12.4% 和 WTQ 上 10.9%），与现有水平相当。 -艺术微调方法（CWQ 为 67%，WebQSP 为 74.7%），并大幅提升普通法学硕士（CWQ 为 14.9%）。我们的代码将在发布后提供。

Title: DevBench: A Comprehensive Benchmark for Software Development

• Authors: Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, Zhiyin Yu, He Du, Ping Yang, Dahua Lin, Chao Peng, Kai Chen
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2403.08604
• Pdf URL: https://arxiv.org/pdf/2403.08604
• Copy Paste: [[2403.08604]] DevBench: A Comprehensive Benchmark for Software Development(https://arxiv.org/abs/2403.08604)
• Keywords: language model, gpt, llm
• Abstract: Recent advancements in large language models (LLMs) have significantly enhanced their coding capabilities. However, existing benchmarks predominantly focused on simplified or isolated aspects of programming, such as single-file code generation or repository issue debugging, falling short of measuring the full spectrum of challenges raised by real-world programming activities. To this end, we propose DevBench, a comprehensive benchmark that evaluates LLMs across various stages of the software development lifecycle, including software design, environment setup, implementation, acceptance testing, and unit testing. DevBench features a wide range of programming languages and domains, high-quality data collection, and carefully designed and verified metrics for each task. Empirical studies show that current LLMs, including GPT-4-Turbo, fail to solve the challenges presented within DevBench. Analyses reveal that models struggle with understanding the complex structures in the repository, managing the compilation process, and grasping advanced programming concepts. Our findings offer actionable insights for the future development of LLMs toward real-world programming applications. Our benchmark is available at https://github.com/open-compass/DevBench
• 摘要：大型语言模型 (LLM) 的最新进展显着增强了其编码能力。然而，现有的基准测试主要关注编程的简化或孤立方面，例如单文件代码生成或存储库问题调试，无法衡量现实世界编程活动带来的全部挑战。为此，我们提出了 DevBench，这是一个综合基准，用于评估软件开发生命周期各个阶段的法学硕士，包括软件设计、环境设置、实施、验收测试和单元测试。 DevBench 具有广泛的编程语言和领域、高质量的数据收集以及针对每项任务精心设计和验证的指标。实证研究表明，当前的法学硕士（包括 GPT-4-Turbo）未能解决 DevBench 中提出的挑战。分析表明，模型很难理解存储库中的复杂结构、管理编译过程以及掌握高级编程概念。我们的研究结果为法学硕士未来向现实世界编程应用的发展提供了可行的见解。我们的基准测试可在 https://github.com/open-compass/DevBench 上找到

Title: MedInsight: A Multi-Source Context Augmentation Framework for Generating Patient-Centric Medical Responses using Large Language Models

• Authors: Subash Neupane, Shaswata Mitra, Sudip Mittal, Noorbakhsh Amiri Golilarz, Shahram Rahimi, Amin Amirlatifi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08607
• Pdf URL: https://arxiv.org/pdf/2403.08607
• Copy Paste: [[2403.08607]] MedInsight: A Multi-Source Context Augmentation Framework for Generating Patient-Centric Medical Responses using Large Language Models(https://arxiv.org/abs/2403.08607)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have shown impressive capabilities in generating human-like responses. However, their lack of domain-specific knowledge limits their applicability in healthcare settings, where contextual and comprehensive responses are vital. To address this challenge and enable the generation of patient-centric responses that are contextually relevant and comprehensive, we propose MedInsight:a novel retrieval augmented framework that augments LLM inputs (prompts) with relevant background information from multiple sources. MedInsight extracts pertinent details from the patient's medical record or consultation transcript. It then integrates information from authoritative medical textbooks and curated web resources based on the patient's health history and condition. By constructing an augmented context combining the patient's record with relevant medical knowledge, MedInsight generates enriched, patient-specific responses tailored for healthcare applications such as diagnosis, treatment recommendations, or patient education. Experiments on the MTSamples dataset validate MedInsight's effectiveness in generating contextually appropriate medical responses. Quantitative evaluation using the Ragas metric and TruLens for answer similarity and answer correctness demonstrates the model's efficacy. Furthermore, human evaluation studies involving Subject Matter Expert (SMEs) confirm MedInsight's utility, with moderate inter-rater agreement on the relevance and correctness of the generated responses.
• 摘要：大型语言模型 (LLM) 在生成类似人类的响应方面表现出了令人印象深刻的能力。然而，它们缺乏特定领域的知识限制了它们在医疗保健环境中的适用性，在医疗保健环境中，上下文和全面的响应至关重要。为了应对这一挑战并能够生成上下文相关且全面的以患者为中心的响应，我们提出了 MedInsight：一种新颖的检索增强框架，可以使用来自多个来源的相关背景信息来增强 LLM 输入（提示）。 MedInsight 从患者的医疗记录或咨询笔录中提取相关详细信息。然后，它会根据患者的健康史和状况整合来自权威医学教科书和精选网络资源的信息。通过构建将患者记录与相关医学知识相结合的增强环境，MedInsight 可以生成针对诊断、治疗建议或患者教育等医疗保健应用量身定制的丰富的、针对患者的响应。 MTSamples 数据集上的实验验证了 MedInsight 在生成适合上下文的医疗响应方面的有效性。使用 Ragas 指标和 TruLens 对答案相似性和答案正确性进行定量评估证明了该模型的有效性。此外，涉及主题专家 (SME) 的人类评估研究证实了 MedInsight 的实用性，评估者之间对生成的响应的相关性和正确性达成了适度的一致。

Title: Zero-shot and Few-shot Generation Strategies for Artificial Clinical Records

• Authors: Erlend Frayling, Jake Lever, Graham McDonald
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.08664
• Pdf URL: https://arxiv.org/pdf/2403.08664
• Copy Paste: [[2403.08664]] Zero-shot and Few-shot Generation Strategies for Artificial Clinical Records(https://arxiv.org/abs/2403.08664)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: The challenge of accessing historical patient data for clinical research, while adhering to privacy regulations, is a significant obstacle in medical science. An innovative approach to circumvent this issue involves utilising synthetic medical records that mirror real patient data without compromising individual privacy. The creation of these synthetic datasets, particularly without using actual patient data to train Large Language Models (LLMs), presents a novel solution as gaining access to sensitive patient information to train models is also a challenge. This study assesses the capability of the Llama 2 LLM to create synthetic medical records that accurately reflect real patient information, employing zero-shot and few-shot prompting strategies for comparison against fine-tuned methodologies that do require sensitive patient data during training. We focus on generating synthetic narratives for the History of Present Illness section, utilising data from the MIMIC-IV dataset for comparison. In this work introduce a novel prompting technique that leverages a chain-of-thought approach, enhancing the model's ability to generate more accurate and contextually relevant medical narratives without prior fine-tuning. Our findings suggest that this chain-of-thought prompted approach allows the zero-shot model to achieve results on par with those of fine-tuned models, based on Rouge metrics evaluation.
• 摘要：在遵守隐私法规的同时访问历史患者数据进行临床研究是医学科学中的一个重大障碍。规避此问题的创新方法涉及利用合成医疗记录来反映真实患者数据，而不损害个人隐私。这些合成数据集的创建，特别是在不使用实际患者数据来训练大型语言模型 (LLM) 的情况下，提供了一种新颖的解决方案，因为获取敏感患者信息来训练模型也是一个挑战。本研究评估了 Llama 2 LLM 创建准确反映真实患者信息的综合医疗记录的能力，采用零样本和少样本提示策略与训练期间确实需要敏感患者数据的微调方法进行比较。我们专注于为当前疾病史部分生成综合叙述，利用 MIMIC-IV 数据集的数据进行比较。在这项工作中，介绍了一种新颖的提示技术，该技术利用思想链方法，增强模型生成更准确且与上下文相关的医学叙述的能力，而无需事先进行微调。我们的研究结果表明，这种思想链提示方法使零样本模型能够获得与基于 Rouge 指标评估的微调模型相同的结果。

Title: Token Alignment via Character Matching for Subword Completion

• Authors: Ben Athiwaratkun, Shiqi Wang, Mingyue Shang, Yuchen Tian, Zijian Wang, Sujan Kumar Gonugondla, Sanjay Krishna Gouda, Rob Kwiatowski, Ramesh Nallapati, Bing Xiang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08688
• Pdf URL: https://arxiv.org/pdf/2403.08688
• Copy Paste: [[2403.08688]] Token Alignment via Character Matching for Subword Completion(https://arxiv.org/abs/2403.08688)
• Keywords: prompt
• Abstract: Generative models, widely utilized in various applications, can often struggle with prompts corresponding to partial tokens. This struggle stems from tokenization, where partial tokens fall out of distribution during inference, leading to incorrect or nonsensical outputs. This paper examines a technique to alleviate the tokenization artifact on text completion in generative models, maintaining performance even in regular non-subword cases. The method, termed token alignment, involves backtracking to the last complete tokens and ensuring the model's generation aligns with the prompt. This approach showcases marked improvement across many partial token scenarios, including nuanced cases like space-prefix and partial indentation, with only a minor time increase. The technique and analysis detailed in this paper contribute to the continuous advancement of generative models in handling partial inputs, bearing relevance for applications like code completion and text autocompletion.
• 摘要：广泛应用于各种应用程序的生成模型常常会遇到与部分标记相对应的提示。这种斗争源于标记化，部分标记在推理过程中脱离分布，导致不正确或无意义的输出。本文研究了一种技术，可以减轻生成模型中文本完成时的标记化伪影，即使在常规非子词情况下也能保持性能。该方法称为令牌对齐，涉及回溯到最后一个完整的令牌并确保模型的生成与提示保持一致。这种方法在许多部分令牌场景中展示了显着的改进，包括空格前缀和部分缩进等细微差别的情况，而仅增加了很小的时间。本文详细介绍的技术和分析有助于生成模型在处理部分输入方面的不断进步，与代码完成和文本自动完成等应用程序相关。

Title: Do Language Models Care About Text Quality? Evaluating Web-Crawled Corpora Across 11 Languages

• Authors: Rik van Noord, Taja Kuzman, Peter Rupnik, Nikola Ljubešić, Miquel Esplà-Gomis, Gema Ramírez-Sánchez, Antonio Toral
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08693
• Pdf URL: https://arxiv.org/pdf/2403.08693
• Copy Paste: [[2403.08693]] Do Language Models Care About Text Quality? Evaluating Web-Crawled Corpora Across 11 Languages(https://arxiv.org/abs/2403.08693)
• Keywords: language model, gpt
• Abstract: Large, curated, web-crawled corpora play a vital role in training language models (LMs). They form the lion's share of the training data in virtually all recent LMs, such as the well-known GPT, LLaMA and XLM-RoBERTa models. However, despite this importance, relatively little attention has been given to the quality of these corpora. In this paper, we compare four of the currently most relevant large, web-crawled corpora (CC100, MaCoCu, mC4 and OSCAR) across eleven lower-resourced European languages. Our approach is two-fold: first, we perform an intrinsic evaluation by performing a human evaluation of the quality of samples taken from different corpora; then, we assess the practical impact of the qualitative differences by training specific LMs on each of the corpora and evaluating their performance on downstream tasks. We find that there are clear differences in quality of the corpora, with MaCoCu and OSCAR obtaining the best results. However, during the extrinsic evaluation, we actually find that the CC100 corpus achieves the highest scores. We conclude that, in our experiments, the quality of the web-crawled corpora does not seem to play a significant role when training LMs.
• 摘要：大型、精心策划的网络爬虫语料库在训练语言模型 (LM) 中发挥着至关重要的作用。它们构成了几乎所有近期 LM 训练数据的大部分，例如著名的 GPT、LLaMA 和 XLM-RoBERTa 模型。然而，尽管如此重要，但人们对这些语料库的质量的关注相对较少。在本文中，我们比较了 11 种资源匮乏的欧洲语言中目前最相关的四种大型网络爬行语料库（CC100、MaCoCu、mC4 和 OSCAR）。我们的方法有两个：首先，我们通过对从不同语料库中获取的样本的质量进行人工评估来进行内在评估；然后，我们通过在每个语料库上训练特定的语言模型并评估它们在下游任务中的表现来评估定性差异的实际影响。我们发现语料库的质量存在明显差异，MaCoCu 和 OSCAR 获得了最好的结果。然而，在外部评估过程中，我们实际上发现CC100语料库得分最高。我们的结论是，在我们的实验中，网络爬取语料库的质量在训练 LM 时似乎并没有发挥重要作用。

Title: TeaMs-RL: Teaching LLMs to Teach Themselves Better Instructions via Reinforcement Learning

• Authors: Shangding Gu, Alois Knoll, Ming Jin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08694
• Pdf URL: https://arxiv.org/pdf/2403.08694
• Copy Paste: [[2403.08694]] TeaMs-RL: Teaching LLMs to Teach Themselves Better Instructions via Reinforcement Learning(https://arxiv.org/abs/2403.08694)
• Keywords: language model, llm
• Abstract: The development of Large Language Models (LLMs) often confronts challenges stemming from the heavy reliance on human annotators in the reinforcement learning with human feedback (RLHF) framework, or the frequent and costly external queries tied to the self-instruct paradigm. In this work, we pivot to Reinforcement Learning (RL) -- but with a twist. Diverging from the typical RLHF, which refines LLMs following instruction data training, we use RL to directly generate the foundational instruction dataset that alone suffices for fine-tuning. Our method, TeaMs-RL, uses a suite of textual operations and rules, prioritizing the diversification of training datasets. It facilitates the generation of high-quality data without excessive reliance on external advanced models, paving the way for a single fine-tuning step and negating the need for subsequent RLHF stages. Our findings highlight key advantages of our approach: reduced need for human involvement and fewer model queries (only $5.73\%$ of WizardLM's total), along with enhanced capabilities of LLMs in crafting and comprehending complex instructions compared to strong baselines, and substantially improved model privacy protection.
• 摘要：大型语言模型（LLM）的开发经常面临挑战，这些挑战源于人类反馈强化学习（RLHF）框架中对人类注释者的严重依赖，或者与自指导范式相关的频繁且昂贵的外部查询。在这项工作中，我们转向强化学习（RL）——但有所不同。与典型的 RLHF（根据指令数据训练来完善 LLM）不同，我们使用 RL 直接生成基础指令数据集，仅此数据集就足以进行微调。我们的方法 TeaMs-RL 使用一套文本操作和规则，优先考虑训练数据集的多样化。它有助于生成高质量数据，而无需过度依赖外部高级模型，为单个微调步骤铺平道路，并无需后续 RLHF 阶段。我们的研究结果突出了我们方法的主要优势：减少了对人类参与的需求和更少的模型查询（仅占 WizardLM 总数的 5.73\%$），与强大的基线相比，法学硕士在制定和理解复杂指令方面的能力得到了增强，并且模型得到了显着改进隐私保护。

Title: SOTOPIA-$π$: Interactive Learning of Socially Intelligent Language Agents

• Authors: Ruiyi Wang, Haofei Yu, Wenxin Zhang, Zhengyang Qi, Maarten Sap, Graham Neubig, Yonatan Bisk, Hao Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.08715
• Pdf URL: https://arxiv.org/pdf/2403.08715
• Copy Paste: [[2403.08715]] SOTOPIA-$π$: Interactive Learning of Socially Intelligent Language Agents(https://arxiv.org/abs/2403.08715)
• Keywords: language model, gpt, llm, agent
• Abstract: Humans learn social skills through both imitation and social interaction. This social learning process is largely understudied by existing research on building language agents. Motivated by this gap, we propose an interactive learning method, SOTOPIA-$\pi$, improving the social intelligence of language agents. This method leverages behavior cloning and self-reinforcement training on filtered social interaction data according to large language model (LLM) ratings. We show that our training method allows a 7B LLM to reach the social goal completion ability of an expert model (GPT-4-based agent), while improving the safety of language agents and maintaining general QA ability on the MMLU benchmark. We also find that this training paradigm uncovers some difficulties in LLM-based evaluation of social intelligence: LLM-based evaluators overestimate the abilities of the language agents trained specifically for social interaction.
• 摘要：人类通过模仿和社交互动来学习社交技能。现有的构建语言代理的研究很大程度上没有充分研究这种社会学习过程。受这一差距的启发，我们提出了一种交互式学习方法 SOTOPIA-$\pi$，以提高语言代理的社交智能。该方法根据大语言模型 (LLM) 评级，利用对过滤后的社交互动数据进行行为克隆和自我强化训练。我们表明，我们的训练方法允许 7B LLM 达到专家模型（基于 GPT-4 的代理）的社会目标完成能力，同时提高语言代理的安全性并保持 MMLU 基准上的一般 QA 能力。我们还发现，这种训练范式揭示了基于法学硕士的社交智力评估中的一些困难：基于法学硕士的评估者高估了专门针对社交互动训练的语言代理的能力。

Title: Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization

• Authors: Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, Tong Zhang
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.08730
• Pdf URL: https://arxiv.org/pdf/2403.08730
• Copy Paste: [[2403.08730]] Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization(https://arxiv.org/abs/2403.08730)
• Keywords: language model, llm
• Abstract: Multimodal Large Language Models (MLLMs) excel in generating responses based on visual inputs. However, they often suffer from a bias towards generating responses similar to their pretraining corpus, overshadowing the importance of visual information. We treat this bias as a "preference" for pretraining statistics, which hinders the model's grounding in visual input. To mitigate this issue, we propose Bootstrapped Preference Optimization (BPO), which conducts preference learning with datasets containing negative responses bootstrapped from the model itself. Specifically, we propose the following two strategies: 1) using distorted image inputs to the MLLM for eliciting responses that contain signified pretraining bias; 2) leveraging text-based LLM to explicitly inject erroneous but common elements into the original response. Those undesirable responses are paired with original annotated responses from the datasets to construct the preference dataset, which is subsequently utilized to perform preference learning. Our approach effectively suppresses pretrained LLM bias, enabling enhanced grounding in visual inputs. Extensive experimentation demonstrates significant performance improvements across multiple benchmarks, advancing the state-of-the-art in multimodal conversational systems.
• 摘要：多模态大语言模型 (MLLM) 擅长根据视觉输入生成响应。然而，他们经常偏向于生成类似于预训练语料库的响应，从而掩盖了视觉信息的重要性。我们将这种偏差视为预训练统计的“偏好”，这阻碍了模型在视觉输入方面的基础。为了缓解这个问题，我们提出了引导偏好优化（BPO），它使用包含从模型本身引导的负面响应的数据集进行偏好学习。具体来说，我们提出以下两种策略：1）使用 MLLM 的扭曲图像输入来引发包含明显预训练偏差的响应； 2）利用基于文本的LLM将错误但常见的元素明确地注入到原始响应中。这些不需要的响应与来自数据集的原始注释响应配对以构建偏好数据集，该数据集随后用于执行偏好学习。我们的方法有效地抑制了预训练的 LLM 偏差，从而增强了视觉输入的基础。广泛的实验证明了多个基准的显着性能改进，推动了多模式对话系统的最先进水平。

Title: The Garden of Forking Paths: Observing Dynamic Parameters Distribution in Large Language Models

• Authors: Carlo Nicolini, Jacopo Staiano, Bruno Lepri, Raffaele Marino
• Subjects: cs.CL, cond-mat.dis-nn, cond-mat.stat-mech, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.08739
• Pdf URL: https://arxiv.org/pdf/2403.08739
• Copy Paste: [[2403.08739]] The Garden of Forking Paths: Observing Dynamic Parameters Distribution in Large Language Models(https://arxiv.org/abs/2403.08739)
• Keywords: language model
• Abstract: A substantial gap persists in understanding the reasons behind the exceptional performance of the Transformer architecture in NLP. A particularly unexplored area involves the mechanistic description of how the distribution of parameters evolves over time during training. In this work we suggest that looking at the time evolution of the statistic distribution of model parameters, and specifically at bifurcation effects, can help understanding the model quality, potentially reducing training costs and evaluation efforts and empirically showing the reasons behind the effectiveness of weights sparsification.
• 摘要：对于 NLP 中 Transformer 架构卓越性能背后的原因的理解仍然存在很大差距。一个特别未经探索的领域涉及训练过程中参数分布如何随时间演变的机械描述。在这项工作中，我们建议观察模型参数统计分布的时间演化，特别是分叉效应，可以帮助理解模型质量，潜在地减少训练成本和评估工作，并凭经验显示权重稀疏有效性背后的原因。

Title: Steering LLMs Towards Unbiased Responses: A Causality-Guided Debiasing Framework

• Authors: Jingling Li, Zeyu Tang, Xiaoyu Liu, Peter Spirtes, Kun Zhang, Liu Leqi, Yang Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.08743
• Pdf URL: https://arxiv.org/pdf/2403.08743
• Copy Paste: [[2403.08743]] Steering LLMs Towards Unbiased Responses: A Causality-Guided Debiasing Framework(https://arxiv.org/abs/2403.08743)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) can easily generate biased and discriminative responses. As LLMs tap into consequential decision-making (e.g., hiring and healthcare), it is of crucial importance to develop strategies to mitigate these biases. This paper focuses on social bias, tackling the association between demographic information and LLM outputs. We propose a causality-guided debiasing framework that utilizes causal understandings of (1) the data-generating process of the training corpus fed to LLMs, and (2) the internal reasoning process of LLM inference, to guide the design of prompts for debiasing LLM outputs through selection mechanisms. Our framework unifies existing de-biasing prompting approaches such as inhibitive instructions and in-context contrastive examples, and sheds light on new ways of debiasing by encouraging bias-free reasoning. Our strong empirical performance on real-world datasets demonstrates that our framework provides principled guidelines on debiasing LLM outputs even with only the black-box access.
• 摘要：大型语言模型（LLM）很容易产生有偏见和歧视性的反应。当法学硕士涉足后续决策（例如招聘和医疗保健）时，制定减轻这些偏见的策略至关重要。本文重点关注社会偏见，解决人口统计信息与法学硕士输出之间的关联。我们提出了一个因果引导的去偏框架，利用对（1）提供给法学硕士的训练语料库的数据生成过程和（2）法学硕士推理的内部推理过程的因果理解，来指导法学硕士去偏提示的设计通过选择机制产出。我们的框架统一了现有的去偏见提示方法，例如抑制性指令和上下文对比示例，并通过鼓励无偏见推理来揭示新的去偏见方法。我们在现实世界数据集上的强大实证表现表明，即使只有黑盒访问，我们的框架也为消除 LLM 输出的偏差提供了原则性指导方针。