2024-08-16

Title: Language Driven Slice Discovery and Error Rectification

• Authors: Shantanu Ghosh, Chenyu Wang, Kayhan Batmanghelich
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.07832
• Pdf URL: https://arxiv.org/pdf/2408.07832
• Copy Paste: [[2408.07832]] Language Driven Slice Discovery and Error Rectification(https://arxiv.org/abs/2408.07832)
• Keywords: language model, llm
• Abstract: Error slice discovery associates structured patterns with model errors. Existing methods discover error slices by clustering the error-prone samples with similar patterns or assigning discrete attributes to each sample for post-hoc analysis. While these methods aim for interpretability and easier mitigation through reweighting or rebalancing, they may not capture the full complexity of error patterns due to incomplete or missing attributes. Contrary to the existing approach, this paper utilizes the reasoning capabilities of the Large Language Model (LLM) to analyze complex error patterns and generate testable hypotheses. This paper proposes LADDER: Language Driven slice Discovery and Error Rectification. It first projects the model's representation into a language-aligned feature space (\eg CLIP) to preserve semantics in the original model feature space. This ensures the accurate retrieval of sentences that highlight the model's errors. Next, the LLM utilizes the sentences and generates hypotheses to discover error slices. Finally, we mitigate the error by fine-tuning the classification head by creating a group-balanced dataset using the hypotheses. Our entire method does not require any attribute annotation, either explicitly or through external tagging models. We validate our method with \textbf{five} image classification datasets. The code is available\footnote{\url{this https URL}}
• 摘要：错误切片发现将结构化模式与模型错误关联起来。现有方法通过将具有相似模式的易错样本聚类或为每个样本分配离散属性以进行事后分析来发现错误切片。虽然这些方法旨在通过重新加权或重新平衡来实现可解释性和更容易的缓解，但由于属性不完整或缺失，它们可能无法捕捉错误模式的全部复杂性。与现有方法相反，本文利用大型语言模型 (LLM) 的推理能力来分析复杂的错误模式并生成可测试的假设。本文提出了 LADDER：语言驱动的切片发现和错误纠正。它首先将模型的表示投影到语言对齐的特征空间（\eg CLIP）中，以保留原始模型特征空间中的语义。这确保了准确检索突出模型错误的句子。接下来，LLM 利用这些句子并生成假设来发现错误切片。最后，我们通过使用假设创建组平衡数据集来微调分类头，从而减轻错误。我们的整个方法不需要任何属性注释，无论是显式注释还是通过外部标记模型。我们使用 \textbf{five} 图像分类数据集验证了我们的方法。代码可从 \footnote{\url{此 https URL}} 获取

Title: ONSEP: A Novel Online Neural-Symbolic Framework for Event Prediction Based on Large Language Model

• Authors: Xuanqing Yu, Wangtao Sun, Jingwei Li, Kang Liu, Chengbao Liu, Jie Tan
• Subjects: cs.CL, cs.AI, cs.SC
• Abstract URL: https://arxiv.org/abs/2408.07840
• Pdf URL: https://arxiv.org/pdf/2408.07840
• Copy Paste: [[2408.07840]] ONSEP: A Novel Online Neural-Symbolic Framework for Event Prediction Based on Large Language Model(https://arxiv.org/abs/2408.07840)
• Keywords: language model, llm
• Abstract: In the realm of event prediction, temporal knowledge graph forecasting (TKGF) stands as a pivotal technique. Previous approaches face the challenges of not utilizing experience during testing and relying on a single short-term history, which limits adaptation to evolving data. In this paper, we introduce the Online Neural-Symbolic Event Prediction (ONSEP) framework, which innovates by integrating dynamic causal rule mining (DCRM) and dual history augmented generation (DHAG). DCRM dynamically constructs causal rules from real-time data, allowing for swift adaptation to new causal relationships. In parallel, DHAG merges short-term and long-term historical contexts, leveraging a bi-branch approach to enrich event prediction. Our framework demonstrates notable performance enhancements across diverse datasets, with significant Hit@k (k=1,3,10) improvements, showcasing its ability to augment large language models (LLMs) for event prediction without necessitating extensive retraining. The ONSEP framework not only advances the field of TKGF but also underscores the potential of neural-symbolic approaches in adapting to dynamic data environments.
• 摘要：在事件预测领域，时间知识图谱预测 (TKGF) 是一项关键技术。以前的方法面临的挑战是，在测试期间不利用经验，并且依赖单一的短期历史，这限制了对不断变化的数据的适应。在本文中，我们介绍了在线神经符号事件预测 (ONSEP) 框架，该框架通过集成动态因果规则挖掘 (DCRM) 和双重历史增强生成 (DHAG) 进行创新。DCRM 从实时数据中动态构建因果规则，从而能够快速适应新的因果关系。同时，DHAG 融合了短期和长期历史背景，利用双分支方法来丰富事件预测。我们的框架在不同数据集上表现出显着的性能增强，Hit@k（k=1,3,10）显著改进，展示了它能够增强大型语言模型 (LLM) 以进行事件预测，而无需进行大量再训练。 ONSEP 框架不仅推动了 TKGF 领域的发展，而且还强调了神经符号方法在适应动态数据环境方面的潜力。

Title: Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability

• Authors: Jiri Hron, Laura Culp, Gamaleldin Elsayed, Rosanne Liu, Ben Adlam, Maxwell Bileschi, Bernd Bohnet, JD Co-Reyes, Noah Fiedel, C. Daniel Freeman, Izzeddin Gur, Kathleen Kenealy, Jaehoon Lee, Peter J. Liu, Gaurav Mishra, Igor Mordatch, Azade Nova, Roman Novak, Aaron Parisi, Jeffrey Pennington, Alex Rizkowsky, Isabelle Simpson, Hanie Sedghi, Jascha Sohl-dickstein, Kevin Swersky, Sharad Vikram, Tris Warkentin, Lechao Xiao, Kelvin Xu, Jasper Snoek, Simon Kornblith
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.07852
• Pdf URL: https://arxiv.org/pdf/2408.07852
• Copy Paste: [[2408.07852]] Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability(https://arxiv.org/abs/2408.07852)
• Keywords: language model, hallucination
• Abstract: While many capabilities of language models (LMs) improve with increased training budget, the influence of scale on hallucinations is not yet fully understood. Hallucinations come in many forms, and there is no universally accepted definition. We thus focus on studying only those hallucinations where a correct answer appears verbatim in the training set. To fully control the training data content, we construct a knowledge graph (KG)-based dataset, and use it to train a set of increasingly large LMs. We find that for a fixed dataset, larger and longer-trained LMs hallucinate less. However, hallucinating on $\leq5$% of the training data requires an order of magnitude larger model, and thus an order of magnitude more compute, than Hoffmann et al. (2022) reported was optimal. Given this costliness, we study how hallucination detectors depend on scale. While we see detector size improves performance on fixed LM's outputs, we find an inverse relationship between the scale of the LM and the detectability of its hallucinations.
• 摘要：虽然语言模型 (LM) 的许多功能会随着训练预算的增加而提高，但规模对幻觉的影响尚未完全了解。幻觉有多种形式，没有普遍接受的定义。因此，我们专注于研究那些在训练集中逐字出现正确答案的幻觉。为了完全控制训练数据内容，我们构建了一个基于知识图谱 (KG) 的数据集，并使用它来训练一组越来越大的 LM。我们发现，对于固定数据集，更大、训练时间更长的 LM 幻觉较少。然而，对 $\leq5$% 的训练数据产生幻觉需要比 Hoffmann 等人 (2022) 报告的最佳值大一个数量级的模型，因此计算量也要大一个数量级。考虑到这种成本高昂，我们研究了幻觉检测器如何依赖于规模。虽然我们看到检测器大小可以提高固定 LM 输出的性能，但我们发现 LM 的规模与其幻觉的可检测性之间存在反比关系。

Title: Words Matter: Reducing Stigma in Online Conversations about Substance Use with Large Language Models

• Authors: Layla Bouzoubaa, Elham Aghakhani, Shadi Rezapour
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07873
• Pdf URL: https://arxiv.org/pdf/2408.07873
• Copy Paste: [[2408.07873]] Words Matter: Reducing Stigma in Online Conversations about Substance Use with Large Language Models(https://arxiv.org/abs/2408.07873)
• Keywords: language model, llm
• Abstract: Stigma is a barrier to treatment for individuals struggling with substance use disorders (SUD), which leads to significantly lower treatment engagement rates. With only 7% of those affected receiving any form of help, societal stigma not only discourages individuals with SUD from seeking help but isolates them, hindering their recovery journey and perpetuating a cycle of shame and self-doubt. This study investigates how stigma manifests on social media, particularly Reddit, where anonymity can exacerbate discriminatory behaviors. We analyzed over 1.2 million posts, identifying 3,207 that exhibited stigmatizing language towards people who use substances (PWUS). Using Informed and Stylized LLMs, we develop a model for de-stigmatization of these expressions into empathetic language, resulting in 1,649 reformed phrase pairs. Our paper contributes to the field by proposing a computational framework for analyzing stigma and destigmatizing online content, and delving into the linguistic features that propagate stigma towards PWUS. Our work not only enhances understanding of stigma's manifestations online but also provides practical tools for fostering a more supportive digital environment for those affected by SUD. Code and data will be made publicly available upon acceptance.
• 摘要：污名化是物质使用障碍 (SUD) 患者接受治疗的一大障碍，导致治疗参与率大幅下降。只有 7% 的患者能得到任何形式的帮助，社会污名化不仅会阻止 SUD 患者寻求帮助，还会孤立他们，阻碍他们的康复之旅，并导致羞耻和自我怀疑的循环。本研究调查了污名化在社交媒体（尤其是 Reddit）上的表现方式，匿名性会加剧歧视行为。我们分析了超过 120 万个帖子，发现有 3,207 个帖子表现出对物质使用者 (PWUS) 的污名化语言。使用知情和风格化的 LLM，我们开发了一个模型，将这些表达去污名化为富有同情心的语言，最终得到 1,649 个改良短语对。我们的论文提出了一个用于分析污名和消除网络内容污名的计算框架，并深入研究了传播针对 PWUS 的污名的语言特征，为该领域做出了贡献。我们的工作不仅增强了对污名在网上表现形式的理解，还提供了实用工具，为受 SUD 影响的人营造一个更具支持性的数字环境。代码和数据将在论文被接受后公开。

Title: Instruct Large Language Models to Generate Scientific Literature Survey Step by Step

• Authors: Yuxuan Lai, Yupeng Wu, Yidan Wang, Wenpeng Hu, Chen Zheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07884
• Pdf URL: https://arxiv.org/pdf/2408.07884
• Copy Paste: [[2408.07884]] Instruct Large Language Models to Generate Scientific Literature Survey Step by Step(https://arxiv.org/abs/2408.07884)
• Keywords: language model, llm, prompt
• Abstract: Abstract. Automatically generating scientific literature surveys is a valuable task that can significantly enhance research efficiency. However, the diverse and complex nature of information within a literature survey poses substantial challenges for generative models. In this paper, we design a series of prompts to systematically leverage large language models (LLMs), enabling the creation of comprehensive literature surveys through a step-by-step approach. Specifically, we design prompts to guide LLMs to sequentially generate the title, abstract, hierarchical headings, and the main content of the literature survey. We argue that this design enables the generation of the headings from a high-level perspective. During the content generation process, this design effectively harnesses relevant information while minimizing costs by restricting the length of both input and output content in LLM queries. Our implementation with Qwen-long achieved third place in the NLPCC 2024 Scientific Literature Survey Generation evaluation task, with an overall score only 0.03% lower than the second-place team. Additionally, our soft heading recall is 95.84%, the second best among the submissions. Thanks to the efficient prompt design and the low cost of the Qwen-long API, our method reduces the expense for generating each literature survey to 0.1 RMB, enhancing the practical value of our method.
• 摘要：摘要。自动生成科学文献调查是一项有价值的任务，可以显著提高研究效率。然而，文献调查中信息的多样性和复杂性对生成模型提出了巨大的挑战。在本文中，我们设计了一系列提示，以系统地利用大型语言模型 (LLM)，从而能够通过循序渐进的方式创建全面的文献调查。具体来说，我们设计提示来指导 LLM 依次生成文献调查的标题、摘要、分层标题和主要内容。我们认为这种设计能够从高级角度生成标题。在内容生成过程中，此设计通过限制 LLM 查询中输入和输出内容的长度，有效地利用相关信息，同时最大限度地降低成本。我们与 Qwen-long 的实现在 NLPCC 2024 科学文献调查生成评估任务中获得了第三名，总分仅比第二名团队低 0.03%。此外，我们的软标题召回率为 95.84%，在提交的作品中排名第二。得益于高效的快捷设计和Qwen-long API的低成本，我们的方法将生成每个文献调查的费用降低到0.1元人民币，增强了我们方法的实用价值。

Title: Fine-tuning Large Language Models with Human-inspired Learning Strategies in Medical Question Answering

• Authors: Yushi Yang, Andrew M. Bean, Robert McCraith, Adam Mahdi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07888
• Pdf URL: https://arxiv.org/pdf/2408.07888
• Copy Paste: [[2408.07888]] Fine-tuning Large Language Models with Human-inspired Learning Strategies in Medical Question Answering(https://arxiv.org/abs/2408.07888)
• Keywords: language model, llm
• Abstract: Training Large Language Models (LLMs) incurs substantial data-related costs, motivating the development of data-efficient training methods through optimised data ordering and selection. Human-inspired learning strategies, such as curriculum learning, offer possibilities for efficient training by organising data according to common human learning practices. Despite evidence that fine-tuning with curriculum learning improves the performance of LLMs for natural language understanding tasks, its effectiveness is typically assessed using a single model. In this work, we extend previous research by evaluating both curriculum-based and non-curriculum-based learning strategies across multiple LLMs, using human-defined and automated data labels for medical question answering. Our results indicate a moderate impact of using human-inspired learning strategies for fine-tuning LLMs, with maximum accuracy gains of 1.77% per model and 1.81% per dataset. Crucially, we demonstrate that the effectiveness of these strategies varies significantly across different model-dataset combinations, emphasising that the benefits of a specific human-inspired strategy for fine-tuning LLMs do not generalise. Additionally, we find evidence that curriculum learning using LLM-defined question difficulty outperforms human-defined difficulty, highlighting the potential of using model-generated measures for optimal curriculum design.
• 摘要：训练大型语言模型 (LLM) 会产生大量与数据相关的成本，这促使人们通过优化数据排序和选择来开发数据高效的训练方法。人类启发式学习策略（例如课程学习）通过根据常见的人类学习实践组织数据，为高效训练提供了可能性。尽管有证据表明，使用课程学习进行微调可以提高 LLM 在自然语言理解任务中的表现，但其有效性通常使用单一模型来评估。在这项工作中，我们扩展了以前的研究，通过评估多个 LLM 中的基于课程和非基于课程的学习策略，使用人类定义和自动化的数据标签来回答医学问题。我们的结果表明，使用人类启发式学习策略对微调 LLM 的影响适中，每个模型的最大准确率提高 1.77%，每个数据集的最大准确率提高 1.81%。至关重要的是，我们证明了这些策略的有效性在不同的模型数据集组合中存在显著差异，强调特定的人类启发式策略对微调 LLM 的好处并不具有普遍性。此外，我们发现证据表明，使用 LLM 定义的问题难度的课程学习优于人类定义的难度，这凸显了使用模型生成的测量方法进行最佳课程设计的潜力。

Title: Assessing Language Models' Worldview for Fiction Generation

• Authors: Aisha Khatun, Daniel G. Brown
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07904
• Pdf URL: https://arxiv.org/pdf/2408.07904
• Copy Paste: [[2408.07904]] Assessing Language Models' Worldview for Fiction Generation(https://arxiv.org/abs/2408.07904)
• Keywords: language model, llm
• Abstract: The use of Large Language Models (LLMs) has become ubiquitous, with abundant applications in computational creativity. One such application is fictional story generation. Fiction is a narrative that occurs in a story world that is slightly different than ours. With LLMs becoming writing partners, we question how suitable they are to generate fiction. This study investigates the ability of LLMs to maintain a state of world essential to generate fiction. Through a series of questions to nine LLMs, we find that only two models exhibit consistent worldview, while the rest are self-conflicting. Subsequent analysis of stories generated by four models revealed a strikingly uniform narrative pattern. This uniformity across models further suggests a lack of `state' necessary for fiction. We highlight the limitations of current LLMs in fiction writing and advocate for future research to test and create story worlds for LLMs to reside in. All code, dataset, and the generated responses can be found in this https URL.
• 摘要：大型语言模型 (LLM) 的使用已经无处不在，在计算创造力方面有着丰富的应用。其中一种应用是虚构故事生成。小说是一种发生在与我们略有不同的故事世界中的叙述。随着 LLM 成为写作伙伴，我们质疑他们是否适合创作小说。本研究调查了 LLM 维持创作小说所必需的世界状态的能力。通过对九位 LLM 提出一系列问题，我们发现只有两个模型表现出一致的世界观，而其余的都是自相矛盾的。随后对四个模型生成的故事的分析揭示了一种惊人统一的叙事模式。模型之间的这种一致性进一步表明缺乏小说所必需的“状态”。我们强调了当前 LLM 在小说写作中的局限性，并提倡未来的研究来测试和创建 LLM 居住的故事世界。所有代码、数据集和生成的响应都可以在此 https URL 中找到。

Title: MAG-SQL: Multi-Agent Generative Approach with Soft Schema Linking and Iterative Sub-SQL Refinement for Text-to-SQL

• Authors: Wenxuan Xie, Gaochen Wu, Bowen Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.07930
• Pdf URL: https://arxiv.org/pdf/2408.07930
• Copy Paste: [[2408.07930]] MAG-SQL: Multi-Agent Generative Approach with Soft Schema Linking and Iterative Sub-SQL Refinement for Text-to-SQL(https://arxiv.org/abs/2408.07930)
• Keywords: gpt, agent
• Abstract: Recent In-Context Learning based methods have achieved remarkable success in Text-to-SQL task. However, there is still a large gap between the performance of these models and human performance on datasets with complex database schema and difficult questions, such as BIRD. Besides, existing work has neglected to supervise intermediate steps when solving questions iteratively with question decomposition methods, and the schema linking methods used in these works are very rudimentary. To address these issues, we propose MAG-SQL, a multi-agent generative approach with soft schema linking and iterative Sub-SQL refinement. In our framework, an entity-based method with tables' summary is used to select the columns in database, and a novel targets-conditions decomposition method is introduced to decompose those complex questions. Additionally, we build a iterative generating module which includes a Sub-SQL Generator and Sub-SQL Refiner, introducing external oversight for each step of generation. Through a series of ablation studies, the effectiveness of each agent in our framework has been demonstrated. When evaluated on the BIRD benchmark with GPT-4, MAG-SQL achieves an execution accuracy of 61.08\%, compared to the baseline accuracy of 46.35\% for vanilla GPT-4 and the baseline accuracy of 57.56\% for MAC-SQL. Besides, our approach makes similar progress on Spider.
• 摘要：最近基于上下文学习的方法在文本到 SQL 任务中取得了显著的成功。然而，在具有复杂数据库模式和困难问题的数据集（如 BIRD）上，这些模型的性能与人类性能之间仍然存在很大差距。此外，现有工作在使用问题分解方法迭代解决问题时忽略了监督中间步骤，并且这些工作中使用的模式链接方法非常简陋。为了解决这些问题，我们提出了 MAG-SQL，这是一种具有软模式链接和迭代 Sub-SQL 细化的多智能体生成方法。在我们的框架中，使用基于表摘要的实体方法来选择数据库中的列，并引入了一种新颖的目标条件分解方法来分解这些复杂问题。此外，我们构建了一个迭代生成模块，其中包括一个 Sub-SQL 生成器和 Sub-SQL 细化器，为生成的每个步骤引入外部监督。通过一系列消融研究，证明了我们框架中每个智能体的有效性。在 BIRD 基准上使用 GPT-4 进行评估时，MAG-SQL 的执行准确率达到 61.08%，而 vanilla GPT-4 的基线准确率仅为 46.35%，MAC-SQL 的基线准确率仅为 57.56%。此外，我们的方法在 Spider 上也取得了类似的进展。

Title: Predicting Lung Cancer Patient Prognosis with Large Language Models

• Authors: Danqing Hu, Bing Liu, Xiang Li, Xiaofeng Zhu, Nan Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07971
• Pdf URL: https://arxiv.org/pdf/2408.07971
• Copy Paste: [[2408.07971]] Predicting Lung Cancer Patient Prognosis with Large Language Models(https://arxiv.org/abs/2408.07971)
• Keywords: language model, gpt, llm
• Abstract: Prognosis prediction is crucial for determining optimal treatment plans for lung cancer patients. Traditionally, such predictions relied on models developed from retrospective patient data. Recently, large language models (LLMs) have gained attention for their ability to process and generate text based on extensive learned knowledge. In this study, we evaluate the potential of GPT-4o mini and GPT-3.5 in predicting the prognosis of lung cancer patients. We collected two prognosis datasets, i.e., survival and post-operative complication datasets, and designed multiple tasks to assess the models' performance comprehensively. Logistic regression models were also developed as baselines for comparison. The experimental results demonstrate that LLMs can achieve competitive, and in some tasks superior, performance in lung cancer prognosis prediction compared to data-driven logistic regression models despite not using additional patient data. These findings suggest that LLMs can be effective tools for prognosis prediction in lung cancer, particularly when patient data is limited or unavailable.
• 摘要：预后预测对于确定肺癌患者的最佳治疗方案至关重要。传统上，这种预测依赖于从回顾性患者数据中开发的模型。最近，大型语言模型 (LLM) 因其基于大量学习知识处理和生成文本的能力而受到关注。在本研究中，我们评估了 GPT-4o mini 和 GPT-3.5 在预测肺癌患者预后方面的潜力。我们收集了两个预后数据集，即生存期和术后并发症数据集，并设计了多个任务来全面评估模型的性能。还开发了逻辑回归模型作为比较的基线。实验结果表明，尽管不使用额外的患者数据，但与数据驱动的逻辑回归模型相比，LLM 在肺癌预后预测方面可以实现具有竞争力的性能，并且在某些任务中性能更优。这些发现表明，LLM 可以成为肺癌预后预测的有效工具，尤其是在患者数据有限或不可用的情况下。

Title: ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models

• Authors: Faris Hijazi (1), Somayah AlHarbi (1), Abdulaziz AlHussein (1), Harethah Abu Shairah (2), Reem AlZahrani (2), Hebah AlShamlan (1), Omar Knio (2), George Turkiyyah (2) ((1) THIQAH, (2) KAUST)
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.07983
• Pdf URL: https://arxiv.org/pdf/2408.07983
• Copy Paste: [[2408.07983]] ArabLegalEval: A Multitask Benchmark for Assessing Arabic Legal Knowledge in Large Language Models(https://arxiv.org/abs/2408.07983)
• Keywords: language model, gpt, llm
• Abstract: The rapid advancements in Large Language Models (LLMs) have led to significant improvements in various natural language processing tasks. However, the evaluation of LLMs' legal knowledge, particularly in non-English languages such as Arabic, remains under-explored. To address this gap, we introduce ArabLegalEval, a multitask benchmark dataset for assessing the Arabic legal knowledge of LLMs. Inspired by the MMLU and LegalBench datasets, ArabLegalEval consists of multiple tasks sourced from Saudi legal documents and synthesized questions. In this work, we aim to analyze the capabilities required to solve legal problems in Arabic and benchmark the performance of state-of-the-art LLMs. We explore the impact of in-context learning and investigate various evaluation methods. Additionally, we explore workflows for generating questions with automatic validation to enhance the dataset's quality. We benchmark multilingual and Arabic-centric LLMs, such as GPT-4 and Jais, respectively. We also share our methodology for creating the dataset and validation, which can be generalized to other domains. We hope to accelerate AI research in the Arabic Legal domain by releasing the ArabLegalEval dataset and code: this https URL
• 摘要：大型语言模型 (LLM) 的快速发展已导致各种自然语言处理任务得到显着改进。然而，对 LLM 的法律知识的评估，特别是对阿拉伯语等非英语语言的法律知识的评估，仍然没有得到充分探索。为了弥补这一差距，我们引入了 ArabLegalEval，这是一个用于评估 LLM 的阿拉伯语法律知识的多任务基准数据集。受 MMLU 和 LegalBench 数据集的启发，ArabLegalEval 由来自沙特法律文件和综合问题的多个任务组成。在这项工作中，我们旨在分析解决阿拉伯语法律问题所需的能力，并对最先进的 LLM 的性能进行基准测试。我们探索了上下文学习的影响并研究了各种评估方法。此外，我们还探索了生成具有自动验证的问题的工作流程，以提高数据集的质量。我们分别对多语言和以阿拉伯语为中心的 LLM（例如 GPT-4 和 Jais）进行了基准测试。我们还分享了创建数据集和验证的方法，这些方法可以推广到其他领域。我们希望通过发布 ArabLegalEval 数据集和代码来加速阿拉伯法律领域的人工智能研究：此 https URL

Title: FuseChat: Knowledge Fusion of Chat Models

• Authors: Fanqi Wan, Longguang Zhong, Ziyi Yang, Ruijun Chen, Xiaojun Quan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.07990
• Pdf URL: https://arxiv.org/pdf/2408.07990
• Copy Paste: [[2408.07990]] FuseChat: Knowledge Fusion of Chat Models(https://arxiv.org/abs/2408.07990)
• Keywords: language model, gpt, llm, chat
• Abstract: While training large language models (LLMs) from scratch can indeed lead to models with distinct capabilities and strengths, it incurs substantial costs and may lead to redundancy in competencies. Knowledge fusion aims to integrate existing LLMs of diverse architectures and capabilities into a more potent LLM through lightweight continual training, thereby reducing the need for costly LLM development. In this work, we propose a new framework for the knowledge fusion of chat LLMs through two main stages, resulting in FuseChat. Firstly, we conduct pairwise knowledge fusion on source chat LLMs of varying structures and scales to create multiple target LLMs with identical structure and size via lightweight fine-tuning. During this process, a statistics-based token alignment approach is introduced as the cornerstone for fusing LLMs with different structures. Secondly, we merge these target LLMs within the parameter space, where we propose a novel method for determining the merging coefficients based on the magnitude of parameter updates before and after fine-tuning. We implement and validate FuseChat using six prominent chat LLMs with diverse architectures and scales, including OpenChat-3.5-7B, Starling-LM-7B-alpha, NH2-SOLAR-10.7B, InternLM2-Chat-20B, Mixtral-8x7B-Instruct, and Qwen-1.5-Chat-72B. Experimental results on two instruction-following benchmarks, AlpacaEval 2.0 and MT-Bench, demonstrate the superiority of FuseChat-7B over baselines of various sizes. Our model is even comparable to the larger Mixtral-8x7B-Instruct and approaches GPT-3.5-Turbo-1106 on MT-Bench. Our code, model weights, and data are public at \url{this https URL}.
• 摘要：虽然从头开始训练大型语言模型 (LLM) 确实可以产生具有独特能力和优势的模型，但它会产生大量成本并可能导致能力冗余。知识融合旨在通过轻量级持续训练将现有的不同架构和能力的 LLM 集成到更强大的 LLM 中，从而减少对昂贵的 LLM 开发的需求。在这项工作中，我们提出了一个通过两个主要阶段进行聊天 LLM 知识融合的新框架，从而产生了 FuseChat。首先，我们对不同结构和规模的源聊天 LLM 进行成对知识融合，以通过轻量级微调创建具有相同结构和大小的多个目标 LLM。在此过程中，引入了基于统计的标记对齐方法作为融合具有不同结构的 LLM 的基石。其次，我们在参数空间内合并这些目标 LLM，其中我们提出了一种新方法，用于根据微调前后参数更新的幅度确定合并系数。我们使用六种具有不同架构和规模的著名聊天 LLM 实现并验证了 FuseChat，包括 OpenChat-3.5-7B、Starling-LM-7B-alpha、NH2-SOLAR-10.7B、InternLM2-Chat-20B、Mixtral-8x7B-Instruct 和 Qwen-1.5-Chat-72B。在两个指令跟踪基准 AlpacaEval 2.0 和 MT-Bench 上的实验结果表明，FuseChat-7B 优于各种规模的基线。我们的模型甚至可以与更大的 Mixtral-8x7B-Instruct 相媲美，并且在 MT-Bench 上接近 GPT-3.5-Turbo-1106。我们的代码、模型权重和数据在 \url{此 https URL} 上公开。

Title: Leveraging Web-Crawled Data for High-Quality Fine-Tuning

• Authors: Jing Zhou, Chenglin Jiang, Wei Shen, Xiao Zhou, Xiaonan He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08003
• Pdf URL: https://arxiv.org/pdf/2408.08003
• Copy Paste: [[2408.08003]] Leveraging Web-Crawled Data for High-Quality Fine-Tuning(https://arxiv.org/abs/2408.08003)
• Keywords: language model, gpt
• Abstract: Most large language models are fine-tuned using either expensive human-annotated data or GPT-4 generated data which cannot guarantee performance in certain domains. We argue that although the web-crawled data often has formatting errors causing semantic inaccuracies, it can still serve as a valuable source for high-quality supervised fine-tuning in specific domains without relying on advanced models like GPT-4. To this end, we create a paired training dataset automatically by aligning web-crawled data with a smaller set of high-quality data. By training a language model on this dataset, we can convert web data with irregular formats into high-quality ones. Our experiments show that training with the model-transformed data yields better results, surpassing training with only high-quality data by an average score of 9.4% in Chinese math problems. Additionally, our 7B model outperforms several open-source models larger than 32B and surpasses well-known closed-source models such as GPT-3.5, highlighting the efficacy of our approach.
• 摘要：大多数大型语言模型都是使用昂贵的人工注释数据或 GPT-4 生成的数据进行微调的，而这些数据无法保证在某些领域的性能。我们认为，尽管网络爬取的数据经常存在格式错误，导致语义不准确，但它仍然可以作为特定领域高质量监督微调的宝贵来源，而无需依赖 GPT-4 等高级模型。为此，我们通过将网络爬取的数据与一组较小的高质量数据对齐，自动创建配对训练数据集。通过在该数据集上训练语言模型，我们可以将格式不规则的网络数据转换为高质量数据。我们的实验表明，使用模型转换后的数据进行训练可以产生更好的结果，在中国数学问题中比仅使用高质量数据的训练平均高出 9.4%。此外，我们的 7B 模型优于几个大于 32B 的开源模型，并超越了 GPT-3.5 等知名闭源模型，凸显了我们方法的有效性。

Title: RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation

• Authors: Dongyu Ru, Lin Qiu, Xiangkun Hu, Tianhang Zhang, Peng Shi, Shuaichen Chang, Jiayang Cheng, Cunxiang Wang, Shichao Sun, Huanyu Li, Zizhao Zhang, Binjie Wang, Jiarong Jiang, Tong He, Zhiguo Wang, Pengfei Liu, Yue Zhang, Zheng Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.08067
• Pdf URL: https://arxiv.org/pdf/2408.08067
• Copy Paste: [[2408.08067]] RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation(https://arxiv.org/abs/2408.08067)
• Keywords: retrieval-augmented generation
• Abstract: Despite Retrieval-Augmented Generation (RAG) has shown promising capability in leveraging external knowledge, a comprehensive evaluation of RAG systems is still challenging due to the modular nature of RAG, evaluation of long-form responses and reliability of measurements. In this paper, we propose a fine-grained evaluation framework, RAGChecker, that incorporates a suite of diagnostic metrics for both the retrieval and generation modules. Meta evaluation verifies that RAGChecker has significantly better correlations with human judgments than other evaluation metrics. Using RAGChecker, we evaluate 8 RAG systems and conduct an in-depth analysis of their performance, revealing insightful patterns and trade-offs in the design choices of RAG architectures. The metrics of RAGChecker can guide researchers and practitioners in developing more effective RAG systems.
• 摘要：尽管检索增强生成 (RAG) 在利用外部知识方面表现出良好的能力，但由于 RAG 的模块化特性、对长格式响应的评估和测量的可靠性，对 RAG 系统的全面评估仍然具有挑战性。在本文中，我们提出了一个细粒度评估框架 RAGChecker，它结合了一套针对检索和生成模块的诊断指标。元评估验证了 RAGChecker 与人类判断的相关性明显优于其他评估指标。使用 RAGChecker，我们评估了 8 个 RAG 系统并对其性能进行了深入分析，揭示了 RAG 架构设计选择中的深刻模式和权衡。RAGChecker 的指标可以指导研究人员和从业人员开发更有效的 RAG 系统。

Title: I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigm

• Authors: Yiming Liang, Ge Zhang, Xingwei Qu, Tianyu Zheng, Jiawei Guo, Xinrun Du, Zhenzhu Yang, Jiaheng Liu, Chenghua Lin, Lei Ma, Wenhao Huang, Jiajun Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08072
• Pdf URL: https://arxiv.org/pdf/2408.08072
• Copy Paste: [[2408.08072]] I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigm(https://arxiv.org/abs/2408.08072)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have achieved significant advancements, however, the common learning paradigm treats LLMs as passive information repositories, neglecting their potential for active learning and alignment. Some approaches train LLMs using their own generated synthetic data, exploring the possibility of active alignment. However, there is still a huge gap between these one-time alignment methods and the continuous automatic alignment of humans. In this paper, we introduce \textbf{I-SHEEP}, an \textbf{I}terative \textbf{S}elf-En\textbf{H}anc\textbf{E}m\textbf{E}nt \textbf{P}aradigm.This human-like paradigm enables LLMs to \textbf{continuously self-align from scratch with nothing}. Compared to the one-time alignment method Dromedary \cite{sun2023principledriven}, which refers to the first iteration in this paper, I-SHEEP can significantly enhance capacities on both Qwen and Llama models. I-SHEEP achieves a maximum relative improvement of 78.2\% in the Alpaca Eval, 24.0\% in the MT Bench, and an absolute increase of 8.88\% in the IFEval accuracy over subsequent iterations in Qwen-1.5 72B model. Additionally, I-SHEEP surpasses the base model in various standard benchmark generation tasks, achieving an average improvement of 24.77\% in code generation tasks, 12.04\% in TrivialQA, and 20.29\% in SQuAD. We also provide new insights based on the experiment results. Our codes, datasets, and models are available at \textbf{https://anonymous.4open.science/r/I-SHEEP}.
• 摘要：大型语言模型 (LLM) 取得了重大进展，然而，常见的学习范式将 LLM 视为被动信息存储库，忽视了其主动学习和对齐的潜力。一些方法使用自己生成的合成数据训练 LLM，探索主动对齐的可能性。然而，这些一次性对齐方法与人类的持续自动对齐之间仍然存在巨大差距。在本文中，我们引入了 \textbf{I} 迭代 \textbf{S}anc\textbf{E}m\textbf{E}nt \textbf{P} 范式。这种类似人类的范式使 LLM 能够 \textbf{从头开始持续自我对齐}。与本文第一次迭代的一次性比对方法 Dromedary \cite{sun2023principledriven} 相比，I-SHEEP 在 Qwen 和 Llama 模型上均能显著提升性能。I-SHEEP 在 Alpaca Eval 中实现了 78.2\% 的最大相对提升，在 MT Bench 中实现了 24.0\% 的最大相对提升，在 Qwen-1.5 72B 模型中，IFEval 准确率在后续迭代中绝对提升了 8.88\%。此外，I-SHEEP 在各种标准基准生成任务中均超越了基础模型，在代码生成任务中平均提升了 24.77\%，在 TrivialQA 中提升了 12.04\%，在 SQuAD 中提升了 20.29\%。我们还根据实验结果提供了新的见解。我们的代码、数据集和模型可在 \textbf{https://anonymous.4open.science/r/I-SHEEP} 上找到。

Title: Extracting Sentence Embeddings from Pretrained Transformer Models

• Authors: Lukas Stankevičius, Mantas Lukoševičius
• Subjects: cs.CL, cs.IR, cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.08073
• Pdf URL: https://arxiv.org/pdf/2408.08073
• Copy Paste: [[2408.08073]] Extracting Sentence Embeddings from Pretrained Transformer Models(https://arxiv.org/abs/2408.08073)
• Keywords: prompt, retrieval-augmented generation
• Abstract: Background/introduction: Pre-trained transformer models shine in many natural language processing tasks and therefore are expected to bear the representation of the input sentence or text meaning. These sentence-level embeddings are also important in retrieval-augmented generation. But do commonly used plain averaging or prompt templates surface it enough? Methods: Given 110M parameters BERT's hidden representations from multiple layers and multiple tokens we tried various ways to extract optimal sentence representations. We tested various token aggregation and representation post-processing techniques. We also tested multiple ways of using a general Wikitext dataset to complement BERTs sentence representations. All methods were tested on 8 Semantic Textual Similarity (STS), 6 short text clustering, and 12 classification tasks. We also evaluated our representation-shaping techniques on other static models, including random token representations. Results: Proposed representation extraction methods improved the performance on STS and clustering tasks for all models considered. Very high improvements for static token-based models, especially random embeddings for STS tasks almost reach the performance of BERT-derived representations. Conclusions: Our work shows that for multiple tasks simple baselines with representation shaping techniques reach or even outperform more complex BERT-based models or are able to contribute to their performance.
• 摘要：背景 / 介绍：预训练的 Transformer 模型在许多自然语言处理任务中大放异彩，因此有望承担输入句子或文本含义的表示。这些句子级嵌入在检索增强生成中也很重要。但常用的普通平均或提示模板是否足以显示它？方法：给定 110M 参数 BERT 的隐藏表示（来自多个层和多个标记），我们尝试了各种方法来提取最佳句子表示。我们测试了各种标记聚合和表示后处理技术。我们还测试了使用通用 Wikitext 数据集来补充 BERT 句子表示的多种方法。所有方法都在 8 个语义文本相似度 (STS)、6 个短文本聚类和 12 个分类任务上进行了测试。我们还在其他静态模型（包括随机标记表示）上评估了我们的表示塑造技术。结果：对于所有考虑的模型，提出的表示提取方法提高了 STS 和聚类任务的性能。对于基于静态 token 的模型，尤其是 STS 任务的随机嵌入，其性能得到了极大的提升，几乎达到了 BERT 衍生表示的性能。结论：我们的工作表明，对于多项任务，使用表示塑造技术的简单基线可以达到甚至超越更复杂的基于 BERT 的模型，或者能够为其性能做出贡献。

Title: AgentCourt: Simulating Court with Adversarial Evolvable Lawyer Agents

• Authors: Guhong Chen, Liyang Fan, Zihan Gong, Nan Xie, Zixuan Li, Ziqiang Liu, Chengming Li, Qiang Qu, Shiwen Ni, Min Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.08089
• Pdf URL: https://arxiv.org/pdf/2408.08089
• Copy Paste: [[2408.08089]] AgentCourt: Simulating Court with Adversarial Evolvable Lawyer Agents(https://arxiv.org/abs/2408.08089)
• Keywords: language model, llm, agent
• Abstract: In this paper, we present a simulation system called AgentCourt that simulates the entire courtroom process. The judge, plaintiff's lawyer, defense lawyer, and other participants are autonomous agents driven by large language models (LLMs). Our core goal is to enable lawyer agents to learn how to argue a case, as well as improving their overall legal skills, through courtroom process simulation. To achieve this goal, we propose an adversarial evolutionary approach for the lawyer-agent. Since AgentCourt can simulate the occurrence and development of court hearings based on a knowledge base and LLM, the lawyer agents can continuously learn and accumulate experience from real court cases. The simulation experiments show that after two lawyer-agents have engaged in a thousand adversarial legal cases in AgentCourt (which can take a decade for real-world lawyers), compared to their pre-evolutionary state, the evolved lawyer agents exhibit consistent improvement in their ability to handle legal tasks. To enhance the credibility of our experimental results, we enlisted a panel of professional lawyers to evaluate our simulations. The evaluation indicates that the evolved lawyer agents exhibit notable advancements in responsiveness, as well as expertise and logical rigor. This work paves the way for advancing LLM-driven agent technology in legal scenarios. Code is available at this https URL.
• 摘要：在本文中，我们提出了一个名为 AgentCourt 的模拟系统，该系统模拟了整个法庭流程。法官、原告律师、辩护律师和其他参与者都是由大型语言模型 (LLM) 驱动的自主代理。我们的核心目标是让律师代理通过法庭流程模拟学习如何辩论案件，并提高他们的整体法律技能。为了实现这一目标，我们提出了一种针对律师代理的对抗性进化方法。由于 AgentCourt 可以基于知识库和 LLM 模拟法庭听证会的发生和发展，律师代理可以不断从真实的法庭案件中学习和积累经验。模拟实验表明，在两位律师代理在 AgentCourt 中参与了一千起对抗性法律案件后（对于现实世界的律师来说，这可能需要十年的时间），与进化前的状态相比，进化后的律师代理在处理法律任务的能力方面表现出持续的提高。为了提高实验结果的可信度，我们聘请了一组专业律师来评估我们的模拟。评估表明，进化后的律师代理在响应能力、专业知识和逻辑严谨性方面表现出显著的进步。这项工作为在法律场景中推进 LLM 驱动的代理技术铺平了道路。代码可在此 https URL 上获取。

Title: MIDAS: Multi-level Intent, Domain, And Slot Knowledge Distillation for Multi-turn NLU

• Authors: Yan Li, So-Eon Kim, Seong-Bae Park, Soyeon Caren Han
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08144
• Pdf URL: https://arxiv.org/pdf/2408.08144
• Copy Paste: [[2408.08144]] MIDAS: Multi-level Intent, Domain, And Slot Knowledge Distillation for Multi-turn NLU(https://arxiv.org/abs/2408.08144)
• Keywords: language model, llm
• Abstract: Although Large Language Models(LLMs) can generate coherent and contextually relevant text, they often struggle to recognise the intent behind the human user's query. Natural Language Understanding (NLU) models, however, interpret the purpose and key information of user's input to enable responsive interactions. Existing NLU models generally map individual utterances to a dual-level semantic frame, involving sentence-level intent and word-level slot labels. However, real-life conversations primarily consist of multi-turn conversations, involving the interpretation of complex and extended dialogues. Researchers encounter challenges addressing all facets of multi-turn dialogue conversations using a unified single NLU model. This paper introduces a novel approach, MIDAS, leveraging a multi-level intent, domain, and slot knowledge distillation for multi-turn NLU. To achieve this, we construct distinct teachers for varying levels of conversation knowledge, namely, sentence-level intent detection, word-level slot filling, and conversation-level domain classification. These teachers are then fine-tuned to acquire specific knowledge of their designated levels. A multi-teacher loss is proposed to facilitate the combination of these multi-level teachers, guiding a student model in multi-turn dialogue tasks. The experimental results demonstrate the efficacy of our model in improving the overall multi-turn conversation understanding, showcasing the potential for advancements in NLU models through the incorporation of multi-level dialogue knowledge distillation techniques.
• 摘要：尽管大型语言模型 (LLM) 可以生成连贯且与上下文相关的文本，但它们通常难以识别人类用户查询背后的意图。然而，自然语言理解 (NLU) 模型会解释用户输入的目的和关键信息，以实现响应式交互。现有的 NLU 模型通常将单个话语映射到双层语义框架，包括句子级意图和单词级槽位标签。然而，现实生活中的对话主要由多轮对话组成，涉及复杂和扩展对话的解释。研究人员在使用统一的单一 NLU 模型解决多轮对话的所有方面时遇到了挑战。本文介绍了一种新方法 MIDAS，利用多级意图、领域和槽位知识提炼进行多轮 NLU。为了实现这一点，我们为不同级别的对话知识构建了不同的教师，即句子级意图检测、单词级槽位填充和对话级领域分类。然后对这些教师进行微调以获取其指定级别的特定知识。提出了多教师损失，以促进这些多级教师的结合，指导学生模型进行多轮对话任务。实验结果证明了我们的模型在提高整体多轮对话理解方面的有效性，展示了通过结合多级对话知识提炼技术在 NLU 模型中取得进步的潜力。

Title: KOALA: Enhancing Speculative Decoding for LLM via Multi-Layer Draft Heads with Adversarial Learning

• Authors: Kaiqi Zhang, Jing Zhao, Rui Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08146
• Pdf URL: https://arxiv.org/pdf/2408.08146
• Copy Paste: [[2408.08146]] KOALA: Enhancing Speculative Decoding for LLM via Multi-Layer Draft Heads with Adversarial Learning(https://arxiv.org/abs/2408.08146)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) exhibit high inference latency due to their autoregressive decoding nature. While the draft head in speculative decoding mitigates this issue, its full potential remains unexplored. In this paper, we introduce KOALA (K-layer Optimized Adversarial Learning Architecture), an orthogonal approach to the draft head. By transforming the conventional single-layer draft head into a multi-layer architecture and incorporating adversarial learning into the traditional supervised training, KOALA significantly improves the accuracy of the draft head in predicting subsequent tokens, thus more closely mirroring the functionality of LLMs. Although this improvement comes at the cost of slightly increased drafting overhead, KOALA substantially unlocks the draft head's potential, greatly enhancing speculative decoding. We conducted comprehensive evaluations of KOALA, including both autoregressive and non-autoregressive draft heads across various tasks, demonstrating a latency speedup ratio improvement of 0.24x-0.41x, which is 10.57%-14.09% faster than the original draft heads.
• 摘要：大型语言模型 (LLM) 由于其自回归解码特性而表现出较高的推理延迟。虽然推测解码中的草稿头可以缓解这个问题，但其全部潜力仍未得到充分挖掘。在本文中，我们介绍了 KOALA（K 层优化对抗学习架构），一种与草稿头正交的方法。通过将传统的单层草稿头转变为多层架构，并将对抗学习纳入传统的监督训练中，KOALA 显著提高了草稿头预测后续标记的准确性，从而更紧密地反映了 LLM 的功能。虽然这种改进是以略微增加草稿开销为代价的，但 KOALA 大大释放了草稿头的潜力，大大增强了推测解码。我们对 KOALA 进行了全面的评估，包括各种任务中的自回归和非自回归草案头，结果显示延迟加速比提高了 0.24 倍至 0.41 倍，比原始草案头快 10.57% 至 14.09%。

Title: DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search

• Authors: Huajian Xin, Z.Z. Ren, Junxiao Song, Zhihong Shao, Wanjia Zhao, Haocheng Wang, Bo Liu, Liyue Zhang, Xuan Lu, Qiushi Du, Wenjun Gao, Qihao Zhu, Dejian Yang, Zhibin Gou, Z.F. Wu, Fuli Luo, Chong Ruan
• Subjects: cs.CL, cs.AI, cs.LG, cs.LO
• Abstract URL: https://arxiv.org/abs/2408.08152
• Pdf URL: https://arxiv.org/pdf/2408.08152
• Copy Paste: [[2408.08152]] DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search(https://arxiv.org/abs/2408.08152)
• Keywords: language model
• Abstract: We introduce DeepSeek-Prover-V1.5, an open-source language model designed for theorem proving in Lean 4, which enhances DeepSeek-Prover-V1 by optimizing both training and inference processes. Pre-trained on DeepSeekMath-Base with specialization in formal mathematical languages, the model undergoes supervised fine-tuning using an enhanced formal theorem proving dataset derived from DeepSeek-Prover-V1. Further refinement is achieved through reinforcement learning from proof assistant feedback (RLPAF). Beyond the single-pass whole-proof generation approach of DeepSeek-Prover-V1, we propose RMaxTS, a variant of Monte-Carlo tree search that employs an intrinsic-reward-driven exploration strategy to generate diverse proof paths. DeepSeek-Prover-V1.5 demonstrates significant improvements over DeepSeek-Prover-V1, achieving new state-of-the-art results on the test set of the high school level miniF2F benchmark ($63.5\%$) and the undergraduate level ProofNet benchmark ($25.3\%$).
• 摘要：我们在 Lean 4 中引入了 DeepSeek-Prover-V1.5，这是一个专为定理证明而设计的开源语言模型，它通过优化训练和推理过程增强了 DeepSeek-Prover-V1。该模型在 DeepSeekMath-Base 上进行了预训练，专门用于形式数学语言，并使用源自 DeepSeek-Prover-V1 的增强形式定理证明数据集进行监督微调。通过从证明助手反馈 (RLPAF) 进行强化学习可以进一步细化。除了 DeepSeek-Prover-V1 的单遍整体证明生成方法之外，我们还提出了 RMaxTS，这是蒙特卡洛树搜索的一种变体，它采用内在奖励驱动的探索策略来生成不同的证明路径。 DeepSeek-Prover-V1.5 相较于 DeepSeek-Prover-V1 有显著的改进，在高中水平 miniF2F 基准 ($63.5\%$) 和本科水平 ProofNet 基准 ($25.3\%$) 的测试集上取得了新的最优结果。

Title: Covert Bias: The Severity of Social Views' Unalignment Towards Implicit and Explicit Opinion

• Authors: Abeer Aldayel, Areej Alokaili, Rehab Alahmadi
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.08212
• Pdf URL: https://arxiv.org/pdf/2408.08212
• Copy Paste: [[2408.08212]] Covert Bias: The Severity of Social Views' Unalignment Towards Implicit and Explicit Opinion(https://arxiv.org/abs/2408.08212)
• Keywords: llm
• Abstract: While various approaches have recently been studied for bias identification, little is known about how implicit language that does not explicitly convey a viewpoint affects bias amplification in large language this http URL examine the severity of bias toward a view, we evaluated the performance of two downstream tasks where the implicit and explicit knowledge of social groups were used. First, we present a stress test evaluation by using a biased model in edge cases of excessive bias scenarios. Then, we evaluate how LLMs calibrate linguistically in response to both implicit and explicit opinions when they are aligned with conflicting viewpoints. Our findings reveal a discrepancy in LLM performance in identifying implicit and explicit opinions, with a general tendency of bias toward explicit opinions of opposing stances. Moreover, the bias-aligned models generate more cautious responses using uncertainty phrases compared to the unaligned (zero-shot) base models. The direct, incautious responses of the unaligned models suggest a need for further refinement of decisiveness by incorporating uncertainty markers to enhance their reliability, especially on socially nuanced topics with high subjectivity.
• 摘要：虽然最近已经研究了各种偏见识别方法，但人们对未明确传达观点的隐性语言如何影响大语言中的偏见放大知之甚少。为了检查对观点的偏见的严重程度，我们评估了两个下游任务的表现，其中使用了社会群体的隐性和显性知识。首先，我们通过在过度偏见场景的边缘情况下使用有偏见的模型进行压力测试评估。然后，我们评估当 LLM 与相互冲突的观点一致时，它们在语言上如何校准以响应隐性和显性意见。我们的研究结果表明，LLM 在识别隐性和显性意见方面的表现存在差异，并且普遍倾向于对立立场的显性意见产生偏见。此外，与未对齐（零样本）基础模型相比，偏见对齐模型使用不确定性短语会生成更谨慎的响应。未对齐模型的直接、不谨慎的反应表明需要通过结合不确定性标记来进一步改进决策性，以提高其可靠性，尤其是在具有高度主观性的社会细微差别主题上。

Title: mhGPT: A Lightweight Generative Pre-Trained Transformer for Mental Health Text Analysis

• Authors: Dae-young Kim (1), Rebecca Hwa (2), Muhammad Mahbubur Rahman (1) ((1) Children's National Hospital, Washington, DC, (2) George Washington University, Washington, DC)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08261
• Pdf URL: https://arxiv.org/pdf/2408.08261
• Copy Paste: [[2408.08261]] mhGPT: A Lightweight Generative Pre-Trained Transformer for Mental Health Text Analysis(https://arxiv.org/abs/2408.08261)
• Keywords: gpt
• Abstract: This paper introduces mhGPT, a lightweight generative pre-trained transformer trained on mental health-related social media and PubMed articles. Fine-tuned for specific mental health tasks, mhGPT was evaluated under limited hardware constraints and compared with state-of-the-art models like MentaLLaMA and Gemma. Despite having only 1.98 billion parameters and using just 5% of the dataset, mhGPT outperformed larger models and matched the performance of models trained on significantly more data. The key contributions include integrating diverse mental health data, creating a custom tokenizer, and optimizing a smaller architecture for low-resource settings. This research could advance AI-driven mental health care, especially in areas with limited computing power.
• 摘要：本文介绍了 mhGPT，这是一种轻量级的生成式预训练转换器，在心理健康相关的社交媒体和 PubMed 文章上进行训练。mhGPT 针对特定的心理健康任务进行了微调，在有限的硬件限制下进行了评估，并与 MentaLLaMA 和 Gemma 等最先进的模型进行了比较。尽管只有 19.8 亿个参数，并且只使用了 5% 的数据集，但 mhGPT 的表现优于更大的模型，并且与在更多数据上训练的模型的性能相当。主要贡献包括整合各种心理健康数据、创建自定义标记器以及针对资源匮乏的环境优化较小的架构。这项研究可以推动人工智能驱动的心理健康护理，尤其是在计算能力有限的地区。

Title: The ShareLM Collection and Plugin: Contributing Human-Model Chats for the Benefit of the Community

• Authors: Shachar Don-Yehiya, Leshem Choshen, Omri Abend
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08291
• Pdf URL: https://arxiv.org/pdf/2408.08291
• Copy Paste: [[2408.08291]] The ShareLM Collection and Plugin: Contributing Human-Model Chats for the Benefit of the Community(https://arxiv.org/abs/2408.08291)
• Keywords: language model, chat
• Abstract: Human-model conversations provide a window into users' real-world scenarios, behavior, and needs, and thus are a valuable resource for model development and research. While for-profit companies collect user data through the APIs of their models, using it internally to improve their own models, the open source and research community lags behind. We introduce the ShareLM collection, a unified set of human conversations with large language models, and its accompanying plugin, a Web extension for voluntarily contributing user-model conversations. Where few platforms share their chats, the ShareLM plugin adds this functionality, thus, allowing users to share conversations from most platforms. The plugin allows the user to rate their conversations, both at the conversation and the response levels, and delete conversations they prefer to keep private before they ever leave the user's local storage. We release the plugin conversations as part of the ShareLM collection, and call for more community effort in the field of open human-model data. The code, plugin, and data are available.
• 摘要：人机对话为用户提供了一个了解真实场景、行为和需求的窗口，因此是模型开发和研究的宝贵资源。虽然营利性公司通过其模型的 API 收集用户数据，并在内部使用这些数据来改进自己的模型，但开源和研究社区却落后了。我们推出了 ShareLM 集合，这是一组具有大型语言模型的统一人机对话，以及其附带的插件，这是一个用于自愿贡献用户模型对话的 Web 扩展。很少有平台会共享他们的聊天记录，而 ShareLM 插件则添加了此功能，从而允许用户共享来自大多数平台的对话。该插件允许用户在对话和响应级别对他们的对话进行评分，并在他们希望保持私密的对话离开用户的本地存储之前删除它们。我们将插件对话作为 ShareLM 集合的一部分发布，并呼吁社区在开放人机模型数据领域做出更多努力。代码、插件和数据均可用。

Title: ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws

• Authors: Ruihang Li, Yixuan Wei, Miaosen Zhang, Nenghai Yu, Han Hu, Houwen Peng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.08310
• Pdf URL: https://arxiv.org/pdf/2408.08310
• Copy Paste: [[2408.08310]] ScalingFilter: Assessing Data Quality through Inverse Utilization of Scaling Laws(https://arxiv.org/abs/2408.08310)
• Keywords: language model
• Abstract: High-quality data is crucial for the pre-training performance of large language models. Unfortunately, existing quality filtering methods rely on a known high-quality dataset as reference, which can introduce potential bias and compromise diversity. In this paper, we propose ScalingFilter, a novel approach that evaluates text quality based on the perplexity difference between two language models trained on the same data, thereby eliminating the influence of the reference dataset in the filtering process. An theoretical analysis shows that ScalingFilter is equivalent to an inverse utilization of scaling laws. Through training models with 1.3B parameters on the same data source processed by various quality filters, we find ScalingFilter can improve zero-shot performance of pre-trained models in downstream tasks. To assess the bias introduced by quality filtering, we introduce semantic diversity, a metric of utilizing text embedding models for semantic representations. Extensive experiments reveal that semantic diversity is a reliable indicator of dataset diversity, and ScalingFilter achieves an optimal balance between downstream performance and semantic diversity.
• 摘要：高质量数据对于大型语言模型的预训练性能至关重要。不幸的是，现有的质量过滤方法依赖于已知的高质量数据集作为参考，这可能会引入潜在的偏差并损害多样性。在本文中，我们提出了一种新颖的方法 ScalingFilter，它基于在同一数据上训练的两个语言模型之间的困惑度差异来评估文本质量，从而消除了参考数据集在过滤过程中的影响。理论分析表明，ScalingFilter 相当于缩放定律的逆利用。通过在经过各种质量过滤器处理的同一数据源上训练具有 1.3B 参数的模型，我们发现 ScalingFilter 可以提高预训练模型在下游任务中的零样本性能。为了评估质量过滤引入的偏差，我们引入了语义多样性，这是利用文本嵌入模型进行语义表示的度量。大量实验表明，语义多样性是数据集多样性的可靠指标，ScalingFilter 在下游性能和语义多样性之间实现了最佳平衡。