2024-09-26

Title: Exploring the traditional NMT model and Large Language Model for chat translation

• Authors: Jinlong Yang, Hengchao Shang, Daimeng Wei, Jiaxin Guo, Zongyao Li, Zhanglin Wu, Zhiqiang Rao, Shaojun Li, Yuhao Xie, Yuanchang Luo, Jiawei Zheng, Bin Wei, Hao Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16331
• Pdf URL: https://arxiv.org/pdf/2409.16331
• Copy Paste: [[2409.16331]] Exploring the traditional NMT model and Large Language Model for chat translation(https://arxiv.org/abs/2409.16331)
• Keywords: language model, chat
• Abstract: This paper describes the submissions of Huawei Translation Services Center(HW-TSC) to WMT24 chat translation shared task on English$\leftrightarrow$Germany (en-de) bidirection. The experiments involved fine-tuning models using chat data and exploring various strategies, including Minimum Bayesian Risk (MBR) decoding and self-training. The results show significant performance improvements in certain directions, with the MBR self-training method achieving the best results. The Large Language Model also discusses the challenges and potential avenues for further research in the field of chat translation.
• 摘要：本文介绍了华为翻译服务中心 (HW-TSC) 在英语和德语 (en-de) 双向 WMT24 聊天翻译共享任务中的提交情况。实验涉及使用聊天数据微调模型并探索各种策略，包括最小贝叶斯风险 (MBR) 解码和自训练。结果显示在某些方向上性能有显著提升，其中 MBR 自训练方法取得了最佳效果。大型语言模型还讨论了聊天翻译领域面临的挑战和进一步研究的潜在途径。

Title: Do the Right Thing, Just Debias! Multi-Category Bias Mitigation Using LLMs

• Authors: Amartya Roy, Danush Khanna, Devanshu Mahapatra, Vasanthakumar, Avirup Das, Kripabandhu Ghosh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16371
• Pdf URL: https://arxiv.org/pdf/2409.16371
• Copy Paste: [[2409.16371]] Do the Right Thing, Just Debias! Multi-Category Bias Mitigation Using LLMs(https://arxiv.org/abs/2409.16371)
• Keywords: llm
• Abstract: This paper tackles the challenge of building robust and generalizable bias mitigation models for language. Recognizing the limitations of existing datasets, we introduce ANUBIS, a novel dataset with 1507 carefully curated sentence pairs encompassing nine social bias categories. We evaluate state-of-the-art models like T5, utilizing Supervised Fine-Tuning (SFT), Reinforcement Learning (PPO, DPO), and In-Context Learning (ICL) for effective bias mitigation. Our analysis focuses on multi-class social bias reduction, cross-dataset generalizability, and environmental impact of the trained models. ANUBIS and our findings offer valuable resources for building more equitable AI systems and contribute to the development of responsible and unbiased technologies with broad societal impact.
• 摘要：本文旨在解决构建稳健且可推广的语言偏见缓解模型的挑战。认识到现有数据集的局限性，我们引入了 ANUBIS，这是一个新数据集，包含 1507 个精心挑选的句子对，涵盖九个社会偏见类别。我们评估了 T5 等最先进的模型，利用监督微调 (SFT)、强化学习 (PPO、DPO) 和上下文学习 (ICL) 来有效缓解偏见。我们的分析侧重于多类别社会偏见减少、跨数据集通用性和训练模型的环境影响。ANUBIS 和我们的研究结果为构建更公平的人工智能系统提供了宝贵的资源，并有助于开发具有广泛社会影响的负责任和无偏见的技术。

Title: RISCORE: Enhancing In-Context Riddle Solving in Language Models through Context-Reconstructed Example Augmentation

• Authors: Ioannis Panagiotopoulos, Giorgos Filandrianos, Maria Lymperaiou, Giorgos Stamou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16383
• Pdf URL: https://arxiv.org/pdf/2409.16383
• Copy Paste: [[2409.16383]] RISCORE: Enhancing In-Context Riddle Solving in Language Models through Context-Reconstructed Example Augmentation(https://arxiv.org/abs/2409.16383)
• Keywords: language model, llm, prompt
• Abstract: Riddle-solving requires advanced reasoning skills, pushing LLMs to engage in abstract thinking and creative problem-solving, often revealing limitations in their cognitive abilities. In this paper, we examine the riddle-solving capabilities of LLMs using a multiple-choice format, exploring how different prompting techniques impact performance on riddles that demand diverse reasoning skills. To enhance results, we introduce RISCORE (RIddle Solving with COntext REcontruciton) a novel fully automated prompting method that generates and utilizes contextually reconstructed sentence-based puzzles in conjunction with the original examples to create few-shot exemplars. Our experiments demonstrate that RISCORE significantly improves the performance of language models in both vertical and lateral thinking tasks, surpassing traditional exemplar selection strategies across a variety of few-shot settings.
• 摘要：解谜需要高级推理能力，这迫使法学硕士进行抽象思维和创造性解决问题，这通常会暴露出他们认知能力的局限性。在本文中，我们使用多项选择题形式检查法学硕士的解谜能力，探索不同的提示技术如何影响需要多种推理能力的谜语的表现。为了提高结果，我们引入了 RISCORE（使用上下文重构的谜题解决），这是一种新颖的全自动提示方法，它生成并利用上下文重构的基于句子的谜题与原始示例结合来创建少量样本样本。我们的实验表明，RISCORE 显著提高了语言模型在垂直和横向思维任务中的表现，超越了各种少量设置中的传统样本选择策略。

Title: A Comprehensive Survey of Bias in LLMs: Current Landscape and Future Directions

• Authors: Rajesh Ranjan, Shailja Gupta, Surya Narayan Singh
• Subjects: cs.CL, cs.AI, cs.CY, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.16430
• Pdf URL: https://arxiv.org/pdf/2409.16430
• Copy Paste: [[2409.16430]] A Comprehensive Survey of Bias in LLMs: Current Landscape and Future Directions(https://arxiv.org/abs/2409.16430)
• Keywords: language model, llm
• Abstract: Large Language Models(LLMs) have revolutionized various applications in natural language processing (NLP) by providing unprecedented text generation, translation, and comprehension capabilities. However, their widespread deployment has brought to light significant concerns regarding biases embedded within these models. This paper presents a comprehensive survey of biases in LLMs, aiming to provide an extensive review of the types, sources, impacts, and mitigation strategies related to these biases. We systematically categorize biases into several dimensions. Our survey synthesizes current research findings and discusses the implications of biases in real-world applications. Additionally, we critically assess existing bias mitigation techniques and propose future research directions to enhance fairness and equity in LLMs. This survey serves as a foundational resource for researchers, practitioners, and policymakers concerned with addressing and understanding biases in LLMs.
• 摘要：大型语言模型 (LLM) 提供了前所未有的文本生成、翻译和理解功能，彻底改变了自然语言处理 (NLP) 中的各种应用。然而，它们的广泛部署也引发了人们对这些模型中嵌入的偏见的重大担忧。本文对 LLM 中的偏见进行了全面的调查，旨在广泛回顾与这些偏见相关的类型、来源、影响和缓解策略。我们系统地将偏见分为几个维度。我们的调查综合了当前的研究结果，并讨论了偏见在实际应用中的影响。此外，我们批判性地评估了现有的偏见缓解技术，并提出了未来的研究方向，以提高 LLM 的公平性和公正性。这项调查为关注解决和理解 LLM 中偏见的研究人员、从业者和政策制定者提供了基础资源。

Title: FMDLlama: Financial Misinformation Detection based on Large Language Models

• Authors: Zhiwei Liu, Xin Zhang, Kailai Yang, Qianqian Xie, Jimin Huang, Sophia Ananiadou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16452
• Pdf URL: https://arxiv.org/pdf/2409.16452
• Copy Paste: [[2409.16452]] FMDLlama: Financial Misinformation Detection based on Large Language Models(https://arxiv.org/abs/2409.16452)
• Keywords: language model, gpt, llm, chat
• Abstract: The emergence of social media has made the spread of misinformation easier. In the financial domain, the accuracy of information is crucial for various aspects of financial market, which has made financial misinformation detection (FMD) an urgent problem that needs to be addressed. Large language models (LLMs) have demonstrated outstanding performance in various fields. However, current studies mostly rely on traditional methods and have not explored the application of LLMs in the field of FMD. The main reason is the lack of FMD instruction tuning datasets and evaluation benchmarks. In this paper, we propose FMDLlama, the first open-sourced instruction-following LLMs for FMD task based on fine-tuning Llama3.1 with instruction data, the first multi-task FMD instruction dataset (FMDID) to support LLM instruction tuning, and a comprehensive FMD evaluation benchmark (FMD-B) with classification and explanation generation tasks to test the FMD ability of LLMs. We compare our models with a variety of LLMs on FMD-B, where our model outperforms all other open-sourced LLMs as well as ChatGPT.
• 摘要：社交媒体的出现使得虚假信息的传播更加容易。在金融领域，信息的准确性对金融市场的各个方面都至关重要，这使得金融虚假信息检测（FMD）成为亟待解决的问题。大型语言模型（LLM）在各个领域都表现出色。然而，当前的研究大多依赖于传统方法，尚未探索LLM在FMD领域的应用。主要原因是缺乏FMD指令调优数据集和评估基准。在本文中，我们提出了第一个基于指令数据微调Llama3.1的用于FMD任务的开源指令跟踪LLM——FMDLlama，第一个支持LLM指令调优的多任务FMD指令数据集（FMDID），以及一个具有分类和解释生成任务的综合FMD评估基准（FMD-B）来测试LLM的FMD能力。我们将我们的模型与 FMD-B 上的各种 LLM 进行比较，我们的模型优于所有其他开源 LLM 以及 ChatGPT。

Title: Strategies for Improving NL-to-FOL Translation with LLMs: Data Generation, Incremental Fine-Tuning, and Verification

• Authors: Ramya Keerthy Thatikonda, Jiuzhou Han, Wray Buntine, Ehsan Shareghi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16461
• Pdf URL: https://arxiv.org/pdf/2409.16461
• Copy Paste: [[2409.16461]] Strategies for Improving NL-to-FOL Translation with LLMs: Data Generation, Incremental Fine-Tuning, and Verification(https://arxiv.org/abs/2409.16461)
• Keywords: language model, gpt, llm
• Abstract: Logical reasoning is a fundamental task in natural language processing that presents significant challenges to Large Language Models (LLMs). The inherent characteristics of logical reasoning makes it well-suited for symbolic representations such as first-order logic (FOL). Research in symbolic logical reasoning explored FOL generation using state-of-the-art LLMs (i.e., GPT-4) to produce FOL translations of natural language (NL) statements, but errors in translation are usually not the focus. We address this by categorizing the translation errors in FOL statements generated by LLMs. To make progress towards improving the quality of FOL translations for smaller language models such as LLaMA-2 13B and Mistral 7B, we create ProofFOL, a high-quality FOL-annotated subset of ProofWriter dataset using GPT-4o. The models fine-tuned on this silver standard data achieve a significant gain in performance when compared to larger language models such as LLaMA-2 70B. In addition to improving the model using large data, we also tackle the issue of data scarcity and introduce an incremental framework encompassing of data augmentation and verification steps. In the augmentation process, a single pair of (premises, conclusion) is split into multiple new instances based on the predicates and FOLs. This data is used for fine-tuning, and the inference on this model generates FOLs with fewer errors over the model trained on the original data. Our investigation on the translation errors leads to generation of a perturbation dataset, which is used to train a verifier that corrects potential syntactic and semantic FOL translation errors. We demonstrate an efficient method for making the most of a limited existing human-annotated dataset. Our results show state-of-the-art performance for ProofWriter and ProntoQA datasets using ProofFOL on LLaMA-2 and Mistral models.
• 摘要：逻辑推理是自然语言处理中的一项基本任务，对大型语言模型 (LLM) 提出了重大挑战。逻辑推理的固有特性使其非常适合符号表示，例如一阶逻辑 (FOL)。符号逻辑推理研究探索了使用最先进的 LLM（即 GPT-4）生成自然语言 (NL) 语句的 FOL 翻译，但翻译中的错误通常不是重点。我们通过对 LLM 生成的 FOL 语句中的翻译错误进行分类来解决这个问题。为了提高 LLaMA-2 13B 和 Mistral 7B 等较小语言模型的 FOL 翻译质量，我们使用 GPT-4o 创建了 ProofFOL，这是 ProofWriter 数据集的高质量 FOL 注释子集。与 LLaMA-2 70B 等较大的语言模型相比，在此银标准数据上微调的模型在性能上取得了显着提升。除了使用大数据改进模型外，我们还解决了数据稀缺问题，并引入了一个包含数据增强和验证步骤的增量框架。在增强过程中，一对（前提，结论）根据谓词和 FOL 被拆分为多个新实例。这些数据用于微调，并且对该模型的推理会生成比在原始数据上训练的模型错误更少的 FOL。我们对翻译错误的调查导致生成扰动数据集，该数据集用于训练验证器以纠正潜在的句法和语义 FOL 翻译错误。我们展示了一种充分利用现有有限的人工注释数据集的有效方法。我们的结果显示，在 LLaMA-2 和 Mistral 模型上使用 ProofFOL 的 ProofWriter 和 ProntoQA 数据集具有最佳性能。

Title: Exploring Knowledge Tracing in Tutor-Student Dialogues

• Authors: Alexander Scarlatos, Andrew Lan
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16490
• Pdf URL: https://arxiv.org/pdf/2409.16490
• Copy Paste: [[2409.16490]] Exploring Knowledge Tracing in Tutor-Student Dialogues(https://arxiv.org/abs/2409.16490)
• Keywords: language model, llm, prompt, chat
• Abstract: Recent advances in large language models (LLMs) have led to the development of artificial intelligence (AI)-powered tutoring chatbots, showing promise in providing broad access to high-quality personalized education. Existing works have primarily studied how to make LLMs follow tutoring principles but not how to model student behavior in dialogues. However, analyzing student dialogue turns can serve as a formative assessment, since open-ended student discourse may indicate their knowledge levels and reveal specific misconceptions. In this work, we present a first attempt at performing knowledge tracing (KT) in tutor-student dialogues. We propose LLM prompting methods to identify the knowledge components/skills involved in each dialogue turn and diagnose whether the student responds correctly to the tutor, and verify the LLM's effectiveness via an expert human evaluation. We then apply a range of KT methods on the resulting labeled data to track student knowledge levels over an entire dialogue. We conduct experiments on two tutoring dialogue datasets, and show that a novel yet simple LLM-based method, LLMKT, significantly outperforms existing KT methods in predicting student response correctness in dialogues. We perform extensive qualitative analyses to highlight the challenges in dialogue KT and outline multiple avenues for future work.
• 摘要：大型语言模型 (LLM) 的最新进展推动了人工智能 (AI) 驱动的辅导聊天机器人的发展，有望为人们提供广泛的高质量个性化教育。现有研究主要研究如何让 LLM 遵循辅导原则，而不是如何模拟学生在对话中的行为。然而，分析学生对话轮次可以作为一种形成性评估，因为开放式的学生话语可能表明他们的知识水平并揭示特定的误解。在这项工作中，我们首次尝试在导师与学生的对话中进行知识追踪 (KT)。我们提出了 LLM 提示方法来识别每个对话轮次中涉及的知识成分/技能，并诊断学生是否正确回应导师，并通过专家人工评估验证 LLM 的有效性。然后，我们对得到的标记数据应用一系列 KT 方法来跟踪整个对话中学生的知识水平。我们对两个辅导对话数据集进行了实验，并表明一种新颖而简单的基于 LLM 的方法 LLMKT 在预测对话中学生回答正确性方面明显优于现有的 KT 方法。我们进行了广泛的定性分析，以突出对话 KT 中的挑战并概述了未来工作的多种途径。

Title: Understanding the Cognitive Complexity in Language Elicited by Product Images

• Authors: Yan-Ying Chen, Shabnam Hakimi, Monica Van, Francine Chen, Matthew Hong, Matt Klenk, Charlene Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16521
• Pdf URL: https://arxiv.org/pdf/2409.16521
• Copy Paste: [[2409.16521]] Understanding the Cognitive Complexity in Language Elicited by Product Images(https://arxiv.org/abs/2409.16521)
• Keywords: language model, llm
• Abstract: Product images (e.g., a phone) can be used to elicit a diverse set of consumer-reported features expressed through language, including surface-level perceptual attributes (e.g., "white") and more complex ones, like perceived utility (e.g., "battery"). The cognitive complexity of elicited language reveals the nature of cognitive processes and the context required to understand them; cognitive complexity also predicts consumers' subsequent choices. This work offers an approach for measuring and validating the cognitive complexity of human language elicited by product images, providing a tool for understanding the cognitive processes of human as well as virtual respondents simulated by Large Language Models (LLMs). We also introduce a large dataset that includes diverse descriptive labels for product images, including human-rated complexity. We demonstrate that human-rated cognitive complexity can be approximated using a set of natural language models that, combined, roughly capture the complexity construct. Moreover, this approach is minimally supervised and scalable, even in use cases with limited human assessment of complexity.
• 摘要：产品图像（例如手机）可用于引出消费者通过语言表达的多种报告特征，包括表面感知属性（例如“白色”）和更复杂的属性，如感知效用（例如“电池”）。引出语言的认知复杂性揭示了认知过程的性质以及理解它们所需的背景；认知复杂性还可以预测消费者的后续选择。这项研究提供了一种测量和验证产品图像引发的人类语言认知复杂性的方法，提供了一种理解人类认知过程的工具，以及由大型语言模型 (LLM) 模拟的虚拟受访者的认知过程。我们还引入了一个大型数据集，其中包括产品图像的各种描述性标签，包括人类评定的复杂性。我们证明，可以使用一组自然语言模型来近似人类评定的认知复杂性，这些模型结合起来大致捕捉了复杂性构造。此外，这种方法是最低限度监督和可扩展的，即使在人类对复杂性的评估有限的用例中也是如此。

Title: Disentangling Questions from Query Generation for Task-Adaptive Retrieval

• Authors: Yoonsang Lee, Minsoo Kim, Seung-won Hwang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16570
• Pdf URL: https://arxiv.org/pdf/2409.16570
• Copy Paste: [[2409.16570]] Disentangling Questions from Query Generation for Task-Adaptive Retrieval(https://arxiv.org/abs/2409.16570)
• Keywords: llm
• Abstract: This paper studies the problem of information retrieval, to adapt to unseen tasks. Existing work generates synthetic queries from domain-specific documents to jointly train the retriever. However, the conventional query generator assumes the query as a question, thus failing to accommodate general search intents. A more lenient approach incorporates task-adaptive elements, such as few-shot learning with an 137B LLM. In this paper, we challenge a trend equating query and question, and instead conceptualize query generation task as a "compilation" of high-level intent into task-adaptive query. Specifically, we propose EGG, a query generator that better adapts to wide search intents expressed in the BeIR benchmark. Our method outperforms baselines and existing models on four tasks with underexplored intents, while utilizing a query generator 47 times smaller than the previous state-of-the-art. Our findings reveal that instructing the LM with explicit search intent is a key aspect of modeling an effective query generator.
• 摘要：本文研究信息检索问题，以适应看不见的任务。现有工作从特定领域的文档生成合成查询以联合训练检索器。但是，传统的查询生成器将查询视为问题，因此无法适应一般的搜索意图。更宽松的方法结合了任务自适应元素，例如使用 137B LLM 进行少量学习。在本文中，我们挑战了将查询和问题等同起来的趋势，而是将查询生成任务概念化为将高级意图“编译”为任务自适应查询。具体来说，我们提出了 EGG，这是一种查询生成器，可以更好地适应 BeIR 基准中表达的广泛搜索意图。我们的方法在四个意图未被充分探索的任务上优于基线和现有模型，同时使用的查询生成器比以前最先进的方法小 47 倍。我们的研究结果表明，用明确的搜索意图指导 LM 是建模有效查询生成器的关键方面。

Title: Evaluating and Enhancing Large Language Models for Novelty Assessment in Scholarly Publications

• Authors: Ethan Lin, Zhiyuan Peng, Yi Fang
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16605
• Pdf URL: https://arxiv.org/pdf/2409.16605
• Copy Paste: [[2409.16605]] Evaluating and Enhancing Large Language Models for Novelty Assessment in Scholarly Publications(https://arxiv.org/abs/2409.16605)
• Keywords: language model, llm
• Abstract: Recent studies have evaluated the creativity/novelty of large language models (LLMs) primarily from a semantic perspective, using benchmarks from cognitive science. However, accessing the novelty in scholarly publications is a largely unexplored area in evaluating LLMs. In this paper, we introduce a scholarly novelty benchmark (SchNovel) to evaluate LLMs' ability to assess novelty in scholarly papers. SchNovel consists of 15000 pairs of papers across six fields sampled from the arXiv dataset with publication dates spanning 2 to 10 years apart. In each pair, the more recently published paper is assumed to be more novel. Additionally, we propose RAG-Novelty, which simulates the review process taken by human reviewers by leveraging the retrieval of similar papers to assess novelty. Extensive experiments provide insights into the capabilities of different LLMs to assess novelty and demonstrate that RAG-Novelty outperforms recent baseline models.
• 摘要：最近的研究主要从语义角度评估了大型语言模型 (LLM) 的创造力/新颖性，使用的是认知科学的基准。然而，在评估 LLM 时，评估学术出版物的新颖性是一个尚未探索的领域。在本文中，我们引入了一个学术新颖性基准 (SchNovel) 来评估 LLM 评估学术论文新颖性的能力。SchNovel 由 arXiv 数据集中抽样的六个领域的 15000 对论文组成，出版日期相隔 2 至 10 年。在每一对中，最近发表的论文被认为更新颖。此外，我们提出了 RAG-Novelty，它通过利用检索类似论文来评估新颖性，从而模拟人类审阅者的审阅过程。大量实验深入了解了不同 LLM 评估新颖性的能力，并证明 RAG-Novelty 优于最近的基线模型。

Title: Claim-Guided Textual Backdoor Attack for Practical Applications

• Authors: Minkyoo Song, Hanna Kim, Jaehan Kim, Youngjin Jin, Seungwon Shin
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.16618
• Pdf URL: https://arxiv.org/pdf/2409.16618
• Copy Paste: [[2409.16618]] Claim-Guided Textual Backdoor Attack for Practical Applications(https://arxiv.org/abs/2409.16618)
• Keywords: language model
• Abstract: Recent advances in natural language processing and the increased use of large language models have exposed new security vulnerabilities, such as backdoor attacks. Previous backdoor attacks require input manipulation after model distribution to activate the backdoor, posing limitations in real-world applicability. Addressing this gap, we introduce a novel Claim-Guided Backdoor Attack (CGBA), which eliminates the need for such manipulations by utilizing inherent textual claims as triggers. CGBA leverages claim extraction, clustering, and targeted training to trick models to misbehave on targeted claims without affecting their performance on clean data. CGBA demonstrates its effectiveness and stealthiness across various datasets and models, significantly enhancing the feasibility of practical backdoor attacks. Our code and data will be available at this https URL.
• 摘要：自然语言处理的最新进展和大型语言模型的广泛使用暴露了新的安全漏洞，例如后门攻击。以前的后门攻击需要在模型分发后进行输入操作才能激活后门，这在现实世界的适用性方面造成了限制。为了解决这一问题，我们引入了一种新颖的声明引导后门攻击 (CGBA)，它利用固有的文本声明作为触发器，消除了此类操作的需要。CGBA 利用声明提取、聚类和有针对性的训练来诱使模型对有针对性的声明做出不当行为，而不会影响其在干净数据上的性能。CGBA 在各种数据集和模型中展示了其有效性和隐蔽性，大大提高了实际后门攻击的可行性。我们的代码和数据将在此 https URL 上提供。

Title: Training Language Models to Win Debates with Self-Play Improves Judge Accuracy

• Authors: Samuel Arnesen, David Rein, Julian Michael
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16636
• Pdf URL: https://arxiv.org/pdf/2409.16636
• Copy Paste: [[2409.16636]] Training Language Models to Win Debates with Self-Play Improves Judge Accuracy(https://arxiv.org/abs/2409.16636)
• Keywords: language model
• Abstract: We test the robustness of debate as a method of scalable oversight by training models to debate with data generated via self-play. In a long-context reading comprehension task, we find that language model based evaluators answer questions more accurately when judging models optimized to win debates. By contrast, we find no such relationship for consultancy models trained to persuade a judge without an opposing debater present. In quantitative and qualitative comparisons between our debate models and novel consultancy baselines, we find evidence that debate training encourages stronger and more informative arguments, showing promise that it can help provide high-quality supervision for tasks that are difficult to directly evaluate.
• 摘要：我们通过训练模型使用通过自我对弈生成的数据进行辩论，测试了辩论作为一种可扩展监督方法的稳健性。在一项长语境阅读理解任务中，我们发现基于语言模型的评估者在评判为赢得辩论而优化的模型时，回答问题更准确。相比之下，我们发现，在没有对方辩手在场的情况下，经过训练以说服法官的咨询模型不存在这种关系。在我们的辩论模型和新颖的咨询基线的定量和定性比较中，我们发现有证据表明，辩论训练鼓励更强大、更具信息量的论据，表明它有望为难以直接评估的任务提供高质量的监督。

Title: Pre-trained Language Models Return Distinguishable Probability Distributions to Unfaithfully Hallucinated Texts

• Authors: Taehun Cha, Donghun Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16658
• Pdf URL: https://arxiv.org/pdf/2409.16658
• Copy Paste: [[2409.16658]] Pre-trained Language Models Return Distinguishable Probability Distributions to Unfaithfully Hallucinated Texts(https://arxiv.org/abs/2409.16658)
• Keywords: language model, hallucination
• Abstract: In this work, we show the pre-trained language models return distinguishable generation probability and uncertainty distribution to unfaithfully hallucinated texts, regardless of their size and structure. By examining 24 models on 6 data sets, we find out that 88-98% of cases return statistically significantly distinguishable generation probability and uncertainty distributions. Using this general phenomenon, we showcase a hallucination-reducing training algorithm. Our algorithm outperforms other baselines by achieving higher faithfulness metrics while maintaining sound general text quality measures.
• 摘要：在这项研究中，我们展示了预训练语言模型对不忠实的幻觉文本返回可区分的生成概率和不确定性分布，无论其大小和结构如何。通过检查 6 个数据集上的 24 个模型，我们发现 88-98% 的案例返回统计上明显可区分的生成概率和不确定性分布。利用这一普遍现象，我们展示了一种减少幻觉的训练算法。我们的算法通过实现更高的忠实度指标同时保持良好的一般文本质量指标，优于其他基线。

Title: A Character-Centric Creative Story Generation via Imagination

• Authors: Kyeongman Park, Minbeom Kim, Kyomin Jung
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16667
• Pdf URL: https://arxiv.org/pdf/2409.16667
• Copy Paste: [[2409.16667]] A Character-Centric Creative Story Generation via Imagination(https://arxiv.org/abs/2409.16667)
• Keywords: language model, llm
• Abstract: Creative story generation with diverse and detailed story elements is a long-standing goal for large language models. While existing methodologies generate long and coherent stories, they fall significantly short of human capabilities in terms of diversity and character detail. To address this, we introduce a novel story generation framework called CCI (Character-centric Creative story generation via Imagination). CCI features two innovative modules for creative story generation: IG (Image-Guided Imagination) and MW (Multi-Writer model). In the IG module, we utilize DALL-E 3 to create visual representations of key story elements. The IG generates more novel and concrete characters, backgrounds, and main plots than text-only methods. The MW module uses these story elements created by IG to generate multiple description candidates for the protagonist and select the best one. This method incorporates vivid and rich character descriptions into the story. We compared the stories generated by CCI and baseline models through human evaluation and statistical analysis. The results showed significant improvements in the creativity. Furthermore, by enabling interactive multi-modal story generation with users, we have opened up possibilities for human-LLM integration in cultural development.
• 摘要：具有多样化和详细故事元素的创意故事生成是大型语言模型的长期目标。虽然现有的方法可以生成长篇连贯的故事，但它们在多样性和人物细节方面远远落后于人类的能力。为了解决这个问题，我们引入了一个新颖的故事生成框架，称为 CCI（通过想象力以人物为中心进行创意故事生成）。CCI 具有两个用于创意故事生成的创新模块：IG（图像引导想象力）和 MW（多作家模型）。在 IG 模块中，我们利用 DALL-E 3 创建关键故事元素的视觉表示。与纯文本方法相比，IG 可以生成更多新颖和具体的人物、背景和主要情节。MW 模块使用 IG 创建的这些故事元素为主角生成多个描述候选，并选择最佳的候选。这种方法将生动丰富的人物描述融入故事中。我们通过人工评估和统计分析比较了 CCI 和基线模型生成的故事。结果显示创造力显着提高。此外，通过与用户实现交互式多模式故事生成，我们为人类与法学硕士在文化发展中的融合开辟了可能性。

Title: SynTQA: Synergistic Table-based Question Answering via Mixture of Text-to-SQL and E2E TQA

• Authors: Siyue Zhang, Anh Tuan Luu, Chen Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16682
• Pdf URL: https://arxiv.org/pdf/2409.16682
• Copy Paste: [[2409.16682]] SynTQA: Synergistic Table-based Question Answering via Mixture of Text-to-SQL and E2E TQA(https://arxiv.org/abs/2409.16682)
• Keywords: llm
• Abstract: Text-to-SQL parsing and end-to-end question answering (E2E TQA) are two main approaches for Table-based Question Answering task. Despite success on multiple benchmarks, they have yet to be compared and their synergy remains unexplored. In this paper, we identify different strengths and weaknesses through evaluating state-of-the-art models on benchmark datasets: Text-to-SQL demonstrates superiority in handling questions involving arithmetic operations and long tables; E2E TQA excels in addressing ambiguous questions, non-standard table schema, and complex table contents. To combine both strengths, we propose a Synergistic Table-based Question Answering approach that integrate different models via answer selection, which is agnostic to any model types. Further experiments validate that ensembling models by either feature-based or LLM-based answer selector significantly improves the performance over individual models.
• 摘要：文本到 SQL 解析和端到端问答 (E2E TQA) 是基于表格的问答任务的两种主要方法。尽管在多个基准测试中取得了成功，但它们尚未进行比较，并且它们的协同作用仍未得到探索。在本文中，我们通过在基准数据集上评估最先进的模型来确定不同的优势和劣势：文本到 SQL 在处理涉及算术运算和长表的问题方面表现出色；E2E TQA 擅长解决模棱两可的问题、非标准表格模式和复杂的表格内容。为了结合这两种优势，我们提出了一种协同的基于表格的问答方法，该方法通过答案选择集成不同的模型，这与任何模型类型无关。进一步的实验验证了通过基于特征或基于 LLM 的答案选择器集成模型可以显着提高单个模型的性能。

Title: PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning

• Authors: Qibin Wang, Xiaolin Hu, Weikai Xu, Wei Liu, Jian Luan, Bin Wang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16722
• Pdf URL: https://arxiv.org/pdf/2409.16722
• Copy Paste: [[2409.16722]] PMSS: Pretrained Matrices Skeleton Selection for LLM Fine-tuning(https://arxiv.org/abs/2409.16722)
• Keywords: llm
• Abstract: Low-rank adaptation (LoRA) and its variants have recently gained much interest due to their ability to avoid excessive inference costs. However, LoRA still encounters the following challenges: (1) Limitation of low-rank assumption; and (2) Its initialization method may be suboptimal. To this end, we propose PMSS(Pre-trained Matrices Skeleton Selection), which enables high-rank updates with low costs while leveraging semantic and linguistic information inherent in pre-trained weight. It achieves this by selecting skeletons from the pre-trained weight matrix and only learning a small matrix instead. Experiments demonstrate that PMSS outperforms LoRA and other fine-tuning methods across tasks with much less trainable parameters. We demonstrate its effectiveness, especially in handling complex tasks such as DROP benchmark(+3.4%/+5.9% on LLaMA2-7B/13B) and math reasoning(+12.89%/+5.61%/+3.11% on LLaMA2-7B, Mistral-7B and Gemma-7B of GSM8K). The code and model will be released soon.
• 摘要：低秩自适应 (LoRA) 及其变体最近引起了广泛关注，因为它们能够避免过多的推理成本。然而，LoRA 仍然面临以下挑战：(1) 低秩假设的局限性；(2) 其初始化方法可能不是最优的。为此，我们提出了 PMSS（预训练矩阵骨架选择），它能够以低成本实现高秩更新，同时利用预训练权重中固有的语义和语言信息。它通过从预训练权重矩阵中选择骨架并仅学习一个小矩阵来实现这一点。实验表明，PMSS 在可训练参数少得多的任务中的表现优于 LoRA 和其他微调方法。我们展示了它的有效性，特别是在处理复杂任务时，例如 DROP 基准测试（LLaMA2-7B/13B 上 +3.4%/+5.9%）和数学推理（GSM8K 的 LLaMA2-7B、Mistral-7B 和 Gemma-7B 上 +12.89%/+5.61%/+3.11%）。代码和模型将很快发布。

Title: RoleBreak: Character Hallucination as a Jailbreak Attack in Role-Playing Systems

• Authors: Yihong Tang, Bo Wang, Xu Wang, Dongming Zhao, Jing Liu, Jijun Zhang, Ruifang He, Yuexian Hou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16727
• Pdf URL: https://arxiv.org/pdf/2409.16727
• Copy Paste: [[2409.16727]] RoleBreak: Character Hallucination as a Jailbreak Attack in Role-Playing Systems(https://arxiv.org/abs/2409.16727)
• Keywords: language model, llm, hallucination
• Abstract: Role-playing systems powered by large language models (LLMs) have become increasingly influential in emotional communication applications. However, these systems are susceptible to character hallucinations, where the model deviates from predefined character roles and generates responses that are inconsistent with the intended persona. This paper presents the first systematic analysis of character hallucination from an attack perspective, introducing the RoleBreak framework. Our framework identifies two core mechanisms-query sparsity and role-query conflict-as key factors driving character hallucination. Leveraging these insights, we construct a novel dataset, RoleBreakEval, to evaluate existing hallucination mitigation techniques. Our experiments reveal that even enhanced models trained to minimize hallucination remain vulnerable to attacks. To address these vulnerabilities, we propose a novel defence strategy, the Narrator Mode, which generates supplemental context through narration to mitigate role-query conflicts and improve query generalization. Experimental results demonstrate that Narrator Mode significantly outperforms traditional refusal-based strategies by reducing hallucinations, enhancing fidelity to character roles and queries, and improving overall narrative coherence.
• 摘要：以大型语言模型 (LLM) 为驱动力的角色扮演系统在情感交流应用中的影响力越来越大。然而，这些系统容易受到角色幻觉的影响，即模型偏离预定义的角色角色并生成与预期角色不一致的响应。本文首次从攻击角度对角色幻觉进行了系统分析，介绍了 RoleBreak 框架。我们的框架确定了两种核心机制——查询稀疏性和角色查询冲突——是驱动角色幻觉的关键因素。利用这些见解，我们构建了一个新数据集 RoleBreakEval，以评估现有的幻觉缓解技术。我们的实验表明，即使是经过训练以最大限度地减少幻觉的增强模型仍然容易受到攻击。为了解决这些弱点，我们提出了一种新颖的防御策略，即叙述者模式，它通过叙述生成补充背景以缓解角色查询冲突并提高查询泛化能力。实验结果表明，叙述者模式通过减少幻觉、增强对角色和查询的保真度以及提高整体叙述连贯性，明显优于传统的基于拒绝的策略。

Title: E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQL

• Authors: Hasan Alp Caferoğlu, Özgür Ulusoy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16751
• Pdf URL: https://arxiv.org/pdf/2409.16751
• Copy Paste: [[2409.16751]] E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQL(https://arxiv.org/abs/2409.16751)
• Keywords: language model, llm
• Abstract: Translating Natural Language Queries into Structured Query Language (Text-to-SQL or NLQ-to-SQL) is a critical task extensively studied by both the natural language processing and database communities, aimed at providing a natural language interface to databases (NLIDB) and lowering the barrier for non-experts. Despite recent advancements made through the use of Large Language Models (LLMs), significant challenges remain. These include handling complex database schemas, resolving ambiguity in user queries, and generating SQL queries with intricate structures that accurately reflect the user's intent. In this work, we introduce E-SQL, a novel pipeline specifically designed to address these challenges through direct schema linking and candidate predicate augmentation. E-SQL enhances the natural language query by incorporating relevant database items (i.e., tables, columns, and values) and conditions directly into the question, bridging the gap between the query and the database structure. The pipeline leverages candidate predicate augmentation to mitigate erroneous or incomplete predicates in generated SQLs. We further investigate the impact of schema filtering, a technique widely explored in previous work, and demonstrate its diminishing returns when applied alongside advanced large language models. Comprehensive evaluations on the BIRD benchmark illustrate that E-SQL achieves competitive performance, particularly excelling in complex queries with a 66.29% execution accuracy on the test set. All code required to reproduce the reported results is publicly available on our GitHub repository.
• 摘要：将自然语言查询转换为结构化查询语言（文本到 SQL 或 NLQ 到 SQL）是一项关键任务，自然语言处理和数据库社区都对此进行了广泛研究，旨在为数据库提供自然语言接口 (NLIDB) 并降低非专家的门槛。尽管最近通过使用大型语言模型 (LLM) 取得了进展，但仍然存在重大挑战。这些挑战包括处理复杂的数据库模式、解决用户查询中的歧义以及生成具有复杂结构的 SQL 查询以准确反映用户的意图。在这项工作中，我们引入了 E-SQL，这是一种新颖的管道，专门设计用于通过直接模式链接和候选谓词增强来解决这些挑战。E-SQL 通过将相关数据库项（即表、列和值）和条件直接合并到问题中来增强自然语言查询，从而弥合查询和数据库结构之间的差距。管道利用候选谓词增强来缓解生成的 SQL 中的错误或不完整谓词。我们进一步研究了模式过滤（一种在以前的工作中广泛探索的技术）的影响，并展示了其与高级大型语言模型一起应用时收益递减的情况。对 BIRD 基准的全面评估表明，E-SQL 实现了具有竞争力的性能，尤其是在复杂查询中表现出色，在测试集上的执行准确率为 66.29%。重现报告结果所需的所有代码均可在我们的 GitHub 存储库中公开获取。

Title: Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction

• Authors: Jinchuan Zhang, Yan Zhou, Yaxin Liu, Ziming Li, Songlin Hu
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.16783
• Pdf URL: https://arxiv.org/pdf/2409.16783
• Copy Paste: [[2409.16783]] Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction(https://arxiv.org/abs/2409.16783)
• Keywords: language model, llm
• Abstract: Automated red teaming is an effective method for identifying misaligned behaviors in large language models (LLMs). Existing approaches, however, often focus primarily on improving attack success rates while overlooking the need for comprehensive test case coverage. Additionally, most of these methods are limited to single-turn red teaming, failing to capture the multi-turn dynamics of real-world human-machine interactions. To overcome these limitations, we propose HARM (Holistic Automated Red teaMing), which scales up the diversity of test cases using a top-down approach based on an extensible, fine-grained risk taxonomy. Our method also leverages a novel fine-tuning strategy and reinforcement learning techniques to facilitate multi-turn adversarial probing in a human-like manner. Experimental results demonstrate that our framework enables a more systematic understanding of model vulnerabilities and offers more targeted guidance for the alignment process.
• 摘要：自动红队演练是识别大型语言模型 (LLM) 中错位行为的有效方法。然而，现有方法通常主要侧重于提高攻击成功率，而忽略了全面测试用例覆盖的必要性。此外，大多数这些方法仅限于单轮红队演练，无法捕捉现实世界人机交互的多轮动态。为了克服这些限制，我们提出了 HARM（整体自动红队演练），它使用基于可扩展、细粒度风险分类法的自上而下方法扩大测试用例的多样性。我们的方法还利用了一种新颖的微调策略和强化学习技术，以类似人类的方式促进多轮对抗性探测。实验结果表明，我们的框架能够更系统地理解模型漏洞，并为对齐过程提供更有针对性的指导。

Title: Mitigating the Bias of Large Language Model Evaluation

• Authors: Hongli Zhou, Hui Huang, Yunfei Long, Bing Xu, Conghui Zhu, Hailong Cao, Muyun Yang, Tiejun Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16788
• Pdf URL: https://arxiv.org/pdf/2409.16788
• Copy Paste: [[2409.16788]] Mitigating the Bias of Large Language Model Evaluation(https://arxiv.org/abs/2409.16788)
• Keywords: language model, llm, prompt
• Abstract: Recently, there has been a trend of evaluating the Large Language Model (LLM) quality in the flavor of LLM-as-a-Judge, namely leveraging another LLM to evaluate the current output quality. However, existing judges are proven to be biased, namely they would favor answers which present better superficial quality (such as verbosity, fluency) while ignoring the instruction following ability. In this work, we propose systematic research about the bias of LLM-as-a-Judge. Specifically, for closed-source judge models, we apply calibration to mitigate the significance of superficial quality, both on probability level and prompt level. For open-source judge models, we propose to mitigate the bias by contrastive training, with curated negative samples that deviate from instruction but present better superficial quality. We apply our methods on the bias evaluation benchmark, and experiment results show our methods mitigate the bias by a large margin while maintaining a satisfactory evaluation accuracy.
• 摘要：最近，出现了一种以 LLM-as-a-Judge 的形式评估大型语言模型 (LLM) 质量的趋势，即利用另一个 LLM 来评估当前的输出质量。然而，现有的评判者被证明是有偏见的，即他们会青睐那些呈现出更好的表面质量（如冗长性、流畅性）的答案，而忽略了指令遵循能力。在本文中，我们提出了关于 LLM-as-a-Judge 偏见的系统研究。具体而言，对于闭源评判模型，我们应用校准来减轻表面质量的显著性，包括概率级别和提示级别。对于开源评判模型，我们建议通过对比训练来减轻偏见，使用偏离指令但呈现出更好表面质量的精选负样本。我们将我们的方法应用于偏见评估基准，实验结果表明我们的方法在保持令人满意的评估准确率的同时大大减轻了偏见。

Title: A Few Hypocrites: Few-Shot Learning and Subtype Definitions for Detecting Hypocrisy Accusations in Online Climate Change Debates

• Authors: Paulina Garcia Corral, Avishai Green, Hendrik Meyer, Anke Stoll, Xiaoyue Yan, Myrthe Reuver
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16807
• Pdf URL: https://arxiv.org/pdf/2409.16807
• Copy Paste: [[2409.16807]] A Few Hypocrites: Few-Shot Learning and Subtype Definitions for Detecting Hypocrisy Accusations in Online Climate Change Debates(https://arxiv.org/abs/2409.16807)
• Keywords: language model, gpt, llm
• Abstract: The climate crisis is a salient issue in online discussions, and hypocrisy accusations are a central rhetorical element in these debates. However, for large-scale text analysis, hypocrisy accusation detection is an understudied tool, most often defined as a smaller subtask of fallacious argument detection. In this paper, we define hypocrisy accusation detection as an independent task in NLP, and identify different relevant subtypes of hypocrisy accusations. Our Climate Hypocrisy Accusation Corpus (CHAC) consists of 420 Reddit climate debate comments, expert-annotated into two different types of hypocrisy accusations: personal versus political hypocrisy. We evaluate few-shot in-context learning with 6 shots and 3 instruction-tuned Large Language Models (LLMs) for detecting hypocrisy accusations in this dataset. Results indicate that the GPT-4o and Llama-3 models in particular show promise in detecting hypocrisy accusations (F1 reaching 0.68, while previous work shows F1 of 0.44). However, context matters for a complex semantic concept such as hypocrisy accusations, and we find models struggle especially at identifying political hypocrisy accusations compared to personal moral hypocrisy. Our study contributes new insights in hypocrisy detection and climate change discourse, and is a stepping stone for large-scale analysis of hypocrisy accusation in online climate debates.
• 摘要：气候危机是网络讨论中的一个突出问题，而虚伪指控是这些辩论中的一个核心修辞元素。然而，对于大规模文本分析，虚伪指控检测是一种研究不足的工具，通常被定义为谬误论证检测的一个较小的子任务。在本文中，我们将虚伪指控检测定义为 NLP 中的一项独立任务，并确定虚伪指控的不同相关子类型。我们的气候虚伪指控语料库 (CHAC) 包含 420 条 Reddit 气候辩论评论，专家将其注释为两种不同类型的虚伪指控：个人虚伪和政治虚伪。我们评估了使用 6 个样本和 3 个指令调整的大型语言模型 (LLM) 的少样本上下文学习来检测此数据集中的虚伪指控。结果表明，GPT-4o 和 Llama-3 模型在检测虚伪指控方面尤其有前景（F1 达到 0.68，而之前的研究显示 F1 为 0.44）。然而，对于虚伪指控等复杂的语义概念，上下文很重要，我们发现模型在识别政治虚伪指控方面尤其困难，而个人道德虚伪则不然。我们的研究为虚伪检测和气候变化话语提供了新的见解，并为大规模分析在线气候辩论中的虚伪指控奠定了基础。

Title: CodeInsight: A Curated Dataset of Practical Coding Solutions from Stack Overflow

• Authors: Nathanaël Beau, Benoît Crabbé
• Subjects: cs.CL, cs.SE
• Abstract URL: https://arxiv.org/abs/2409.16819
• Pdf URL: https://arxiv.org/pdf/2409.16819
• Copy Paste: [[2409.16819]] CodeInsight: A Curated Dataset of Practical Coding Solutions from Stack Overflow(https://arxiv.org/abs/2409.16819)
• Keywords: gpt
• Abstract: We introduce a novel dataset tailored for code generation, aimed at aiding developers in common tasks. Our dataset provides examples that include a clarified intent, code snippets associated, and an average of three related unit tests. It encompasses a range of libraries such as \texttt{Pandas}, \texttt{Numpy}, and \texttt{Regex}, along with more than 70 standard libraries in Python code derived from Stack Overflow. Comprising 3,409 crafted examples by Python experts, our dataset is designed for both model finetuning and standalone evaluation. To complete unit tests evaluation, we categorize examples in order to get more fine grained analysis, enhancing the understanding of models' strengths and weaknesses in specific coding tasks. The examples have been refined to reduce data contamination, a process confirmed by the performance of three leading models: Mistral 7B, CodeLLaMa 13B, and Starcoder 15B. We further investigate data-contamination testing GPT-4 performance on a part of our dataset. The benchmark can be accessed at \url{this https URL}.
• 摘要：我们引入了一个专为代码生成量身定制的新数据集，旨在帮助开发人员完成常见任务。我们的数据集提供的示例包括明确的意图、相关的代码片段和平均三个相关的单元测试。它包含一系列库，例如 \texttt{Pandas}、\texttt{Numpy} 和 \texttt{Regex}，以及来自 Stack Overflow 的 70 多个 Python 代码标准库。我们的数据集包含由 Python 专家制作的 3,409 个示例，专为模型微调和独立评估而设计。为了完成单元测试评估，我们对示例进行了分类，以便获得更细粒度的分析，增强对模型在特定编码任务中的优势和劣势的理解。这些示例已经过改进以减少数据污染，这一过程已由三个领先模型的性能证实：Mistral 7B、CodeLLaMa 13B 和 Starcoder 15B。我们进一步调查了数据污染测试 GPT-4 在我们数据集的一部分上的性能。您可以通过 \url{此 https URL} 访问该基准。

Title: Shifting from endangerment to rebirth in the Artificial Intelligence Age: An Ensemble Machine Learning Approach for Hawrami Text Classification

• Authors: Aram Khaksar, Hossein Hassani
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16884
• Pdf URL: https://arxiv.org/pdf/2409.16884
• Copy Paste: [[2409.16884]] Shifting from endangerment to rebirth in the Artificial Intelligence Age: An Ensemble Machine Learning Approach for Hawrami Text Classification(https://arxiv.org/abs/2409.16884)
• Keywords: language model
• Abstract: Hawrami, a dialect of Kurdish, is classified as an endangered language as it suffers from the scarcity of data and the gradual loss of its speakers. Natural Language Processing projects can be used to partially compensate for data availability for endangered languages/dialects through a variety of approaches, such as machine translation, language model building, and corpora development. Similarly, NLP projects such as text classification are in language documentation. Several text classification studies have been conducted for Kurdish, but they were mainly dedicated to two particular dialects: Sorani (Central Kurdish) and Kurmanji (Northern Kurdish). In this paper, we introduce various text classification models using a dataset of 6,854 articles in Hawrami labeled into 15 categories by two native speakers. We use K-nearest Neighbor (KNN), Linear Support Vector Machine (Linear SVM), Logistic Regression (LR), and Decision Tree (DT) to evaluate how well those methods perform the classification task. The results indicate that the Linear SVM achieves a 96% of accuracy and outperforms the other approaches.
• 摘要：库尔德语的一种方言 Hawrami 被列为濒危语言，因为它面临数据稀缺和使用者逐渐消失的问题。自然语言处理项目可通过多种方法（如机器翻译、语言模型构建和语料库开发）部分弥补濒危语言/方言的数据可用性。同样，文本分类等 NLP 项目也属于语言文档。已经对库尔德语进行了多项文本分类研究，但主要针对两种特定方言：索拉尼语（中库尔德语）和库尔曼吉语（北库尔德语）。在本文中，我们使用由两位母语人士分为 15 个类别的 6,854 篇 Hawrami 文章数据集介绍了各种文本分类模型。我们使用 K 最近邻 (KNN)、线性支持向量机 (Linear SVM)、逻辑回归 (LR) 和决策树 (DT) 来评估这些方法在分类任务中的表现。结果表明，线性 SVM 的准确率达到了 96%，优于其他方法。

Title: Enhancing Temporal Sensitivity and Reasoning for Time-Sensitive Question Answering

• Authors: Wanqi Yang, Yanda Li, Meng Fang, Ling Chen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16909
• Pdf URL: https://arxiv.org/pdf/2409.16909
• Copy Paste: [[2409.16909]] Enhancing Temporal Sensitivity and Reasoning for Time-Sensitive Question Answering(https://arxiv.org/abs/2409.16909)
• Keywords: language model, llm
• Abstract: Time-Sensitive Question Answering (TSQA) demands the effective utilization of specific temporal contexts, encompassing multiple time-evolving facts, to address time-sensitive questions. This necessitates not only the parsing of temporal information within questions but also the identification and understanding of time-evolving facts to generate accurate answers. However, current large language models still have limited sensitivity to temporal information and their inadequate temporal reasoning this http URL this paper, we propose a novel framework that enhances temporal awareness and reasoning through Temporal Information-Aware Embedding and Granular Contrastive Reinforcement Learning. Experimental results on four TSQA datasets demonstrate that our framework significantly outperforms existing LLMs in TSQA tasks, marking a step forward in bridging the performance gap between machine and human temporal understanding and reasoning.
• 摘要：时间敏感型问答系统 (TSQA) 要求有效利用特定的时间上下文（包含多个随时间变化的事实）来解决时间敏感型问题。这不仅需要解析问题中的时间信息，还需要识别和理解随时间变化的事实以生成准确的答案。然而，当前的大型语言模型对时间信息的敏感度仍然有限​​，并且其时间推理能力不足。本文提出了一种新颖的框架，通过时间信息感知嵌入和粒度对比强化学习来增强时间意识和推理能力。在四个 TSQA 数据集上的实验结果表明，我们的框架在 TSQA 任务中的表现明显优于现有的 LLM，标志着在缩小机器和人类时间理解和推理之间的性能差距方面迈出了一步。

Title: Pruning Multilingual Large Language Models for Multilingual Inference

• Authors: Hwichan Kim, Jun Suzuki, Tosho Hirasawa, Mamoru Komachi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16911
• Pdf URL: https://arxiv.org/pdf/2409.16911
• Copy Paste: [[2409.16911]] Pruning Multilingual Large Language Models for Multilingual Inference(https://arxiv.org/abs/2409.16911)
• Keywords: language model, llm
• Abstract: Multilingual large language models (MLLMs), trained on multilingual balanced data, demonstrate better zero-shot learning performance in non-English languages compared to large language models trained on English-dominant data. However, the disparity in performance between English and non-English languages remains a challenge yet to be fully addressed. A distinctive characteristic of MLLMs is their high-quality translation capabilities, indicating an acquired proficiency in aligning between languages. This study explores how to enhance the zero-shot performance of MLLMs in non-English languages by leveraging their alignment capability between English and non-English languages. To achieve this, we first analyze the behavior of MLLMs when performing translation and reveal that there are large magnitude features that play a critical role in the translation process. Inspired by these findings, we retain the weights associated with operations involving the large magnitude features and prune other weights to force MLLMs to rely on these features for tasks beyond translation. We empirically demonstrate that this pruning strategy can enhance the MLLMs' performance in non-English language.
• 摘要：在多语言平衡数据上训练的多语言大型语言模型 (MLLM) 在非英语语言中表现出比在以英语为主的数据上训练的大型语言模型更好的零样本学习性能。然而，英语和非英语语言之间的性能差异仍然是一个尚未完全解决的挑战。MLLM 的一个显着特点是它们具有高质量的翻译能力，表明在语言之间对齐方面具有熟练的技能。本研究探讨了如何利用 MLLM 在英语和非英语语言之间的对齐能力来提高其在非英语语言中的零样本性能。为此，我们首先分析了 MLLM 在执行翻译时的行为，并揭示了在翻译过程中起关键作用的大量特征。受这些发现的启发，我们保留了与涉及大量特征的操作相关的权重，并修剪了其他权重，以迫使 MLLM 依赖这些特征来完成翻译以外的任务。我们通过经验证明，这种修剪策略可以提高 MLLM 在非英语语言中的表现。

Title: Zero-Shot Detection of LLM-Generated Text using Token Cohesiveness

• Authors: Shixuan Ma, Quan Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.16914
• Pdf URL: https://arxiv.org/pdf/2409.16914
• Copy Paste: [[2409.16914]] Zero-Shot Detection of LLM-Generated Text using Token Cohesiveness(https://arxiv.org/abs/2409.16914)
• Keywords: language model, llm
• Abstract: The increasing capability and widespread usage of large language models (LLMs) highlight the desirability of automatic detection of LLM-generated text. Zero-shot detectors, due to their training-free nature, have received considerable attention and notable success. In this paper, we identify a new feature, token cohesiveness, that is useful for zero-shot detection, and we demonstrate that LLM-generated text tends to exhibit higher token cohesiveness than human-written text. Based on this observation, we devise TOCSIN, a generic dual-channel detection paradigm that uses token cohesiveness as a plug-and-play module to improve existing zero-shot detectors. To calculate token cohesiveness, TOCSIN only requires a few rounds of random token deletion and semantic difference measurement, making it particularly suitable for a practical black-box setting where the source model used for generation is not accessible. Extensive experiments with four state-of-the-art base detectors on various datasets, source models, and evaluation settings demonstrate the effectiveness and generality of the proposed approach. Code available at: \url{this https URL}.
• 摘要：大型语言模型 (LLM) 的功能不断增强，使用范围越来越广，这凸显了自动检测 LLM 生成文本的必要性。零样本检测器由于其无需训练的特性，受到了广泛关注并取得了显著的成功。在本文中，我们确定了一个对零样本检测有用的新功能，即标记内聚性，并证明了 LLM 生成的文本往往比人工书写的文本表现出更高的标记内聚性。基于这一观察，我们设计了 TOCSIN，这是一种通用的双通道检测范例，它使用标记内聚性作为即插即用模块来改进现有的零样本检测器。为了计算标记内聚性，TOCSIN 只需要几轮随机标记删除和语义差异测量，这使其特别适合于无法访问用于生成的源模型的实际黑盒设置。在各种数据集、源模型和评估设置上使用四个最先进的基础检测器进行的大量实验证明了所提出方法的有效性和通用性。代码位于：\url{此 https URL}。

Title: Investigating OCR-Sensitive Neurons to Improve Entity Recognition in Historical Documents

• Authors: Emanuela Boros, Maud Ehrmann
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16934
• Pdf URL: https://arxiv.org/pdf/2409.16934
• Copy Paste: [[2409.16934]] Investigating OCR-Sensitive Neurons to Improve Entity Recognition in Historical Documents(https://arxiv.org/abs/2409.16934)
• Keywords: language model
• Abstract: This paper investigates the presence of OCR-sensitive neurons within the Transformer architecture and their influence on named entity recognition (NER) performance on historical documents. By analysing neuron activation patterns in response to clean and noisy text inputs, we identify and then neutralise OCR-sensitive neurons to improve model performance. Based on two open access large language models (Llama2 and Mistral), experiments demonstrate the existence of OCR-sensitive regions and show improvements in NER performance on historical newspapers and classical commentaries, highlighting the potential of targeted neuron modulation to improve models' performance on noisy text.
• 摘要：本文研究了 Transformer 架构中 OCR 敏感神经元的存在及其对历史文献命名实体识别 (NER) 性能的影响。通过分析响应干净和嘈杂文本输入的神经元激活模式，我们识别并中和 OCR 敏感神经元以提高模型性能。基于两个开放获取大型语言模型 (Llama2 和 Mistral)，实验证明了 OCR 敏感区域的存在，并表明历史报纸和古典评论的 NER 性能有所提高，凸显了有针对性的神经元调节在提高模型对嘈杂文本的性能方面的潜力。

Title: Adaptive Self-Supervised Learning Strategies for Dynamic On-Device LLM Personalization

• Authors: Rafael Mendoza, Isabella Cruz, Richard Liu, Aarav Deshmukh, David Williams, Jesscia Peng, Rohan Iyer
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.16973
• Pdf URL: https://arxiv.org/pdf/2409.16973
• Copy Paste: [[2409.16973]] Adaptive Self-Supervised Learning Strategies for Dynamic On-Device LLM Personalization(https://arxiv.org/abs/2409.16973)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have revolutionized how we interact with technology, but their personalization to individual user preferences remains a significant challenge, particularly in on-device applications. Traditional methods often depend heavily on labeled datasets and can be resource-intensive. To address these issues, we present Adaptive Self-Supervised Learning Strategies (ASLS), which utilizes self-supervised learning techniques to personalize LLMs dynamically. The framework comprises a user profiling layer for collecting interaction data and a neural adaptation layer for real-time model fine-tuning. This innovative approach enables continuous learning from user feedback, allowing the model to generate responses that align closely with user-specific contexts. The adaptive mechanisms of ASLS minimize computational demands and enhance personalization efficiency. Experimental results across various user scenarios illustrate the superior performance of ASLS in boosting user engagement and satisfaction, highlighting its potential to redefine LLMs as highly responsive and context-aware systems on-device.
• 摘要：大型语言模型 (LLM) 彻底改变了我们与技术交互的方式，但根据个人用户偏好进行个性化仍然是一项重大挑战，尤其是在设备端应用中。传统方法通常严重依赖标记数据集，并且可能耗费大量资源。为了解决这些问题，我们提出了自适应自监督学习策略 (ASLS)，它利用自监督学习技术动态个性化 LLM。该框架包括用于收集交互数据的用户分析层和用于实时模型微调的神经自适应层。这种创新方法能够从用户反馈中不断学习，使模型能够生成与用户特定上下文紧密结合的响应。ASLS 的自适应机制最大限度地减少了计算需求并提高了个性化效率。在各种用户场景中的实验结果表明 ASLS 在提高用户参与度和满意度方面的卓越性能，凸显了其将 LLM 重新定义为高度响应和上下文感知的设备端系统的潜力。

Title: Decoding Large-Language Models: A Systematic Overview of Socio-Technical Impacts, Constraints, and Emerging Questions

• Authors: Zeyneb N. Kaya, Souvick Ghosh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.16974
• Pdf URL: https://arxiv.org/pdf/2409.16974
• Copy Paste: [[2409.16974]] Decoding Large-Language Models: A Systematic Overview of Socio-Technical Impacts, Constraints, and Emerging Questions(https://arxiv.org/abs/2409.16974)
• Keywords: language model, llm
• Abstract: There have been rapid advancements in the capabilities of large language models (LLMs) in recent years, greatly revolutionizing the field of natural language processing (NLP) and artificial intelligence (AI) to understand and interact with human language. Therefore, in this work, we conduct a systematic investigation of the literature to identify the prominent themes and directions of LLM developments, impacts, and limitations. Our findings illustrate the aims, methodologies, limitations, and future directions of LLM research. It includes responsible development considerations, algorithmic improvements, ethical challenges, and societal implications of LLM development. Overall, this paper provides a rigorous and comprehensive overview of current research in LLM and identifies potential directions for future development. The article highlights the application areas that could have a positive impact on society along with the ethical considerations.
• 摘要：近年来，大型语言模型 (LLM) 的功能得到了快速发展，极大地改变了自然语言处理 (NLP) 和人工智能 (AI) 领域，使其能够理解和与人类语言交互。因此，在本文中，我们对文献进行了系统的调查，以确定 LLM 发展、影响和局限性的突出主题和方向。我们的研究结果说明了 LLM 研究的目的、方法、局限性和未来方向。它包括负责任的发展考虑、算法改进、道德挑战和 LLM 发展的社会影响。总体而言，本文对 LLM 的当前研究进行了严格而全面的概述，并确定了未来发展的潜在方向。本文强调了可能对社会产生积极影响的应用领域以及道德考虑。

Title: LLM-CARD: Towards a Description and Landscape of Large Language Models

• Authors: Shengwei Tian, Lifeng Han, Erick Mendez Guzman, Goran Nenadic
• Subjects: cs.CL, cs.DL
• Abstract URL: https://arxiv.org/abs/2409.17011
• Pdf URL: https://arxiv.org/pdf/2409.17011
• Copy Paste: [[2409.17011]] LLM-CARD: Towards a Description and Landscape of Large Language Models(https://arxiv.org/abs/2409.17011)
• Keywords: language model, llm
• Abstract: With the rapid growth of the Natural Language Processing (NLP) field, a vast variety of Large Language Models (LLMs) continue to emerge for diverse NLP tasks. As an increasing number of papers are presented, researchers and developers face the challenge of information overload. Thus, it is particularly important to develop a system that can automatically extract and organise key information about LLMs from academic papers (\textbf{LLM model card}). This work is to develop such a pioneer system by using Named Entity Recognition (\textbf{NER}) and Relation Extraction (\textbf{RE}) methods that automatically extract key information about large language models from the papers, helping researchers to efficiently access information about LLMs. These features include model \textit{licence}, model \textit{name}, and model \textit{application}. With these features, we can form a model card for each paper. \textbf{Data-contribution} wise, 106 academic papers were processed by defining three dictionaries - LLMs name, licence, and application. 11,051 sentences were extracted through dictionary lookup, and the dataset was constructed through manual review of the final selection of 129 sentences that have a link between the name and the licence, and 106 sentences that have a link between the model name and the application.
• 摘要：随着自然语言处理 (NLP) 领域的快速发展，各种各样的大型语言模型 (LLM) 不断涌现，用于各种 NLP 任务。随着论文数量的增加，研究人员和开发人员面临着信息过载的挑战。因此，开发一个能够自动从学术论文中提取和组织有关 LLM 的关键信息的系统 (\textbf{LLM 模型卡}) 尤为重要。这项工作是通过使用命名实体识别 (\textbf{NER}) 和关系提取 (\textbf{RE}) 方法开发这样一个先锋系统，该系统可自动从论文中提取有关大型语言模型的关键信息，帮助研究人员高效地访问有关 LLM 的信息。这些功能包括模型 \textit{licence}、模型 \textit{name} 和模型 \textit{application}。利用这些功能，我们可以为每篇论文形成一个模型卡。\textbf{数据贡献} 方面，通过定义三个词典（LLM 名称、许可证和应用程序）处理了 106 篇学术论文。通过字典查找提取了 11,051 句句子，并通过人工审查最终选定 129 句名称与许可证有联系的句子，以及 106 句模型名称与应用程序有联系的句子，构建了数据集。

Title: How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Not

• Authors: Francesco Verdini, Pierfrancesco Melucci, Stefano Perna, Francesco Cariaggi, Marco Gaido, Sara Papi, Szymon Mazurek, Marek Kasztelnik, Luisa Bentivogli, Sébastien Bratières, Paolo Merialdo, Simone Scardapane
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17044
• Pdf URL: https://arxiv.org/pdf/2409.17044
• Copy Paste: [[2409.17044]] How to Connect Speech Foundation Models and Large Language Models? What Matters and What Does Not(https://arxiv.org/abs/2409.17044)
• Keywords: language model, llm
• Abstract: The remarkable performance achieved by Large Language Models (LLM) has driven research efforts to leverage them for a wide range of tasks and input modalities. In speech-to-text (S2T) tasks, the emerging solution consists of projecting the output of the encoder of a Speech Foundational Model (SFM) into the LLM embedding space through an adapter module. However, no work has yet investigated how much the downstream-task performance depends on each component (SFM, adapter, LLM) nor whether the best design of the adapter depends on the chosen SFM and LLM. To fill this gap, we evaluate the combination of 5 adapter modules, 2 LLMs (Mistral and Llama), and 2 SFMs (Whisper and SeamlessM4T) on two widespread S2T tasks, namely Automatic Speech Recognition and Speech Translation. Our results demonstrate that the SFM plays a pivotal role in downstream performance, while the adapter choice has moderate impact and depends on the SFM and LLM.
• 摘要：大型语言模型 (LLM) 所取得的卓越性能推动了研究工作，以将其用于广泛的任务和输入模式。在语音转文本 (S2T) 任务中，新兴的解决方案包括通过适配器模块将语音基础模型 (SFM) 编码器的输出投影到 LLM 嵌入空间中。但是，尚未有研究调查下游任务性能在多大程度上取决于每个组件（SFM、适配器、LLM），也未调查适配器的最佳设计是否取决于所选的 SFM 和 LLM。为了填补这一空白，我们在两个广泛的 S2T 任务（即自动语音识别和语音翻译）上评估了 5 个适配器模块、2 个 LLM（Mistral 和 Llama）和 2 个 SFM（Whisper 和 SeamlessM4T）的组合。我们的结果表明，SFM 在下游性能中起着关键作用，而适配器选择具有中等影响，并且取决于 SFM 和 LLM。

Title: Enhancing Post-Hoc Attributions in Long Document Comprehension via Coarse Grained Answer Decomposition

• Authors: Pritika Ramu, Koustava Goswami, Apoorv Saxena, Balaji Vasan Srinivavsan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.17073
• Pdf URL: https://arxiv.org/pdf/2409.17073
• Copy Paste: [[2409.17073]] Enhancing Post-Hoc Attributions in Long Document Comprehension via Coarse Grained Answer Decomposition(https://arxiv.org/abs/2409.17073)
• Keywords: llm
• Abstract: Accurately attributing answer text to its source document is crucial for developing a reliable question-answering system. However, attribution for long documents remains largely unexplored. Post-hoc attribution systems are designed to map answer text back to the source document, yet the granularity of this mapping has not been addressed. Furthermore, a critical question arises: What precisely should be attributed, with an emphasis on identifying the information units within an answer that necessitate grounding? In this paper, we propose and investigate a novel approach to the factual decomposition of generated answers for attribution, employing template-based in-context learning. To accomplish this, we utilize the question and integrate negative sampling during few-shot in-context learning for decomposition. This approach enhances the semantic understanding of both abstractive and extractive answers. We examine the impact of answer decomposition by providing a thorough examination of various attribution approaches, ranging from retrieval-based techniques to LLM-based attributors.
• 摘要：准确地将答案文本归因于其源文档对于开发可靠的问答系统至关重要。然而，长文档的归因在很大程度上仍未得到探索。事后归因系统旨在将答案文本映射回源文档，但这种映射的粒度尚未得到解决。此外，出现了一个关键问题：应该归因什么，重点是识别答案中需要基础的信息单元？在本文中，我们提出并研究了一种新颖的方法来对生成的答案进行事实分解以进行归因，采用基于模板的上下文学习。为了实现这一点，我们利用这个问题并在少数情况下的上下文学习过程中整合负采样进行分解。这种方法增强了对抽象和提取答案的语义理解。我们通过全面检查各种归因方法（从基于检索的技术到基于 LLM 的归因器）来研究答案分解的影响。

Title: Programming Every Example: Lifting Pre-training Data Quality like Experts at Scale

• Authors: Fan Zhou, Zengzhi Wang, Qian Liu, Junlong Li, Pengfei Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17115
• Pdf URL: https://arxiv.org/pdf/2409.17115
• Copy Paste: [[2409.17115]] Programming Every Example: Lifting Pre-training Data Quality like Experts at Scale(https://arxiv.org/abs/2409.17115)
• Keywords: language model, llm
• Abstract: Large language model pre-training has traditionally relied on human experts to craft heuristics for improving the corpora quality, resulting in numerous rules developed to date. However, these rules lack the flexibility to address the unique characteristics of individual example effectively. Meanwhile, applying tailored rules to every example is impractical for human experts. In this paper, we demonstrate that even small language models, with as few as 0.3B parameters, can exhibit substantial data refining capabilities comparable to those of human experts. We introduce Programming Every Example (ProX), a novel framework that treats data refinement as a programming task, enabling models to refine corpora by generating and executing fine-grained operations, such as string normalization, for each individual example at scale. Experimental results show that models pre-trained on ProX-curated data outperform either original data or data filtered by other selection methods by more than 2% across various downstream benchmarks. Its effectiveness spans various model sizes and pre-training corpora, including C4, RedPajama-V2, and FineWeb. Furthermore, ProX exhibits significant potential in domain-specific continual pre-training: without domain specific design, models trained on OpenWebMath refined by ProX outperform human-crafted rule-based methods, improving average accuracy by 7.6% over Mistral-7B, with 14.6% for Llama-2-7B and 20.3% for CodeLlama-7B, all within 10B tokens to be comparable to models like Llemma-7B trained on 200B tokens. Further analysis highlights that ProX significantly saves training FLOPs, offering a promising path for efficient LLM pre-training.We are open-sourcing ProX with >100B corpus, models, and sharing all training and implementation details for reproducible research and future innovation. Code: this https URL
• 摘要：大型语言模型预训练传统上依靠人类专家制定启发式方法来提高语料库质量，迄今为止已开发出许多规则。然而，这些规则缺乏灵活性，无法有效解决单个示例的独特特征。同时，对每个示例应用定制规则对人类专家来说是不切实际的。在本文中，我们证明，即使是只有 0.3B 个参数的小型语言模型，也可以表现出与人类专家相当的强大数据提炼能力。我们引入了 Programming Every Example (ProX)，这是一个将数据提炼视为编程任务的新框架，使模型能够通过为每个单独的示例大规模生成和执行细粒度操作（例如字符串规范化）来提炼语料库。实验结果表明，在各种下游基准测试中，使用 ProX 整理的数据进行预训练的模型比原始数据或通过其他选择方法过滤的数据高出 2% 以上。它的有效性涵盖各种模型大小和预训练语料库，包括 C4、RedPajama-V2 和 FineWeb。此外，ProX 在特定领域的持续预训练中表现出巨大的潜力：无需特定领域的设计，经过 ProX 改进的 OpenWebMath 训练模型的表现优于人工制定的基于规则的方法，平均准确率比 Mistral-7B 提高了 7.6%，Llama-2-7B 提高了 14.6%，CodeLlama-7B 提高了 20.3%，所有模型都在 100 亿个 token 内，与 Llemma-7B 等在 2000 亿个 token 上训练的模型相当。进一步的分析表明，ProX 显著节省了训练 FLOP，为高效的 LLM 预训练提供了一条有希望的道路。我们正在开源 ProX，拥有超过 1000 亿个语料库和模型，并分享所有训练和实施细节，以供可重复的研究和未来创新。代码：此 https URL

Title: Deep Learning and Machine Learning, Advancing Big Data Analytics and Management: Handy Appetizer

• Authors: Benji Peng, Xuanhe Pan, Yizhu Wen, Ziqian Bi, Keyu Chen, Ming Li, Ming Liu, Qian Niu, Junyu Liu, Jinlang Wang, Sen Zhang, Jiawei Xu, Pohsun Feng
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17120
• Pdf URL: https://arxiv.org/pdf/2409.17120
• Copy Paste: [[2409.17120]] Deep Learning and Machine Learning, Advancing Big Data Analytics and Management: Handy Appetizer(https://arxiv.org/abs/2409.17120)
• Keywords: gpt
• Abstract: This book explores the role of Artificial Intelligence (AI), Machine Learning (ML), and Deep Learning (DL) in driving the progress of big data analytics and management. The book focuses on simplifying the complex mathematical concepts behind deep learning, offering intuitive visualizations and practical case studies to help readers understand how neural networks and technologies like Convolutional Neural Networks (CNNs) work. It introduces several classic models and technologies such as Transformers, GPT, ResNet, BERT, and YOLO, highlighting their applications in fields like natural language processing, image recognition, and autonomous driving. The book also emphasizes the importance of pre-trained models and how they can enhance model performance and accuracy, with instructions on how to apply these models in various real-world scenarios. Additionally, it provides an overview of key big data management technologies like SQL and NoSQL databases, as well as distributed computing frameworks such as Apache Hadoop and Spark, explaining their importance in managing and processing vast amounts of data. Ultimately, the book underscores the value of mastering deep learning and big data management skills as critical tools for the future workforce, making it an essential resource for both beginners and experienced professionals.
• 摘要：本书探讨了人工智能 (AI)、机器学习 (ML) 和深度学习 (DL) 在推动大数据分析和管理进步方面的作用。本书着重于简化深度学习背后复杂的数学概念，提供直观的可视化和实际案例研究，帮助读者了解神经网络和卷积神经网络 (CNN) 等技术的工作原理。它介绍了几种经典模型和技术，如 Transformers、GPT、ResNet、BERT 和 YOLO，重点介绍了它们在自然语言处理、图像识别和自动驾驶等领域的应用。本书还强调了预训练模型的重要性以及它们如何提高模型性能和准确性，并指导如何在各种实际场景中应用这些模型。此外，它概述了关键的大数据管理技术，如 SQL 和 NoSQL 数据库，以及分布式计算框架，如 Apache Hadoop 和 Spark，解释了它们在管理和处理海量数据方面的重要性。最终，本书强调了掌握深度学习和大数据管理技能作为未来劳动力的关键工具的价值，使其成为初学者和经验丰富的专业人士的重要资源。

Title: FineZip : Pushing the Limits of Large Language Models for Practical Lossless Text Compression

• Authors: Fazal Mittu, Yihuan Bu, Akshat Gupta, Ashok Devireddy, Alp Eren Ozdarendeli, Anant Singh, Gopala Anumanchipalli
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.17141
• Pdf URL: https://arxiv.org/pdf/2409.17141
• Copy Paste: [[2409.17141]] FineZip : Pushing the Limits of Large Language Models for Practical Lossless Text Compression(https://arxiv.org/abs/2409.17141)
• Keywords: language model, llm
• Abstract: While the language modeling objective has been shown to be deeply connected with compression, it is surprising that modern LLMs are not employed in practical text compression systems. In this paper, we provide an in-depth analysis of neural network and transformer-based compression techniques to answer this question. We compare traditional text compression systems with neural network and LLM-based text compression methods. Although LLM-based systems significantly outperform conventional compression methods, they are highly impractical. Specifically, LLMZip, a recent text compression system using Llama3-8B requires 9.5 days to compress just 10 MB of text, although with huge improvements in compression ratios. To overcome this, we present FineZip - a novel LLM-based text compression system that combines ideas of online memorization and dynamic context to reduce the compression time immensely. FineZip can compress the above corpus in approximately 4 hours compared to 9.5 days, a 54 times improvement over LLMZip and comparable performance. FineZip outperforms traditional algorithmic compression methods with a large margin, improving compression ratios by approximately 50\%. With this work, we take the first step towards making lossless text compression with LLMs a reality. While FineZip presents a significant step in that direction, LLMs are still not a viable solution for large-scale text compression. We hope our work paves the way for future research and innovation to solve this problem.
• 摘要：虽然语言建模目标已被证明与压缩密切相关，但令人惊讶的是，现代 LLM 并未用于实际的文本压缩系统。在本文中，我们深入分析了基于神经网络和变压器的压缩技术，以回答这个问题。我们将传统文本压缩系统与基于神经网络和 LLM 的文本压缩方法进行了比较。虽然基于 LLM 的系统明显优于传统压缩方法，但它们非常不切实际。具体来说，LLMZip 是一种使用 Llama3-8B 的最新文本压缩系统，需要 9.5 天才能压缩仅 10 MB 的文本，尽管压缩率有很大的提高。为了克服这个问题，我们提出了 FineZip——一种新颖的基于 LLM 的文本压缩系统，它结合了在线记忆和动态上下文的思想，大大缩短了压缩时间。FineZip 可以在大约 4 小时内压缩上述语料库，而 LLMZip 则需要 9.5 天，比 LLMZip 提高了 54 倍，性能相当。FineZip 的表现远远优于传统的算法压缩方法，压缩率提高了约 50%。通过这项工作，我们朝着使用 LLM 实现无损文本压缩迈出了第一步。虽然 FineZip 朝着这个方向迈出了重要一步，但 LLM 仍然不是大规模文本压缩的可行解决方案。我们希望我们的工作为未来解决这一问题的研究和创新铺平道路。