2024-12-05

Title: Optimizing Large Language Models for Turkish: New Methodologies in Corpus Selection and Training

• Authors: H. Toprak Kesgin, M. Kaan Yuce, Eren Dogan, M. Egemen Uzun, Atahan Uz, Elif Ince, Yusuf Erdem, Osama Shbib, Ahmed Zeer, M. Fatih Amasyali
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.02775
• Pdf URL: https://arxiv.org/pdf/2412.02775
• Copy Paste: [[2412.02775]] Optimizing Large Language Models for Turkish: New Methodologies in Corpus Selection and Training(https://arxiv.org/abs/2412.02775)
• Keywords: language model
• Abstract: In this study, we develop and assess new corpus selection and training methodologies to improve the effectiveness of Turkish language models. Specifically, we adapted Large Language Model generated datasets and translated English datasets into Turkish, integrating these resources into the training process. This approach led to substantial enhancements in model accuracy for both few-shot and zero-shot learning scenarios. Furthermore, the merging of these adapted models was found to markedly improve their performance. Human evaluative metrics, including task-specific performance assessments, further demonstrated that these adapted models possess a greater aptitude for comprehending the Turkish language and addressing logic-based queries. This research underscores the importance of refining corpus selection strategies to optimize the performance of multilingual models, particularly for under-resourced languages like Turkish.
• 摘要：在本研究中，我们开发并评估了新的语料库选择和训练方法，以提高土耳其语模型的有效性。具体来说，我们调整了大型语言模型生成的数据集，并将英语数据集翻译成土耳其语，将这些资源整合到训练过程中。这种方法大大提高了小样本和零样本学习场景的模型准确性。此外，我们发现，合并这些调整后的模型可以显著提高其性能。人工评估指标（包括特定任务的绩效评估）进一步表明，这些调整后的模型更善于理解土耳其语和解决基于逻辑的查询。这项研究强调了改进语料库选择策略以优化多语言模型性能的重要性，尤其是对于土耳其语等资源不足的语言。

Title: Hybrid-SQuAD: Hybrid Scholarly Question Answering Dataset

• Authors: Tilahun Abedissa Taffa, Debayan Baneerje, Yaregal Assabie, Ricardo Usbeck
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.02788
• Pdf URL: https://arxiv.org/pdf/2412.02788
• Copy Paste: [[2412.02788]] Hybrid-SQuAD: Hybrid Scholarly Question Answering Dataset(https://arxiv.org/abs/2412.02788)
• Keywords: language model
• Abstract: Existing Scholarly Question Answering (QA) methods typically target homogeneous data sources, relying solely on either text or Knowledge Graphs (KGs). However, scholarly information often spans heterogeneous sources, necessitating the development of QA systems that can integrate information from multiple heterogeneous data sources. To address this challenge, we introduce Hybrid-SQuAD (Hybrid Scholarly Question Answering Dataset), a novel large-scale QA dataset designed to facilitate answering questions incorporating both text and KG facts. The dataset consists of 10.5K question-answer pairs generated by a large language model, leveraging the KGs - DBLP and SemOpenAlex alongside corresponding text from Wikipedia. In addition, we propose a RAG-based baseline hybrid QA model, achieving an exact match score of 69.65 on the Hybrid-SQuAD test set.
• 摘要：现有的学术问答 (QA) 方法通常针对同质数据源，仅依赖于文本或知识图谱 (KG)。然而，学术信息通常跨越异构来源，因此需要开发能够整合来自多个异构数据源的信息的 QA 系统。为了应对这一挑战，我们推出了 Hybrid-SQuAD（混合学术问答数据集），这是一个新颖的大规模 QA 数据集，旨在帮助回答结合文本和 KG 事实的问题。该数据集由大型语言模型生成的 10.5K 个问答对组成，利用 KG - DBLP 和 SemOpenAlex 以及来自维基百科的相应文本。此外，我们提出了一种基于 RAG 的基线混合 QA 模型，在 Hybrid-SQuAD 测试集上实现了 69.65 的精确匹配分数。

Title: An Evolutionary Large Language Model for Hallucination Mitigation

• Authors: Abdennour Boulesnane, Abdelhakim Souilah
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.02790
• Pdf URL: https://arxiv.org/pdf/2412.02790
• Copy Paste: [[2412.02790]] An Evolutionary Large Language Model for Hallucination Mitigation(https://arxiv.org/abs/2412.02790)
• Keywords: language model, gpt, llm, hallucination, chat
• Abstract: The emergence of LLMs, like ChatGPT and Gemini, has marked the modern era of artificial intelligence applications characterized by high-impact applications generating text, images, and videos. However, these models usually ensue with one critical challenge called hallucination: confident presentation of inaccurate or fabricated information. This problem attracts serious concern when these models are applied to specialized domains, including healthcare and law, where the accuracy and preciseness of information are absolute conditions. In this paper, we propose EvoLLMs, an innovative framework inspired by Evolutionary Computation, which automates the generation of high-quality Question-answering (QA) datasets while minimizing hallucinations. EvoLLMs employs genetic algorithms, mimicking evolutionary processes like selection, variation, and mutation, to guide LLMs in generating accurate, contextually relevant question-answer pairs. Comparative analysis shows that EvoLLMs consistently outperforms human-generated datasets in key metrics such as Depth, Relevance, and Coverage, while nearly matching human performance in mitigating hallucinations. These results highlight EvoLLMs as a robust and efficient solution for QA dataset generation, significantly reducing the time and resources required for manual curation.
• 摘要：ChatGPT 和 Gemini 等 LLM 的出现标志着人工智能应用的现代时代的到来，其特点是高影响力的应用可以生成文本、图像和视频。然而，这些模型通常会带来一个关键挑战，即幻觉：自信地呈现不准确或虚假的信息。当这些模型应用于医疗保健和法律等专业领域时，这个问题引起了严重的担忧，因为在这些领域，信息的准确性和精确性是绝对的条件。在本文中，我们提出了 EvoLLMs，这是一个受进化计算启发的创新框架，它可以自动生成高质量的问答 (QA) 数据集，同时最大限度地减少幻觉。EvoLLMs 采用遗传算法，模仿选择、变异和突变等进化过程，以指导 LLM 生成准确、上下文相关的问答对。比较分析表明，EvoLLMs 在深度、相关性和覆盖率等关键指标方面始终优于人类生成的数据集，同时在缓解幻觉方面几乎与人类的表现相匹配。这些结果强调了 EvoLLM 是生成 QA 数据集的强大而有效的解决方案，可显著减少手动管理所需的时间和资源。

Title: CNNSum: Exploring Long-Conext Summarization with Large Language Models in Chinese Novels

• Authors: Lingxiao Wei, He Yan, Xiangju Lu, Junmin Zhu, Jun Wang, Wei Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.02819
• Pdf URL: https://arxiv.org/pdf/2412.02819
• Copy Paste: [[2412.02819]] CNNSum: Exploring Long-Conext Summarization with Large Language Models in Chinese Novels(https://arxiv.org/abs/2412.02819)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: Large Language Models (LLMs) have been well-researched in many long-context tasks. However, due to high annotation costs, high-quality long-context summary datasets for training or evaluation are scarce, limiting further research. In this work, we introduce CNNSum, a new multi-scale Chinese long-context novel summarization benchmark, including four subsets, length covering 16k\textasciitilde128k, 695 samples in total, the annotations are human-driven. We evaluate commercial and open-source models on CNNSum and conduct a detailed analysis. Based on the observations, we further conduct fine-tuning exploration with short-context summary data. In our study: (1) GPT-4o underperformed, due to excessive subjective commentary. (2) Currently, long-context summarization mainly relies on memory ability, small LLMs with stable longer context lengths are the most cost-effective. Using long data concatenated from short-context summaries makes a significant improvement. (3) Prompt templates may cause a large performance gap but can be mitigated through fine-tuning. (4) Fine-tuned Chat or Instruction versions may harm the Base model and further fine-tuning cannot bridge performance gap. (5) while models with RoPE base scaling exhibit strong extrapolation potential, their performance may vary significantly when combined with other interpolation methods and need careful selection. (6) CNNSum provides more reliable and insightful evaluation results than other benchmarks. We release CNNSum to advance research in this field.
• 摘要：大型语言模型（LLM）在许多长上下文任务中得到了很好的研究。然而，由于标注成本高，用于训练或评估的高质量长上下文摘要数据集很少，限制了进一步的研究。在本文中，我们引入了一个新的多尺度中文长上下文小说摘要基准 CNNSum，包括四个子集，长度覆盖 16k\textasciitilde128k，共 695 个样本，标注由人工驱动。我们在 CNNSum 上评估了商业和开源模型并进行了详细分析。基于观察，我们进一步使用短上下文摘要数据进行微调探索。在我们的研究中：（1）GPT-4o 表现不佳，原因是主观评论过多。（2）目前，长上下文摘要主要依赖于记忆能力，具有稳定的较长上下文长度的小型 LLM 最具成本效益。使用由短上下文摘要连接起来的长数据可以显着改善。 （3）提示模板可能会导致较大的性能差距，但可以通过微调来缓解。 （4）微调后的聊天或指令版本可能会损害基础模型，进一步的微调无法弥补性能差距。 （5）虽然具有 RoPE 基础缩放的模型表现出强大的外推潜力，但与其他插值方法结合时，它们的性能可能会有很大差异，需要仔细选择。 （6）CNNSum 比其他基准提供了更可靠、更有见地的评估结果。 我们发布 CNNSum 以推动该领域的研究。

Title: Minimization of Boolean Complexity in In-Context Concept Learning

• Authors: Leroy Z. Wang, R. Thomas McCoy, Shane Steinert-Threlkeld
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.02823
• Pdf URL: https://arxiv.org/pdf/2412.02823
• Copy Paste: [[2412.02823]] Minimization of Boolean Complexity in In-Context Concept Learning(https://arxiv.org/abs/2412.02823)
• Keywords: language model, llm
• Abstract: What factors contribute to the relative success and corresponding difficulties of in-context learning for Large Language Models (LLMs)? Drawing on insights from the literature on human concept learning, we test LLMs on carefully designed concept learning tasks, and show that task performance highly correlates with the Boolean complexity of the concept. This suggests that in-context learning exhibits a learning bias for simplicity in a way similar to humans.
• 摘要：哪些因素导致了大型语言模型 (LLM) 的语境学习相对成功和相应的困难？借鉴人类概念学习文献中的见解，我们在精心设计的概念学习任务上测试了 LLM，并表明任务表现与概念的布尔复杂度高度相关。这表明，语境学习表现出与人类类似的学习倾向，倾向于简单性。

Title: RARE: Retrieval-Augmented Reasoning Enhancement for Large Language Models

• Authors: Hieu Tran, Zonghai Yao, Junda Wang, Yifan Zhang, Zhichao Yang, Hong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.02830
• Pdf URL: https://arxiv.org/pdf/2412.02830
• Copy Paste: [[2412.02830]] RARE: Retrieval-Augmented Reasoning Enhancement for Large Language Models(https://arxiv.org/abs/2412.02830)
• Keywords: language model, gpt, llm
• Abstract: This work introduces RARE (Retrieval-Augmented Reasoning Enhancement), a versatile extension to the mutual reasoning framework (rStar), aimed at enhancing reasoning accuracy and factual integrity across large language models (LLMs) for complex, knowledge-intensive tasks such as commonsense and medical reasoning. RARE incorporates two innovative actions within the Monte Carlo Tree Search (MCTS) framework: A6, which generates search queries based on the initial problem statement, performs information retrieval using those queries, and augments reasoning with the retrieved data to formulate the final answer; and A7, which leverages information retrieval specifically for generated sub-questions and re-answers these sub-questions with the relevant contextual information. Additionally, a Retrieval-Augmented Factuality Scorer is proposed to replace the original discriminator, prioritizing reasoning paths that meet high standards of factuality. Experimental results with LLaMA 3.1 show that RARE enables open-source LLMs to achieve competitive performance with top open-source models like GPT-4 and GPT-4o. This research establishes RARE as a scalable solution for improving LLMs in domains where logical coherence and factual integrity are critical.
• 摘要：这项工作引入了 RARE（检索增强推理增强），这是相互推理框架（rStar）的多功能扩展，旨在提高大型语言模型（LLM）的推理准确性和事实完整性，以应对常识和医学推理等复杂、知识密集型任务。RARE 在蒙特卡洛树搜索（MCTS）框架中整合了两项创新操作：A6，它根据初始问题陈述生成搜索查询，使用这些查询执行信息检索，并使用检索到的数据增强推理以形成最终答案；A7，它专门针对生成的子问题利用信息检索，并使用相关的上下文信息重新回答这些子问题。此外，还提出了一种检索增强事实性评分器来取代原始鉴别器，优先考虑符合高事实性标准的推理路径。使用 LLaMA 3.1 的实验结果表明，RARE 可使开源 LLM 实现与 GPT-4 和 GPT-4o 等顶级开源模型相媲美的性能。这项研究确立了 RARE 是一种可扩展的解决方案，可用于在逻辑连贯性和事实完整性至关重要的领域改进 LLM。

Title: CAISSON: Concept-Augmented Inference Suite of Self-Organizing Neural Networks

• Authors: Igor Halperin
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.02835
• Pdf URL: https://arxiv.org/pdf/2412.02835
• Copy Paste: [[2412.02835]] CAISSON: Concept-Augmented Inference Suite of Self-Organizing Neural Networks(https://arxiv.org/abs/2412.02835)
• Keywords: retrieval-augmented generation
• Abstract: We present CAISSON, a novel hierarchical approach to Retrieval-Augmented Generation (RAG) that transforms traditional single-vector search into a multi-view clustering framework. At its core, CAISSON leverages dual Self-Organizing Maps (SOMs) to create complementary organizational views of the document space, where each view captures different aspects of document relationships through specialized embeddings. The first view processes combined text and metadata embeddings, while the second operates on metadata enriched with concept embeddings, enabling a comprehensive multi-view analysis that captures both fine-grained semantic relationships and high-level conceptual patterns. This dual-view approach enables more nuanced document discovery by combining evidence from different organizational perspectives. To evaluate CAISSON, we develop SynFAQA, a framework for generating synthetic financial analyst notes and question-answer pairs that systematically tests different aspects of information retrieval capabilities. Drawing on HotPotQA's methodology for constructing multi-step reasoning questions, SynFAQA generates controlled test cases where each question is paired with the set of notes containing its ground-truth answer, progressing from simple single-entity queries to complex multi-hop retrieval tasks involving multiple entities and concepts. Our experimental results demonstrate substantial improvements over both basic and enhanced RAG implementations, particularly for complex multi-entity queries, while maintaining practical response times suitable for interactive applications.
• 摘要：我们提出了 CAISSON，一种新颖的检索增强生成 (RAG) 分层方法，它将传统的单向量搜索转变为多视图聚类框架。CAISSON 的核心是利用双自组织映射 (SOM) 来创建文档空间的互补组织视图，其中每个视图通过专门的嵌入捕获文档关系的不同方面。第一个视图处理组合的文本和元数据嵌入，而第二个视图对富含概念嵌入的元数据进行操作，从而实现全面的多视图分析，既能捕获细粒度的语义关系，又能捕获高级概念模式。这种双视图方法通过结合来自不同组织视角的证据，实现了更细致入微的文档发现。为了评估 CAISSON，我们开发了 SynFAQA，这是一个用于生成合成金融分析师笔记和问答对的框架，可以系统地测试信息检索能力的不同方面。 SynFAQA 借鉴 HotPotQA 构建多步推理问题的方法，生成受控测试用例，其中每个问题都与包含其基本答案的一组注释配对，从简单的单实体查询发展到涉及多个实体和概念的复杂多跳检索任务。我们的实验结果表明，与基本和增强型 RAG 实现相比，都有显著改进，特别是对于复杂的多实体查询，同时保持了适合交互式应用程序的实际响应时间。

Title: Removing Spurious Correlation from Neural Network Interpretations

• Authors: Milad Fotouhi, Mohammad Taha Bahadori, Oluwaseyi Feyisetan, Payman Arabshahi, David Heckerman
• Subjects: cs.CL, cs.AI, cs.LG, stat.AP, stat.ME
• Abstract URL: https://arxiv.org/abs/2412.02893
• Pdf URL: https://arxiv.org/pdf/2412.02893
• Copy Paste: [[2412.02893]] Removing Spurious Correlation from Neural Network Interpretations(https://arxiv.org/abs/2412.02893)
• Keywords: language model
• Abstract: The existing algorithms for identification of neurons responsible for undesired and harmful behaviors do not consider the effects of confounders such as topic of the conversation. In this work, we show that confounders can create spurious correlations and propose a new causal mediation approach that controls the impact of the topic. In experiments with two large language models, we study the localization hypothesis and show that adjusting for the effect of conversation topic, toxicity becomes less localized.
• 摘要：现有的识别导致不良和有害行为的神经元的算法没有考虑混杂因素（例如谈话主题）的影响。在这项研究中，我们表明混杂因素可以产生虚假相关性，并提出了一种控制话题影响的新因果中介方法。在对两个大型语言模型进行的实验中，我们研究了局部化假设，并表明调整谈话主题的影响后，毒性的局部性会降低。

Title: MLD-EA: Check and Complete Narrative Coherence by Introducing Emotions and Actions

• Authors: Jinming Zhang, Yunfei Long
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.02897
• Pdf URL: https://arxiv.org/pdf/2412.02897
• Copy Paste: [[2412.02897]] MLD-EA: Check and Complete Narrative Coherence by Introducing Emotions and Actions(https://arxiv.org/abs/2412.02897)
• Keywords: language model, llm
• Abstract: Narrative understanding and story generation are critical challenges in natural language processing (NLP), with much of the existing research focused on summarization and question-answering tasks. While previous studies have explored predicting plot endings and generating extended narratives, they often neglect the logical coherence within stories, leaving a significant gap in the field. To address this, we introduce the Missing Logic Detector by Emotion and Action (MLD-EA) model, which leverages large language models (LLMs) to identify narrative gaps and generate coherent sentences that integrate seamlessly with the story's emotional and logical flow. The experimental results demonstrate that the MLD-EA model enhances narrative understanding and story generation, highlighting LLMs' potential as effective logic checkers in story writing with logical coherence and emotional consistency. This work fills a gap in NLP research and advances border goals of creating more sophisticated and reliable story-generation systems.
• 摘要：叙事理解和故事生成是自然语言处理 (NLP) 中的关键挑战，现有的大部分研究都集中在总结和问答任务上。虽然先前的研究已经探索了预测情节结局和生成扩展叙事，但它们往往忽视了故事中的逻辑连贯性，导致该领域存在重大空白。为了解决这个问题，我们引入了情感和动作缺失逻辑检测器 (MLD-EA) 模型，该模型利用大型语言模型 (LLM) 来识别叙事空白并生成与故事的情感和逻辑流程无缝集成的连贯句子。实验结果表明，MLD-EA 模型增强了叙事理解和故事生成，凸显了 LLM 作为具有逻辑连贯性和情感一致性的故事写作中有效逻辑检查器的潜力。这项工作填补了 NLP 研究的空白，并推进了创建更复杂、更可靠的故事生成系统的边界目标。

Title: Enhancing Trust in Large Language Models with Uncertainty-Aware Fine-Tuning

• Authors: Ranganath Krishnan, Piyush Khanna, Omesh Tickoo
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.02904
• Pdf URL: https://arxiv.org/pdf/2412.02904
• Copy Paste: [[2412.02904]] Enhancing Trust in Large Language Models with Uncertainty-Aware Fine-Tuning(https://arxiv.org/abs/2412.02904)
• Keywords: language model, llm, hallucination, prompt
• Abstract: Large language models (LLMs) have revolutionized the field of natural language processing with their impressive reasoning and question-answering capabilities. However, these models are sometimes prone to generating credible-sounding but incorrect information, a phenomenon known as LLM hallucinations. Reliable uncertainty estimation in LLMs is essential for fostering trust in their generated responses and serves as a critical tool for the detection and prevention of erroneous or hallucinated outputs. To achieve reliable and well-calibrated uncertainty quantification in open-ended and free-form natural language generation, we propose an uncertainty-aware fine-tuning approach for LLMs. This approach enhances the model's ability to provide reliable uncertainty estimates without compromising accuracy, thereby guiding them to produce more trustworthy responses. We introduce a novel uncertainty-aware causal language modeling loss function, grounded in the principles of decision theory. Through rigorous evaluation on multiple free-form question-answering datasets and models, we demonstrate that our uncertainty-aware fine-tuning approach yields better calibrated uncertainty estimates in natural language generation tasks than fine-tuning with the standard causal language modeling loss. Furthermore, the experimental results show that the proposed method significantly improves the model's ability to detect hallucinations and identify out-of-domain prompts.
• 摘要：大型语言模型 (LLM) 凭借其出色的推理和问答能力彻底改变了自然语言处理领域。然而，这些模型有时容易生成看似可信但实际上不正确的信息，这种现象称为 LLM 幻觉。LLM 中可靠的不确定性估计对于培养对其生成的响应的信任至关重要，并且是检测和预防错误或幻觉输出的关键工具。为了在开放式和自由形式的自然语言生成中实现可靠且经过良好校准的不确定性量化，我们提出了一种不确定性感知的 LLM 微调方法。这种方法增强了模型在不影响准确性的情况下提供可靠不确定性估计的能力，从而引导它们产生更值得信赖的响应。我们引入了一种新颖的不确定性感知因果语言建模损失函数，该函数以决策理论原理为基础。通过对多个自由形式问答数据集和模型进行严格评估，我们证明了我们的不确定性感知微调方法在自然语言生成任务中比使用标准因果语言建模损失进行微调能产生更好的校准不确定性估计。此外，实验结果表明，所提出的方法显著提高了模型检测幻觉和识别域外提示的能力。

Title: Single-Cell Omics Arena: A Benchmark Study for Large Language Models on Cell Type Annotation Using Single-Cell Data

• Authors: Junhao Liu, Siwei Xu, Lei Zhang, Jing Zhang
• Subjects: cs.CL, q-bio.GN
• Abstract URL: https://arxiv.org/abs/2412.02915
• Pdf URL: https://arxiv.org/pdf/2412.02915
• Copy Paste: [[2412.02915]] Single-Cell Omics Arena: A Benchmark Study for Large Language Models on Cell Type Annotation Using Single-Cell Data(https://arxiv.org/abs/2412.02915)
• Keywords: language model, llm, prompt, chain-of-thought
• Abstract: Over the past decade, the revolution in single-cell sequencing has enabled the simultaneous molecular profiling of various modalities across thousands of individual cells, allowing scientists to investigate the diverse functions of complex tissues and uncover underlying disease mechanisms. Among all the analytical steps, assigning individual cells to specific types is fundamental for understanding cellular heterogeneity. However, this process is usually labor-intensive and requires extensive expert knowledge. Recent advances in large language models (LLMs) have demonstrated their ability to efficiently process and synthesize vast corpora of text to automatically extract essential biological knowledge, such as marker genes, potentially promoting more efficient and automated cell type annotations. To thoroughly evaluate the capability of modern instruction-tuned LLMs in automating the cell type identification process, we introduce SOAR, a comprehensive benchmarking study of LLMs for cell type annotation tasks in single-cell genomics. Specifically, we assess the performance of 8 instruction-tuned LLMs across 11 datasets, spanning multiple cell types and species. Our study explores the potential of LLMs to accurately classify and annotate cell types in single-cell RNA sequencing (scRNA-seq) data, while extending their application to multiomics data through cross-modality translation. Additionally, we evaluate the effectiveness of chain-of-thought (CoT) prompting techniques in generating detailed biological insights during the annotation process. The results demonstrate that LLMs can provide robust interpretations of single-cell data without requiring additional fine-tuning, advancing the automation of cell type annotation in genomics research.
• 摘要：在过去十年中，单细胞测序的革命使得人们能够同时对数千个单个细胞的各种模式进行分子分析，从而使科学家能够研究复杂组织的多种功能并揭示潜在的疾病机制。在所有分析步骤中，将单个细胞分配到特定类型对于理解细胞异质性至关重要。然而，这个过程通常是劳动密集型的，需要大量的专业知识。大型语言模型 (LLM) 的最新进展已经证明了它们能够有效地处理和合成大量文本语料库，以自动提取必要的生物学知识，例如标记基因，从而有可能促进更高效和自动化的细胞类型注释。为了全面评估现代指令调整的 LLM 在自动化细胞类型识别过程中的能力，我们引入了 SOAR，这是一项全面的 LLM 基准研究，用于单细胞基因组学中的细胞类型注释任务。具体来说，我们评估了 8 个指令调整的 LLM 在 11 个数据集中的表现，涵盖多种细胞类型和物种。我们的研究探索了 LLM 在单细胞 RNA 测序 (scRNA-seq) 数据中准确分类和注释细胞类型的潜力，同时通过跨模态翻译将其应用扩展到多组学数据。此外，我们评估了思路链 (CoT) 提示技术在注释过程中产生详细生物学见解的有效性。结果表明，LLM 可以提供对单细胞数据的可靠解释，而无需额外的微调，从而推进基因组学研究中细胞类型注释的自动化。

Title: Curriculum-style Data Augmentation for LLM-based Metaphor Detection

• Authors: Kaidi Jia, Yanxia Wu, Rongsheng Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.02956
• Pdf URL: https://arxiv.org/pdf/2412.02956
• Copy Paste: [[2412.02956]] Curriculum-style Data Augmentation for LLM-based Metaphor Detection(https://arxiv.org/abs/2412.02956)
• Keywords: language model, llm
• Abstract: Recently, utilizing large language models (LLMs) for metaphor detection has achieved promising results. However, these methods heavily rely on the capabilities of closed-source LLMs, which come with relatively high inference costs and latency. To address this, we propose a method for metaphor detection by fine-tuning open-source LLMs, effectively reducing inference costs and latency with a single inference step. Furthermore, metaphor detection suffers from a severe data scarcity problem, which hinders effective fine-tuning of LLMs. To tackle this, we introduce Curriculum-style Data Augmentation (CDA). Specifically, before fine-tuning, we evaluate the training data to identify correctly predicted instances for fine-tuning, while incorrectly predicted instances are used as seed data for data augmentation. This approach enables the model to quickly learn simpler knowledge and progressively acquire more complex knowledge, thereby improving performance incrementally. Experimental results demonstrate that our method achieves state-of-the-art performance across all baselines. Additionally, we provide detailed ablation studies to validate the effectiveness of CDA.
• 摘要：最近，利用大型语言模型 (LLM) 进行隐喻检测取得了令人鼓舞的成果。然而，这些方法严重依赖于闭源 LLM 的功能，而闭源 LLM 的推理成本和延迟相对较高。为了解决这个问题，我们提出了一种通过微调开源 LLM 进行隐喻检测的方法，通过单个推理步骤有效地降低了推理成本和延迟。此外，隐喻检测存在严重的数据稀缺问题，这阻碍了 LLM 的有效微调。为了解决这个问题，我们引入了课程式数据增强 (CDA)。具体来说，在微调之前，我们评估训练数据以识别正确预测的实例进行微调，而错误预测的实例则用作数据增强的种子数据。这种方法使模型能够快速学习更简单的知识并逐步获取更复杂的知识，从而逐步提高性能。实验结果表明，我们的方法在所有基线上都实现了最先进的性能。此外，我们还提供了详细的消融研究来验证 CDA 的有效性。

Title: Advancing Conversational Psychotherapy: Integrating Privacy, Dual-Memory, and Domain Expertise with Large Language Models

• Authors: XiuYu Zhang, Zening Luo
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.02987
• Pdf URL: https://arxiv.org/pdf/2412.02987
• Copy Paste: [[2412.02987]] Advancing Conversational Psychotherapy: Integrating Privacy, Dual-Memory, and Domain Expertise with Large Language Models(https://arxiv.org/abs/2412.02987)
• Keywords: language model, llm, prompt, chat, retrieval augmented generation
• Abstract: Mental health has increasingly become a global issue that reveals the limitations of traditional conversational psychotherapy, constrained by location, time, expense, and privacy concerns. In response to these challenges, we introduce SoulSpeak, a Large Language Model (LLM)-enabled chatbot designed to democratize access to psychotherapy. SoulSpeak improves upon the capabilities of standard LLM-enabled chatbots by incorporating a novel dual-memory component that combines short-term and long-term context via Retrieval Augmented Generation (RAG) to offer personalized responses while ensuring the preservation of user privacy and intimacy through a dedicated privacy module. In addition, it leverages a counseling chat dataset of therapist-client interactions and various prompting techniques to align the generated responses with psychotherapeutic methods. We introduce two fine-tuned BERT models to evaluate the system against existing LLMs and human therapists: the Conversational Psychotherapy Preference Model (CPPM) to simulate human preference among responses and another to assess response relevance to user input. CPPM is useful for training and evaluating psychotherapy-focused language models independent from SoulSpeak, helping with the constrained resources available for psychotherapy. Furthermore, the effectiveness of the dual-memory component and the robustness of the privacy module are also examined. Our findings highlight the potential and challenge of enhancing mental health care by offering an alternative that combines the expertise of traditional therapy with the advantages of LLMs, providing a promising way to address the accessibility and personalization gap in current mental health services.
• 摘要：心理健康日益成为一个全球性问题，这揭示了传统对话心理治疗的局限性，受到地点、时间、费用和隐私问题的限制。为了应对这些挑战，我们推出了 SoulSpeak，这是一款支持大型语言模型 (LLM) 的聊天机器人，旨在使心理治疗的普及。SoulSpeak 通过整合一种新颖的双记忆组件来改进标准 LLM 聊天机器人的功能，该组件通过检索增强生成 (RAG) 结合短期和长期背景来提供个性化响应，同时通过专用隐私模块确保保护用户隐私和亲密关系。此外，它利用治疗师与客户互动的咨询聊天数据集和各种提示技术来使生成的响应与心理治疗方法保持一致。我们引入了两个经过微调的 BERT 模型来评估该系统与现有 LLM 和人类治疗师的比较：对话心理治疗偏好模型 (CPPM) 用于模拟人类对响应的偏好，另一个用于评估响应与用户输入的相关性。 CPPM 可用于训练和评估独立于 SoulSpeak 的以心理治疗为中心的语言模型，从而帮助解决可用于心理治疗的有限资源问题。此外，我们还检查了双记忆组件的有效性和隐私模块的稳健性。我们的研究结果强调了通过提供一种结合传统疗法的专业知识和 LLM 优势的替代方案来增强心理健康护理的潜力和挑战，为解决当前心理健康服务中的可及性和个性化差距提供了一种有希望的方法。

Title: Human Variability vs. Machine Consistency: A Linguistic Analysis of Texts Generated by Humans and Large Language Models

• Authors: Sergio E. Zanotto, Segun Aroyehun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03025
• Pdf URL: https://arxiv.org/pdf/2412.03025
• Copy Paste: [[2412.03025]] Human Variability vs. Machine Consistency: A Linguistic Analysis of Texts Generated by Humans and Large Language Models(https://arxiv.org/abs/2412.03025)
• Keywords: language model, llm
• Abstract: The rapid advancements in large language models (LLMs) have significantly improved their ability to generate natural language, making texts generated by LLMs increasingly indistinguishable from human-written texts. Recent research has predominantly focused on using LLMs to classify text as either human-written or machine-generated. In our study, we adopt a different approach by profiling texts spanning four domains based on 250 distinct linguistic features. We select the M4 dataset from the Subtask B of SemEval 2024 Task 8. We automatically calculate various linguistic features with the LFTK tool and additionally measure the average syntactic depth, semantic similarity, and emotional content for each document. We then apply a two-dimensional PCA reduction to all the calculated features. Our analyses reveal significant differences between human-written texts and those generated by LLMs, particularly in the variability of these features, which we find to be considerably higher in human-written texts. This discrepancy is especially evident in text genres with less rigid linguistic style constraints. Our findings indicate that humans write texts that are less cognitively demanding, with higher semantic content, and richer emotional content compared to texts generated by LLMs. These insights underscore the need for incorporating meaningful linguistic features to enhance the understanding of textual outputs of LLMs.
• 摘要：大型语言模型 (LLM) 的快速发展显著提高了其生成自然语言的能力，使得 LLM 生成的文本与人类书写的文本越来越难以区分。最近的研究主要集中在使用 LLM 将文本分类为人类书写的或机器生成的。在我们的研究中，我们采用了一种不同的方法，根据 250 种不同的语言特征对四个领域的文本进行分析。我们从 SemEval 2024 任务 8 的子任务 B 中选择了 M4 数据集。我们使用 LFTK 工具自动计算各种语言特征，并另外测量每个文档的平均句法深度、语义相似性和情感内容。然后，我们对所有计算出的特征应用二维 PCA 约简。我们的分析揭示了人类书写的文本与 LLM 生成的文本之间存在显著差异，特别是在这些特征的可变性方面，我们发现人类书写的文本中的可变性要高得多。这种差异在语言风格约束不那么严格的文本类型中尤其明显。我们的研究结果表明，与 LLM 生成的文本相比，人类撰写的文本对认知的要求较低，语义内容更丰富，情感内容更丰富。这些见解强调了需要结合有意义的语言特征来增强对 LLM 文本输出的理解。

Title: ASR-EC Benchmark: Evaluating Large Language Models on Chinese ASR Error Correction

• Authors: Victor Junqiu Wei, Weicheng Wang, Di Jiang, Yuanfeng Song, Lu Wang
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.03075
• Pdf URL: https://arxiv.org/pdf/2412.03075
• Copy Paste: [[2412.03075]] ASR-EC Benchmark: Evaluating Large Language Models on Chinese ASR Error Correction(https://arxiv.org/abs/2412.03075)
• Keywords: language model, llm, prompt
• Abstract: Automatic speech Recognition (ASR) is a fundamental and important task in the field of speech and natural language processing. It is an inherent building block in many applications such as voice assistant, speech translation, etc. Despite the advancement of ASR technologies in recent years, it is still inevitable for modern ASR systems to have a substantial number of erroneous recognition due to environmental noise, ambiguity, etc. Therefore, the error correction in ASR is crucial. Motivated by this, this paper studies ASR error correction in the Chinese language, which is one of the most popular languages and enjoys a large number of users in the world. We first create a benchmark dataset named \emph{ASR-EC} that contains a wide spectrum of ASR errors generated by industry-grade ASR systems. To the best of our knowledge, it is the first Chinese ASR error correction benchmark. Then, inspired by the recent advances in \emph{large language models (LLMs)}, we investigate how to harness the power of LLMs to correct ASR errors. We apply LLMs to ASR error correction in three paradigms. The first paradigm is prompting, which is further categorized as zero-shot, few-shot, and multi-step. The second paradigm is finetuning, which finetunes LLMs with ASR error correction data. The third paradigm is multi-modal augmentation, which collectively utilizes the audio and ASR transcripts for error correction. Extensive experiments reveal that prompting is not effective for ASR error correction. Finetuning is effective only for a portion of LLMs. Multi-modal augmentation is the most effective method for error correction and achieves state-of-the-art performance.
• 摘要：自动语音识别 (ASR) 是语音和自然语言处理领域的一项基本且重要的任务。它是语音助手、语音翻译等许多应用程序的固有构建块。尽管近年来 ASR 技术取得了进步，但现代 ASR 系统仍不可避免地会由于环境噪声、歧义等原因而出现大量错误识别。因此，ASR 中的错​​误纠正至关重要。受此启发，本文研究了中文的 ASR 错误纠正，中文是世界上最流行的语言之一，拥有大量用户。我们首先创建一个名为 \emph{ASR-EC} 的基准数据集，其中包含由工业级 ASR 系统生成的各种 ASR 错误。据我们所知，这是第一个中文 ASR 错误纠正基准。然后，受到 \emph{大型语言模型 (LLM)} 最新进展的启发，我们研究如何利用 LLM 的强大功能来纠正 ASR 错误。我们将 LLM 以三种范式应用于 ASR 纠错。第一种范式是提示，进一步分为零样本、少样本和多步骤。第二种范式是微调，使用 ASR 纠错数据对 LLM 进行微调。第三种范式是多模态增强，它共同利用音频和 ASR 转录进行纠错。大量实验表明，提示对于 ASR 纠错无效。微调仅对部分 LLM 有效。多模态增强是最有效的纠错方法，并实现了最佳性能。

Title: Revolve: Optimizing AI Systems by Tracking Response Evolution in Textual Optimization

• Authors: Peiyan Zhang, Haibo Jin, Leyang Hu, Xinnuo Li, Liying Kang, Man Luo, Yangqiu Song, Haohan Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.03092
• Pdf URL: https://arxiv.org/pdf/2412.03092
• Copy Paste: [[2412.03092]] Revolve: Optimizing AI Systems by Tracking Response Evolution in Textual Optimization(https://arxiv.org/abs/2412.03092)
• Keywords: language model, llm, prompt
• Abstract: Recent advancements in large language models (LLMs) have significantly enhanced the ability of LLM-based systems to perform complex tasks through natural language processing and tool interaction. However, optimizing these LLM-based systems for specific tasks remains challenging, often requiring manual interventions like prompt engineering and hyperparameter tuning. Existing automatic optimization methods, such as textual feedback-based techniques (e.g., TextGrad), tend to focus on immediate feedback, analogous to using immediate derivatives in traditional numerical gradient descent. However, relying solely on such feedback can be limited when the adjustments made in response to this feedback are either too small or fluctuate irregularly, potentially slowing down or even stalling the optimization process. To overcome these challenges, more adaptive methods are needed, especially in situations where the system's response is evolving slowly or unpredictably. In this paper, we introduce REVOLVE, an optimization method that tracks how "R"esponses "EVOLVE" across iterations in LLM systems. By focusing on the evolution of responses over time, REVOLVE enables more stable and effective optimization by making thoughtful, progressive adjustments at each step. Experimental results demonstrate that REVOLVE outperforms competitive baselines, achieving a 7.8% improvement in prompt optimization, a 20.72% gain in solution refinement, and a 29.17% increase in code optimization. Additionally, REVOLVE converges in fewer iterations, resulting in significant computational savings. These advantages highlight its adaptability and efficiency, positioning REVOLVE as a valuable tool for optimizing LLM-based systems and accelerating the development of next-generation AI technologies. Code is available at: this https URL.
• 摘要：大型语言模型 (LLM) 的最新进展显著增强了基于 LLM 的系统通过自然语言处理和工具交互执行复杂任务的能力。然而，针对特定任务优化这些基于 LLM 的系统仍然具有挑战性，通常需要手动干预，如及时工程和超参数调整。现有的自动优化方法，例如基于文本反馈的技术（例如 TextGrad），往往侧重于即时反馈，类似于在传统数值梯度下降中使用即时导数。然而，当响应这种反馈而做出的调整太小或波动不规律时，仅仅依靠这种反馈可能会受到限制，从而可能减慢甚至停滞优化过程。为了克服这些挑战，需要更多自适应方法，特别是在系统响应缓慢或不可预测的情况下。在本文中，我们介绍了 REVOLVE，这是一种优化方法，可跟踪 LLM 系统中“R”响应在迭代过程中如何“EVOLVE”。 REVOLVE 关注响应随时间的变化，通过在每个步骤进行深思熟虑的渐进式调整，实现更稳定、更有效的优化。实验结果表明，REVOLVE 的表现优于竞争基线，在快速优化方面提高了 7.8%，在解决方案改进方面提高了 20.72%，在代码优化方面提高了 29.17%。此外，REVOLVE 可以在更少的迭代中收敛，从而节省大量计算资源。这些优势凸显了其适应性和效率，使 REVOLVE 成为优化基于 LLM 的系统和加速下一代 AI 技术开发的宝贵工具。代码可在以下网址获取：此 https URL。

Title: TOOL-ED: Enhancing Empathetic Response Generation with the Tool Calling Capability of LLM

• Authors: Huiying Cao, Yiqun Zhang, Shi Feng, Xiaocui Yang, Daling Wang, Yifei Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03096
• Pdf URL: https://arxiv.org/pdf/2412.03096
• Copy Paste: [[2412.03096]] TOOL-ED: Enhancing Empathetic Response Generation with the Tool Calling Capability of LLM(https://arxiv.org/abs/2412.03096)
• Keywords: language model, llm
• Abstract: Empathetic conversation is a crucial characteristic in daily conversations between individuals. Nowadays, Large Language models (LLMs) have shown outstanding performance in generating empathetic responses. Knowledge bases like COMET can assist LLMs in mitigating illusions and enhancing the understanding of users' intentions and emotions. However, models remain heavily reliant on fixed knowledge bases and unrestricted incorporation of external knowledge can introduce noise. Tool learning is a flexible end-to-end approach that assists LLMs in handling complex problems. In this paper, we propose Emotional Knowledge Tool Calling (EKTC) framework, which encapsulates the commonsense knowledge bases as empathetic tools, enabling LLMs to integrate external knowledge flexibly through tool calling. In order to adapt the models to the new task, we construct a novel dataset TOOL-ED based on the EMPATHETICMPATHETIC DIALOGUE (ED) dataset. We validate EKTC on the ED dataset, and the experimental results demonstrate that our framework can enhance the ability of LLMs to generate empathetic responses effectively.
• 摘要：富有同理心的对话是人与人之间日常对话中的一个重要特征。如今，大型语言模型 (LLM) 在生成富有同理心的回应方面表现出色。像 COMET 这样的知识库可以帮助 LLM 减轻错觉并增强对用户意图和情绪的理解。然而，模型仍然严重依赖于固定的知识库，而不受限制地整合外部知识可能会引入噪音。工具学习是一种灵活的端到端方法，可帮助 LLM 处理复杂问题。在本文中，我们提出了情感知识工具调用 (EKTC) 框架，该框架将常识知识库封装为富有同理心的工具，使 LLM 能够通过工具调用灵活地整合外部知识。为了让模型适应新任务，我们基于 EMPATHETICMPATHETIC DIALOGUE (ED) 数据集构建了一个新数据集 TOOL-ED。我们在 ED 数据集上验证了 EKTC，实验结果表明我们的框架可以有效增强 LLM 生成共情反应的能力。

Title: A surprisal oracle for when every layer counts

• Authors: Xudong Hong, Sharid Loáiciga, Asad Sayeed
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03098
• Pdf URL: https://arxiv.org/pdf/2412.03098
• Copy Paste: [[2412.03098]] A surprisal oracle for when every layer counts(https://arxiv.org/abs/2412.03098)
• Keywords: language model
• Abstract: Active Curriculum Language Modeling (ACLM; Hong et al., 2023) is a learner directed approach to training a language model. We proposed the original version of this process in our submission to the BabyLM 2023 task, and now we propose an updated ACLM process for the BabyLM 2024 task. ACLM involves an iteratively- and dynamically-constructed curriculum informed over the training process by a model of uncertainty; other training items that are similarly uncertain to a least certain candidate item are prioritized. Our new process improves the similarity model so that it is more dynamic, and we run ACLM over the most successful model from the BabyLM 2023 task: ELC-BERT (Charpentier and Samuel, 2023). We find that while our models underperform on fine-grained grammatical inferences, they outperform the BabyLM 2024 official base-lines on common-sense and world-knowledge tasks. We make our code available at https: //github.com/asayeed/ActiveBaby.
• 摘要：主动课程语言建模 (ACLM；Hong 等人，2023) 是一种以学习者为导向的语言模型训练方法。我们在提交给 BabyLM 2023 任务的论文中提出了此过程的原始版本，现在我们为 BabyLM 2024 任务提出了更新的 ACLM 过程。ACLM 涉及一个迭代和动态构建的课程，该课程在训练过程中由不确定性模型提供信息；其他与最不确定候选项目同样不确定的训练项目将被优先考虑。我们的新流程改进了相似性模型，使其更具动态性，并且我们在 BabyLM 2023 任务中最成功的模型上运行 ACLM：ELC-BERT（Charpentier 和 Samuel，2023）。我们发现，虽然我们的模型在细粒度语法推理方面表现不佳，但它们在常识和世界知识任务上的表现优于 BabyLM 2024 官方基线。我们的代码可在https://github.com/asayeed/ActiveBaby上获取。

Title: Fine-Grained Behavior Simulation with Role-Playing Large Language Model on Social Media

• Authors: Kun Li, Chenwei Dai, Wei Zhou, Songlin Hu
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.03148
• Pdf URL: https://arxiv.org/pdf/2412.03148
• Copy Paste: [[2412.03148]] Fine-Grained Behavior Simulation with Role-Playing Large Language Model on Social Media(https://arxiv.org/abs/2412.03148)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) have demonstrated impressive capabilities in role-playing tasks. However, there is limited research on whether LLMs can accurately simulate user behavior in real-world scenarios, such as social media. This requires models to effectively analyze a user's history and simulate their role. In this paper, we introduce \textbf{FineRob}, a novel fine-grained behavior simulation dataset. We collect the complete behavioral history of 1,866 distinct users across three social media platforms. Each behavior is decomposed into three fine-grained elements: object, type, and content, resulting in 78.6k QA records. Based on FineRob, we identify two dominant reasoning patterns in LLMs' behavior simulation processes and propose the \textbf{OM-CoT} fine-tuning method to enhance the capability. Through comprehensive experiments, we conduct an in-depth analysis of key factors of behavior simulation and also demonstrate the effectiveness of OM-CoT approach\footnote{Code and dataset are available at \url{this https URL}}
• 摘要：大型语言模型 (LLM) 在角色扮演任务中表现出色。然而，关于 LLM 是否能够准确模拟社交媒体等现实场景中的用户行为的研究有限。这需要模型有效地分析用户的历史并模拟他们的角色。在本文中，我们介绍了 \textbf{FineRob}，这是一个新颖的细粒度行为模拟数据集。我们收集了三个社交媒体平台上 1,866 个不同用户的完整行为历史。每个行为被分解为三个细粒度元素：对象、类型和内容，从而产生 78.6k 条 QA 记录。基于 FineRob，我们确定了 LLM 行为模拟过程中的两种主要推理模式，并提出了 \textbf{OM-CoT} 微调方法来增强其能力。通过全面的实验，我们对行为模拟的关键因素进行了深入分析，并证明了 OM-CoT 方法的有效性\footnote{代码和数据集可在 \url{此 https URL}} 获得

Title: Byte BPE Tokenization as an Inverse string Homomorphism

• Authors: Saibo Geng, Sankalp Gambhir, Chris Wendler, Robert West
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03160
• Pdf URL: https://arxiv.org/pdf/2412.03160
• Copy Paste: [[2412.03160]] Byte BPE Tokenization as an Inverse string Homomorphism(https://arxiv.org/abs/2412.03160)
• Keywords: language model, llm
• Abstract: Tokenization is an important preprocessing step in the training and inference of large language models (LLMs). While there has been extensive research on the expressive power of the neural achitectures used in LLMs, the impact of tokenization has not been well understood. In this work, we demonstrate that tokenization, irrespective of the algorithm used, acts as an inverse homomorphism between strings and tokens. This suggests that the character space of the source language and the token space of the tokenized language are homomorphic, preserving the structural properties of the source language. Additionally, we explore the concept of proper tokenization, which refers to an unambiguous tokenization returned from the tokenizer. Our analysis reveals that the expressiveness of neural architectures in recognizing context-free languages is not affected by tokenization.
• 摘要：标记化是大型语言模型 (LLM) 训练和推理中的一个重要预处理步骤。虽然已经对 LLM 中使用的神经架构的表达能力进行了广泛的研究，但标记化的影响尚未得到很好的理解。在这项工作中，我们证明了标记化，无论使用哪种算法，都充当字符串和标记之间的逆同态。这表明源语言的字符空间和标记化语言的标记空间是同态的，保留了源语言的结构属性。此外，我们探索了正确标记化的概念，它指的是从标记器返回的明确标记化。我们的分析表明，神经架构在识别上下文无关语言方面的表达能力不受标记化的影响。

Title: Automatic detection of diseases in Spanish clinical notes combining medical language models and ontologies

• Authors: Leon-Paul Schaub Torre, Pelayo Quiros, Helena Garcia Mieres
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03176
• Pdf URL: https://arxiv.org/pdf/2412.03176
• Copy Paste: [[2412.03176]] Automatic detection of diseases in Spanish clinical notes combining medical language models and ontologies(https://arxiv.org/abs/2412.03176)
• Keywords: language model
• Abstract: In this paper we present a hybrid method for the automatic detection of dermatological pathologies in medical reports. We use a large language model combined with medical ontologies to predict, given a first appointment or follow-up medical report, the pathology a person may suffer from. The results show that teaching the model to learn the type, severity and location on the body of a dermatological pathology, as well as in which order it has to learn these three features, significantly increases its accuracy. The article presents the demonstration of state-of-the-art results for classification of medical texts with a precision of 0.84, micro and macro F1-score of 0.82 and 0.75, and makes both the method and the data set used available to the community.
• 摘要：在本文中，我们介绍了一种用于自动检测医疗报告中的皮肤病的混合方法。我们使用大型语言模型与医学本体相结合，根据首次预约或后续医疗报告预测患者可能患上的疾病。结果表明，教会模型学习皮肤病的类型、严重程度和身体部位，以及学习这三个特征的顺序，可以显著提高其准确性。本文展示了最先进的医学文本分类结果，精度为 0.84，微观和宏观 F1 分数为 0.82 和 0.75，并将该方法和所使用的数据集提供给社区。

Title: Weighted-Reward Preference Optimization for Implicit Model Fusion

• Authors: Ziyi Yang, Fanqi Wan, Longguang Zhong, Tianyuan Shi, Xiaojun Quan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03187
• Pdf URL: https://arxiv.org/pdf/2412.03187
• Copy Paste: [[2412.03187]] Weighted-Reward Preference Optimization for Implicit Model Fusion(https://arxiv.org/abs/2412.03187)
• Keywords: gpt, llm
• Abstract: While fusing heterogeneous open-source LLMs with varying architectures and sizes can potentially integrate the strengths of different models, existing fusion methods face significant challenges, such as vocabulary alignment and merging distribution matrices. These procedures are not only complex but also prone to introducing noise and errors. In this paper, we propose an implicit fusion method, Weighted-Reward Preference Optimization (WRPO), which leverages preference optimization between the source LLMs and the target LLM to transfer their capabilities effectively. WRPO eliminates the need for vocabulary alignment and matrix fusion and can be efficiently scaled to accommodate various LLMs. To address distributional deviations between the source and target LLMs, WRPO introduces a progressive adaptation strategy that gradually shifts reliance on preferred examples from the target LLM to the source LLMs. Extensive experiments on the MT-Bench, AlpacaEval-2, and Arena-Hard benchmarks demonstrate that WRPO consistently outperforms existing knowledge fusion methods and various fine-tuning baselines. When applied to LLaMA3-8B-Instruct as the target model, WRPO achieves a length-controlled win rate of 55.9% against GPT-4-Preview-1106 on AlpacaEval-2 and a win rate of 46.2% against GPT-4-0314 on Arena-Hard. Our code is available at \url{this https URL}.
• 摘要：虽然融合具有不同架构和大小的异构开源 LLM 可以整合不同模型的优势，但现有的融合方法面临着重大挑战，例如词汇对齐和合并分布矩阵。这些过程不仅复杂，而且容易引入噪音和错误。在本文中，我们提出了一种隐式融合方法，即加权奖励偏好优化 (WRPO)，它利用源 LLM 和目标 LLM 之间的偏好优化来有效地转移它们的能力。WRPO 消除了词汇对齐和矩阵融合的需要，并且可以有效地扩展以适应各种 LLM。为了解决源和目标 LLM 之间的分布偏差，WRPO 引入了一种渐进式适应策略，该策略逐渐将对首选示例的依赖从目标 LLM 转移到源 LLM。在 MT-Bench、AlpacaEval-2 和 Arena-Hard 基准上进行的大量实验表明，WRPO 始终优于现有的知识融合方法和各种微调基线。当将 LLaMA3-8B-Instruct 用作​​目标模型时，WRPO 在 AlpacaEval-2 上对 GPT-4-Preview-1106 的长度控制胜率为 55.9%，在 Arena-Hard 上对 GPT-4-0314 的长度控制胜率为 46.2%。我们的代码可在 \url{此 https URL} 上找到。

Title: U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMs

• Authors: Konstantin Chernyshev, Vitaliy Polshkov, Ekaterina Artemova, Alex Myasnikov, Vlad Stepanov, Alexei Miasnikov, Sergei Tilga
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.03205
• Pdf URL: https://arxiv.org/pdf/2412.03205
• Copy Paste: [[2412.03205]] U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMs(https://arxiv.org/abs/2412.03205)
• Keywords: llm
• Abstract: The current evaluation of mathematical skills in LLMs is limited, as existing benchmarks are either relatively small, primarily focus on elementary and high-school problems, or lack diversity in topics. Additionally, the inclusion of visual elements in tasks remains largely under-explored. To address these gaps, we introduce U-MATH, a novel benchmark of 1,100 unpublished open-ended university-level problems sourced from teaching materials. It is balanced across six core subjects, with 20% of multimodal problems. Given the open-ended nature of U-MATH problems, we employ an LLM to judge the correctness of generated solutions. To this end, we release $\mu$-MATH, a dataset to evaluate the LLMs' capabilities in judging solutions. The evaluation of general domain, math-specific, and multimodal LLMs highlights the challenges presented by U-MATH. Our findings reveal that LLMs achieve a maximum accuracy of only 63% on text-based tasks, with even lower 45% on visual problems. The solution assessment proves challenging for LLMs, with the best LLM judge having an F1-score of 80% on $\mu$-MATH.
• 摘要：目前对 LLM 数学技能的评估有限，因为现有的基准要么相对较小，主要关注小学和高中问题，要么缺乏主题多样性。此外，在任务中加入视觉元素仍然在很大程度上未被充分探索。为了弥补这些差距，我们引入了 U-MATH，这是一个新颖的基准，包含 1,100 个未发表的开放式大学水平问题，这些问题来自教学材料。它在六个核心科目中保持平衡，其中 20% 是多模态问题。鉴于 U-MATH 问题的开放性，我们使用 LLM 来判断生成的解决方案的正确性。为此，我们发布了 $\mu$-MATH，这是一个用于评估 LLM 判断解决方案能力的数据集。对一般领域、数学特定和多模态 LLM 的评估凸显了 U-MATH 所带来的挑战。我们的研究结果表明，LLM 在基于文本的任务上的最高准确率仅为 63%，在视觉问题上的准确率甚至更低，仅为 45%。解决方案评估对于 LLM 来说具有挑战性，最佳 LLM 评委在 $\mu$-MATH 上的 F1 分数为 80%。

Title: Linq-Embed-Mistral Technical Report

• Authors: Chanyeol Choi, Junseong Kim, Seolhwa Lee, Jihoon Kwon, Sangmo Gu, Yejin Kim, Minkyung Cho, Jy-yong Sohn
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03223
• Pdf URL: https://arxiv.org/pdf/2412.03223
• Copy Paste: [[2412.03223]] Linq-Embed-Mistral Technical Report(https://arxiv.org/abs/2412.03223)
• Keywords: language model, llm
• Abstract: This report explores the enhancement of text retrieval performance using advanced data refinement techniques. We develop Linq-Embed-Mistral\footnote{\url{this https URL}} by building on the E5-mistral and Mistral-7B-v0.1 models, focusing on sophisticated data crafting, data filtering, and negative mining methods, which are highly tailored to each task, applied to both existing benchmark dataset and highly tailored synthetic dataset generated via large language models (LLMs). Linq-Embed-Mistral excels in the MTEB benchmarks (as of May 29, 2024), achieving an average score of 68.2 across 56 datasets, and ranks 1st among all models for retrieval tasks on the MTEB leaderboard with a performance score of 60.2. This performance underscores its superior capability in enhancing search precision and reliability. Our contributions include advanced data refinement methods that significantly improve model performance on benchmark and synthetic datasets, techniques for homogeneous task ordering and mixed task fine-tuning to enhance model generalization and stability, and a streamlined evaluation process using 4-bit precision and a light retrieval evaluation set, which accelerates validation without sacrificing accuracy.
• 摘要：本报告探讨了使用高级数据细化技术来增强文本检索性能。我们在 E5-mistral 和 Mistral-7B-v0.1 模型的基础上开发了 Linq-Embed-Mistral\footnote{\url{this https URL}}，重点关注复杂的数据制作、数据过滤和负面挖掘方法，这些方法针对每项任务进行了高度定制，应用于现有基准数据集和通过大型语言模型 (LLM) 生成的高度定制的合成数据集。Linq-Embed-Mistral 在 MTEB 基准测试中表现出色（截至 2024 年 5 月 29 日），在 56 个数据集中的平均得分为 68.2，在 MTEB 排行榜上以 60.2 的性能得分在所有检索任务模型中排名第一。这一表现凸显了其在提高搜索精度和可靠性方面的卓越能力。我们的贡献包括先进的数据细化方法，可显著提高基准和合成数据集上的模型性能，同质任务排序和混合任务微调技术可增强模型泛化和稳定性，以及使用 4 位精度和轻量检索评估集的简化评估流程，可在不牺牲准确性的情况下加速验证。

Title: PERL: Pinyin Enhanced Rephrasing Language Model for Chinese ASR N-best Error Correction

• Authors: Junhong Liang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03230
• Pdf URL: https://arxiv.org/pdf/2412.03230
• Copy Paste: [[2412.03230]] PERL: Pinyin Enhanced Rephrasing Language Model for Chinese ASR N-best Error Correction(https://arxiv.org/abs/2412.03230)
• Keywords: language model
• Abstract: ASR correction methods have predominantly focused on general datasets and have not effectively utilized Pinyin information, unique to the Chinese language. In this study, we address this gap by proposing a Pinyin Enhanced Rephrasing Language Model (PERL), specifically designed for N-best correction scenarios. Additionally, we implement a length predictor module to address the variable-length problem. We conduct experiments on the Aishell-1 dataset and our newly proposed DoAD dataset. The results show that our approach outperforms baseline methods, achieving a 29.11% reduction in Character Error Rate (CER) on Aishell-1 and around 70% CER reduction on domain-specific datasets. Furthermore, our approach leverages Pinyin similarity at the token level, providing an advantage over baselines and leading to superior performance.
• 摘要：ASR 校正方法主要侧重于一般数据集，未能有效利用汉语独有的拼音信息。在本研究中，我们通过提出一种拼音增强复述语言模型 (PERL) 来解决这一问题，该模型专为 N-best 校正场景而设计。此外，我们还实现了一个长度预测器模块来解决可变长度问题。我们在 Aishell-1 数据集和我们新提出的 DoAD 数据集上进行了实验。结果表明，我们的方法优于基线方法，在 Aishell-1 上实现了字符错误率 (CER) 降低 29.11%，在特定领域数据集上实现了 CER 降低约 70%。此外，我们的方法在 token 级别利用了拼音相似性，比基线更具优势，并实现了卓越的性能。

Title: Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?

• Authors: Sravanti Addepalli, Yerram Varun, Arun Suggala, Karthikeyan Shanmugam, Prateek Jain
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.03235
• Pdf URL: https://arxiv.org/pdf/2412.03235
• Copy Paste: [[2412.03235]] Does Safety Training of LLMs Generalize to Semantically Related Natural Prompts?(https://arxiv.org/abs/2412.03235)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large Language Models (LLMs) are known to be susceptible to crafted adversarial attacks or jailbreaks that lead to the generation of objectionable content despite being aligned to human preferences using safety fine-tuning methods. While the large dimensionality of input token space makes it inevitable to find adversarial prompts that can jailbreak these models, we aim to evaluate whether safety fine-tuned LLMs are safe against natural prompts which are semantically related to toxic seed prompts that elicit safe responses after alignment. We surprisingly find that popular aligned LLMs such as GPT-4 can be compromised using naive prompts that are NOT even crafted with an objective of jailbreaking the model. Furthermore, we empirically show that given a seed prompt that elicits a toxic response from an unaligned model, one can systematically generate several semantically related natural prompts that can jailbreak aligned LLMs. Towards this, we propose a method of Response Guided Question Augmentation (ReG-QA) to evaluate the generalization of safety aligned LLMs to natural prompts, that first generates several toxic answers given a seed question using an unaligned LLM (Q to A), and further leverages an LLM to generate questions that are likely to produce these answers (A to Q). We interestingly find that safety fine-tuned LLMs such as GPT-4o are vulnerable to producing natural jailbreak questions from unsafe content (without denial) and can thus be used for the latter (A to Q) step. We obtain attack success rates that are comparable to/ better than leading adversarial attack methods on the JailbreakBench leaderboard, while being significantly more stable against defenses such as Smooth-LLM and Synonym Substitution, which are effective against existing all attacks on the leaderboard.
• 摘要：众所周知，大型语言模型 (LLM) 容易受到精心设计的对抗性攻击或越狱，尽管使用安全微调方法与人类偏好保持一致，但仍会导致生成令人反感的内容。虽然输入标记空间的维数很大，因此不可避免地会找到可以越狱这些模型的对抗性提示，但我们的目标是评估安全微调的 LLM 是否可以抵御自然提示，这些提示在语义上与在对齐后引发安全响应的有毒种子提示相关。我们惊讶地发现，流行的对齐 LLM（例如 GPT-4）可能会被天真的提示所破坏，这些提示甚至不是以越狱模型为目的而制作的。此外，我们通过经验表明，给定一个从未对齐的模型中引发有毒响应的种子提示，人们可以系统地生成几个可以越狱对齐 LLM 的语义相关的自然提示。为此，我们提出了一种响应引导问题增强 (ReG-QA) 方法来评估与安全性一致的 LLM 对自然提示的泛化，该方法首先使用未对齐的 LLM（Q 到 A）根据种子问题生成几个有毒答案，然后进一步利用 LLM 生成可能产生这些答案的问题（A 到 Q）。有趣的是，我们发现，安全性微调的 LLM（例如 GPT-4o）很容易从不安全内容中产生自然越狱问题（不会拒绝），因此可以用于后者（A 到 Q）步骤。我们获得的攻击成功率与 JailbreakBench 排行榜上领先的对抗性攻击方法相当/更好，同时对 Smooth-LLM 和同义词替换等防御措施更加稳定，这些防御措施对排行榜上现有的所有攻击都有效。

Title: Benchmarking terminology building capabilities of ChatGPT on an English-Russian Fashion Corpus

• Authors: Anastasiia Bezobrazova, Miriam Seghiri, Constantin Orasan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03242
• Pdf URL: https://arxiv.org/pdf/2412.03242
• Copy Paste: [[2412.03242]] Benchmarking terminology building capabilities of ChatGPT on an English-Russian Fashion Corpus(https://arxiv.org/abs/2412.03242)
• Keywords: gpt, chat
• Abstract: This paper compares the accuracy of the terms extracted using SketchEngine, TBXTools and ChatGPT. In addition, it evaluates the quality of the definitions produced by ChatGPT for these terms. The research is carried out on a comparable corpus of fashion magazines written in English and Russian collected from the web. A gold standard for the fashion terminology was also developed by identifying web pages that can be harvested automatically and contain definitions of terms from the fashion domain in English and Russian. This gold standard was used to evaluate the quality of the extracted terms and of the definitions produced. Our evaluation shows that TBXTools and SketchEngine, while capable of high recall, suffer from reduced precision as the number of terms increases, which affects their overall performance. Conversely, ChatGPT demonstrates superior performance, maintaining or improving precision as more terms are considered. Analysis of the definitions produced by ChatGPT for 60 commonly used terms in English and Russian shows that ChatGPT maintains a reasonable level of accuracy and fidelity across languages, but sometimes the definitions in both languages miss crucial specifics and include unnecessary deviations. Our research reveals that no single tool excels universally; each has strengths suited to particular aspects of terminology extraction and application.
• 摘要：本文比较了使用 SketchEngine、TBXTools 和 ChatGPT 提取的术语的准确性。此外，它还评估了 ChatGPT 为这些术语生成的定义的质量。这项研究是在从网络上收集的用英语和俄语编写的时尚杂志的可比语料库上进行的。还通过识别可以自动收集的网页并包含英语和俄语时尚领域术语的定义，制定了时尚术语的黄金标准。该黄金标准用于评估提取的术语和生成的定义的质量。我们的评估表明，虽然 TBXTools 和 SketchEngine 具有高召回率，但随着术语数量的增加，其精确度会降低，从而影响它们的整体性能。相反，ChatGPT 表现出卓越的性能，在考虑更多术语时保持或提高精确度。对 ChatGPT 为英语和俄语中 60 个常用术语生成的定义的分析表明，ChatGPT 在跨语言方面保持了合理的准确性和保真度，但有时两种语言的定义都会遗漏关键细节并包含不必要的偏差。我们的研究表明，没有任何一种工具能够普遍发挥作用；每种工具都有适合术语提取和应用特定方面的优势。

Title: Alignment at Pre-training! Towards Native Alignment for Arabic LLMs

• Authors: Juhao Liang, Zhenyang Cai, Jianqing Zhu, Huang Huang, Kewei Zong, Bang An, Mosen Alharthi, Juncai He, Lian Zhang, Haizhou Li, Benyou Wang, Jinchao Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03253
• Pdf URL: https://arxiv.org/pdf/2412.03253
• Copy Paste: [[2412.03253]] Alignment at Pre-training! Towards Native Alignment for Arabic LLMs(https://arxiv.org/abs/2412.03253)
• Keywords: language model, llm
• Abstract: The alignment of large language models (LLMs) is critical for developing effective and safe language models. Traditional approaches focus on aligning models during the instruction tuning or reinforcement learning stages, referred to in this paper as `post alignment'. We argue that alignment during the pre-training phase, which we term `native alignment', warrants investigation. Native alignment aims to prevent unaligned content from the beginning, rather than relying on post-hoc processing. This approach leverages extensively aligned pre-training data to enhance the effectiveness and usability of pre-trained models. Our study specifically explores the application of native alignment in the context of Arabic LLMs. We conduct comprehensive experiments and ablation studies to evaluate the impact of native alignment on model performance and alignment stability. Additionally, we release open-source Arabic LLMs that demonstrate state-of-the-art performance on various benchmarks, providing significant benefits to the Arabic LLM community.
• 摘要：大型语言模型 (LLM) 的对齐对于开发有效且安全的语言模型至关重要。传统方法侧重于在指令调整或强化学习阶段对齐模型，本文将其称为“后对齐”。我们认为，预训练阶段的对齐（我们称之为“本机对齐”）值得研究。本机对齐旨在从一开始就防止未对齐的内容，而不是依赖于事后处理。这种方法利用广泛对齐的预训练数据来增强预训练模型的有效性和可用性。我们的研究专门探讨了本机对齐在阿拉伯语 LLM 中的应用。我们进行了全面的实验和消融研究，以评估本机对齐对模型性能和对齐稳定性的影响。此外，我们发布了开源阿拉伯语 LLM，这些 LLM 在各种基准测试中都表现出最先进的性能，为阿拉伯语 LLM 社区带来了巨大的利益。

Title: Intent-driven In-context Learning for Few-shot Dialogue State Tracking

• Authors: Zihao Yi, Zhe Xu, Ying Shen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.03270
• Pdf URL: https://arxiv.org/pdf/2412.03270
• Copy Paste: [[2412.03270]] Intent-driven In-context Learning for Few-shot Dialogue State Tracking(https://arxiv.org/abs/2412.03270)
• Keywords: language model
• Abstract: Dialogue state tracking (DST) plays an essential role in task-oriented dialogue systems. However, user's input may contain implicit information, posing significant challenges for DST tasks. Additionally, DST data includes complex information, which not only contains a large amount of noise unrelated to the current turn, but also makes constructing DST datasets expensive. To address these challenges, we introduce Intent-driven In-context Learning for Few-shot DST (IDIC-DST). By extracting user's intent, we propose an Intent-driven Dialogue Information Augmentation module to augment the dialogue information, which can track dialogue states more effectively. Moreover, we mask noisy information from DST data and rewrite user's input in the Intent-driven Examples Retrieval module, where we retrieve similar examples. We then utilize a pre-trained large language model to update the dialogue state using the augmented dialogue information and examples. Experimental results demonstrate that IDIC-DST achieves state-of-the-art performance in few-shot settings on MultiWOZ 2.1 and MultiWOZ 2.4 datasets.
• 摘要：对话状态跟踪 (DST) 在面向任务的对话系统中起着至关重要的作用。然而，用户的输入可能包含隐式信息，这对 DST 任务构成了重大挑战。此外，DST 数据包含复杂信息，不仅包含大量与当前回合无关的噪音，而且使构建 DST 数据集的成本很高。为了应对这些挑战，我们引入了意图驱动的少样本 DST 上下文学习 (IDIC-DST)。通过提取用户意图，我们提出了一个意图驱动的对话信息增强模块来增强对话信息，从而可以更有效地跟踪对话状态。此外，我们从 DST 数据中屏蔽了噪声信息，并在意图驱动的示例检索模块中重写用户的输入，在该模块中我们检索类似的示例。然后，我们利用预先训练的大型语言模型，使用增强的对话信息和示例来更新对话状态。实验结果表明，IDIC-DST 在 MultiWOZ 2.1 和 MultiWOZ 2.4 数据集的少样本设置中实现了最佳性能。

Title: AntLM: Bridging Causal and Masked Language Models

• Authors: Xinru Yu, Bin Guo, Shiwei Luo, Jie Wang, Tao Ji, Yuanbin Wu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03275
• Pdf URL: https://arxiv.org/pdf/2412.03275
• Copy Paste: [[2412.03275]] AntLM: Bridging Causal and Masked Language Models(https://arxiv.org/abs/2412.03275)
• Keywords: language model
• Abstract: Causal Language Modeling (CLM) and Masked Language Modeling (MLM) are two mainstream learning paradigms based on Transformer networks, specifically the Decoder-only and Encoder-only architectures. The strengths of each paradigm in downstream tasks have shown a mix of advantages and disadvantages. In the past BabyLM Challenge 2023, although the MLM paradigm achieved the best average performance, the CLM paradigm demonstrated significantly faster convergence rates. For the BabyLM Challenge 2024, we propose a novel language modeling paradigm named $\textbf{AntLM}$, which integrates both CLM and MLM to leverage the advantages of these two classic paradigms. We chose the strict-small track and conducted experiments on two foundation models: BabyLlama, representing CLM, and LTG-BERT, representing MLM. During the training process for specific foundation models, we alternate between applying CLM or MLM training objectives and causal or bidirectional attention masks. Experimental results show that combining the two pretraining objectives leverages their strengths, enhancing overall training performance. Under the same epochs, $AntLM_{BabyLlama}$ improves Macro-average by 1%, and $AntLM_{LTG-BERT}$ achieves a 2.2% increase over the baselines.
• 摘要：因果语言模型（CLM）和掩码语言模型（MLM）是基于Transformer网络的两种主流学习范式，具体包括Decoder-only和Encoder-only架构。每种范式在下游任务中的优势都表现出了优缺点并存的现象。在过去的BabyLM Challenge 2023中，虽然MLM范式取得了最佳平均表现，但CLM范式的收敛速度明显更快。在BabyLM Challenge 2024中，我们提出了一种新的语言建模范式AntLM，将CLM与MLM融合在一起，充分发挥两种经典范式的优势。我们选择了strict-small tr​​ack，在两个基础模型上进行了实验：代表CLM的BabyLlama和代表MLM的LTG-BERT。在具体基础模型的训练过程中，我们交替应用CLM或MLM的训练目标和因果或双向注意力掩码。实验结果表明，将两个预训练目标结合起来可以发挥它们的优势，提高整体的训练效果。在相同的 epoch 下，$AntLM_{BabyLlama}$ 将 Macro-average 提高了 1%，而 $AntLM_{LTG-BERT}$ 比基线提高了 2.2%。

Title: Improving Linguistic Diversity of Large Language Models with Possibility Exploration Fine-Tuning

• Authors: Long Mai, Julie Carson-Berndsen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.03343
• Pdf URL: https://arxiv.org/pdf/2412.03343
• Copy Paste: [[2412.03343]] Improving Linguistic Diversity of Large Language Models with Possibility Exploration Fine-Tuning(https://arxiv.org/abs/2412.03343)
• Keywords: language model, llm, prompt, chat
• Abstract: While Large Language Models (LLMs) have made significant strides in replicating human-like abilities, there are concerns about a reduction in the linguistic diversity of their outputs. This results in the homogenization of viewpoints and perspectives, as well as the underrepresentation of specific demographic groups. Although several fine-tuning and prompting techniques have been suggested to tackle the issue, they are often tailored to specific tasks or come with a substantial increase in computational cost and latency. This makes them challenging to apply to applications that demand very low latency, such as chatbots and virtual assistants. We propose Possibility Exploration Fine-Tuning (PEFT), a task-agnostic framework that enhances the text diversity of LLMs without increasing latency or computational cost. Given the same prompt, models fine-tuned with PEFT can simultaneously generate multiple diverse responses, each corresponding with a controllable possibility number. Experiments on dialogue and story generation tasks demonstrate that PEFT significantly enhances the diversity of LLM outputs, as evidenced by lower similarity between candidate responses. Since PEFT emphasizes semantic diversity over lexical diversity, it can also notably reduce demographic bias in dialogue systems. The implementations and datasets are available in our repository: this https URL
• 摘要：虽然大型语言模型 (LLM) 在复制类似人类的能力方面取得了重大进展，但人们担心其输出的语言多样性会降低。这导致观点和视角的同质化，以及特定人口群体的代表性不足。尽管已经提出了几种微调和提示技术来解决这一问题，但它们通常是针对特定任务量身定制的，或者会大幅增加计算成本和延迟。这使得它们很难应用于需要极低延迟的应用程序，例如聊天机器人和虚拟助手。我们提出了可能性探索微调 (PEFT)，这是一个与任务无关的框架，可在不增加延迟或计算成本的情况下增强 LLM 的文本多样性。给定相同的提示，使用 PEFT 微调的模型可以同时生成多个不同的响应，每个响应都对应一个可控的可能性数。对话和故事生成任务的实验表明，PEFT 显著增强了 LLM 输出的多样性，候选响应之间的相似性较低就是明证。由于 PEFT 强调语义多样性而非词汇多样性，它还可以显著减少对话系统中的人口统计学偏见。实现和数据集可在我们的存储库中找到：此 https URL

Title: RedStone: Curating General, Code, Math, and QA Data for Large Language Models

• Authors: Yaoyao Chang, Lei Cui, Li Dong, Shaohan Huang, Yangyu Huang, Yupan Huang, Scarlett Li, Tengchao Lv, Shuming Ma, Qinzheng Sun, Wenhui Wang, Furu Wei, Ying Xin, Mao Yang, Qiufeng Yin, Xingxing Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.03398
• Pdf URL: https://arxiv.org/pdf/2412.03398
• Copy Paste: [[2412.03398]] RedStone: Curating General, Code, Math, and QA Data for Large Language Models(https://arxiv.org/abs/2412.03398)
• Keywords: language model, llm
• Abstract: Pre-training Large Language Models (LLMs) on high-quality, meticulously curated datasets is widely recognized as critical for enhancing their performance and generalization capabilities. This study explores the untapped potential of Common Crawl as a comprehensive and flexible resource for pre-training LLMs, addressing both general-purpose language understanding and specialized domain knowledge. We introduce RedStone, an innovative and scalable pipeline engineered to extract and process data from Common Crawl, facilitating the creation of extensive and varied pre-training datasets. Unlike traditional datasets, which often require expensive curation and domain-specific expertise, RedStone leverages the breadth of Common Crawl to deliver datasets tailored to a wide array of domains. In this work, we exemplify its capability by constructing pre-training datasets across multiple fields, including general language understanding, code, mathematics, and question-answering tasks. The flexibility of RedStone allows for easy adaptation to other specialized domains, significantly lowering the barrier to creating valuable domain-specific datasets. Our findings demonstrate that Common Crawl, when harnessed through effective pipelines like RedStone, can serve as a rich, renewable source of pre-training data, unlocking new avenues for domain adaptation and knowledge discovery in LLMs. This work also underscores the importance of innovative data acquisition strategies and highlights the role of web-scale data as a powerful resource in the continued evolution of LLMs. RedStone code and data samples will be publicly available at \url{this https URL}.
• 摘要：人们普遍认为，在高质量、精心策划的数据集上对大型语言模型 (LLM) 进行预训练对于提高其性能和泛化能力至关重要。本研究探索了 Common Crawl 作为预训练 LLM 的全面灵活资源的尚未开发的潜力，既解决了通用语言理解问题，也解决了专业领域的知识问题。我们推出了 RedStone，这是一种创新且可扩展的管道，旨在从 Common Crawl 中提取和处理数据，从而促进创建广泛而多样的预训练数据集。与通常需要昂贵的管理和领域特定专业知识的传统数据集不同，RedStone 利用 Common Crawl 的广度来提供针对广泛领域量身定制的数据集。在这项工作中，我们通过构建跨多个领域的预训练数据集来展示其能力，包括一般语言理解、代码、数学和问答任务。RedStone 的灵活性使其可以轻松适应其他专业领域，大大降低了创建有价值的领域特定数据集的障碍。我们的研究结果表明，当通过 RedStone 等有效管道加以利用时，Common Crawl 可以成为丰富的可再生预训练数据源，为 LLM 中的领域适应和知识发现开辟新途径。这项工作还强调了创新数据获取策略的重要性，并强调了网络规模数据作为 LLM 持续发展中强大资源的作用。RedStone 代码和数据样本将在 \url{此 https URL} 上公开提供。

Title: FANAL -- Financial Activity News Alerting Language Modeling Framework

• Authors: Urjitkumar Patel, Fang-Chun Yeh, Chinmay Gondhalekar, Hari Nalluri
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.03527
• Pdf URL: https://arxiv.org/pdf/2412.03527
• Copy Paste: [[2412.03527]] FANAL -- Financial Activity News Alerting Language Modeling Framework(https://arxiv.org/abs/2412.03527)
• Keywords: language model, gpt
• Abstract: In the rapidly evolving financial sector, the accurate and timely interpretation of market news is essential for stakeholders needing to navigate unpredictable events. This paper introduces FANAL (Financial Activity News Alerting Language Modeling Framework), a specialized BERT-based framework engineered for real-time financial event detection and analysis, categorizing news into twelve distinct financial categories. FANAL leverages silver-labeled data processed through XGBoost and employs advanced fine-tuning techniques, alongside ORBERT (Odds Ratio BERT), a novel variant of BERT fine-tuned with ORPO (Odds Ratio Preference Optimization) for superior class-wise probability calibration and alignment with financial event relevance. We evaluate FANAL's performance against leading large language models, including GPT-4o, Llama-3.1 8B, and Phi-3, demonstrating its superior accuracy and cost efficiency. This framework sets a new standard for financial intelligence and responsiveness, significantly outstripping existing models in both performance and affordability.
• 摘要：在快速发展的金融领域，准确及时地解读市场新闻对于需要应对不可预测事件的利益相关者至关重要。本文介绍了 FANAL（金融活动新闻警报语言建模框架），这是一种基于 BERT 的专门框架，专为实时金融事件检测和分析而设计，将新闻分为十二个不同的金融类别。FANAL 利用通过 XGBoost 处理的银标数据，并采用先进的微调技术，以及 ORBERT（比值比 BERT），这是 BERT 的一种新变体，通过 ORPO（比值比偏好优化）进行了微调，以实现卓越的类别概率校准和与金融事件相关性的对齐。我们将 FANAL 与领先的大型语言模型（包括 GPT-4o、Llama-3.1 8B 和 Phi-3）进行了性能评估，证明了其卓越的准确性和成本效益。该框架为金融智能和响应能力树立了新标准，在性能和可负担性方面远远超过现有模型。

Title: A Review on Scientific Knowledge Extraction using Large Language Models in Biomedical Sciences

• Authors: Gabriel Lino Garcia, João Renato Ribeiro Manesco, Pedro Henrique Paiola, Lucas Miranda, Maria Paola de Salvo, João Paulo Papa
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.03531
• Pdf URL: https://arxiv.org/pdf/2412.03531
• Copy Paste: [[2412.03531]] A Review on Scientific Knowledge Extraction using Large Language Models in Biomedical Sciences(https://arxiv.org/abs/2412.03531)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: The rapid advancement of large language models (LLMs) has opened new boundaries in the extraction and synthesis of medical knowledge, particularly within evidence synthesis. This paper reviews the state-of-the-art applications of LLMs in the biomedical domain, exploring their effectiveness in automating complex tasks such as evidence synthesis and data extraction from a biomedical corpus of documents. While LLMs demonstrate remarkable potential, significant challenges remain, including issues related to hallucinations, contextual understanding, and the ability to generalize across diverse medical tasks. We highlight critical gaps in the current research literature, particularly the need for unified benchmarks to standardize evaluations and ensure reliability in real-world applications. In addition, we propose directions for future research, emphasizing the integration of state-of-the-art techniques such as retrieval-augmented generation (RAG) to enhance LLM performance in evidence synthesis. By addressing these challenges and utilizing the strengths of LLMs, we aim to improve access to medical literature and facilitate meaningful discoveries in healthcare.
• 摘要：大型语言模型 (LLM) 的快速发展为医学知识的提取和综合开辟了新的领域，特别是在证据综合方面。本文回顾了 LLM 在生物医学领域的最新应用，探讨了它们在自动执行复杂任务（例如证据综合和从生物医学文档语料库中提取数据）方面的有效性。虽然 LLM 表现出巨大的潜力，但仍存在重大挑战，包括与幻觉、情境理解和跨各种医疗任务概括的能力相关的问题。我们强调了当前研究文献中的关键差距，特别是需要统一的基准来标准化评估并确保实际应用中的可靠性。此外，我们提出了未来研究的方向，强调整合最先进的技术，例如检索增强生成 (RAG)，以增强 LLM 在证据综合中的表现。通过应对这些挑战并利用 LLM 的优势，我们旨在改善医学文献的获取并促进医疗保健领域有意义的发现。

Title: Evaluating Gender Bias Transfer between Pre-trained and Prompt-Adapted Language Models

• Authors: Natalie Mackraz, Nivedha Sivakumar, Samira Khorshidi, Krishna Patel, Barry-John Theobald, Luca Zappella, Nicholas Apostoloff
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.03537
• Pdf URL: https://arxiv.org/pdf/2412.03537
• Copy Paste: [[2412.03537]] Evaluating Gender Bias Transfer between Pre-trained and Prompt-Adapted Language Models(https://arxiv.org/abs/2412.03537)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) are increasingly being adapted to achieve task-specificity for deployment in real-world decision systems. Several previous works have investigated the bias transfer hypothesis (BTH) by studying the effect of the fine-tuning adaptation strategy on model fairness to find that fairness in pre-trained masked language models have limited effect on the fairness of models when adapted using fine-tuning. In this work, we expand the study of BTH to causal models under prompt adaptations, as prompting is an accessible, and compute-efficient way to deploy models in real-world systems. In contrast to previous works, we establish that intrinsic biases in pre-trained Mistral, Falcon and Llama models are strongly correlated (rho >= 0.94) with biases when the same models are zero- and few-shot prompted, using a pronoun co-reference resolution task. Further, we find that bias transfer remains strongly correlated even when LLMs are specifically prompted to exhibit fair or biased behavior (rho >= 0.92), and few-shot length and stereotypical composition are varied (rho >= 0.97). Our findings highlight the importance of ensuring fairness in pre-trained LLMs, especially when they are later used to perform downstream tasks via prompt adaptation.
• 摘要：大型语言模型 (LLM) 越来越多地被调整以实现任务特定性，以便在现实世界的决策系统中部署。之前的几项研究通过研究微调适应策略对模型公平性的影响来调查偏见转移假设 (BTH)，结果发现预训练的掩蔽语言模型中的公平性在使用微调进行调整时对模型公平性的影响有限。在这项工作中，我们将 BTH 的研究扩展到提示适应下的因果模型，因为提示是一种可访问且计算效率高的方法，可以在现实世界系统中部署模型。与之前的研究相反，我们确定预训练的 Mistral、Falcon 和 Llama 模型中的内在偏见与使用代词共指消解任务对相同模型进行零次和少次提示时的偏见密切相关（rho >= 0.94）。此外，我们发现，即使 LLM 被明确提示表现出公平或有偏见的行为（rho >= 0.92），并且小样本长度和刻板构图各不相同（rho >= 0.97），偏见转移仍然保持着强烈的相关性。我们的研究结果强调了确保预训练 LLM 公平性的重要性，尤其是当它们后来通过即时适应用于执行下游任务时。

Title: Best-of-N Jailbreaking

• Authors: John Hughes, Sara Price, Aengus Lynch, Rylan Schaeffer, Fazl Barez, Sanmi Koyejo, Henry Sleight, Erik Jones, Ethan Perez, Mrinank Sharma
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.03556
• Pdf URL: https://arxiv.org/pdf/2412.03556
• Copy Paste: [[2412.03556]] Best-of-N Jailbreaking(https://arxiv.org/abs/2412.03556)
• Keywords: language model, gpt, prompt
• Abstract: We introduce Best-of-N (BoN) Jailbreaking, a simple black-box algorithm that jailbreaks frontier AI systems across modalities. BoN Jailbreaking works by repeatedly sampling variations of a prompt with a combination of augmentations - such as random shuffling or capitalization for textual prompts - until a harmful response is elicited. We find that BoN Jailbreaking achieves high attack success rates (ASRs) on closed-source language models, such as 89% on GPT-4o and 78% on Claude 3.5 Sonnet when sampling 10,000 augmented prompts. Further, it is similarly effective at circumventing state-of-the-art open-source defenses like circuit breakers. BoN also seamlessly extends to other modalities: it jailbreaks vision language models (VLMs) such as GPT-4o and audio language models (ALMs) like Gemini 1.5 Pro, using modality-specific augmentations. BoN reliably improves when we sample more augmented prompts. Across all modalities, ASR, as a function of the number of samples (N), empirically follows power-law-like behavior for many orders of magnitude. BoN Jailbreaking can also be composed with other black-box algorithms for even more effective attacks - combining BoN with an optimized prefix attack achieves up to a 35% increase in ASR. Overall, our work indicates that, despite their capability, language models are sensitive to seemingly innocuous changes to inputs, which attackers can exploit across modalities.
• 摘要：我们引入了 Best-of-N (BoN) 越狱，这是一种简单的黑盒算法，可以越狱跨模态的前沿 AI 系统。BoN 越狱的工作原理是反复采样提示的变化，并使用增强功能组合（例如随机打乱或将文本提示大写），直到引发有害响应。我们发现，在采样 10,000 个增强提示时，BoN 越狱在闭源语言模型上实现了高攻击成功率 (ASR)，例如在 GPT-4o 上为 89%，在 Claude 3.5 Sonnet 上为 78%。此外，它在绕过断路器等最先进的开源防御方面同样有效。BoN 还可以无缝扩展到其他模态：它使用特定于模态的增强功能越狱视觉语言模型 (VLM)，例如 GPT-4o 和音频语言模型 (ALM)，例如 Gemini 1.5 Pro。当我们采样更多增强提示时，BoN 会得到可靠的改进。在所有模态中，ASR 作为样本数量 (N) 的函数，经验上遵循幂律行为，数量级数之多。BoN 越狱还可以与其他黑盒算法组合，以实现更有效的攻击 - 将 BoN 与优化的前缀攻击相结合，可将 ASR 提高 35%。总体而言，我们的工作表明，尽管语言模型功能强大，但它们对看似无害的输入变化很敏感，攻击者可以跨模态利用这些变化。

Title: From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents

• Authors: Xinyi Mou, Xuanwen Ding, Qi He, Liang Wang, Jingcong Liang, Xinnong Zhang, Libo Sun, Jiayu Lin, Jie Zhou, Xuanjing Huang, Zhongyu Wei
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.03563
• Pdf URL: https://arxiv.org/pdf/2412.03563
• Copy Paste: [[2412.03563]] From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents(https://arxiv.org/abs/2412.03563)
• Keywords: language model, llm, agent
• Abstract: Traditional sociological research often relies on human participation, which, though effective, is expensive, challenging to scale, and with ethical concerns. Recent advancements in large language models (LLMs) highlight their potential to simulate human behavior, enabling the replication of individual responses and facilitating studies on many interdisciplinary studies. In this paper, we conduct a comprehensive survey of this field, illustrating the recent progress in simulation driven by LLM-empowered agents. We categorize the simulations into three types: (1) Individual Simulation, which mimics specific individuals or demographic groups; (2) Scenario Simulation, where multiple agents collaborate to achieve goals within specific contexts; and (3) Society Simulation, which models interactions within agent societies to reflect the complexity and variety of real-world dynamics. These simulations follow a progression, ranging from detailed individual modeling to large-scale societal phenomena. We provide a detailed discussion of each simulation type, including the architecture or key components of the simulation, the classification of objectives or scenarios and the evaluation method. Afterward, we summarize commonly used datasets and benchmarks. Finally, we discuss the trends across these three types of simulation. A repository for the related sources is at {\url{this https URL}}.
• 摘要：传统的社会学研究往往依赖于人类的参与，这种参与虽然有效，但成本高昂、难以扩大规模，而且存在伦理问题。大型语言模型 (LLM) 的最新进展凸显了它们模拟人类行为的潜力，使复制个体反应成为可能，并促进了许多跨学科研究。在本文中，我们对这一领域进行了全面的调查，展示了由 LLM 赋能的代理推动的模拟的最新进展。我们将模拟分为三类：（1）个体模拟，模仿特定的个体或人口群体；（2）场景模拟，多个代理在特定环境中合作实现目标；（3）社会模拟，模拟代理社会中的互动，以反映现实世界动态的复杂性和多样性。这些模拟遵循从详细的个体建模到大规模社会现象的进展。我们对每种模拟类型进行了详细的讨论，包括模拟的架构或关键组件、目标或场景的分类以及评估方法。之后，我们总结了常用的数据集和基准。最后，我们讨论了这三种模拟类型的趋势。相关资源的存储库位于 {\url{this https URL}}。