2025-01-07

Title: Cross-model Transferability among Large Language Models on the Platonic Representations of Concepts

• Authors: Youcheng Huang, Chen Huang, Duanyu Feng, Wenqiang Lei, Jiancheng Lv
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02009
• Pdf URL: https://arxiv.org/pdf/2501.02009
• Copy Paste: [[2501.02009]] Cross-model Transferability among Large Language Models on the Platonic Representations of Concepts(https://arxiv.org/abs/2501.02009)
• Keywords: language model, llm
• Abstract: Understanding the inner workings of Large Language Models (LLMs) is a critical research frontier. Prior research has shown that a single LLM's concept representations can be captured as steering vectors (SVs), enabling the control of LLM behavior (e.g., towards generating harmful content). Our work takes a novel approach by exploring the intricate relationships between concept representations across different LLMs, drawing an intriguing parallel to Plato's Allegory of the Cave. In particular, we introduce a linear transformation method to bridge these representations and present three key findings: 1) Concept representations across different LLMs can be effectively aligned using simple linear transformations, enabling efficient cross-model transfer and behavioral control via SVs. 2) This linear transformation generalizes across concepts, facilitating alignment and control of SVs representing different concepts across LLMs. 3) A weak-to-strong transferability exists between LLM concept representations, whereby SVs extracted from smaller LLMs can effectively control the behavior of larger LLMs.
• 摘要：了解大型语言模型 (LLM) 的内部工作原理是一项重要的研究前沿。先前的研究表明，单个 LLM 的概念表示可以作为控制向量 (SV) 捕获，从而实现对 LLM 行为的控制（例如，生成有害内容）。我们的工作采用了一种新颖的方法，探索了不同 LLM 之间概念表示之间的复杂关系，与柏拉图的“洞穴寓言”形成了有趣的对比。具体来说，我们引入了一种线性变换方法来连接这些表示，并提出了三个关键发现：1) 可以使用简单的线性变换有效地对齐不同 LLM 之间概念表示，从而通过 SV 实现高效的跨模型传输和行为控制。2) 这种线性变换跨概念推广，有助于对齐和控制跨 LLM 表示不同概念的 SV。3) LLM 概念表示之间存在从弱到强的可转移性，因此从较小的 LLM 中提取的 SV 可以有效地控制较大的 LLM 的行为。

Title: Safeguarding Large Language Models in Real-time with Tunable Safety-Performance Trade-offs

• Authors: Joao Fonseca, Andrew Bell, Julia Stoyanovich
• Subjects: cs.CL, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02018
• Pdf URL: https://arxiv.org/pdf/2501.02018
• Copy Paste: [[2501.02018]] Safeguarding Large Language Models in Real-time with Tunable Safety-Performance Trade-offs(https://arxiv.org/abs/2501.02018)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have been shown to be susceptible to jailbreak attacks, or adversarial attacks used to illicit high risk behavior from a model. Jailbreaks have been exploited by cybercriminals and blackhat actors to cause significant harm, highlighting the critical need to safeguard widely-deployed models. Safeguarding approaches, which include fine-tuning models or having LLMs "self-reflect", may lengthen the inference time of a model, incur a computational penalty, reduce the semantic fluency of an output, and restrict ``normal'' model behavior. Importantly, these Safety-Performance Trade-offs (SPTs) remain an understudied area. In this work, we introduce a novel safeguard, called SafeNudge, that combines Controlled Text Generation with "nudging", or using text interventions to change the behavior of a model. SafeNudge triggers during text-generation while a jailbreak attack is being executed, and can reduce successful jailbreak attempts by 30% by guiding the LLM towards a safe responses. It adds minimal latency to inference and has a negligible impact on the semantic fluency of outputs. Further, we allow for tunable SPTs. SafeNudge is open-source and available through this https URL, and is compatible with models loaded with the Hugging Face "transformers" library.
• 摘要：大型语言模型 (LLM) 已被证明容易受到越狱攻击或用于从模型中引出高风险行为的对抗性攻击。网络犯罪分子和黑帽攻击者利用越狱造成重大伤害，凸显了保护广泛部署的模型的迫切需要。保护方法包括微调模型或让 LLM“自我反思”，可能会延长模型的推理时间、产生计算惩罚、降低输出的语义流畅性并限制“正常”模型行为。重要的是，这些安全性能权衡 (SPT) 仍然是一个研究不足的领域。在这项工作中，我们引入了一种名为 SafeNudge 的新型保护措施，它将受控文本生成与“推动”相结合，或使用文本干预来改变模型的行为。SafeNudge 在执行越狱攻击时在文本生成期间触发，并且可以通过引导 LLM 走向安全响应将成功的越狱尝试减少 30%。它对推理的延迟影响最小，对输出的语义流畅性的影响可以忽略不计。此外，我们允许可调的 SPT。SafeNudge 是开源的，可通过此 https URL 获得，并且与加载了 Hugging Face“transformers”库的模型兼容。

Title: Enhancing Uncertainty Modeling with Semantic Graph for Hallucination Detection

• Authors: Kedi Chen, Qin Chen, Jie Zhou, Xinqi Tao, Bowen Ding, Jingwen Xie, Mingchen Xie, Peilong Li, Feng Zheng, Liang He
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02020
• Pdf URL: https://arxiv.org/pdf/2501.02020
• Copy Paste: [[2501.02020]] Enhancing Uncertainty Modeling with Semantic Graph for Hallucination Detection(https://arxiv.org/abs/2501.02020)
• Keywords: language model, llm, hallucination
• Abstract: Large Language Models (LLMs) are prone to hallucination with non-factual or unfaithful statements, which undermines the applications in real-world scenarios. Recent researches focus on uncertainty-based hallucination detection, which utilizes the output probability of LLMs for uncertainty calculation and does not rely on external knowledge or frequent sampling from LLMs. Whereas, most approaches merely consider the uncertainty of each independent token, while the intricate semantic relations among tokens and sentences are not well studied, which limits the detection of hallucination that spans over multiple tokens and sentences in the passage. In this paper, we propose a method to enhance uncertainty modeling with semantic graph for hallucination detection. Specifically, we first construct a semantic graph that well captures the relations among entity tokens and sentences. Then, we incorporate the relations between two entities for uncertainty propagation to enhance sentence-level hallucination detection. Given that hallucination occurs due to the conflict between sentences, we further present a graph-based uncertainty calibration method that integrates the contradiction probability of the sentence with its neighbors in the semantic graph for uncertainty calculation. Extensive experiments on two datasets show the great advantages of our proposed approach. In particular, we obtain substantial improvements with 19.78% in passage-level hallucination detection.
• 摘要：大型语言模型 (LLM) 容易因非事实或不真实的陈述而产生幻觉，这会破坏其在实际场景中的应用。最近的研究集中于基于不确定性的幻觉检测，它利用 LLM 的输出概率进行不确定性计算，而不依赖于外部知识或来自 LLM 的频繁采样。然而，大多数方法仅考虑每个独立 token 的不确定性，而 token 和句子之间复杂的语义关系尚未得到很好的研究，这限制了对跨越段落中多个 token 和句子的幻觉的检测。在本文中，我们提出了一种使用语义图增强不确定性建模以进行幻觉检测的方法。具体而言，我们首先构建一个可以很好地捕捉实体 token 和句子之间关系的语义图。然后，我们结合两个实体之间的关系进行不确定性传播，以增强句子级幻觉检测。考虑到幻觉是由于句子之间的冲突而发生的，我们进一步提出了一种基于图的不确定性校准方法，该方法将句子与其在语义图中的邻居的矛盾概率结合起来进行不确定性计算。在两个数据集上进行的大量实验表明，我们提出的方法具有巨大优势。特别是在段落级幻觉检测方面，我们取得了 19.78% 的显著改进。

Title: Recursive Decomposition of Logical Thoughts: Framework for Superior Reasoning and Knowledge Propagation in Large Language Models

• Authors: Kaleem Ullah Qasim, Jiashu Zhang, Tariq Alsahfi, Ateeq Ur Rehman Butt
• Subjects: cs.CL, cs.AI, cs.LG, cs.LO
• Abstract URL: https://arxiv.org/abs/2501.02026
• Pdf URL: https://arxiv.org/pdf/2501.02026
• Copy Paste: [[2501.02026]] Recursive Decomposition of Logical Thoughts: Framework for Superior Reasoning and Knowledge Propagation in Large Language Models(https://arxiv.org/abs/2501.02026)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: Enhancing the reasoning capabilities of Large Language Models remains a critical challenge in artificial intelligence. We introduce RDoLT, Recursive Decomposition of Logical Thought prompting, a novel framework that significantly boosts LLM reasoning performance. RDoLT is built on three key innovations: (1) recursively breaking down complex reasoning tasks into sub-tasks of progressive complexity; (2) employing an advanced selection and scoring mechanism to identify the most promising reasoning thoughts; and (3) integrating a knowledge propagation module that mimics human learning by keeping track of strong and weak thoughts for information propagation. Our approach was evaluated across multiple benchmarks, including GSM8K, SVAMP, MultiArith, LastLetterConcatenation, and Gaokao2023 Math. The results demonstrate that RDoLT consistently outperforms existing state-of-the-art techniques, achieving a 90.98 percent accuracy on GSM8K with ChatGPT-4, surpassing state-of-the-art techniques by 6.28 percent. Similar improvements were observed on other benchmarks, with accuracy gains ranging from 5.5 percent to 6.75 percent. These findings highlight RDoLT's potential to advance prompt engineering, offering a more effective and generalizable approach to complex reasoning tasks.
• 摘要：增强大型语言模型的推理能力仍然是人工智能领域的一大挑战。我们引入了 RDoLT（逻辑思维提示的递归分解），这是一个可显著提高 LLM 推理性能的新颖框架。RDoLT 基于三项关键创新：（1）将复杂的推理任务递归分解为复杂程度逐渐提高的子任务；（2）采用先进的选择和评分机制来识别最有前途的推理思维；（3）集成知识传播模块，通过跟踪强弱思维来模仿人类学习以进行信息传播。我们的方法在多个基准上进行了评估，包括 GSM8K、SVAMP、MultiArith、LastLetterConcatenation 和 Gaokao2023 Math。结果表明，RDoLT 始终优于现有的最先进技术，使用 ChatGPT-4 在 GSM8K 上实现了 90.98% 的准确率，比最先进技术高出 6.28%。在其他基准测试中也观察到了类似的改进，准确率提高了 5.5% 到 6.75%。这些发现凸显了 RDoLT 推动快速工程的潜力，为复杂的推理任务提供了更有效、更通用的方法。

Title: CarbonChat: Large Language Model-Based Corporate Carbon Emission Analysis and Climate Knowledge Q&A System

• Authors: Zhixuan Cao, Ming Han, Jingtao Wang, Meng Jia
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02031
• Pdf URL: https://arxiv.org/pdf/2501.02031
• Copy Paste: [[2501.02031]] CarbonChat: Large Language Model-Based Corporate Carbon Emission Analysis and Climate Knowledge Q&A System(https://arxiv.org/abs/2501.02031)
• Keywords: language model, hallucination, prompt, chat, retrieval-augmented generation
• Abstract: As the impact of global climate change intensifies, corporate carbon emissions have become a focal point of global attention. In response to issues such as the lag in climate change knowledge updates within large language models, the lack of specialization and accuracy in traditional augmented generation architectures for complex problems, and the high cost and time consumption of sustainability report analysis, this paper proposes CarbonChat: Large Language Model-based corporate carbon emission analysis and climate knowledge Q&A system, aimed at achieving precise carbon emission analysis and policy this http URL, a diversified index module construction method is proposed to handle the segmentation of rule-based and long-text documents, as well as the extraction of structured data, thereby optimizing the parsing of key this http URL, an enhanced self-prompt retrieval-augmented generation architecture is designed, integrating intent recognition, structured reasoning chains, hybrid retrieval, and Text2SQL, improving the efficiency of semantic understanding and query this http URL, based on the greenhouse gas accounting framework, 14 dimensions are established for carbon emission analysis, enabling report summarization, relevance evaluation, and customized this http URL, through a multi-layer chunking mechanism, timestamps, and hallucination detection features, the accuracy and verifiability of the analysis results are ensured, reducing hallucination rates and enhancing the precision of the responses.
• 摘要：随着全球气候变化影响不断加剧，企业碳排放成为全球关注的焦点。针对大型语言模型中气候变化知识更新滞后、传统增强生成架构对复杂问题缺乏专业化和准确性、可持续发展报告分析成本和时间消耗高等问题，本文提出了CarbonChat：基于大型语言模型的企业碳排放分析与气候知识问答系统，旨在实现精准的碳排放分析与政策制定，提出多样化的索引模块构建方法，处理基于规则和长文本的分词，以及结构化数据的提取，从而优化关键字段的解析，设计增强型自提示检索增强生成架构，融合意图识别、结构化推理链、混合检索和Text2SQL，提升语义理解和查询效率，基于温室气体核算框架，建立碳排放分析的14个维度，通过多层分块机制、时间戳和幻觉检测特征，实现报告汇总、相关性评估和自定义，保证了分析结果的准确性和可验证性，降低了幻觉发生率，提高了回答的准确性。

Title: An Investigation into Value Misalignment in LLM-Generated Texts for Cultural Heritage

• Authors: Fan Bu, Zheng Wang, Siyi Wang, Ziyao Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02039
• Pdf URL: https://arxiv.org/pdf/2501.02039
• Copy Paste: [[2501.02039]] An Investigation into Value Misalignment in LLM-Generated Texts for Cultural Heritage(https://arxiv.org/abs/2501.02039)
• Keywords: language model, llm
• Abstract: As Large Language Models (LLMs) become increasingly prevalent in tasks related to cultural heritage, such as generating descriptions of historical monuments, translating ancient texts, preserving oral traditions, and creating educational content, their ability to produce accurate and culturally aligned texts is being increasingly relied upon by users and researchers. However, cultural value misalignments may exist in generated texts, such as the misrepresentation of historical facts, the erosion of cultural identity, and the oversimplification of complex cultural narratives, which may lead to severe consequences. Therefore, investigating value misalignment in the context of LLM for cultural heritage is crucial for mitigating these risks, yet there has been a significant lack of systematic and comprehensive study and investigation in this area. To fill this gap, we systematically assess the reliability of LLMs in generating culturally aligned texts for cultural heritage-related tasks. We conduct a comprehensive evaluation by compiling an extensive set of 1066 query tasks covering 5 widely recognized categories with 17 aspects within the knowledge framework of cultural heritage across 5 open-source LLMs, and examine both the type and rate of cultural value misalignments in the generated texts. Using both automated and manual approaches, we effectively detect and analyze the cultural value misalignments in LLM-generated texts. Our findings are concerning: over 65% of the generated texts exhibit notable cultural misalignments, with certain tasks demonstrating almost complete misalignment with key cultural values. Beyond these findings, this paper introduces a benchmark dataset and a comprehensive evaluation workflow that can serve as a valuable resource for future research aimed at enhancing the cultural sensitivity and reliability of LLMs.
• 摘要：随着大型语言模型 (LLM) 在与文化遗产相关的任务中越来越普遍，例如生成历史古迹描述、翻译古代文献、保存口头传统和创建教育内容，用户和研究人员越来越依赖它们生成准确且符合文化的文本的能力。然而，生成的文本中可能存在文化价值观错位，例如对历史事实的错误表述、文化认同的侵蚀以及对复杂文化叙事的过度简化，这可能会导致严重后果。因此，调查文化遗产 LLM 背景下的价值观错位对于减轻这些风险至关重要，但该领域缺乏系统而全面的研究和调查。为了填补这一空白，我们系统地评估了 LLM 在为文化遗产相关任务生成文化一致文本方面的可靠性。我们通过编制 5 个开源 LLM 中的 1066 个查询任务集（涵盖文化遗产知识框架中 5 个被广泛认可的类别和 17 个方面）进行了全面评估，并检查了生成的文本中文化价值观错位的类型和比率。使用自动和手动方法，我们有效地检测和分析了 LLM 生成的文本中的文化价值观错位。我们的发现令人担忧：超过 65% 的生成文本表现出明显的文化错位，某些任务几乎完全显示出与关键文化价值观的错位。除了这些发现之外，本文还介绍了一个基准数据集和一个全面的评估工作流程，它们可以作为未来研究的宝贵资源，旨在提高 LLM 的文化敏感性和可靠性。

Title: AGGA: A Dataset of Academic Guidelines for Generative AI and Large Language Models

• Authors: Junfeng Jiao, Saleh Afroogh, Kevin Chen, David Atkinson, Amit Dhurandhar
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.02063
• Pdf URL: https://arxiv.org/pdf/2501.02063
• Copy Paste: [[2501.02063]] AGGA: A Dataset of Academic Guidelines for Generative AI and Large Language Models(https://arxiv.org/abs/2501.02063)
• Keywords: language model, llm
• Abstract: This study introduces AGGA, a dataset comprising 80 academic guidelines for the use of Generative AIs (GAIs) and Large Language Models (LLMs) in academic settings, meticulously collected from official university websites. The dataset contains 188,674 words and serves as a valuable resource for natural language processing tasks commonly applied in requirements engineering, such as model synthesis, abstraction identification, and document structure assessment. Additionally, AGGA can be further annotated to function as a benchmark for various tasks, including ambiguity detection, requirements categorization, and the identification of equivalent requirements. Our methodologically rigorous approach ensured a thorough examination, with a selection of universities that represent a diverse range of global institutions, including top-ranked universities across six continents. The dataset captures perspectives from a variety of academic fields, including humanities, technology, and both public and private institutions, offering a broad spectrum of insights into the integration of GAIs and LLMs in academia.
• 摘要：本研究引入了 AGGA，这是一个包含 80 条学术指南的数据集，这些指南用于在学术环境中使用生成式人工智能 (GAI) 和大型语言模型 (LLM)，这些指南是从大学官方网站精心收集的。该数据集包含 188,674 个单词，是需求工程中常用的自然语言处理任务的宝贵资源，例如模型合成、抽象识别和文档结构评估。此外，AGGA 可以进一步注释，作为各种任务的基准，包括歧义检测、需求分类和等效需求识别。我们的方法严谨，确保了彻底的审查，所选大学代表了全球各种机构，包括六大洲的顶尖大学。该数据集涵盖了人文、技术以及公立和私立机构等各种学术领域的观点，为 GAI 和 LLM 在学术界的整合提供了广泛的见解。

Title: The interplay between domain specialization and model size: a case study in the legal domain

• Authors: Roseval Malaquias Junior, Ramon Pires, Thales Sales Almeida, Kenzo Sakiyama, Roseli Romero, Rodrigo Nogueira
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02068
• Pdf URL: https://arxiv.org/pdf/2501.02068
• Copy Paste: [[2501.02068]] The interplay between domain specialization and model size: a case study in the legal domain(https://arxiv.org/abs/2501.02068)
• Keywords: language model
• Abstract: Scaling laws for language models so far focused on finding the compute-optimal model size and token count for training from scratch. However, achieving this optimal balance requires significant compute resources due to the extensive data demands when training models from randomly-initialized weights. Continual pre-training offers a cost-effective alternative, leveraging the compute investment from pre-trained models to incorporate new knowledge without requiring extensive new data. Recent findings suggest that data quality influences constants in scaling laws, thereby altering the optimal parameter-token allocation ratio. Building on this insight, we investigate the interplay between domain specialization and model size during continual pre-training under compute-constrained scenarios. Our goal is to identify a compute-efficient training regime for this scenario and, potentially, detect patterns in this interplay that can be generalized across different model sizes and domains. To compare general and specialized training, we filtered a web-based dataset to extract legal domain data. We pre-trained models with 1.5B, 3B, 7B and 14B parameters on both the unfiltered and filtered datasets, then evaluated their performance on legal exams. Results show that as model size increases, the compute-effectiveness gap between specialized and general models widens.
• 摘要：到目前为止，语言模型的缩放定律主要集中在从头开始寻找计算最优的模型大小和标记计数。但是，由于从随机初始化的权重训练模型时需要大量数据，因此实现这种最佳平衡需要大量的计算资源。持续预训练提供了一种经济高效的替代方案，利用预训练模型的计算投资来整合新知识，而无需大量新数据。最近的研究结果表明，数据质量会影响缩放定律中的常数，从而改变最佳参数标记分配率。基于这一见解，我们研究了在计算受限场景下持续预训练期间领域专业化和模型大小之间的相互作用。我们的目标是为这种场景确定一种计算效率高的训练方案，并可能检测出这种相互作用中的模式，这些模式可以推广到不同的模型大小和领域。为了比较一般和专门的训练，我们过滤了一个基于网络的数据集以提取合法领域的数据。我们在未过滤和过滤后的数据集上分别对具有 1.5B、3B、7B 和 14B 个参数的模型进行了预训练，然后评估了它们在法律考试中的表现。结果表明，随着模型规模的增加，专用模型和通用模型之间的计算效率差距越来越大。

Title: Instruction-Following Pruning for Large Language Models

• Authors: Bairu Hou, Qibin Chen, Jianyu Wang, Guoli Yin, Chong Wang, Nan Du, Ruoming Pang, Shiyu Chang, Tao Lei
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02086
• Pdf URL: https://arxiv.org/pdf/2501.02086
• Copy Paste: [[2501.02086]] Instruction-Following Pruning for Large Language Models(https://arxiv.org/abs/2501.02086)
• Keywords: language model, llm
• Abstract: With the rapid scaling of large language models (LLMs), structured pruning has become a widely used technique to learn efficient, smaller models from larger ones, delivering superior performance compared to training similarly sized models from scratch. In this paper, we move beyond the traditional static pruning approach of determining a fixed pruning mask for a model, and propose a dynamic approach to structured pruning. In our method, the pruning mask is input-dependent and adapts dynamically based on the information described in a user instruction. Our approach, termed "instruction-following pruning", introduces a sparse mask predictor that takes the user instruction as input and dynamically selects the most relevant model parameters for the given task. To identify and activate effective parameters, we jointly optimize the sparse mask predictor and the LLM, leveraging both instruction-following data and the pre-training corpus. Experimental results demonstrate the effectiveness of our approach on a wide range of evaluation benchmarks. For example, our 3B activated model improves over the 3B dense model by 5-8 points of absolute margin on domains such as math and coding, and rivals the performance of a 9B model.
• 摘要：随着大型语言模型 (LLM) 的快速扩展，结构化剪枝已成为一种广泛使用的技术，用于从大型模型中学习高效、较小的模型，与从头开始训练类似大小的模型相比，其性能更出色。在本文中，我们超越了传统的静态剪枝方法（即为模型确定固定的剪枝掩码），并提出了一种动态的结构化剪枝方法。在我们的方法中，剪枝掩码依赖于输入，并根据用户指令中描述的信息进行动态调整。我们的方法称为“指令跟踪剪枝”，它引入了一个稀疏掩码预测器，它将用户指令作为输入并动态选择与给定任务最相关的模型参数。为了识别和激活有效参数，我们联合优化了稀疏掩码预测器和 LLM，同时利用指令跟踪数据和预训练语料库。实验结果证明了我们的方法在各种评估基准上的有效性。例如，我们的 3B 激活模型在数学和编码等领域比 3B 密集模型提高了 5-8 个点的绝对幅度，并可与 9B 模型的性能相媲美。

Title: Personalized Graph-Based Retrieval for Large Language Models

• Authors: Steven Au, Cameron J. Dimacali, Ojasmitha Pedirappagari, Namyong Park, Franck Dernoncourt, Yu Wang, Nikos Kanakaris, Hanieh Deilamsalehy, Ryan A. Rossi, Nesreen K. Ahmed
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02157
• Pdf URL: https://arxiv.org/pdf/2501.02157
• Copy Paste: [[2501.02157]] Personalized Graph-Based Retrieval for Large Language Models(https://arxiv.org/abs/2501.02157)
• Keywords: language model, llm, prompt, retrieval-augmented generation
• Abstract: As large language models (LLMs) evolve, their ability to deliver personalized and context-aware responses offers transformative potential for improving user experiences. Existing personalization approaches, however, often rely solely on user history to augment the prompt, limiting their effectiveness in generating tailored outputs, especially in cold-start scenarios with sparse data. To address these limitations, we propose Personalized Graph-based Retrieval-Augmented Generation (PGraphRAG), a framework that leverages user-centric knowledge graphs to enrich personalization. By directly integrating structured user knowledge into the retrieval process and augmenting prompts with user-relevant context, PGraphRAG enhances contextual understanding and output quality. We also introduce the Personalized Graph-based Benchmark for Text Generation, designed to evaluate personalized text generation tasks in real-world settings where user history is sparse or unavailable. Experimental results show that PGraphRAG significantly outperforms state-of-the-art personalization methods across diverse tasks, demonstrating the unique advantages of graph-based retrieval for personalization.
• 摘要：随着大型语言模型 (LLM) 的发展，它们提供个性化和情境感知响应的能力为改善用户体验提供了变革性的潜力。然而，现有的个性化方法通常仅依靠用户历史记录来增强提示，这限制了它们在生成定制输出方面的有效性，尤其是在数据稀疏的冷启动场景中。为了解决这些限制，我们提出了基于个性化图谱的检索增强生成 (PGraphRAG)，这是一个利用以用户为中心的知识图谱来丰富个性化的框架。通过将结构化用户知识直接集成到检索过程中并使用与用户相关的上下文来增强提示，PGraphRAG 增强了上下文理解和输出质量。我们还引入了基于个性化图谱的文本生成基准，旨在评估用户历史记录稀疏或不可用的现实环境中的个性化文本生成任务。实验结果表明，PGraphRAG 在各种任务中的表现明显优于最先进的个性化方法，展示了基于图谱的检索在个性化方面的独特优势。

Title: CPTuning: Contrastive Prompt Tuning for Generative Relation Extraction

• Authors: Jiaxin Duan, Fengyu Lu, Junfei Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02196
• Pdf URL: https://arxiv.org/pdf/2501.02196
• Copy Paste: [[2501.02196]] CPTuning: Contrastive Prompt Tuning for Generative Relation Extraction(https://arxiv.org/abs/2501.02196)
• Keywords: language model, prompt
• Abstract: Generative relation extraction (RE) commonly involves first reformulating RE as a linguistic modeling problem easily tackled with pre-trained language models (PLM) and then fine-tuning a PLM with supervised cross-entropy loss. Although having achieved promising performance, existing approaches assume only one deterministic relation between each pair of entities without considering real scenarios where multiple relations may be valid, i.e., entity pair overlap, causing their limited applications. To address this problem, we introduce a novel contrastive prompt tuning method for RE, CPTuning, which learns to associate a candidate relation between two in-context entities with a probability mass above or below a threshold, corresponding to whether the relation exists. Beyond learning schema, CPTuning also organizes RE as a verbalized relation generation task and uses Trie-constrained decoding to ensure a model generates valid relations. It adaptively picks out the generated candidate relations with a high estimated likelihood in inference, thereby achieving multi-relation extraction. We conduct extensive experiments on four widely used datasets to validate our method. Results show that T5-large fine-tuned with CPTuning significantly outperforms previous methods, regardless of single or multiple relations extraction.
• 摘要：生成关系提取 (RE) 通常涉及首先将 RE 重新表述为语言建模问题，该问题可通过预训练语言模型 (PLM) 轻松解决，然后使用监督交叉熵损失对 PLM 进行微调。尽管现有方法取得了令人鼓舞的性能，但它们仅假设每对实体之间存在一种确定性关系，而没有考虑多个关系可能有效的真实场景，即实体对重叠，这导致其应用受限。为了解决这个问题，我们引入了一种新颖的 RE 对比快速调整方法 CPTuning，它学习将两个上下文实体之间的候选关系与高于或低于阈值的概率质量相关联，对应于关系是否存在。除了学习模式之外，CPTuning 还将 RE 组织为语言化的关系生成任务，并使用 Trie 约束解码来确保模型生成有效的关系。它自适应地挑选出在推理中具有高估计可能性的生成候选关系，从而实现多关系提取。我们在四个广泛使用的数据集上进行了广泛的实验来验证我们的方法。结果表明，无论是单个还是多个关系提取，使用 CPTuning 微调的 T5-large 都明显优于以前的方法。

Title: Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trends

• Authors: Camille Barboule, Benjamin Piwowarski, Yoan Chabot
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02235
• Pdf URL: https://arxiv.org/pdf/2501.02235
• Copy Paste: [[2501.02235]] Survey on Question Answering over Visually Rich Documents: Methods, Challenges, and Trends(https://arxiv.org/abs/2501.02235)
• Keywords: language model, llm
• Abstract: Using Large Language Models (LLMs) for Visually-rich Document Understanding (VrDU) has significantly improved performance on tasks requiring both comprehension and generation, such as question answering, albeit introducing new challenges. This survey explains how VrDU models enhanced by LLMs function, covering methods for integrating VrD features into LLMs and highlighting key challenges.
• 摘要：使用大型语言模型 (LLM) 进行视觉丰富文档理解 (VrDU) 显著提高了需要理解和生成的任务（例如问答）的性能，尽管这带来了新的挑战。本调查介绍了通过 LLM 增强的 VrDU 模型如何发挥作用，涵盖了将 VrD 功能集成到 LLM 中的方法并强调了关键挑战。

Title: Financial Named Entity Recognition: How Far Can LLM Go?

• Authors: Yi-Te Lu, Yintong Huo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02237
• Pdf URL: https://arxiv.org/pdf/2501.02237
• Copy Paste: [[2501.02237]] Financial Named Entity Recognition: How Far Can LLM Go?(https://arxiv.org/abs/2501.02237)
• Keywords: language model, llm, prompt
• Abstract: The surge of large language models (LLMs) has revolutionized the extraction and analysis of crucial information from a growing volume of financial statements, announcements, and business news. Recognition for named entities to construct structured data poses a significant challenge in analyzing financial documents and is a foundational task for intelligent financial analytics. However, how effective are these generic LLMs and their performance under various prompts are yet need a better understanding. To fill in the blank, we present a systematic evaluation of state-of-the-art LLMs and prompting methods in the financial Named Entity Recognition (NER) problem. Specifically, our experimental results highlight their strengths and limitations, identify five representative failure types, and provide insights into their potential and challenges for domain-specific tasks.
• 摘要：大型语言模型 (LLM) 的激增彻底改变了从越来越多的财务报表、公告和商业新闻中提取和分析关键信息的方式。识别命名实体以构建结构化数据是分析财务文档的重大挑战，也是智能财务分析的基础任务。然而，这些通用 LLM 的有效性以及它们在各种提示下的表现如何仍需要更好的了解。为了填补这一空白，我们对金融命名实体识别 (NER) 问题中最先进的 LLM 和提示方法进行了系统评估。具体来说，我们的实验结果突出了它们的优势和局限性，确定了五种有代表性的失败类型，并深入了解了它们在特定领域任务中的潜力和挑战。

Title: LLMzSz{\L}: a comprehensive LLM benchmark for Polish

• Authors: Krzysztof Jassem, Michał Ciesiółka, Filip Graliński, Piotr Jabłoński, Jakub Pokrywka, Marek Kubis, Monika Jabłońska, Ryszard Staruch
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02266
• Pdf URL: https://arxiv.org/pdf/2501.02266
• Copy Paste: [[2501.02266]] LLMzSz{\L}: a comprehensive LLM benchmark for Polish(https://arxiv.org/abs/2501.02266)
• Keywords: llm
• Abstract: This article introduces the first comprehensive benchmark for the Polish language at this scale: LLMzSzŁ (LLMs Behind the School Desk). It is based on a coherent collection of Polish national exams, including both academic and professional tests extracted from the archives of the Polish Central Examination Board. It covers 4 types of exams, coming from 154 domains. Altogether, it consists of almost 19k closed-ended questions. We investigate the performance of open-source multilingual, English, and Polish LLMs to verify LLMs' abilities to transfer knowledge between languages. Also, the correlation between LLMs and humans at model accuracy and exam pass rate levels is examined. We show that multilingual LLMs can obtain superior results over monolingual ones; however, monolingual models may be beneficial when model size matters. Our analysis highlights the potential of LLMs in assisting with exam validation, particularly in identifying anomalies or errors in examination tasks.
• 摘要：本文介绍了波兰语首个如此规模的综合基准：LLMzSzŁ（LLMs Behind the School Desk）。它基于波兰国家考试的连贯集合，包括从波兰中央考试委员会档案中提取的学术和专业考试。它涵盖 4 种类型的考试，来自 154 个领域。总共包含近 19,000 个封闭式问题。我们调查了开源多语言、英语和波兰语 LLM 的性能，以验证 LLM 在语言之间传递知识的能力。此外，我们还研究了 LLM 与人类在模型准确率和考试通过率水平上的相关性。我们表明，多语言 LLM 可以获得比单语 LLM 更好的结果；然而，当模型大小很重要时，单语模型可能会有所帮助。我们的分析强调了 LLM 在协助考试验证方面的潜力，特别是在识别考试任务中的异常或错误方面。

Title: Explicit vs. Implicit: Investigating Social Bias in Large Language Models through Self-Reflection

• Authors: Yachao Zhao, Bo Wang, Yan Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02295
• Pdf URL: https://arxiv.org/pdf/2501.02295
• Copy Paste: [[2501.02295]] Explicit vs. Implicit: Investigating Social Bias in Large Language Models through Self-Reflection(https://arxiv.org/abs/2501.02295)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have been shown to exhibit various biases and stereotypes in their generated content. While extensive research has investigated bias in LLMs, prior work has predominantly focused on explicit bias, leaving the more nuanced implicit biases largely unexplored. This paper presents a systematic framework grounded in social psychology theories to investigate and compare explicit and implicit biases in LLMs. We propose a novel "self-reflection" based evaluation framework that operates in two phases: first measuring implicit bias through simulated psychological assessment methods, then evaluating explicit bias by prompting LLMs to analyze their own generated content. Through extensive experiments on state-of-the-art LLMs across multiple social dimensions, we demonstrate that LLMs exhibit a substantial inconsistency between explicit and implicit biases, where explicit biases manifest as mild stereotypes while implicit biases show strong stereotypes. Furthermore, we investigate the underlying factors contributing to this explicit-implicit bias inconsistency. Our experiments examine the effects of training data scale, model parameters, and alignment techniques. Results indicate that while explicit bias diminishes with increased training data and model size, implicit bias exhibits a contrasting upward trend. Notably, contemporary alignment methods (e.g., RLHF, DPO) effectively suppress explicit bias but show limited efficacy in mitigating implicit bias. These findings suggest that while scaling up models and alignment training can address explicit bias, the challenge of implicit bias requires novel approaches beyond current methodologies.
• 摘要：大型语言模型 (LLM) 已被证明在其生成的内容中表现出各种偏见和刻板印象。虽然大量研究已经调查了 LLM 中的偏见，但先前的研究主要集中在显性偏见上，而更细微的隐性偏见则基本未被探索。本文提出了一个基于社会心理学理论的系统框架，用于研究和比较 LLM 中的显性和隐性偏见。我们提出了一种基于“自我反思”的新型评估框架，该框架分两个阶段运行：首先通过模拟心理评估方法测量隐性偏见，然后通过提示 LLM 分析自己生成的内容来评估显性偏见。通过对多个社会维度的最先进的 LLM 进行大量实验，我们证明 LLM 在显性和隐性偏见之间表现出相当大的不一致，其中显性偏见表现为轻微的刻板印象，而隐性偏见则表现出强烈的刻板印象。此外，我们研究了导致这种显性-隐性偏见不一致的潜在因素。我们的实验检查了训练数据规模、模型参数和对齐技术的影响。结果表明，虽然显性偏见随着训练数据和模型规模的增加而减少，但隐性偏见却呈现出相反的上升趋势。值得注意的是，当代对齐方法（例如 RLHF、DPO）有效地抑制了显性偏见，但在减轻隐性偏见方面效果有限。这些发现表明，虽然扩大模型和对齐训练可以解决显性偏见，但隐性偏见的挑战需要超越当前方法的新方法。

Title: Validity Arguments For Constructed Response Scoring Using Generative Artificial Intelligence Applications

• Authors: Jodi M. Casabianca, Daniel F. McCaffrey, Matthew S. Johnson, Naim Alper, Vladimir Zubenko
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.02334
• Pdf URL: https://arxiv.org/pdf/2501.02334
• Copy Paste: [[2501.02334]] Validity Arguments For Constructed Response Scoring Using Generative Artificial Intelligence Applications(https://arxiv.org/abs/2501.02334)
• Keywords: language model
• Abstract: The rapid advancements in large language models and generative artificial intelligence (AI) capabilities are making their broad application in the high-stakes testing context more likely. Use of generative AI in the scoring of constructed responses is particularly appealing because it reduces the effort required for handcrafting features in traditional AI scoring and might even outperform those methods. The purpose of this paper is to highlight the differences in the feature-based and generative AI applications in constructed response scoring systems and propose a set of best practices for the collection of validity evidence to support the use and interpretation of constructed response scores from scoring systems using generative AI. We compare the validity evidence needed in scoring systems using human ratings, feature-based natural language processing AI scoring engines, and generative AI. The evidence needed in the generative AI context is more extensive than in the feature-based NLP scoring context because of the lack of transparency and other concerns unique to generative AI such as consistency. Constructed response score data from standardized tests demonstrate the collection of validity evidence for different types of scoring systems and highlights the numerous complexities and considerations when making a validity argument for these scores. In addition, we discuss how the evaluation of AI scores might include a consideration of how a contributory scoring approach combining multiple AI scores (from different sources) will cover more of the construct in the absence of human ratings.
• 摘要：大型语言模型和生成式人工智能 (AI) 能力的快速发展使得它们更有可能在高风险测试环境中得到广泛应用。在构造反应评分中使用生成式人工智能尤其具有吸引力，因为它减少了在传统人工智能评分中手工制作特征所需的工作量，甚至可能优于这些方法。本文的目的是强调基于特征和生成式人工智能在构造反应评分系统中的应用差异，并提出一套收集有效性证据的最佳实践，以支持使用生成式人工智能评分系统构造反应分数的使用和解释。我们比较了使用人工评分、基于特征的自然语言处理人工智能评分引擎和生成式人工智能的评分系统所需的有效性证据。由于缺乏透明度和生成式人工智能特有的其他问题（例如一致性），生成式人工智能环境中所需的证据比基于特征的 NLP 评分环境中所需的证据更广泛。标准化测试的构造反应分数数据展示了不同类型评分系统的有效性证据的收集，并强调了在为这些分数进行有效性论证时的许多复杂性和注意事项。此外，我们讨论了人工智能分数的评估可能包括考虑如何在没有人工评分的情况下，结合多个人工智能分数（来自不同来源）的贡献评分方法覆盖更多的构造。

Title: AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inference

• Authors: Zhuomin He, Yizhen Yao, Pengfei Zuo, Bin Gao, Qinya Li, Zhenzhe Zheng, Fan Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02336
• Pdf URL: https://arxiv.org/pdf/2501.02336
• Copy Paste: [[2501.02336]] AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inference(https://arxiv.org/abs/2501.02336)
• Keywords: language model, llm
• Abstract: Long-context large language models (LLMs) inference is increasingly critical, motivating a number of studies devoted to alleviating the substantial storage and computational costs in such scenarios. Layer-wise skipping methods are promising optimizations but rarely explored in long-context inference. We observe that existing layer-wise skipping strategies have several limitations when applied in long-context inference, including the inability to adapt to model and context variability, disregard for sublayer significance, and inapplicability for the prefilling phase. This paper proposes \sysname, an adaptive sublayer skipping method specifically designed for long-context inference. \sysname adaptively identifies less important layers by leveraging on-the-fly similarity information, enables sublayer-wise skipping, and accelerates both the prefilling and decoding phases. The effectiveness of \sysname is demonstrated through extensive experiments on various long-context benchmarks and models, showcasing its superior inference performance over existing baselines.
• 摘要：长上下文大型语言模型 (LLM) 推理越来越重要，这促使许多研究致力于减轻此类场景中大量的存储和计算成本。分层跳过方法是有前途的优化方法，但在长上下文推理中很少被探索。我们观察到，现有的分层跳过策略在应用于长上下文推理时有几个局限性，包括无法适应模型和上下文的变化、忽视子层的重要性以及不适用于预填充阶段。本文提出了 \sysname，一种专为长上下文推理设计的自适应子层跳过方法。 \sysname 通过利用动态相似性信息自适应地识别不太重要的层，实现子层跳过，并加速预填充和解码阶段。通过在各种长上下文基准和模型上进行的大量实验证明了 \sysname 的有效性，展示了其优于现有基线的推理性能。

Title: Thinking with Many Minds: Using Large Language Models for Multi-Perspective Problem-Solving

• Authors: Sanghyun Park, Boris Maciejovsky, Phanish Puranam
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2501.02348
• Pdf URL: https://arxiv.org/pdf/2501.02348
• Copy Paste: [[2501.02348]] Thinking with Many Minds: Using Large Language Models for Multi-Perspective Problem-Solving(https://arxiv.org/abs/2501.02348)
• Keywords: language model, gpt, llm, agent
• Abstract: Complex problem-solving requires cognitive flexibility--the capacity to entertain multiple perspectives while preserving their distinctiveness. This flexibility replicates the "wisdom of crowds" within a single individual, allowing them to "think with many minds." While mental simulation enables imagined deliberation, cognitive constraints limit its effectiveness. We propose synthetic deliberation, a Large Language Model (LLM)-based method that simulates discourse between agents embodying diverse perspectives, as a solution. Using a custom GPT-based model, we showcase its benefits: concurrent processing of multiple viewpoints without cognitive degradation, parallel exploration of perspectives, and precise control over viewpoint synthesis. By externalizing the deliberative process and distributing cognitive labor between parallel search and integration, synthetic deliberation transcends mental simulation's limitations. This approach shows promise for strategic planning, policymaking, and conflict resolution.
• 摘要：解决复杂问题需要认知灵活性——即在保留多种观点的同时兼顾多种观点的能力。这种灵活性在单个个体中复制了“群体智慧”，使他们能够“用多种思维思考”。虽然心理模拟可以实现想象中的审议，但认知限制限制了其有效性。我们提出了一种解决方案，即综合审议，这是一种基于大型语言模型 (LLM) 的方法，可以模拟体现不同观点的代理之间的对话。使用基于 GPT 的自定义模型，我们展示了它的好处：同时处理多个观点而不会降低认知能力，并行探索观点，以及对观点综合的精确控制。通过将审议过程外部化并在并行搜索和集成之间分配认知劳动，综合审议超越了心理模拟的局限性。这种方法在战略规划、政策制定和冲突解决方面大有可为。

Title: Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison

• Authors: Tsz Kin Lam, Marco Gaido, Sara Papi, Luisa Bentivogli, Barry Haddow
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.02370
• Pdf URL: https://arxiv.org/pdf/2501.02370
• Copy Paste: [[2501.02370]] Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison(https://arxiv.org/abs/2501.02370)
• Keywords: language model, llm
• Abstract: Following the remarkable success of Large Language Models (LLMs) in NLP tasks, there is increasing interest in extending their capabilities to speech -- the most common form in communication. To integrate speech into LLMs, one promising approach is dense feature prepending (DFP) which prepends the projected speech representations to the textual representations, allowing end-to-end training with the speech encoder. However, DFP typically requires connecting a text decoder to a speech encoder. This raises questions about the importance of having a sophisticated speech encoder for DFP, and how its performance compares with a standard encoder-decoder (i.e. cross-attention) architecture. In order to perform a controlled architectural comparison, we train all models from scratch, rather than using large pretrained models, and use comparable data and parameter settings, testing speech-to-text recognition (ASR) and translation (ST) on MuST-C v1.0 and CoVoST2 datasets. We study the influence of a speech encoder in DFP. More importantly, we compare DFP and cross-attention under a variety of configurations, such as CTC compression, sequence-level knowledge distillation, generation speed and GPU memory footprint on monolingual, bilingual and multilingual models. Despite the prevalence of DFP over cross-attention, our overall results do not indicate a clear advantage of DFP.
• 摘要：随着大型语言模型 (LLM) 在 NLP 任务中取得显著成功，人们越来越有兴趣将其功能扩展到语音——这是最常见的通信形式。为了将语音集成到 LLM 中，一种有前途的方法是密集特征前置 (DFP)，它将投影的语音表示前置到​​文本表示，从而允许使用语音编码器进行端到端训练。但是，DFP 通常需要将文本解码器连接到语音编码器。这引发了关于为 DFP 配备复杂的语音编码器的重要性的问题，以及其性能与标准编码器-解码器（即交叉注意）架构相比如何。为了进行受控的架构比较，我们从头开始训练所有模型，而不是使用大型预训练模型，并使用可比较的数据和参数设置，在 MuST-C v1.0 和 CoVoST2 数据集上测试语音到文本识别 (ASR) 和翻译 (ST)。我们研究语音编码器对 DFP 的影响。更重要的是，我们在单语、双语和多语模型上比较了各种配置下的 DFP 和交叉注意力，例如 CTC 压缩、序列级知识蒸馏、生成速度和 GPU 内存占用。尽管 DFP 比交叉注意力更受欢迎，但我们的整体结果并未表明 DFP 具有明显的优势。

Title: Anonymization by Design of Language Modeling

• Authors: Antoine Boutet, Zakaria El Kazdam, Lucas Magnana, Helain Zimmermann
• Subjects: cs.CL, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02407
• Pdf URL: https://arxiv.org/pdf/2501.02407
• Copy Paste: [[2501.02407]] Anonymization by Design of Language Modeling(https://arxiv.org/abs/2501.02407)
• Keywords: language model, gpt
• Abstract: Rapid advances in Natural Language Processing (NLP) have revolutionized many fields, including healthcare. However, these advances raise significant privacy concerns, especially when models specialized on sensitive data can memorize and then expose and regurgitate confidential information. This paper presents a privacy-by-design language modeling approach to address the problem of language models anonymization, and thus promote their sharing. Specifically, we propose both a Masking Language Modeling (MLM) methodology to specialize a BERT-like language model, and a Causal Language Modeling (CLM) methodology to specialize a GPT-like model that avoids the model from memorizing direct and indirect identifying information present in the training data. We have comprehensively evaluated our approaches using medical datasets and compared them against different baselines. Our results indicate that by avoiding memorizing both direct and indirect identifiers during model specialization, our masking and causal language modeling schemes offer the best tradeoff for maintaining high privacy while retaining high utility.
• 摘要：自然语言处理 (NLP) 的快速发展彻底改变了许多领域，包括医疗保健。然而，这些进步引发了严重的隐私问题，尤其是当专门处理敏感数据的模型可以记忆然后暴露和复述机密信息时。本文提出了一种隐私设计语言建模方法来解决语言模型匿名化问题，从而促进其共享。具体来说，我们提出了一种屏蔽语言建模 (MLM) 方法来专门化类似 BERT 的语言模型，以及一种因果语言建模 (CLM) 方法来专门化类似 GPT 的模型，避免模型记忆训练数据中存在的直接和间接识别信息。我们已经使用医疗数据集全面评估了我们的方法，并将它们与不同的基线进行了比较。我们的结果表明，通过避免在模型专门化期间记忆直接和间接标识符，我们的屏蔽和因果语言建模方案在保持高隐私的同时保持高实用性方面提供了最佳权衡。

Title: Towards Multimodal Metaphor Understanding: A Chinese Dataset and Model for Metaphor Mapping Identification

• Authors: Dongyu Zhang, Shengcheng Yin, Jingwei Yu, Zhiyao Wu, Zhen Li, Chengpei Xu, Xiaoxia Wang, Feng Xia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02434
• Pdf URL: https://arxiv.org/pdf/2501.02434
• Copy Paste: [[2501.02434]] Towards Multimodal Metaphor Understanding: A Chinese Dataset and Model for Metaphor Mapping Identification(https://arxiv.org/abs/2501.02434)
• Keywords: prompt, chain-of-thought
• Abstract: Metaphors play a crucial role in human communication, yet their comprehension remains a significant challenge for natural language processing (NLP) due to the cognitive complexity involved. According to Conceptual Metaphor Theory (CMT), metaphors map a target domain onto a source domain, and understanding this mapping is essential for grasping the nature of metaphors. While existing NLP research has focused on tasks like metaphor detection and sentiment analysis of metaphorical expressions, there has been limited attention to the intricate process of identifying the mappings between source and target domains. Moreover, non-English multimodal metaphor resources remain largely neglected in the literature, hindering a deeper understanding of the key elements involved in metaphor interpretation. To address this gap, we developed a Chinese multimodal metaphor advertisement dataset (namely CM3D) that includes annotations of specific target and source domains. This dataset aims to foster further research into metaphor comprehension, particularly in non-English languages. Furthermore, we propose a Chain-of-Thought (CoT) Prompting-based Metaphor Mapping Identification Model (CPMMIM), which simulates the human cognitive process for identifying these mappings. Drawing inspiration from CoT reasoning and Bi-Level Optimization (BLO), we treat the task as a hierarchical identification problem, enabling more accurate and interpretable metaphor mapping. Our experimental results demonstrate the effectiveness of CPMMIM, highlighting its potential for advancing metaphor comprehension in NLP. Our dataset and code are both publicly available to encourage further advancements in this field.
• 摘要：隐喻在人类交流中起着至关重要的作用，但由于涉及认知复杂性，对隐喻的理解仍然是自然语言处理 (NLP) 的一大挑战。根据概念隐喻理论 (CMT)，隐喻将目标域映射到源域，理解这种映射对于掌握隐喻的本质至关重要。虽然现有的 NLP 研究主要集中在隐喻检测和隐喻表达的情感分析等任务上，但对识别源域和目标域之间映射的复杂过程的关注有限。此外，非英语多模态隐喻资源在文献中仍然被很大程度上忽视，阻碍了对隐喻解释中涉及的关键要素的更深入理解。为了弥补这一差距，我们开发了一个中文多模态隐喻广告数据集 (即 CM3D)，其中包括特定目标域和源域的注释。该数据集旨在促进对隐喻理解的进一步研究，特别是在非英语语言中。此外，我们提出了一种基于思维链 (CoT) 提示的隐喻映射识别模型 (CPMMIM)，该模型模拟了人类识别这些映射的认知过程。从 CoT 推理和双层优化 (BLO) 中汲取灵感，我们将该任务视为分层识别问题，从而实现更准确、更易于解释的隐喻映射。我们的实验结果证明了 CPMMIM 的有效性，凸显了其在 NLP 中推进隐喻理解的潜力。我们的数据集和代码均已公开，以鼓励该领域的进一步发展。

Title: Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap

• Authors: Hyunwoo Ko, Guijin Son, Dasol Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02448
• Pdf URL: https://arxiv.org/pdf/2501.02448
• Copy Paste: [[2501.02448]] Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap(https://arxiv.org/abs/2501.02448)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) demonstrate exceptional performance on complex reasoning tasks. However, despite their strong reasoning capabilities in high-resource languages (e.g., English and Chinese), a significant performance gap persists in other languages. To investigate this gap in Korean, we introduce HRM8K, a benchmark comprising 8,011 English-Korean parallel bilingual math problems. Through systematic analysis of model behaviors, we identify a key finding: these performance disparities stem primarily from difficulties in comprehending non-English inputs, rather than limitations in reasoning capabilities. Based on these findings, we propose UST (Understand, Solve, and Translate), a method that strategically uses English as an anchor for reasoning and solution generation. By fine-tuning the model on 130k synthetically generated data points, UST achieves a 10.91% improvement on the HRM8K benchmark and reduces the multilingual performance gap from 11.6% to 0.7%. Additionally, we show that improvements from UST generalize effectively to different Korean domains, demonstrating that capabilities acquired from machine-verifiable content can be generalized to other areas. We publicly release the benchmark, training dataset, and models.
• 摘要：大型语言模型 (LLM) 在复杂的推理任务上表现出色。然而，尽管它们在资源丰富的语言（例如英语和中文）中具有强大的推理能力，但在其他语言中仍然存在显著的性能差距。为了研究韩语中的这种差距，我们引入了 HRM8K，这是一个包含 8,011 个英韩并行双语数学问题的基准。通过对模型行为的系统分析，我们发现了一个关键发现：这些性能差异主要源于理解非英语输入的困难，而不是推理能力的限制。基于这些发现，我们提出了 UST（理解、解决和翻译），这是一种策略性地使用英语作为推理和解决方案生成基础的方法。通过在 130k 个合成生成的数据点上对模型进行微调，UST 在 HRM8K 基准上实现了 10.91% 的改进，并将多语言性能差距从 11.6% 缩小到 0.7%。此外，我们还展示了 UST 的改进可以有效地推广到不同的韩语领域，表明从机器可验证内容中获得的能力可以推广到其他领域。我们公开发布了基准、训练数据集和模型。

Title: Towards Omni-RAG: Comprehensive Retrieval-Augmented Generation for Large Language Models in Medical Applications

• Authors: Zhe Chen, Yusheng Liao, Shuyang Jiang, Pingjie Wang, Yiqiu Guo, Yanfeng Wang, Yu Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02460
• Pdf URL: https://arxiv.org/pdf/2501.02460
• Copy Paste: [[2501.02460]] Towards Omni-RAG: Comprehensive Retrieval-Augmented Generation for Large Language Models in Medical Applications(https://arxiv.org/abs/2501.02460)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: Large language models (LLMs) hold promise for addressing healthcare challenges but often generate hallucinations due to limited integration of medical knowledge. Incorporating external medical knowledge is therefore critical, especially considering the breadth and complexity of medical content, which necessitates effective multi-source knowledge acquisition. We address this challenge by framing it as a source planning problem, where the task is to formulate context-appropriate queries tailored to the attributes of diverse knowledge sources. Existing approaches either overlook source planning or fail to achieve it effectively due to misalignment between the model's expectation of the sources and their actual content. To bridge this gap, we present MedOmniKB, a comprehensive repository comprising multigenre and multi-structured medical knowledge sources. Leveraging these sources, we propose the Source Planning Optimisation (SPO) method, which enhances multi-source utilisation through explicit planning optimisation. Our approach involves enabling an expert model to explore and evaluate potential plans while training a smaller model to learn source alignment using positive and negative planning samples. Experimental results demonstrate that our method substantially improves multi-source planning performance, enabling the optimised small model to achieve state-of-the-art results in leveraging diverse medical knowledge sources.
• 摘要：大型语言模型 (LLM) 有望解决医疗保健挑战，但由于对医学知识的整合有限，通常会产生幻觉。因此，整合外部医学知识至关重要，尤其是考虑到医学内容的广度和复杂性，这需要有效的多源知识获取。我们将这一挑战定义为源规划问题，其任务是制定适合不同知识源属性的上下文查询。现有方法要么忽视源规划，要么由于模型对源的期望与其实际内容不一致而无法有效实现源规划。为了弥补这一差距，我们提出了 MedOmniKB，这是一个包含多类型和多结构医学知识源的综合存储库。利用这些来源，我们提出了源规划优化 (SPO) 方法，该方法通过明确的规划优化来增强多源利用率。我们的方法包括使专家模型能够探索和评估潜在计划，同时训练较小的模型以使用正负规划样本学习源对齐。实验结果表明，我们的方法显著提高了多源规划性能，使得优化的小模型在利用多样化的医学知识源方面取得最先进的结果。

Title: Hengqin-RA-v1: Advanced Large Language Model for Diagnosis and Treatment of Rheumatoid Arthritis with Dataset based Traditional Chinese Medicine

• Authors: Yishen Liu, Shengda Luo, Zishao Zhong, Tongtong Wu, Jianguo Zhang, Peiyao Ou, Yong Liang, Liang Liu, Hudan Pan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02471
• Pdf URL: https://arxiv.org/pdf/2501.02471
• Copy Paste: [[2501.02471]] Hengqin-RA-v1: Advanced Large Language Model for Diagnosis and Treatment of Rheumatoid Arthritis with Dataset based Traditional Chinese Medicine(https://arxiv.org/abs/2501.02471)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) primarily trained on English texts, often face biases and inaccuracies in Chinese contexts. Their limitations are pronounced in fields like Traditional Chinese Medicine (TCM), where cultural and clinical subtleties are vital, further hindered by a lack of domain-specific data, such as rheumatoid arthritis (RA). To address these issues, this paper introduces Hengqin-RA-v1, the first large language model specifically tailored for TCM with a focus on diagnosing and treating RA. We also present HQ-GCM-RA-C1, a comprehensive RA-specific dataset curated from ancient Chinese medical literature, classical texts, and modern clinical studies. This dataset empowers Hengqin-RA-v1 to deliver accurate and culturally informed responses, effectively bridging the gaps left by general-purpose models. Extensive experiments demonstrate that Hengqin-RA-v1 outperforms state-of-the-art models, even surpassing the diagnostic accuracy of TCM practitioners in certain cases.
• 摘要：大型语言模型 (LLM) 主要在英文文本上进行训练，在中文语境中经常会出现偏差和不准确性。它们的局限性在传统中医 (TCM) 等领域尤为明显，因为文化和临床细节至关重要，而且缺乏特定领域的数据，例如类风湿性关节炎 (RA)。为了解决这些问题，本文介绍了 Hengqin-RA-v1，这是第一个专门为中医量身定制的大型语言模型，专注于诊断和治疗 RA。我们还介绍了 HQ-GCM-RA-C1，这是一个全面的 RA 特定数据集，来自古代中医文献、经典文本和现代临床研究。该数据集使 Hengqin-RA-v1 能够提供准确且具有文化意义的响应，有效地弥补了通用模型留下的空白。大量实验表明，Hengqin-RA-v1 的表现优于最先进的模型，甚至在某些情况下超过了中医师的诊断准确率。

Title: Decoding News Bias: Multi Bias Detection in News Articles

• Authors: Bhushan Santosh Shah, Deven Santosh Shah, Vahida Attar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02482
• Pdf URL: https://arxiv.org/pdf/2501.02482
• Copy Paste: [[2501.02482]] Decoding News Bias: Multi Bias Detection in News Articles(https://arxiv.org/abs/2501.02482)
• Keywords: language model, llm
• Abstract: News Articles provides crucial information about various events happening in the society but they unfortunately come with different kind of biases. These biases can significantly distort public opinion and trust in the media, making it essential to develop techniques to detect and address them. Previous works have majorly worked towards identifying biases in particular domains e.g., Political, gender biases. However, more comprehensive studies are needed to detect biases across diverse domains. Large language models (LLMs) offer a powerful way to analyze and understand natural language, making them ideal for constructing datasets and detecting these biases. In this work, we have explored various biases present in the news articles, built a dataset using LLMs and present results obtained using multiple detection techniques. Our approach highlights the importance of broad-spectrum bias detection and offers new insights for improving the integrity of news articles.
• 摘要：新闻文章提供了有关社会上发生的各种事件的重要信息，但不幸的是，它们带有不同类型的偏见。这些偏见会严重扭曲公众舆论和对媒体的信任，因此开发检测和解决这些偏见的技术至关重要。以前的研究主要致力于识别特定领域的偏见，例如政治、性别偏见。然而，需要更全面的研究来检测不同领域的偏见。大型语言模型 (LLM) 提供了一种分析和理解自然语言的强大方法，使其成为构建数据集和检测这些偏见的理想选择。在这项工作中，我们探讨了新闻文章中存在的各种偏见，使用 LLM 构建了一个数据集，并展示了使用多种检测技术获得的结果。我们的方法强调了广谱偏见检测的重要性，并为提高新闻文章的完整性提供了新的见解。

Title: ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use

• Authors: Junjie Ye, Zhengyin Du, Xuesong Yao, Weijian Lin, Yufei Xu, Zehui Chen, Zaiyuan Wang, Sining Zhu, Zhiheng Xi, Siyu Yuan, Tao Gui, Qi Zhang, Xuanjing Huang, Jiechao Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02506
• Pdf URL: https://arxiv.org/pdf/2501.02506
• Copy Paste: [[2501.02506]] ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use(https://arxiv.org/abs/2501.02506)
• Keywords: language model, gpt, llm
• Abstract: Effective evaluation of multi-hop tool use is critical for analyzing the understanding, reasoning, and function-calling capabilities of large language models (LLMs). However, progress has been hindered by a lack of reliable evaluation datasets. To address this, we present ToolHop, a dataset comprising 995 user queries and 3,912 associated tools, specifically designed for rigorous evaluation of multi-hop tool use. ToolHop ensures diverse queries, meaningful interdependencies, locally executable tools, detailed feedback, and verifiable answers through a novel query-driven data construction approach that includes tool creation, document refinement, and code generation. We evaluate 14 LLMs across five model families (i.e., LLaMA3.1, Qwen2.5, Gemini1.5, Claude3.5, and GPT), uncovering significant challenges in handling multi-hop tool-use scenarios. The leading model, GPT-4o, achieves an accuracy of 49.04%, underscoring substantial room for improvement. Further analysis reveals variations in tool-use strategies for various families, offering actionable insights to guide the development of more effective approaches. Code and data can be found in this https URL.
• 摘要：有效评估多跳工具的使用对于分析大型语言模型 (LLM) 的理解、推理和函数调用能力至关重要。然而，由于缺乏可靠的评估数据集，进展受到阻碍。为了解决这个问题，我们提出了 ToolHop，这是一个包含 995 个用户查询和 3,912 个相关工具的数据集，专门用于严格评估多跳工具的使用。ToolHop 通过一种新颖的查询驱动数据构造方法（包括工具创建、文档细化和代码生成）确保多样化的查询、有意义的相互依赖性、本地可执行的工具、详细的反馈和可验证的答案。我们评估了五个模型系列（即 LLaMA3.1、Qwen2.5、Gemini1.5、Claude3.5 和 GPT）中的 14 个 LLM，发现在处理多跳工具使用场景方面存在重大挑战。领先的模型 GPT-4o 的准确率达到了 49.04%，凸显了巨大的改进空间。进一步的分析揭示了不同家庭的工具使用策略的差异，提供了可操作的见解来指导开发更有效的方法。代码和数据可在此 https URL 中找到。

Title: CHAIR-Classifier of Hallucination as Improver

• Authors: Ao Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02518
• Pdf URL: https://arxiv.org/pdf/2501.02518
• Copy Paste: [[2501.02518]] CHAIR-Classifier of Hallucination as Improver(https://arxiv.org/abs/2501.02518)
• Keywords: language model, hallucination
• Abstract: This paper presents a supervised method for detecting hallucinations in large language models. By analyzing token scores (logitis) across layers of the LLaMA model, we derive a small set, aiming to reduce overfitting, of features-including maximum, minimum, mean, standard deviation, and slope. We use logistic regression for classification and validate the model on the TruthfulQA and MMLU datasets. The results demonstrate significant performance gains, especially in zero-shot scenarios, highlighting the effectiveness and potential for generalization.
• 摘要：本文介绍了一种用于检测大型语言模型中的幻觉的监督方法。通过分析 LLaMA 模型各层的标记分数 (logitis)，我们得出了一个小型特征集，旨在减少过度拟合，这些特征包括最大值、最小值、平均值、标准差和斜率。我们使用逻辑回归进行分类，并在 TruthfulQA 和 MMLU 数据集上验证该模型。结果显示性能显著提升，尤其是在零样本场景中，凸显了其有效性和泛化潜力。

Title: Evaluating Large Language Models Against Human Annotators in Latent Content Analysis: Sentiment, Political Leaning, Emotional Intensity, and Sarcasm

• Authors: Ljubisa Bojic, Olga Zagovora, Asta Zelenkauskaite, Vuk Vukovic, Milan Cabarkapa, Selma Veseljević Jerkovic, Ana Jovančevic
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.02532
• Pdf URL: https://arxiv.org/pdf/2501.02532
• Copy Paste: [[2501.02532]] Evaluating Large Language Models Against Human Annotators in Latent Content Analysis: Sentiment, Political Leaning, Emotional Intensity, and Sarcasm(https://arxiv.org/abs/2501.02532)
• Keywords: language model, gpt, llm
• Abstract: In the era of rapid digital communication, vast amounts of textual data are generated daily, demanding efficient methods for latent content analysis to extract meaningful insights. Large Language Models (LLMs) offer potential for automating this process, yet comprehensive assessments comparing their performance to human annotators across multiple dimensions are lacking. This study evaluates the reliability, consistency, and quality of seven state-of-the-art LLMs, including variants of OpenAI's GPT-4, Gemini, Llama, and Mixtral, relative to human annotators in analyzing sentiment, political leaning, emotional intensity, and sarcasm detection. A total of 33 human annotators and eight LLM variants assessed 100 curated textual items, generating 3,300 human and 19,200 LLM annotations, with LLMs evaluated across three time points to examine temporal consistency. Inter-rater reliability was measured using Krippendorff's alpha, and intra-class correlation coefficients assessed consistency over time. The results reveal that both humans and LLMs exhibit high reliability in sentiment analysis and political leaning assessments, with LLMs demonstrating higher internal consistency than humans. In emotional intensity, LLMs displayed higher agreement compared to humans, though humans rated emotional intensity significantly higher. Both groups struggled with sarcasm detection, evidenced by low agreement. LLMs showed excellent temporal consistency across all dimensions, indicating stable performance over time. This research concludes that LLMs, especially GPT-4, can effectively replicate human analysis in sentiment and political leaning, although human expertise remains essential for emotional intensity interpretation. The findings demonstrate the potential of LLMs for consistent and high-quality performance in certain areas of latent content analysis.
• 摘要：在快速数字通信时代，每天都会产生大量文本数据，需要有效的潜在内容分析方法来提取有意义的见解。大型语言模型 (LLM) 提供了自动化此过程的潜力，但缺乏将其性能与多维度的人工注释者进行比较的综合评估。这项研究评估了七种最先进的 LLM（包括 OpenAI 的 GPT-4、Gemini、Llama 和 Mixtral 的变体）相对于人工注释者在分析情绪、政治倾向、情绪强度和讽刺检测方面的可靠性、一致性和质量。共有 33 名人工注释者和 8 个 LLM 变体评估了 100 个精选的文本项目，生成了 3,300 个人工注释和 19,200 个 LLM 注释，并在三个时间点评估 LLM 以检查时间一致性。使用 Krippendorff 的 alpha 测量评分者间信度，并使用类内相关系数评估随时间变化的一致性。结果表明，人类和 LLM 在情绪分析和政治倾向评估中都表现出很高的可靠性，而且 LLM 表现出比人类更高的内部一致性。在情绪强度方面，LLM 表现出比人类更高的一致性，尽管人类对情绪强度的评价明显更高。两组人在讽刺检测方面都遇到了困难，一致性较低就是明证。LLM 在所有维度上都表现出极好的时间一致性，表明随着时间的推移其性能稳定。这项研究的结论是，LLM，尤其是 GPT-4，可以有效地复制人类在情绪和政治倾向方面的分析，尽管人类的专业知识对于情绪强度的解释仍然至关重要。研究结果表明，LLM 在某些潜在内容分析领域具有一致且高质量的性能潜力。

Title: Multi-LLM Collaborative Caption Generation in Scientific Documents

• Authors: Jaeyoung Kim, Jongho Lee, Hong-Jun Choi, Ting-Yao Hsu, Chieh-Yang Huang, Sungchul Kim, Ryan Rossi, Tong Yu, Clyde Lee Giles, Ting-Hao 'Kenneth' Huang, Sungchul Choi
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02552
• Pdf URL: https://arxiv.org/pdf/2501.02552
• Copy Paste: [[2501.02552]] Multi-LLM Collaborative Caption Generation in Scientific Documents(https://arxiv.org/abs/2501.02552)
• Keywords: language model, llm, prompt
• Abstract: Scientific figure captioning is a complex task that requires generating contextually appropriate descriptions of visual content. However, existing methods often fall short by utilizing incomplete information, treating the task solely as either an image-to-text or text summarization problem. This limitation hinders the generation of high-quality captions that fully capture the necessary details. Moreover, existing data sourced from arXiv papers contain low-quality captions, posing significant challenges for training large language models (LLMs). In this paper, we introduce a framework called Multi-LLM Collaborative Figure Caption Generation (MLBCAP) to address these challenges by leveraging specialized LLMs for distinct sub-tasks. Our approach unfolds in three key modules: (Quality Assessment) We utilize multimodal LLMs to assess the quality of training data, enabling the filtration of low-quality captions. (Diverse Caption Generation) We then employ a strategy of fine-tuning/prompting multiple LLMs on the captioning task to generate candidate captions. (Judgment) Lastly, we prompt a prominent LLM to select the highest quality caption from the candidates, followed by refining any remaining inaccuracies. Human evaluations demonstrate that informative captions produced by our approach rank better than human-written captions, highlighting its effectiveness. Our code is available at this https URL
• 摘要：科学图形字幕制作是一项复杂的任务，需要生成符合上下文的视觉内容描述。然而，现有的方法往往因利用不完整的信息而不足，将任务仅仅视为图像到文本或文本摘要问题。这种限制阻碍了生成完全捕捉必要细节的高质量字幕。此外，来自 arXiv 论文的现有数据包含低质量字幕，这对训练大型语言模型 (LLM) 构成了重大挑战。在本文中，我们引入了一个名为多 LLM 协作图形字幕生成 (MLBCAP) 的框架，通过利用专门的 LLM 来完成不同的子任务来解决这些挑战。我们的方法分为三个关键模块：（质量评估）我们利用多模态 LLM 来评估训练数据的质量，从而过滤掉低质量的字幕。（多样化字幕生成）然后，我们采用一种策略，在字幕任务上微调/提示多个 LLM 来生成候选字幕。 （判断）最后，我们让一位著名的法学硕士从候选字幕中选择出质量最高的字幕，然后改进任何剩余的不准确之处。人工评估表明，我们的方法生成的信息性字幕排名优于人工编写的字幕，凸显了其有效性。我们的代码可在此 https URL 上找到

Title: Prune or Retrain: Optimizing the Vocabulary of Multilingual Models for Estonian

• Authors: Aleksei Dorkin, Taido Purason, Kairit Sirts
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02631
• Pdf URL: https://arxiv.org/pdf/2501.02631
• Copy Paste: [[2501.02631]] Prune or Retrain: Optimizing the Vocabulary of Multilingual Models for Estonian(https://arxiv.org/abs/2501.02631)
• Keywords: language model
• Abstract: Adapting multilingual language models to specific languages can enhance both their efficiency and performance. In this study, we explore how modifying the vocabulary of a multilingual encoder model to better suit the Estonian language affects its downstream performance on the Named Entity Recognition (NER) task. The motivations for adjusting the vocabulary are twofold: practical benefits affecting the computational cost, such as reducing the input sequence length and the model size, and performance enhancements by tailoring the vocabulary to the particular language. We evaluate the effectiveness of two vocabulary adaptation approaches -- retraining the tokenizer and pruning unused tokens -- and assess their impact on the model's performance, particularly after continual training. While retraining the tokenizer degraded the performance of the NER task, suggesting that longer embedding tuning might be needed, we observed no negative effects on pruning.
• 摘要：将多语言语言模型调整为特定语言可以提高其效率和性能。在本研究中，我们探讨了修改多语言编码器模型的词汇表以更好地适应爱沙尼亚语会如何影响其在命名实体识别 (NER) 任务中的下游性能。调整词汇表的动机有两个方面：影响计算成本的实际好处，例如减少输入序列长度和模型大小，以及通过将词汇表调整为特定语言来提高性能。我们评估了两种词汇调整方法的有效性——重新训练标记器和修剪未使用的标记——并评估它们对模型性能的影响，尤其是在持续训练之后。虽然重新训练标记器会降低 NER 任务的性能，表明可能需要更长的嵌入调整，但我们没有观察到对修剪的负面影响。

Title: From Superficial Patterns to Semantic Understanding: Fine-Tuning Language Models on Contrast Sets

• Authors: Daniel Petrov
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02683
• Pdf URL: https://arxiv.org/pdf/2501.02683
• Copy Paste: [[2501.02683]] From Superficial Patterns to Semantic Understanding: Fine-Tuning Language Models on Contrast Sets(https://arxiv.org/abs/2501.02683)
• Keywords: language model
• Abstract: Large scale pretrained language models have demonstrated high performance on standard datasets for natural language inference (NLI) tasks. Unfortunately, these evaluations can be misleading, as although the models can perform well on in-distribution data, they perform poorly on out-of-distribution test sets, such as contrast sets. Contrast sets consist of perturbed instances of data that have very minor, but meaningful, changes to the input that alter the gold label, revealing how models can learn superficial patterns in the training data rather than learning more sophisticated language nuances. As an example, the ELECTRA-small language model achieves nearly 90% accuracy on an SNLI dataset but drops to 75% when tested on an out-of-distribution contrast set. The research performed in this study explores how a language models' robustness can be improved by exposing it to small amounts of more complex contrast sets during training to help it better learn language patterns. With this approach, the model regains performance and achieves nearly 90% accuracy on contrast sets, highlighting the importance of diverse and challenging training data.
• 摘要：大规模预训练语言模型在自然语言推理 (NLI) 任务的标准数据集上表现出色。不幸的是，这些评估可能会产生误导，因为尽管这些模型在分布内的数据上表现良好，但它们在分布外的测试集（例如对比集）上表现不佳。对比集由受干扰的数据实例组成，这些实例对输入进行了非常小但有意义的更改，从而改变了黄金标签，揭示了模型如何学习训练数据中的表面模式，而不是学习更复杂的语言细微差别。例如，ELECTRA-small 语言模型在 SNLI 数据集上的准确率接近 90%，但在分布外的对比集上测试时下降到 75%。本研究进行的研究探索了如何通过在训练期间将语言模型暴露于少量更复杂的对比集来帮助其更好地学习语言模式来提高语言模型的稳健性。通过这种方法，模型恢复了性能并在对比集上实现了近 90% 的准确率，凸显了多样化和具有挑战性的训练数据的重要性。

Title: Decoding specialised feature neurons in LLMs with the final projection layer

• Authors: Harry J Davies
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02688
• Pdf URL: https://arxiv.org/pdf/2501.02688
• Copy Paste: [[2501.02688]] Decoding specialised feature neurons in LLMs with the final projection layer(https://arxiv.org/abs/2501.02688)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) typically have billions of parameters and are thus often difficult to interpret in their operation. Such black-box models can pose a significant risk to safety when trusted to make important decisions. The lack of interpretability of LLMs is more related to their sheer size, rather than the complexity of their individual components. The TARS method for knowledge removal (Davies et al 2024) provides strong evidence for the hypothesis that that linear layer weights which act directly on the residual stream may have high correlation with different concepts encoded in the residual stream. Building upon this, we attempt to decode neuron weights directly into token probabilities through the final projection layer of the model (the LM-head). Firstly, we show that with Llama 3.1 8B we can utilise the LM-head to decode specialised feature neurons that respond strongly to certain concepts, with examples such as "dog" and "California". This is then confirmed by demonstrating that these neurons can be clamped to affect the probability of the concept in the output. This extends to the fine-tuned assistant Llama 3.1 8B instruct model, where we find that over 75% of neurons in the up-projection layers have the same top associated token compared to the pretrained model. Finally, we demonstrate that clamping the "dog" neuron leads the instruct model to always discuss dogs when asked about its favourite animal. Through our method, it is possible to map the entirety of Llama 3.1 8B's up-projection neurons in less than 15 minutes with no parallelization.
• 摘要：大型语言模型 (LLM) 通常具有数十亿个参数，因此在操作过程中通常难以解释。当信任此类黑盒模型做出重要决策时，可能会对安全造成重大风险。LLM 缺乏可解释性与其庞大的规模有关，而不是其各个组件的复杂性。TARS 知识移除方法 (Davies et al 2024) 为以下假设提供了强有力的证据：直接作用于残差流的线性层权重可能与残差流中编码的不同概念具有高度相关性。在此基础上，我们尝试通过模型的最终投影层 (LM-head) 将神经元权重直接解码为标记概率。首先，我们展示了使用 Llama 3.1 8B，我们可以利用 LM-head 解码对某些概念反应强烈的特殊特征神经元，例如“狗”和“加利福尼亚”。然后通过证明这些神经元可以被钳制以影响输出中概念的概率来证实这一点。这扩展到经过微调的助手 Llama 3.1 8B 指示模型，我们发现与预训练模型相比，上投影层中超过 75% 的神经元具有相同的顶部相关标记。最后，我们证明，限制“狗”神经元会导致指示模型在被问及它最喜欢的动物时总是讨论狗。通过我们的方法，可以在不到 15 分钟的时间内映射 Llama 3.1 8B 的全部上投影神经元，而无需并行化。

Title: QuIM-RAG: Advancing Retrieval-Augmented Generation with Inverted Question Matching for Enhanced QA Performance

• Authors: Binita Saha, Utsha Saha, Muhammad Zubair Malik
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02702
• Pdf URL: https://arxiv.org/pdf/2501.02702
• Copy Paste: [[2501.02702]] QuIM-RAG: Advancing Retrieval-Augmented Generation with Inverted Question Matching for Enhanced QA Performance(https://arxiv.org/abs/2501.02702)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: This work presents a novel architecture for building Retrieval-Augmented Generation (RAG) systems to improve Question Answering (QA) tasks from a target corpus. Large Language Models (LLMs) have revolutionized the analyzing and generation of human-like text. These models rely on pre-trained data and lack real-time updates unless integrated with live data tools. RAG enhances LLMs by integrating online resources and databases to generate contextually appropriate responses. However, traditional RAG still encounters challenges like information dilution and hallucinations when handling vast amounts of data. Our approach addresses these challenges by converting corpora into a domain-specific dataset and RAG architecture is constructed to generate responses from the target document. We introduce QuIM-RAG (Question-to-question Inverted Index Matching), a novel approach for the retrieval mechanism in our system. This strategy generates potential questions from document chunks and matches these with user queries to identify the most relevant text chunks for generating accurate answers. We have implemented our RAG system on top of the open-source Meta-LLaMA3-8B-instruct model by Meta Inc. that is available on Hugging Face. We constructed a custom corpus of 500+ pages from a high-traffic website accessed thousands of times daily for answering complex questions, along with manually prepared ground truth QA for evaluation. We compared our approach with traditional RAG models using BERT-Score and RAGAS, state-of-the-art metrics for evaluating LLM applications. Our evaluation demonstrates that our approach outperforms traditional RAG architectures on both metrics.
• 摘要：这项工作提出了一种用于构建检索增强生成 (RAG) 系统的新架构，以改进目标语料库中的问答 (QA) 任务。大型语言模型 (LLM) 彻底改变了类人文本的分析和生成。这些模型依赖于预先训练的数据，除非与实时数据工具集成，否则缺乏实时更新。RAG 通过集成在线资源和数据库来生成适合上下文的响应，从而增强了 LLM。然而，传统的 RAG 在处理大量数据时仍然面临信息稀释和幻觉等挑战。我们的方法通过将语料库转换为特定领域的数据集来解决这些挑战，并构建 RAG 架构以从目标文档生成响应。我们引入了 QuIM-RAG（问题到问题倒排索引匹配），这是我们系统中检索机制的一种新方法。该策略从文档块中生成潜在问题，并将这些问题与用户查询进行匹配，以确定最相关的文本块以生成准确的答案。我们在 Meta Inc. 的开源 Meta-LLaMA3-8B-instruct 模型之上实现了 RAG 系统，该模型可在 Hugging Face 上使用。我们从一个流量很大的网站构建了一个包含 500 多页的自定义语料库，该网站每天访问数千次，用于回答复杂问题，同时还手动准备了用于评估的真值 QA。我们使用 BERT-Score 和 RAGAS（用于评估 LLM 应用程序的最先进的指标）将我们的方法与传统 RAG 模型进行了比较。我们的评估表明，我们的方法在这两个指标上都优于传统的 RAG 架构。

Title: TARDiS : Text Augmentation for Refining Diversity and Separability

• Authors: Kyungmin Kim, SangHun Im, GiBaeg Kim, Heung-Seon Oh
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02739
• Pdf URL: https://arxiv.org/pdf/2501.02739
• Copy Paste: [[2501.02739]] TARDiS : Text Augmentation for Refining Diversity and Separability(https://arxiv.org/abs/2501.02739)
• Keywords: llm, prompt
• Abstract: Text augmentation (TA) is a critical technique for text classification, especially in few-shot settings. This paper introduces a novel LLM-based TA method, TARDiS, to address challenges inherent in the generation and alignment stages of two-stage TA methods. For the generation stage, we propose two generation processes, SEG and CEG, incorporating multiple class-specific prompts to enhance diversity and separability. For the alignment stage, we introduce a class adaptation (CA) method to ensure that generated examples align with their target classes through verification and modification. Experimental results demonstrate TARDiS's effectiveness, outperforming state-of-the-art LLM-based TA methods in various few-shot text classification tasks. An in-depth analysis confirms the detailed behaviors at each stage.
• 摘要：文本增强 (TA) 是文本分类的一项关键技术，尤其是在小样本设置中。本文介绍了一种基于 LLM 的新型 TA 方法 TARDiS，以解决两阶段 TA 方法的生成和对齐阶段所固有的挑战。对于生成阶段，我们提出了两个生成过程，SEG 和 CEG，结合了多个特定于类的提示来增强多样性和可分离性。对于对齐阶段，我们引入了一种类自适应 (CA) 方法，以确保生成的示例通过验证和修改与其目标类对齐。实验结果证明了 TARDiS 的有效性，在各种小样本文本分类任务中均优于最先进的基于 LLM 的 TA 方法。深入分析证实了每个阶段的详细行为。

Title: Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model

• Authors: Yueqin Yin, Shentao Yang, Yujia Xie, Ziyi Yang, Yuting Sun, Hany Awadalla, Weizhu Chen, Mingyuan Zhou
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02790
• Pdf URL: https://arxiv.org/pdf/2501.02790
• Copy Paste: [[2501.02790]] Segmenting Text and Learning Their Rewards for Improved RLHF in Language Model(https://arxiv.org/abs/2501.02790)
• Keywords: language model
• Abstract: Reinforcement learning from human feedback (RLHF) has been widely adopted to align language models (LMs) with human preference. Prior RLHF works typically take a bandit formulation, which, though intuitive, ignores the sequential nature of LM generation and can suffer from the sparse reward issue. While recent works propose dense token-level RLHF, treating each token as an action may be oversubtle to proper reward assignment. In this paper, we seek to get the best of both by training and utilizing a segment-level reward model, which assigns a reward to each semantically complete text segment that spans over a short sequence of tokens. For reward learning, our method allows dynamic text segmentation and compatibility with standard sequence-preference datasets. For effective RL-based LM training against segment reward, we generalize the classical scalar bandit reward normalizers into location-aware normalizer functions and interpolate the segment reward for further densification. With these designs, our method performs competitively on three popular RLHF benchmarks for LM policy: AlpacaEval 2.0, Arena-Hard, and MT-Bench. Ablation studies are conducted to further demonstrate our method.
• 摘要：人类反馈强化学习 (RLHF) 已被广泛采用，以使语言模型 (LM) 与人类偏好保持一致。先前的 RLHF 工作通常采用 bandit 公式，虽然直观，但忽略了 LM 生成的顺序性，并且可能受到稀疏奖励问题的影响。虽然最近的工作提出了密集的 token 级 RLHF，但将每个 token 视为一个动作可能过于微妙，无法正确分配奖励。在本文中，我们寻求通过训练和使用段级奖励模型来获得两者的最佳效果，该模型为每个跨越短序列 token 的语义完整的文本段分配奖励。对于奖励学习，我们的方法允许动态文本分段并与标准序列偏好数据集兼容。为了针对段奖励进行有效的基于 RL 的 LM 训练，我们将经典的标量 bandit 奖励规范化器推广到位置感知规范化器函数，并插入段奖励以进一步密集化。凭借这些设计，我们的方法在三个流行的 LM 策略 RLHF 基准上表现出色：AlpacaEval 2.0、Arena-Hard 和 MT-Bench。进行了消融研究以进一步证明我们的方法。

Title: InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusion

• Authors: Zhaoyi Yan, Zhijie Sang, Yiming Zhang, Yuhao Fu, Baoyi He, Qi Zhou, Yining Di, Chunlin Ji, Shengyu Zhang, Fei Wu, Hongxia Yang
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.02795
• Pdf URL: https://arxiv.org/pdf/2501.02795
• Copy Paste: [[2501.02795]] InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusion(https://arxiv.org/abs/2501.02795)
• Keywords: language model, llm
• Abstract: Large Language Models (LLMs) have demonstrated strong performance across various reasoning tasks, yet building a single model that consistently excels across all domains remains challenging. This paper addresses this problem by exploring strategies to integrate multiple domain-specialized models into an efficient pivot this http URL propose two fusion strategies to combine the strengths of multiple LLMs: (1) a pairwise, multi-step fusion approach that sequentially distills each source model into the pivot model, followed by a weight merging step to integrate the distilled models into the final model. This method achieves strong performance but requires substantial training effort; and (2) a unified fusion approach that aggregates all source models' outputs this http URL improve the fusion process, we introduce a novel Rate-Skewness Adaptive Fusion (RSAF) technique, which dynamically adjusts top-K ratios during parameter merging for enhanced flexibility and this http URL, we propose an uncertainty-based weighting method for the unified approach, which dynamically balances the contributions of source models and outperforms other logits/distribution ensemble this http URL achieved accuracy improvements of 9.27%, 8.80%, and 8.89% on the GSM8K, MATH, and HumanEval tasks, respectively.
• 摘要：大型语言模型 (LLM) 在各种推理任务中都表现出色，但构建一个在所有领域都表现出色的单一模型仍然具有挑战性。本文通过探索将多个领域专门模型集成为高效枢轴的策略来解决此问题，此 http URL 提出了两种融合策略来结合多个 LLM 的优势：(1) 成对、多步骤融合方法，该方法依次将每个源模型提炼到枢轴模型中，然后进行权重合并步骤以将提炼后的模型集成到最终模型中。此方法实现了强大的性能，但需要大量的训练工作；以及（2）一种聚合所有源模型输出的统一融合方法，此 http URL 改进融合过程，我们引入了一种新颖的速率偏斜自适应融合（RSAF）技术，该技术在参数合并期间动态调整 top-K 比率以增强灵活性，并且此 http URL，我们提出了一种基于不确定性的统一方法加权方法，它动态平衡源模型的贡献并优于其他 logits/分布集成此 http URL 分别在 GSM8K、MATH 和 HumanEval 任务上实现了 9.27%、8.80% 和 8.89% 的准确度提高。

Title: Graph-based Retrieval Augmented Generation for Dynamic Few-shot Text Classification

• Authors: Yubo Wang, Haoyang Li, Fei Teng, Lei Chen
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.02844
• Pdf URL: https://arxiv.org/pdf/2501.02844
• Copy Paste: [[2501.02844]] Graph-based Retrieval Augmented Generation for Dynamic Few-shot Text Classification(https://arxiv.org/abs/2501.02844)
• Keywords: language model, llm, retrieval augmented generation, retrieval-augmented generation
• Abstract: Text classification is a fundamental task in natural language processing, pivotal to various applications such as query optimization, data integration, and schema matching. While neural network-based models, such as CNN and BERT, have demonstrated remarkable performance in text classification, their effectiveness heavily relies on abundant labeled training data. This dependency makes these models less effective in dynamic few-shot text classification, where labeled data is scarce, and target labels frequently evolve based on application needs. Recently, large language models (LLMs) have shown promise due to their extensive pretraining and contextual understanding. Current approaches provide LLMs with text inputs, candidate labels, and additional side information (e.g., descriptions) to predict text labels. However, their effectiveness is hindered by the increased input size and the noise introduced through side information processing. To address these limitations, we propose a graph-based online retrieval-augmented generation framework, namely GORAG, for dynamic few-shot text classification. GORAG constructs and maintains an adaptive information graph by extracting side information across all target texts, rather than treating each input independently. It employs a weighted edge mechanism to prioritize the importance and reliability of extracted information and dynamically retrieves relevant context using a minimum-cost spanning tree tailored for each text input. Empirical evaluations demonstrate that GORAG outperforms existing approaches by providing more comprehensive and accurate contextual information.
• 摘要：文本分类是自然语言处理中的一项基本任务，对查询优化、数据集成和模式匹配等各种应用至关重要。虽然基于神经网络的模型（例如 CNN 和 BERT）在文本分类中表现出色，但它们的有效性在很大程度上依赖于丰富的标记训练数据。这种依赖性使得这些模型在动态小样本文本分类中效果较差，因为标记数据稀缺，并且目标标签经常根据应用需求而变化。最近，大型语言模型 (LLM) 因其广泛的预训练和上下文理解而显示出良好的前景。当前的方法为 LLM 提供文本输入、候选标签和额外的辅助信息（例如描述）来预测文本标签。然而，它们的有效性受到输入大小增加和辅助信息处理引入的噪声的阻碍。为了解决这些限制，我们提出了一个基于图的在线检索增强生成框架，即 GORAG，用于动态小样本文本分类。GORAG 通过提取所有目标文本中的辅助信息来构建和维护自适应信息图，而不是独立处理每个输入。它采用加权边缘机制来优先考虑所提取信息的重要性和可靠性，并使用针对每个文本输入量身定制的最小成本生成树动态检索相关上下文。实证评估表明，GORAG 的表现优于现有方法，因为它可以提供更全面、更准确的上下文信息。

Title: IIMedGPT: Promoting Large Language Model Capabilities of Medical Tasks by Efficient Human Preference Alignment

• Authors: Yiming Zhang, Zheng Chang, Wentao Cai, MengXing Ren, Kang Yuan, Yining Sun, Zenghui Ding
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.02869
• Pdf URL: https://arxiv.org/pdf/2501.02869
• Copy Paste: [[2501.02869]] IIMedGPT: Promoting Large Language Model Capabilities of Medical Tasks by Efficient Human Preference Alignment(https://arxiv.org/abs/2501.02869)
• Keywords: language model, gpt, llm
• Abstract: Recent researches of large language models(LLM), which is pre-trained on massive general-purpose corpora, have achieved breakthroughs in responding human queries. However, these methods face challenges including limited data insufficiency to support extensive pre-training and can not align responses with users' instructions. To address these issues, we introduce a medical instruction dataset, CMedINS, containing six medical instructions derived from actual medical tasks, which effectively fine-tunes LLM in conjunction with other data. Subsequently, We launch our medical model, IIMedGPT, employing an efficient preference alignment method, Direct preference Optimization(DPO). The results show that our final model outperforms existing medical models in medical this http URL, Code and model checkpoints will be released upon acceptance.
• 摘要：大型语言模型 (LLM) 的最新研究在响应人类查询方面取得了突破，该模型是在大量通用语料库上进行预训练的。然而，这些方法面临着挑战，包括有限的数​​据不足以支持广泛的预训练，并且无法将响应与用户的指令对齐。为了解决这些问题，我们引入了一个医疗指令数据集 CMedINS，其中包含来自实际医疗任务的六个医疗指令，它与其他数据一起有效地对 LLM 进行了微调。随后，我们推出了我们的医疗模型 IIMedGPT，采用了一种有效的偏好对齐方法，直接偏好优化 (DPO)。结果表明，我们的最终模型在医学方面优于现有的医学模型。此 http URL、代码和模型检查点将在接受后发布。

Title: Registering Source Tokens to Target Language Spaces in Multilingual Neural Machine Translation

• Authors: Zhi Qu, Yiran Wang, Jiannan Mao, Chenchen Ding, Hideki Tanaka, Masao Utiyama, Taro Watanabe
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.02979
• Pdf URL: https://arxiv.org/pdf/2501.02979
• Copy Paste: [[2501.02979]] Registering Source Tokens to Target Language Spaces in Multilingual Neural Machine Translation(https://arxiv.org/abs/2501.02979)
• Keywords: language model, llm
• Abstract: The multilingual neural machine translation (MNMT) enables arbitrary translations across multiple languages by training a model with limited parameters using parallel data only. However, the performance of such MNMT models still lags behind that of large language models (LLMs), limiting their practicality. In this work, we address this limitation by introducing registering to achieve the new state-of-the-art of decoder-only MNMT models. Specifically, we insert a set of artificial tokens specifying the target language, called registers, into the input sequence between the source and target tokens. By modifying the attention mask, the target token generation only pays attention to the activation of registers, representing the source tokens in the target language space. Experiments on EC-40, a large-scale benchmark, show that our method outperforms related methods driven by optimizing multilingual representations. We further scale up and collect 9.3 billion sentence pairs across 24 languages from public datasets to pre-train two models, namely MITRE (multilingual translation with registers). One of them, MITRE-913M, outperforms NLLB-3.3B, achieves comparable performance with commercial LLMs, and shows strong adaptability in fine-tuning. Finally, we open-source our models to facilitate further research and development in MNMT: this https URL.
• 摘要：多语言神经机器翻译 (MNMT) 仅使用并行数据训练具有有限参数的模型，即可实现跨多种语言的任意翻译。然而，此类 MNMT 模型的性能仍然落后于大型语言模型 (LLM)，限制了它们的实用性。在这项工作中，我们通过引入注册来解决这一限制，以实现仅解码器的 MNMT 模型的全新最佳状态。具体来说，我们在源和目标标记之间的输入序列中插入一组指定目标语言的人工标记（称为寄存器）。通过修改注意力掩码，目标标记生成仅关注寄存器的激活，表示目标语言空间中的源标记。在大型基准 EC-40 上的实验表明，我们的方法优于通过优化多语言表示驱动的相关方法。我们进一步扩大规模，并从公共数据集中收集了 24 种语言的 93 亿个句子对，以预训练两个模型，即 MITRE（带寄存器的多语言翻译）。其中之一 MITRE-913M 的表现优于 NLLB-3.3B，与商用 LLM 的性能相当，并且在微调方面表现出很强的适应性。最后，我们将我们的模型开源，以促进 MNMT 的进一步研究和开发：此 https URL。

Title: Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning

• Authors: Zhen Li, Yupeng Su, Runming Yang, Zhongwei Xie, Ngai Wong, Hongxia Yang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03035
• Pdf URL: https://arxiv.org/pdf/2501.03035
• Copy Paste: [[2501.03035]] Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning(https://arxiv.org/abs/2501.03035)
• Keywords: language model, llm
• Abstract: Large language models have achieved significant advancements in complex mathematical reasoning benchmarks, such as MATH. However, their substantial computational requirements present challenges for practical deployment. Model quantization has emerged as an effective strategy to reduce memory usage and computational costs by employing lower precision and bit-width representations. In this study, we systematically evaluate the impact of quantization on mathematical reasoning tasks. We introduce a multidimensional evaluation framework that qualitatively assesses specific capability dimensions and conduct quantitative analyses on the step-by-step outputs of various quantization methods. Our results demonstrate that quantization differentially affects numerical computation and reasoning planning abilities, identifying key areas where quantized models experience performance degradation.
• 摘要：大型语言模型在复杂的数学推理基准测试（如 MATH）中取得了重大进展。然而，它们大量的计算要求给实际部署带来了挑战。模型量化已成为一种有效的策略，它通过使用较低的精度和位宽表示来减少内存使用和计算成本。在本研究中，我们系统地评估了量化对数学推理任务的影响。我们引入了一个多维评估框架，该框架定性评估特定能力维度，并对各种量化方法的逐步输出进行定量分析。我们的结果表明，量化对数值计算和推理规划能力有不同的影响，并确定了量化模型性能下降的关键领域。

Title: Trust Modeling in Counseling Conversations: A Benchmark Study

• Authors: Aseem Srivastava, Zuhair Hasan Shaik, Tanmoy Chakraborty, Md Shad Akhtar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03064
• Pdf URL: https://arxiv.org/pdf/2501.03064
• Copy Paste: [[2501.03064]] Trust Modeling in Counseling Conversations: A Benchmark Study(https://arxiv.org/abs/2501.03064)
• Keywords: language model
• Abstract: In mental health counseling, a variety of earlier studies have focused on dialogue modeling. However, most of these studies give limited to no emphasis on the quality of interaction between a patient and a therapist. The therapeutic bond between a patient and a therapist directly correlates with effective mental health counseling. It involves developing the patient's trust on the therapist over the course of counseling. To assess the therapeutic bond in counseling, we introduce trust as a therapist-assistive metric. Our definition of trust involves patients' willingness and openness to express themselves and, consequently, receive better care. We conceptualize it as a dynamic trajectory observable through textual interactions during the counseling. To facilitate trust modeling, we present MENTAL-TRUST, a novel counseling dataset comprising manual annotation of 212 counseling sessions with first-of-its-kind seven expert-verified ordinal trust levels. We project our problem statement as an ordinal classification task for trust quantification and propose a new benchmark, TrustBench, comprising a suite of classical and state-of-the-art language models on MENTAL-TRUST. We evaluate the performance across a suite of metrics and lay out an exhaustive set of findings. Our study aims to unfold how trust evolves in therapeutic interactions.
• 摘要：在心理健康咨询中，早期的各种研究都集中于对话建模。然而，这些研究中的大多数都没有强调患者和治疗师之间的互动质量。患者和治疗师之间的治疗纽带与有效的心理健康咨询直接相关。它涉及在咨询过程中培养患者对治疗师的信任。为了评估咨询中的治疗纽带，我们引入了信任作为治疗师的辅助指标。我们对信任的定义包括患者表达自己的意愿和开放程度，从而得到更好的护理。我们将其概念化为可通过咨询期间的文本交互观察到的动态轨迹。为了促进信任建模，我们提出了 MENTAL-TRUST，这是一个新颖的咨询数据集，包含 212 个咨询会话的手动注释，具有首创的七个专家验证的序数信任级别。我们将问题陈述投射为信任量化的有序分类任务，并提出了一个新的基准 TrustBench，它包含一套经典和最先进的 MENTAL-TRUST 语言模型。我们通过一套指标评估其性能，并列出一系列详尽的发现。我们的研究旨在揭示信任在治疗互动中是如何演变的。

Title: Sentiment-guided Commonsense-aware Response Generation for Mental Health Counseling

• Authors: Aseem Srivastava, Gauri Naik, Alison Cerezo, Tanmoy Chakraborty, Md. Shad Akhtar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03088
• Pdf URL: https://arxiv.org/pdf/2501.03088
• Copy Paste: [[2501.03088]] Sentiment-guided Commonsense-aware Response Generation for Mental Health Counseling(https://arxiv.org/abs/2501.03088)
• Keywords: chat
• Abstract: The crisis of mental health issues is escalating. Effective counseling serves as a critical lifeline for individuals suffering from conditions like PTSD, stress, etc. Therapists forge a crucial therapeutic bond with clients, steering them towards positivity. Unfortunately, the massive shortage of professionals, high costs, and mental health stigma pose significant barriers to consulting therapists. As a substitute, Virtual Mental Health Assistants (VMHAs) have emerged in the digital healthcare space. However, most existing VMHAs lack the commonsense to understand the nuanced sentiments of clients to generate effective responses. To this end, we propose EmpRes, a novel sentiment-guided mechanism incorporating commonsense awareness for generating responses. By leveraging foundation models and harnessing commonsense knowledge, EmpRes aims to generate responses that effectively shape the client's sentiment towards positivity. We evaluate the performance of EmpRes on HOPE, a benchmark counseling dataset, and observe a remarkable performance improvement compared to the existing baselines across a suite of qualitative and quantitative metrics. Moreover, our extensive empirical analysis and human evaluation show that the generation ability of EmpRes is well-suited and, in some cases, surpasses the gold standard. Further, we deploy EmpRes as a chat interface for users seeking mental health support. We address the deployed system's effectiveness through an exhaustive user study with a significant positive response. Our findings show that 91% of users find the system effective, 80% express satisfaction, and over 85.45% convey a willingness to continue using the interface and recommend it to others, demonstrating the practical applicability of EmpRes in addressing the pressing challenges of mental health support, emphasizing user feedback, and ethical considerations in a real-world context.
• 摘要：心理健康问题危机正在不断升级。有效的咨询是患有创伤后应激障碍、压力等疾病的个人的重要生命线。治疗师与客户建立重要的治疗纽带，引导他们走向积极。不幸的是，专业人员的严重短缺、高昂的成本和心理健康耻辱感给咨询治疗师带来了重大障碍。作为替代方案，虚拟心理健康助理 (VMHA) 已出现在数字医疗领域。然而，大多数现有的 VMHA 缺乏常识来理解客户的细微情绪，从而产生有效的反应。为此，我们提出了 EmpRes，这是一种新颖的情绪引导机制，结合常识意识来生成反应。通过利用基础模型和利用常识知识，EmpRes 旨在生成有效塑造客户积极情绪的反应。我们评估了 EmpRes 在基准咨询数据集 HOPE 上的表现，并观察到与现有基线相比，在一系列定性和定量指标中性能显着提高。此外，我们广泛的实证分析和人工评估表明，EmpRes 的生成能力非常合适，在某些情况下甚至超过了黄金标准。此外，我们将 EmpRes 部署为寻求心理健康支持的用户的聊天界面。我们通过详尽的用户研究解决了部署系统的有效性问题，并获得了显著的积极响应。我们的研究结果表明，91% 的用户认为该系统有效，80% 的用户表示满意，超过 85.45% 的用户表示愿意继续使用该界面并将其推荐给其他人，证明了 EmpRes 在解决心理健康支持的紧迫挑战、强调用户反馈和现实环境中的道德考虑方面的实际适用性。

Title: LangFair: A Python Package for Assessing Bias and Fairness in Large Language Model Use Cases

• Authors: Dylan Bouchard, Mohit Singh Chauhan, David Skarbrevik, Viren Bajaj, Zeya Ahmad
• Subjects: cs.CL, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03112
• Pdf URL: https://arxiv.org/pdf/2501.03112
• Copy Paste: [[2501.03112]] LangFair: A Python Package for Assessing Bias and Fairness in Large Language Model Use Cases(https://arxiv.org/abs/2501.03112)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have been observed to exhibit bias in numerous ways, potentially creating or worsening outcomes for specific groups identified by protected attributes such as sex, race, sexual orientation, or age. To help address this gap, we introduce LangFair, an open-source Python package that aims to equip LLM practitioners with the tools to evaluate bias and fairness risks relevant to their specific use cases. The package offers functionality to easily generate evaluation datasets, comprised of LLM responses to use-case-specific prompts, and subsequently calculate applicable metrics for the practitioner's use case. To guide in metric selection, LangFair offers an actionable decision framework.
• 摘要：据观察，大型语言模型 (LLM) 以多种方式表现出偏见，可能会对受保护属性（例如性别、种族、性取向或年龄）识别的特定群体造成或恶化结果。为了帮助解决这一差距，我们推出了 LangFair，这是一个开源 Python 包，旨在为 LLM 从业者提供工具来评估与其特定用例相关的偏见和公平风险。该包提供的功能可以轻松生成评估数据集，包括 LLM 对用例特定提示的响应，然后计算适用于从业者用例的指标。为了指导指标选择，LangFair 提供了一个可操作的决策框架。

Title: PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models

• Authors: Mingyang Song, Zhaochen Su, Xiaoye Qu, Jiawei Zhou, Yu Cheng
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03124
• Pdf URL: https://arxiv.org/pdf/2501.03124
• Copy Paste: [[2501.03124]] PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models(https://arxiv.org/abs/2501.03124)
• Keywords: language model, prompt
• Abstract: Process-level Reward Models (PRMs) are crucial for complex reasoning and decision-making tasks, where each intermediate step plays an important role in the reasoning process. Since language models are prone to various types of errors during the reasoning process, PRMs are required to possess nuanced capabilities for detecting various implicit error types in real-world scenarios. However, current benchmarks primarily focus on step correctness, failing to evaluate PRMs' performance systematically. To address this gap, we introduce PRMBench, a process-level benchmark specifically designed to assess the fine-grained error detection capabilities of PRMs. PRMBench comprises 6,216 carefully designed problems and 83,456 step-level labels, evaluating models across multiple dimensions, including simplicity, soundness, and sensitivity. In our experiments on 15 models, spanning both open-source PRMs and closed-source large language models prompted as critic models, we uncover significant weaknesses in current PRMs. These findings underscore the challenges inherent in process-level evaluation and highlight key directions for future research. We hope PRMBench can be a robust bench for advancing research on PRM evaluation and development.
• 摘要：过程级奖励模型 (PRM) 对于复杂的推理和决策任务至关重要，其中每个中间步骤在推理过程中都发挥着重要作用。由于语言模型在推理过程中容易出现各种类型的错误，因此 PRM 需要具备在现实场景中检测各种隐式错误类型的细微能力。然而，当前的基准测试主要关注步骤正确性，未能系统地评估 PRM 的性能。为了解决这一差距，我们引入了 PRMBench，这是一个专门用于评估 PRM 细粒度错误检测能力的过程级基准测试。PRMBench 包含 6,216 个精心设计的问题和 83,456 个步骤级标签，从多个维度评估模型，包括简单性、健全性和敏感性。在我们对 15 个模型进行的实验中，我们涵盖了开源 PRM 和作为批评模型的闭源大型语言模型，我们发现了当前 PRM 中存在重大缺陷。这些发现强调了过程级评估固有的挑战，并强调了未来研究的关键方向。我们希望 PRMBench 能够成为推动 PRM 评估和开发研究的强大平台。

Title: VicSim: Enhancing Victim Simulation with Emotional and Linguistic Fidelity

• Authors: Yerong Li, Yiren Liu, Yun Huang
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2501.03139
• Pdf URL: https://arxiv.org/pdf/2501.03139
• Copy Paste: [[2501.03139]] VicSim: Enhancing Victim Simulation with Emotional and Linguistic Fidelity(https://arxiv.org/abs/2501.03139)
• Keywords: language model, gpt, llm, prompt
• Abstract: Scenario-based training has been widely adopted in many public service sectors. Recent advancements in Large Language Models (LLMs) have shown promise in simulating diverse personas to create these training scenarios. However, little is known about how LLMs can be developed to simulate victims for scenario-based training purposes. In this paper, we introduce VicSim (victim simulator), a novel model that addresses three key dimensions of user simulation: informational faithfulness, emotional dynamics, and language style (e.g., grammar usage). We pioneer the integration of scenario-based victim modeling with GAN-based training workflow and key-information-based prompting, aiming to enhance the realism of simulated victims. Our adversarial training approach teaches the discriminator to recognize grammar and emotional cues as reliable indicators of synthetic content. According to evaluations by human raters, the VicSim model outperforms GPT-4 in terms of human-likeness.
• 摘要：基于场景的培训已在许多公共服务部门得到广泛采用。大型语言模型 (LLM) 的最新进展已显示出模拟各种角色以创建这些训练场景的前景。然而，对于如何开发 LLM 来模拟受害者以进行基于场景的训练，人们知之甚少。在本文中，我们介绍了 VicSim（受害者模拟器），这是一种新颖的模型，它解决了用户模拟的三个关键维度：信息忠实度、情感动态和语言风格（例如语法使用）。我们率先将基于场景的受害者建模与基于 GAN 的训练工作流程和基于关键信息的提示相结合，旨在增强模拟受害者的真实感。我们的对抗性训练方法教会鉴别器识别语法和情感线索作为合成内容的可靠指标。根据人类评分者的评估，VicSim 模型在人类相似性方面优于 GPT-4。

Title: Semantic Captioning: Benchmark Dataset and Graph-Aware Few-Shot In-Context Learning for SQL2Text

• Authors: Ali Al-Lawati, Jason Lucas, Prasenjit Mitra
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03166
• Pdf URL: https://arxiv.org/pdf/2501.03166
• Copy Paste: [[2501.03166]] Semantic Captioning: Benchmark Dataset and Graph-Aware Few-Shot In-Context Learning for SQL2Text(https://arxiv.org/abs/2501.03166)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large Language Models (LLMs) have demonstrated remarkable performance in various NLP tasks, including semantic parsing, which trans lates natural language into formal code representations. However, the reverse process, translating code into natural language, termed semantic captioning, has received less attention. This task is becoming increasingly important as LLMs are integrated into platforms for code generation, security analysis, and educational purposes. In this paper, we focus on the captioning of SQL query (SQL2Text) to address the critical need for understanding and explaining SQL queries in an era where LLM-generated code poses potential security risks. We repurpose Text2SQL datasets for SQL2Text by introducing an iterative ICL prompt using GPT-4o to generate multiple additional utterances, which enhances the robustness of the datasets for the reverse task. We conduct our experiments using in-context learning (ICL) based on different sample selection methods, emphasizing smaller, more computationally efficient LLMs. Our findings demonstrate that leveraging the inherent graph properties of SQL for ICL sample selection significantly outperforms random selection by up to 39% on BLEU score and provides better results than alternative methods. Dataset and codes are published: \url{this https URL}.
• 摘要：大型语言模型 (LLM) 在各种 NLP 任务中表现出色，包括语义解析，将自然语言转换为正式的代码表示。然而，将代码转换为自然语言的反向过程，称为语义字幕，却没有受到太多关注。随着 LLM 被集成到代码生成、安全分析和教育目的的平台中，这项任务变得越来越重要。在本文中，我们专注于 SQL 查询的字幕 (SQL2Text)，以满足在 LLM 生成的代码存在潜在安全风险的时代理解和解释 SQL 查询的迫切需求。我们通过使用 GPT-4o 引入迭代 ICL 提示来生成多个额外的话语，将 Text2SQL 数据集重新用于 SQL2Text，这增强了数据集对于反向任务的鲁棒性。我们使用基于不同样本选择方法的上下文学习 (ICL) 进行实验，强调更小、计算效率更高的 LLM。我们的研究结果表明，利用 SQL 固有的图形属性进行 ICL 样本选择，其 BLEU 得分显著优于随机选择，最高可达 39%，并且比其他方法提供更好的结果。数据集和代码已发布：\url{此 https URL}。

Title: Boosting Explainability through Selective Rationalization in Pre-trained Language Models

• Authors: Libing Yuan, Shuaibo Hu, Kui Yu, Le Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.03182
• Pdf URL: https://arxiv.org/pdf/2501.03182
• Copy Paste: [[2501.03182]] Boosting Explainability through Selective Rationalization in Pre-trained Language Models(https://arxiv.org/abs/2501.03182)
• Keywords: language model
• Abstract: The widespread application of pre-trained language models (PLMs) in natural language processing (NLP) has led to increasing concerns about their explainability. Selective rationalization is a self-explanatory framework that selects human-intelligible input subsets as rationales for predictions. Recent studies have shown that applying existing rationalization frameworks to PLMs will result in severe degeneration and failure problems, producing sub-optimal or meaningless rationales. Such failures severely damage trust in rationalization methods and constrain the application of rationalization techniques on PLMs. In this paper, we find that the homogeneity of tokens in the sentences produced by PLMs is the primary contributor to these problems. To address these challenges, we propose a method named Pre-trained Language Model's Rationalization (PLMR), which splits PLMs into a generator and a predictor to deal with NLP tasks while providing interpretable rationales. The generator in PLMR also alleviates homogeneity by pruning irrelevant tokens, while the predictor uses full-text information to standardize predictions. Experiments conducted on two widely used datasets across multiple PLMs demonstrate the effectiveness of the proposed method PLMR in addressing the challenge of applying selective rationalization to PLMs. Codes: this https URL.
• 摘要：预训练语言模型 (PLM) 在自然语言处理 (NLP) 中的广泛应用导致人们越来越担心其可解释性。选择性合理化是一个自解释框架，它选择人类可理解的输入子集作为预测的原理。最近的研究表明，将现有的合理化框架应用于 PLM 将导致严重的退化和故障问题，产生次优或毫无意义的原理。此类故障严重损害了对合理化方法的信任，并限制了合理化技术在 PLM 上的应用。在本文中，我们发现 PLM 生成的句子中标记的同质性是导致这些问题的主要原因。为了应对这些挑战，我们提出了一种名为预训练语言模型合理化 (PLMR) 的方法，它将 PLM 拆分为生成器和预测器以处理 NLP 任务，同时提供可解释的原理。PLMR 中的生成器还通过修剪不相关的标记来缓解同质性，而预测器使用全文信息来标准化预测。在多个 PLM 中对两个广泛使用的数据集进行的实验证明了所提出的方法 PLMR 在解决将选择性合理化应用于 PLM 的挑战方面的有效性。代码：此 https URL。

Title: Classifier-Guided Captioning Across Modalities

• Authors: Ariel Shaulov, Tal Shaharabany, Eitan Shaar, Gal Chechik, Lior Wolf
• Subjects: cs.CL, cs.AI, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.03183
• Pdf URL: https://arxiv.org/pdf/2501.03183
• Copy Paste: [[2501.03183]] Classifier-Guided Captioning Across Modalities(https://arxiv.org/abs/2501.03183)
• Keywords: language model, gpt, prompt
• Abstract: Most current captioning systems use language models trained on data from specific settings, such as image-based captioning via Amazon Mechanical Turk, limiting their ability to generalize to other modality distributions and contexts. This limitation hinders performance in tasks like audio or video captioning, where different semantic cues are needed. Addressing this challenge is crucial for creating more adaptable and versatile captioning frameworks applicable across diverse real-world contexts. In this work, we introduce a method to adapt captioning networks to the semantics of alternative settings, such as capturing audibility in audio captioning, where it is crucial to describe sounds and their sources. Our framework consists of two main components: (i) a frozen captioning system incorporating a language model (LM), and (ii) a text classifier that guides the captioning system. The classifier is trained on a dataset automatically generated by GPT-4, using tailored prompts specifically designed to enhance key aspects of the generated captions. Importantly, the framework operates solely during inference, eliminating the need for further training of the underlying captioning model. We evaluate the framework on various models and modalities, with a focus on audio captioning, and report promising results. Notably, when combined with an existing zero-shot audio captioning system, our framework improves its quality and sets state-of-the-art performance in zero-shot audio captioning.
• 摘要：目前大多数字幕系统都使用基于特定设置的数据训练的语言模型，例如通过 Amazon Mechanical Turk 进行的图像字幕，这限制了它们推广到其他模态分布和上下文的能力。这种限制阻碍了音频或视频字幕等任务的性能，因为这些任务需要不同的语义提示。解决这一挑战对于创建适用于各种现实世界环境的适应性更强、用途更广的字幕框架至关重要。在这项工作中，我们介绍了一种方法，使字幕网络适应替代设置的语义，例如在音频字幕中捕捉可听性，其中描述声音及其来源至关重要。我们的框架由两个主要组件组成：(i) 包含语言模型 (LM) 的冻结字幕系统，以及 (ii) 指导字幕系统的文本分类器。分类器在 GPT-4 自动生成的数据集上进行训练，使用专门为增强生成字幕的关键方面而设计的定制提示。重要的是，该框架仅在推理期间运行，无需进一步训练底层字幕模型。我们在各种模型和模式下评估了该框架，重点关注音频字幕，并报告了令人鼓舞的结果。值得注意的是，当与现有的零样本音频字幕系统结合使用时，我们的框架可以提高其质量并在零样本音频字幕中树立最佳性能。

Title: CLIX: Cross-Lingual Explanations of Idiomatic Expressions

• Authors: Aaron Gluck, Katharina von der Wense, Maria Pacheco
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03191
• Pdf URL: https://arxiv.org/pdf/2501.03191
• Copy Paste: [[2501.03191]] CLIX: Cross-Lingual Explanations of Idiomatic Expressions(https://arxiv.org/abs/2501.03191)
• Keywords: language model
• Abstract: Automated definition generation systems have been proposed to support vocabulary expansion for language learners. The main barrier to the success of these systems is that learners often struggle to understand definitions due to the presence of potentially unfamiliar words and grammar, particularly when non-standard language is involved. To address these challenges, we propose CLIX, the task of Cross-Lingual explanations of Idiomatic eXpressions. We explore the capabilities of current NLP models for this task, and observe that while it remains challenging, large language models show promise. Finally, we perform a detailed error analysis to highlight the key challenges that need to be addressed before we can reliably incorporate these systems into educational tools.
• 摘要：已经有人提出使用自动定义生成系统来支持语言学习者的词汇扩展。这些系统成功的主要障碍是，由于可能存在不熟悉的单词和语法，学习者往往很难理解定义，尤其是涉及非标准语言时。为了应对这些挑战，我们提出了 CLIX，即跨语言解释惯用表达的任务。我们探索了当前 NLP 模型执行此任务的能力，并观察到虽然它仍然具有挑战性，但大型语言模型显示出前景。最后，我们进行了详细的错误分析，以强调在将这些系统可靠地纳入教育工具之前需要解决的关键挑战。

Title: The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input

• Authors: Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska, Lukas Haas, Michelle Liu, Nate Keating, Adam Bloniarz, Carl Saroufim, Corey Fry, Dror Marcus, Doron Kukliansky, Gaurav Singh Tomar, James Swirhun, Jinwei Xing, Lily Wang, Madhu Gurumurthy, Michael Aaron, Moran Ambar, Rachana Fellinger, Rui Wang, Zizhao Zhang, Sasha Goldshtein, Dipanjan Das
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.03200
• Pdf URL: https://arxiv.org/pdf/2501.03200
• Copy Paste: [[2501.03200]] The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input(https://arxiv.org/abs/2501.03200)
• Keywords: language model, llm, prompt
• Abstract: We introduce FACTS Grounding, an online leaderboard and associated benchmark that evaluates language models' ability to generate text that is factually accurate with respect to given context in the user prompt. In our benchmark, each prompt includes a user request and a full document, with a maximum length of 32k tokens, requiring long-form responses. The long-form responses are required to be fully grounded in the provided context document while fulfilling the user request. Models are evaluated using automated judge models in two phases: (1) responses are disqualified if they do not fulfill the user request; (2) they are judged as accurate if the response is fully grounded in the provided document. The automated judge models were comprehensively evaluated against a held-out test-set to pick the best prompt template, and the final factuality score is an aggregate of multiple judge models to mitigate evaluation bias. The FACTS Grounding leaderboard will be actively maintained over time, and contains both public and private splits to allow for external participation while guarding the integrity of the leaderboard. It can be found at this https URL.
• 摘要：我们引入了 FACTS Grounding，这是一个在线排行榜和相关基准，用于评估语言模型生成与用户提示中给定上下文相关的事实准确文本的能力。在我们的基准中，每个提示都包含一个用户请求和一个完整文档，最大长度为 32k 个标记，需要长格式响应。长格式响应需要在满足用户请求的同时完全基于提供的上下文文档。使用自动判断模型分两个阶段评估模型：(1) 如果响应不符合用户请求，则取消其资格；(2) 如果响应完全基于提供的文档，则判断其准确。根据保留的测试集对自动判断模型进行了全面评估，以选出最佳提示模板，最终的事实性分数是多个判断模型的总和，以减轻评估偏差。FACTS Grounding 排行榜将随着时间的推移得到积极维护，并包含公共和私人分割，以允许外部参与，同时保护排行榜的完整性。可以在此 https URL 中找到它。

Title: Detecting AI-Generated Text in Educational Content: Leveraging Machine Learning and Explainable AI for Academic Integrity

• Authors: Ayat A. Najjar, Huthaifa I. Ashqar, Omar A. Darwish, Eman Hammad
• Subjects: cs.CL, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.03203
• Pdf URL: https://arxiv.org/pdf/2501.03203
• Copy Paste: [[2501.03203]] Detecting AI-Generated Text in Educational Content: Leveraging Machine Learning and Explainable AI for Academic Integrity(https://arxiv.org/abs/2501.03203)
• Keywords: language model, gpt, llm, chat
• Abstract: This study seeks to enhance academic integrity by providing tools to detect AI-generated content in student work using advanced technologies. The findings promote transparency and accountability, helping educators maintain ethical standards and supporting the responsible integration of AI in education. A key contribution of this work is the generation of the CyberHumanAI dataset, which has 1000 observations, 500 of which are written by humans and the other 500 produced by ChatGPT. We evaluate various machine learning (ML) and deep learning (DL) algorithms on the CyberHumanAI dataset comparing human-written and AI-generated content from Large Language Models (LLMs) (i.e., ChatGPT). Results demonstrate that traditional ML algorithms, specifically XGBoost and Random Forest, achieve high performance (83% and 81% accuracies respectively). Results also show that classifying shorter content seems to be more challenging than classifying longer content. Further, using Explainable Artificial Intelligence (XAI) we identify discriminative features influencing the ML model's predictions, where human-written content tends to use a practical language (e.g., use and allow). Meanwhile AI-generated text is characterized by more abstract and formal terms (e.g., realm and employ). Finally, a comparative analysis with GPTZero show that our narrowly focused, simple, and fine-tuned model can outperform generalized systems like GPTZero. The proposed model achieved approximately 77.5% accuracy compared to GPTZero's 48.5% accuracy when tasked to classify Pure AI, Pure Human, and mixed class. GPTZero showed a tendency to classify challenging and small-content cases as either mixed or unrecognized while our proposed model showed a more balanced performance across the three classes.
• 摘要：本研究旨在通过使用先进技术提供工具来检测学生作业中 AI 生成的内容，从而提高学术诚信度。研究结果促进了透明度和问责制，帮助教育工作者保持道德标准，并支持负责任地将 AI 融入教育。这项工作的一个关键贡献是生成了 Cyber​​HumanAI 数据集，该数据集有 1000 个观测值，其中 500 个由人类编写，另外 500 个由 ChatGPT 生成。我们在 Cyber​​HumanAI 数据集上评估了各种机器学习 (ML) 和深度学习 (DL) 算法，比较了大型语言模型 (LLM)（即 ChatGPT）中人类编写的内容和 AI 生成的内容。结果表明，传统的 ML 算法，特别是 XGBoost 和随机森林，实现了高性能（准确率分别为 83% 和 81%）。结果还表明，对较短内容进行分类似乎比对较长内容进行分类更具挑战性。此外，使用可解释人工智能 (XAI)，我们可以识别影响 ML 模型预测的判别特征，其中人类编写的内容倾向于使用实用语言（例如，使用和允许）。同时，人工智能生成的文本以更抽象和正式的术语为特征（例如，领域和雇用）。最后，与 GPTZero 的比较分析表明，我们专注、简单且经过微调的模型可以胜过 GPTZero 等通用系统。在对纯人工智能、纯人类和混合类进行分类时，所提出的模型的准确率约为 77.5%，而 GPTZero 的准确率仅为 48.5%。GPTZero 倾向于将具有挑战性和小内容的案例归类为混合或无法识别，而我们提出的模型在这三个类别中表现出更均衡的性能。

Title: Leveraging Explainable AI for LLM Text Attribution: Differentiating Human-Written and Multiple LLMs-Generated Text

• Authors: Ayat Najjar, Huthaifa I. Ashqar, Omar Darwish, Eman Hammad
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.03212
• Pdf URL: https://arxiv.org/pdf/2501.03212
• Copy Paste: [[2501.03212]] Leveraging Explainable AI for LLM Text Attribution: Differentiating Human-Written and Multiple LLMs-Generated Text(https://arxiv.org/abs/2501.03212)
• Keywords: language model, gpt, llm, chat
• Abstract: The development of Generative AI Large Language Models (LLMs) raised the alarm regarding identifying content produced through generative AI or humans. In one case, issues arise when students heavily rely on such tools in a manner that can affect the development of their writing or coding skills. Other issues of plagiarism also apply. This study aims to support efforts to detect and identify textual content generated using LLM tools. We hypothesize that LLMs-generated text is detectable by machine learning (ML), and investigate ML models that can recognize and differentiate texts generated by multiple LLMs tools. We leverage several ML and Deep Learning (DL) algorithms such as Random Forest (RF), and Recurrent Neural Networks (RNN), and utilized Explainable Artificial Intelligence (XAI) to understand the important features in attribution. Our method is divided into 1) binary classification to differentiate between human-written and AI-text, and 2) multi classification, to differentiate between human-written text and the text generated by the five different LLM tools (ChatGPT, LLaMA, Google Bard, Claude, and Perplexity). Results show high accuracy in the multi and binary classification. Our model outperformed GPTZero with 98.5\% accuracy to 78.3\%. Notably, GPTZero was unable to recognize about 4.2\% of the observations, but our model was able to recognize the complete test dataset. XAI results showed that understanding feature importance across different classes enables detailed author/source profiles. Further, aiding in attribution and supporting plagiarism detection by highlighting unique stylistic and structural elements ensuring robust content originality verification.
• 摘要：生成式人工智能大型语言模型 (LLM) 的发展敲响了警钟，提醒人们如何识别通过生成式人工智能或人类生成的内容。在一种情况下，当学生过度依赖此类工具时，问题就会出现，这可能会影响他们写作或编码技能的发展。其他抄袭问题也适用。本研究旨在支持检测和识别使用 LLM 工具生成的文本内容的努力。我们假设 LLM 生成的文本可以通过机器学习 (ML) 检测，并研究可以识别和区分由多个 LLM 工具生成的文本的 ML 模型。我们利用多种 ML 和深度学习 (DL) 算法，例如随机森林 (RF) 和循环神经网络 (RNN)，并利用可解释人工智能 (XAI) 来了解归因中的重要特征。我们的方法分为 1) 二分类，以区分人工书写文本和 AI 文本；2) 多分类，以区分人工书写文本和五种不同 LLM 工具（ChatGPT、LLaMA、Google Bard、Claude 和 Perplexity）生成的文本。结果显示，多分类和二分类的准确率很高。我们的模型表现优于 GPTZero，准确率为 98.5% 对 78.3%。值得注意的是，GPTZero 无法识别约 4.2% 的观察结果，但我们的模型能够识别完整的测试数据集。XAI 结果表明，了解不同类别中特征的重要性可以实现详细的作者/来源资料。此外，通过突出显示独特的风格和结构元素，帮助归因并支持抄袭检测，确保强大的内容原创性验证。

Title: BoostStep: Boosting mathematical capability of Large Language Models via improved single-step reasoning

• Authors: Beichen Zhang, Yuhong Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Haodong Duan, Yuhang Cao, Dahua Lin, Jiaqi Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.03226
• Pdf URL: https://arxiv.org/pdf/2501.03226
• Copy Paste: [[2501.03226]] BoostStep: Boosting mathematical capability of Large Language Models via improved single-step reasoning(https://arxiv.org/abs/2501.03226)
• Keywords: language model, gpt, llm
• Abstract: Cutting-edge large language models (LLMs) demonstrate promising performance in solving complex math problems with a divide-and-conquer pipeline and the assistance of in-context learning (ICL) examples. However, their potential for improvement is limited by two critical problems within their ICL examples: granularity-mismatch and the ensuing negative-effect noise problem. Specifically, the LLMs are capable of the dividing process yet mostly failed by inaccurate reasoning within a few conquer steps, while the ICL examples retrieved in question-grained sometimes lack relevant steps for a specific challenging reasoning step. Further, this disconnect may hinder the correct reasoning due to its irrelevance. To this end, we focus on improving the reasoning quality within each step and present BoostStep. BoostStep aligns the granularity between the retrieving and reasoning on step grained, and provides highly related ICL examples for each reasoning step with a novel `first-try' strategy. BoostStep provides more relevant examples than the coarse question-grained strategy, enhancing the model reasoning quality within each step steadily. BoostStep is a general and robust reasoning-enhancing method that not only improves standalone reasoning performance but also integrates seamlessly with Monte Carlo Tree Search methods (MCTS) to refine both candidate generation and decision-making. Quantitatively, it improves GPT-4o and Qwen2.5-Math-72B by 3.6\% and 2.0\% respectively on various mathematical benchmarks, and 7.5\% gain combined with MCTS.
• 摘要：先进的大型语言模型 (LLM) 在利用分治流水线和上下文学习 (ICL) 示例的帮助解决复杂数学问题方面表现出色。然而，它们的改进潜力受到 ICL 示例中两个关键问题的限制：粒度不匹配和随之而来的负面影响噪声问题。具体而言，LLM 能够进行分治过程，但大多因几个征服步骤内的不准确推理而失败，而问题粒度中检索到的 ICL 示例有时缺少特定具有挑战性的推理步骤的相关步骤。此外，由于其不相关性，这种脱节可能会阻碍正确的推理。为此，我们专注于提高每个步骤中的推理质量并提出了 BoostStep。BoostStep 在每个步骤粒度上对齐检索和推理之间的粒度，并为每个推理步骤提供高度相关的 ICL 示例，并采用新颖的“首次尝试”策略。与粗粒度问题策略相比，BoostStep 提供了更多相关示例，从而稳步提升了每个步骤中的模型推理质量。BoostStep 是一种通用且强大的推理增强方法，不仅可以提高独立推理性能，还可以与蒙特卡洛树搜索方法 (MCTS) 无缝集成，以改进候选生成和决策。从数量上看，它在各种数学基准上分别将 GPT-4o 和 Qwen2.5-Math-72B 提高了 3.6% 和 2.0%，与 MCTS 结合后提高了 7.5%。