2025-01-20

Title: Enhancing the De-identification of Personally Identifiable Information in Educational Data

• Authors: Y. Shen, Z. Ji, J. Lin, K. R. Koedginer
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.09765
• Pdf URL: https://arxiv.org/pdf/2501.09765
• Copy Paste: [[2501.09765]] Enhancing the De-identification of Personally Identifiable Information in Educational Data(https://arxiv.org/abs/2501.09765)
• Keywords: gpt, prompt
• Abstract: Protecting Personally Identifiable Information (PII), such as names, is a critical requirement in learning technologies to safeguard student and teacher privacy and maintain trust. Accurate PII detection is an essential step toward anonymizing sensitive information while preserving the utility of educational data. Motivated by recent advancements in artificial intelligence, our study investigates the GPT-4o-mini model as a cost-effective and efficient solution for PII detection tasks. We explore both prompting and fine-tuning approaches and compare GPT-4o-mini's performance against established frameworks, including Microsoft Presidio and Azure AI Language. Our evaluation on two public datasets, CRAPII and TSCC, demonstrates that the fine-tuned GPT-4o-mini model achieves superior performance, with a recall of 0.9589 on CRAPII. Additionally, fine-tuned GPT-4o-mini significantly improves precision scores (a threefold increase) while reducing computational costs to nearly one-tenth of those associated with Azure AI Language. Furthermore, our bias analysis reveals that the fine-tuned GPT-4o-mini model consistently delivers accurate results across diverse cultural backgrounds and genders. The generalizability analysis using the TSCC dataset further highlights its robustness, achieving a recall of 0.9895 with minimal additional training data from TSCC. These results emphasize the potential of fine-tuned GPT-4o-mini as an accurate and cost-effective tool for PII detection in educational data. It offers robust privacy protection while preserving the data's utility for research and pedagogical analysis. Our code is available on GitHub: this https URL
• 摘要：保护个人身份信息 (PII)（例如姓名）是学习技术中保护学生和教师隐私并维持信任的关键要求。准确的 PII 检测是匿名化敏感信息同时保留教育数据实用性的重要一步。受人工智能最新进展的推动，我们的研究调查了 GPT-4o-mini 模型作为一种经济高效的 PII 检测任务解决方案。我们探索了提示和微调方法，并将 GPT-4o-mini 的性能与现有框架（包括 Microsoft Presidio 和 Azure AI Language）进行了比较。我们对两个公共数据集 CRAPII 和 TSCC 的评估表明，经过微调的 GPT-4o-mini 模型实现了卓越的性能，CRAPII 的召回率为 0.9589。此外，经过微调的 GPT-4o-mini 显着提高了准确率（增加了三倍），同时将计算成本降低到与 Azure AI Language 相关的计算成本的近十分之一。此外，我们的偏差分析表明，经过微调的 GPT-4o-mini 模型在不同文化背景和性别之间始终提供准确的结果。使用 TSCC 数据集的通用性分析进一步凸显了其稳健性，在 TSCC 的极少额外训练数据的情况下实现了 0.9895 的召回率。这些结果强调了经过微调的 GPT-4o-mini 作为一种准确且经济高效的教育数据 PII 检测工具的潜力。它提供了强大的隐私保护，同时保留了数据对研究和教学分析的实用性。我们的代码可在 GitHub 上找到：此 https URL

Title: Boosting Tool Use of Large Language Models via Iterative Reinforced Fine-Tuning

• Authors: Yirong Zeng, Xiao Ding, Yuxian Wang, Weiwen Liu, Wu Ning, Yutai Hou, Xu Huang, Bing Qin, Ting Liu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.09766
• Pdf URL: https://arxiv.org/pdf/2501.09766
• Copy Paste: [[2501.09766]] Boosting Tool Use of Large Language Models via Iterative Reinforced Fine-Tuning(https://arxiv.org/abs/2501.09766)
• Keywords: language model, llm
• Abstract: Augmenting large language models (LLMs) with external tools is a promising approach to enhance their capabilities. Effectively leveraging this potential for complex tasks hinges crucially on improving their ability to use tools. Synthesizing tool use data by simulating the real world is an effective approach. Nevertheless, our investigation reveals that training gains significantly decay as the scale of these data increases. The primary factor is the model's poor performance (a.k.a deficiency) in complex scenarios, which hinders learning from data using SFT. Driven by this objective, we propose an iterative reinforced fine-tuning strategy to continually guide the model to alleviate it. Specifically, we first identify deficiency-related data based on feedback from the policy model, then perform a Monte Carlo Tree Search to collect fine-grained preference pairs to pinpoint deficiencies. Subsequently, we update the policy model using preference optimization to align with ground truth and misalign with deficiencies. This process can be iterated. Moreover, before the iteration, we propose an easy-to-hard warm-up SFT strategy to facilitate learning from challenging data. The experiments demonstrate our models go beyond the same parametric models, outperforming many larger open-source and closed-source models. Additionally, it has achieved notable training gains in complex tool use scenarios.
• 摘要：使用外部工具增强大型语言模型 (LLM) 是一种增强其能力的有效方法。有效利用这种潜力完成复杂任务的关键在于提高它们使用工具的能力。通过模拟现实世界来合成工具使用数据是一种有效的方法。然而，我们的调查显示，随着这些数据规模的增加，训练收益会显著下降。主要因素是模型在复杂场景中的表现不佳（即缺陷），这阻碍了使用 SFT 从数据中学习。在这一目标的驱动下，我们提出了一种迭代强化微调策略，以不断指导模型缓解这一问题。具体来说，我们首先根据策略模型的反馈识别与缺陷相关的数据，然后执行蒙特卡洛树搜索以收集细粒度的偏好对以查明缺陷。随后，我们使用偏好优化来更新策略模型，使其与基本事实保持一致并与缺陷不一致。这个过程可以迭代。此外，在迭代之前，我们提出了一种由易到难的热身 SFT 策略，以促进从具有挑战性的数据中学习。实验表明，我们的模型超越了相同的参数模型，优于许多较大的开源和闭源模型。此外，它在复杂的工具使用场景中取得了显著的训练效果。

Title: LeMo: Enabling LEss Token Involvement for MOre Context Fine-tuning

• Authors: Tuowei Wang, Xingyu Chen, Kun Li, Ting Cao, Ju Ren, Yaoxue Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.09767
• Pdf URL: https://arxiv.org/pdf/2501.09767
• Copy Paste: [[2501.09767]] LeMo: Enabling LEss Token Involvement for MOre Context Fine-tuning(https://arxiv.org/abs/2501.09767)
• Keywords: llm
• Abstract: The escalating demand for long-context applications has intensified the necessity of extending the LLM context windows. Despite recent fine-tuning approaches successfully expanding context lengths, their high memory footprints, especially for activations, present a critical practical limitation. Current parameter-efficient fine-tuning methods prioritize reducing parameter update overhead over addressing activation memory constraints. Similarly, existing sparsity mechanisms improve computational efficiency but overlook activation memory optimization due to the phenomenon of Shadowy Activation. In this paper, we propose LeMo, the first LLM fine-tuning system that explores and exploits a new token-level sparsity mechanism inherent in long-context scenarios, termed Contextual Token Sparsity. LeMo minimizes redundant token involvement by assessing the informativeness of token embeddings while preserving model accuracy. Specifically, LeMo introduces three key techniques: (1) Token Elimination, dynamically identifying and excluding redundant tokens across varying inputs and layers. (2) Pattern Prediction, utilizing well-trained predictors to approximate token sparsity patterns with minimal overhead. (3) Kernel Optimization, employing permutation-free and segment-based strategies to boost system performance. We implement LeMo as an end-to-end fine-tuning system compatible with various LLM architectures and other optimization techniques. Comprehensive evaluations demonstrate that LeMo reduces memory consumption by up to 1.93x and achieves up to 1.36x speedups, outperforming state-of-the-art fine-tuning systems.
• 摘要：对长上下文应用的需求不断增加，这加剧了扩展 LLM 上下文窗口的必要性。尽管最近的微调方法成功地扩展了上下文长度，但它们的高内存占用（尤其是对于激活而言）带来了关键的实际限制。当前参数高效的微调方法优先考虑减少参数更新开销，而不是解决激活内存限制。同样，现有的稀疏机制提高了计算效率，但由于 Shadowy Activation 现象而忽略了激活内存优化。在本文中，我们提出了 LeMo，这是第一个探索和利用长上下文场景中固有的新的 token 级稀疏机制（称为上下文 token 稀疏）的 LLM 微调系统。LeMo 通过评估 token 嵌入的信息量来最大限度地减少冗余 token 的参与，同时保持模型准确性。具体来说，LeMo 引入了三种关键技术：(1) Token Elimination，动态识别和排除不同输入和层中的冗余 token。(2) 模式预测，利用训练有素的预测器以最小的开销近似 token 稀疏模式。 （3）内核优化，采用无置换和基于段的策略来提高系统性能。我们将 LeMo 实现为一个端到端微调系统，与各种 LLM 架构和其他优化技术兼容。综合评估表明，LeMo 将内存消耗降低了 1.93 倍，速度提高了 1.36 倍，优于最先进的微调系统。

Title: Can Large Language Models Predict the Outcome of Judicial Decisions?

• Authors: Mohamed Bayan Kmainasi, Ali Ezzat Shahroor, Amani Al-Ghraibah
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.09768
• Pdf URL: https://arxiv.org/pdf/2501.09768
• Copy Paste: [[2501.09768]] Can Large Language Models Predict the Outcome of Judicial Decisions?(https://arxiv.org/abs/2501.09768)
• Keywords: language model, llm, prompt
• Abstract: Large Language Models (LLMs) have shown exceptional capabilities in Natural Language Processing (NLP) across diverse domains. However, their application in specialized tasks such as Legal Judgment Prediction (LJP) for low-resource languages like Arabic remains underexplored. In this work, we address this gap by developing an Arabic LJP dataset, collected and preprocessed from Saudi commercial court judgments. We benchmark state-of-the-art open-source LLMs, including LLaMA-3.2-3B and LLaMA-3.1-8B, under varying configurations such as zero-shot, one-shot, and fine-tuning using QLoRA. Additionally, we used a comprehensive evaluation framework combining quantitative metrics (BLEU and ROUGE) and qualitative assessments (Coherence, legal language, clarity). Our results demonstrate that fine-tuned smaller models achieve comparable performance to larger models in task-specific contexts while offering significant resource efficiency. Furthermore, we investigate the effects of prompt engineering and fine-tuning on model outputs, providing insights into performance variability and instruction sensitivity. By making the dataset, implementation code, and models publicly available, we establish a robust foundation for future research in Arabic legal NLP.
• 摘要：大型语言模型 (LLM) 已在不同领域的自然语言处理 (NLP) 中展现出卓越的能力。然而，它们在针对阿拉伯语等资源匮乏的语言的法律判决预测 (LJP) 等专门任务中的应用仍未得到充分探索。在这项工作中，我们通过开发一个阿拉伯语 LJP 数据集来解决这一差距，该数据集是从沙特商业法院判决中收集和预处理的。我们在零样本、单样本和使用 QLoRA 进行微调等不同配置下对最先进的开源 LLM（包括 LLaMA-3.2-3B 和 LLaMA-3.1-8B）进行了基准测试。此外，我们使用了一个综合评估框架，结合了定量指标（BLEU 和 ROUGE）和定性评估（连贯性、法律语言、清晰度）。我们的结果表明，经过微调的小型模型在特定于任务的环境中实现了与大型模型相当的性能，同时提供了显着的资源效率。此外，我们研究了快速工程和微调对模型输出的影响，从而深入了解了性能变化和指令敏感性。通过公开数据集、实现代码和模型，我们为未来阿拉伯法律 NLP 的研究奠定了坚实的基础。

Title: Multiple Choice Questions: Reasoning Makes Large Language Models (LLMs) More Self-Confident Even When They Are Wrong

• Authors: Tairan Fu, Javier Conde, Gonzalo Martínez, María Grandury, Pedro Reviriego
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.09775
• Pdf URL: https://arxiv.org/pdf/2501.09775
• Copy Paste: [[2501.09775]] Multiple Choice Questions: Reasoning Makes Large Language Models (LLMs) More Self-Confident Even When They Are Wrong(https://arxiv.org/abs/2501.09775)
• Keywords: language model, llm, prompt
• Abstract: One of the most widely used methods to evaluate LLMs are Multiple Choice Question (MCQ) tests. MCQ benchmarks enable the testing of LLM knowledge on almost any topic at scale as the results can be processed automatically. To help the LLM answer, a few examples called few shots can be included in the prompt. Moreover, the LLM can be asked to answer the question directly with the selected option or to first provide the reasoning and then the selected answer, which is known as chain of thought. In addition to checking whether the selected answer is correct, the evaluation can look at the LLM-estimated probability of its response as an indication of the confidence of the LLM in the response. In this paper, we study how the LLM confidence in its answer depends on whether the model has been asked to answer directly or to provide the reasoning before answering. The results of the evaluation of questions on a wide range of topics in seven different models show that LLMs are more confident in their answers when they provide reasoning before the answer. This occurs regardless of whether the selected answer is correct. Our hypothesis is that this behavior is due to the reasoning that modifies the probability of the selected answer, as the LLM predicts the answer based on the input question and the reasoning that supports the selection made. Therefore, LLM estimated probabilities seem to have intrinsic limitations that should be understood in order to use them in evaluation procedures. Interestingly, the same behavior has been observed in humans, for whom explaining an answer increases confidence in its correctness.
• 摘要：评估法学硕士最广泛使用的方法之一是多项选择题 (MCQ) 测试。MCQ 基准测试可以大规模测试法学硕士对几乎任何主题的知识，因为结果可以自动处理。为了帮助法学硕士回答问题，可以在提示中包含一些称为“few shots”的示例。此外，可以要求法学硕士直接使用所选选项回答问题，或者先提供推理，然后再提供所选答案，这被称为思路链。除了检查所选答案是否正确之外，评估还可以查看法学硕士对其回答的估计概率，作为法学硕士对回答的信心的指标。在本文中，我们研究了法学硕士对其答案的信心如何取决于模型是否被要求直接回答或在回答之前提供推理。对七种不同模型中广泛主题的问题的评估结果表明，如果法学硕士在回答之前提供推理，他们对答案更有信心。无论所选答案是否正确，都会发生这种情况。我们的假设是，这种行为是由于推理改变了所选答案的概率，因为 LLM 根据输入问题和支持所做选择的推理来预测答案。因此，LLM 估计的概率似乎具有内在的局限性，应该理解这些局限性才能在评估过程中使用它们。有趣的是，在人类身上也观察到了同样的行为，对人类来说，解释答案会增加对其正确性的信心。

Title: Qwen it detect machine-generated text?

• Authors: Teodor-George Marchitan, Claudiu Creanga, Liviu P. Dinu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.09813
• Pdf URL: https://arxiv.org/pdf/2501.09813
• Copy Paste: [[2501.09813]] Qwen it detect machine-generated text?(https://arxiv.org/abs/2501.09813)
• Keywords: language model
• Abstract: This paper describes the approach of the Unibuc - NLP team in tackling the Coling 2025 GenAI Workshop, Task 1: Binary Multilingual Machine-Generated Text Detection. We explored both masked language models and causal models. For Subtask A, our best model achieved first-place out of 36 teams when looking at F1 Micro (Auxiliary Score) of 0.8333, and second-place when looking at F1 Macro (Main Score) of 0.8301
• 摘要：本文介绍了 Unibuc - NLP 团队在解决 Coling 2025 GenAI 研讨会任务 1：二进制多语言机器生成文本检测方面的方法。我们探索了掩码语言模型和因果模型。对于子任务 A，我们的最佳模型在 36 个团队中排名第一，F1 Micro（辅助分数）为 0.8333，在 F1 Macro（主分数）为 0.8301 时排名第二

Title: Bridging Language Barriers in Healthcare: A Study on Arabic LLMs

• Authors: Nada Saadi, Tathagata Raha, Clément Christophe, Marco AF Pimentel, Ronnie Rajan, Praveen K Kanithi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.09825
• Pdf URL: https://arxiv.org/pdf/2501.09825
• Copy Paste: [[2501.09825]] Bridging Language Barriers in Healthcare: A Study on Arabic LLMs(https://arxiv.org/abs/2501.09825)
• Keywords: language model, llm
• Abstract: This paper investigates the challenges of developing large language models (LLMs) proficient in both multilingual understanding and medical knowledge. We demonstrate that simply translating medical data does not guarantee strong performance on clinical tasks in the target language. Our experiments reveal that the optimal language mix in training data varies significantly across different medical tasks. We find that larger models with carefully calibrated language ratios achieve superior performance on native-language clinical tasks. Furthermore, our results suggest that relying solely on fine-tuning may not be the most effective approach for incorporating new language knowledge into LLMs. Instead, data and computationally intensive pretraining methods may still be necessary to achieve optimal performance in multilingual medical settings. These findings provide valuable guidance for building effective and inclusive medical AI systems for diverse linguistic communities.
• 摘要：本文探讨了开发精通多语言理解和医学知识的大型语言模型 (LLM) 所面临的挑战。我们证明，仅仅翻译医学数据并不能保证在目标语言的临床任务上取得优异的表现。我们的实验表明，训练数据中的最佳语言组合在不同的医疗任务中存在显著差异。我们发现，具有精心校准的语言比率的大型模型在母语临床任务上取得了优异的表现。此外，我们的结果表明，仅仅依靠微调可能不是将新语言知识融入 LLM 的最有效方法。相反，数据和计算密集型预训练方法可能仍然是在多语言医疗环境中实现最佳性能所必需的。这些发现为为不同语言社区构建有效且包容的医疗 AI 系统提供了宝贵的指导。

Title: Dialogue Benchmark Generation from Knowledge Graphs with Cost-Effective Retrieval-Augmented LLMs

• Authors: Reham Omar, Omij Mangukiya, Essam Mansour
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.09928
• Pdf URL: https://arxiv.org/pdf/2501.09928
• Copy Paste: [[2501.09928]] Dialogue Benchmark Generation from Knowledge Graphs with Cost-Effective Retrieval-Augmented LLMs(https://arxiv.org/abs/2501.09928)
• Keywords: gpt, llm, hallucination, chat, retrieval-augmented generation
• Abstract: Dialogue benchmarks are crucial in training and evaluating chatbots engaging in domain-specific conversations. Knowledge graphs (KGs) represent semantically rich and well-organized data spanning various domains, such as DBLP, DBpedia, and YAGO. Traditionally, dialogue benchmarks have been manually created from documents, neglecting the potential of KGs in automating this process. Some question-answering benchmarks are automatically generated using extensive preprocessing from KGs, but they do not support dialogue generation. This paper introduces Chatty-Gen, a novel multi-stage retrieval-augmented generation platform for automatically generating high-quality dialogue benchmarks tailored to a specific domain using a KG. Chatty-Gen decomposes the generation process into manageable stages and uses assertion rules for automatic validation between stages. Our approach enables control over intermediate results to prevent time-consuming restarts due to hallucinations. It also reduces reliance on costly and more powerful commercial LLMs. Chatty-Gen eliminates upfront processing of the entire KG using efficient query-based retrieval to find representative subgraphs based on the dialogue context. Our experiments with several real and large KGs demonstrate that Chatty-Gen significantly outperforms state-of-the-art systems and ensures consistent model and system performance across multiple LLMs of diverse capabilities, such as GPT-4o, Gemini 1.5, Llama 3, and Mistral.
• 摘要：对话基准对于训练和评估参与特定领域对话的聊天机器人至关重要。知识图谱 (KG) 表示语义丰富且组织良好的数据，涵盖各个领域，例如 DBLP、DBpedia 和 YAGO。传统上，对话基准是从文档中手动创建的，忽略了 KG 在自动化此过程方面的潜力。一些问答基准是使用 KG 的大量预处理自动生成的，但它们不支持对话生成。本文介绍了 Chatty-Gen，这是一种新颖的多阶段检索增强生成平台，可使用 KG 自动生成针对特定领域的高质量对话基准。Chatty-Gen 将生成过程分解为可管理的阶段，并使用断言规则在阶段之间自动验证。我们的方法可以控制中间结果，以防止由于幻觉而导致耗时的重启。它还减少了对昂贵且功能更强大的商业 LLM 的依赖。 Chatty-Gen 使用高效的基于查询的检索来根据对话上下文查找代表性子图，从而消除了对整个 KG 的前期处理。我们对几个真实的大型 KG 进行的实验表明，Chatty-Gen 的表现明显优于最先进的系统，并确保在具有不同功能的多个 LLM（例如 GPT-4o、Gemini 1.5、Llama 3 和 Mistral）中保持一致的模型和系统性能。

Title: Passage Segmentation of Documents for Extractive Question Answering

• Authors: Zuhong Liu, Charles-Elie Simon, Fabien Caspani
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2501.09940
• Pdf URL: https://arxiv.org/pdf/2501.09940
• Copy Paste: [[2501.09940]] Passage Segmentation of Documents for Extractive Question Answering(https://arxiv.org/abs/2501.09940)
• Keywords: retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) has proven effective in open-domain question answering. However, the chunking process, which is essential to this pipeline, often receives insufficient attention relative to retrieval and synthesis components. This study emphasizes the critical role of chunking in improving the performance of both dense passage retrieval and the end-to-end RAG pipeline. We then introduce the Logits-Guided Multi-Granular Chunker (LGMGC), a novel framework that splits long documents into contextualized, self-contained chunks of varied granularity. Our experimental results, evaluated on two benchmark datasets, demonstrate that LGMGC not only improves the retrieval step but also outperforms existing chunking methods when integrated into a RAG pipeline.
• 摘要：检索增强生成 (RAG) 已被证明在开放域问答中非常有效。然而，与检索和合成组件相比，该流程中必不可少的分块过程往往没有得到足够的重视。本研究强调了分块在提高密集段落检索和端到端 RAG 流程性能方面的关键作用。然后，我们介绍了 Logits 引导多粒度分块器 (LGMGC)，这是一种新颖的框架，可将长文档拆分为具有不同粒度的上下文化、自包含的块。我们在两个基准数据集上评估的实验结果表明，LGMGC 不仅改进了检索步骤，而且在集成到 RAG 流程中时也优于现有的分块方法。

Title: FRAG: A Flexible Modular Framework for Retrieval-Augmented Generation based on Knowledge Graphs

• Authors: Zengyi Gao, Yukun Cao, Hairu Wang, Ao Ke, Yuan Feng, Xike Xie, S Kevin Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.09957
• Pdf URL: https://arxiv.org/pdf/2501.09957
• Copy Paste: [[2501.09957]] FRAG: A Flexible Modular Framework for Retrieval-Augmented Generation based on Knowledge Graphs(https://arxiv.org/abs/2501.09957)
• Keywords: language model, llm, hallucination, retrieval-augmented generation
• Abstract: To mitigate the hallucination and knowledge deficiency in large language models (LLMs), Knowledge Graph (KG)-based Retrieval-Augmented Generation (RAG) has shown promising potential by utilizing KGs as external resource to enhance LLMs this http URL, existing KG-RAG approaches struggle with a trade-off between flexibility and retrieval this http URL methods prioritize flexibility by avoiding the use of KG-fine-tuned models during retrieval, leading to fixed retrieval strategies and suboptimal retrieval this http URL, coupled methods embed KG information within models to improve retrieval quality, but at the expense of this http URL this paper, we propose a novel flexible modular KG-RAG framework, termed FRAG, which synergizes the advantages of both this http URL estimates the hop range of reasoning paths based solely on the query and classify it as either simple or this http URL match the complexity of the query, tailored pipelines are applied to ensure efficient and accurate reasoning path retrieval, thus fostering the final reasoning this http URL using the query text instead of the KG to infer the structural information of reasoning paths and employing adaptable retrieval strategies, FRAG improves retrieval quality while maintaining this http URL, FRAG does not require extra LLMs fine-tuning or calls, significantly boosting efficiency and conserving this http URL experiments show that FRAG achieves state-of-the-art performance with high efficiency and low resource consumption.
• 摘要：为了缓解大型语言模型（LLM）中的幻觉和知识缺陷，基于知识图谱（KG）的检索增强生成（RAG）通过利用 KG 作为外部资源来增强 LLM，显示出了良好的潜力。现有的 KG-RAG 方法在灵活性和检索之间难以取得平衡。这些方法优先考虑灵活性，避免在检索过程中使用 KG 微调模型，导致检索策略固定，检索效果不理想。耦合方法将 KG 信息嵌入模型中以提高检索质量，但代价是牺牲了性能。在本文中，我们提出了一种新颖的灵活模块化 KG-RAG 框架，称为 FRAG，它结合了两者的优点。仅根据查询估计推理路径的跳数范围，并将其分类为简单或与查询的复杂性相匹配，应用量身定制的管道以确保高效准确的推理路径检索，从而促进最终的通过使用查询文本而不是KG来推理该http URL以推断推理路径的结构信息并采用自适应的检索策略，FRAG在保持该http URL的同时提高了检索质量，FRAG不需要额外的LLM微调或调用，大大提高了效率并节省了该http URL实验表明FRAG以高效率和低资源消耗实现了最先进的性能。

Title: A Survey on Multi-Turn Interaction Capabilities of Large Language Models

• Authors: Chen Zhang, Xinyi Dai, Yaxiong Wu, Qu Yang, Yasheng Wang, Ruiming Tang, Yong Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.09959
• Pdf URL: https://arxiv.org/pdf/2501.09959
• Copy Paste: [[2501.09959]] A Survey on Multi-Turn Interaction Capabilities of Large Language Models(https://arxiv.org/abs/2501.09959)
• Keywords: language model, llm, chat, agent
• Abstract: Multi-turn interaction in the dialogue system research refers to a system's ability to maintain context across multiple dialogue turns, enabling it to generate coherent and contextually relevant responses. Recent advancements in large language models (LLMs) have significantly expanded the scope of multi-turn interaction, moving beyond chatbots to enable more dynamic agentic interactions with users or environments. In this paper, we provide a focused review of the multi-turn capabilities of LLMs, which are critical for a wide range of downstream applications, including conversational search and recommendation, consultation services, and interactive tutoring. This survey explores four key aspects: (1) the core model capabilities that contribute to effective multi-turn interaction, (2) how multi-turn interaction is evaluated in current practice, (3) the general algorithms used to enhance multi-turn interaction, and (4) potential future directions for research in this field.
• 摘要：对话系统研究中的多轮交互是指系统在多个对话轮次中保持上下文的能力，从而使其能够生成连贯且与上下文相关的响应。大型语言模型 (LLM) 的最新进展大大扩展了多轮交互的范围，超越了聊天机器人，实现了与用户或环境的更动态​​的代理交互。在本文中，我们重点回顾了 LLM 的多轮功能，这些功能对于广泛的下游应用至关重要，包括对话搜索和推荐、咨询服务和交互式辅导。本综述探讨了四个关键方面：(1) 有助于有效多轮交互的核心模型功能，(2) 当前实践中如何评估多轮交互，(3) 用于增强多轮交互的通用算法，以及 (4) 该领域未来研究的潜在方向。

Title: Agent-as-Judge for Factual Summarization of Long Narratives

• Authors: Yeonseok Jeong, Minsoo Kim, Seung-won Hwang, Byung-Hak Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.09993
• Pdf URL: https://arxiv.org/pdf/2501.09993
• Copy Paste: [[2501.09993]] Agent-as-Judge for Factual Summarization of Long Narratives(https://arxiv.org/abs/2501.09993)
• Keywords: language model, llm, agent
• Abstract: Large Language Models (LLMs) have demonstrated near-human performance in summarization tasks based on traditional metrics such as ROUGE and BERTScore. However, these metrics do not adequately capture critical aspects of summarization quality, such as factual accuracy, particularly for long narratives (>100K tokens). Recent advances, such as LLM-as-a-Judge, address the limitations of metrics based on lexical similarity but still exhibit factual inconsistencies, especially in understanding character relationships and states. In this work, we introduce NarrativeFactScore, a novel "Agent-as-a-Judge" framework for evaluating and refining summaries. By leveraging a Character Knowledge Graph (CKG) extracted from input and generated summaries, NarrativeFactScore assesses the factual consistency and provides actionable guidance for refinement, such as identifying missing or erroneous facts. We demonstrate the effectiveness of NarrativeFactScore through a detailed workflow illustration and extensive validation on widely adopted benchmarks, achieving superior performance compared to competitive methods. Our results highlight the potential of agent-driven evaluation systems to improve the factual reliability of LLM-generated summaries.
• 摘要：大型语言模型 (LLM) 在基于传统指标（例如 ROUGE 和 BERTScore）的摘要任务中表现出接近人类的表现。然而，这些指标不能充分捕捉摘要质量的关键方面，例如事实准确性，特别是对于长篇叙述（>100K 个标记）。最近的进展，例如 LLM-as-a-Judge，解决了基于词汇相似性的指标的局限性，但仍然表现出事实不一致，特别是在理解人物关系和状态方面。在这项工作中，我们引入了 NarrativeFactScore，这是一种用于评估和改进摘要的新颖“Agent-as-a-Judge”框架。通过利用从输入和生成的摘要中提取的字符知识图谱 (CKG)，NarrativeFactScore 评估事实一致性并提供可操作的改进指导，例如识别缺失或错误的事实。我们通过详细的工作流程说明和对广泛采用的基准的广泛验证证明了 NarrativeFactScore 的有效性，与竞争方法相比实现了卓越的性能。我们的研究结果强调了代理驱动评估系统提高 LLM 生成的摘要的事实可靠性的潜力。

Title: Attention-guided Self-reflection for Zero-shot Hallucination Detection in Large Language Models

• Authors: Qiang Liu, Xinlong Chen, Yue Ding, Shizhen Xu, Shu Wu, Liang Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.09997
• Pdf URL: https://arxiv.org/pdf/2501.09997
• Copy Paste: [[2501.09997]] Attention-guided Self-reflection for Zero-shot Hallucination Detection in Large Language Models(https://arxiv.org/abs/2501.09997)
• Keywords: language model, llm, hallucination
• Abstract: Hallucination has emerged as a significant barrier to the effective application of Large Language Models (LLMs). In this work, we introduce a novel Attention-Guided SElf-Reflection (AGSER) approach for zero-shot hallucination detection in LLMs. The AGSER method utilizes attention contributions to categorize the input query into attentive and non-attentive queries. Each query is then processed separately through the LLMs, allowing us to compute consistency scores between the generated responses and the original answer. The difference between the two consistency scores serves as a hallucination estimator. In addition to its efficacy in detecting hallucinations, AGSER notably reduces computational complexity, requiring only three passes through the LLM and utilizing two sets of tokens. We have conducted extensive experiments with four widely-used LLMs across three different hallucination benchmarks, demonstrating that our approach significantly outperforms existing methods in zero-shot hallucination detection.
• 摘要：幻觉已成为有效应用大型语言模型 (LLM) 的重大障碍。在这项工作中，我们引入了一种新颖的注意力引导自我反思 (AGSER) 方法，用于 LLM 中的零样本幻觉检测。AGSER 方法利用注意力贡献将输入查询分类为注意和非注意查询。然后通过 LLM 分别处理每个查询，使我们能够计算生成的响应与原始答案之间的一致性分数。两个一致性分数之间的差异可作为幻觉估计器。除了在检测幻觉方面的功效外，AGSER 还显着降低了计算复杂度，只需要通过 LLM 三次并使用两组标记。我们在三个不同的幻觉基准上对四种广泛使用的 LLM 进行了广泛的实验，表明我们的方法在零样本幻觉检测方面明显优于现有方法。

Title: MSTS: A Multimodal Safety Test Suite for Vision-Language Models

• Authors: Paul Röttger, Giuseppe Attanasio, Felix Friedrich, Janis Goldzycher, Alicia Parrish, Rishabh Bhardwaj, Chiara Di Bonaventura, Roman Eng, Gaia El Khoury Geagea, Sujata Goswami, Jieun Han, Dirk Hovy, Seogyeong Jeong, Paloma Jeretič, Flor Miriam Plaza-del-Arco, Donya Rooein, Patrick Schramowski, Anastassia Shaitarova, Xudong Shen, Richard Willats, Andrea Zugarini, Bertie Vidgen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.10057
• Pdf URL: https://arxiv.org/pdf/2501.10057
• Copy Paste: [[2501.10057]] MSTS: A Multimodal Safety Test Suite for Vision-Language Models(https://arxiv.org/abs/2501.10057)
• Keywords: language model, prompt, chat
• Abstract: Vision-language models (VLMs), which process image and text inputs, are increasingly integrated into chat assistants and other consumer AI applications. Without proper safeguards, however, VLMs may give harmful advice (e.g. how to self-harm) or encourage unsafe behaviours (e.g. to consume drugs). Despite these clear hazards, little work so far has evaluated VLM safety and the novel risks created by multimodal inputs. To address this gap, we introduce MSTS, a Multimodal Safety Test Suite for VLMs. MSTS comprises 400 test prompts across 40 fine-grained hazard categories. Each test prompt consists of a text and an image that only in combination reveal their full unsafe meaning. With MSTS, we find clear safety issues in several open VLMs. We also find some VLMs to be safe by accident, meaning that they are safe because they fail to understand even simple test prompts. We translate MSTS into ten languages, showing non-English prompts to increase the rate of unsafe model responses. We also show models to be safer when tested with text only rather than multimodal prompts. Finally, we explore the automation of VLM safety assessments, finding even the best safety classifiers to be lacking.
• 摘要：视觉语言模型 (VLM) 处理图像和文本输入，越来越多地集成到聊天助手和其他消费者 AI 应用程序中。然而，如果没有适当的保护措施，VLM 可能会给出有害的建议（例如如何自残）或鼓励不安全的行为（例如吸毒）。尽管存在这些明显的危害，但迄今为止很少有研究评估 VLM 的安全性以及多模态输入带来的新风险。为了解决这一差距，我们引入了 MSTS，这是一种用于 VLM 的多模态安全测试套件。MSTS 包含 400 个测试提示，涵盖 40 个细粒度危险类别。每个测试提示都包含一个文本和一个图像，只有结合起来才能揭示它们的全部不安全含义。通过 MSTS，我们在几个开放的 VLM 中发现了明显的安全问题。我们还发现一些 VLM 是意外安全的，这意味着它们是安全的，因为它们甚至无法理解简单的测试提示。我们将 MSTS 翻译成十种语言，显示非英语提示以提高不安全模型响应的速率。我们还表明，仅使用文本而非多模式提示进行测试时，模型更安全。最后，我们探索了 VLM 安全评估的自动化，发现即使是最好的安全分类器也存在不足。

Title: ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario

• Authors: Lucen Zhong, Zhengxiao Du, Xiaohan Zhang, Haiyi Hu, Jie Tang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.10132
• Pdf URL: https://arxiv.org/pdf/2501.10132
• Copy Paste: [[2501.10132]] ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario(https://arxiv.org/abs/2501.10132)
• Keywords: language model, llm, long context
• Abstract: Enhancing large language models (LLMs) with real-time APIs can help generate more accurate and up-to-date responses. However, evaluating the function calling abilities of LLMs in real-world scenarios remains under-explored due to the complexity of data collection and evaluation. In this work, we introduce ComplexFuncBench, a benchmark for complex function calling across five real-world scenarios. Compared to existing benchmarks, ComplexFuncBench encompasses multi-step and constrained function calling, which requires long-parameter filing, parameter value reasoning, and 128k long context. Additionally, we propose an automatic framework, ComplexEval, for quantitatively evaluating complex function calling tasks. Through comprehensive experiments, we demonstrate the deficiencies of state-of-the-art LLMs in function calling and suggest future directions for optimizing these capabilities. The data and code are available at \url{this https URL}.
• 摘要：使用实时 API 增强大型语言模型 (LLM) 有助于生成更准确、更最新的响应。然而，由于数据收集和评估的复杂性，在现实场景中评估 LLM 的函数调用能力仍未得到充分探索。在这项工作中，我们引入了 ComplexFuncBench，这是五个现实场景中复杂函数调用的基准。与现有基准相比，ComplexFuncBench 包含多步骤和受约束的函数调用，需要长参数归档、参数值推理和 128k 长的上下文。此外，我们提出了一个自动框架 ComplexEval，用于定量评估复杂函数调用任务。通过全面的实验，我们展示了最先进的 LLM 在函数调用方面的不足，并提出了优化这些功能的未来方向。数据和代码可在 \url{this https URL} 上找到。

Title: Dual Debiasing: Remove Stereotypes and Keep Factual Gender for Fair Language Modeling and Translation

• Authors: Tomasz Limisiewicz, David Mareček, Tomáš Musil
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.10150
• Pdf URL: https://arxiv.org/pdf/2501.10150
• Copy Paste: [[2501.10150]] Dual Debiasing: Remove Stereotypes and Keep Factual Gender for Fair Language Modeling and Translation(https://arxiv.org/abs/2501.10150)
• Keywords: language model
• Abstract: Mitigation of biases, such as language models' reliance on gender stereotypes, is a crucial endeavor required for the creation of reliable and useful language technology. The crucial aspect of debiasing is to ensure that the models preserve their versatile capabilities, including their ability to solve language tasks and equitably represent various genders. To address this issue, we introduce a streamlined Dual Dabiasing Algorithm through Model Adaptation (2DAMA). Novel Dual Debiasing enables robust reduction of stereotypical bias while preserving desired factual gender information encoded by language models. We show that 2DAMA effectively reduces gender bias in English and is one of the first approaches facilitating the mitigation of stereotypical tendencies in translation. The proposed method's key advantage is the preservation of factual gender cues, which are useful in a wide range of natural language processing tasks.
• 摘要：减轻偏见（例如语言模型对性别刻板印象的依赖）是创建可靠且有用的语言技术所需的一项关键努力。去偏见的关键方面是确保模型保留其多功能性，包括解决语言任务和公平代表各种性别的能力。为了解决这个问题，我们引入了一种通过模型自适应 (2DAMA) 简化的双重去偏见算法。新颖的双重去偏见能够稳健地减少刻板偏见，同时保留语言模型编码所需的事实性别信息。我们表明，2DAMA 有效地减少了英语中的性别偏见，并且是首批有助于减轻翻译中刻板印象倾向的方法之一。所提出方法的主要优势是保留了事实性别线索，这在广泛的自然语言处理任务中很有用。

Title: Multi-stage Training of Bilingual Islamic LLM for Neural Passage Retrieval

• Authors: Vera Pavlova
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.10175
• Pdf URL: https://arxiv.org/pdf/2501.10175
• Copy Paste: [[2501.10175]] Multi-stage Training of Bilingual Islamic LLM for Neural Passage Retrieval(https://arxiv.org/abs/2501.10175)
• Keywords: language model, llm
• Abstract: This study examines the use of Natural Language Processing (NLP) technology within the Islamic domain, focusing on developing an Islamic neural retrieval model. By leveraging the robust XLM-R model, the research employs a language reduction technique to create a lightweight bilingual large language model (LLM). Our approach for domain adaptation addresses the unique challenges faced in the Islamic domain, where substantial in-domain corpora exist only in Arabic while limited in other languages, including English. The work utilizes a multi-stage training process for retrieval models, incorporating large retrieval datasets, such as MS MARCO, and smaller, in-domain datasets to improve retrieval performance. Additionally, we have curated an in-domain retrieval dataset in English by employing data augmentation techniques and involving a reliable Islamic source. This approach enhances the domain-specific dataset for retrieval, leading to further performance gains. The findings suggest that combining domain adaptation and a multi-stage training method for the bilingual Islamic neural retrieval model enables it to outperform monolingual models on downstream retrieval tasks.
• 摘要：本研究考察了自然语言处理 (NLP) 技术在伊斯兰领域的应用，重点是开发伊斯兰神经检索模型。通过利用强大的 XLM-R 模型，该研究采用语言缩减技术来创建轻量级双语大型语言模型 (LLM)。我们的领域适应方法解决了伊斯兰领域面临的独特挑战，其中大量的领域内语料库仅存在于阿拉伯语中，而其他语言（包括英语）则有限。这项工作利用多阶段训练过程来训练检索模型，结合大型检索数据集（例如 MS MARCO）和较小的领域内数据集来提高检索性能。此外，我们还通过采用数据增强技术并涉及可靠的伊斯兰来源，整理了英语领域内检索数据集。这种方法增强了领域特定数据集的检索能力，从而进一步提高了性能。研究结果表明，将领域适应和双语伊斯兰神经检索模型的多阶段训练方法相结合，使其在下游检索任务中的表现优于单语模型。

Title: Towards Preventing Overreliance on Task-Oriented Conversational AI Through Accountability Modeling

• Authors: Suvodip Dey, Yi-Jyun Sun, Gokhan Tur, Dilek Hakkani-Tur
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.10316
• Pdf URL: https://arxiv.org/pdf/2501.10316
• Copy Paste: [[2501.10316]] Towards Preventing Overreliance on Task-Oriented Conversational AI Through Accountability Modeling(https://arxiv.org/abs/2501.10316)
• Keywords: llm, agent
• Abstract: Recent LLMs have enabled significant advancements for conversational agents. However, they are also well-known to hallucinate, i.e., they often produce responses that seem plausible but are not factually correct. On the other hand, users tend to over-rely on LLM-based AI agents; they accept the AI's suggestion even when it is wrong. Adding good friction, such as explanations or getting user confirmations, has been proposed as a mitigation in AI-supported decision-making systems. In this paper, we propose an accountability model for LLM-based task-oriented dialogue agents to address user overreliance via friction turns in cases of model uncertainty and errors associated with dialogue state tracking (DST). The accountability model is an augmented LLM with an additional accountability head, which functions as a binary classifier to predict the slots of the dialogue states. We perform our experiments with three backbone LLMs (Llama, Mistral, Gemma) on two established task-oriented datasets (MultiWOZ and Snips). Our empirical findings demonstrate that this approach not only enables reliable estimation of AI agent errors but also guides the LLM decoder in generating more accurate actions. We observe around 3% absolute improvement in joint goal accuracy by incorporating accountability heads in modern LLMs for the MultiWOZ dataset. We also show that this method enables the agent to self-correct its actions, further boosting its performance by 3%. Finally, we discuss the application of accountability modeling to prevent user overreliance by introducing friction.
• 摘要：最近的 LLM 为对话代理带来了重大进步。然而，它们也会产生幻觉，即它们经常产生看似合理但实际上并不正确的反应。另一方面，用户倾向于过度依赖基于 LLM 的 AI 代理；即使 AI 的建议是错误的，他们也会接受。在 AI 支持的决策系统中，增加良好的摩擦（例如解释或获得用户确认）已被提议作为一种缓解措施。在本文中，我们提出了一种基于 LLM 的任务导向型对话代理的责任模型，以在模型不确定性和与对话状态跟踪 (DST) 相关的错误的情况下通过摩擦转折来解决用户过度依赖的问题。责任模型是一个增强型 LLM，带有一个额外的责任头，它充当二元分类器来预测对话状态的时隙。我们在两个已建立的任务导向型数据集（MultiWOZ 和 Snips）上使用三个主干 LLM（Llama、Mistral、Gemma）进行实验。我们的实证结果表明，这种方法不仅可以可靠地估计 AI 代理的错误，还可以指导 LLM 解码器生成更准确的操作。通过在 MultiWOZ 数据集的现代 LLM 中结合责任头，我们观察到联合目标准确率绝对提高了约 3%。我们还表明，这种方法使代理能够自我纠正其行为，从而将其性能进一步提高 3%。最后，我们讨论了责任建模的应用，以防止因引入摩擦而导致用户过度依赖。

Title: Hierarchical Autoregressive Transformers: Combining Byte-~and Word-Level Processing for Robust, Adaptable Language Models

• Authors: Pit Neitemeier, Björn Deiseroth, Constantin Eichenberg, Lukas Balles
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.10322
• Pdf URL: https://arxiv.org/pdf/2501.10322
• Copy Paste: [[2501.10322]] Hierarchical Autoregressive Transformers: Combining Byte-~and Word-Level Processing for Robust, Adaptable Language Models(https://arxiv.org/abs/2501.10322)
• Keywords: language model
• Abstract: Tokenization is a fundamental step in natural language processing, breaking text into units that computational models can process. While learned subword tokenizers have become the de-facto standard, they present challenges such as large vocabularies, limited adaptability to new domains or languages, and sensitivity to spelling errors and variations. To overcome these limitations, we investigate a hierarchical architecture for autoregressive language modelling that combines character-level and word-level processing. It employs a lightweight character-level encoder to convert character sequences into word embeddings, which are then processed by a word-level backbone model and decoded back into characters via a compact character-level decoder. This method retains the sequence compression benefits of word-level tokenization without relying on a rigid, predefined vocabulary. We demonstrate, at scales up to 7 billion parameters, that hierarchical transformers match the downstream task performance of subword-tokenizer-based models while exhibiting significantly greater robustness to input perturbations. Additionally, during continued pretraining on an out-of-domain language, our model trains almost twice as fast, achieves superior performance on the target language, and retains more of its previously learned knowledge. Hierarchical transformers pave the way for NLP systems that are more robust, flexible, and generalizable across languages and domains.
• 摘要：标记化是自然语言处理中的一个基本步骤，将文本分解为计算模型可以处理的单元。虽然学习到的子词标记器已成为事实上的标准，但它们也带来了一些挑战，例如词汇量大、对新领域或语言的适应性有限以及对拼写错误和变化的敏感性。为了克服这些限制，我们研究了一种结合字符级和词级处理的自回归语言建模的分层架构。它采用轻量级字符级编码器将字符序列转换为词嵌入，然后由词级主干模型处理，并通过紧凑的字符级解码器解码回字符。此方法保留了词级标记化的序列压缩优势，而无需依赖严格的预定义词汇表。我们证明，在高达 70 亿个参数的规模下，分层转换器可与基于子词标记器的模型的下游任务性能相匹配，同时对输入扰动表现出显著更高的鲁棒性。此外，在对领域外语言进行持续预训练的过程中，我们的模型训练速度几乎提高了一倍，在目标语言上取得了卓越的表现，并保留了更多之前学到的知识。分层转换器为更强大、更灵活、跨语言和领域通用的 NLP 系统铺平了道路。

Title: BoK: Introducing Bag-of-Keywords Loss for Interpretable Dialogue Response Generation

• Authors: Suvodip Dey, Maunendra Sankar Desarkar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2501.10328
• Pdf URL: https://arxiv.org/pdf/2501.10328
• Copy Paste: [[2501.10328]] BoK: Introducing Bag-of-Keywords Loss for Interpretable Dialogue Response Generation(https://arxiv.org/abs/2501.10328)
• Keywords: language model, gpt, chat
• Abstract: The standard language modeling (LM) loss by itself has been shown to be inadequate for effective dialogue modeling. As a result, various training approaches, such as auxiliary loss functions and leveraging human feedback, are being adopted to enrich open-domain dialogue systems. One such auxiliary loss function is Bag-of-Words (BoW) loss, defined as the cross-entropy loss for predicting all the words/tokens of the next utterance. In this work, we propose a novel auxiliary loss named Bag-of-Keywords (BoK) loss to capture the central thought of the response through keyword prediction and leverage it to enhance the generation of meaningful and interpretable responses in open-domain dialogue systems. BoK loss upgrades the BoW loss by predicting only the keywords or critical words/tokens of the next utterance, intending to estimate the core idea rather than the entire response. We incorporate BoK loss in both encoder-decoder (T5) and decoder-only (DialoGPT) architecture and train the models to minimize the weighted sum of BoK and LM (BoK-LM) loss. We perform our experiments on two popular open-domain dialogue datasets, DailyDialog and Persona-Chat. We show that the inclusion of BoK loss improves the dialogue generation of backbone models while also enabling post-hoc interpretability. We also study the effectiveness of BoK-LM loss as a reference-free metric and observe comparable performance to the state-of-the-art metrics on various dialogue evaluation datasets.
• 摘要：标准语言建模 (LM) 损失本身已被证明不足以进行有效的对话建模。因此，人们正在采用各种训练方法，例如辅助损失函数和利用人工反馈，来丰富开放域对话系统。一种这样的辅助损失函数是词袋 (BoW) 损失，定义为预测下一个话语的所有单词/标记的交叉熵损失。在这项工作中，我们提出了一种名为关键词袋 (BoK) 损失的新型辅助损失，通过关键词预测来捕捉响应的中心思想，并利用它来增强开放域对话系统中有意义且可解释的响应的生成。BoK 损失通过仅预测下一个话语的关键词或关键单词/标记来升级 BoW 损失，旨在估计核心思想而不是整个响应。我们在编码器-解码器 (T5) 和仅解码器 (DialoGPT) 架构中都加入了 BoK 损失，并训练模型以最小化 BoK 和 LM (BoK-LM) 损失的加权和。我们在两个流行的开放域对话数据集 DailyDialog 和 Persona-Chat 上进行了实验。我们表明，加入 BoK 损失可以改善骨干模型的对话生成，同时还可以实现事后可解释性。我们还研究了 BoK-LM 损失作为无参考指标的有效性，并观察到在各种对话评估数据集上与最先进指标相当的性能。