2024-12-13

Title: LatentQA: Teaching LLMs to Decode Activations Into Natural Language

• Authors: Alexander Pan, Lijie Chen, Jacob Steinhardt
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.08686
• Pdf URL: https://arxiv.org/pdf/2412.08686
• Copy Paste: [[2412.08686]] LatentQA: Teaching LLMs to Decode Activations Into Natural Language(https://arxiv.org/abs/2412.08686)
• Keywords: language model, llm, prompt
• Abstract: Interpretability methods seek to understand language model representations, yet the outputs of most such methods -- circuits, vectors, scalars -- are not immediately human-interpretable. In response, we introduce LatentQA, the task of answering open-ended questions about model activations in natural language. Towards solving LatentQA, we propose Latent Interpretation Tuning (LIT), which finetunes a decoder LLM on a dataset of activations and associated question-answer pairs, similar to how visual instruction tuning trains on question-answer pairs associated with images. We use the decoder for diverse reading applications, such as extracting relational knowledge from representations or uncovering system prompts governing model behavior. Our decoder also specifies a differentiable loss that we use to control models, such as debiasing models on stereotyped sentences and controlling the sentiment of generations. Finally, we extend LatentQA to reveal harmful model capabilities, such as generating recipes for bioweapons and code for hacking.
• 摘要：可解释性方法旨在理解语言模型表示，但大多数此类方法的输出（电路、向量、标量）都不是人类可以立即解释的。为此，我们引入了 LatentQA，即用自然语言回答有关模型激活的开放式问题。为了解决 LatentQA，我们提出了潜在解释调整 (LIT)，它在激活数据集和相关问答对上对解码器 LLM 进行微调，类似于视觉指令调整在与图像相关的问答对上进行训练的方式。我们将解码器用于各种阅读应用，例如从表示中提取关系知识或发现控制模型行为的系统提示。我们的解码器还指定了可区分损失，我们用它来控制模型，例如消除刻板句子上的模型偏差并控制代际情绪。最后，我们扩展了 LatentQA 以揭示有害的模型功能，例如生成生物武器配方和黑客代码。

Title: In-Context Learning with Topological Information for Knowledge Graph Completion

• Authors: Udari Madhushani Sehwag, Kassiani Papasotiriou, Jared Vann, Sumitra Ganesh
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.08742
• Pdf URL: https://arxiv.org/pdf/2412.08742
• Copy Paste: [[2412.08742]] In-Context Learning with Topological Information for Knowledge Graph Completion(https://arxiv.org/abs/2412.08742)
• Keywords: language model, llm
• Abstract: Knowledge graphs (KGs) are crucial for representing and reasoning over structured information, supporting a wide range of applications such as information retrieval, question answering, and decision-making. However, their effectiveness is often hindered by incompleteness, limiting their potential for real-world impact. While knowledge graph completion (KGC) has been extensively studied in the literature, recent advances in generative AI models, particularly large language models (LLMs), have introduced new opportunities for innovation. In-context learning has recently emerged as a promising approach for leveraging pretrained knowledge of LLMs across a range of natural language processing tasks and has been widely adopted in both academia and industry. However, how to utilize in-context learning for effective KGC remains relatively underexplored. We develop a novel method that incorporates topological information through in-context learning to enhance KGC performance. By integrating ontological knowledge and graph structure into the context of LLMs, our approach achieves strong performance in the transductive setting i.e., nodes in the test graph dataset are present in the training graph dataset. Furthermore, we apply our approach to KGC in the more challenging inductive setting, i.e., nodes in the training graph dataset and test graph dataset are disjoint, leveraging the ontology to infer useful information about missing nodes which serve as contextual cues for the LLM during inference. Our method demonstrates superior performance compared to baselines on the ILPC-small and ILPC-large datasets.
• 摘要：知识图谱 (KG) 对于结构化信息的表示和推理至关重要，支持信息检索、问答和决策等广泛的应用。然而，它们的有效性往往受到不完整性的阻碍，限制了它们对现实世界产生影响的潜力。虽然知识图谱补全 (KGC) 在文献中得到了广泛的研究，但生成式人工智能模型，尤其是大型语言模型 (LLM) 的最新进展为创新带来了新的机遇。上下文学习最近成为一种有前途的方法，可在一系列自然语言处理任务中利用 LLM 的预训练知识，并已在学术界和工业界得到广泛采用。然而，如何利用上下文学习进行有效的 KGC 仍然相对未被充分探索。我们开发了一种新方法，通过上下文学习结合拓扑信息来增强 KGC 性能。通过将本体知识和图结构集成到 LLM 的上下文中，我们的方法在传导设置中实现了强大的性能，即测试图数据集中的节点存在于训练图数据集中。此外，我们在更具挑战性的归纳设置中将我们的方法应用于 KGC，即训练图数据集和测试图数据集中的节点是不相交的，利用本体推断有关缺失节点的有用信息，这些信息在推理过程中充当 LLM 的上下文线索。与 ILPC-small 和 ILPC-large 数据集上的基线相比，我们的方法表现出卓越的性能。

Title: Coverage-based Fairness in Multi-document Summarization

• Authors: Haoyuan Li, Yusen Zhang, Rui Zhang, Snigdha Chaturvedi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.08795
• Pdf URL: https://arxiv.org/pdf/2412.08795
• Copy Paste: [[2412.08795]] Coverage-based Fairness in Multi-document Summarization(https://arxiv.org/abs/2412.08795)
• Keywords: llm
• Abstract: Fairness in multi-document summarization (MDS) measures whether a system can generate a summary fairly representing information from documents with different social attribute values. Fairness in MDS is crucial since a fair summary can offer readers a comprehensive view. Previous works focus on quantifying summary-level fairness using Proportional Representation, a fairness measure based on Statistical Parity. However, Proportional Representation does not consider redundancy in input documents and overlooks corpus-level unfairness. In this work, we propose a new summary-level fairness measure, Equal Coverage, which is based on coverage of documents with different social attribute values and considers the redundancy within documents. To detect the corpus-level unfairness, we propose a new corpus-level measure, Coverage Parity. Our human evaluations show that our measures align more with our definition of fairness. Using our measures, we evaluate the fairness of thirteen different LLMs. We find that Claude3-sonnet is the fairest among all evaluated LLMs. We also find that almost all LLMs overrepresent different social attribute values.
• 摘要：多文档摘要 (MDS) 中的公平性衡量系统是否能够公平地生成摘要，以代表具有不同社会属性值的文档的信息。MDS 中的公平性至关重要，因为公平的摘要可以为读者提供全面的视角。以前的研究侧重于使用比例代表法（一种基于统计均等性的公平性衡量标准）来量化摘要级别的公平性。然而，比例代表法不考虑输入文档中的冗余，并且忽略了语料库级别的不公平性。在这项工作中，我们提出了一种新的摘要级别公平性衡量标准，即平等覆盖率，它基于具有不同社会属性值的文档的覆盖率，并考虑了文档内的冗余。为了检测语料库级别的不公平性，我们提出了一种新的语料库级别衡量标准，即覆盖率均等性。我们的人工评估表明，我们的衡量标准更符合我们对公平性的定义。使用我们的衡量标准，我们评估了十三个不同 LLM 的公平性。我们发现 Claude3-sonnet 是所有评估的 LLM 中最公平的。我们还发现，几乎所有的 LLM 都过度体现了不同的社会属性值。

Title: Large Concept Models: Language Modeling in a Sentence Representation Space

• Authors: The LCM team, Loïc Barrault, Paul-Ambroise Duquenne, Maha Elbayad, Artyom Kozhevnikov, Belen Alastruey, Pierre Andrews, Mariano Coria, Guillaume Couairon, Marta R. Costa-jussà, David Dale, Hady Elsahar, Kevin Heffernan, João Maria Janeiro, Tuan Tran, Christophe Ropers, Eduardo Sánchez, Robin San Roman, Alexandre Mourachko, Safiyyah Saleem, Holger Schwenk
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.08821
• Pdf URL: https://arxiv.org/pdf/2412.08821
• Copy Paste: [[2412.08821]] Large Concept Models: Language Modeling in a Sentence Representation Space(https://arxiv.org/abs/2412.08821)
• Keywords: language model, llm
• Abstract: LLMs have revolutionized the field of artificial intelligence and have emerged as the de-facto tool for many tasks. The current established technology of LLMs is to process input and generate output at the token level. This is in sharp contrast to humans who operate at multiple levels of abstraction, well beyond single words, to analyze information and to generate creative content. In this paper, we present an attempt at an architecture which operates on an explicit higher-level semantic representation, which we name a concept. Concepts are language- and modality-agnostic and represent a higher level idea or action in a flow. Hence, we build a "Large Concept Model". In this study, as proof of feasibility, we assume that a concept corresponds to a sentence, and use an existing sentence embedding space, SONAR, which supports up to 200 languages in both text and speech modalities. The Large Concept Model is trained to perform autoregressive sentence prediction in an embedding space. We explore multiple approaches, namely MSE regression, variants of diffusion-based generation, and models operating in a quantized SONAR space. These explorations are performed using 1.6B parameter models and training data in the order of 1.3T tokens. We then scale one architecture to a model size of 7B parameters and training data of about 2.7T tokens. We perform an experimental evaluation on several generative tasks, namely summarization and a new task of summary expansion. Finally, we show that our model exhibits impressive zero-shot generalization performance to many languages, outperforming existing LLMs of the same size. The training code of our models is freely available.
• 摘要：LLM 彻底改变了人工智能领域，并已成为许多任务的实际工具。LLM 的现有技术是在 token 级别处理输入并生成输出。这与人类形成了鲜明对比，人类在多个抽象层次上进行操作，远远超出单个单词，以分析信息并生成创意内容。在本文中，我们尝试了一种架构，该架构在显式高级语义表示上运行，我们将其称为概念。概念与语言和模态无关，表示流程中更高级别的想法或动作。因此，我们建立了一个“大概念模型”。在本研究中，作为可行性的证明，我们假设一个概念对应于一个句子，并使用现有的句子嵌入空间 SONAR，它在文本和语音模态中支持多达 200 种语言。大概念模型经过训练可在嵌入空间中执行自回归句子预测。我们探索了多种方法，即 MSE 回归、基于扩散的生成的变体以及在量化 SONAR 空间中运行的模型。这些探索是使用 1.6B 参数模型和 1.3T 标记量级的训练数据进行的。然后，我们将一个架构扩展到 7B 参数的模型大小和约 2.7T 标记的训练数据。我们对几个生成任务（即摘要和新的摘要扩展任务）进行了实验评估。最后，我们表明我们的模型对许多语言表现出令人印象深刻的零样本泛化性能，优于现有同等规模的 LLM。我们模型的训练代码是免费提供的。

Title: Exploring Large Language Models on Cross-Cultural Values in Connection with Training Methodology

• Authors: Minsang Kim, Seungjun Baek
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.08846
• Pdf URL: https://arxiv.org/pdf/2412.08846
• Copy Paste: [[2412.08846]] Exploring Large Language Models on Cross-Cultural Values in Connection with Training Methodology(https://arxiv.org/abs/2412.08846)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) closely interact with humans, and thus need an intimate understanding of the cultural values of human society. In this paper, we explore how open-source LLMs make judgments on diverse categories of cultural values across countries, and its relation to training methodology such as model sizes, training corpus, alignment, etc. Our analysis shows that LLMs can judge socio-cultural norms similar to humans but less so on social systems and progress. In addition, LLMs tend to judge cultural values biased toward Western culture, which can be improved with training on the multilingual corpus. We also find that increasing model size helps a better understanding of social values, but smaller models can be enhanced by using synthetic data. Our analysis reveals valuable insights into the design methodology of LLMs in connection with their understanding of cultural values.
• 摘要：大型语言模型 (LLM) 与人类密切相关，因此需要对人类社会的文化价值观有深入的理解。在本文中，我们探讨了开源 LLM 如何对不同国家/地区的不同类别的文化价值观做出判断，以及它与模型大小、训练语料库、对齐等训练方法的关系。我们的分析表明，LLM 可以判断与人类相似的社会文化规范，但对社会制度和进步的判断则较差。此外，LLM 倾向于判断偏向西方文化的文化价值观，这可以通过在多语言语料库上进行训练来改进。我们还发现，增加模型大小有助于更好地理解社会价值观，但较小的模型可以通过使用合成数据得到增强。我们的分析为 LLM 的设计方法及其对文化价值观的理解提供了宝贵的见解。

Title: A Graph-Based Synthetic Data Pipeline for Scaling High-Quality Reasoning Instructions

• Authors: Jiankang Wang, Jianjun Xu, Xiaorui Wang, Yuxin Wang, Mengting Xing, Shancheng Fang, Zhineng Chen, Hongtao Xie, Yongdong Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.08864
• Pdf URL: https://arxiv.org/pdf/2412.08864
• Copy Paste: [[2412.08864]] A Graph-Based Synthetic Data Pipeline for Scaling High-Quality Reasoning Instructions(https://arxiv.org/abs/2412.08864)
• Keywords: language model, gpt, llm
• Abstract: Synthesizing high-quality reasoning data for continual training has been proven to be effective in enhancing the performance of Large Language Models (LLMs). However, previous synthetic approaches struggle to easily scale up data and incur high costs in the pursuit of high quality. In this paper, we propose the Graph-based Synthetic Data Pipeline (GSDP), an economical and scalable framework for high-quality reasoning data synthesis. Inspired by knowledge graphs, we extracted knowledge points from seed data and constructed a knowledge point relationships graph to explore their interconnections. By exploring the implicit relationships among knowledge, our method achieves $\times$255 data expansion. Furthermore, GSDP led by open-source models, achieves synthesis quality comparable to GPT-4-0613 while maintaining $\times$100 lower costs. To tackle the most challenging mathematical reasoning task, we present the GSDP-MATH dataset comprising over 1.91 million pairs of math problems and answers. After fine-tuning on GSDP-MATH, GSDP-7B based on Mistral-7B achieves 37.7% accuracy on MATH and 78.4% on GSM8K, demonstrating the effectiveness of our method. The dataset and models trained in this paper will be available.
• 摘要：合成高质量推理数据进行持续训练已被证明可有效提高大型语言模型 (LLM) 的性能。然而，以前的合成方法难以轻松扩大数据规模，并且在追求高质量时成本高昂。在本文中，我们提出了基于图的合成数据管道 (GSDP)，这是一种经济且可扩展的高质量推理数据合成框架。受知识图谱的启发，我们从种子数据中提取知识点并构建知识点关系图以探索它们之间的互连。通过探索知识之间的隐式关系，我们的方法实现了 $\times$255 数据扩展。此外，由开源模型引领的 GSDP 实现了与 GPT-4-0613 相当的合成质量，同时保持了 $\times$100 更低的成本。为了解决最具挑战性的数学推理任务，我们提出了包含超过 191 万对数学问题和答案的 GSDP-MATH 数据集。基于 Mistral-7B 的 GSDP-7B 在 GSDP-MATH 上进行微调后，在 MATH 上达到了 37.7% 的准确率，在 GSM8K 上达到了 78.4%，证明了我们方法的有效性。本文训练的数据集和模型将可用。

Title: AI-assisted Knowledge Discovery in Biomedical Literature to Support Decision-making in Precision Oncology

• Authors: Ting He, Kory Kreimeyer, Mimi Najjar, Jonathan Spiker, Maria Fatteh, Valsamo Anagnostou, Taxiarchis Botsis
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.08900
• Pdf URL: https://arxiv.org/pdf/2412.08900
• Copy Paste: [[2412.08900]] AI-assisted Knowledge Discovery in Biomedical Literature to Support Decision-making in Precision Oncology(https://arxiv.org/abs/2412.08900)
• Keywords: language model
• Abstract: The delivery of appropriate targeted therapies to cancer patients requires the complete analysis of the molecular profiling of tumors and the patient's clinical characteristics in the context of existing knowledge and recent findings described in biomedical literature and several other sources. We evaluated the potential contributions of specific natural language processing solutions to support knowledge discovery from biomedical literature. Two models from the Bidirectional Encoder Representations from Transformers (BERT) family, two Large Language Models, and PubTator 3.0 were tested for their ability to support the named entity recognition (NER) and the relation extraction (RE) tasks. PubTator 3.0 and the BioBERT model performed best in the NER task (best F1-score equal to 0.93 and 0.89, respectively), while BioBERT outperformed all other solutions in the RE task (best F1-score 0.79) and a specific use case it was applied to by recognizing nearly all entity mentions and most of the relations.
• 摘要：为癌症患者提供适当的靶向治疗需要在现有知识和生物医学文献及其他来源中描述的最新发现的背景下，对肿瘤的分子分析和患者的临床特征进行全面分析。我们评估了特定自然语言处理解决方案对支持从生物医学文献中发现知识的潜在贡献。对 Transformers 双向编码器表示 (BERT) 系列中的两个模型、两个大型语言模型和 PubTator 3.0 进行了测试，以了解它们对命名实体识别 (NER) 和关系提取 (RE) 任务的支持能力。PubTator 3.0 和 BioBERT 模型在 NER 任务中表现最佳（最佳 F1 分数分别等于 0.93 和 0.89），而 BioBERT 在 RE 任务（最佳 F1 分数 0.79）和应用它的特定用例中的表现优于所有其他解决方案，因为它可以识别几乎所有的实体提及和大多数关系。

Title: Phi-4 Technical Report

• Authors: Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, Yi Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.08905
• Pdf URL: https://arxiv.org/pdf/2412.08905
• Copy Paste: [[2412.08905]] Phi-4 Technical Report(https://arxiv.org/abs/2412.08905)
• Keywords: language model, gpt
• Abstract: We present phi-4, a 14-billion parameter language model developed with a training recipe that is centrally focused on data quality. Unlike most language models, where pre-training is based primarily on organic data sources such as web content or code, phi-4 strategically incorporates synthetic data throughout the training process. While previous models in the Phi family largely distill the capabilities of a teacher model (specifically GPT-4), phi-4 substantially surpasses its teacher model on STEM-focused QA capabilities, giving evidence that our data-generation and post-training techniques go beyond distillation. Despite minimal changes to the phi-3 architecture, phi-4 achieves strong performance relative to its size -- especially on reasoning-focused benchmarks -- due to improved data, training curriculum, and innovations in the post-training scheme.
• 摘要：我们推出了 phi-4，这是一个拥有 140 亿个参数的语言模型，其开发方法主要关注数据质量。与大多数语言模型不同，大多数语言模型的预训练主要基于网络内容或代码等有机数据源，而 phi-4 在整个训练过程中策略性地整合了合成数据。虽然 Phi 系列中的先前模型在很大程度上提炼了教师模型（特别是 GPT-4）的功能，但 phi-4 在以 STEM 为中心的 QA 功能方面大大超越了教师模型，这证明我们的数据生成和后训练技术超越了提炼。尽管对 phi-3 架构的改动很小，但由于数据、训练课程的改进和后训练方案的创新，phi-4 相对于其规模实现了强劲的性能——尤其是在以推理为重点的基准上。

Title: From Text to Trajectory: Exploring Complex Constraint Representation and Decomposition in Safe Reinforcement Learning

• Authors: Pusen Dong, Tianchen Zhu, Yue Qiu, Haoyi Zhou, Jianxin Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.08920
• Pdf URL: https://arxiv.org/pdf/2412.08920
• Copy Paste: [[2412.08920]] From Text to Trajectory: Exploring Complex Constraint Representation and Decomposition in Safe Reinforcement Learning(https://arxiv.org/abs/2412.08920)
• Keywords: agent
• Abstract: Safe reinforcement learning (RL) requires the agent to finish a given task while obeying specific constraints. Giving constraints in natural language form has great potential for practical scenarios due to its flexible transfer capability and accessibility. Previous safe RL methods with natural language constraints typically need to design cost functions manually for each constraint, which requires domain expertise and lacks flexibility. In this paper, we harness the dual role of text in this task, using it not only to provide constraint but also as a training signal. We introduce the Trajectory-level Textual Constraints Translator (TTCT) to replace the manually designed cost function. Our empirical results demonstrate that TTCT effectively comprehends textual constraint and trajectory, and the policies trained by TTCT can achieve a lower violation rate than the standard cost function. Extra studies are conducted to demonstrate that the TTCT has zero-shot transfer capability to adapt to constraint-shift environments.
• 摘要：安全的强化学习 (RL) 要求代理在遵守特定约束的同时完成给定的任务。以自然语言形式给出约束由于其灵活的迁移能力和可访问性而在实际场景中具有巨大的潜力。以前具有自然语言约束的安全 RL 方法通常需要为每个约束手动设计成本函数，这需要领域专业知识并且缺乏灵活性。在本文中，我们利用文本在此任务中的双重作用，不仅将其用作提供约束，还将其用作训练信号。我们引入了轨迹级文本约束转换器 (TTCT) 来取代手动设计的成本函数。我们的实证结果表明，TTCT 有效地理解了文本约束和轨迹，并且用 TTCT 训练的策略可以实现比标准成本函数更低的违规率。进行了额外的研究表明 TTCT 具有零样本迁移能力以适应约束转移环境。

Title: Align, Generate, Learn: A Novel Closed-Loop Framework for Cross-Lingual In-Context Learning

• Authors: Mateo Alejandro Rojas, Rafael Carranza
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.08955
• Pdf URL: https://arxiv.org/pdf/2412.08955
• Copy Paste: [[2412.08955]] Align, Generate, Learn: A Novel Closed-Loop Framework for Cross-Lingual In-Context Learning(https://arxiv.org/abs/2412.08955)
• Keywords: language model, llm
• Abstract: Cross-lingual in-context learning (XICL) has emerged as a transformative paradigm for leveraging large language models (LLMs) to tackle multilingual tasks, especially for low-resource languages. However, existing approaches often rely on external retrievers or task-specific fine-tuning, limiting their scalability and generalizability. In this paper, we propose a novel self-supervised framework that harnesses the generative capabilities of LLMs to internally select and utilize task-relevant examples. Our method introduces two key objectives: a retrieval-generation alignment loss to optimize the quality of selected examples and a semantic coherence loss to ensure cross-lingual consistency. Through extensive experiments on multilingual benchmarks, our approach achieves state-of-the-art performance, significantly outperforming existing baselines. Further analysis highlights its robustness across diverse language families and its ability to generalize to unseen tasks. Human evaluations confirm the superior fluency, relevance, and semantic correctness of outputs generated by our method. This work provides a scalable, effective, and generalizable solution for cross-lingual in-context learning.
• 摘要：跨语言上下文学习 (XICL) 已成为一种变革性范式，利用大型语言模型 (LLM) 来解决多语言任务，尤其是对于资源匮乏的语言。然而，现有方法通常依赖于外部检索器或特定于任务的微调，从而限制了它们的可扩展性和通用性。在本文中，我们提出了一种新颖的自监督框架，该框架利用 LLM 的生成能力在内部选择和利用与任务相关的示例。我们的方法引入了两个关键目标：检索生成对齐损失以优化所选示例的质量，语义连贯性损失以确保跨语言一致性。通过在多语言基准上进行大量实验，我们的方法实现了最先进的性能，大大优于现有基线。进一步的分析突出了它在各种语言家族中的稳健性以及它推广到看不见的任务的能力。人工评估证实了我们的方法生成的输出具有出色的流畅性、相关性和语义正确性。这项工作为跨语言上下文学习提供了一种可扩展、有效且可推广的解决方案。

Title: Reasoning-Aware Query-Focused Summarization over Multi-Table Data

• Authors: Xiaochuan Lin, Xiangyong Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.08970
• Pdf URL: https://arxiv.org/pdf/2412.08970
• Copy Paste: [[2412.08970]] Reasoning-Aware Query-Focused Summarization over Multi-Table Data(https://arxiv.org/abs/2412.08970)
• Keywords: language model, llm
• Abstract: Query-focused summarization over multi-table data is a challenging yet critical task for extracting precise and relevant information from structured data. Existing methods often rely on complex preprocessing steps and struggle to generalize across domains or handle the logical reasoning required for multi-table queries. In this paper, we propose QueryTableSummarizer++, an end-to-end generative framework leveraging large language models (LLMs) enhanced with table-aware pre-training, query-aligned fine-tuning, and reinforcement learning with feedback. Our method eliminates the need for intermediate serialization steps and directly generates query-relevant summaries. Experiments on a benchmark dataset demonstrate that QueryTableSummarizer++ significantly outperforms state-of-the-art baselines in terms of BLEU, ROUGE, and F1-score. Additional analyses highlight its scalability, generalization across domains, and robust handling of complex queries. Human evaluation further validates the superior quality and practical applicability of the generated summaries, establishing QueryTableSummarizer++ as a highly effective solution for multi-table summarization tasks.
• 摘要：针对多表数据进行以查询为中心的摘要是一项具有挑战性但又至关重要的任务，它可从结构化数据中提取精确且相关的信息。现有方法通常依赖于复杂的预处理步骤，并且难以跨域推广或处理多表查询所需的逻辑推理。在本文中，我们提出了 QueryTableSummarizer++，这​​是一个端到端的生成框架，利用大型语言模型 (LLM)，并通过表感知预训练、查询对齐微调和带反馈的强化学习进行了增强。我们的方法消除了中间序列化步骤的需要，并直接生成与查询相关的摘要。在基准数据集上的实验表明，QueryTableSummarizer++ 在 BLEU、ROUGE 和 F1 分数方面明显优于最先进的基线。其他分析突出了它的可扩展性、跨域推广以及对复杂查询的强大处理能力。人工评估进一步验证了所生成摘要的卓越品质和实际适用性，从而确立了 QueryTableSummarizer++ 成为多表摘要任务的高效解决方案。

Title: RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios

• Authors: Ruiwen Zhou, Wenyue Hua, Liangming Pan, Sitao Cheng, Xiaobao Wu, En Yu, William Yang Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.08972
• Pdf URL: https://arxiv.org/pdf/2412.08972
• Copy Paste: [[2412.08972]] RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios(https://arxiv.org/abs/2412.08972)
• Keywords: language model, llm
• Abstract: This paper introduces RuleArena, a novel and challenging benchmark designed to evaluate the ability of large language models (LLMs) to follow complex, real-world rules in reasoning. Covering three practical domains -- airline baggage fees, NBA transactions, and tax regulations -- RuleArena assesses LLMs' proficiency in handling intricate natural language instructions that demand long-context understanding, logical reasoning, and accurate mathematical computation. Two key attributes distinguish RuleArena from traditional rule-based reasoning benchmarks: (1) it extends beyond standard first-order logic representations, and (2) it is grounded in authentic, practical scenarios, providing insights into the suitability and reliability of LLMs for real-world applications. Our findings reveal several notable limitations in LLMs: (1) they struggle to identify and apply the appropriate rules, frequently becoming confused by similar but distinct regulations, (2) they cannot consistently perform accurate mathematical computations, even when they correctly identify the relevant rules, and (3) in general, they perform poorly in the benchmark. These results highlight significant challenges in advancing LLMs' rule-guided reasoning capabilities in real-life applications.
• 摘要：本文介绍了 RuleArena，这是一种新颖且具有挑战性的基准，旨在评估大型语言模型 (LLM) 在推理中遵循复杂的现实世界规则的能力。RuleArena 涵盖三个实际领域——航空行李费、NBA 交易和税收法规——评估 LLM 处理复杂自然语言指令的能力，这些指令需要长上下文理解、逻辑推理和准确的数学计算。两个关键属性将 RuleArena 与传统的基于规则的推理基准区分开来：(1) 它超越了标准的一阶逻辑表示，(2) 它以真实的实际场景为基础，为 LLM 在现实世界应用中的适用性和可靠性提供了见解。我们的研究结果揭示了 LLM 的几个显着局限性：(1) 它们很难识别和应用适当的规则，经常被相似但不同的规则所混淆，(2) 它们无法始终如一地执行准确的数学计算，即使它们正确识别了相关规则，(3) 总体而言，它们在基准测试中表现不佳。这些结果凸显了在实际应用中提升 LLM 规则引导推理能力所面临的重大挑战。

Title: Assessing the Robustness of Retrieval-Augmented Generation Systems in K-12 Educational Question Answering with Knowledge Discrepancies

• Authors: Tianshi Zheng, Weihan Li, Jiaxin Bai, Weiqi Wang, Yangqiu Song
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.08985
• Pdf URL: https://arxiv.org/pdf/2412.08985
• Copy Paste: [[2412.08985]] Assessing the Robustness of Retrieval-Augmented Generation Systems in K-12 Educational Question Answering with Knowledge Discrepancies(https://arxiv.org/abs/2412.08985)
• Keywords: language model, llm, retrieval-augmented generation
• Abstract: Retrieval-Augmented Generation (RAG) systems have demonstrated remarkable potential as question answering systems in the K-12 Education domain, where knowledge is typically queried within the restricted scope of authoritative textbooks. However, the discrepancy between textbooks and the parametric knowledge in Large Language Models (LLMs) could undermine the effectiveness of RAG systems. To systematically investigate the robustness of RAG systems under such knowledge discrepancies, we present EduKDQA, a question answering dataset that simulates knowledge discrepancies in real applications by applying hypothetical knowledge updates in answers and source documents. EduKDQA includes 3,005 questions covering five subjects, under a comprehensive question typology from the perspective of context utilization and knowledge integration. We conducted extensive experiments on retrieval and question answering performance. We find that most RAG systems suffer from a substantial performance drop in question answering with knowledge discrepancies, while questions that require integration of contextual knowledge and parametric knowledge pose a challenge to LLMs.
• 摘要：检索增强生成 (RAG) 系统已显示出作为 K-12 教育领域问答系统的巨大潜力，在该领域，知识通常在权威教科书的有限范围内查询。然而，教科书与大型语言模型 (LLM) 中的参数知识之间的差异可能会削弱 RAG 系统的有效性。为了系统地研究 RAG 系统在这种知识差异下的稳健性，我们提出了 EduKDQA，这是一个问答数据集，通过在答案和源文档中应用假设的知识更新来模拟实际应用中的知识差异。EduKDQA 包含 3,005 个问题，涵盖五个主题，从上下文利用和知识整合的角度提供了一个全面的问题类型。我们对检索和问答性能进行了广泛的实验。我们发现大多数 RAG 系统在知识差异的问答中性能大幅下降，而需要整合上下文知识和参数知识的问题对 LLM 提出了挑战。

Title: What Makes Cryptic Crosswords Challenging for LLMs?

• Authors: Abdelrahman Sadallah, Daria Kotova, Ekaterina Kochmar
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.09012
• Pdf URL: https://arxiv.org/pdf/2412.09012
• Copy Paste: [[2412.09012]] What Makes Cryptic Crosswords Challenging for LLMs?(https://arxiv.org/abs/2412.09012)
• Keywords: language model, gpt, llm, chat
• Abstract: Cryptic crosswords are puzzles that rely on general knowledge and the solver's ability to manipulate language on different levels, dealing with various types of wordplay. Previous research suggests that solving such puzzles is challenging even for modern NLP models, including Large Language Models (LLMs). However, there is little to no research on the reasons for their poor performance on this task. In this paper, we establish the benchmark results for three popular LLMs: Gemma2, LLaMA3 and ChatGPT, showing that their performance on this task is still significantly below that of humans. We also investigate why these models struggle to achieve superior performance. We release our code and introduced datasets at this https URL.
• 摘要：填字游戏是一种依靠常识和解谜者在不同层面操纵语言的能力的谜题，涉及各种类型的文字游戏。先前的研究表明，即使对于现代 NLP 模型（包括大型语言模型 (LLM)）来说，解决此类谜题也是一项挑战。然而，几乎没有研究研究它们在此任务上表现不佳的原因。在本文中，我们为三种流行的 LLM：Gemma2、LLaMA3 和 ChatGPT 建立了基准测试结果，结果表明它们在此任务上的表现仍然远低于人类。我们还研究了这些模型为何难以实现卓越性能。我们在此 https URL 上发布了我们的代码并引入了数据集。

Title: Dialogue Language Model with Large-Scale Persona Data Engineering

• Authors: Mengze Hong, Chen Zhang, Chaotao Chen, Rongzhong Lian, Di Jiang
• Subjects: cs.CL, cs.HC
• Abstract URL: https://arxiv.org/abs/2412.09034
• Pdf URL: https://arxiv.org/pdf/2412.09034
• Copy Paste: [[2412.09034]] Dialogue Language Model with Large-Scale Persona Data Engineering(https://arxiv.org/abs/2412.09034)
• Keywords: language model, gpt, chat
• Abstract: Maintaining persona consistency is paramount in the application of open-domain dialogue systems, as exemplified by models like ChatGPT. Despite significant advancements, the limited scale and diversity of current persona dialogue datasets remain challenges to achieving robust persona-consistent dialogue models. In this study, drawing inspiration from the success of large-scale pre-training, we introduce PPDS, an open-domain persona dialogue system that employs extensive generative pre-training on a persona dialogue dataset to enhance persona consistency. Specifically, we present a persona extraction model designed to autonomously and precisely generate vast persona dialogue datasets. Additionally, we unveil a pioneering persona augmentation technique to address the invalid persona bias inherent in the constructed dataset. Both quantitative and human evaluations consistently highlight the superior response quality and persona consistency of our proposed model, underscoring its effectiveness.
• 摘要：在开放域对话系统的应用中，保持角色一致性至关重要，ChatGPT 等模型就是一个典型例子。尽管取得了重大进展，但当前角色对话数据集的规模和多样性有限，这仍然是实现稳健的角色一致对话模型的挑战。在本研究中，我们借鉴大规模预训练的成功经验，引入了 PPDS，这是一个开放域角色对话系统，它在角色对话数据集上采用广泛的生成预训练来增强角色一致性。具体来说，我们提出了一个角色提取模型，旨在自主、准确地生成大量角色对话数据集。此外，我们还推出了一种开创性的角色增强技术，以解决构建数据集中固有的无效角色偏差。定量和人工评估都一致强调了我们提出的模型的卓越响应质量和角色一致性，凸显了其有效性。

Title: ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty

• Authors: Meizhi Zhong, Xikai Liu, Chen Zhang, Yikun Lei, Yan Gao, Yao Hu, Kehai Chen, Min Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.09036
• Pdf URL: https://arxiv.org/pdf/2412.09036
• Copy Paste: [[2412.09036]] ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based on Layer Uncertainty(https://arxiv.org/abs/2412.09036)
• Keywords: language model, llm
• Abstract: Large Language models (LLMs) have become a research hotspot. To accelerate the inference of LLMs, storing computed caches in memory has become the standard technique. However, as the inference length increases, growing KV caches might lead to out-of-memory issues. Many existing methods address this issue through KV cache compression, primarily by preserving key tokens throughout all layers to reduce information loss. Most of them allocate a uniform budget size for each layer to retain. However, we observe that the minimum budget sizes needed to retain essential information vary across layers and models based on the perspectives of attention and hidden state output. Building on this observation, this paper proposes a simple yet effective KV cache compression method that leverages layer uncertainty to allocate budget size for each layer. Experimental results show that the proposed method can reduce memory usage of the KV caches to only $\sim$20\% when compared to Full KV inference while achieving nearly lossless performance.
• 摘要：大型语言模型 (LLM) 已成为研究热点。为了加速 LLM 的推理，将计算缓存存储在内存中已成为标准技术。然而，随着推理长度的增加，增加 KV 缓存可能会导致内存不足的问题。许多现有方法通过 KV 缓存压缩来解决此问题，主要是通过在所有层中保留关键标记来减少信息丢失。他们中的大多数为每一层分配统一的预算大小来保留。然而，我们观察到，根据注意力和隐藏状态输出的角度，保留基本信息所需的最小预算大小在不同的层和模型之间有所不同。基于这一观察，本文提出了一种简单而有效的 KV 缓存压缩方法，利用层的不确定性为每一层分配预算大小。实验结果表明，与全 KV 推理相比，所提出的方法可以将 KV 缓存的内存使用量降低到仅 $\sim$20\% 同时实现几乎无损的性能。

Title: Multi-Task Learning with LLMs for Implicit Sentiment Analysis: Data-level and Task-level Automatic Weight Learning

• Authors: Wenna Lai, Haoran Xie, Guandong Xu, Qing Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.09046
• Pdf URL: https://arxiv.org/pdf/2412.09046
• Copy Paste: [[2412.09046]] Multi-Task Learning with LLMs for Implicit Sentiment Analysis: Data-level and Task-level Automatic Weight Learning(https://arxiv.org/abs/2412.09046)
• Keywords: language model, llm, hallucination
• Abstract: Implicit sentiment analysis (ISA) presents significant challenges due to the absence of salient cue words. Previous methods have struggled with insufficient data and limited reasoning capabilities to infer underlying opinions. Integrating multi-task learning (MTL) with large language models (LLMs) offers the potential to enable models of varying sizes to reliably perceive and recognize genuine opinions in ISA. However, existing MTL approaches are constrained by two sources of uncertainty: data-level uncertainty, arising from hallucination problems in LLM-generated contextual information, and task-level uncertainty, stemming from the varying capacities of models to process contextual information. To handle these uncertainties, we introduce MT-ISA, a novel MTL framework that enhances ISA by leveraging the generation and reasoning capabilities of LLMs through automatic MTL. Specifically, MT-ISA constructs auxiliary tasks using generative LLMs to supplement sentiment elements and incorporates automatic MTL to fully exploit auxiliary data. We introduce data-level and task-level automatic weight learning (AWL), which dynamically identifies relationships and prioritizes more reliable data and critical tasks, enabling models of varying sizes to adaptively learn fine-grained weights based on their reasoning capabilities. We investigate three strategies for data-level AWL, while also introducing homoscedastic uncertainty for task-level AWL. Extensive experiments reveal that models of varying sizes achieve an optimal balance between primary prediction and auxiliary tasks in MT-ISA. This underscores the effectiveness and adaptability of our approach.
• 摘要：由于缺乏突出的提示词，隐性情绪分析 (ISA) 面临巨大挑战。以前的方法一直面临数据不足和推理能力有限的问题，无法推断潜在的观点。将多任务学习 (MTL) 与大型语言模型 (LLM) 相结合，有可能使不同大小的模型能够可靠地感知和识别 ISA 中的真实观点。然而，现有的 MTL 方法受到两个不确定性来源的限制：数据级不确定性，源于 LLM 生成的上下文信息中的幻觉问题，以及任务级不确定性，源于模型处理上下文信息的能力不同。为了处理这些不确定性，我们引入了 MT-ISA，这是一种新颖的 MTL 框架，它通过自动 MTL 利用 LLM 的生成和推理能力来增强 ISA。具体而言，MT-ISA 使用生成 LLM 构建辅助任务来补充情绪元素，并结合自动 MTL 来充分利用辅助数据。我们引入了数据级和任务级自动权重学习 (AWL)，它可以动态识别关系并优先考虑更可靠的数据和关键任务，使不同大小的模型能够根据其推理能力自适应地学习细粒度权重。我们研究了数据级 AWL 的三种策略，同时还为任务级 AWL 引入了同方差不确定性。大量实验表明，不同大小的模型在 MT-ISA 中实现了主要预测和辅助任务之间的最佳平衡。这强调了我们方法的有效性和适应性。

Title: Dial-In LLM: Human-Aligned Dialogue Intent Clustering with LLM-in-the-loop

• Authors: Mengze Hong, Yuanfeng Song, Di Jiang, Wailing Ng, Yanjie Sun, Chen Jason Zhang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.09049
• Pdf URL: https://arxiv.org/pdf/2412.09049
• Copy Paste: [[2412.09049]] Dial-In LLM: Human-Aligned Dialogue Intent Clustering with LLM-in-the-loop(https://arxiv.org/abs/2412.09049)
• Keywords: language model, llm
• Abstract: The discovery of customer intention from dialogue plays an important role in automated support system. However, traditional text clustering methods are poorly aligned with human perceptions due to the shift from embedding distance to semantic distance, and existing quantitative metrics for text clustering may not accurately reflect the true quality of intent clusters. In this paper, we leverage the superior language understanding capabilities of Large Language Models (LLMs) for designing better-calibrated intent clustering algorithms. We first establish the foundation by verifying the robustness of fine-tuned LLM utility in semantic coherence evaluation and cluster naming, resulting in an accuracy of 97.50% and 94.40%, respectively, when compared to the human-labeled ground truth. Then, we propose an iterative clustering algorithm that facilitates cluster-level refinement and the continuous discovery of high-quality intent clusters. Furthermore, we present several LLM-in-the-loop semi-supervised clustering techniques tailored for intent discovery from customer service dialogue. Experiments on a large-scale industrial dataset comprising 1,507 intent clusters demonstrate the effectiveness of the proposed techniques. The methods outperformed existing counterparts, achieving 6.25% improvement in quantitative metrics and 12% enhancement in application-level performance when constructing an intent classifier.
• 摘要：从对话中发现客户意图在自动化支持系统中起着重要作用。然而，由于从嵌入距离到语义距离的转变，传统的文本聚类方法与人类感知的一致性较差，现有的文本聚类定量指标可能无法准确反映意图聚类的真实质量。在本文中，我们利用大型语言模型 (LLM) 的卓越语言理解能力来设计更精确的意图聚类算法。我们首先通过验证微调的 LLM 效用在语义一致性评估和聚类命名中的稳健性来建立基础，与人类标记的地面实况相比，准确率分别达到 97.50% 和 94.40%。然后，我们提出了一种迭代聚类算法，该算法有助于聚类级细化和高质量意图聚类的持续发现。此外，我们提出了几种专为从客户服务对话中发现意图而定制的 LLM-in-the-loop 半监督聚类技术。在包含 1,507 个意图聚类的大规模工业数据集上的实验证明了所提技术的有效性。与现有方法相比，该方法在构建意图分类器时实现了定量指标 6.25% 的提升和应用级性能 12% 的提升。

Title: Forest-of-Thought: Scaling Test-Time Compute for Enhancing LLM Reasoning

• Authors: Zhenni Bi, Kai Han, Chuanjian Liu, Yehui Tang, Yunhe Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.09078
• Pdf URL: https://arxiv.org/pdf/2412.09078
• Copy Paste: [[2412.09078]] Forest-of-Thought: Scaling Test-Time Compute for Enhancing LLM Reasoning(https://arxiv.org/abs/2412.09078)
• Keywords: language model, llm, prompt, chain-of-thought, tree-of-thought
• Abstract: Large Language Models (LLMs) have shown remarkable abilities across various language tasks, but solving complex reasoning problems remains a challenge. While existing methods like Chain-of-Thought (CoT) and Tree-of-Thought (ToT) enhance reasoning by decomposing problems or structuring prompts, they typically perform a single pass of reasoning and may fail to revisit flawed paths, compromising accuracy. To address this, we propose a novel reasoning framework called Forest-of-Thought (FoT), which integrates multiple reasoning trees to leverage collective decision-making for solving complex logical problems. FoT utilizes sparse activation strategies to select the most relevant reasoning paths, improving both efficiency and accuracy. Additionally, we introduce a dynamic self-correction strategy that enables real-time error correction and learning from past mistakes, as well as consensus-guided decision making strategies to optimize correctness and computational resources. Experimental results demonstrate that the FoT framework, combined with these strategies, significantly enhances the reasoning capabilities of LLMs, enabling them to solve complex tasks with greater precision and efficiency.
• 摘要：大型语言模型 (LLM) 在各种语言任务中都表现出非凡的能力，但解决复杂的推理问题仍然是一个挑战。虽然现有的方法，如思维链 (CoT) 和思维树 (ToT)，通过分解问题或构造提示来增强推理能力，但它们通常只执行一次推理，可能无法重新访问有缺陷的路径，从而影响准确性。为了解决这个问题，我们提出了一种名为思维森林 (FoT) 的新型推理框架，它集成了多个推理树，利用集体决策来解决复杂的逻辑问题。FoT 利用稀疏激活策略来选择最相关的推理路径，从而提高效率和准确性。此外，我们引入了一种动态自我纠正策略，可以实时纠正错误并从过去的错误中学习，以及共识引导的决策策略，以优化正确性和计算资源。实验结果表明，FoT 框架与这些策略相结合，显著增强了 LLM 的推理能力，使它们能够以更高的精度和效率解决复杂任务。

Title: Evaluating Pixel Language Models on Non-Standardized Languages

• Authors: Alberto Muñoz-Ortiz, Verena Blaschke, Barbara Plank
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.09084
• Pdf URL: https://arxiv.org/pdf/2412.09084
• Copy Paste: [[2412.09084]] Evaluating Pixel Language Models on Non-Standardized Languages(https://arxiv.org/abs/2412.09084)
• Keywords: language model
• Abstract: We explore the potential of pixel-based models for transfer learning from standard languages to dialects. These models convert text into images that are divided into patches, enabling a continuous vocabulary representation that proves especially useful for out-of-vocabulary words common in dialectal data. Using German as a case study, we compare the performance of pixel-based models to token-based models across various syntactic and semantic tasks. Our results show that pixel-based models outperform token-based models in part-of-speech tagging, dependency parsing and intent detection for zero-shot dialect evaluation by up to 26 percentage points in some scenarios, though not in Standard German. However, pixel-based models fall short in topic classification. These findings emphasize the potential of pixel-based models for handling dialectal data, though further research should be conducted to assess their effectiveness in various linguistic contexts.
• 摘要：我们探索了基于像素的模型在从标准语言到方言的迁移学习方面的潜力。这些模型将文本转换为分成块的图像，从而实现连续的词汇表示，这对于方言数据中常见的词汇之外的单词特别有用。以德语为例，我们比较了基于像素的模型和基于标记的模型在各种句法和语义任务中的表现。我们的结果表明，在零样本方言评估的词性标注、依存关系解析和意图检测方面，基于像素的模型在某些情况下比基于标记的模型表现更好，最高可达 26 个百分点，但在标准德语中并非如此。然而，基于像素的模型在主题分类方面有所欠缺。这些发现强调了基于像素的模型处理方言数据的潜力，但应进行进一步研究以评估它们在各种语言环境中的有效性。

Title: Filter-then-Generate: Large Language Models with Structure-Text Adapter for Knowledge Graph Completion

• Authors: Ben Liu, Jihai Zhang, Fangquan Lin, Cheng Yang, Min Peng
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.09094
• Pdf URL: https://arxiv.org/pdf/2412.09094
• Copy Paste: [[2412.09094]] Filter-then-Generate: Large Language Models with Structure-Text Adapter for Knowledge Graph Completion(https://arxiv.org/abs/2412.09094)
• Keywords: language model, llm, hallucination, prompt
• Abstract: Large Language Models (LLMs) present massive inherent knowledge and superior semantic comprehension capability, which have revolutionized various tasks in natural language processing. Despite their success, a critical gap remains in enabling LLMs to perform knowledge graph completion (KGC). Empirical evidence suggests that LLMs consistently perform worse than conventional KGC approaches, even through sophisticated prompt design or tailored instruction-tuning. Fundamentally, applying LLMs on KGC introduces several critical challenges, including a vast set of entity candidates, hallucination issue of LLMs, and under-exploitation of the graph structure. To address these challenges, we propose a novel instruction-tuning-based method, namely FtG. Specifically, we present a \textit{filter-then-generate} paradigm and formulate the KGC task into a multiple-choice question format. In this way, we can harness the capability of LLMs while mitigating the issue casused by hallucinations. Moreover, we devise a flexible ego-graph serialization prompt and employ a structure-text adapter to couple structure and text information in a contextualized manner. Experimental results demonstrate that FtG achieves substantial performance gain compared to existing state-of-the-art methods. The instruction dataset and code are available at \url{this https URL}.
• 摘要：大型语言模型 (LLM) 具有海量固有知识和卓越的语义理解能力，彻底改变了自然语言处理中的各种任务。尽管它们取得了成功，但在使 LLM 能够执行知识图谱补全 (KGC) 方面仍然存在关键差距。经验证据表明，即使通过复杂的提示设计或量身定制的指令调整，LLM 的表现始终不如传统的 KGC 方法。从根本上讲，将 LLM 应用于 KGC 会带来几个关键挑战，包括大量的实体候选集、LLM 的幻觉问题以及图结构的未充分利用。为了应对这些挑战，我们提出了一种基于指令调整的新方法，即 FtG。具体来说，我们提出了一个 \textit{过滤-然后-生成} 范式，并将 KGC 任务制定为多项选择题格式。通过这种方式，我们可以利用 LLM 的能力，同时缓解幻觉引起的问题。此外，我们设计了一个灵活的自我图序列化提示，并使用结构文本适配器以情境化的方式耦合结构和文本信息。实验结果表明，与现有的最先进方法相比，FtG 实现了显着的性能提升。指令数据集和代码可在 \url{此 https URL} 处获得。

Title: When Text Embedding Meets Large Language Model: A Comprehensive Survey

• Authors: Zhijie Nie, Zhangchi Feng, Mingxin Li, Cunwang Zhang, Yanzhao Zhang, Dingkun Long, Richong Zhang
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.09165
• Pdf URL: https://arxiv.org/pdf/2412.09165
• Copy Paste: [[2412.09165]] When Text Embedding Meets Large Language Model: A Comprehensive Survey(https://arxiv.org/abs/2412.09165)
• Keywords: language model, llm
• Abstract: Text embedding has become a foundational technology in natural language processing (NLP) during the deep learning era, driving advancements across a wide array of downstream tasks. While many natural language understanding challenges can now be modeled using generative paradigms and leverage the robust generative and comprehension capabilities of large language models (LLMs), numerous practical applications, such as semantic matching, clustering, and information retrieval, continue to rely on text embeddings for their efficiency and effectiveness. In this survey, we categorize the interplay between LLMs and text embeddings into three overarching themes: (1) LLM-augmented text embedding, enhancing traditional embedding methods with LLMs; (2) LLMs as text embedders, utilizing their innate capabilities for embedding generation; and (3) Text embedding understanding with LLMs, leveraging LLMs to analyze and interpret embeddings. By organizing these efforts based on interaction patterns rather than specific downstream applications, we offer a novel and systematic overview of contributions from various research and application domains in the era of LLMs. Furthermore, we highlight the unresolved challenges that persisted in the pre-LLM era with pre-trained language models (PLMs) and explore the emerging obstacles brought forth by LLMs. Building on this analysis, we outline prospective directions for the evolution of text embedding, addressing both theoretical and practical opportunities in the rapidly advancing landscape of NLP.
• 摘要：在深度学习时代，文本嵌入已成为自然语言处理 (NLP) 的一项基础技术，推动了一系列下游任务的进步。虽然现在可以使用生成范式对许多自然语言理解挑战进行建模，并利用大型语言模型 (LLM) 强大的生成和理解能力，但许多实际应用（例如语义匹配、聚类和信息检索）仍然依赖于文本嵌入来实现其效率和效果。在本调查中，我们将 LLM 和文本嵌入之间的相互作用分为三个主要主题：(1) LLM 增强文本嵌入，使用 LLM 增强传统嵌入方法；(2) LLM 作为文本嵌入器，利用其固有的嵌入生成能力；(3) 使用 LLM 理解文本嵌入，利用 LLM 分析和解释嵌入。通过根据交互模式而不是特定的下游应用来组织这些工作，我们提供了一个新颖而系统的概述，概述了 LLM 时代各个研究和应用领域的贡献。此外，我们强调了 LLM 时代之前使用预训练语言模型 (PLM) 时仍然存在的未解决的挑战，并探讨了 LLM 带来的新障碍。在此分析的基础上，我们概述了文本嵌入发展的未来方向，解决了 NLP 快速发展领域中的理论和实践机会。

Title: ReFF: Reinforcing Format Faithfulness in Language Models across Varied Tasks

• Authors: Jiashu Yao, Heyan Huang, Zeming Liu, Haoyu Wen, Wei Su, Boao Qian, Yuhang Guo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.09173
• Pdf URL: https://arxiv.org/pdf/2412.09173
• Copy Paste: [[2412.09173]] ReFF: Reinforcing Format Faithfulness in Language Models across Varied Tasks(https://arxiv.org/abs/2412.09173)
• Keywords: language model, llm, chat
• Abstract: Following formatting instructions to generate well-structured content is a fundamental yet often unmet capability for large language models (LLMs). To study this capability, which we refer to as format faithfulness, we present FormatBench, a comprehensive format-related benchmark. Compared to previous format-related benchmarks, FormatBench involves a greater variety of tasks in terms of application scenes (traditional NLP tasks, creative works, autonomous agency tasks), human-LLM interaction styles (single-turn instruction, multi-turn chat), and format types (inclusion, wrapping, length, coding). Moreover, each task in FormatBench is attached with a format checker program. Extensive experiments on the benchmark reveal that state-of-the-art open- and closed-source LLMs still suffer from severe deficiency in format faithfulness. By virtue of the decidable nature of formats, we propose to Reinforce Format Faithfulness (ReFF) to help LLMs generate formatted output as instructed without compromising general quality. Without any annotated data, ReFF can substantially improve the format faithfulness rate (e.g., from 21.6% in original LLaMA3 to 95.0% on caption segmentation task), while keep the general quality comparable (e.g., from 47.3 to 46.4 in F1 scores). Combined with labeled training data, ReFF can simultaneously improve both format faithfulness (e.g., from 21.6% in original LLaMA3 to 75.5%) and general quality (e.g., from 47.3 to 61.6 in F1 scores). We further offer an interpretability analysis to explain how ReFF improves both format faithfulness and general quality.
• 摘要：遵循格式化指令来生成结构良好的内容是大型语言模型 (LLM) 的基本但经常未得到满足的能力。为了研究这种我们称之为格式忠实度的能力，我们提出了 FormatBench，这是一个全面的格式相关基准。与以前的格式相关基准相比，FormatBench 涉及的应用场景（传统 NLP 任务、创意作品、自主代理任务）、人机交互风格（单轮指令、多轮聊天）和格式类型（包含、包装、长度、编码）方面更加多样化。此外，FormatBench 中的每个任务都附带一个格式检查程序。对基准进行的大量实验表明，最先进的开源和闭源 LLM 仍然存在严重的格式忠实度缺陷。凭借格式的可判定性质，我们提出强化格式忠实度 (ReFF) 来帮助 LLM 按照指令生成格式化输出，而不会影响整体质量。在没有任何注释数据的情况下，ReFF 可以显著提高格式忠实度（例如，在字幕分割任务中从原始 LLaMA3 中的 21.6% 提高到 95.0%），同时保持总体质量相当（例如，F1 分数从 47.3 提高到 46.4）。结合标记训练数据，ReFF 可以同时提高格式忠实度（例如，从原始 LLaMA3 中的 21.6% 提高到 75.5%）和总体质量（例如，F1 分数从 47.3 提高到 61.6）。我们进一步提供了可解释性分析来解释 ReFF 如何提高格式忠实度和总体质量。

Title: Make Satire Boring Again: Reducing Stylistic Bias of Satirical Corpus by Utilizing Generative LLMs

• Authors: Asli Umay Ozturk, Recep Firat Cekinel, Asli Umay Ozturk
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.09247
• Pdf URL: https://arxiv.org/pdf/2412.09247
• Copy Paste: [[2412.09247]] Make Satire Boring Again: Reducing Stylistic Bias of Satirical Corpus by Utilizing Generative LLMs(https://arxiv.org/abs/2412.09247)
• Keywords: language model, llm
• Abstract: Satire detection is essential for accurately extracting opinions from textual data and combating misinformation online. However, the lack of diverse corpora for satire leads to the problem of stylistic bias which impacts the models' detection performances. This study proposes a debiasing approach for satire detection, focusing on reducing biases in training data by utilizing generative large language models. The approach is evaluated in both cross-domain (irony detection) and cross-lingual (English) settings. Results show that the debiasing method enhances the robustness and generalizability of the models for satire and irony detection tasks in Turkish and English. However, its impact on causal language models, such as Llama-3.1, is limited. Additionally, this work curates and presents the Turkish Satirical News Dataset with detailed human annotations, with case studies on classification, debiasing, and explainability.
• 摘要：讽刺检测对于从文本数据中准确提取观点和打击网上虚假信息至关重要。然而，讽刺语料库的缺乏导致了风格偏见的问题，从而影响了模型的检测性能。本研究提出了一种讽刺检测的去偏方法，重点是通过利用生成式大型语言模型来减少训练数据中的偏见。该方法在跨领域（讽刺检测）和跨语言（英语）环境中进行了评估。结果表明，去偏方法增强了土耳其语和英语讽刺和讽刺检测任务模型的稳健性和通用性。然而，它对因果语言模型（如 Llama-3.1）的影响有限。此外，这项研究整理并展示了土耳其讽刺新闻数据集，其中包含详细的人工注释，以及关于分类、去偏和可解释性的案例研究。

Title: Towards Understanding the Robustness of LLM-based Evaluations under Perturbations

• Authors: Manav Chaudhary, Harshit Gupta, Savita Bhat, Vasudeva Varma
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.09269
• Pdf URL: https://arxiv.org/pdf/2412.09269
• Copy Paste: [[2412.09269]] Towards Understanding the Robustness of LLM-based Evaluations under Perturbations(https://arxiv.org/abs/2412.09269)
• Keywords: language model, llm, prompt
• Abstract: Traditional evaluation metrics like BLEU and ROUGE fall short when capturing the nuanced qualities of generated text, particularly when there is no single ground truth. In this paper, we explore the potential of Large Language Models (LLMs), specifically Google Gemini 1, to serve as automatic evaluators for non-standardized metrics in summarization and dialog-based tasks. We conduct experiments across multiple prompting strategies to examine how LLMs fare as quality evaluators when compared with human judgments on the SummEval and USR datasets, asking the model to generate both a score as well as a justification for the score. Furthermore, we explore the robustness of the LLM evaluator by using perturbed inputs. Our findings suggest that while LLMs show promise, their alignment with human evaluators is limited, they are not robust against perturbations and significant improvements are required for their standalone use as reliable evaluators for subjective metrics.
• 摘要：BLEU 和 ROUGE 等传统评估指标在捕捉生成文本的细微差别时会失效，尤其是在没有单一基本事实的情况下。在本文中，我们探索了大型语言模型 (LLM)，特别是 Google Gemini 1，作为摘要和对话任务中非标准化指标的自动评估器的潜力。我们通过多种提示策略进行实验，以检查 LLM 作为质量评估器与 SummEval 和 USR 数据集上的人类判断相比表现如何，要求模型生成分数以及分数的理由。此外，我们通过使用扰动输入探索 LLM 评估器的稳健性。我们的研究结果表明，虽然 LLM 很有前途，但它们与人类评估者的一致性有限，它们对扰动的稳健性不强，需要进行重大改进才能独立用作主观指标的可靠评估器。

Title: Learning to Solve Domain-Specific Calculation Problems with Knowledge-Intensive Programs Generator

• Authors: Chengyuan Liu, Shihang Wang, Lizhi Qing, Jun Lin, Ji Zhang, Fei Wu, Kun Kuang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.09280
• Pdf URL: https://arxiv.org/pdf/2412.09280
• Copy Paste: [[2412.09280]] Learning to Solve Domain-Specific Calculation Problems with Knowledge-Intensive Programs Generator(https://arxiv.org/abs/2412.09280)
• Keywords: language model, llm
• Abstract: Domain Large Language Models (LLMs) are developed for domain-specific tasks based on general LLMs. But it still requires professional knowledge to facilitate the expertise for some domain-specific tasks. In this paper, we investigate into knowledge-intensive calculation problems. We find that the math problems to be challenging for LLMs, when involving complex domain-specific rules and knowledge documents, rather than simple formulations of terminologies. Therefore, we propose a pipeline to solve the domain-specific calculation problems with Knowledge-Intensive Programs Generator more effectively, named as KIPG. It generates knowledge-intensive programs according to the domain-specific documents. For each query, key variables are extracted, then outcomes which are dependent on domain knowledge are calculated with the programs. By iterative preference alignment, the code generator learns to improve the logic consistency with the domain knowledge. Taking legal domain as an example, we have conducted experiments to prove the effectiveness of our pipeline, and extensive analysis on the modules. We also find that the code generator is also adaptable to other domains, without training on the new knowledge.
• 摘要：领域大型语言模型（LLM）是基于通用LLM开发的用于领域特定任务的模型。但是，它仍然需要专业知识来促进某些领域特定任务的专业知识。在本文中，我们研究了知识密集型计算问题。我们发现，当涉及复杂的领域特定规则和知识文档而不是简单的术语表述时，LLM的数学问题具有挑战性。因此，我们提出了一种使用知识密集型程序生成器更有效地解决领域特定计算问题的流程，称为KIPG。它根据领域特定文档生成知识密集型程序。对于每个查询，提取关键变量，然后使用程序计算依赖于领域知识的结果。通过迭代偏好对齐，代码生成器学习提高与领域知识的逻辑一致性。以法律领域为例，我们进行了实验以证明我们流程的有效性，并对模块进行了广泛的分析。我们还发现代码生成器也适用于其他领域，而无需对新知识进行训练。

Title: Benchmarking LLMs for Mimicking Child-Caregiver Language in Interaction

• Authors: Jing Liu, Abdellah Fourtassi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.09318
• Pdf URL: https://arxiv.org/pdf/2412.09318
• Copy Paste: [[2412.09318]] Benchmarking LLMs for Mimicking Child-Caregiver Language in Interaction(https://arxiv.org/abs/2412.09318)
• Keywords: gpt, llm
• Abstract: LLMs can generate human-like dialogues, yet their ability to simulate early child-adult interactions remains largely unexplored. In this paper, we examined how effectively LLMs can capture the distinctive features of child-caregiver language in interaction, using both static and interactive benchmarking methods. We found that state-of-the-art LLMs like Llama 3 and GPT-4o can approximate child-caregiver dialogues at the word and utterance level, but they struggle to reproduce the child and caregiver's discursive patterns, exaggerate alignment, and fail to reach the level of diversity shown by humans. The broader goal of this work is to initiate the development of a comprehensive benchmark for LLMs in child-oriented applications.
• 摘要：LLM 可以生成类似人类的对话，但它们模拟早期儿童与成人互动的能力仍未得到充分探索。在本文中，我们使用静态和交互式基准测试方法，研究了 LLM 如何有效地捕捉儿童看护者语言在互动中的独特特征。我们发现，像 Llama 3 和 GPT-4o 这样的最先进的 LLM 可以在单词和话语层面上近似儿童看护者的对话，但它们很难重现儿童和看护者的话语模式，夸大一致性，并且无法达到人类所表现出的多样性水平。这项工作的更广泛目标是启动面向儿童应用的 LLM 综合基准的开发。

Title: Falcon-UI: Understanding GUI Before Following User Instructions

• Authors: Huawen Shen, Chang Liu, Gengluo Li, Xinlong Wang, Yu Zhou, Can Ma, Xiangyang Ji
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.09362
• Pdf URL: https://arxiv.org/pdf/2412.09362
• Copy Paste: [[2412.09362]] Falcon-UI: Understanding GUI Before Following User Instructions(https://arxiv.org/abs/2412.09362)
• Keywords: agent
• Abstract: Pursuing human-like interaction for Graphical User Interface (GUI) agents requires understanding the GUI context and following user instructions. However, existing works typically couple these two aspects and focus more on instruct-following abilities, while ignoring the importance of understanding the GUI context. In this paper, we introduce an instruction-free GUI navigation dataset, termed Insight-UI Dataset, to enhance model comprehension of GUI environments. Insight-UI Dataset is automatically generated from the Common Crawl corpus, simulating various platforms -- including iOS, Android, Windows, and Linux -- across multiple resolutions on 312K domains. Although GUI interactions vary by context, diverse interfaces share common internal patterns, such as clicking an item to view its details. It implies the feasibility of independent GUI operation learning, followed by joint optimization with instruction tuning. Thereby, we develop the GUI agent model Falcon-UI, which is initially pretrained on Insight-UI Dataset and subsequently fine-tuned on Android and Web GUI datasets, including AITW, AITZ, Android Control, and Mind2Web. With 7 billion parameters, Falcon-UI achieves accuracy comparable to the 72 billion-parameter Qwen2VL on AITZ, validating the alignment between GUI context comprehension and agent performance. Our code and dataset will be open-sourced.
• 摘要：追求图形用户界面 (GUI) 代理的类人交互需要理解 GUI 上下文并遵循用户指令。然而，现有的研究通常将这两个方面结合起来，更多地关注遵循指令的能力，而忽略了理解 GUI 上下文的重要性。在本文中，我们引入了一个无指令的 GUI 导航数据集，称为 Insight-UI 数据集，以增强模型对 GUI 环境的理解。Insight-UI 数据集是从 Common Crawl 语料库自动生成的，模拟了 312K 域上跨多种分辨率的各种平台——包括 iOS、Android、Windows 和 Linux。虽然 GUI 交互因上下文而异，但不同的界面共享共同的内部模式，例如单击某个项目以查看其详细信息。这意味着独立 GUI 操作学习的可行性，然后通过指令调整进行联合优化。因此，我们开发了 GUI 代理模型 Falcon-UI，该模型最初在 Insight-UI 数据集上进行预训练，随后在 Android 和 Web GUI 数据集（包括 AITW、AITZ、Android Control 和 Mind2Web）上进行微调。Falcon-UI 拥有 70 亿个参数，其准确度可与 AITZ 上的 720 亿个参数的 Qwen2VL 相媲美，从而验证了 GUI 上下文理解与代理性能之间的一致性。我们的代码和数据集将开源。

Title: Text Generation Models for Luxembourgish with Limited Data: A Balanced Multilingual Strategy

• Authors: Alistair Plum, Tharindu Ranasinghe, Christoph Purschke
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.09415
• Pdf URL: https://arxiv.org/pdf/2412.09415
• Copy Paste: [[2412.09415]] Text Generation Models for Luxembourgish with Limited Data: A Balanced Multilingual Strategy(https://arxiv.org/abs/2412.09415)
• Keywords: language model
• Abstract: This paper addresses the challenges in developing language models for less-represented languages, with a focus on Luxembourgish. Despite its active development, Luxembourgish faces a digital data scarcity, exacerbated by Luxembourg's multilingual context. We propose a novel text generation model based on the T5 architecture, combining limited Luxembourgish data with equal amounts, in terms of size and type, of German and French data. We hypothesise that a model trained on Luxembourgish, German, and French will improve the model's cross-lingual transfer learning capabilities and outperform monolingual and large multilingual models. To verify this, the study at hand explores whether multilingual or monolingual training is more beneficial for Luxembourgish language generation. For the evaluation, we introduce LuxGen, a text generation benchmark that is the first of its kind for Luxembourgish.
• 摘要：本文探讨了为代表性较低的语言开发语言模型所面临的挑战，重点研究了卢森堡语。尽管卢森堡语正在积极发展，但它面临着数字数据稀缺的问题，而卢森堡的多语言环境又加剧了这一问题。我们提出了一种基于 T5 架构的新型文本生成模型，将有限的卢森堡语数据与大小和类型相等的德语和法语数据相结合。我们假设，用卢森堡语、德语和法语训练的模型将提高模型的跨语言迁移学习能力，并优于单语和大型多语言模型。为了验证这一点，本研究探讨了多语言或单语训练是否更有利于卢森堡语的生成。为了进行评估，我们引入了 LuxGen，这是一个文本生成基准，也是卢森堡语的首个同类基准。

Title: Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutors

• Authors: Kaushal Kumar Maurya, KV Aditya Srivatsa, Kseniia Petukhova, Ekaterina Kochmar
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.09416
• Pdf URL: https://arxiv.org/pdf/2412.09416
• Copy Paste: [[2412.09416]] Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical Ability Assessment of LLM-Powered AI Tutors(https://arxiv.org/abs/2412.09416)
• Keywords: language model, llm
• Abstract: In this paper, we investigate whether current state-of-the-art large language models (LLMs) are effective as AI tutors and whether they demonstrate pedagogical abilities necessary for good AI tutoring in educational dialogues. Previous efforts towards evaluation have been limited to subjective protocols and benchmarks. To bridge this gap, we propose a unified evaluation taxonomy with eight pedagogical dimensions based on key learning sciences principles, which is designed to assess the pedagogical value of LLM-powered AI tutor responses grounded in student mistakes or confusion in the mathematical domain. We release MRBench -- a new evaluation benchmark containing 192 conversations and 1,596 responses from seven state-of-the-art LLM-based and human tutors, providing gold annotations for eight pedagogical dimensions. We assess reliability of the popular Prometheus2 LLM as an evaluator and analyze each tutor's pedagogical abilities, highlighting which LLMs are good tutors and which ones are more suitable as question-answering systems. We believe that the presented taxonomy, benchmark, and human-annotated labels will streamline the evaluation process and help track the progress in AI tutors' development.
• 摘要：在本文中，我们研究了当前最先进的大型语言模型 (LLM) 是否可有效用作 AI 导师，以及它们是否具备在教育对话中进行良好 AI 辅导所必需的教学能力。以前的评估工作仅限于主观协议和基准。为了弥合这一差距，我们提出了一个基于关键学习科学原则的统一评估分类法，该分类法包含八个教学维度，旨在评估基于学生在数学领域的错误或困惑的 LLM 驱动的 AI 导师回答的教学价值。我们发布了 MRBench——一个新的评估基准，包含来自七位最先进的 LLM 导师和人类导师的 192 段对话和 1,596 个回复，为八个教学维度提供了黄金注释。我们评估了流行的 Prometheus2 LLM 作为评估者的可靠性，并分析了每位导师的教学能力，重点介绍了哪些 LLM 是好的导师，哪些更适合用作问答系统。我们相信，所提出的分类法、基准和人工注释的标签将简化评估流程，并有助于跟踪人工智能导师发展的进度。

Title: The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspective

• Authors: Javier de la Rosa, Vladislav Mikhailov, Lemei Zhang, Freddy Wetjen, David Samuel, Peng Liu, Rolv-Arild Braaten, Petter Mæhlum, Magnus Breder Birkenes, Andrey Kutuzov, Tita Enstad, Svein Arne Brygfjeld, Jon Atle Gulla, Stephan Oepen, Erik Velldal, Wilfred Østgulen, Liljia Øvrelid, Aslak Sira Myhre
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.09460
• Pdf URL: https://arxiv.org/pdf/2412.09460
• Copy Paste: [[2412.09460]] The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspective(https://arxiv.org/abs/2412.09460)
• Keywords: language model, llm
• Abstract: The use of copyrighted materials in training generative language models raises critical legal and ethical questions. This paper presents a framework for and the results of empirically assessing the impact of copyrighted materials on the performance of large language models (LLMs) for Norwegian. We found that both books and newspapers contribute positively when the models are evaluated on a diverse set of Norwegian benchmarks, while fiction works possibly lead to decreased performance. Our experiments could inform the creation of a compensation scheme for authors whose works contribute to AI development.
• 摘要：使用受版权保护的材料来训练生成式语言模型引发了严重的法律和道德问题。本文提出了一个框架，并给出了实证评估受版权保护的材料对挪威语大型语言模型 (LLM) 性能影响的结果。我们发现，当根据一组不同的挪威语基准对模型进行评估时，书籍和报纸都会产生积极影响，而小说作品可能会导致性能下降。我们的实验可以为那些作品对人工智能发展做出贡献的作者制定补偿计划提供参考。

Title: JuStRank: Benchmarking LLM Judges for System Ranking

• Authors: Ariel Gera, Odellia Boni, Yotam Perlitz, Roy Bar-Haim, Lilach Eden, Asaf Yehudai
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.09569
• Pdf URL: https://arxiv.org/pdf/2412.09569
• Copy Paste: [[2412.09569]] JuStRank: Benchmarking LLM Judges for System Ranking(https://arxiv.org/abs/2412.09569)
• Keywords: llm
• Abstract: Given the rapid progress of generative AI, there is a pressing need to systematically compare and choose between the numerous models and configurations available. The scale and versatility of such evaluations make the use of LLM-based judges a compelling solution for this challenge. Crucially, this approach requires first to validate the quality of the LLM judge itself. Previous work has focused on instance-based assessment of LLM judges, where a judge is evaluated over a set of responses, or response pairs, while being agnostic to their source systems. We argue that this setting overlooks critical factors affecting system-level ranking, such as a judge's positive or negative bias towards certain systems. To address this gap, we conduct the first large-scale study of LLM judges as system rankers. System scores are generated by aggregating judgment scores over multiple system outputs, and the judge's quality is assessed by comparing the resulting system ranking to a human-based ranking. Beyond overall judge assessment, our analysis provides a fine-grained characterization of judge behavior, including their decisiveness and bias.
• 摘要：鉴于生成式人工智能的快速发展，迫切需要系统地比较和选择众多可用的模型和配置。此类评估的规模和多功能性使得使用基于 LLM 的评委成为应对这一挑战的有力解决方案。至关重要的是，这种方法首先需要验证 LLM 评委本身的质量。之前的工作重点是 LLM 评委的基于实例的评估，其中对评委进行一组响应或响应对的评估，同时不考虑其源系统。我们认为这种设置忽略了影响系统级排名的关键因素，例如评委对某些系统的积极或消极偏见。为了解决这一差距，我们首次对 LLM 评委作为系统排名者进行了大规模研究。系统分数是通过汇总多个系统输出的判断分数生成的，并且通过将得到的系统排名与基于人类的排名进行比较来评估评委的质量。除了整体评委评估之外，我们的分析还提供了对评委行为的细粒度描述，包括他们的果断性和偏见。

Title: DiverseAgentEntropy: Quantifying Black-Box LLM Uncertainty through Diverse Perspectives and Multi-Agent Interaction

• Authors: Yu Feng, Phu Mon Htut, Zheng Qi, Wei Xiao, Manuel Mager, Nikolaos Pappas, Kishaloy Halder, Yang Li, Yassine Benajiba, Dan Roth
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.09572
• Pdf URL: https://arxiv.org/pdf/2412.09572
• Copy Paste: [[2412.09572]] DiverseAgentEntropy: Quantifying Black-Box LLM Uncertainty through Diverse Perspectives and Multi-Agent Interaction(https://arxiv.org/abs/2412.09572)
• Keywords: language model, llm, hallucination, agent
• Abstract: Quantifying the uncertainty in the factual parametric knowledge of Large Language Models (LLMs), especially in a black-box setting, poses a significant challenge. Existing methods, which gauge a model's uncertainty through evaluating self-consistency in responses to the original query, do not always capture true uncertainty. Models might respond consistently to the origin query with a wrong answer, yet respond correctly to varied questions from different perspectives about the same query, and vice versa. In this paper, we propose a novel method, DiverseAgentEntropy, for evaluating a model's uncertainty using multi-agent interaction under the assumption that if a model is certain, it should consistently recall the answer to the original query across a diverse collection of questions about the same original query. We further implement an abstention policy to withhold responses when uncertainty is high. Our method offers a more accurate prediction of the model's reliability and further detects hallucinations, outperforming other self-consistency-based methods. Additionally, it demonstrates that existing models often fail to consistently retrieve the correct answer to the same query under diverse varied questions even when knowing the correct answer.
• 摘要：量化大型语言模型 (LLM) 事实参数知识中的不确定性，尤其是在黑盒环境中，是一项重大挑战。现有方法通过评估对原始查询的响应中的自洽性来衡量模型的不确定性，但并不总是能捕捉到真正的不确定性。模型可能会对原始查询始终给出错误答案，但对同一查询的不同角度的各种问题却能做出正确回答，反之亦然。在本文中，我们提出了一种新方法 DiverseAgentEntropy，用于使用多智能体交互评估模型的不确定性，假设如果模型是确定的，它应该在关于同一原始查询的各种问题中始终如一地回忆起原始查询的答案。我们进一步实施弃权政策，在高度不确定时不予回应。我们的方法可以更准确地预测模型的可靠性，并进一步检测幻觉，优于其他基于自洽的方法。此外，它表明现有模型即使知道正确答案，也常常无法在不同的问题下始终如一地检索到同一查询的正确答案。

Title: OpenNER 1.0: Standardized Open-Access Named Entity Recognition Datasets in 50+ Languages

• Authors: Chester Palen-Michel, Maxwell Pickering, Maya Kruse, Jonne Sälevä, Constantine Lignos
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.09587
• Pdf URL: https://arxiv.org/pdf/2412.09587
• Copy Paste: [[2412.09587]] OpenNER 1.0: Standardized Open-Access Named Entity Recognition Datasets in 50+ Languages(https://arxiv.org/abs/2412.09587)
• Keywords: language model
• Abstract: We present OpenNER 1.0, a standardized collection of openly available named entity recognition (NER) datasets. OpenNER contains 34 datasets spanning 51 languages, annotated in varying named entity ontologies. We correct annotation format issues, standardize the original datasets into a uniform representation, map entity type names to be more consistent across corpora, and provide the collection in a structure that enables research in multilingual and multi-ontology NER. We provide baseline models using three pretrained multilingual language models to compare the performance of recent models and facilitate future research in NER.
• 摘要：我们推出了 OpenNER 1.0，这是一套标准化的公开命名实体识别 (NER) 数据集。OpenNER 包含 34 个数据集，涵盖 51 种语言，并以不同的命名实体本体进行注释。我们纠正了注释格式问题，将原始数据集标准化为统一的表示形式，将实体类型名称映射到语料库中以使其更加一致，并以支持多语言和多本体 NER 研究的结构提供该集合。我们使用三种预训练的多语言语言模型提供基线模型，以比较近期模型的性能并促进 NER 的未来研究。

Title: AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials

• Authors: Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, Tao Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2412.09605
• Pdf URL: https://arxiv.org/pdf/2412.09605
• Copy Paste: [[2412.09605]] AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials(https://arxiv.org/abs/2412.09605)
• Keywords: language model, agent
• Abstract: Graphical User Interface (GUI) agents hold great potential for automating complex tasks across diverse digital environments, from web applications to desktop software. However, the development of such agents is hindered by the lack of high-quality, multi-step trajectory data required for effective training. Existing approaches rely on expensive and labor-intensive human annotation, making them unsustainable at scale. To address this challenge, we propose AgentTrek, a scalable data synthesis pipeline that generates high-quality GUI agent trajectories by leveraging web tutorials. Our method automatically gathers tutorial-like texts from the internet, transforms them into task goals with step-by-step instructions, and employs a visual-language model agent to simulate their execution in a real digital environment. A VLM-based evaluator ensures the correctness of the generated trajectories. We demonstrate that training GUI agents with these synthesized trajectories significantly improves their grounding and planning performance over the current models. Moreover, our approach is more cost-efficient compared to traditional human annotation methods. This work underscores the potential of guided replay with web tutorials as a viable strategy for large-scale GUI agent training, paving the way for more capable and autonomous digital agents.
• 摘要：图形用户界面 (GUI) 代理在从 Web 应用程序到桌面软件等各种数字环境中自动执行复杂任务方面具有巨大潜力。然而，由于缺乏有效训练所需的高质量、多步骤轨迹数据，此类代理的开发受到阻碍。现有方法依赖于昂贵且劳动密集型的人工注释，因此无法大规模使用。为了应对这一挑战，我们提出了 AgentTrek，这是一种可扩展的数据合成管道，它利用 Web 教程生成高质量的 GUI 代理轨迹。我们的方法会自动从互联网上收集类似教程的文本，将它们转换为带有分步说明的任务目标，并使用视觉语言模型代理在真实数字环境中模拟它们的执行。基于 VLM 的评估器可确保生成的轨迹的正确性。我们证明，使用这些合成轨迹训练 GUI 代理可显著提高其与当前模型相比的接地和规划性能。此外，与传统的人工注释方法相比，我们的方法更具成本效益。这项工作强调了通过网络教程进行引导重放的潜力，作为大规模 GUI 代理训练的可行策略，为更强大、更自主的数字代理铺平了道路。