2024-08-26

Title: Macro-Queries: An Exploration into Guided Chart Generation from High Level Prompts

• Authors: Christopher J. Lee, Giorgio Tran, Roderick Tabalba, Jason Leigh, Ryan Longman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12726
• Pdf URL: https://arxiv.org/pdf/2408.12726
• Copy Paste: [[2408.12726]] Macro-Queries: An Exploration into Guided Chart Generation from High Level Prompts(https://arxiv.org/abs/2408.12726)
• Keywords: language model, llm, prompt
• Abstract: This paper explores the intersection of data visualization and Large Language Models (LLMs). Driven by the need to make a broader range of data visualization types accessible for novice users, we present a guided LLM-based pipeline designed to transform data, guided by high-level user questions (referred to as macro-queries), into a diverse set of useful visualizations. This approach leverages various prompting techniques, fine-tuning inspired by Abela's Chart Taxonomy, and integrated SQL tool usage.
• 摘要：本文探讨了数据可视化与大型语言模型 (LLM) 的交集。为了使新手用户能够使用更广泛的数据可视化类型，我们提出了一种基于 LLM 的引导式管道，旨在将数据在高级用户问题（称为宏查询）的引导下转换为一组多样化的有用可视化。这种方法利用了各种提示技术、受 Abela 图表分类法启发的微调以及集成的 SQL 工具使用。

Title: SLM Meets LLM: Balancing Latency, Interpretability and Consistency in Hallucination Detection

• Authors: Mengya Hu, Rui Xu, Deren Lei, Yaxi Li, Mingyu Wang, Emily Ching, Eslam Kamal, Alex Deng
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12748
• Pdf URL: https://arxiv.org/pdf/2408.12748
• Copy Paste: [[2408.12748]] SLM Meets LLM: Balancing Latency, Interpretability and Consistency in Hallucination Detection(https://arxiv.org/abs/2408.12748)
• Keywords: language model, llm, hallucination, prompt
• Abstract: Large language models (LLMs) are highly capable but face latency challenges in real-time applications, such as conducting online hallucination detection. To overcome this issue, we propose a novel framework that leverages a small language model (SLM) classifier for initial detection, followed by a LLM as constrained reasoner to generate detailed explanations for detected hallucinated content. This study optimizes the real-time interpretable hallucination detection by introducing effective prompting techniques that align LLM-generated explanations with SLM decisions. Empirical experiment results demonstrate its effectiveness, thereby enhancing the overall user experience.
• 摘要：大型语言模型 (LLM) 功能强大，但在实时应用中面临延迟挑战，例如进行在线幻觉检测。为了解决这个问题，我们提出了一个新颖的框架，该框架利用小型语言模型 (SLM) 分类器进行初始检测，然后使用 LLM 作为受约束的推理器为检测到的幻觉内容生成详细解释。本研究通过引入有效的提示技术优化了实时可解释的幻觉检测，这些提示技术将 LLM 生成的解释与 SLM 决策相结合。实证实验结果证明了其有效性，从而提升了整体用户体验。

Title: Investigating LLM Applications in E-Commerce

• Authors: Chester Palen-Michel, Ruixiang Wang, Yipeng Zhang, David Yu, Canran Xu, Zhe Wu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12779
• Pdf URL: https://arxiv.org/pdf/2408.12779
• Copy Paste: [[2408.12779]] Investigating LLM Applications in E-Commerce(https://arxiv.org/abs/2408.12779)
• Keywords: language model, llm
• Abstract: The emergence of Large Language Models (LLMs) has revolutionized natural language processing in various applications especially in e-commerce. One crucial step before the application of such LLMs in these fields is to understand and compare the performance in different use cases in such tasks. This paper explored the efficacy of LLMs in the e-commerce domain, focusing on instruction-tuning an open source LLM model with public e-commerce datasets of varying sizes and comparing the performance with the conventional models prevalent in industrial applications. We conducted a comprehensive comparison between LLMs and traditional pre-trained language models across specific tasks intrinsic to the e-commerce domain, namely classification, generation, summarization, and named entity recognition (NER). Furthermore, we examined the effectiveness of the current niche industrial application of very large LLM, using in-context learning, in e-commerce specific tasks. Our findings indicate that few-shot inference with very large LLMs often does not outperform fine-tuning smaller pre-trained models, underscoring the importance of task-specific model optimization.Additionally, we investigated different training methodologies such as single-task training, mixed-task training, and LoRA merging both within domain/tasks and between different tasks. Through rigorous experimentation and analysis, this paper offers valuable insights into the potential effectiveness of LLMs to advance natural language processing capabilities within the e-commerce industry.
• 摘要：大型语言模型 (LLM) 的出现彻底改变了各种应用尤其是电子商务中的自然语言处理。在这些领域应用此类 LLM 之前的一个关键步骤是了解和比较这些任务中不同用例的性能。本文探讨了 LLM 在电子商务领域的功效，重点使用不同大小的公共电子商务数据集对开源 LLM 模型进行指令调整，并将其性能与工业应用中流行的传统模型进行比较。我们在电子商务领域固有的特定任务（即分类、生成、摘要和命名实体识别 (NER)）中对 LLM 与传统预训练语言模型进行了全面比较。此外，我们使用上下文学习，研究了当前非常大型 LLM 在电子商务特定任务中的小众工业应用的有效性。我们的研究结果表明，使用非常大的 LLM 进行少量推理通常不如微调较小的预训练模型，这凸显了针对特定任务的模型优化的重要性。此外，我们还研究了不同的训练方法，例如单任务训练、混合任务训练和领域/任务内以及不同任务之间的 LoRA 合并。通过严格的实验和分析，本文提供了宝贵的见解，表明 LLM 在电子商务行业中提升自然语言处理能力的潜在有效性。

Title: Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translation

• Authors: Vivek Iyer, Bhavitvya Malik, Pavel Stepachev, Pinzhen Chen, Barry Haddow, Alexandra Birch
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12780
• Pdf URL: https://arxiv.org/pdf/2408.12780
• Copy Paste: [[2408.12780]] Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translation(https://arxiv.org/abs/2408.12780)
• Keywords: language model, llm
• Abstract: Despite the recent popularity of Large Language Models (LLMs) in Machine Translation (MT), their performance in low-resource translation still lags significantly behind Neural Machine Translation (NMT) models. In this paper, we explore what it would take to adapt LLMs for low-resource settings. In particular, we re-examine the role of two factors: a) the importance and application of parallel data, and b) diversity in Supervised Fine-Tuning (SFT). Recently, parallel data has been shown to be less important for MT using LLMs than in previous MT research. Similarly, diversity during SFT has been shown to promote significant transfer in LLMs across languages and tasks. However, for low-resource LLM-MT, we show that the opposite is true for both of these considerations: a) parallel data is critical during both pretraining and SFT, and b) diversity tends to cause interference, not transfer. Our experiments, conducted with 3 LLMs across 2 low-resourced language groups - indigenous American and North-East Indian - reveal consistent patterns in both cases, underscoring the generalizability of our findings. We believe these insights will be valuable for scaling to massively multilingual LLM-MT models that can effectively serve lower-resource languages.
• 摘要：尽管大型语言模型 (LLM) 最近在机器翻译 (MT) 中大受欢迎，但它们在低资源翻译中的表现仍然远远落后于神经机器翻译 (NMT) 模型。在本文中，我们探讨了如何使 LLM 适应低资源环境。特别是，我们重新审视了两个因素的作用：a) 并行数据的重要性和应用，以及 b) 监督微调 (SFT) 中的多样性。最近，研究表明，使用 LLM 进行机器翻译时，并行数据的重要性不如以前的机器翻译研究。同样，研究表明 SFT 期间的多样性可以促进 LLM 在语言和任务之间的显著迁移。然而，对于低资源 LLM-MT，我们表明这两个考虑因素的实际情况正好相反：a) 并行数据在预训练和 SFT 期间都至关重要，b) 多样性往往会造成干扰，而不是迁移。我们针对 2 个资源匮乏的语言群体（美洲土著和东北印第安）的 3 名 LLM 进行了实验，结果显示两种情况下都存在一致的模式，这凸显了我们的研究结果具有普遍性。我们相信这些见解对于扩展到能够有效服务于资源匮乏语言的大规模多语言 LLM-MT 模型将大有裨益。

Title: Less for More: Enhancing Preference Learning in Generative Language Models with Automated Self-Curation of Training Corpora

• Authors: JoonHo Lee, JuYoun Son, Juree Seok, Wooseok Jang, Yeong-Dae Kwon
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12799
• Pdf URL: https://arxiv.org/pdf/2408.12799
• Copy Paste: [[2408.12799]] Less for More: Enhancing Preference Learning in Generative Language Models with Automated Self-Curation of Training Corpora(https://arxiv.org/abs/2408.12799)
• Keywords: language model
• Abstract: Ambiguity in language presents challenges in developing more enhanced language models, particularly in preference learning, where variability among annotators results in inconsistently annotated datasets used for model alignment. To address this issue, we introduce a self-curation method that preprocesses annotated datasets by leveraging proxy models trained directly on these datasets. Our method enhances preference learning by automatically detecting and removing ambiguous annotations within the dataset. The proposed approach is validated through extensive experiments, demonstrating a marked improvement in performance across various instruction-following tasks. Our work provides a straightforward and reliable method to overcome annotation inconsistencies, serving as an initial step towards the development of more advanced preference learning techniques.
• 摘要：语言的歧义性给开发更强大的语言模型带来了挑战，尤其是在偏好学习中，注释者之间的差异导致用于模型对齐的注释数据集不一致。为了解决这个问题，我们引入了一种自我管理方法，该方法利用直接在这些数据集上训练的代理模型对注释数据集进行预处理。我们的方法通过自动检测和删除数据集中的歧义注释来增强偏好学习。所提出的方法通过大量实验得到验证，表明在各种指令遵循任务中性能显着提高。我们的工作提供了一种直接可靠的方法来克服注释不一致问题，这是开发更高级偏好学习技术的第一步。

Title: Grounding Fallacies Misrepresenting Scientific Publications in Evidence

• Authors: Max Glockner, Yufang Hou, Preslav Nakov, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12812
• Pdf URL: https://arxiv.org/pdf/2408.12812
• Copy Paste: [[2408.12812]] Grounding Fallacies Misrepresenting Scientific Publications in Evidence(https://arxiv.org/abs/2408.12812)
• Keywords: llm
• Abstract: Health-related misinformation claims often falsely cite a credible biomedical publication as evidence, which superficially appears to support the false claim. The publication does not really support the claim, but a reader could believe it thanks to the use of logical fallacies. Here, we aim to detect and to highlight such fallacies, which requires carefully assessing the exact content of the misrepresented publications. To achieve this, we introduce MissciPlus, an extension of the fallacy detection dataset Missci. MissciPlus builds on Missci by grounding the applied fallacies in real-world passages from misrepresented studies. This creates a realistic test-bed for detecting and verbalizing these fallacies under real-world input conditions, and enables novel passage-retrieval tasks. MissciPlus is the first logical fallacy dataset which pairs the real-world misrepresented evidence with incorrect claims, identical to the input to evidence-based fact-checking models. With MissciPlus, we i) benchmark retrieval models in identifying passages that support claims only when fallacies are applied, ii) evaluate how well LLMs articulate fallacious reasoning from misrepresented scientific passages, and iii) assess the effectiveness of fact-checking models in refuting claims that misrepresent biomedical research. Our findings show that current fact-checking models struggle to use relevant passages from misrepresented publications to refute misinformation. Moreover, these passages can mislead LLMs into accepting false claims as true.
• 摘要：与健康相关的错误信息经常错误地引用可信的生物医学出版物作为证据，从表面上看，这些证据似乎支持了错误的说法。该出版物实际上并不支持该说法，但由于使用了逻辑谬误，读者可能会相信它。在这里，我们的目标是检测和突出此类谬误，这需要仔细评估被歪曲的出版物的确切内容。为了实现这一目标，我们推出了 MissciPlus，它是谬误检测数据集 Missci 的扩展。MissciPlus 在 Missci 的基础上，将所应用的谬误建立在被歪曲研究的真实段落中。这为在真实输入条件下检测和表达这些谬误创建了一个现实的测试平台，并支持新颖的段落检索任务。MissciPlus 是第一个逻辑谬误数据集，它将真实世界中的歪曲证据与不正确的说法配对，与基于证据的事实核查模型的输入相同。使用 MissciPlus，我们 i) 对检索模型进行基准测试，以识别仅在应用谬误时才支持主张的段落，ii) 评估法学硕士如何表达来自歪曲的科学段落的谬误推理，以及 iii) 评估事实核查模型在驳斥歪曲生物医学研究的主张方面的有效性。我们的研究结果表明，当前的事实核查模型很难使用来自歪曲出版物的相关段落来驳斥错误信息。此外，这些段落可能会误导法学硕士将虚假主张视为真实。

Title: LIMP: Large Language Model Enhanced Intent-aware Mobility Prediction

• Authors: Songwei Li, Jie Feng, Jiawei Chi, Xinyuan Hu, Xiaomeng Zhao, Fengli Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12832
• Pdf URL: https://arxiv.org/pdf/2408.12832
• Copy Paste: [[2408.12832]] LIMP: Large Language Model Enhanced Intent-aware Mobility Prediction(https://arxiv.org/abs/2408.12832)
• Keywords: language model, llm, agent
• Abstract: Human mobility prediction is essential for applications like urban planning and transportation management, yet it remains challenging due to the complex, often implicit, intentions behind human behavior. Existing models predominantly focus on spatiotemporal patterns, paying less attention to the underlying intentions that govern movements. Recent advancements in large language models (LLMs) offer a promising alternative research angle for integrating commonsense reasoning into mobility prediction. However, it is a non-trivial problem because LLMs are not natively built for mobility intention inference, and they also face scalability issues and integration difficulties with spatiotemporal models. To address these challenges, we propose a novel LIMP (LLMs for Intent-ware Mobility Prediction) framework. Specifically, LIMP introduces an "Analyze-Abstract-Infer" (A2I) agentic workflow to unleash LLM's commonsense reasoning power for mobility intention inference. Besides, we design an efficient fine-tuning scheme to transfer reasoning power from commercial LLM to smaller-scale, open-source language model, ensuring LIMP's scalability to millions of mobility records. Moreover, we propose a transformer-based intention-aware mobility prediction model to effectively harness the intention inference ability of LLM. Evaluated on two real-world datasets, LIMP significantly outperforms baseline models, demonstrating improved accuracy in next-location prediction and effective intention inference. The interpretability of intention-aware mobility prediction highlights our LIMP framework's potential for real-world applications. Codes and data can be found in this https URL .
• 摘要：人类流动预测对于城市规划和交通管理等应用至关重要，但由于人类行为背后有着复杂且通常隐含的意图，因此它仍然具有挑战性。现有模型主要关注时空模式，较少关注控制运动的根本意图。大型语言模型 (LLM) 的最新进展为将常识推理融入流动预测提供了一个有前途的替代研究角度。然而，这是一个不小的问题，因为 LLM 不是为移动意图推理而构建的，而且它们还面临可扩展性问题和与时空模型的集成困难。为了应对这些挑战，我们提出了一种新颖的 LIMP（用于意图软件移动预测的 LLM）框架。具体来说，LIMP 引入了“分析-抽象-推断”（A2I）代理工作流，以释放 LLM 的常识推理能力来进行移动意图推理。此外，我们设计了一种有效的微调方案，将推理能力从商业 LLM 转移到规模较小的开源语言模型，确保 LIMP 可扩展到数百万条移动记录。此外，我们提出了一种基于变换器的意图感知移动预测模型，以有效利用 LLM 的意图推理能力。在两个真实世界数据集上进行评估后，LIMP 的表现明显优于基线模型，显示出下一个位置预测的准确性有所提高，并且意图推理有效。意图感知移动预测的可解释性凸显了我们的 LIMP 框架在实际应用中的潜力。代码和数据可以在这个 https URL 中找到。

Title: CLLMFS: A Contrastive Learning enhanced Large Language Model Framework for Few-Shot Named Entity Recognition

• Authors: Yafeng Zhang, Zilan Yu, Yuang Huang, Jing Tang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12834
• Pdf URL: https://arxiv.org/pdf/2408.12834
• Copy Paste: [[2408.12834]] CLLMFS: A Contrastive Learning enhanced Large Language Model Framework for Few-Shot Named Entity Recognition(https://arxiv.org/abs/2408.12834)
• Keywords: language model, llm, prompt
• Abstract: Few-shot Named Entity Recognition (NER), the task of identifying named entities with only a limited amount of labeled data, has gained increasing significance in natural language processing. While existing methodologies have shown some effectiveness, such as enriching label semantics through various prompting modes or employing metric learning techniques, their performance exhibits limited robustness across diverse domains due to the lack of rich knowledge in their pre-trained models. To address this issue, we propose CLLMFS, a Contrastive Learning enhanced Large Language Model (LLM) Framework for Few-Shot Named Entity Recognition, achieving promising results with limited training data. Considering the impact of LLM's internal representations on downstream tasks, CLLMFS integrates Low-Rank Adaptation (LoRA) and contrastive learning mechanisms specifically tailored for few-shot NER. By enhancing the model's internal representations, CLLMFS effectively improves both entity boundary awareness ability and entity recognition accuracy. Our method has achieved state-of-the-art performance improvements on F1-score ranging from 2.58\% to 97.74\% over existing best-performing methods across several recognized benchmarks. Furthermore, through cross-domain NER experiments conducted on multiple datasets, we have further validated the robust generalization capability of our method. Our code will be released in the near future.
• 摘要：少样本命名实体识别 (NER) 是指使用有限数量的标记数据来识别命名实体的任务，其在自然语言处理中的重要性日益凸显。尽管现有方法已展现出一些有效性，例如通过各种提示模式或采用度量学习技术来丰富标签语义，但由于预训练模型缺乏丰富的知识，其性能在不同领域中表现出有限的鲁棒性。针对这一问题，我们提出了 CLLMFS，一种用于少样本命名实体识别的对比学习增强型大型语言模型 (LLM) 框架，在有限的训练数据下取得了良好的效果。考虑到 LLM 的内部表示对下游任务的影响，CLLMFS 集成了专门针对少样本 NER 的低秩自适应 (LoRA) 和对比学习机制。通过增强模型的内部表示，CLLMFS 有效地提高了实体边界感知能力和实体识别准确率。我们的方法在 F1 得分上取得了最先进的性能改进，在多个公认的基准测试中比现有的最佳方法提高了 2.58% 到 97.74%。此外，通过在多个数据集上进行的跨域 NER 实验，我们进一步验证了我们方法的强大泛化能力。我们的代码将在不久的将来发布。

Title: Causal-Guided Active Learning for Debiasing Large Language Models

• Authors: Zhouhao Sun, Li Du, Xiao Ding, Yixuan Ma, Kaitao Qiu, Ting Liu, Bing Qin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12942
• Pdf URL: https://arxiv.org/pdf/2408.12942
• Copy Paste: [[2408.12942]] Causal-Guided Active Learning for Debiasing Large Language Models(https://arxiv.org/abs/2408.12942)
• Keywords: language model, llm
• Abstract: Although achieving promising performance, recent analyses show that current generative large language models (LLMs) may still capture dataset biases and utilize them for generation, leading to poor generalizability and harmfulness of LLMs. However, due to the diversity of dataset biases and the over-optimization problem, previous prior-knowledge-based debiasing methods and fine-tuning-based debiasing methods may not be suitable for current LLMs. To address this issue, we explore combining active learning with the causal mechanisms and propose a casual-guided active learning (CAL) framework, which utilizes LLMs itself to automatically and autonomously identify informative biased samples and induce the bias patterns. Then a cost-effective and efficient in-context learning based method is employed to prevent LLMs from utilizing dataset biases during generation. Experimental results show that CAL can effectively recognize typical biased instances and induce various bias patterns for debiasing LLMs.
• 摘要：尽管取得了令人鼓舞的性能，但最近的分析表明，当前的生成式大型语言模型 (LLM) 仍可能捕获数据集偏差并将其用于生成，从而导致 LLM 的泛化能力较差且有害。然而，由于数据集偏差的多样性和过度优化问题，以前基于先验知识的去偏方法和基于微调的去偏方法可能不适合当前的 LLM。针对这一问题，我们探索将主动学习与因果机制相结合，并提出了一种因果引导的主动学习 (CAL) 框架，该框架利用 LLM 本身自动和自主地识别信息偏差样本并归纳偏差模式。然后采用一种经济高效且基于上下文学习的方法来防止 LLM 在生成过程中利用数据集偏差。实验结果表明，CAL 可以有效识别典型的偏差实例并归纳出各种偏差模式来去偏 LLM。

Title: Multimodal Contrastive In-Context Learning

• Authors: Yosuke Miyanishi, Minh Le Nguyen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.12959
• Pdf URL: https://arxiv.org/pdf/2408.12959
• Copy Paste: [[2408.12959]] Multimodal Contrastive In-Context Learning(https://arxiv.org/abs/2408.12959)
• Keywords: language model, llm
• Abstract: The rapid growth of Large Language Models (LLMs) usage has highlighted the importance of gradient-free in-context learning (ICL). However, interpreting their inner workings remains challenging. This paper introduces a novel multimodal contrastive in-context learning framework to enhance our understanding of ICL in LLMs. First, we present a contrastive learning-based interpretation of ICL in real-world settings, marking the distance of the key-value representation as the differentiator in ICL. Second, we develop an analytical framework to address biases in multimodal input formatting for real-world datasets. We demonstrate the effectiveness of ICL examples where baseline performance is poor, even when they are represented in unseen formats. Lastly, we propose an on-the-fly approach for ICL (Anchored-by-Text ICL) that demonstrates effectiveness in detecting hateful memes, a task where typical ICL struggles due to resource limitations. Extensive experiments on multimodal datasets reveal that our approach significantly improves ICL performance across various scenarios, such as challenging tasks and resource-constrained environments. Moreover, it provides valuable insights into the mechanisms of in-context learning in LLMs. Our findings have important implications for developing more interpretable, efficient, and robust multimodal AI systems, especially in challenging tasks and resource-constrained environments.
• 摘要：大型语言模型 (LLM) 的使用量快速增长，凸显了无梯度上下文学习 (ICL) 的重要性。然而，解释其内部工作原理仍然具有挑战性。本文介绍了一种新颖的多模态对比上下文学习框架，以增强我们对 LLM 中 ICL 的理解。首先，我们介绍了一种基于对比学习的 ICL 在现实世界环境中的解释，将键值表示的距离标记为 ICL 中的区分因素。其次，我们开发了一个分析框架来解决现实世界数据集的多模态输入格式中的偏差。我们证明了 ICL 示例的有效性，即使它们以看不见的格式表示，基线性能也很差。最后，我们提出了一种即时 ICL（文本锚定 ICL）方法，该方法证明了在检测仇恨模因方面的有效性，而典型的 ICL 由于资源限制而难以完成这项任务。在多模态数据集上进行的大量实验表明，我们的方法显着提高了各种场景中的 ICL 性能，例如具有挑战性的任务和资源受限的环境。此外，它还为 LLM 中的情境学习机制提供了宝贵见解。我们的研究结果对于开发更具解释性、更高效、更强大的多模态 AI 系统具有重要意义，尤其是在具有挑战性的任务和资源受限的环境中。

Title: Open Llama2 Model for the Lithuanian Language

• Authors: Artūras Nakvosas, Povilas Daniušis, Vytas Mulevičius
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.12963
• Pdf URL: https://arxiv.org/pdf/2408.12963
• Copy Paste: [[2408.12963]] Open Llama2 Model for the Lithuanian Language(https://arxiv.org/abs/2408.12963)
• Keywords: language model, llm
• Abstract: In this paper, we propose and describe the first open Llama2 large language models (LLMs) for the Lithuanian language, including an accompanying question/answer (Q/A) dataset and translations of popular LLM benchmarks. We provide a brief review of open regional LLMs and detailed information on the proposed LLMs and their training process. We also conduct an empirical evaluation, comparing the perplexities of the proposed LLMs with those of other modern open LLMs. In addition, benchmarking the proposed LLMs against language understanding tasks reveals that high-quality pretraining datasets may be essential for achieving models that perform efficiently on these benchmarks. The full realisations of the described LLMs are available in the accompanying open repository~\url{this https URL}.
• 摘要：在本文中，我们提出并描述了第一个针对立陶宛语的开放 Llama2 大型语言模型 (LLM)，包括随附的问答 (Q/A) 数据集和流行 LLM 基准的翻译。我们简要回顾了开放的区域 LLM，并详细介绍了所提出的 LLM 及其训练过程。我们还进行了实证评估，将所提出的 LLM 的困惑度与其他现代开放 LLM 的困惑度进行了比较。此外，根据语言理解任务对所提出的 LLM 进行基准测试表明，高质量的预训练数据集对于实现在这些基准上表现高效的模型可能至关重要。所述 LLM 的完整实现可在随附的开放存储库~\url{此 https URL} 中找到。

Title: Internal and External Knowledge Interactive Refinement Framework for Knowledge-Intensive Question Answering

• Authors: Haowei Du, Dongyan Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.12979
• Pdf URL: https://arxiv.org/pdf/2408.12979
• Copy Paste: [[2408.12979]] Internal and External Knowledge Interactive Refinement Framework for Knowledge-Intensive Question Answering(https://arxiv.org/abs/2408.12979)
• Keywords: llm, hallucination, prompt
• Abstract: Recent works have attempted to integrate external knowledge into LLMs to address the limitations and potential factual errors in LLM-generated content. However, how to retrieve the correct knowledge from the large amount of external knowledge imposes a challenge. To this end, we empirically observe that LLMs have already encoded rich knowledge in their pretrained parameters and utilizing these internal knowledge improves the retrieval of external knowledge when applying them to knowledge-intensive tasks. In this paper, we propose a new internal and external knowledge interactive refinement paradigm dubbed IEKR to utilize internal knowledge in LLM to help retrieve relevant knowledge from the external knowledge base, as well as exploit the external knowledge to refine the hallucination of generated internal knowledge. By simply adding a prompt like 'Tell me something about' to the LLMs, we try to review related explicit knowledge and insert them with the query into the retriever for external retrieval. The external knowledge is utilized to complement the internal knowledge into input of LLM for answers. We conduct experiments on 3 benchmark datasets in knowledge-intensive question answering task with different LLMs and domains, achieving the new state-of-the-art. Further analysis shows the effectiveness of different modules in our approach.
• 摘要：最近的研究尝试将外部知识整合到 LLM 中，以解决 LLM 生成内容的局限性和潜在的事实错误。然而，如何从大量的外部知识中检索正确的知识是一个挑战。为此，我们通过经验观察到 LLM 已经在其预训练参数中编码了丰富的知识，利用这些内部知识可以在将其应用于知识密集型任务时提高外部知识的检索。在本文中，我们提出了一种新的内部和外部知识交互细化范式 IEKR，利用 LLM 中的内部知识帮助从外部知识库中检索相关知识，并利用外部知识来细化生成的内部知识的幻觉。通过在 LLM 中简单地添加“告诉我一些关于”之类的提示，我们尝试查看相关的显性知识并将它们与查询一起插入到检索器中进行外部检索。外部知识被用来补充内部知识作为 LLM 的答案输入。我们在知识密集型问答任务的 3 个基准数据集上用不同的 LLM 和领域进行了实验，取得了新的最佳成果。进一步的分析表明了我们方法中不同模块的有效性。

Title: Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates

• Authors: Hui Wei, Shenghua He, Tian Xia, Andy Wong, Jingyang Lin, Mei Han
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13006
• Pdf URL: https://arxiv.org/pdf/2408.13006
• Copy Paste: [[2408.13006]] Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates(https://arxiv.org/abs/2408.13006)
• Keywords: language model, gpt, llm, prompt
• Abstract: Alignment approaches such as RLHF and DPO are actively investigated to align large language models (LLMs) with human preferences. Commercial large language models (LLMs) like GPT-4 have been recently employed to evaluate and compare different LLM alignment approaches. These models act as surrogates for human evaluators due to their promising abilities to approximate human preferences with remarkably faster feedback and lower costs. This methodology is referred to as LLM-as-a-judge. However, concerns regarding its reliability have emerged, attributed to LLM judges' biases and inconsistent decision-making. Previous research has sought to develop robust evaluation frameworks for assessing the reliability of LLM judges and their alignment with human preferences. However, the employed evaluation metrics often lack adequate explainability and fail to address the internal inconsistency of LLMs. Additionally, existing studies inadequately explore the impact of various prompt templates when applying LLM-as-a-judge methods, which leads to potentially inconsistent comparisons between different alignment algorithms. In this work, we systematically evaluate LLM judges on alignment tasks (e.g. summarization) by defining evaluation metrics with improved theoretical interpretability and disentangling reliability metrics with LLM internal inconsistency. We develop a framework to evaluate, compare, and visualize the reliability and alignment of LLM judges to provide informative observations that help choose LLM judges for alignment tasks. Our results indicate a significant impact of prompt templates on LLM judge performance, as well as a mediocre alignment level between the tested LLM judges and human evaluators.
• 摘要：人们积极研究对齐方法（例如 RLHF 和 DPO），以使大型语言模型 (LLM) 与人类偏好对齐。最近，人们使用 GPT-4 等商业大型语言模型 (LLM) 来评估和比较不同的 LLM 对齐方法。这些模型可以替代人类评估者，因为它们具有以更快的反馈速度和更低的成本近似人类偏好的潜力。这种方法被称为 LLM-as-a-judge。然而，人们开始担心它的可靠性，这归因于 LLM 评委的偏见和不一致的决策。先前的研究试图开发强大的评估框架来评估 LLM 评委的可靠性及其与人类偏好的一致性。然而，所采用的评估指标通常缺乏足够的可解释性，无法解决 LLM 的内部不一致性问题。此外，现有研究在应用 LLM-as-a-judge 方法时没有充分探索各种提示模板的影响，这导致不同对齐算法之间的比较可能不一致。在这项工作中，我们通过定义具有改进的理论可解释性的评估指标并将可靠性指标与 LLM 内部不一致性区分开来，系统地评估 LLM 评委在对齐任务（例如摘要）上的表现。我们开发了一个框架来评估、比较和可视化 LLM 评委的可靠性和对齐性，以提供有用的观察结果，帮助选择 LLM 评委执行对齐任务。我们的结果表明，提示模板对 LLM 评委的表现有显著影响，并且测试的 LLM 评委和人类评估者之间的对齐水平一般。

Title: In-Context Learning with Reinforcement Learning for Incomplete Utterance Rewriting

• Authors: Haowei Du, Dongyan Zhao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13028
• Pdf URL: https://arxiv.org/pdf/2408.13028
• Copy Paste: [[2408.13028]] In-Context Learning with Reinforcement Learning for Incomplete Utterance Rewriting(https://arxiv.org/abs/2408.13028)
• Keywords: language model, llm
• Abstract: In-context learning (ICL) of large language models (LLMs) has attracted increasing attention in the community where LLMs make predictions only based on instructions augmented with a few examples. Existing example selection methods for ICL utilize sparse or dense retrievers and derive effective performance. However, these methods do not utilize direct feedback of LLM to train the retriever and the examples selected can not necessarily improve the analogy ability of LLM. To tackle this, we propose our policy-based reinforcement learning framework for example selection (RLS), which consists of a language model (LM) selector and an LLM generator. The LM selector encodes the candidate examples into dense representations and selects the top-k examples into the demonstration for LLM. The outputs of LLM are adopted to compute the reward and policy gradient to optimize the LM selector. We conduct experiments on different datasets and significantly outperform existing example selection methods. Moreover, our approach shows advantages over supervised finetuning (SFT) models in few shot setting. Further experiments show the balance of abundance and the similarity with the test case of examples is important for ICL performance of LLM.
• 摘要：大型语言模型 (LLM) 的上下文学习 (ICL) 在社区中引起了越来越多的关注，其中 LLM 仅基于用一些示例增强的指令进行预测。现有的 ICL 示例选择方法利用稀疏或密集检索器并获得有效的性能。然而，这些方法不利用 LLM 的直接反馈来训练检索器，所选示例不一定能提高 LLM 的类比能力。为了解决这个问题，我们提出了基于策略的强化学习示例选择 (RLS) 框架，它由语言模型 (LM) 选择器和 LLM 生成器组成。LM 选择器将候选示例编码为密集表示，并选择前 k 个示例作为 LLM 的演示。采用 LLM 的输出来计算奖励和策略梯度以优化 LM 选择器。我们在不同的数据集上进行了实验，并且明显优于现有的示例选择方法。此外，我们的方法在少量设置中显示出优于监督微调 (SFT) 模型。进一步的实验表明，示例的丰富度和与测试用例的相似性的平衡对于LLM的ICL性能至关重要。

Title: Can LLM be a Good Path Planner based on Prompt Engineering? Mitigating the Hallucination for Path Planning

• Authors: Hourui Deng, Hongjie Zhang, Jie Ou, Chaosheng Feng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13184
• Pdf URL: https://arxiv.org/pdf/2408.13184
• Copy Paste: [[2408.13184]] Can LLM be a Good Path Planner based on Prompt Engineering? Mitigating the Hallucination for Path Planning(https://arxiv.org/abs/2408.13184)
• Keywords: language model, llm, hallucination, prompt
• Abstract: Spatial reasoning in Large Language Models (LLMs) is the foundation for embodied intelligence. However, even in simple maze environments, LLMs still encounter challenges in long-term path-planning, primarily influenced by their spatial hallucination and context inconsistency hallucination by long-term reasoning. To address this challenge, this study proposes an innovative model, Spatial-to-Relational Transformation and Curriculum Q-Learning (S2RCQL). To address the spatial hallucination of LLMs, we propose the Spatial-to-Relational approach, which transforms spatial prompts into entity relations and paths representing entity relation chains. This approach fully taps the potential of LLMs in terms of sequential thinking. As a result, we design a path-planning algorithm based on Q-learning to mitigate the context inconsistency hallucination, which enhances the reasoning ability of LLMs. Using the Q-value of state-action as auxiliary information for prompts, we correct the hallucinations of LLMs, thereby guiding LLMs to learn the optimal path. Finally, we propose a reverse curriculum learning technique based on LLMs to further mitigate the context inconsistency hallucination. LLMs can rapidly accumulate successful experiences by reducing task difficulty and leveraging them to tackle more complex tasks. We performed comprehensive experiments based on Baidu's self-developed LLM: ERNIE-Bot 4.0. The results showed that our S2RCQL achieved a 23%--40% improvement in both success and optimality rates compared with advanced prompt engineering.
• 摘要：大型语言模型（LLM）的空间推理是具身智能的基础。然而，即使在简单的迷宫环境中，LLM 在长期路径规划中仍然会遇到挑战，这主要受到长期推理产生的空间幻觉和上下文不一致幻觉的影响。为了应对这一挑战，本研究提出了一个创新模型——空间到关系转换和课程 Q 学习（S2RCQL）。为了解决 LLM 的空间幻觉，我们提出了空间到关系的方法，将空间提示转换为实体关系和表示实体关系链的路径。这种方法充分挖掘了 LLM 在顺序思维方面的潜力。因此，我们设计了一种基于 Q 学习的路径规划算法来缓解上下文不一致幻觉，从而增强 LLM 的推理能力。利用状态-动作的Q值作为提示的辅助信息，我们纠正了LLM的幻觉，从而引导LLM学习到最优路径。最后，我们提出了一种基于LLM的逆向课程学习技术，以进一步缓解上下文不一致的幻觉。通过降低任务难度，LLM可以快速积累成功经验，并利用这些经验来解决更复杂的任务。我们基于百度自研的LLM：ERNIE-Bot 4.0进行了全面的实验。结果表明，与先进的提示工程相比，我们的S2RCQL在成功率和最优率方面都实现了23%--40%的提升。

Title: Domain-specific long text classification from sparse relevant information

• Authors: Célia D'Cruz, Jean-Marc Bereder, Frédéric Precioso, Michel Riveill
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.13253
• Pdf URL: https://arxiv.org/pdf/2408.13253
• Copy Paste: [[2408.13253]] Domain-specific long text classification from sparse relevant information(https://arxiv.org/abs/2408.13253)
• Keywords: language model
• Abstract: Large Language Models have undoubtedly revolutionized the Natural Language Processing field, the current trend being to promote one-model-for-all tasks (sentiment analysis, translation, etc.). However, the statistical mechanisms at work in the larger language models struggle to exploit the relevant information when it is very sparse, when it is a weak signal. This is the case, for example, for the classification of long domain-specific documents, when the relevance relies on a single relevant word or on very few relevant words from technical jargon. In the medical domain, it is essential to determine whether a given report contains critical information about a patient's condition. This critical information is often based on one or few specific isolated terms. In this paper, we propose a hierarchical model which exploits a short list of potential target terms to retrieve candidate sentences and represent them into the contextualized embedding of the target term(s) they contain. A pooling of the term(s) embedding(s) entails the document representation to be classified. We evaluate our model on one public medical document benchmark in English and on one private French medical dataset. We show that our narrower hierarchical model is better than larger language models for retrieving relevant long documents in a domain-specific context.
• 摘要：大型语言模型无疑彻底改变了自然语言处理领域，目前的趋势是推广一模型适用于所有任务（情感分析、翻译等）。然而，当相关信息非常稀疏、信号微弱时，大型语言模型中的统计机制很难利用相关信息。例如，在对长领域特定文档进行分类时，相关性依赖于单个相关词或技术术语中的极少数相关词。在医学领域，确定给定报告是否包含有关患者病情的关键信息至关重要。这些关键信息通常基于一个或几个特定的​​孤立术语。在本文中，我们提出了一个分层模型，该模型利用潜在目标术语的简短列表来检索候选句子，并将它们表示为它们包含的目标术语的上下文化嵌入。术语嵌入的池化需要对文档表示进行分类。我们在一个公共的英文医学文档基准和一个私人的法文医学数据集上评估我们的模型。我们表明，我们的较窄层次模型比较大的语言模型更适合在特定领域的上下文中检索相关的长文档。