2024-09-02

Title: Inductive Learning of Logical Theories with LLMs: A Complexity-graded Analysis

• Authors: João Pedro Gandarela, Danilo S. Carvalho, André Freitas
• Subjects: cs.CL, cs.AI, cs.LO
• Abstract URL: https://arxiv.org/abs/2408.16779
• Pdf URL: https://arxiv.org/pdf/2408.16779
• Copy Paste: [[2408.16779]] Inductive Learning of Logical Theories with LLMs: A Complexity-graded Analysis(https://arxiv.org/abs/2408.16779)
• Keywords: language model, llm
• Abstract: This work presents a novel systematic methodology to analyse the capabilities and limitations of Large Language Models (LLMs) with feedback from a formal inference engine, on logic theory induction. The analysis is complexity-graded w.r.t. rule dependency structure, allowing quantification of specific inference challenges on LLM performance. Integrating LLMs with formal methods is a promising frontier in the Natural Language Processing field, as an important avenue for improving model inference control and explainability. In particular, inductive learning over complex sets of facts and rules, poses unique challenges for current autoregressive models, as they lack explicit symbolic grounding. While they can be complemented by formal systems, the properties delivered by LLMs regarding inductive learning, are not well understood and quantified. Empirical results indicate that the largest LLMs can achieve competitive results against a SOTA Inductive Logic Programming (ILP) system baseline, but also that tracking long predicate relationship chains is a more difficult obstacle than theory complexity for the LLMs.
• 摘要：这项研究提出了一种新颖的系统方法来分析大型语言模型 (LLM) 在逻辑理论归纳方面的能力和局限性，这些模型具有来自形式推理引擎的反馈。分析是相对于规则依赖结构的复杂性分级的，可以量化 LLM 性能的特定推理挑战。将 LLM 与形式化方法相结合是自然语言处理领域的一个有前途的前沿，是提高模型推理控制和可解释性的重要途径。特别是，对复杂的事实和规则集进行归纳学习，对当前的自回归模型提出了独特的挑战，因为它们缺乏明确的符号基础。虽然它们可以通过形式化系统来补充，但 LLM 提供的关于归纳学习的属性尚未得到很好的理解和量化。实证结果表明，最大的 LLM 可以取得与 SOTA 归纳逻辑编程 (ILP) 系统基线相媲美的结果，但跟踪长谓词关系链对 LLM 来说是一个比理论复杂性更困难的障碍。

Title: LLaVA-Chef: A Multi-modal Generative Model for Food Recipes

• Authors: Fnu Mohbat, Mohammed J. Zaki
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.16889
• Pdf URL: https://arxiv.org/pdf/2408.16889
• Copy Paste: [[2408.16889]] LLaVA-Chef: A Multi-modal Generative Model for Food Recipes(https://arxiv.org/abs/2408.16889)
• Keywords: language model, gpt, llm, prompt
• Abstract: In the rapidly evolving landscape of online recipe sharing within a globalized context, there has been a notable surge in research towards comprehending and generating food recipes. Recent advancements in large language models (LLMs) like GPT-2 and LLaVA have paved the way for Natural Language Processing (NLP) approaches to delve deeper into various facets of food-related tasks, encompassing ingredient recognition and comprehensive recipe generation. Despite impressive performance and multi-modal adaptability of LLMs, domain-specific training remains paramount for their effective application. This work evaluates existing LLMs for recipe generation and proposes LLaVA-Chef, a novel model trained on a curated dataset of diverse recipe prompts in a multi-stage approach. First, we refine the mapping of visual food image embeddings to the language space. Second, we adapt LLaVA to the food domain by fine-tuning it on relevant recipe data. Third, we utilize diverse prompts to enhance the model's recipe comprehension. Finally, we improve the linguistic quality of generated recipes by penalizing the model with a custom loss function. LLaVA-Chef demonstrates impressive improvements over pretrained LLMs and prior works. A detailed qualitative analysis reveals that LLaVA-Chef generates more detailed recipes with precise ingredient mentions, compared to existing approaches.
• 摘要：在全球化背景下，在线菜谱共享领域发展迅速，对理解和生成食物菜谱的研究出现了显著增长。GPT-2 和 LLaVA 等大型语言模型 (LLM) 的最新进展为自然语言处理 (NLP) 方法深入研究与食物相关的任务的各个方面铺平了道路，包括成分识别和全面的菜谱生成。尽管 LLM 具有令人印象深刻的性能和多模态适应性，但特定领域的训练对于其有效应用仍然至关重要。这项工作评估了现有的用于菜谱生成的 LLM，并提出了 LLaVA-Chef，这是一种采用多阶段方法在精选的多样化菜谱提示数据集上进行训练的新模型。首先，我们改进了视觉食物图像嵌入到语言空间的映射。其次，我们通过对相关菜谱数据进行微调，使 LLaVA 适应食物领域。第三，我们利用不同的提示来增强模型对菜谱的理解。最后，我们通过使用自定义损失函数惩罚模型来提高生成的食谱的语言质量。LLaVA-Chef 比预训练的 LLM 和之前的作品表现出了显著的改进。详细的定性分析表明，与现有方法相比，LLaVA-Chef 可以生成更详细的食谱，并精确提及配料。

Title: Plausible-Parrots @ MSP2023: Enhancing Semantic Plausibility Modeling using Entity and Event Knowledge

• Authors: Chong Shen, Chenyue Zhou
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.16937
• Pdf URL: https://arxiv.org/pdf/2408.16937
• Copy Paste: [[2408.16937]] Plausible-Parrots @ MSP2023: Enhancing Semantic Plausibility Modeling using Entity and Event Knowledge(https://arxiv.org/abs/2408.16937)
• Keywords: language model, llm
• Abstract: In this work, we investigate the effectiveness of injecting external knowledge to a large language model (LLM) to identify semantic plausibility of simple events. Specifically, we enhance the LLM with fine-grained entity types, event types and their definitions extracted from an external knowledge base. These knowledge are injected into our system via designed templates. We also augment the data to balance the label distribution and adapt the task setting to real world scenarios in which event mentions are expressed as natural language sentences. The experimental results show the effectiveness of the injected knowledge on modeling semantic plausibility of events. An error analysis further emphasizes the importance of identifying non-trivial entity and event types.
• 摘要：在这项工作中，我们研究了将外部知识注入大型语言模型 (LLM) 以识别简单事件的语义合理性的有效性。具体来说，我们使用从外部知识库中提取的细粒度实体类型、事件类型及其定义来增强 LLM。这些知识通过设计的模板注入到我们的系统中。我们还扩充数据以平衡标签分布并使任务设置适应现实世界场景，其中事件提及以自然语言句子的形式表达。实验结果表明注入的知识在建模事件的语义合理性方面是有效的。错误分析进一步强调了识别非平凡实体和事件类型的重要性。

Title: A longitudinal sentiment analysis of Sinophobia during COVID-19 using large language models

• Authors: Chen Wang, Rohitash Chandra
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.16942
• Pdf URL: https://arxiv.org/pdf/2408.16942
• Copy Paste: [[2408.16942]] A longitudinal sentiment analysis of Sinophobia during COVID-19 using large language models(https://arxiv.org/abs/2408.16942)
• Keywords: language model, llm
• Abstract: The COVID-19 pandemic has exacerbated xenophobia, particularly Sinophobia, leading to widespread discrimination against individuals of Chinese descent. Large language models (LLMs) are pre-trained deep learning models used for natural language processing (NLP) tasks. The ability of LLMs to understand and generate human-like text makes them particularly useful for analysing social media data to detect and evaluate sentiments. We present a sentiment analysis framework utilising LLMs for longitudinal sentiment analysis of the Sinophobic sentiments expressed in X (Twitter) during the COVID-19 pandemic. The results show a significant correlation between the spikes in Sinophobic tweets, Sinophobic sentiments and surges in COVID-19 cases, revealing that the evolution of the pandemic influenced public sentiment and the prevalence of Sinophobic discourse. Furthermore, the sentiment analysis revealed a predominant presence of negative sentiments, such as annoyance and denial, which underscores the impact of political narratives and misinformation shaping public opinion. The lack of empathetic sentiment which was present in previous studies related to COVID-19 highlights the way the political narratives in media viewed the pandemic and how it blamed the Chinese community. Our study highlights the importance of transparent communication in mitigating xenophobic sentiments during global crises.
• 摘要：COVID-19 疫情加剧了仇外心理，尤其是仇华情绪，导致对华裔人士的普遍歧视。大型语言模型 (LLM) 是用于自然语言处理 (NLP) 任务的预训练深度学习模型。LLM 能够理解和生成类似人类的文本，这使得它们特别适合用于分析社交媒体数据以检测和评估情绪。我们提出了一个情绪分析框架，利用 LLM 对 COVID-19 疫情期间 X（推特）中表达的仇华情绪进行纵向情绪分析。结果显示，仇华推文的激增、仇华情绪和 COVID-19 病例激增之间存在显著相关性，表明疫情的发展影响了公众情绪和仇华言论的盛行。此外，情绪分析显示，恼怒和否认等负面情绪占主导地位，这凸显了政治叙事和错误信息对公众舆论的影响。此前与 COVID-19 相关的研究中，人们缺乏同理心，这凸显了媒体的政治叙事对疫情的看法以及它如何将责任归咎于华人社区。我们的研究强调了透明沟通对于缓解全球危机期间的仇外情绪的重要性。

Title: MemLong: Memory-Augmented Retrieval for Long Text Modeling

• Authors: Weijie Liu, Zecheng Tang, Juntao Li, Kehai Chen, Min Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.16967
• Pdf URL: https://arxiv.org/pdf/2408.16967
• Copy Paste: [[2408.16967]] MemLong: Memory-Augmented Retrieval for Long Text Modeling(https://arxiv.org/abs/2408.16967)
• Keywords: language model, llm, long context
• Abstract: Recent advancements in Large Language Models (LLMs) have yielded remarkable success across diverse fields. However, handling long contexts remains a significant challenge for LLMs due to the quadratic time and space complexity of attention mechanisms and the growing memory consumption of the key-value cache during generation. This work introduces MemLong: Memory-Augmented Retrieval for Long Text Generation, a method designed to enhance the capabilities of long-context language modeling by utilizing an external retriever for historical information retrieval. MemLong combines a non-differentiable ``ret-mem'' module with a partially trainable decoder-only language model and introduces a fine-grained, controllable retrieval attention mechanism that leverages semantic-level relevant chunks. Comprehensive evaluations on multiple long-context language modeling benchmarks demonstrate that MemLong consistently outperforms other state-of-the-art LLMs. More importantly, MemLong can extend the context length on a single 3090 GPU from 4k up to 80k. Our code is available at this https URL
• 摘要：大型语言模型 (LLM) 的最新进展已在不同领域取得了显著成功。然而，由于注意力机制的二次时间和空间复杂度以及生成过程中键值缓存的不断增长的内存消耗，处理长上下文仍然是 LLM 面临的重大挑战。这项工作引入了 MemLong：用于长文本生成的内存增强检索，这种方法旨在通过利用外部检索器进行历史信息检索来增强长上下文语言建模的能力。MemLong 将不可微分的“ret-mem”模块与部分可训练的解码器专用语言模型相结合，并引入了一种细粒度、可控的检索注意力机制，该机制利用了语义级相关块。对多个长上下文语言建模基准的综合评估表明，MemLong 始终优于其他最先进的 LLM。更重要的是，MemLong 可以将单个 3090 GPU 上的上下文长度从 4k 扩展到 80k。我们的代码可在此 https URL 上获取

Title: Tool-Assisted Agent on SQL Inspection and Refinement in Real-World Scenarios

• Authors: Zhongyuan Wang, Richong Zhang, Zhijie Nie, Jaein Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.16991
• Pdf URL: https://arxiv.org/pdf/2408.16991
• Copy Paste: [[2408.16991]] Tool-Assisted Agent on SQL Inspection and Refinement in Real-World Scenarios(https://arxiv.org/abs/2408.16991)
• Keywords: language model, llm, agent
• Abstract: Recent Text-to-SQL methods leverage large language models (LLMs) by incorporating feedback from the database management system. While these methods effectively address execution errors in SQL queries, they struggle with database mismatches -- errors that do not trigger execution exceptions. Database mismatches include issues such as condition mismatches and stricter constraint mismatches, both of which are more prevalent in real-world scenarios. To address these challenges, we propose a tool-assisted agent framework for SQL inspection and refinement, equipping the LLM-based agent with two specialized tools: a retriever and a detector, designed to diagnose and correct SQL queries with database mismatches. These tools enhance the capability of LLMs to handle real-world queries more effectively. We also introduce Spider-Mismatch, a new dataset specifically constructed to reflect the condition mismatch problems encountered in real-world scenarios. Experimental results demonstrate that our method achieves the highest performance on the averaged results of the Spider and Spider-Realistic datasets in few-shot settings, and it significantly outperforms baseline methods on the more realistic dataset, Spider-Mismatch.
• 摘要：最近的文本到 SQL 方法通过结合数据库管理系统的反馈来利用大型语言模型 (LLM)。虽然这些方法有效地解决了 SQL 查询中的执行错误，但它们在处理数据库不匹配问题时却举步维艰——这些错误不会触发执行异常。数据库不匹配包括条件不匹配和更严格的约束不匹配等问题，这两种问题在现实场景中都更为普遍。为了应对这些挑战，我们提出了一个用于 SQL 检查和改进的工具辅助代理框架，为基于 LLM 的代理配备了两个专用工具：检索器和检测器，旨在诊断和纠正具有数据库不匹配的 SQL 查询。这些工具增强了 LLM 更有效地处理现实查询的能力。我们还引入了 Spider-Mismatch，这是一个专门构建的新数据集，用于反映现实场景中遇到的条件不匹配问题。实验结果表明，我们的方法在少样本设置中对 Spider 和 Spider-Realistic 数据集的平均结果实现了最高性能，并且在更现实的数据集 Spider-Mismatch 上的表现明显优于基线方法。

Title: Dynamic Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Sampling

• Authors: Guangya Wan, Yuqi Wu, Jie Chen, Sheng Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.17017
• Pdf URL: https://arxiv.org/pdf/2408.17017
• Copy Paste: [[2408.17017]] Dynamic Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Sampling(https://arxiv.org/abs/2408.17017)
• Keywords: language model, llm, hallucination, prompt
• Abstract: Self-Consistency (SC) is a widely used method to mitigate hallucinations in Large Language Models (LLMs) by sampling the LLM multiple times and outputting the most frequent solution. Despite its benefits, SC results in significant computational costs proportional to the number of samples generated. Previous early-stopping approaches, such as Early Stopping Self Consistency and Adaptive Consistency, have aimed to reduce these costs by considering output consistency, but they do not analyze the quality of the reasoning paths (RPs) themselves. To address this issue, we propose Reasoning-Aware Self-Consistency (RASC), an innovative early-stopping framework that dynamically adjusts the number of sample generations by considering both the output answer and the RPs from Chain of Thought (CoT) prompting. RASC assigns confidence scores sequentially to the generated samples, stops when certain criteria are met, and then employs weighted majority voting to optimize sample usage and enhance answer reliability. We comprehensively test RASC with multiple LLMs across varied QA datasets. RASC outperformed existing methods and significantly reduces sample usage by an average of 80% while maintaining or improving accuracy up to 5% compared to the original SC
• 摘要：自一致性 (SC) 是一种广泛使用的方法，通过多次采样 LLM 并输出最常见的解决方案来缓解大型语言模型 (LLM) 中的幻觉。尽管 SC 有很多好处，但它会导致与生成的样本数量成比例的大量计算成本。以前的早期停止方法，例如早期停止自一致性和自适应一致性，旨在通过考虑输出一致性来降低这些成本，但它们不会分析推理路径 (RP) 本身的质量。为了解决这个问题，我们提出了推理感知自一致性 (RASC)，这是一种创新的早期停止框架，它通过考虑输出答案和来自思路链 (CoT) 提示的 RP 来动态调整样本生成的数量。RASC 按顺序为生成的样本分配置信度分数，在满足某些条件时停止，然后采用加权多数投票来优化样本使用并提高答案可靠性。我们在不同的 QA 数据集上使用多个 LLM 全面测试了 RASC。 RASC 的表现优于现有方法，与原始 SC 相比，其样本使用量平均减少了 80%，同时准确度保持或提高了 5%

Title: InkubaLM: A small language model for low-resource African languages

• Authors: Atnafu Lambebo Tonja, Bonaventure F. P. Dossou, Jessica Ojo, Jenalea Rajab, Fadel Thior, Eric Peter Wairagala, Aremu Anuoluwapo, Pelonomi Moiloa, Jade Abbott, Vukosi Marivate, Benjamin Rosman
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.17024
• Pdf URL: https://arxiv.org/pdf/2408.17024
• Copy Paste: [[2408.17024]] InkubaLM: A small language model for low-resource African languages(https://arxiv.org/abs/2408.17024)
• Keywords: language model
• Abstract: High-resource language models often fall short in the African context, where there is a critical need for models that are efficient, accessible, and locally relevant, even amidst significant computing and data constraints. This paper introduces InkubaLM, a small language model with 0.4 billion parameters, which achieves performance comparable to models with significantly larger parameter counts and more extensive training data on tasks such as machine translation, question-answering, AfriMMLU, and the AfriXnli task. Notably, InkubaLM outperforms many larger models in sentiment analysis and demonstrates remarkable consistency across multiple languages. This work represents a pivotal advancement in challenging the conventional paradigm that effective language models must rely on substantial resources. Our model and datasets are publicly available \footnote{\url{this https URL}} to encourage research and development on low-resource languages.
• 摘要：高资源语言模型在非洲环境中往往无法满足需求，因为非洲迫切需要高效、可访问且与当地相关的模型，即使在计算和数据受到严重限制的情况下也是如此。本文介绍了 InkubaLM，这是一个具有 0.4 亿个参数的小型语言模型，其在机器翻译、问答、AfriMMLU 和 AfriXnli 任务等任务上的性能可与具有明显更大参数数量和更广泛训练数据的模型相媲美。值得注意的是，InkubaLM 在情绪分析方面的表现优于许多大型模型，并且在多种语言中表现出了显著的一致性。这项工作代表了挑战有效语言模型必须依赖大量资源的传统范式的关键进步。我们的模型和数据集是公开可用的 \footnote{\url{this https URL}}，以鼓励对低资源语言的研究和开发。

Title: From Text to Emotion: Unveiling the Emotion Annotation Capabilities of LLMs

• Authors: Minxue Niu (1), Mimansa Jaiswal (2), Emily Mower Provost (1) ((1) University of Michigan, (2) Independent Researcher)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.17026
• Pdf URL: https://arxiv.org/pdf/2408.17026
• Copy Paste: [[2408.17026]] From Text to Emotion: Unveiling the Emotion Annotation Capabilities of LLMs(https://arxiv.org/abs/2408.17026)
• Keywords: language model, gpt, llm
• Abstract: Training emotion recognition models has relied heavily on human annotated data, which present diversity, quality, and cost challenges. In this paper, we explore the potential of Large Language Models (LLMs), specifically GPT4, in automating or assisting emotion annotation. We compare GPT4 with supervised models and or humans in three aspects: agreement with human annotations, alignment with human perception, and impact on model training. We find that common metrics that use aggregated human annotations as ground truth can underestimate the performance, of GPT-4 and our human evaluation experiment reveals a consistent preference for GPT-4 annotations over humans across multiple datasets and evaluators. Further, we investigate the impact of using GPT-4 as an annotation filtering process to improve model training. Together, our findings highlight the great potential of LLMs in emotion annotation tasks and underscore the need for refined evaluation methodologies.
• 摘要：训练情绪识别模型在很大程度上依赖于人工注释的数据，这些数据带来了多样性、质量和成本方面的挑战。在本文中，我们探讨了大型语言模型 (LLM)，特别是 GPT4，在自动化或协助情绪注释方面的潜力。我们从三个方面将 GPT4 与监督模型和/或人类进行了比较：与人类注释的一致性、与人类感知的一致性以及对模型训练的影响。我们发现，使用聚合的人类注释作为基本事实的常用指标可能会低估 GPT-4 的性能，而我们的人类评估实验表明，在多个数据集和评估者中，GPT-4 注释比人类的偏好一致。此外，我们研究了使用 GPT-4 作为注释过滤过程对改进模型训练的影响。总之，我们的研究结果凸显了 LLM 在情绪注释任务中的巨大潜力，并强调了改进评估方法的必要性。

Title: Novel-WD: Exploring acquisition of Novel World Knowledge in LLMs Using Prefix-Tuning

• Authors: Maxime Méloux, Christophe Cerisara
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.17070
• Pdf URL: https://arxiv.org/pdf/2408.17070
• Copy Paste: [[2408.17070]] Novel-WD: Exploring acquisition of Novel World Knowledge in LLMs Using Prefix-Tuning(https://arxiv.org/abs/2408.17070)
• Keywords: language model, llm
• Abstract: Teaching new information to pre-trained large language models (PLM) is a crucial but challenging task. Model adaptation techniques, such as fine-tuning and parameter-efficient training have been shown to store new facts at a slow rate; continual learning is an option but is costly and prone to catastrophic forgetting. This work studies and quantifies how PLM may learn and remember new world knowledge facts that do not occur in their pre-training corpus, which only contains world knowledge up to a certain date. To that purpose, we first propose Novel-WD, a new dataset consisting of sentences containing novel facts extracted from recent Wikidata updates, along with two evaluation tasks in the form of causal language modeling and multiple choice questions (MCQ). We make this dataset freely available to the community, and release a procedure to later build new versions of similar datasets with up-to-date information. We also explore the use of prefix-tuning for novel information learning, and analyze how much information can be stored within a given prefix. We show that a single fact can reliably be encoded within a single prefix, and that the prefix capacity increases with its length and with the base model size.
• 摘要：将新信息传授给预先训练的大型语言模型 (PLM) 是一项至关重要但具有挑战性的任务。模型适应技术（例如微调和参数高效训练）已被证明可以以较慢的速度存储新事实；持续学习是一种选择，但成本高昂且容易发生灾难性遗忘。这项工作研究并量化了 PLM 如何学习和记住其预训练语料库中未出现的新世界知识事实，该语料库仅包含截至某一日期的世界知识。为此，我们首先提出了 Novel-WD，这是一个新的数据集，由包含从最近的 Wikidata 更新中提取的新事实的句子组成，以及两个评估任务，形式为因果语言建模和多项选择题 (MCQ)。我们将这个数据集免费提供给社区，并发布一个程序，以便以后使用最新信息构建类似数据集的新版本。我们还探索了前缀调整在新信息学习中的应用，并分析了给定前缀中可以存储多少信息。我们表明，单个事实可以可靠地编码在单个前缀内，并且前缀容量随着其长度和基本模型大小的增加而增加。

Title: MaFeRw: Query Rewriting with Multi-Aspect Feedbacks for Retrieval-Augmented Large Language Models

• Authors: Yujing Wang, Hainan Zhang, Liang Pang, Liang Pang, Hongwei Zheng, Zhiming Zheng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.17072
• Pdf URL: https://arxiv.org/pdf/2408.17072
• Copy Paste: [[2408.17072]] MaFeRw: Query Rewriting with Multi-Aspect Feedbacks for Retrieval-Augmented Large Language Models(https://arxiv.org/abs/2408.17072)
• Keywords: language model
• Abstract: In a real-world RAG system, the current query often involves spoken ellipses and ambiguous references from dialogue contexts, necessitating query rewriting to better describe user's information needs. However, traditional context-based rewriting has minimal enhancement on downstream generation tasks due to the lengthy process from query rewriting to response generation. Some researchers try to utilize reinforcement learning with generation feedback to assist the rewriter, but these sparse rewards provide little guidance in most cases, leading to unstable training and generation results. We find that user's needs are also reflected in the gold document, retrieved documents and ground truth. Therefore, by feeding back these multi-aspect dense rewards to query rewriting, more stable and satisfactory responses can be achieved. In this paper, we propose a novel query rewriting method MaFeRw, which improves RAG performance by integrating multi-aspect feedback from both the retrieval process and generated results. Specifically, we first use manual data to train a T5 model for the rewriter initialization. Next, we design three metrics as reinforcement learning feedback: the similarity between the rewritten query and the gold document, the ranking metrics, and ROUGE between the generation and the ground truth. Inspired by RLAIF, we train three kinds of reward models for the above metrics to achieve more efficient training. Finally, we combine the scores of these reward models as feedback, and use PPO algorithm to explore the optimal query rewriting strategy. Experimental results on two conversational RAG datasets demonstrate that MaFeRw achieves superior generation metrics and more stable training compared to baselines.
• 摘要：在现实世界的 RAG 系统中，当前查询通常涉及对话上下文中的省略号和歧义引用，因此需要重写查询以更好地描述用户的信息需求。然而，传统的基于上下文的重写对下游生成任务的增强很小，因为从查询重写到响应生成的过程很长。一些研究人员尝试利用带有生成反馈的强化学习来协助重写器，但这些稀疏的奖励在大多数情况下提供的指导很少，导致训练和生成结果不稳定。我们发现用户的需求也反映在黄金文档、检索到的文档和基本事实中。因此，通过将这些多方面的密集奖励反馈到查询重写中，可以获得更稳定和令人满意的响应。在本文中，我们提出了一种新颖的查询重写方法 MaFeRw，它通过整合来自检索过程和生成结果的多方面反馈来提高 RAG 性能。具体而言，我们首先使用手动数据训练 T5 模型进行重写器初始化。接下来，我们设计了三个指标作为强化学习反馈：重写查询与黄金文档之间的相似度、排名指标以及生成与基本事实之间的 ROUGE。受 RLAIF 的启发，我们针对上述指标训练了三种奖励模型，以实现更高效的训练。最后，我们将这些奖励模型的得分组合作为反馈，并使用 PPO 算法探索最佳的查询重写策略。在两个对话 RAG 数据集上的实验结果表明，与基线相比，MaFeRw 实现了更出色的生成指标和更稳定的训练。

Title: Improving Extraction of Clinical Event Contextual Properties from Electronic Health Records: A Comparative Study

• Authors: Shubham Agarwal, Thomas Searle, Mart Ratas, Anthony Shek, James Teo, Richard Dobson
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.17181
• Pdf URL: https://arxiv.org/pdf/2408.17181
• Copy Paste: [[2408.17181]] Improving Extraction of Clinical Event Contextual Properties from Electronic Health Records: A Comparative Study(https://arxiv.org/abs/2408.17181)
• Keywords: language model
• Abstract: Electronic Health Records are large repositories of valuable clinical data, with a significant portion stored in unstructured text format. This textual data includes clinical events (e.g., disorders, symptoms, findings, medications and procedures) in context that if extracted accurately at scale can unlock valuable downstream applications such as disease prediction. Using an existing Named Entity Recognition and Linking methodology, MedCAT, these identified concepts need to be further classified (contextualised) for their relevance to the patient, and their temporal and negated status for example, to be useful downstream. This study performs a comparative analysis of various natural language models for medical text classification. Extensive experimentation reveals the effectiveness of transformer-based language models, particularly BERT. When combined with class imbalance mitigation techniques, BERT outperforms Bi-LSTM models by up to 28% and the baseline BERT model by up to 16% for recall of the minority classes. The method has been implemented as part of CogStack/MedCAT framework and made available to the community for further research.
• 摘要：电子健康记录是大量有价值的临床数据存储库，其中很大一部分以非结构化文本格式存储。这些文本数据包括临床事件（例如疾病、症状、发现、药物和程序）及其上下文，如果能够大规模准确提取，则可以解锁有价值的下游应用，例如疾病预测。使用现有的命名实体识别和链接方法 MedCAT，需要进一步对这些已识别的概念进行分类（语境化），以确定它们与患者的相关性以及它们的时间和否定状态，以便在下游使用。本研究对各种用于医学文本分类的自然语言模型进行了比较分析。大量实验揭示了基于变换器的语言模型（尤其是 BERT）的有效性。当与类别不平衡缓解技术相结合时，BERT 在少数类别的召回率方面比 Bi-LSTM 模型高出 28%，比基线 BERT 模型高出 16%。该方法已作为 CogStack/MedCAT 框架的一部分实施，并提供给社区以供进一步研究。

Title: Impact of ChatGPT on the writing style of condensed matter physicists

• Authors: Shaojun Xu, Xiaohui Ye, Mengqi Zhang, Pei Wang
• Subjects: cs.CL, cond-mat.dis-nn, cond-mat.stat-mech
• Abstract URL: https://arxiv.org/abs/2408.17325
• Pdf URL: https://arxiv.org/pdf/2408.17325
• Copy Paste: [[2408.17325]] Impact of ChatGPT on the writing style of condensed matter physicists(https://arxiv.org/abs/2408.17325)
• Keywords: gpt, chat
• Abstract: We apply a state-of-the-art difference-in-differences approach to estimate the impact of ChatGPT's release on the writing style of condensed matter papers on arXiv. Our analysis reveals a statistically significant improvement in the English quality of abstracts written by non-native English speakers. Importantly, this improvement remains robust even after accounting for other potential factors, confirming that it can be attributed to the release of ChatGPT. This indicates widespread adoption of the tool. Following the release of ChatGPT, there is a significant increase in the use of unique words, while the frequency of rare words decreases. Across language families, the changes in writing style are significant for authors from the Latin and Ural-Altaic groups, but not for those from the Germanic or other Indo-European groups.
• 摘要：我们采用最先进的差异法来估计 ChatGPT 发布对 arXiv 上凝聚态论文写作风格的影响。我们的分析显示，非英语母语人士撰写的摘要的英语质量在统计上显著提高。重要的是，即使考虑到其他潜在因素，这种改进仍然强劲，证实了这可以归因于 ChatGPT 的发布。这表明该工具已被广泛采用。ChatGPT 发布后，独特词的使用显著增加，而罕见词的频率则下降。在各个语系中，拉丁语和乌拉尔-阿尔泰语系作者的写作风格变化显著，但日耳曼语系或其他印欧语系作者的写作风格变化并不显著。

Title: Assessing Generative Language Models in Classification Tasks: Performance and Self-Evaluation Capabilities in the Environmental and Climate Change Domain

• Authors: Francesca Grasso, Stefano Locci
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.17362
• Pdf URL: https://arxiv.org/pdf/2408.17362
• Copy Paste: [[2408.17362]] Assessing Generative Language Models in Classification Tasks: Performance and Self-Evaluation Capabilities in the Environmental and Climate Change Domain(https://arxiv.org/abs/2408.17362)
• Keywords: language model, gpt, llm
• Abstract: This paper examines the performance of two Large Language Models (LLMs), GPT3.5 and Llama2 and one Small Language Model (SLM) Gemma, across three different classification tasks within the climate change (CC) and environmental domain. Employing BERT-based models as a baseline, we compare their efficacy against these transformer-based models. Additionally, we assess the models' self-evaluation capabilities by analyzing the calibration of verbalized confidence scores in these text classification tasks. Our findings reveal that while BERT-based models generally outperform both the LLMs and SLM, the performance of the large generative models is still noteworthy. Furthermore, our calibration analysis reveals that although Gemma is well-calibrated in initial tasks, it thereafter produces inconsistent results; Llama is reasonably calibrated, and GPT consistently exhibits strong calibration. Through this research, we aim to contribute to the ongoing discussion on the utility and effectiveness of generative LMs in addressing some of the planet's most urgent issues, highlighting their strengths and limitations in the context of ecology and CC.
• 摘要：本文研究了两个大型语言模型 (LLM) GPT3.5 和 Llama2 以及一个小型语言模型 (SLM) Gemma 在气候变化 (CC) 和环境领域的三个不同分类任务中的表现。以基于 BERT 的模型为基线，我们将其效果与这些基于 Transformer 的模型进行比较。此外，我们通过分析这些文本分类任务中言语化置信度分数的校准来评估模型的自我评估能力。我们的研究结果表明，虽然基于 BERT 的模型通常优于 LLM 和 SLM，但大型生成模型的性能仍然值得关注。此外，我们的校准分析表明，尽管 Gemma 在初始任务中校准良好，但此后产生的结果不一致；Llama 校准合理，GPT 始终表现出强大的校准能力。通过这项研究，我们旨在为有关生成性语言模型在解决地球上一些最紧迫的问题方面的实用性和有效性的持续讨论做出贡献，并强调它们在生态学和 CC 背景下的优势和局限性。

Title: NDP: Next Distribution Prediction as a More Broad Target

• Authors: Junhao Ruan, Abudukeyumu Abudula, Xinyu Liu, Bei Li, Yinqiao Li, Chenglong Wang, Yuchun Fan, Yuan Ge, Tong Xiao, Jingbo Zhu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.17377
• Pdf URL: https://arxiv.org/pdf/2408.17377
• Copy Paste: [[2408.17377]] NDP: Next Distribution Prediction as a More Broad Target(https://arxiv.org/abs/2408.17377)
• Keywords: language model, llm
• Abstract: Large language models (LLMs) trained on next-token prediction (NTP) paradigm have demonstrated powerful capabilities. However, the existing NTP paradigm contains several limitations, particularly related to planned task complications and error propagation during inference. In our work, we extend the critique of NTP, highlighting its limitation also due to training with a narrow objective: the prediction of a sub-optimal one-hot distribution. To support this critique, we conducted a pre-experiment treating the output distribution from powerful LLMs as efficient world data compression. By evaluating the similarity between the $n$-gram distribution and the one-hot distribution with LLMs, we observed that the $n$-gram distributions align more closely with the output distribution of LLMs. Based on this insight, we introduce Next Distribution Prediction (NDP), which uses $n$-gram distributions to replace the one-hot targets, enhancing learning without extra online training time. We conducted experiments across translation, general task, language transfer, and medical domain adaptation. Compared to NTP, NDP can achieve up to +2.97 COMET improvement in translation tasks, +0.61 average improvement in general tasks, and incredible +10.75 average improvement in the medical domain. This demonstrates the concrete benefits of addressing the target narrowing problem, pointing to a new direction for future work on improving NTP.
• 摘要：在下一个标记预测 (NTP) 范式上训练的大型语言模型 (LLM) 已展示出强大的功能。然而，现有的 NTP 范式包含几个限制，特别是与计划任务的复杂性和推理过程中的错误传播有关。在我们的工作中，我们扩展了对 NTP 的批评，强调了它的局限性，这也是由于训练目标狭窄：预测次优的独热分布。为了支持这一批评，我们进行了一项预实验，将强大的 LLM 的输出分布视为高效的世界数据压缩。通过评估 LLM 的 $n$-gram 分布和独热分布之间的相似性，我们观察到 $n$-gram 分布与 LLM 的输出分布更接近。基于这一见解，我们引入了下一个分布预测 (NDP)，它使用 $n$-gram 分布来取代独热目标，从而无需额外的在线训练时间即可增强学习。我们在翻译、一般任务、语言迁移和医学领域适应方面进行了实验。与 NTP 相比，NDP 在翻译任务上可实现高达 +2.97 的 COMET 改进，在一般任务上可实现 +0.61 的平均改进，在医学领域可实现令人难以置信的 +10.75 的平均改进。这展示了解决目标缩小问题的具体好处，为未来改进 NTP 的工作指明了新的方向。

Title: CLOCR-C: Context Leveraging OCR Correction with Pre-trained Language Models

• Authors: Jonathan Bourne
• Subjects: cs.CL, cs.DL
• Abstract URL: https://arxiv.org/abs/2408.17428
• Pdf URL: https://arxiv.org/pdf/2408.17428
• Copy Paste: [[2408.17428]] CLOCR-C: Context Leveraging OCR Correction with Pre-trained Language Models(https://arxiv.org/abs/2408.17428)
• Keywords: language model, prompt
• Abstract: The digitisation of historical print media archives is crucial for increasing accessibility to contemporary records. However, the process of Optical Character Recognition (OCR) used to convert physical records to digital text is prone to errors, particularly in the case of newspapers and periodicals due to their complex layouts. This paper introduces Context Leveraging OCR Correction (CLOCR-C), which utilises the infilling and context-adaptive abilities of transformer-based language models (LMs) to improve OCR quality. The study aims to determine if LMs can perform post-OCR correction, improve downstream NLP tasks, and the value of providing the socio-cultural context as part of the correction process. Experiments were conducted using seven LMs on three datasets: the 19th Century Serials Edition (NCSE) and two datasets from the Overproof collection. The results demonstrate that some LMs can significantly reduce error rates, with the top-performing model achieving over a 60% reduction in character error rate on the NCSE dataset. The OCR improvements extend to downstream tasks, such as Named Entity Recognition, with increased Cosine Named Entity Similarity. Furthermore, the study shows that providing socio-cultural context in the prompts improves performance, while misleading prompts lower performance. In addition to the findings, this study releases a dataset of 91 transcribed articles from the NCSE, containing a total of 40 thousand words, to support further research in this area. The findings suggest that CLOCR-C is a promising approach for enhancing the quality of existing digital archives by leveraging the socio-cultural information embedded in the LMs and the text requiring correction.
• 摘要：历史印刷媒体档案的数字化对于提高当代记录的可访问性至关重要。然而，用于将物理记录转换为数字文本的光学字符识别 (OCR) 过程容易出错，尤其是报纸和期刊，因为它们的布局复杂。本文介绍了上下文利用 OCR 校正 (CLOCR-C)，它利用基于转换器的语言模型 (LM) 的填充和上下文自适应能力来提高 OCR 质量。该研究旨在确定 LM 是否可以执行 OCR 后校正、改进下游 NLP 任务，以及在校正过程中提供社会文化背景的价值。使用七个 LM 在三个数据集上进行了实验：19 世纪连续出版物版本 (NCSE) 和两个来自 Overproof 集合的数据集。结果表明，一些 LM 可以显著降低错误率，其中表现最佳的模型在 NCSE 数据集上的字符错误率降低了 60% 以上。 OCR 改进扩展到下游任务，例如命名实体识别，余弦命名实体相似度增加。此外，研究表明，在提示中提供社会文化背景可以提高性能，而误导性提示会降低性能。除了研究结果之外，这项研究还发布了 NCSE 的 91 篇转录文章数据集，共包含 4 万个单词，以支持该领域的进一步研究。研究结果表明，CLOCR-C 是一种很有前途的方法，可以通过利用嵌入在 LM 和需要更正的文本中的社会文化信息来提高现有数字档案的质量。

Title: SYNTHEVAL: Hybrid Behavioral Testing of NLP Models with Synthetic CheckLists

• Authors: Raoyuan Zhao, Abdullatif Köksal, Yihong Liu, Leonie Weissweiler, Anna Korhonen, Hinrich Schütze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.17437
• Pdf URL: https://arxiv.org/pdf/2408.17437
• Copy Paste: [[2408.17437]] SYNTHEVAL: Hybrid Behavioral Testing of NLP Models with Synthetic CheckLists(https://arxiv.org/abs/2408.17437)
• Keywords: language model, llm
• Abstract: Traditional benchmarking in NLP typically involves using static held-out test sets. However, this approach often results in an overestimation of performance and lacks the ability to offer comprehensive, interpretable, and dynamic assessments of NLP models. Recently, works like DynaBench (Kiela et al., 2021) and CheckList (Ribeiro et al., 2020) have addressed these limitations through behavioral testing of NLP models with test types generated by a multistep human-annotated pipeline. Unfortunately, manually creating a variety of test types requires much human labor, often at prohibitive cost. In this work, we propose SYNTHEVAL, a hybrid behavioral testing framework that leverages large language models (LLMs) to generate a wide range of test types for a comprehensive evaluation of NLP models. SYNTHEVAL first generates sentences via LLMs using controlled generation, and then identifies challenging examples by comparing the predictions made by LLMs with task-specific NLP models. In the last stage, human experts investigate the challenging examples, manually design templates, and identify the types of failures the taskspecific models consistently exhibit. We apply SYNTHEVAL to two classification tasks, sentiment analysis and toxic language detection, and show that our framework is effective in identifying weaknesses of strong models on these tasks. We share our code in this https URL.
• 摘要：传统的 NLP 基准测试通常涉及使用静态保留测试集。然而，这种方法往往会导致对性能的估计过高，并且无法提供对 NLP 模型的全面、可解释和动态评估。最近，DynaBench（Kiela 等人，2021 年）和 CheckList（Ribeiro 等人，2020 年）等研究通过使用由多步人工注释管道生成的测试类型对 NLP 模型进行行为测试来解决这些限制。不幸的是，手动创建各种测试类型需要大量人力，而且成本通常过高。在这项工作中，我们提出了 SYNTHEVAL，这是一个混合行为测试框架，它利用大型语言模型 (LLM) 生成各种测试类型，以全面评估 NLP 模型。SYNTHEVAL 首先使用受控生成通过 LLM 生成句子，然后通过将 LLM 的预测与特定于任务的 NLP 模型进行比较来识别具有挑战性的示例。在最后阶段，人类专家调查具有挑战性的示例，手动设计模板，并确定特定于任务的模型始终表现出的故障类型。我们将 SYNTHEVAL 应用于两个分类任务，即情绪分析和有毒语言检测，并表明我们的框架能够有效地识别这些任务上强大模型的弱点。我们在此 https URL 中分享了我们的代码。