2024-08-13

Title: Large Model Strategic Thinking, Small Model Efficiency: Transferring Theory of Mind in Large Language Models

• Authors: Nunzio Lore, Alireza (Sepehr)Ilami, Babak Heydari
• Subjects: cs.CL, cs.AI, cs.CY, cs.ET, cs.GT
• Abstract URL: https://arxiv.org/abs/2408.05241
• Pdf URL: https://arxiv.org/pdf/2408.05241
• Copy Paste: [[2408.05241]] Large Model Strategic Thinking, Small Model Efficiency: Transferring Theory of Mind in Large Language Models(https://arxiv.org/abs/2408.05241)
• Keywords: language model, agent
• Abstract: As the performance of larger, newer Large Language Models continues to improve for strategic Theory of Mind (ToM) tasks, the demand for these state of the art models increases commensurately. However, their deployment is costly both in terms of processing power and time. In this paper, we investigate the feasibility of creating smaller, simulation-ready agents by way of fine-tuning. To do this, we present a large pre-trained model with 20 unique scenarios that combine a social context with a social dilemma, recording its answers, and using them for Q\&A fine-tuning on a smaller model of the same family. Our focus is on in-context game-theoretic decision-making, the same domain within which human interaction occurs and that requires both a theory of mind (or a semblance thereof) and an understanding of social dynamics. We find that the fine-tuned smaller language model exhibited significant performance closer to that of its larger relative, and that their improvements extended in areas and contexts beyond the ones provided in the training examples. On average for all games, through fine-tuning, the smaller model showed a \%46 improvement in aligning with the behavior of the larger model, with \%100 representing complete alignment. This suggests that our pipeline represents an efficient method to transmit some form of theory of mind to smaller models, creating improved and cheaply deployable algorithms in the process. Despite their simplicity and their associated shortcomings and limitations, our findings represent a stepping stone in the pursuit and training of specialized models for strategic and social decision making.
• 摘要：随着更大、更新的大型语言模型在战略性心智理论 (ToM) 任务中的性能不断提高，对这些先进模型的需求也相应增加。然而，它们的部署在处理能力和时间方面都很昂贵。在本文中，我们研究了通过微调创建更小、可模拟的代理的可行性。为此，我们提出了一个大型预训练模型，该模型具有 20 个独特的场景，将社交背景与社交困境相结合，记录其答案，并将它们用于同一家族的较小模型上的问答微调。我们的重点是情境博弈论决策，即人类互动发生的同一领域，它既需要心智理论（或类似理论），也需要对社会动态的理解。我们发现，经过微调的较小语言模型表现出与其较大相关模型更接近的显著性能，并且它们的改进扩展到训练示例中提供的领域和情境之外。在所有游戏中，通过微调，较小的模型在与较大模型的行为保持一致方面平均提高了 46%，100% 代表完全一致。这表明我们的流程代表了一种将某种形式的心理理论传输到较小模型的有效方法，从而在此过程中创建了改进的、可廉价部署的算法。尽管这些算法很简单，并且存在相关的缺点和局限性，但我们的研究结果代表了寻求和训练用于战略和社会决策的专门模型的垫脚石。

Title: A Psychology-based Unified Dynamic Framework for Curriculum Learning

• Authors: Guangyu Meng, Qingkai Zeng, John P. Lalor, Hong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.05326
• Pdf URL: https://arxiv.org/pdf/2408.05326
• Copy Paste: [[2408.05326]] A Psychology-based Unified Dynamic Framework for Curriculum Learning(https://arxiv.org/abs/2408.05326)
• Keywords: language model
• Abstract: Directly learning from examples of random difficulty levels is often challenging for both humans and machine learning models. A more effective strategy involves exposing learners to examples in a progressive order, from easy to difficult. Curriculum Learning (CL) has been proposed to implement this strategy in machine learning model training. However, two key challenges persist in CL framework design: defining the difficulty of training data and determining the appropriate amount of data to input at each training step. This paper presents a Psychology-based Unified Dynamic Framework for Curriculum Learning (PUDF), drawing inspiration from psychometrics. We quantify the difficulty of training data by applying Item Response Theory (IRT) to responses from Artificial Crowds (AC). This theory-driven IRT-AC approach leads to global (i.e., model-independent) and interpretable difficulty values. Leveraging IRT, we propose a Dynamic Data Selection via Model Ability Estimation (DDS-MAE) strategy to schedule the appropriate amount of data during model training. Since our difficulty labeling and model ability estimation are based on a consistent theory, namely IRT, their values are comparable within the same scope, potentially leading to a faster convergence compared to the other CL methods. Experimental results demonstrate that fine-tuning pre-trained language models with PUDF enhances their performance on the GLUE benchmark. Moreover, PUDF surpasses other state-of-the-art (SOTA) CL methods on the GLUE benchmark. We further explore the components of PUDF, namely the difficulty measurer (IRT-AC) and the training scheduler (DDS-MAE) qualitatively and quantitatively. Lastly, we conduct an ablation study to clarify which components of PUDF contribute to faster convergence and higher accuracy.
• 摘要：直接从随机难度级别的示例中学习通常对人类和机器学习模型都具有挑战性。更有效的策略是让学习者按照从易到难的渐进顺序接触示例。课程学习 (CL) 已被提出用于在机器学习模型训练中实现此策略。然而，CL 框架设计中仍然存在两个关键挑战：定义训练数据的难度并确定在每个训练步骤中输入的适当数据量。本文提出了一种基于心理学的课程学习统一动态框架 (PUDF)，灵感来自心理测量学。我们通过将项目反应理论 (IRT) 应用于来自人工人群 (AC) 的反应来量化训练数据的难度。这种理论驱动的 IRT-AC 方法可产生全局（即独立于模型）和可解释的难度值。利用 IRT，我们提出了一种通过模型能力估计进行动态数据选择 (DDS-MAE) 策略来在模型训练期间安排适当量的数据。由于我们的难度标记和模型能力估计基于一致的理论，即 IRT，因此它们的值在同一范围内是可比较的，与其他 CL 方法相比，可能带来更快的收敛速度。实验结果表明，使用 PUDF 对预训练语言模型进行微调可提高其在 GLUE 基准上的性能。此外，PUDF 在 GLUE 基准上超越了其他最先进的 (SOTA) CL 方法。我们进一步定性和定量探索了 PUDF 的组成部分，即难度测量器 (IRT-AC) 和训练调度器 (DDS-MAE)。最后，我们进行了一项消融研究，以阐明 PUDF 的哪些组件有助于加快收敛速度​​和提高准确性。

Title: From Text to Insight: Leveraging Large Language Models for Performance Evaluation in Management

• Authors: Ning Li, Huaikang Zhou, Mingze Xu
• Subjects: cs.CL, cs.AI, cs.ET, cs.HC, econ.GN
• Abstract URL: https://arxiv.org/abs/2408.05328
• Pdf URL: https://arxiv.org/pdf/2408.05328
• Copy Paste: [[2408.05328]] From Text to Insight: Leveraging Large Language Models for Performance Evaluation in Management(https://arxiv.org/abs/2408.05328)
• Keywords: language model, gpt, llm
• Abstract: This study explores the potential of Large Language Models (LLMs), specifically GPT-4, to enhance objectivity in organizational task performance evaluations. Through comparative analyses across two studies, including various task performance outputs, we demonstrate that LLMs can serve as a reliable and even superior alternative to human raters in evaluating knowledge-based performance outputs, which are a key contribution of knowledge workers. Our results suggest that GPT ratings are comparable to human ratings but exhibit higher consistency and reliability. Additionally, combined multiple GPT ratings on the same performance output show strong correlations with aggregated human performance ratings, akin to the consensus principle observed in performance evaluation literature. However, we also find that LLMs are prone to contextual biases, such as the halo effect, mirroring human evaluative biases. Our research suggests that while LLMs are capable of extracting meaningful constructs from text-based data, their scope is currently limited to specific forms of performance evaluation. By highlighting both the potential and limitations of LLMs, our study contributes to the discourse on AI role in management studies and sets a foundation for future research to refine AI theoretical and practical applications in management.
• 摘要：本研究探讨了大型语言模型 (LLM)，特别是 GPT-4，在提高组织任务绩效评估的客观性方面的潜力。通过对两项研究（包括各种任务绩效输出）的比较分析，我们证明 LLM 可以作为人类评估者的可靠甚至更优越的替代方案，用于评估知识型绩效输出，这是知识型员工的一项关键贡献。我们的结果表明，GPT 评分与人类评分相当，但表现出更高的一致性和可靠性。此外，对同一绩效输出的多个 GPT 评分与汇总的人类绩效评分显示出很强的相关性，类似于绩效评估文献中观察到的共识原则。然而，我们还发现 LLM 容易受到情境偏见的影响，例如光环效应，反映了人类的评价偏见。我们的研究表明，虽然 LLM 能够从基于文本的数据中提取有意义的结构，但它们的范围目前仅限于特定形式的绩效评估。通过强调法学硕士的潜力和局限性，我们的研究为人工智能在管理研究中的作用的讨论做出了贡献，并为未来完善人工智能在管理中的理论和实际应用的研究奠定了基础。

Title: DataNarrative: Automated Data-Driven Storytelling with Visualizations and Texts

• Authors: Mohammed Saidul Islam, Enamul Hoque, Shafiq Joty, Md Tahmid Rahman Laskar, Md Rizwan Parvez
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.05346
• Pdf URL: https://arxiv.org/pdf/2408.05346
• Copy Paste: [[2408.05346]] DataNarrative: Automated Data-Driven Storytelling with Visualizations and Texts(https://arxiv.org/abs/2408.05346)
• Keywords: language model, llm, agent
• Abstract: Data-driven storytelling is a powerful method for conveying insights by combining narrative techniques with visualizations and text. These stories integrate visual aids, such as highlighted bars and lines in charts, along with textual annotations explaining insights. However, creating such stories requires a deep understanding of the data and meticulous narrative planning, often necessitating human intervention, which can be time-consuming and mentally taxing. While Large Language Models (LLMs) excel in various NLP tasks, their ability to generate coherent and comprehensive data stories remains underexplored. In this work, we introduce a novel task for data story generation and a benchmark containing 1,449 stories from diverse sources. To address the challenges of crafting coherent data stories, we propose a multiagent framework employing two LLM agents designed to replicate the human storytelling process: one for understanding and describing the data (Reflection), generating the outline, and narration, and another for verification at each intermediary step. While our agentic framework generally outperforms non-agentic counterparts in both model-based and human evaluations, the results also reveal unique challenges in data story generation.
• 摘要：数据驱动的故事讲述是一种将叙事技巧与可视化和文本相结合来传达见解的有效方法。这些故事整合了视觉辅助工具，例如图表中突出显示的条形图和线条，以及解释见解的文本注释。然而，创建这样的故事需要对数据有深入的理解和细致的叙事规划，通常需要人工干预，这可能既耗时又费脑力。虽然大型语言模型 (LLM) 在各种 NLP 任务中表现出色，但它们生成连贯而全面的数据故事的能力仍未得到充分探索。在这项工作中，我们引入了一个用于数据故事生成的新任务和一个包含来自不同来源的 1,449 个故事的基准。为了应对编写连贯数据故事的挑战，我们提出了一个多智能体框架，该框架采用两个 LLM 智能体，旨在复制人类的故事讲述过程：一个用于理解和描述数据（反思）、生成大纲和叙述，另一个用于在每个中间步骤进行验证。虽然我们的代理框架在基于模型和人工的评估中通常优于非代理框架，但结果也揭示了数据故事生成中的独特挑战。

Title: FiST-Financial Style Transfer with Hallucination and Creativity Control Framework

• Authors: Sohini Roychowdhury, Marko Krema, Brian Moore, Xingjian Lai, Dike Effedua, Bharat Jethwani
• Subjects: cs.CL, cs.AI, cs.CE
• Abstract URL: https://arxiv.org/abs/2408.05365
• Pdf URL: https://arxiv.org/pdf/2408.05365
• Copy Paste: [[2408.05365]] FiST-Financial Style Transfer with Hallucination and Creativity Control Framework(https://arxiv.org/abs/2408.05365)
• Keywords: language model, llm, hallucination, prompt, retrieval augmented generation
• Abstract: Financial report generation using general purpose large language models pose two major challenges, including the lack of compound sentences and hallucinations. Advanced prompt engineering and retrieval augmented generation (RAG) techniques are incapable of curing the writing style discrepancies. In this work we propose a novel two-stage fine-tuning process wherein public domain financial reports are processed into prompt-completions and augmented using simple LLM prompts to then enable sectional financial report generation using minimal instructions and tabular data inputs. Our proposed fine-tuning framework results doubles the number of correct questions answers and reduces hallucinations by over 50%. Additionally, the two-stage fine tuned models have lower perplexity, improved ROUGE, TER and BLEU scores, higher creativity and knowledge density with lower uncertainty and cross entropy.
• 摘要：使用通用大型语言模型生成财务报告面临两大挑战，包括缺乏复合句和幻觉。先进的提示工程和检索增强生成 (RAG) 技术无法解决写作风格差异。在这项工作中，我们提出了一种新颖的两阶段微调流程，其中将公共领域财务报告处理为提示完成，并使用简单的 LLM 提示进行增强，然后使用最少的指令和表格数据输入实现分段财务报告生成。我们提出的微调框架使正确问题答案的数量增加了一倍，并将幻觉减少了 50% 以上。此外，两阶段微调模型的困惑度更低，ROUGE、TER 和 BLEU 分数更高，创造力和知识密度更高，不确定性和交叉熵更低。

Title: LaiDA: Linguistics-aware In-context Learning with Data Augmentation for Metaphor Components Identification

• Authors: Hongde Liu, Chenyuan He, Feiyang Meng, Changyong Niu, Yuxiang Jia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.05404
• Pdf URL: https://arxiv.org/pdf/2408.05404
• Copy Paste: [[2408.05404]] LaiDA: Linguistics-aware In-context Learning with Data Augmentation for Metaphor Components Identification(https://arxiv.org/abs/2408.05404)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: Metaphor Components Identification (MCI) contributes to enhancing machine understanding of metaphors, thereby advancing downstream natural language processing tasks. However, the complexity, diversity, and dependency on context and background knowledge pose significant challenges for MCI. Large language models (LLMs) offer new avenues for accurate comprehension of complex natural language texts due to their strong semantic analysis and extensive commonsense knowledge. In this research, a new LLM-based framework is proposed, named Linguistics-aware In-context Learning with Data Augmentation (LaiDA). Specifically, ChatGPT and supervised fine-tuning are utilized to tailor a high-quality dataset. LaiDA incorporates a simile dataset for pre-training. A graph attention network encoder generates linguistically rich feature representations to retrieve similar examples. Subsequently, LLM is fine-tuned with prompts that integrate linguistically similar examples. LaiDA ranked 2nd in Subtask 2 of NLPCC2024 Shared Task 9, demonstrating its effectiveness. Code and data are available at this https URL.
• 摘要：隐喻成分识别 (MCI) 有助于增强机器对隐喻的理解，从而推动下游自然语言处理任务的发展。然而，隐喻的复杂性、多样性以及对上下文和背景知识的依赖对 MCI 提出了重大挑战。大型语言模型 (LLM) 凭借其强大的语义分析能力和广泛的常识性知识，为准确理解复杂的自然语言文本提供了新的途径。在本研究中，提出了一种基于 LLM 的新框架，即基于数据增强的语言感知上下文学习 (LaiDA)。具体而言，使用 ChatGPT 和监督微调来定制高质量的数据集。LaiDA 结合了明喻数据集进行预训练。图注意网络编码器生成语言丰富的特征表示以检索类似的示例。随后，使用集成语言相似示例的提示对 LLM 进行微调。LaiDA 在 NLPCC2024 共享任务 9 的子任务 2 中排名第二，证明了其有效性。代码和数据可在此 https URL 上获取。

Title: Chain of Condition: Construct, Verify and Solve Conditions for Conditional Question Answering

• Authors: Jiuheng Lin, Yuxuan Lai, Yansong Feng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.05442
• Pdf URL: https://arxiv.org/pdf/2408.05442
• Copy Paste: [[2408.05442]] Chain of Condition: Construct, Verify and Solve Conditions for Conditional Question Answering(https://arxiv.org/abs/2408.05442)
• Keywords: gpt, prompt
• Abstract: Conditional question answering (CQA) is an important task that aims to find probable answers and identify conditions that need to be satisfied to support the answer. Existing approaches struggle with CQA due to two main challenges: (1) precisely identifying conditions and their logical relationship, and (2) verifying and solving the conditions. To address these challenges, we propose Chain of Condition, a novel prompting approach by firstly identifying all conditions and constructing their logical relationships explicitly according to the document, then verifying whether these conditions are satisfied, finally solving the logical expression by tools to indicate any missing conditions and generating the answer based on the resolved conditions. The experiments on two benchmark conditional question answering datasets shows chain of condition outperforms existing prompting baselines, establishing a new state-of-the-art. Furthermore, with backbone models like GPT-3.5-Turbo or GPT-4, it surpasses all supervised baselines with only few-shot settings.
• 摘要：条件问答 (CQA) 是一项重要任务，旨在找到可能的答案并确定支持答案需要满足的条件。现有方法在 CQA 方面遇到困难，主要有两个挑战：(1) 准确识别条件及其逻辑关系，以及 (2) 验证和解决条件。为了应对这些挑战，我们提出了条件链，这是一种新颖的提示方法，首先根据文档明确识别所有条件并构建它们的逻辑关系，然后验证这些条件是否满足，最后使用工具解决逻辑表达式以指出任何缺失的条件并根据已解决的条件生成答案。在两个基准条件问答数据集上的实验表明，条件链的表现优于现有的提示基线，建立了新的最先进水平。此外，借助 GPT-3.5-Turbo 或 GPT-4 等骨干模型，它仅通过少量设置就超越了所有监督基线。

Title: Path-LLM: A Shortest-Path-based LLM Learning for Unified Graph Representation

• Authors: Wenbo Shang, Xuliang Zhu, Xin Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.05456
• Pdf URL: https://arxiv.org/pdf/2408.05456
• Copy Paste: [[2408.05456]] Path-LLM: A Shortest-Path-based LLM Learning for Unified Graph Representation(https://arxiv.org/abs/2408.05456)
• Keywords: language model, llm
• Abstract: Unified graph representation learning aims to produce node embeddings, which can be applied to multiple downstream applications. However, existing studies based on graph neural networks and language models either suffer from the limitations of numerous training needed toward specific downstream predictions or have shallow semantic features. In this work, we propose a novel Path-LLM model to learn unified graph representation, which leverages a powerful large language model (LLM) to incorporate our proposed path features. Our Path-LLM framework consists of several well-designed techniques. First, we develop a new mechanism of long-to-short shortest path (L2SP) selection, which covers essential connections between different dense groups. An in-depth comparison of different path selection plans is offered to illustrate the strength of our designed L2SP. Then, we design path textualization to obtain L2SP-based training texts. Next, we feed the texts into a self-supervised LLM training process to learn embeddings. Extensive experiments on benchmarks validate the superiority of Path-LLM against the state-of-the-art WalkLM method on two classical graph learning tasks (node classification and link prediction) and one NP-hard graph query processing task (keyword search), meanwhile saving more than 90% of training paths.
• 摘要：统一图表示学习旨在生成节点嵌入，可应用于多个下游应用程序。然而，现有的基于图神经网络和语言模型的研究要么受到需要大量训练才能实现特定下游预测的限制，要么语义特征较浅。在这项工作中，我们提出了一种新颖的 Path-LLM 模型来学习统一图表示，该模型利用强大的大型语言模型 (LLM) 来整合我们提出的路径特征。我们的 Path-LLM 框架由几种精心设计的技术组成。首先，我们开发了一种新的长到短最短路径 (L2SP) 选择机制，该机制涵盖了不同密集组之间的基本连接。对不同的路径选择方案进行了深入比较，以说明我们设计的 L2SP 的优势。然后，我们设计路径文本化以获得基于 L2SP 的训练文本。接下来，我们将文本输入自监督的 LLM 训练过程以学习嵌入。大量基准测试验证了 Path-LLM 在两个经典图学习任务（节点分类和链接预测）和一个 NP-hard 图查询处理任务（关键字搜索）上相对于最先进的 WalkLM 方法的优越性，同时节省了 90% 以上的训练路径。

Title: Investigating Instruction Tuning Large Language Models on Graphs

• Authors: Kerui Zhu, Bo-Wei Huang, Bowen Jin, Yizhu Jiao, Ming Zhong, Kevin Chang, Shou-De Lin, Jiawei Han
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.05457
• Pdf URL: https://arxiv.org/pdf/2408.05457
• Copy Paste: [[2408.05457]] Investigating Instruction Tuning Large Language Models on Graphs(https://arxiv.org/abs/2408.05457)
• Keywords: language model, llm
• Abstract: Inspired by the recent advancements of Large Language Models (LLMs) in NLP tasks, there's growing interest in applying LLMs to graph-related tasks. This study delves into the capabilities of instruction-following LLMs for engaging with real-world graphs, aiming to offer empirical insights into how LLMs can effectively interact with graphs and generalize across graph tasks. We begin by constructing a dataset designed for instruction tuning, which comprises a diverse collection of 79 graph-related tasks from academic and e-commerce domains, featuring 44,240 training instances and 18,960 test samples. Utilizing this benchmark, our initial investigation focuses on identifying the optimal graph representation that serves as a conduit for LLMs to understand complex graph structures. Our findings indicate that JSON format for graph representation consistently outperforms natural language and code formats across various LLMs and graph types. Furthermore, we examine the key factors that influence the generalization abilities of instruction-tuned LLMs by evaluating their performance on both in-domain and out-of-domain graph tasks.
• 摘要：受大型语言模型 (LLM) 在 NLP 任务中的最新进展的启发，人们对将 LLM 应用于图相关任务的兴趣日益浓厚。本研究深入研究了遵循指令的 LLM 与现实世界图交互的能力，旨在提供关于 LLM 如何有效地与图交互并在图任务中推广的经验见解。我们首先构建一个专为指令调整而设计的数据集，该数据集包含来自学术和电子商务领域的 79 个图相关任务的多样化集合，具有 44,240 个训练实例和 18,960 个测试样本。利用这个基准，我们的初步研究重点是确定最佳图表示，作为 LLM 理解复杂图结构的渠道。我们的研究结果表明，在各种 LLM 和图类型中，用于图表示的 JSON 格式始终优于自然语言和代码格式。此外，我们通过评估指令调整的 LLM 在域内和域外图形任务上的性能来研究影响其泛化能力的关键因素。

Title: Your Context Is Not an Array: Unveiling Random Access Limitations in Transformers

• Authors: MohammadReza Ebrahimi, Sunny Panchal, Roland Memisevic
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.05506
• Pdf URL: https://arxiv.org/pdf/2408.05506
• Copy Paste: [[2408.05506]] Your Context Is Not an Array: Unveiling Random Access Limitations in Transformers(https://arxiv.org/abs/2408.05506)
• Keywords: language model
• Abstract: Despite their recent successes, Transformer-based large language models show surprising failure modes. A well-known example of such failure modes is their inability to length-generalize: solving problem instances at inference time that are longer than those seen during training. In this work, we further explore the root cause of this failure by performing a detailed analysis of model behaviors on the simple parity task. Our analysis suggests that length generalization failures are intricately related to a model's inability to perform random memory accesses within its context window. We present supporting evidence for this hypothesis by demonstrating the effectiveness of methodologies that circumvent the need for indexing or that enable random token access indirectly, through content-based addressing. We further show where and how the failure to perform random memory access manifests through attention map visualizations.
• 摘要：尽管最近取得了成功，但基于 Transformer 的大型语言模型却表现出令人惊讶的失败模式。这种失败模式的一个众所周知的例子是它们无法进行长度泛化：在推理时解决的问题实例比训练期间遇到的问题实例更长。在这项工作中，我们通过对简单奇偶校验任务上的模型行为进行详细分析，进一步探索了这种失败的根本原因。我们的分析表明，长度泛化失败与模型无法在其上下文窗口内执行随机内存访问密切相关。我们通过展示避免索引需要或通过基于内容的寻址间接实现随机令牌访问的方法的有效性，为这一假设提供支持证据。我们进一步通过注意力图可视化展示了执行随机内存访问失败的位置和方式。

Title: SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning

• Authors: Yuze Zhao, Jintao Huang, Jinghan Hu, Daoze Zhang, Zeyinzi Jiang, Zhikai Wu, Baole Ai, Ang Wang, Wenmeng Zhou, Yingda Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.05517
• Pdf URL: https://arxiv.org/pdf/2408.05517
• Copy Paste: [[2408.05517]] SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning(https://arxiv.org/abs/2408.05517)
• Keywords: language model, llm, hallucination, agent
• Abstract: Recent development in Large Language Models (LLMs) and Multi-modal Large Language Models (MLLMs) have leverage Attention-based Transformer architectures and achieved superior performance and generalization capabilities. They have since covered extensive areas of traditional learning tasks. For instance, text-based tasks such as text-classification and sequence-labeling, as well as multi-modal tasks like Visual Question Answering (VQA) and Optical Character Recognition (OCR), which were previously addressed using different models, can now be tackled based on one foundation model. Consequently, the training and lightweight fine-tuning of LLMs and MLLMs, especially those based on Transformer architecture, has become particularly important. In recognition of these overwhelming needs, we develop SWIFT, a customizable one-stop infrastructure for large models. With support of over $300+$ LLMs and $50+$ MLLMs, SWIFT stands as the open-source framework that provide the \textit{most comprehensive support} for fine-tuning large models. In particular, it is the first training framework that provides systematic support for MLLMs. In addition to the core functionalities of fine-tuning, SWIFT also integrates post-training processes such as inference, evaluation, and model quantization, to facilitate fast adoptions of large models in various application scenarios. With a systematic integration of various training techniques, SWIFT offers helpful utilities such as benchmark comparisons among different training techniques for large models. For fine-tuning models specialized in agent framework, we show that notable improvements on the ToolBench leader-board can be achieved by training with customized dataset on SWIFT, with an increase of 5.2\%-21.8\% in the Act.EM metric over various baseline models, a reduction in hallucination by 1.6\%-14.1\%, and an average performance improvement of 8\%-17\%.
• 摘要：大型语言模型 (LLM) 和多模态大型语言模型 (MLLM) 的最新发展利用了基于注意力机制的 Transformer 架构，并实现了卓越的性能和泛化能力。它们已经覆盖了传统学习任务的广泛领域。例如，基于文本的任务（如文本分类和序列标记）以及多模态任务（如视觉问答 (VQA) 和光学字符识别 (OCR)），以前使用不同的模型来解决，现在可以基于一个基础模型来解决。因此，LLM 和 MLLM（尤其是基于 Transformer 架构的 LLM 和 MLLM）的训练和轻量级微调变得尤为重要。为了满足这些迫切的需求，我们开发了 SWIFT，这是一种可定制的大型模型一站式基础设施。凭借对超过 300+ 美元的 LLM 和 50+ 美元的 MLLM 的支持，SWIFT 是提供 \textit{最全面支持} 大型模型微调的开源框架。特别是，它是第一个为 MLLM 提供系统支持的训练框架。除了微调的核心功能之外，SWIFT 还集成了推理、评估和模型量化等训练后流程，以便于在各种应用场景中快速采用大型模型。通过系统地整合各种训练技术，SWIFT 提供了有用的实用程序，例如对大型模型的不同训练技术之间的基准比较。对于专门用于代理框架的微调模型，我们表明，通过在 SWIFT 上使用定制数据集进行训练可以显著提高 ToolBench 排行榜上的排名，与各种基线模型相比，Act.EM 指标提高了 5.2\%-21.8\%，幻觉减少了 1.6\%-14.1\%，平均性能提高了 8\%-17\%。

Title: Context-Driven Index Trimming: A Data Quality Perspective to Enhancing Precision of RALMs

• Authors: Kexin Ma, Ruochun Jin, Xi Wang, Huan Chen, Jing Ren, Yuhua Tang
• Subjects: cs.CL, cs.DB
• Abstract URL: https://arxiv.org/abs/2408.05524
• Pdf URL: https://arxiv.org/pdf/2408.05524
• Copy Paste: [[2408.05524]] Context-Driven Index Trimming: A Data Quality Perspective to Enhancing Precision of RALMs(https://arxiv.org/abs/2408.05524)
• Keywords: language model, llm
• Abstract: Retrieval-Augmented Large Language Models (RALMs) have made significant strides in enhancing the accuracy of generated responses.However, existing research often overlooks the data quality issues within retrieval results, often caused by inaccurate existing vector-distance-based retrieval methods.We propose to boost the precision of RALMs' answers from a data quality perspective through the Context-Driven Index Trimming (CDIT) framework, where Context Matching Dependencies (CMDs) are employed as logical data quality rules to capture and regulate the consistency between retrieved contexts.Based on the semantic comprehension capabilities of Large Language Models (LLMs), CDIT can effectively identify and discard retrieval results that are inconsistent with the query context and further modify indexes in the database, thereby improving answer quality.Experiments demonstrate on challenging question-answering tasks.Also, the flexibility of CDIT is verified through its compatibility with various language models and indexing methods, which offers a promising approach to bolster RALMs' data quality and retrieval precision jointly.
• 摘要：检索增强大型语言模型 (RALM) 在提高生成响应的准确性方面取得了重大进展。然而，现有的研究往往忽略了检索结果中的数据质量问题，而这些问题通常是由现有的基于向量距离的检索方法不准确造成的。我们建议通过上下文驱动的索引修剪 (CDIT) 框架从数据质量的角度提高 RALM 答案的精度，其中上下文匹配依赖关系 (CMD) 被用作逻辑数据质量规则来捕获和规范检索上下文之间的一致性。基于大型语言模型 (LLM) 的语义理解能力，CDIT 可以有效识别和丢弃与查询上下文不一致的检索结果，并进一步修改数据库中的索引，从而提高答案质量。实验证明了在具有挑战性的问答任务上的效果。此外，通过与各种语言模型和索引方法的兼容性验证了 CDIT 的灵活性，这为提高 RALM 的数据质量和检索精度。

Title: P3: A Policy-Driven, Pace-Adaptive, and Diversity-Promoted Framework for Optimizing LLM Training

• Authors: Yingxuan Yang, Huayi Wang, Muning Wen, Weinan Zhang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.05541
• Pdf URL: https://arxiv.org/pdf/2408.05541
• Copy Paste: [[2408.05541]] P3: A Policy-Driven, Pace-Adaptive, and Diversity-Promoted Framework for Optimizing LLM Training(https://arxiv.org/abs/2408.05541)
• Keywords: language model, llm
• Abstract: In the rapidly evolving field of Large Language Models (LLMs), selecting high-quality data for fine-tuning is essential. This paper focuses on task-specific data pruning and selection to enhance fine-tuning. We introduce an innovative framework, termed P3, which improves LLM performance through a dynamic, adaptive training strategy. Specifically, P3 comprises the following components: (1) Policy-driven Difficulty Measurement: we begin by measuring the difficulty of data based on the model's real-time performance, transitioning from static, predefined metrics to more dynamic and adaptable ones. (2) Pace-adaptive Selection: we employ self-paced learning (SPL) to gradually select increasingly challenging data, thereby progressively enhancing the model's performance. (3) Diversity Promotion: we integrate Determinantal Point Process (DPP) into the selection process to promote the diversity within and between samples, enriching the learning process. We have validated our method on two well-known LLM datasets, APPS and MATH, designed for logical reasoning scenarios. The results show that our P3 framework significantly improves training outcomes compared to traditional methods. By fundamentally refining data selection and utilization strategies, P3 not only advances theoretical understanding of dynamic training approaches but also provides a versatile framework that can revolutionize model training in natural language processing.
• 摘要：在快速发展的大型语言模型 (LLM) 领域，选择高质量数据进行微调至关重要。本文重点介绍针对特定任务的数据修剪和选择，以增强微调。我们引入了一个创新框架，称为 P3，它通过动态、自适应的训练策略提高 LLM 性能。具体来说，P3 包含以下组件：（1）策略驱动的难度测量：我们首先根据模型的实时性能测量数据的难度，从静态的预定义指标过渡到更动态和适应性更强的指标。（2）步调自适应选择：我们采用自定进度学习 (SPL) 逐步选择越来越具有挑战性的数据，从而逐步提高模型的性能。（3）多样性促进：我们将行列式点过程 (DPP) 集成到选择过程中，以促进样本内和样本之间的多样性，丰富学习过程。我们已经在两个著名的 LLM 数据集 APPS 和 MATH 上验证了我们的方法，这两个数据集是为逻辑推理场景设计的。结果表明，与传统方法相比，我们的 P3 框架显著改善了训练结果。通过从根本上改进数据选择和利用策略，P3 不仅推进了对动态训练方法的理论理解，而且还提供了一个多功能框架，可以彻底改变自然语言处理中的模型训练。

Title: Large Language Model-based Role-Playing for Personalized Medical Jargon Extraction

• Authors: Jung Hoon Lim, Sunjae Kwon, Zonghai Yao, John P.Lalor, Hong Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.05555
• Pdf URL: https://arxiv.org/pdf/2408.05555
• Copy Paste: [[2408.05555]] Large Language Model-based Role-Playing for Personalized Medical Jargon Extraction(https://arxiv.org/abs/2408.05555)
• Keywords: language model, gpt, llm, chat
• Abstract: Previous studies reveal that Electronic Health Records (EHR), which have been widely adopted in the U.S. to allow patients to access their personal medical information, do not have high readability to patients due to the prevalence of medical jargon. Tailoring medical notes to individual comprehension by identifying jargon that is difficult for each person will enhance the utility of generative models. We present the first quantitative analysis to measure the impact of role-playing in LLM in medical term extraction. By comparing the results of Mechanical Turk workers over 20 sentences, our study demonstrates that LLM role-playing improves F1 scores in 95% of cases across 14 different socio-demographic backgrounds. Furthermore, applying role-playing with in-context learning outperformed the previous state-of-the-art models. Our research showed that ChatGPT can improve traditional medical term extraction systems by utilizing role-play to deliver personalized patient education, a potential that previous models had not achieved.
• 摘要：先前的研究表明，由于医学术语的盛行，美国广泛采用的电子健康记录 (EHR) 允许患者访问其个人医疗信息，但对患者而言，可读性不高。通过识别每个人难以理解的术语，根据个人理解量身定制医疗笔记将提高生成模型的效用。我们首次提出了定量分析，以衡量 LLM 中角色扮演对医学术语提取的影响。通过比较 Mechanical Turk 工作者对 20 个句子的结果，我们的研究表明，LLM 角色扮演在 14 种不同社会人口背景下的 95% 案例中提高了 F1 分数。此外，将角色扮演与情境学习相结合的表现优于以前最先进的模型。我们的研究表明，ChatGPT 可以通过利用角色扮演来提供个性化的患者教育，从而改进传统的医学术语提取系统，这是以前的模型无法实现的潜力。

Title: Document-Level Event Extraction with Definition-Driven ICL

• Authors: Zhuoyuan Liu, Yilin Luo
• Subjects: cs.CL, cs.AI, cs.CY, cs.IR
• Abstract URL: https://arxiv.org/abs/2408.05566
• Pdf URL: https://arxiv.org/pdf/2408.05566
• Copy Paste: [[2408.05566]] Document-Level Event Extraction with Definition-Driven ICL(https://arxiv.org/abs/2408.05566)
• Keywords: language model, llm, prompt
• Abstract: In the field of Natural Language Processing (NLP), Large Language Models (LLMs) have shown great potential in document-level event extraction tasks, but existing methods face challenges in the design of prompts. To address this issue, we propose an optimization strategy called "Definition-driven Document-level Event Extraction (DDEE)." By adjusting the length of the prompt and enhancing the clarity of heuristics, we have significantly improved the event extraction performance of LLMs. We used data balancing techniques to solve the long-tail effect problem, enhancing the model's generalization ability for event types. At the same time, we refined the prompt to ensure it is both concise and comprehensive, adapting to the sensitivity of LLMs to the style of prompts. In addition, the introduction of structured heuristic methods and strict limiting conditions has improved the precision of event and argument role extraction. These strategies not only solve the prompt engineering problems of LLMs in document-level event extraction but also promote the development of event extraction technology, providing new research perspectives for other tasks in the NLP field.
• 摘要：在自然语言处理（NLP）领域，大型语言模型（LLM）在文档级事件抽取任务中展现出巨大潜力，但现有方法在提示符的设计上面临挑战。针对这一问题，我们提出了一种优化策略，称为“定义驱动的文档级事件抽取（DDEE）”。通过调整提示符的长度和增强启发式的清晰度，我们显著提高了LLM的事件抽取性能。我们使用数据平衡技术来解决长尾效应问题，增强了模型对事件类型的泛化能力。同时，我们细化提示符，使其既简洁又全面，适应LLM对提示符风格的敏感性。此外，结构化启发式方法和严格限制条件的引入提高了事件和论元角色抽取的精度。这些策略不仅解决了LLM在文档级事件抽取中的提示工程问题，而且促进了事件抽取技术的发展，为NLP领域的其他任务提供了新的研究视角。

Title: Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion

• Authors: Jacob K Christopher, Brian R Bartoldson, Bhavya Kailkhura, Ferdinando Fioretto
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.05636
• Pdf URL: https://arxiv.org/pdf/2408.05636
• Copy Paste: [[2408.05636]] Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion(https://arxiv.org/abs/2408.05636)
• Keywords: language model
• Abstract: Speculative decoding has emerged as a widely adopted method to accelerate large language model inference without sacrificing the quality of the model outputs. While this technique has facilitated notable speed improvements by enabling parallel sequence verification, its efficiency remains inherently limited by the reliance on incremental token generation in existing draft models. To overcome this limitation, this paper proposes an adaptation of speculative decoding which uses discrete diffusion models to generate draft sequences. This allows parallelization of both the drafting and verification steps, providing significant speed-ups to the inference process. Our proposed approach, \textit{Speculative Diffusion Decoding (SpecDiff)}, is validated on standard language generation benchmarks and empirically demonstrated to provide a \textbf{up to 8.7x speed-up over standard generation processes and up to 2.5x speed-up over existing speculative decoding approaches.}
• 摘要：推测解码已成为一种广泛采用的方法，可在不牺牲模型输出质量的情况下加速大型语言模型推理。虽然这种技术通过实现并行序列验证显著提高了速度，但其效率仍然受到现有草稿模型中对增量标记生成的依赖的固有限制。为了克服这一限制，本文提出了一种推测解码的改编方法，该方法使用离散扩散模型来生成草稿序列。这允许并行化起草和验证步骤，从而显著加快推理过程。我们提出的方法 \textit{推测扩散解码 (SpecDiff)} 在标准语言生成基准上得到了验证，并通过经验证明可提供 \textbf{比标准生成过程快 8.7 倍，比现有推测解码方法快 2.5 倍。}

Title: Reference-free Hallucination Detection for Large Vision-Language Models

• Authors: Qing Li, Chenyang Lyu, Jiahui Geng, Derui Zhu, Maxim Panov, Fakhri Karray
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.05767
• Pdf URL: https://arxiv.org/pdf/2408.05767
• Copy Paste: [[2408.05767]] Reference-free Hallucination Detection for Large Vision-Language Models(https://arxiv.org/abs/2408.05767)
• Keywords: language model, hallucination
• Abstract: Large vision-language models (LVLMs) have made significant progress in recent years. While LVLMs exhibit excellent ability in language understanding, question answering, and conversations of visual inputs, they are prone to producing hallucinations. While several methods are proposed to evaluate the hallucinations in LVLMs, most are reference-based and depend on external tools, which complicates their practical application. To assess the viability of alternative methods, it is critical to understand whether the reference-free approaches, which do not rely on any external tools, can efficiently detect hallucinations. Therefore, we initiate an exploratory study to demonstrate the effectiveness of different reference-free solutions in detecting hallucinations in LVLMs. In particular, we conduct an extensive study on three kinds of techniques: uncertainty-based, consistency-based, and supervised uncertainty quantification methods on four representative LVLMs across two different tasks. The empirical results show that the reference-free approaches are capable of effectively detecting non-factual responses in LVLMs, with the supervised uncertainty quantification method outperforming the others, achieving the best performance across different settings.
• 摘要：大型视觉语言模型 (LVLM) 近年来取得了重大进展。虽然 LVLM 在语言理解、问答和视觉输入对话方面表现出色，但它们容易产生幻觉。虽然提出了几种方法来评估 LVLM 中的幻觉，但大多数都是基于参考的，并且依赖于外部工具，这使它们的实际应用变得复杂。为了评估替代方法的可行性，至关重要的是要了解不依赖任何外部工具的无参考方法是否能有效地检测幻觉。因此，我们发起了一项探索性研究，以证明不同的无参考解决方案在检测 LVLM 中的幻觉方面的有效性。具体而言，我们对三种技术进行了广泛的研究：基于不确定性、基于一致性和监督不确定性量化方法，在两个不同任务的四个代表性 LVLM 上进行了研究。实证结果表明，无参考方法能够有效地检测 LVLM 中的非事实响应，其中监督不确定性量化方法优于其他方法，在不同设置中取得最佳性能。

Title: LI-TTA: Language Informed Test-Time Adaptation for Automatic Speech Recognition

• Authors: Eunseop Yoon, Hee Suk Yoon, John Harvill, Mark Hasegawa-Johnson, Chang D. Yoo
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.05769
• Pdf URL: https://arxiv.org/pdf/2408.05769
• Copy Paste: [[2408.05769]] LI-TTA: Language Informed Test-Time Adaptation for Automatic Speech Recognition(https://arxiv.org/abs/2408.05769)
• Keywords: language model
• Abstract: Test-Time Adaptation (TTA) has emerged as a crucial solution to the domain shift challenge, wherein the target environment diverges from the original training environment. A prime exemplification is TTA for Automatic Speech Recognition (ASR), which enhances model performance by leveraging output prediction entropy minimization as a self-supervision signal. However, a key limitation of this self-supervision lies in its primary focus on acoustic features, with minimal attention to the linguistic properties of the input. To address this gap, we propose Language Informed Test-Time Adaptation (LI-TTA), which incorporates linguistic insights during TTA for ASR. LI-TTA integrates corrections from an external language model to merge linguistic with acoustic information by minimizing the CTC loss from the correction alongside the standard TTA loss. With extensive experiments, we show that LI-TTA effectively improves the performance of TTA for ASR in various distribution shift situations.
• 摘要：测试时自适应 (TTA) 已成为解决领域转移挑战的关键解决方案，其中目标环境与原始训练环境不同。一个主要的例子是用于自动语音识别 (ASR) 的 TTA，它通过利用输出预测熵最小化作为自监督信号来提高模型性能。然而，这种自监督的一个关键限制在于它主要关注声学特征，而很少关注输入的语言属性。为了解决这一差距，我们提出了语言知情测试时自适应 (LI-TTA)，它在 ASR 的 TTA 过程中融入了语言学见解。LI-TTA 集成了来自外部语言模型的校正，通过最小化校正中的 CTC 损失以及标准 TTA 损失来将语言信息与声学信息融合在一起。通过大量实验，我们表明 LI-TTA 有效地提高了各种分布转移情况下 TTA 在 ASR 中的性能。

Title: SAGA: A Participant-specific Examination of Story Alternatives and Goal Applicability for a Deeper Understanding of Complex Events

• Authors: Sai Vallurupalli, Katrin Erk, Francis Ferraro
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.05793
• Pdf URL: https://arxiv.org/pdf/2408.05793
• Copy Paste: [[2408.05793]] SAGA: A Participant-specific Examination of Story Alternatives and Goal Applicability for a Deeper Understanding of Complex Events(https://arxiv.org/abs/2408.05793)
• Keywords: language model
• Abstract: Interpreting and assessing goal driven actions is vital to understanding and reasoning over complex events. It is important to be able to acquire the knowledge needed for this understanding, though doing so is challenging. We argue that such knowledge can be elicited through a participant achievement lens. We analyze a complex event in a narrative according to the intended achievements of the participants in that narrative, the likely future actions of the participants, and the likelihood of goal success. We collect 6.3K high quality goal and action annotations reflecting our proposed participant achievement lens, with an average weighted Fleiss-Kappa IAA of 80%. Our collection contains annotated alternate versions of each narrative. These alternate versions vary minimally from the "original" story, but can license drastically different inferences. Our findings suggest that while modern large language models can reflect some of the goal-based knowledge we study, they find it challenging to fully capture the design and intent behind concerted actions, even when the model pretraining included the data from which we extracted the goal knowledge. We show that smaller models fine-tuned on our dataset can achieve performance surpassing larger models.
• 摘要：解释和评估目标驱动的行动对于理解和推理复杂事件至关重要。能够获得理解所需的知识很重要，尽管这样做很有挑战性。我们认为，这种知识可以通过参与者成就视角来获取。我们根据叙述中参与者的预期成就、参与者未来可能采取的行动以及目标成功的可能性来分析叙述中的复杂事件。我们收集了 6.3K 个高质量的目标和行动注释，反映了我们提出的参与者成就视角，平均加权 Fleiss-Kappa IAA 为 80%。我们的集合包含每个叙述的带注释的替代版本。这些替代版本与“原始”故事差别很小，但可以允许截然不同的推论。我们的研究结果表明，虽然现代大型语言模型可以反映我们研究的一些基于目标的知识，但它们很难完全捕捉协同行动背后的设计和意图，即使模型预训练包含了我们从中提取目标知识的数据。我们表明，在我们的数据集上进行微调的小型模型可以实现超越大型模型的性能。

Title: Defining Boundaries: A Spectrum of Task Feasibility for Large Language Models

• Authors: Wenbo Zhang, Zihang Xu, Hengrui Cai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.05873
• Pdf URL: https://arxiv.org/pdf/2408.05873
• Copy Paste: [[2408.05873]] Defining Boundaries: A Spectrum of Task Feasibility for Large Language Models(https://arxiv.org/abs/2408.05873)
• Keywords: language model, llm, hallucination
• Abstract: Large language models (LLMs) have shown remarkable performance in various tasks but often fail to handle queries that exceed their knowledge and capabilities, leading to incorrect or fabricated responses. This paper addresses the need for LLMs to recognize and refuse infeasible tasks due to the required skills surpassing their capabilities. We first systematically conceptualize infeasible tasks for LLMs, providing formal definitions and categorizations that cover a spectrum of related hallucinations. We develop and benchmark a new dataset comprising diverse infeasible and feasible tasks to test multiple LLMs' abilities on task feasibility. Furthermore, we explore the potential of training enhancements to increase LLMs' refusal capabilities with fine-tuning. Experiments validate the effectiveness of our methods, offering promising directions for refining the operational boundaries of LLMs in real applications.
• 摘要：大型语言模型 (LLM) 在各种任务中都表现出色，但通常无法处理超出其知识和能力的查询，从而导致错误或捏造的响应。本文讨论了 LLM 需要识别和拒绝不可行任务，因为所需技能超出了其能力。我们首先系统地概念化 LLM 的不可行任务，提供涵盖一系列相关幻觉的正式定义和分类。我们开发并基准测试了一个包含各种不可行和可行任务的新数据集，以测试多个 LLM 在任务可行性方面的能力。此外，我们探索了通过微调来提高训练增强能力的潜力，以提高 LLM 的拒绝能力。实验验证了我们方法的有效性，为在实际应用中细化 LLM 的操作边界提供了有希望的方向。

Title: LLM-Based Robust Product Classification in Commerce and Compliance

• Authors: Sina Gholamian, Gianfranco Romani, Bartosz Rudnikowicz, Laura Skylaki
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.05874
• Pdf URL: https://arxiv.org/pdf/2408.05874
• Copy Paste: [[2408.05874]] LLM-Based Robust Product Classification in Commerce and Compliance(https://arxiv.org/abs/2408.05874)
• Keywords: language model, llm
• Abstract: Product classification is a crucial task in international trade, as compliance regulations are verified and taxes and duties are applied based on product categories. Manual classification of products is time-consuming and error-prone, and the sheer volume of products imported and exported renders the manual process infeasible. Consequently, e-commerce platforms and enterprises involved in international trade have turned to automatic product classification using machine learning. However, current approaches do not consider the real-world challenges associated with product classification, such as very abbreviated and incomplete product descriptions. In addition, recent advancements in generative Large Language Models (LLMs) and their reasoning capabilities are mainly untapped in product classification and e-commerce. In this research, we explore the real-life challenges of industrial classification and we propose data perturbations that allow for realistic data simulation. Furthermore, we employ LLM-based product classification to improve the robustness of the prediction in presence of incomplete data. Our research shows that LLMs with in-context learning outperform the supervised approaches in the clean-data scenario. Additionally, we illustrate that LLMs are significantly more robust than the supervised approaches when data attacks are present.
• 摘要：产品分类是国际贸易中的一项重要任务，因为需要验证合规性法规并根据产品类别征收税费。手动对产品进行分类既耗时又容易出错，而且进出口产品的数量庞大，使得手动流程不可行。因此，电子商务平台和参与国际贸易的企业已转向使用机器学习进行自动产品分类。然而，当前的方法没有考虑与产品分类相关的现实挑战，例如非常简短和不完整的产品描述。此外，生成大型语言模型 (LLM) 及其推理能力的最新进展在产品分类和电子商务中主要尚未得到利用。在这项研究中，我们探索了工业分类的现实挑战，并提出了允许进行真实数据模拟的数据扰动。此外，我们采用基于 LLM 的产品分类来提高在存在不完整数据的情况下预测的稳健性。我们的研究表明，在干净数据场景中，具有上下文学习的 LLM 优于监督方法。此外，我们说明，当存在数据攻击时，LLM 比监督方法更为稳健。

Title: Creating Arabic LLM Prompts at Scale

• Authors: Abdelrahman El-Sheikh, Ahmed Elmogtaba, Kareem Darwish, Muhammad Elmallah, Ashraf Elneima, Hassan Sawaf
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.05882
• Pdf URL: https://arxiv.org/pdf/2408.05882
• Copy Paste: [[2408.05882]] Creating Arabic LLM Prompts at Scale(https://arxiv.org/abs/2408.05882)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: The debut of chatGPT and BARD has popularized instruction following text generation using LLMs, where a user can interrogate an LLM using natural language requests and obtain natural language answers that matches their requests. Training LLMs to respond in this manner requires a large number of worked out examples of user requests (aka prompts) with corresponding gold responses. In this paper, we introduce two methods for creating such prompts for Arabic cheaply and quickly. The first methods entails automatically translating existing prompt datasets from English, such as PromptSource and Super-NaturalInstructions, and then using machine translation quality estimation to retain high quality translations only. The second method involves creating natural language prompts on top of existing Arabic NLP datasets. Using these two methods we were able to create more than 67.4 million Arabic prompts that cover a variety of tasks including summarization, headline generation, grammar checking, open/closed question answering, creative writing, etc. We show that fine tuning an open 7 billion parameter large language model, namely base Qwen2 7B, enables it to outperform a state-of-the-art 70 billion parameter instruction tuned model, namely Llama3 70B, in handling Arabic prompts.
• 摘要：chatGPT 和 BARD 的首次亮相使使用 LLM 进行文本生成后的指令变得流行，用户可以使用自然语言请求询问 LLM 并获得与其请求相匹配的自然语言答案。训练 LLM 以这种方式做出响应需要大量经过计算的用户请求示例（又称提示）和相应的黄金响应。在本文中，我们介绍了两种廉价快速地为阿拉伯语创建此类提示的方法。第一种方法需要自动翻译现有的英语提示数据集，例如 PromptSource 和 Super-NaturalInstructions，然后使用机器翻译质量评估仅保留高质量翻译。第二种方法涉及在现有阿拉伯语 NLP 数据集之上创建自然语言提示。使用这两种方法，我们能够创建超过 6740 万个阿拉伯语提示，涵盖各种任务，包括总结、标题生成、语法检查、开放式/封闭式问答、创意写作等。我们表明，对开放的 70 亿参数大型语言模型（即基础 Qwen2 7B）进行微调，使其在处理阿拉伯语提示方面胜过最先进的 700 亿参数指令调整模型（即 Llama3 70B）。

Title: AdTEC: A Unified Benchmark for Evaluating Text Quality in Search Engine Advertising

• Authors: Peinan Zhang, Yusuke Sakai, Masato Mita, Hiroki Ouchi, Taro Watanabe
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.05906
• Pdf URL: https://arxiv.org/pdf/2408.05906
• Copy Paste: [[2408.05906]] AdTEC: A Unified Benchmark for Evaluating Text Quality in Search Engine Advertising(https://arxiv.org/abs/2408.05906)
• Keywords: language model
• Abstract: With the increase in the more fluent ad texts automatically created by natural language generation technology, it is in the high demand to verify the quality of these creatives in a real-world setting. We propose AdTEC, the first public benchmark to evaluate ad texts in multiple aspects from the perspective of practical advertising operations. Our contributions are: (i) Defining five tasks for evaluating the quality of ad texts and building a dataset based on the actual operational experience of advertising agencies, which is typically kept in-house. (ii) Validating the performance of existing pre-trained language models (PLMs) and human evaluators on the dataset. (iii) Analyzing the characteristics and providing challenges of the benchmark. The results show that while PLMs have already reached the practical usage level in several tasks, human still outperforms in certain domains, implying that there is significant room for improvement in such area.
• 摘要：随着自然语言生成技术自动生成的广告文案越来越流畅，在真实场景中验证这些创意质量的需求越来越高。我们提出了 AdTEC，这是第一个从实际广告运营的角度对广告文案进行多方面评估的公共基准测试。我们的贡献包括：（i）定义评估广告文案质量的五个任务，并根据广告代理商的实际运营经验构建数据集，该数据集通常保留在内部。（ii）在数据集上验证现有预训练语言模型 (PLM) 和人工评估者的性能。（iii）分析基准测试的特点并提出挑战。结果表明，虽然 PLM 已经在多个任务中达到实际使用水平，但人类在某些领域仍然表现优异，这意味着这些领域还有很大的改进空间。

Title: A New Pipeline For Generating Instruction Dataset via RAG and Self Fine-Tuning

• Authors: Chih-Wei Song, Yu-Kai Lee, Yin-Te Tsai
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.05911
• Pdf URL: https://arxiv.org/pdf/2408.05911
• Copy Paste: [[2408.05911]] A New Pipeline For Generating Instruction Dataset via RAG and Self Fine-Tuning(https://arxiv.org/abs/2408.05911)
• Keywords: language model, llm, retrieval-augmented generation, agent
• Abstract: With the rapid development of large language models in recent years, there has been an increasing demand for domain-specific Agents that can cater to the unique needs of enterprises and organizations. Unlike general models, which strive for broad coverage, these specialized Agents rely on focused datasets tailored to their intended applications. This research proposes a pipeline that leverages the power of LLMs and the Retrieval-Augmented Generation related framework to construct high-quality instruction datasets for fine-tuning on specific domains using custom document collections. By ingesting domain-specific documents, the pipeline generates relevant and contextually appropriate instructions, thus effectively creating a comprehensive dataset for fine-tuning LLMs on the target domain. This approach overcomes the limitations of traditional dataset creation methods, which often rely on manual curation or web-scraping techniques that may introduce noise and irrelevant data. Notably, our pipeline offers a dynamic solution that can quickly adapt to updates or modifications in the domain-specific document collection, eliminating the need for complete retraining. Additionally, it addresses the challenge of data scarcity by enabling the generation of instruction datasets from a limited set of initial documents, rendering it suitable for unpopular or specialized domains where comprehensive datasets are scarce. As a case study, we apply this approach to the domain of psychiatry, a field requiring specialized knowledge and sensitive handling of patient information. The resulting fine-tuned LLM demonstrates showcases the viability of the proposed approach and underscores its potential for widespread adoption across various industries and domains where tailored, accurate, and contextually relevant language models are indispensable.
• 摘要：近年来，随着大型语言模型的快速发展，对能够满足企业和组织独特需求的领域特定代理的需求日益增长。与力求广泛覆盖的通用模型不同，这些专用代理依赖于针对其预期应用量身定制的重点数据集。这项研究提出了一种管道，利用 LLM 和检索增强生成相关框架的强大功能，使用自定义文档集合构建高质量指令数据集，以便在特定领域进行微调。通过提取领域特定文档，管道生成相关且上下文适当的指令，从而有效地创建一个全面的数据集，用于在目标领域微调 LLM。这种方法克服了传统数据集创建方法的局限性，这些方法通常依赖于手动管理或网络抓取技术，可能会引入噪音和不相关的数据。值得注意的是，我们的管道提供了一种动态解决方案，可以快速适应领域特定文档集合中的更新或修改，无需完全重新训练。此外，它还通过从一组有限的初始文档生成指令数据集来解决数据稀缺的挑战，使其适用于综合数据集稀缺的冷门或专业领域。作为案例研究，我们将这种方法应用于精神病学领域，该领域需要专业知识和对患者信息的敏感处理。由此产生的经过微调的 LLM 演示展示了所提出方法的可行性，并强调了其在各个行业和领域广泛采用的潜力，在这些行业和领域中，量身定制、准确且与上下文相关的语言模型是必不可少的。

Title: ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge Graph QA datasets with Large Language Models

• Authors: Ronak Pradeep, Daniel Lee, Ali Mousavi, Jeff Pound, Yisi Sang, Jimmy Lin, Ihab Ilyas, Saloni Potdar, Mostafa Arefiyan, Yunyao Li
• Subjects: cs.CL, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.05948
• Pdf URL: https://arxiv.org/pdf/2408.05948
• Copy Paste: [[2408.05948]] ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge Graph QA datasets with Large Language Models(https://arxiv.org/abs/2408.05948)
• Keywords: language model, llm
• Abstract: The rapid advancement of Large Language Models (LLMs) and conversational assistants necessitates dynamic, scalable, and configurable conversational datasets for training and evaluation. These datasets must accommodate diverse user interaction modes, including text and voice, each presenting unique modeling challenges. Knowledge Graphs (KGs), with their structured and evolving nature, offer an ideal foundation for current and precise knowledge. Although human-curated KG-based conversational datasets exist, they struggle to keep pace with the rapidly changing user information needs. We present ConvKGYarn, a scalable method for generating up-to-date and configurable conversational KGQA datasets. Qualitative psychometric analyses confirm our method can generate high-quality datasets rivaling a popular conversational KGQA dataset while offering it at scale and covering a wide range of human-interaction configurations. We showcase its utility by testing LLMs on diverse conversations - exploring model behavior on conversational KGQA sets with different configurations grounded in the same KG fact set. Our results highlight the ability of ConvKGYarn to improve KGQA foundations and evaluate parametric knowledge of LLMs, thus offering a robust solution to the constantly evolving landscape of conversational assistants.
• 摘要：大型语言模型 (LLM) 和对话助手的快速发展需要动态、可扩展和可配置的对话数据集来进行训练和评估。这些数据集必须适应不同的用户交互模式，包括文本和语音，每种模式都提出了独特的建模挑战。知识图谱 (KG) 具有结构化和不断发展的特性，为当前和精确的知识提供了理想的基础。尽管存在人工策划的基于 KG 的对话数据集，但它们难以跟上快速变化的用户信息需求。我们提出了 ConvKGYarn，这是一种可扩展的方法，用于生成最新且可配置的对话 KGQA 数据集。定性心理测量分析证实，我们的方法可以生成与流行的对话 KGQA 数据集相媲美的高质量数据集，同时提供大规模数据集并涵盖广泛的人机交互配置。我们通过在不同对话中测试 LLM 来展示其实用性 - 探索基于同一 KG 事实集的不同配置的对话 KGQA 集上的模型行为。我们的研究结果强调了 ConvKGYarn 改进 KGQA 基础和评估 LLM 参数知识的能力，从而为不断发展的对话助手领域提供了强大的解决方案。

Title: The Language of Trauma: Modeling Traumatic Event Descriptions Across Domains with Explainable AI

• Authors: Miriam Schirmer, Tobias Leemann, Gjergji Kasneci, Jürgen Pfeffer, David Jurgens
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.05977
• Pdf URL: https://arxiv.org/pdf/2408.05977
• Copy Paste: [[2408.05977]] The Language of Trauma: Modeling Traumatic Event Descriptions Across Domains with Explainable AI(https://arxiv.org/abs/2408.05977)
• Keywords: language model, gpt
• Abstract: Psychological trauma can manifest following various distressing events and is captured in diverse online contexts. However, studies traditionally focus on a single aspect of trauma, often neglecting the transferability of findings across different scenarios. We address this gap by training language models with progressing complexity on trauma-related datasets, including genocide-related court data, a Reddit dataset on post-traumatic stress disorder (PTSD), counseling conversations, and Incel forum posts. Our results show that the fine-tuned RoBERTa model excels in predicting traumatic events across domains, slightly outperforming large language models like GPT-4. Additionally, SLALOM-feature scores and conceptual explanations effectively differentiate and cluster trauma-related language, highlighting different trauma aspects and identifying sexual abuse and experiences related to death as a common traumatic event across all datasets. This transferability is crucial as it allows for the development of tools to enhance trauma detection and intervention in diverse populations and settings.
• 摘要：心理创伤可能在各种令人痛苦的事件后表现出来，并在不同的在线环境中被捕捉到。然而，研究传统上只关注创伤的单一方面，往往忽略了研究结果在不同场景中的可转移性。我们通过在创伤相关数据集上训练复杂度不断提高的语言模型来解决这一差距，这些数据集包括与种族灭绝相关的法庭数据、Reddit 的创伤后应激障碍 (PTSD) 数据集、咨询对话和 Incel 论坛帖子。我们的结果表明，经过微调的 RoBERTa 模型在预测跨领域的创伤事件方面表现出色，略优于 GPT-4 等大型语言模型。此外，SLALOM 特征分数和概念解释可以有效地区分和聚类与创伤相关的语言，突出不同的创伤方面，并将性虐待和与死亡相关的经历确定为所有数据集中的常见创伤事件。这种可转移性至关重要，因为它允许开发工具来增强对不同人群和环境的创伤检测和干预。

Title: Enhancing Dialogue Speech Recognition with Robust Contextual Awareness via Noise Representation Learning

• Authors: Wonjun Lee, San Kim, Gary Geunbae Lee
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.06043
• Pdf URL: https://arxiv.org/pdf/2408.06043
• Copy Paste: [[2408.06043]] Enhancing Dialogue Speech Recognition with Robust Contextual Awareness via Noise Representation Learning(https://arxiv.org/abs/2408.06043)
• Keywords: agent
• Abstract: Recent dialogue systems rely on turn-based spoken interactions, requiring accurate Automatic Speech Recognition (ASR). Errors in ASR can significantly impact downstream dialogue tasks. To address this, using dialogue context from user and agent interactions for transcribing subsequent utterances has been proposed. This method incorporates the transcription of the user's speech and the agent's response as model input, using the accumulated context generated by each turn. However, this context is susceptible to ASR errors because it is generated by the ASR model in an auto-regressive fashion. Such noisy context can further degrade the benefits of context input, resulting in suboptimal ASR performance. In this paper, we introduce Context Noise Representation Learning (CNRL) to enhance robustness against noisy context, ultimately improving dialogue speech recognition accuracy. To maximize the advantage of context awareness, our approach includes decoder pre-training using text-based dialogue data and noise representation learning for a context encoder. Based on the evaluation of speech dialogues, our method shows superior results compared to baselines. Furthermore, the strength of our approach is highlighted in noisy environments where user speech is barely audible due to real-world noise, relying on contextual information to transcribe the input accurately.
• 摘要：最近的对话系统依赖于回合制口头互动，需要准确的自动语音识别 (ASR)。ASR 中的错​​误会严重影响下游对话任务。为了解决这个问题，有人提出了使用用户和代理互动中的对话上下文来转录后续话语的方法。该方法将用户语音和代理响应的转录作为模型输入，使用每个回合生成的累积上下文。然而，这种上下文容易受到 ASR 错误的影响，因为它是由 ASR 模型以自回归方式生成的。这种嘈杂的上下文会进一步降低上下文输入的好处，导致 ASR 性能不佳。在本文中，我们引入了上下文噪声表示学习 (CNRL) 来增强对嘈杂上下文的鲁棒性，最终提高对话语音识别准确性。为了最大限度地发挥上下文感知的优势，我们的方法包括使用基于文本的对话数据对解码器进行预训练，以及对上下文编码器进行噪声表示学习。基于对语音对话的评估，我们的方法与基线相比显示出更优的结果。此外，我们的方法的优势在嘈杂的环境中得到了凸显。在这种环境中，由于现实世界的噪音，用户的语音几乎听不见，只能依靠上下文信息来准确地转录输入。

Title: DiagESC: Dialogue Synthesis for Integrating Depression Diagnosis into Emotional Support Conversation

• Authors: Seungyeon Seo, Gary Geunbae Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06044
• Pdf URL: https://arxiv.org/pdf/2408.06044
• Copy Paste: [[2408.06044]] DiagESC: Dialogue Synthesis for Integrating Depression Diagnosis into Emotional Support Conversation(https://arxiv.org/abs/2408.06044)
• Keywords: prompt
• Abstract: Dialogue systems for mental health care aim to provide appropriate support to individuals experiencing mental distress. While extensive research has been conducted to deliver adequate emotional support, existing studies cannot identify individuals who require professional medical intervention and cannot offer suitable guidance. We introduce the Diagnostic Emotional Support Conversation task for an advanced mental health management system. We develop the DESC dataset to assess depression symptoms while maintaining user experience by utilizing task-specific utterance generation prompts and a strict filtering algorithm. Evaluations by professional psychological counselors indicate that DESC has a superior ability to diagnose depression than existing data. Additionally, conversational quality evaluation reveals that DESC maintains fluent, consistent, and coherent dialogues.
• 摘要：心理健康护理对话系统旨在为遭受心理困扰的个人提供适当的支持。虽然已经进行了广泛的研究来提供足够的情感支持，但现有研究无法识别需要专业医疗干预的个体，也无法提供适当的指导。我们为高级心理健康管理系统引入了诊断性情感支持对话任务。我们开发了 DESC 数据集来评估抑郁症症状，同时通过使用特定于任务的话语生成提示和严格的过滤算法来保持用户体验。专业心理咨询师的评估表明，DESC 诊断抑郁症的能力优于现有数据。此外，对话质量评估表明 DESC 保持流畅、一致和连贯的对话。

Title: Building Decision Making Models Through Language Model Regime

• Authors: Yu Zhang, Haoxiang Liu, Feijun Jiang, Weihua Luo, Kaifu Zhang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.06087
• Pdf URL: https://arxiv.org/pdf/2408.06087
• Copy Paste: [[2408.06087]] Building Decision Making Models Through Language Model Regime(https://arxiv.org/abs/2408.06087)
• Keywords: language model, llm
• Abstract: We propose a novel approach for decision making problems leveraging the generalization capabilities of large language models (LLMs). Traditional methods such as expert systems, planning algorithms, and reinforcement learning often exhibit limited generalization, typically requiring the training of new models for each unique task. In contrast, LLMs demonstrate remarkable success in generalizing across varied language tasks, inspiring a new strategy for training decision making models. Our approach, referred to as "Learning then Using" (LTU), entails a two-stage process. Initially, the \textit{learning} phase develops a robust foundational decision making model by integrating diverse knowledge from various domains and decision making contexts. The subsequent \textit{using} phase refines this foundation model for specific decision making scenarios. Distinct from other studies that employ LLMs for decision making through supervised learning, our LTU method embraces a versatile training methodology that combines broad pre-training with targeted fine-tuning. Experiments in e-commerce domains such as advertising and search optimization have shown that LTU approach outperforms traditional supervised learning regimes in decision making capabilities and generalization. The LTU approach is the first practical training architecture for both single-step and multi-step decision making tasks combined with LLMs, which can be applied beyond game and robot domains. It provides a robust and adaptable framework for decision making, enhances the effectiveness and flexibility of various systems in tackling various challenges.
• 摘要：我们提出了一种利用大型语言模型 (LLM) 的泛化能力来解决决策问题的新方法。专家系统、规划算法和强化学习等传统方法通常表现出有限的泛化能力，通常需要为每个独特任务训练新模型。相比之下，LLM 在跨各种语言任务的泛化方面表现出显著的成功，激发了一种训练决策模型的新策略。我们的方法称为“先学后用”(LTU)，包含一个两阶段的过程。最初，\textit{学习} 阶段通过整合来自不同领域和决策环境的各种知识来开发一个强大的基础决策模型。随后的 \textit{使用} 阶段针对特定的决策场景完善了这个基础模型。与其他通过监督学习使用 LLM 进行决策的研究不同，我们的 LTU 方法采用了一种多功能的训练方法，将广泛的预训练与有针对性的微调相结合。在广告和搜索优化等电子商务领域的实验表明，LTU 方法在决策能力和泛化方面优于传统的监督学习方案。LTU 方法是第一个结合 LLM 的单步和多步决策任务的实用训练架构，可应用于游戏和机器人领域之外。它为决策提供了一个强大且适应性强的框架，提高了各种系统在应对各种挑战时的有效性和灵活性。

Title: How ChatGPT Changed the Media's Narratives on AI: A Semi-Automated Narrative Analysis Through Frame Semantics

• Authors: Igor Ryazanov, Carl Öhman, Johanna Björklund
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2408.06120
• Pdf URL: https://arxiv.org/pdf/2408.06120
• Copy Paste: [[2408.06120]] How ChatGPT Changed the Media's Narratives on AI: A Semi-Automated Narrative Analysis Through Frame Semantics(https://arxiv.org/abs/2408.06120)
• Keywords: gpt, chat
• Abstract: The recent explosion of attention to AI is arguably one of the biggest in the technology's media coverage. To investigate the effects it has on the discourse, we perform a mixed-method frame semantics-based analysis on a dataset of more than 49,000 sentences collected from 5846 news articles that mention AI. The dataset covers the twelve-month period centred around the launch of OpenAI's chatbot ChatGPT and is collected from the most visited open-access English-language news publishers. Our findings indicate that during the half year succeeding the launch, media attention rose tenfold$\unicode{x2014}$from already historically high levels. During this period, discourse has become increasingly centred around experts and political leaders, and AI has become more closely associated with dangers and risks. A deeper review of the data also suggests a qualitative shift in the types of threat AI is thought to represent, as well as the anthropomorphic qualities ascribed to it.
• 摘要：最近，人们对人工智能的关注度激增，这可以说是媒体对该技术报道中最大的一次。为了研究它对话语的影响，我们对从 5846 篇提及人工智能的新闻文章中收集的 49,000 多个句子的数据集进行了基于混合方法框架语义的分析。该数据集涵盖了 OpenAI 聊天机器人 ChatGPT 发布前后的十二个月时间，收集自访问量最大的开放获取英语新闻出版商。我们的研究结果表明，在发布后的半年内，媒体关注度从历史最高水平上升了十倍。在此期间，话语越来越多地集中在专家和政治领袖身上，人工智能与危险和风险的联系也越来越紧密。对数据的更深入审查还表明，人工智能被认为代表的威胁类型以及赋予它的拟人化特质发生了质的转变。

Title: Utilize Transformers for translating Wikipedia category names

• Authors: Hoang-Thang Ta, Quoc Thang La
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06124
• Pdf URL: https://arxiv.org/pdf/2408.06124
• Copy Paste: [[2408.06124]] Utilize Transformers for translating Wikipedia category names(https://arxiv.org/abs/2408.06124)
• Keywords: language model
• Abstract: On Wikipedia, articles are categorized to aid readers in navigating content efficiently. The manual creation of new categories can be laborious and time-intensive. To tackle this issue, we built language models to translate Wikipedia categories from English to Vietnamese with a dataset containing 15,000 English-Vietnamese category pairs. Subsequently, small to medium-scale Transformer pre-trained models with a sequence-to-sequence architecture were fine-tuned for category translation. The experiments revealed that OPUS-MT-en-vi surpassed other models, attaining the highest performance with a BLEU score of 0.73, despite its smaller model storage. We expect our paper to be an alternative solution for translation tasks with limited computer resources.
• 摘要：在维基百科上，文章被分类以帮助读者高效地浏览内容。手动创建新类别可能非常费力且耗时。为了解决这个问题，我们建立了语言模型，使用包含 15,000 个英语-越南语类别对的数据集将维基百科类别从英语翻译成越南语。随后，对具有序列到序列架构的小型到中型 Transformer 预训练模型进行了微调，以进行类别翻译。实验表明，尽管模型存储较小，但 OPUS-MT-en-vi 的表现优于其他模型，BLEU 得分为 0.73，达到最高性能。我们希望我们的论文能够成为计算机资源有限的翻译任务的替代解决方案。

Title: Med42-v2: A Suite of Clinical LLMs

• Authors: Clément Christophe, Praveen K Kanithi, Tathagata Raha, Shadab Khan, Marco AF Pimentel
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06142
• Pdf URL: https://arxiv.org/pdf/2408.06142
• Copy Paste: [[2408.06142]] Med42-v2: A Suite of Clinical LLMs(https://arxiv.org/abs/2408.06142)
• Keywords: language model, gpt, llm, prompt
• Abstract: Med42-v2 introduces a suite of clinical large language models (LLMs) designed to address the limitations of generic models in healthcare settings. These models are built on Llama3 architecture and fine-tuned using specialized clinical data. They underwent multi-stage preference alignment to effectively respond to natural prompts. While generic models are often preference-aligned to avoid answering clinical queries as a precaution, Med42-v2 is specifically trained to overcome this limitation, enabling its use in clinical settings. Med42-v2 models demonstrate superior performance compared to the original Llama3 models in both 8B and 70B parameter configurations and GPT-4 across various medical benchmarks. These LLMs are developed to understand clinical queries, perform reasoning tasks, and provide valuable assistance in clinical environments. The models are now publicly available at \href{this https URL}{this https URL}.
• 摘要：Med42-v2 引入了一套临床大型语言模型 (LLM)，旨在解决通用模型在医疗保健环境中的局限性。这些模型基于 Llama3 架构构建，并使用专门的临床数据进行微调。它们经过多阶段偏好对齐，以有效响应自然提示。虽然通用模型通常会进行偏好对齐，以避免出于预防措施回答临床查询，但 Med42-v2 经过专门训练以克服这一限制，使其能够在临床环境中使用。在各种医学基准测试中，Med42-v2 模型在 8B 和 70B 参数配置和 GPT-4 方面均表现出优于原始 Llama3 模型的性能。这些 LLM 旨在理解临床查询、执行推理任务并在临床环境中提供有价值的帮助。这些模型现已在 \href{this https URL}{this https URL} 上公开提供。

Title: LipidBERT: A Lipid Language Model Pre-trained on METiS de novo Lipid Library

• Authors: Tianhao Yu, Cai Yao, Zhuorui Sun, Feng Shi, Lin Zhang, Kangjie Lyu, Xuan Bai, Andong Liu, Xicheng Zhang, Jiali Zou, Wenshou Wang, Chris Lai, Kai Wang
• Subjects: cs.CL, physics.chem-ph, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2408.06150
• Pdf URL: https://arxiv.org/pdf/2408.06150
• Copy Paste: [[2408.06150]] LipidBERT: A Lipid Language Model Pre-trained on METiS de novo Lipid Library(https://arxiv.org/abs/2408.06150)
• Keywords: language model, gpt
• Abstract: In this study, we generate and maintain a database of 10 million virtual lipids through METiS's in-house de novo lipid generation algorithms and lipid virtual screening techniques. These virtual lipids serve as a corpus for pre-training, lipid representation learning, and downstream task knowledge transfer, culminating in state-of-the-art LNP property prediction performance. We propose LipidBERT, a BERT-like model pre-trained with the Masked Language Model (MLM) and various secondary tasks. Additionally, we compare the performance of embeddings generated by LipidBERT and PhatGPT, our GPT-like lipid generation model, on downstream tasks. The proposed bilingual LipidBERT model operates in two languages: the language of ionizable lipid pre-training, using in-house dry-lab lipid structures, and the language of LNP fine-tuning, utilizing in-house LNP wet-lab data. This dual capability positions LipidBERT as a key AI-based filter for future screening tasks, including new versions of METiS de novo lipid libraries and, more importantly, candidates for in vivo testing for orgran-targeting LNPs. To the best of our knowledge, this is the first successful demonstration of the capability of a pre-trained language model on virtual lipids and its effectiveness in downstream tasks using web-lab data. This work showcases the clever utilization of METiS's in-house de novo lipid library as well as the power of dry-wet lab integration.
• 摘要：在本研究中，我们通过 METiS 的内部从头脂质生成算法和脂质虚拟筛选技术生成并维护了一个包含 1000 万种虚拟脂质的数据库。这些虚拟脂质可作为预训练、脂质表征学习和下游任务知识转移的语料库，最终实现最先进的 LNP 属性预测性能。我们提出了 LipidBERT，这是一个使用 Masked Language Model (MLM) 和各种辅助任务进行预训练的类似 BERT 的模型。此外，我们还比较了 LipidBERT 和我们的类似 GPT 的脂质生成模型 PhatGPT 在下游任务中生成的嵌入的性能。所提出的双语 LipidBERT 模型以两种语言运行：使用内部干实验室脂质结构的可电离脂质预训练语言，以及利用内部 LNP 湿实验室数据的 LNP 微调语言。这种双重功能使 LipidBERT 成为未来筛选任务的关键 AI 过滤器，包括 METiS 从头脂质库的新版本，更重要的是，它是针对器官靶向 LNP 的体内测试候选物。据我们所知，这是首次成功展示预训练语言模型在虚拟脂质上的能力及其在使用网络实验室数据的下游任务中的有效性。这项工作展示了 METiS 内部从头脂质库的巧妙利用以及干湿实验室集成的强大功能。

Title: Improving Structural Diversity of Blackbox LLMs via Chain-of-Specification Prompting

• Authors: Halley Young, Yimeng Zeng, Jacob Gardner, Osbert Bastani
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.06186
• Pdf URL: https://arxiv.org/pdf/2408.06186
• Copy Paste: [[2408.06186]] Improving Structural Diversity of Blackbox LLMs via Chain-of-Specification Prompting(https://arxiv.org/abs/2408.06186)
• Keywords: language model, llm, prompt
• Abstract: The capability to generate diverse text is a key challenge facing large language models (LLMs). Thus far, diversity has been studied via metrics such as $n$-gram diversity or diversity of BERT embeddings. However, for these kinds of diversity, the user has little control over the dimensions along which diversity is considered. For example, in the poetry domain, one might desire diversity in terms of rhyme and meter, whereas in the code domain, one might desire diversity in terms of the kinds of expressions used to solve a problem. We propose a diversity metric called structural diversity, where the user provides a mapping from generated text to features capturing the kinds of diversity that they care about. In addition, we propose a novel strategy called chain-of-specification (CoS) prompting for improving diversity by first having the LLM generate a specification encoding one instance of structural features, and then prompting the LLM to generate text that satisfies these features; notably, our strategy works with blackbox LLMs. In our experiments, we show that for structural diversity in the poetry and code domains, CoS significantly improves diversity compared to several baselines.
• 摘要：生成多样化文本的能力是大型语言模型 (LLM) 面临的一个关键挑战。到目前为止，多样性已经通过诸如 $n$-gram 多样性或 BERT 嵌入多样性等指标进行了研究。然而，对于这些类型的多样性，用户几乎无法控制考虑多样性的维度。例如，在诗歌领域，人们可能希望在韵律和韵律方面具有多样性，而在代码领域，人们可能希望在解决问题所用的表达方式方面具有多样性。我们提出了一种称为结构多样性的多样性指标，其中用户提供从生成的文本到捕获他们关心的多样性类型的特征的映射。此外，我们提出了一种称为规范链 (CoS) 的新策略，通过首先让 LLM 生成一个编码一个结构特征实例的规范，然后提示 LLM 生成满足这些特征的文本来提示提高多样性；值得注意的是，我们的策略适用于黑盒 LLM。在我们的实验中，我们表明，对于诗歌和代码领域的结构多样性，与几个基线相比，CoS 显著提高了多样性。

Title: Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers

• Authors: Zhenting Qi, Mingyuan Ma, Jiahang Xu, Li Lyna Zhang, Fan Yang, Mao Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06195
• Pdf URL: https://arxiv.org/pdf/2408.06195
• Copy Paste: [[2408.06195]] Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers(https://arxiv.org/abs/2408.06195)
• Keywords: language model, llm
• Abstract: This paper introduces rStar, a self-play mutual reasoning approach that significantly improves reasoning capabilities of small language models (SLMs) without fine-tuning or superior models. rStar decouples reasoning into a self-play mutual generation-discrimination process. First, a target SLM augments the Monte Carlo Tree Search (MCTS) with a rich set of human-like reasoning actions to construct higher quality reasoning trajectories. Next, another SLM, with capabilities similar to the target SLM, acts as a discriminator to verify each trajectory generated by the target SLM. The mutually agreed reasoning trajectories are considered mutual consistent, thus are more likely to be correct. Extensive experiments across five SLMs demonstrate rStar can effectively solve diverse reasoning problems, including GSM8K, GSM-Hard, MATH, SVAMP, and StrategyQA. Remarkably, rStar boosts GSM8K accuracy from 12.51% to 63.91% for LLaMA2-7B, from 36.46% to 81.88% for Mistral-7B, from 74.53% to 91.13% for LLaMA3-8B-Instruct. Code will be available at this https URL.
• 摘要：本文介绍了一种自我对弈的相互推理方法 rStar，该方法无需微调或高级模型即可显著提高小型语言模型 (SLM) 的推理能力。rStar 将推理分解为自我对弈的相互生成-判别过程。首先，目标 SLM 使用一组丰富的类似人类的推理动作增强蒙特卡洛树搜索 (MCTS)，以构建更高质量的推理轨迹。接下来，另一个具有与目标 SLM 类似功能的 SLM 充当鉴别器来验证目标 SLM 生成的每条轨迹。相互同意的推理轨迹被认为是相互一致的，因此更有可能是正确的。在五个 SLM 上进行的大量实验表明，rStar 可以有效解决各种推理问题，包括 GSM8K、GSM-Hard、MATH、SVAMP 和 StrategyQA。值得注意的是，rStar 将 LLaMA2-7B 的 GSM8K 准确率从 12.51% 提高到 63.91%，将 Mistral-7B 的准确率从 36.46% 提高到 81.88%，将 LLaMA3-8B-Instruct 的准确率从 74.53% 提高到 91.13%。代码可在此 https URL 上获取。

Title: On Effects of Steering Latent Representation for Large Language Model Unlearning

• Authors: Dang Huu-Tien, Trung-Tin Pham, Hoang Thanh-Tung, Naoya Inoue
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06223
• Pdf URL: https://arxiv.org/pdf/2408.06223
• Copy Paste: [[2408.06223]] On Effects of Steering Latent Representation for Large Language Model Unlearning(https://arxiv.org/abs/2408.06223)
• Keywords: language model, llm
• Abstract: Representation Misdirection for Unlearning (RMU), which steers model representation in the intermediate layer to a target random representation, is an effective method for large language model (LLM) unlearning. Despite its high performance, the underlying cause and explanation remain underexplored. In this paper, we first theoretically demonstrate that steering forget representations in the intermediate layer reduces token confidence, causing LLMs to generate wrong or nonsense responses. Second, we investigate how the coefficient influences the alignment of forget-sample representations with the random direction and hint at the optimal coefficient values for effective unlearning across different network layers. Third, we show that RMU unlearned models are robust against adversarial jailbreak attacks. Last, our empirical analysis shows that RMU is less effective when applied to the middle and later layers in LLMs. To resolve this drawback, we propose Adaptive RMU -- a simple yet effective alternative method that makes unlearning effective with most layers. Extensive experiments demonstrate that Adaptive RMU significantly improves the unlearning performance compared to prior art while incurring no additional computational cost.
• 摘要：表示误导反学习 (RMU) 是一种有效的大型语言模型 (LLM) 反学习方法，它将中间层的模型表示引导到目标随机表示。尽管它具有很高的性能，但其根本原因和解释仍未得到充分探索。在本文中，我们首先从理论上证明，在中间层引导遗忘表示会降低 token 置信度，导致 LLM 生成错误或无意义的响应。其次，我们研究了系数如何影响遗忘样本表示与随机方向的对齐，并提示了跨不同网络层有效反学习的最佳系数值。第三，我们表明 RMU 反学习模型对对抗性越狱攻击具有鲁棒性。最后，我们的实证分析表明，当应用于 LLM 的中间层和后面的层时，RMU 的效果较差。为了解决这个缺点，我们提出了自适应 RMU——一种简单而有效的替代方法，使反学习对大多数层都有效。大量实验表明，与现有技术相比，自适应 RMU 显著提高了反学习性能，同时不产生额外的计算成本。

Title: FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Data

• Authors: Haoran Sun, Renren Jin, Shaoyang Xu, Leiyu Pan, Supryadi, Menglong Cui, Jiangcun Dui, Yikun Lei, Lei Yang, Ling Shi, Juesi Xiao, Shaolin Zhu, Deyi Xiong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06273
• Pdf URL: https://arxiv.org/pdf/2408.06273
• Copy Paste: [[2408.06273]] FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Data(https://arxiv.org/abs/2408.06273)
• Keywords: language model, llm, chat
• Abstract: Large language models (LLMs) have demonstrated prowess in a wide range of tasks. However, many LLMs exhibit significant performance discrepancies between high- and low-resource languages. To mitigate this challenge, we present FuxiTranyu, an open-source multilingual LLM, which is designed to satisfy the need of the research community for balanced and high-performing multilingual capabilities. FuxiTranyu-8B, the base model with 8 billion parameters, is trained from scratch on a meticulously balanced multilingual data repository that contains 600 billion tokens covering 43 natural languages and 16 programming languages. In addition to the base model, we also develop two instruction-tuned models: FuxiTranyu-8B-SFT that is fine-tuned on a diverse multilingual instruction dataset, and FuxiTranyu-8B-DPO that is further refined with DPO on a preference dataset for enhanced alignment ability. Extensive experiments on a wide range of multilingual benchmarks demonstrate the competitive performance of FuxiTranyu against existing multilingual LLMs, e.g., BLOOM-7B, PolyLM-13B, Llama-2-Chat-7B and Mistral-7B-Instruct. Interpretability analyses at both the neuron and representation level suggest that FuxiTranyu is able to learn consistent multilingual representations across different languages. To promote further research into multilingual LLMs and their working mechanisms, we release both the base and instruction-tuned FuxiTranyu models together with 58 pretraining checkpoints at HuggingFace and Github.
• 摘要：大型语言模型 (LLM) 已在各种任务中展现出强大威力。然而，许多 LLM 在高资源语言和低资源语言之间表现出显著的性能差异。为了应对这一挑战，我们推出了 FuxiTranyu，这是一个开源多语言 LLM，旨在满足研究界对平衡且高性能多语言能力的需求。FuxiTranyu-8B 是一个具有 80 亿个参数的基础模型，它在一个精心平衡的多语言数据存储库上从头开始训练，该存储库包含 6000 亿个标记，涵盖 43 种自然语言和 16 种编程语言。除了基础模型之外，我们还开发了两个指令调整模型：FuxiTranyu-8B-SFT 在多样化多语言指令数据集上进行了微调，FuxiTranyu-8B-DPO 在偏好数据集上使用 DPO 进一步细化以增强对齐能力。在各种多语言基准上进行的大量实验表明，FuxiTranyu 与现有的多语言 LLM（例如 BLOOM-7B、PolyLM-13B、Llama-2-Chat-7B 和 Mistral-7B-Instruct）相比具有竞争力。神经元和表示级别的可解释性分析表明，FuxiTranyu 能够学习跨不同语言的一致多语言表示。为了促进对多语言 LLM 及其工作机制的进一步研究，我们在 HuggingFace 和 Github 上发布了基础和指令调整的 FuxiTranyu 模型以及 58 个预训练检查点。

Title: Review-driven Personalized Preference Reasoning with Large Language Models for Recommendation

• Authors: Jieyong Kim, Hyunseo Kim, Hyunjin Cho, SeongKu Kang, Buru Chang, Jinyoung Yeo, Dongha Lee
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06276
• Pdf URL: https://arxiv.org/pdf/2408.06276
• Copy Paste: [[2408.06276]] Review-driven Personalized Preference Reasoning with Large Language Models for Recommendation(https://arxiv.org/abs/2408.06276)
• Keywords: language model, llm
• Abstract: Recent advancements in Large Language Models (LLMs) have demonstrated exceptional performance across a wide range of tasks, generating significant interest in their application to recommendation systems. However, existing methods have not fully capitalized on the potential of LLMs, often constrained by limited input information or failing to fully utilize their advanced reasoning capabilities. To address these limitations, we introduce EXP3RT, a novel LLM-based recommender designed to leverage rich preference information contained in user and item reviews. EXP3RT is basically fine-tuned through distillation from a teacher LLM to perform three key tasks in order: EXP3RT first extracts and encapsulates essential subjective preferences from raw reviews, aggregates and summarizes them according to specific criteria to create user and item profiles. It then generates detailed step-by-step reasoning followed by predicted rating, i.e., reasoning-enhanced rating prediction, by considering both subjective and objective information from user/item profiles and item descriptions. This personalized preference reasoning from EXP3RT enhances rating prediction accuracy and also provides faithful and reasonable explanations for recommendation. Extensive experiments show that EXP3RT outperforms existing methods on both rating prediction and candidate item reranking for top-k recommendation, while significantly enhancing the explainability of recommendation systems.
• 摘要：大型语言模型 (LLM) 的最新进展已在各种任务中表现出色，引起了人们对其在推荐系统中的应用的极大兴趣。然而，现有方法尚未充分利用 LLM 的潜力，通常受到输入信息有限的限制或未能充分利用其高级推理能力的限制。为了解决这些限制，我们引入了 EXP3RT，这是一种基于 LLM 的新型推荐器，旨在利用用户和项目评论中包含的丰富偏好信息。EXP3RT 基本上是通过从教师 LLM 中提炼而微调的，可按顺序执行三个关键任务：EXP3RT 首先从原始评论中提取和封装必要的主观偏好，根据特定标准对其进行汇总和总结，以创建用户和项目配置文件。然后，它通过考虑来自用户/项目配置文件和项目描述的主观和客观信息，生成详细的分步推理，然后进行预测评分，即推理增强的评分预测。EXP3RT 的这种个性化偏好推理提高了评分预测准确性，并为推荐提供了忠实合理的解释。大量实验表明，EXP3RT 在 top-k 推荐的评分预测和候选项目重新排序方面均优于现有方法，同时显著增强了推荐系统的可解释性。

Title: MovieSum: An Abstractive Summarization Dataset for Movie Screenplays

• Authors: Rohit Saxena, Frank Keller
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.06281
• Pdf URL: https://arxiv.org/pdf/2408.06281
• Copy Paste: [[2408.06281]] MovieSum: An Abstractive Summarization Dataset for Movie Screenplays(https://arxiv.org/abs/2408.06281)
• Keywords: language model
• Abstract: Movie screenplay summarization is challenging, as it requires an understanding of long input contexts and various elements unique to movies. Large language models have shown significant advancements in document summarization, but they often struggle with processing long input contexts. Furthermore, while television transcripts have received attention in recent studies, movie screenplay summarization remains underexplored. To stimulate research in this area, we present a new dataset, MovieSum, for abstractive summarization of movie screenplays. This dataset comprises 2200 movie screenplays accompanied by their Wikipedia plot summaries. We manually formatted the movie screenplays to represent their structural elements. Compared to existing datasets, MovieSum possesses several distinctive features: (1) It includes movie screenplays, which are longer than scripts of TV episodes. (2) It is twice the size of previous movie screenplay datasets. (3) It provides metadata with IMDb IDs to facilitate access to additional external knowledge. We also show the results of recently released large language models applied to summarization on our dataset to provide a detailed baseline.
• 摘要：电影剧本摘要具有挑战性，因为它需要理解长输入上下文和电影独有的各种元素。大型语言模型在文档摘要方面取得了重大进展，但它们往往难以处理长输入上下文。此外，虽然电视脚本在最近的研究中受到关注，但电影剧本摘要仍未得到充分探索。为了促进这一领域的研究，我们提出了一个新的数据集 MovieSum，用于对电影剧本进行抽象摘要。该数据集包含 2200 部电影剧本，并附有维基百科情节摘要。我们手动格式化电影剧本以表示其结构元素。与现有数据集相比，MovieSum 具有几个鲜明的特点：（1）它包含电影剧本，比电视剧集脚本更长。（2）它是以前电影剧本数据集的两倍。（3）它提供带有 IMDb ID 的元数据，以方便访问其他外部知识。我们还展示了最近发布的大型语言模型应用于我们数据集摘要的结果，以提供详细的基线。

Title: Synthetic Patient-Physician Dialogue Generation from Clinical Notes Using LLM

• Authors: Trisha Das, Dina Albassam, Jimeng Sun
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.06285
• Pdf URL: https://arxiv.org/pdf/2408.06285
• Copy Paste: [[2408.06285]] Synthetic Patient-Physician Dialogue Generation from Clinical Notes Using LLM(https://arxiv.org/abs/2408.06285)
• Keywords: gpt, llm, prompt
• Abstract: Medical dialogue systems (MDS) enhance patient-physician communication, improve healthcare accessibility, and reduce costs. However, acquiring suitable data to train these systems poses significant challenges. Privacy concerns prevent the use of real conversations, necessitating synthetic alternatives. Synthetic dialogue generation from publicly available clinical notes offers a promising solution to this issue, providing realistic data while safeguarding privacy. Our approach, SynDial, uses a single LLM iteratively with zero-shot prompting and a feedback loop to generate and refine high-quality synthetic dialogues. The feedback consists of weighted evaluation scores for similarity and extractiveness. The iterative process ensures dialogues meet predefined thresholds, achieving superior extractiveness as a result of the feedback loop. Additionally, evaluation shows that the generated dialogues excel in factuality metric compared to the baselines and has comparable diversity scores with GPT4.
• 摘要：医疗对话系统 (MDS) 可增强医患沟通、提高医疗保健的可及性并降低成本。然而，获取合适的数据来训练这些系统带来了重大挑战。隐私问题阻碍了真实对话的使用，因此需要合成替代方案。从公开的临床记录中生成合成对话为这个问题提供了一个有希望的解决方案，既能提供真实的数据，又能保护隐私。我们的方法 SynDial 使用单个 LLM 迭代地进行零样本提示和反馈循环来生成和改进高质量的合成对话。反馈包括相似性和可提取性的加权评估分数。迭代过程确保对话满足预定义的阈值，并通过反馈循环实现卓越的可提取性。此外，评估表明，与基线相比，生成的对话在事实性指标方面表现出色，并且多样性得分与 GPT4 相当。

Title: Long-Form Answers to Visual Questions from Blind and Low Vision People

• Authors: Mina Huh, Fangyuan Xu, Yi-Hao Peng, Chongyan Chen, Hansika Murugu, Danna Gurari, Eunsol Choi, Amy Pavel
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.06303
• Pdf URL: https://arxiv.org/pdf/2408.06303
• Copy Paste: [[2408.06303]] Long-Form Answers to Visual Questions from Blind and Low Vision People(https://arxiv.org/abs/2408.06303)
• Keywords: language model, hallucination, prompt
• Abstract: Vision language models can now generate long-form answers to questions about images - long-form visual question answers (LFVQA). We contribute VizWiz-LF, a dataset of long-form answers to visual questions posed by blind and low vision (BLV) users. VizWiz-LF contains 4.2k long-form answers to 600 visual questions, collected from human expert describers and six VQA models. We develop and annotate functional roles of sentences of LFVQA and demonstrate that long-form answers contain information beyond the question answer such as explanations and suggestions. We further conduct automatic and human evaluations with BLV and sighted people to evaluate long-form answers. BLV people perceive both human-written and generated long-form answers to be plausible, but generated answers often hallucinate incorrect visual details, especially for unanswerable visual questions (e.g., blurry or irrelevant images). To reduce hallucinations, we evaluate the ability of VQA models to abstain from answering unanswerable questions across multiple prompting strategies.
• 摘要：视觉语言模型现在可以生成有关图像的问题的长格式答案 - 长格式视觉问题答案 (LFVQA)。我们贡献了 VizWiz-LF，这是一个由盲人和低视力 (BLV) 用户提出的视觉问题的长格式答案数据集。VizWiz-LF 包含 600 个视觉问题的 4.2k 个长格式答案，这些答案来自人类专家描述者和六个 VQA 模型。我们开发并注释了 LFVQA 句子的功能角色，并证明长格式答案包含问题答案以外的信息，例如解释和建议。我们进一步对 BLV 和视力正常的人进行自动和人工评估，以评估长格式答案。BLV 人认为人类书写和生成的长格式答案都是合理的，但生成的答案通常会产生错误的视觉细节幻觉，尤其是对于无法回答的视觉问题（例如模糊或不相关的图像）。为了减少幻觉，我们评估了 VQA 模型在多种提示策略中避免回答无法回答的问题的能力。

Title: Animate, or Inanimate, That is the Question for Large Language Models

• Authors: Leonardo Ranaldi, Giulia Pucci, Fabio Massimo Zanzotto
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06332
• Pdf URL: https://arxiv.org/pdf/2408.06332
• Copy Paste: [[2408.06332]] Animate, or Inanimate, That is the Question for Large Language Models(https://arxiv.org/abs/2408.06332)
• Keywords: language model, llm, prompt
• Abstract: The cognitive essence of humans is deeply intertwined with the concept of animacy, which plays an essential role in shaping their memory, vision, and multi-layered language understanding. Although animacy appears in language via nuanced constraints on verbs and adjectives, it is also learned and refined through extralinguistic information. Similarly, we assume that the LLMs' limited abilities to understand natural language when processing animacy are motivated by the fact that these models are trained exclusively on text. Hence, the question this paper aims to answer arises: can LLMs, in their digital wisdom, process animacy in a similar way to what humans would do? We then propose a systematic analysis via prompting approaches. In particular, we probe different LLMs by prompting them using animate, inanimate, usual, and stranger contexts. Results reveal that, although LLMs have been trained predominantly on textual data, they exhibit human-like behavior when faced with typical animate and inanimate entities in alignment with earlier studies. Hence, LLMs can adapt to understand unconventional situations by recognizing oddities as animated without needing to interface with unspoken cognitive triggers humans rely on to break down animations.
• 摘要：人类的认知本质与生命概念紧密相连，生命概念在塑造人类记忆、视觉和多层次语言理解方面发挥着重要作用。尽管生命概念在语言中通过对动词和形容词的细微限制而出现，但它也是通过语言外的信息来学习和完善的。同样，我们假设 LLM 在处理生命概念时理解自然语言的能力有限，这是因为这些模型是专门在文本上训练的。因此，本文旨在回答的问题出现了：LLM 能否以其数字智慧以与人类类似的方式处理生命概念？然后，我们提出了通过提示方法进行系统分析。具体来说，我们通过使用有生命、无生命、常见和陌生的上下文来提示不同的 LLM，以探究它们。结果表明，尽管 LLM 主要在文本数据上进行训练，但它们在面对典型的有生命和无生命实体时表现出类似人类的行为，这与早期的研究一致。因此，法学硕士 (LLM) 可以通过将奇怪事物识别为动画来适应理解非常规情况，而无需与人类依赖的分解动画的不言而喻的认知触发因素进行交互。

Title: FastFiD: Improve Inference Efficiency of Open Domain Question Answering via Sentence Selection

• Authors: Yufei Huang, Xu Han, Maosong Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.06333
• Pdf URL: https://arxiv.org/pdf/2408.06333
• Copy Paste: [[2408.06333]] FastFiD: Improve Inference Efficiency of Open Domain Question Answering via Sentence Selection(https://arxiv.org/abs/2408.06333)
• Keywords: language model
• Abstract: Open Domain Question Answering (ODQA) has been advancing rapidly in recent times, driven by significant developments in dense passage retrieval and pretrained language models. Current models typically incorporate the FiD framework, which is composed by a neural retriever alongside an encoder-decoder neural reader. In the answer generation process, the retriever will retrieve numerous passages (around 100 for instance), each of which is then individually encoded by the encoder. Subsequently, the decoder makes predictions based on these encoded passages. Nevertheless, this framework can be relatively time-consuming, particularly due to the extensive length of the gathered passages. To address this, we introduce FastFiD in this paper, a novel approach that executes sentence selection on the encoded passages. This aids in retaining valuable sentences while reducing the context length required for generating answers. Experiments on three commonly used datasets (Natural Questions, TriviaQA and ASQA) demonstrate that our method can enhance the inference speed by 2.3X-5.7X, while simultaneously maintaining the model's performance. Moreover, an in-depth analysis of the model's attention reveals that the selected sentences indeed hold a substantial contribution towards the final answer. The codes are publicly available at this https URL.
• 摘要：近年来，开放域问答 (ODQA) 的发展十分迅速，这得益于密集段落检索和预训练语言模型的重大发展。当前的模型通常采用 FiD 框架，该框架由神经检索器和编码器-解码器神经阅读器组成。在答案生成过程中，检索器将检索大量段落（例如大约 100 个），然后编码器对每个段落进行单独编码。随后，解码器根据这些编码段落进行预测。然而，这个框架可能相对耗时，特别是由于收集的段落很长。为了解决这个问题，我们在本文中引入了 FastFiD，这是一种对编码段落执行句子选择的新方法。这有助于保留有价值的句子，同时减少生成答案所需的上下文长度。在三个常用数据集（Natural Questions、TriviaQA 和 ASQA）上的实验表明，我们的方法可以将推理速度提高 2.3 倍至 5.7 倍，同时保持模型的性能。此外，对模型注意力机制的深入分析表明，所选句子确实对最终答案做出了重大贡献。代码可在此 https URL 上公开获取。