2024-01-16

Title: GOODAT: Towards Test-time Graph Out-of-Distribution Detection

• Authors: Luzhi Wang, Dongxiao He, He Zhang, Yixin Liu, Wenjie Wang, Shirui Pan, Di Jin, Tat-Seng Chua
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06176
• Pdf URL: https://arxiv.org/pdf/2401.06176
• Copy Paste: [[2401.06176]] GOODAT: Towards Test-time Graph Out-of-Distribution Detection(https://arxiv.org/abs/2401.06176)
• Keywords: code
• Abstract: Graph neural networks (GNNs) have found widespread application in modeling graph data across diverse domains. While GNNs excel in scenarios where the testing data shares the distribution of their training counterparts (in distribution, ID), they often exhibit incorrect predictions when confronted with samples from an unfamiliar distribution (out-of-distribution, OOD). To identify and reject OOD samples with GNNs, recent studies have explored graph OOD detection, often focusing on training a specific model or modifying the data on top of a well-trained GNN. Despite their effectiveness, these methods come with heavy training resources and costs, as they need to optimize the GNN-based models on training data. Moreover, their reliance on modifying the original GNNs and accessing training data further restricts their universality. To this end, this paper introduces a method to detect Graph Out-of-Distribution At Test-time (namely GOODAT), a data-centric, unsupervised, and plug-and-play solution that operates independently of training data and modifications of GNN architecture. With a lightweight graph masker, GOODAT can learn informative subgraphs from test samples, enabling the capture of distinct graph patterns between OOD and ID samples. To optimize the graph masker, we meticulously design three unsupervised objective functions based on the graph information bottleneck principle, motivating the masker to capture compact yet informative subgraphs for OOD detection. Comprehensive evaluations confirm that our GOODAT method outperforms state-of-the-art benchmarks across a variety of real-world datasets. The code is available at Github: https://github.com/Ee1s/GOODAT
• 摘要：图神经网络（GNN）在跨不同领域的图数据建模中得到了广泛的应用。虽然 GNN 在测试数据共享其训练对应数据的分布（分布中，ID）的场景中表现出色，但当面对来自不熟悉的分布（分布外，OOD）的样本时，它们经常表现出错误的预测。为了使用 GNN 识别和拒绝 OOD 样本，最近的研究探索了图 OOD 检测，通常侧重于训练特定模型或修改训练有素的 GNN 之上的数据。尽管它们很有效，但这些方法需要大量的训练资源和成本，因为它们需要在训练数据上优化基于 GNN 的模型。此外，它们对修改原始 GNN 和访问训练数据的依赖进一步限制了它们的通用性。为此，本文介绍了一种在测试时检测图分布外的方法（即 GOODAT），这是一种以数据为中心、无监督、即插即用的解决方案，独立于训练数据和 GNN 的修改而运行建筑学。借助轻量级图形屏蔽器，GOODAT 可以从测试样本中学习信息丰富的子图，从而能够捕获 OOD 和 ID 样本之间的不同图形模式。为了优化图掩码器，我们根据图信息瓶颈原理精心设计了三个无监督目标函数，激励掩码器捕获紧凑但信息丰富的子图以进行 OOD 检测。综合评估证实，我们的 GOODAT 方法在各种真实数据集上的表现优于最先进的基准。代码可在 Github 上获取：https://github.com/Ee1s/GOODAT

Title: EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction

• Authors: Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Ren Kan, Dongsheng Li, Deqing Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06201
• Pdf URL: https://arxiv.org/pdf/2401.06201
• Copy Paste: [[2401.06201]] EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction(https://arxiv.org/abs/2401.06201)
• Keywords: language model, llm, code, agent
• Abstract: To address intricate real-world tasks, there has been a rising interest in tool utilization in applications of large language models (LLMs). To develop LLM-based agents, it usually requires LLMs to understand many tool functions from different tool documentation. But these documentations could be diverse, redundant or incomplete, which immensely affects the capability of LLMs in using tools. To solve this, we introduce EASYTOOL, a framework transforming diverse and lengthy tool documentation into a unified and concise tool instruction for easier tool usage. EasyTool purifies essential information from extensive tool documentation of different sources, and elaborates a unified interface (i.e., tool instruction) to offer standardized tool descriptions and functionalities for LLM-based agents. Extensive experiments on multiple different tasks demonstrate that EasyTool can significantly reduce token consumption and improve the performance of tool utilization in real-world scenarios. Our code will be available at \url{https://github.com/microsoft/JARVIS/} in the future.
• 摘要：为了解决复杂的现实任务，人们对大型语言模型 (LLM) 应用中的工具利用越来越感兴趣。要开发基于LLM的代理，通常需要LLM从不同的工具文档中了解许多工具功能。但这些文档可能多种多样、冗余或不完整，这极大地影响了法学硕士使用工具的能力。为了解决这个问题，我们引入了 EASYTOOL 框架，将多样化、冗长的工具文档转换为统一、简洁的工具指令，以方便工具的使用。 EasyTool 从不同来源的广泛工具文档中纯化了基本信息，并详细阐述了统一的界面（即工具指令），为基于 LLM 的代理提供标准化的工具描述和功能。对多个不同任务的大量实验表明，EasyTool 可以显着减少代币消耗并提高实际场景中工具使用的性能。我们的代码将来将在 \url{https://github.com/microsoft/JARVIS/} 中提供。

Title: An Exploratory Assessment of LLM's Potential Toward Flight Trajectory Reconstruction Analysis

• Authors: Qilei Zhang, John H. Mott
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2401.06204
• Pdf URL: https://arxiv.org/pdf/2401.06204
• Copy Paste: [[2401.06204]] An Exploratory Assessment of LLM's Potential Toward Flight Trajectory Reconstruction Analysis(https://arxiv.org/abs/2401.06204)
• Keywords: language model, llm, lora
• Abstract: Large Language Models (LLMs) hold transformative potential in aviation, particularly in reconstructing flight trajectories. This paper investigates this potential, grounded in the notion that LLMs excel at processing sequential data and deciphering complex data structures. Utilizing the LLaMA 2 model, a pre-trained open-source LLM, the study focuses on reconstructing flight trajectories using Automatic Dependent Surveillance-Broadcast (ADS-B) data with irregularities inherent in real-world scenarios. The findings demonstrate the model's proficiency in filtering noise and estimating both linear and curved flight trajectories. However, the analysis also reveals challenges in managing longer data sequences, which may be attributed to the token length limitations of LLM models. The study's insights underscore the promise of LLMs in flight trajectory reconstruction and open new avenues for their broader application across the aviation and transportation sectors.
• 摘要：大型语言模型 (LLM) 在航空领域具有变革潜力，特别是在重建飞行轨迹方面。本文研究了这种潜力，其基础是法学硕士擅长处理顺序数据和破译复杂的数据结构。该研究利用 LLaMA 2 模型（一种预先训练的开源法学硕士），重点关注使用自动相关监视广播 (ADS-B) 数据重建飞行轨迹，并具有现实场景中固有的不规则性。研究结果证明了该模型在过滤噪声和估计线性和弯曲飞行轨迹方面的能力。然而，分析还揭示了管理较长数据序列的挑战，这可能归因于 LLM 模型的令牌长度限制。该研究的见解强调了法学硕士在飞行轨迹重建方面的前景，并为其在航空和运输领域的更广泛应用开辟了新途径。

Title: LEGOBench: Leaderboard Generation Benchmark for Scientific Models

• Authors: Shruti Singh, Shoaib Alam, Mayank Singh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06233
• Pdf URL: https://arxiv.org/pdf/2401.06233
• Copy Paste: [[2401.06233]] LEGOBench: Leaderboard Generation Benchmark for Scientific Models(https://arxiv.org/abs/2401.06233)
• Keywords: language model, code
• Abstract: The ever-increasing volume of paper submissions makes it difficult to stay informed about the latest state-of-the-art research. To address this challenge, we introduce LEGOBench, a benchmark for evaluating systems that generate leaderboards. LEGOBench is curated from 22 years of preprint submission data in arXiv and more than 11,000 machine learning leaderboards in the PapersWithCode portal. We evaluate the performance of four traditional graph-based ranking variants and three recently proposed large language models. Our preliminary results show significant performance gaps in automatic leaderboard generation. The code is available on https://github.com/lingo-iitgn/LEGOBench and the dataset is hosted on https://osf.io/9v2py/?view_only=6f91b0b510df498ba01595f8f278f94c .
• 摘要：论文提交量的不断增加使得人们很难及时了解最新的最先进的研究成果。为了应对这一挑战，我们引入了 LEGOBench，这是一个用于评估生成排行榜的系统的基准。 LEGOBench 根据 arXiv 中 22 年的预印本提交数据以及 PapersWithCode 门户中的 11,000 多个机器学习排行榜进行整理。我们评估了四种传统的基于图的排名变体和三种最近提出的大型语言模型的性能。我们的初步结果显示自动排行榜生成方面存在显着的性能差距。代码可在 https://github.com/lingo-iitgn/LEGOBench 上获取，数据集托管在 https://osf.io/9v2py/?view_only=6f91b0b510df498ba01595f8f278f94c 上。

Title: A Universal Knowledge Model and Cognitive Architecture for Prototyping AGI

• Authors: Artem Sukhobokov, Evgeny Belousov, Danila Gromozdov, Anna Zenger, Ilya Popov
• Subjects: cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06256
• Pdf URL: https://arxiv.org/pdf/2401.06256
• Copy Paste: [[2401.06256]] A Universal Knowledge Model and Cognitive Architecture for Prototyping AGI(https://arxiv.org/abs/2401.06256)
• Keywords: rag, agent
• Abstract: The article identified 42 cognitive architectures for creating general artificial intelligence (AGI) and proposed a set of interrelated functional blocks that an agent approaching AGI in its capabilities should possess. Since the required set of blocks is not found in any of the existing architectures, the article proposes a new cognitive architecture for intelligent systems approaching AGI in their capabilities. As one of the key solutions within the framework of the architecture, a universal method of knowledge representation is proposed, which allows combining various non-formalized, partially and fully formalized methods of knowledge representation in a single knowledge base, such as texts in natural languages, images, audio and video recordings, graphs, algorithms, databases, neural networks, knowledge graphs, ontologies, frames, essence-property-relation models, production systems, predicate calculus models, conceptual models, and others. To combine and structure various fragments of knowledge, archigraph models are used, constructed as a development of annotated metagraphs. As components, the cognitive architecture being developed includes machine consciousness, machine subconsciousness, blocks of interaction with the external environment, a goal management block, an emotional control system, a block of social interaction, a block of reflection, an ethics block and a worldview block, a learning block, a monitoring block, blocks of statement and solving problems, self-organization and meta learning block.
• 摘要：该文章确定了用于创建通用人工智能 (AGI) 的 42 种认知架构，并提出了一组相互关联的功能块，这些功能块是接近 AGI 能力的代理应具备的。由于在任何现有架构中都找不到所需的块集，因此本文提出了一种新的认知架构，用于智能系统在其功能上接近 AGI。作为该架构框架内的关键解决方案之一，提出了一种通用的知识表示方法，该方法允许将各种非形式化、部分和完全形式化的知识表示方法组合在单个知识库中，例如自然语言的文本、图像、音视频记录、图表、算法、数据库、神经网络、知识图谱、本体、框架、本质属性关系模型、产生式系统、谓词演算模型、概念模型等。为了组合和构建各种知识片段，使用了体系图模型，将其构建为带注释的元图的发展。作为组件，正在开发的认知架构包括机器意识、机器潜意识、与外部环境交互的块、目标管理块、情绪控制系统、社交交互块、反思块、道德块和世界观块，学习块，监控块，陈述和解决问题块，自组织和元学习块。

Title: FedTabDiff: Federated Learning of Diffusion Probabilistic Models for Synthetic Mixed-Type Tabular Data Generation

• Authors: Timur Sattarov, Marco Schreyer, Damian Borth
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06263
• Pdf URL: https://arxiv.org/pdf/2401.06263
• Copy Paste: [[2401.06263]] FedTabDiff: Federated Learning of Diffusion Probabilistic Models for Synthetic Mixed-Type Tabular Data Generation(https://arxiv.org/abs/2401.06263)
• Keywords: rag
• Abstract: Realistic synthetic tabular data generation encounters significant challenges in preserving privacy, especially when dealing with sensitive information in domains like finance and healthcare. In this paper, we introduce \textit{Federated Tabular Diffusion} (FedTabDiff) for generating high-fidelity mixed-type tabular data without centralized access to the original tabular datasets. Leveraging the strengths of \textit{Denoising Diffusion Probabilistic Models} (DDPMs), our approach addresses the inherent complexities in tabular data, such as mixed attribute types and implicit relationships. More critically, FedTabDiff realizes a decentralized learning scheme that permits multiple entities to collaboratively train a generative model while respecting data privacy and locality. We extend DDPMs into the federated setting for tabular data generation, which includes a synchronous update scheme and weighted averaging for effective model aggregation. Experimental evaluations on real-world financial and medical datasets attest to the framework's capability to produce synthetic data that maintains high fidelity, utility, privacy, and coverage.
• 摘要：现实的合成表格数据生成在保护隐私方面遇到了重大挑战，特别是在处理金融和医疗保健等领域的敏感信息时。在本文中，我们引入 \textit{Federated Tabular Diffusion} (FedTabDiff)，用于生成高保真混合类型表格数据，而无需集中访问原始表格数据集。利用 \textit{去噪扩散概率模型} (DDPM) 的优势，我们的方法解决了表格数据中固有的复杂性，例如混合属性类型和隐式关系。更关键的是，FedTabDiff 实现了一种去中心化的学习方案，允许多个实体协作训练生成模型，同时尊重数据隐私和局部性。我们将 DDPM 扩展到表格数据生成的联合设置中，其中包括同步更新方案和用于有效模型聚合的加权平均。对现实世界金融和医疗数据集的实验评估证明了该框架能够生成保持高保真度、实用性、隐私性和覆盖范围的合成数据。

Title: Sampling and Uniqueness Sets in Graphon Signal Processing

• Authors: Alejandro Parada-Mayorga, Alejandro Ribeiro
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2401.06279
• Pdf URL: https://arxiv.org/pdf/2401.06279
• Copy Paste: [[2401.06279]] Sampling and Uniqueness Sets in Graphon Signal Processing(https://arxiv.org/abs/2401.06279)
• Keywords: rag
• Abstract: In this work, we study the properties of sampling sets on families of large graphs by leveraging the theory of graphons and graph limits. To this end, we extend to graphon signals the notion of removable and uniqueness sets, which was developed originally for the analysis of signals on graphs. We state the formal definition of a $\Lambda-$removable set and conditions under which a bandlimited graphon signal can be represented in a unique way when its samples are obtained from the complement of a given $\Lambda-$removable set in the graphon. By leveraging such results we show that graphon representations of graphs and graph signals can be used as a common framework to compare sampling sets between graphs with different numbers of nodes and edges, and different node labelings. Additionally, given a sequence of graphs that converges to a graphon, we show that the sequences of sampling sets whose graphon representation is identical in $[0,1]$ are convergent as well. We exploit the convergence results to provide an algorithm that obtains approximately close to optimal sampling sets. Performing a set of numerical experiments, we evaluate the quality of these sampling sets. Our results open the door for the efficient computation of optimal sampling sets in graphs of large size.
• 摘要：在这项工作中，我们利用图子和图极限理论研究大型图族上采样集的属性。为此，我们将可移除和唯一性集的概念扩展到图基信号，该概念最初是为了分析图上的信号而开发的。我们陈述了 $\Lambda-$removable 集的正式定义，以及当带限图子信号的样本是从图子中给定 $\Lambda-$removable 集的补集获得时，可以以独特的方式表示带限图子信号的条件。通过利用这些结果，我们表明图和图信号的图元表示可以用作通用框架来比较具有不同数量的节点和边以及不同节点标签的图之间的采样集。此外，给定收敛到图元的图序列，我们表明，其图元表示在 $[0,1]$ 中相同的采样集序列也是收敛的。我们利用收敛结果来提供一种算法，该算法可以获得近似接近最佳采样集的结果。通过进行一组数值实验，我们评估了这些采样集的质量。我们的结果为有效计算大尺寸图中的最佳采样集打开了大门。

Title: Demystifying Variational Diffusion Models

• Authors: Fabio De Sousa Ribeiro, Ben Glocker
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06281
• Pdf URL: https://arxiv.org/pdf/2401.06281
• Copy Paste: [[2401.06281]] Demystifying Variational Diffusion Models(https://arxiv.org/abs/2401.06281)
• Keywords: rag
• Abstract: Despite the growing popularity of diffusion models, gaining a deep understanding of the model class remains somewhat elusive for the uninitiated in non-equilibrium statistical physics. With that in mind, we present what we believe is a more straightforward introduction to diffusion models using directed graphical modelling and variational Bayesian principles, which imposes relatively fewer prerequisites on the average reader. Our exposition constitutes a comprehensive technical review spanning from foundational concepts like deep latent variable models to recent advances in continuous-time diffusion-based modelling, highlighting theoretical connections between model classes along the way. We provide additional mathematical insights that were omitted in the seminal works whenever possible to aid in understanding, while avoiding the introduction of new notation. We envision this article serving as a useful educational supplement for both researchers and practitioners in the area, and we welcome feedback and contributions from the community at https://github.com/biomedia-mira/demystifying-diffusion.
• 摘要：尽管扩散模型越来越受欢迎，但对于非平衡统计物理学的新手来说，深入了解模型类别仍然有些困难。考虑到这一点，我们使用定向图形建模和变分贝叶斯原理对扩散模型进行了更直接的介绍，这对普通读者提出了相对较少的先决条件。我们的阐述构成了全面的技术回顾，涵盖从深层潜变量模型等基本概念到基于连续时间扩散的建模的最新进展，强调了模型类之间的理论联系。我们尽可能提供在开创性著作中省略的额外数学见解，以帮助理解，同时避免引入新的符号。我们希望这篇文章成为该领域的研究人员和从业者的有用的教育补充，我们欢迎来自社区的反馈和贡献：https://github.com/biomedia-mira/demystifying-diffusion。

Title: MultiSlot ReRanker: A Generic Model-based Re-Ranking Framework in Recommendation Systems

• Authors: Qiang Charles Xiao, Ajith Muralidharan, Birjodh Tiwana, Johnson Jia, Fedor Borisyuk, Aman Gupta, Dawn Woodard
• Subjects: cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2401.06293
• Pdf URL: https://arxiv.org/pdf/2401.06293
• Copy Paste: [[2401.06293]] MultiSlot ReRanker: A Generic Model-based Re-Ranking Framework in Recommendation Systems(https://arxiv.org/abs/2401.06293)
• Keywords: rag
• Abstract: In this paper, we propose a generic model-based re-ranking framework, MultiSlot ReRanker, which simultaneously optimizes relevance, diversity, and freshness. Specifically, our Sequential Greedy Algorithm (SGA) is efficient enough (linear time complexity) for large-scale production recommendation engines. It achieved a lift of $+6\%$ to $ +10\%$ offline Area Under the receiver operating characteristic Curve (AUC) which is mainly due to explicitly modeling mutual influences among items of a list, and leveraging the second pass ranking scores of multiple objectives. In addition, we have generalized the offline replay theory to multi-slot re-ranking scenarios, with trade-offs among multiple objectives. The offline replay results can be further improved by Pareto Optimality. Moreover, we've built a multi-slot re-ranking simulator based on OpenAI Gym integrated with the Ray framework. It can be easily configured for different assumptions to quickly benchmark both reinforcement learning and supervised learning algorithms.
• 摘要：在本文中，我们提出了一种基于通用模型的重排序框架 MultiSlot ReRanker，它同时优化了相关性、多样性和新鲜度。具体来说，我们的顺序贪婪算法（SGA）对于大规模生产推荐引擎来说足够高效（线性时间复杂度）。它在接收者操作特征曲线 (AUC) 下的离线区域实现了 $+6\%$ 到 $ +10\%$ 的提升，这主要是由于明确地建模了列表中项目之间的相互影响，并利用了第二遍排名多个目标的分数。此外，我们将离线重放理论推广到多槽重新排名场景，并在多个目标之间进行权衡。离线重放结果可以通过帕累托最优进一步提高。此外，我们还基于 OpenAI Gym 与 Ray 框架集成构建了多槽重排模拟器。它可以轻松地针对不同的假设进行配置，以快速对强化学习和监督学习算法进行基准测试。

Title: Misconfidence-based Demonstration Selection for LLM In-Context Learning

• Authors: Shangqing Xu, Chao Zhang (Georgia Institute of Technology)
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06301
• Pdf URL: https://arxiv.org/pdf/2401.06301
• Copy Paste: [[2401.06301]] Misconfidence-based Demonstration Selection for LLM In-Context Learning(https://arxiv.org/abs/2401.06301)
• Keywords: language model, llm, rag
• Abstract: In-context learning with large language models (LLMs) excels at adapting to various tasks rapidly. However, its success hinges on carefully selecting demonstrations, which remains an obstacle in practice. Current approaches to this problem either rely on hard-to-acquire external supervision or require frequent interactions with LLMs, resulting in high costs. We propose a new method called In-Context Reflection (ICR) to overcome these challenges. ICR strategically selects demonstrations to reduce the discrepancy between the LLM's outputs and the actual input-output mappings. Specifically, ICR starts with a random set of initial demonstrations, then iteratively refines it. In each step, it analyzes a pool of candidate examples and identifies the ones most likely to challenge the LLM's current understanding, measured by a new metric called misconfidence. These most confusing examples are then selected to replace the less informative demonstrations in the current set. Our comprehensive evaluation across five diverse datasets encompassing 13 subtasks shows the efficacy of ICR. Compared to existing methods, ICR achieves an average performance boost of 4%, while demonstrating remarkable cross-task generalization capabilities.
• 摘要：使用大型语言模型 (LLM) 的情境学习擅长快速适应各种任务。然而，其成功取决于仔细选择示范，这在实践中仍然是一个障碍。目前解决这一问题的方法要么依赖于难以获得的外部监督，要么需要与法学硕士频繁互动，导致成本高昂。我们提出了一种称为上下文反射（ICR）的新方法来克服这些挑战。 ICR 战略性地选择示范，以减少法学硕士的输出与实际输入输出映射之间的差异。具体来说，ICR 从一组随机的初始演示开始，然后迭代地对其进行完善。在每个步骤中，它都会分析一组候选示例，并确定最有可能挑战法学硕士当前理解的示例，并通过称为“不自信”的新指标来衡量。然后选择这些最令人困惑的示例来替换当前集中信息量较少的演示。我们对包含 13 个子任务的 5 个不同数据集进行的综合评估显示了 ICR 的有效性。与现有方法相比，ICR 的平均性能提升了 4%，同时展示了卓越的跨任务泛化能力。

Title: Multi-Task Learning for Front-End Text Processing in TTS

• Authors: Wonjune Kang, Yun Wang, Shun Zhang, Arthur Hinsvark, Qing He
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06321
• Pdf URL: https://arxiv.org/pdf/2401.06321
• Copy Paste: [[2401.06321]] Multi-Task Learning for Front-End Text Processing in TTS(https://arxiv.org/abs/2401.06321)
• Keywords: language model
• Abstract: We propose a multi-task learning (MTL) model for jointly performing three tasks that are commonly solved in a text-to-speech (TTS) front-end: text normalization (TN), part-of-speech (POS) tagging, and homograph disambiguation (HD). Our framework utilizes a tree-like structure with a trunk that learns shared representations, followed by separate task-specific heads. We further incorporate a pre-trained language model to utilize its built-in lexical and contextual knowledge, and study how to best use its embeddings so as to most effectively benefit our multi-task model. Through task-wise ablations, we show that our full model trained on all three tasks achieves the strongest overall performance compared to models trained on individual or sub-combinations of tasks, confirming the advantages of our MTL framework. Finally, we introduce a new HD dataset containing a balanced number of sentences in diverse contexts for a variety of homographs and their pronunciations. We demonstrate that incorporating this dataset into training significantly improves HD performance over only using a commonly used, but imbalanced, pre-existing dataset.
• 摘要：我们提出了一种多任务学习（MTL）模型，用于联合执行文本转语音（TTS）前端中通常解决的三个任务：文本规范化（TN）、词性（POS）标记、和同形异义词消歧（HD）。我们的框架采用树状结构，其主干学习共享表示，后面是单独的特定于任务的头。我们进一步结合预先训练的语言模型来利用其内置的词汇和上下文知识，并研究如何最好地利用其嵌入，以便最有效地使我们的多任务模型受益。通过按任务进行消融，我们表明，与在单个任务或子任务组合上训练的模型相比，在所有三个任务上训练的完整模型实现了最强的整体性能，证实了我们的 MTL 框架的优势。最后，我们引入了一个新的高清数据集，其中包含不同上下文中各种同形异义词及其发音的平衡数量的句子。我们证明，与仅使用常用但不平衡的预先存在的数据集相比，将此数据集纳入训练可显着提高 HD 性能。

Title: How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs

• Authors: Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, Weiyan Shi
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06373
• Pdf URL: https://arxiv.org/pdf/2401.06373
• Copy Paste: [[2401.06373]] How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs(https://arxiv.org/abs/2401.06373)
• Keywords: language model, gpt, llm, prompt, chat
• Abstract: Most traditional AI safety research has approached AI models as machines and centered on algorithm-focused attacks developed by security experts. As large language models (LLMs) become increasingly common and competent, non-expert users can also impose risks during daily interactions. This paper introduces a new perspective to jailbreak LLMs as human-like communicators, to explore this overlooked intersection between everyday language interaction and AI safety. Specifically, we study how to persuade LLMs to jailbreak them. First, we propose a persuasion taxonomy derived from decades of social science research. Then, we apply the taxonomy to automatically generate interpretable persuasive adversarial prompts (PAP) to jailbreak LLMs. Results show that persuasion significantly increases the jailbreak performance across all risk categories: PAP consistently achieves an attack success rate of over $92\%$ on Llama 2-7b Chat, GPT-3.5, and GPT-4 in $10$ trials, surpassing recent algorithm-focused attacks. On the defense side, we explore various mechanisms against PAP and, found a significant gap in existing defenses, and advocate for more fundamental mitigation for highly interactive LLMs
• 摘要：大多数传统的人工智能安全研究都将人工智能模型视为机器，并以安全专家开发的以算法为中心的攻击为中心。随着大型语言模型 (LLM) 变得越来越普遍和强大，非专家用户也可能在日常交互中带来风险。本文介绍了越狱法学硕士作为类人沟通者的新视角，以探索日常语言交互和人工智能安全之间被忽视的交叉点。具体来说，我们研究如何说服法学硕士越狱。首先，我们提出了一个源自数十年社会科学研究的说服分类法。然后，我们应用分类法自动生成可解释的说服性对抗提示（PAP）来越狱法学硕士。结果表明，说服显着提高了所有风险类别的越狱性能：在 10 美元的试验中，PAP 在 Llama 2-7b Chat、GPT-3.5 和 GPT-4 上始终实现了超过 92\%$ 的攻击成功率，超过了最新的算法-集中攻击。在防御方面，我们探索了针对 PAP 的各种机制，发现了现有防御中的重大差距，并主张为高度互动的 LLM 提供更根本的缓解措施

Title: Cognitive BPM as an Equalizer: Improving Access and Efficiency for Employees with (and without) Cognitive Disabilities

• Authors: Gordon Banks, Gates Bierhuizen, Katherine McCrum, Ellen Wengert
• Subjects: cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06375
• Pdf URL: https://arxiv.org/pdf/2401.06375
• Copy Paste: [[2401.06375]] Cognitive BPM as an Equalizer: Improving Access and Efficiency for Employees with (and without) Cognitive Disabilities(https://arxiv.org/abs/2401.06375)
• Keywords: gpt
• Abstract: We examine ProcessGPT, an AI model designed to automate, augment, and improve business processes, to study the challenges of managing business processes within the cognitive limitations of the human workforce, particularly individuals with cognitive disabilities. ProcessGPT provides a blueprint for designing efficient business processes that take into account human cognitive limitations. By viewing this through the lens of cognitive disabilities, we show that ProcessGPT improves process usability for individuals with and without cognitive disabilities. We also demonstrate that organizations implementing ProcessGPT-like capabilities will realize increased productivity, morale, and inclusion.
• 摘要：我们研究了 ProcessGPT，这是一种旨在自动化、增强和改进业务流程的人工智能模型，以研究在人类劳动力（尤其是有认知障碍的个人）的认知限制内管理业务流程的挑战。 ProcessGPT 为设计高效的业务流程提供了蓝图，同时考虑了人类认知的局限性。通过从认知障碍的角度来看这一点，我们表明 ProcessGPT 可以提高有或没有认知障碍的个人的流程可用性。我们还证明，实施类似 ProcessGPT 功能的组织将提高生产力、士气和包容性。

Title: Vehicle: Bridging the Embedding Gap in the Verification of Neuro-Symbolic Programs

• Authors: Matthew L. Daggitt, Wen Kokke, Robert Atkey, Natalia Slusarz, Luca Arnaboldi, Ekaterina Komendantskaya
• Subjects: cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06379
• Pdf URL: https://arxiv.org/pdf/2401.06379
• Copy Paste: [[2401.06379]] Vehicle: Bridging the Embedding Gap in the Verification of Neuro-Symbolic Programs(https://arxiv.org/abs/2401.06379)
• Keywords: code
• Abstract: Neuro-symbolic programs -- programs containing both machine learning components and traditional symbolic code -- are becoming increasingly widespread. However, we believe that there is still a lack of a general methodology for verifying these programs whose correctness depends on the behaviour of the machine learning components. In this paper, we identify the ``embedding gap'' -- the lack of techniques for linking semantically-meaningful ``problem-space'' properties to equivalent ``embedding-space'' properties -- as one of the key issues, and describe Vehicle, a tool designed to facilitate the end-to-end verification of neural-symbolic programs in a modular fashion. Vehicle provides a convenient language for specifying ``problem-space'' properties of neural networks and declaring their relationship to the ``embedding-space", and a powerful compiler that automates interpretation of these properties in the language of a chosen machine-learning training environment, neural network verifier, and interactive theorem prover. We demonstrate Vehicle's utility by using it to formally verify the safety of a simple autonomous car equipped with a neural network controller.
• 摘要：神经符号程序——包含机器学习组件和传统符号代码的程序——正变得越来越普遍。然而，我们认为仍然缺乏验证这些程序的通用方法，其正确性取决于机器学习组件的行为。在本文中，我们将“嵌入差距”（缺乏将语义上有意义的“问题空间”属性链接到等效“嵌入空间”属性的技术）视为关键问题之一，并描述了 Vehicle，一种旨在以模块化方式促进神经符号程序端到端验证的工具。 Vehicle 提供了一种方便的语言，用于指定神经网络的“问题空间”属性并声明它们与“嵌入空间”的关系，以及一个强大的编译器，可以用所选机器学习的语言自动解释这些属性训练环境、神经网络验证器和交互式定理证明器。我们通过使用它来正式验证配备神经网络控制器的简单自动驾驶汽车的安全性来展示车辆的实用性。

Title: Generalizing Visual Question Answering from Synthetic to Human-Written Questions via a Chain of QA with a Large Language Model

• Authors: Taehee Kim, Yeongjae Cho, Heejun Shin, Yohan Jo, Dongmyung Shin
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2401.06400
• Pdf URL: https://arxiv.org/pdf/2401.06400
• Copy Paste: [[2401.06400]] Generalizing Visual Question Answering from Synthetic to Human-Written Questions via a Chain of QA with a Large Language Model(https://arxiv.org/abs/2401.06400)
• Keywords: language model
• Abstract: Visual question answering (VQA) is a task where an image is given, and a series of questions are asked about the image. To build an efficient VQA algorithm, a large amount of QA data is required which is very expensive. Generating synthetic QA pairs based on templates is a practical way to obtain data. However, VQA models trained on those data do not perform well on complex, human-written questions. To address this issue, we propose a new method called {\it chain of QA for human-written questions} (CoQAH). CoQAH utilizes a sequence of QA interactions between a large language model and a VQA model trained on synthetic data to reason and derive logical answers for human-written questions. We tested the effectiveness of CoQAH on two types of human-written VQA datasets for 3D-rendered and chest X-ray images and found that it achieved state-of-the-art accuracy in both types of data. Notably, CoQAH outperformed general vision-language models, VQA models, and medical foundation models with no finetuning.
• 摘要：视觉问答（VQA）是一项给定图像并针对该图像提出一系列问题的任务。为了构建高效的VQA算法，需要大量的QA数据，这是非常昂贵的。基于模板生成合成问答对是一种获取数据的实用方法。然而，基于这些数据训练的 VQA 模型在复杂的、人工编写的问题上表现不佳。为了解决这个问题，我们提出了一种新方法，称为{\it chain of QA for human-writing questions} (CoQAH)。 CoQAH 利用大型语言模型和在合成数据上训练的 VQA 模型之间的一系列 QA 交互来推理并得出人类编写的问题的逻辑答案。我们在两种类型的人工编写的 VQA 数据集（针对 3D 渲染和胸部 X 射线图像）上测试了 CoQAH 的有效性，发现它在两种类型的数据中都实现了最先进的准确性。值得注意的是，CoQAH 的性能优于一般视觉语言模型、VQA 模型和无需微调的医学基础模型。

Title: AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters

• Authors: Li Lucy, Suchin Gururangan, Luca Soldaini, Emma Strubell, David Bamman, Lauren Klein, Jesse Dodge
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06408
• Pdf URL: https://arxiv.org/pdf/2401.06408
• Copy Paste: [[2401.06408]] AboutMe: Using Self-Descriptions in Webpages to Document the Effects of English Pretraining Data Filters(https://arxiv.org/abs/2401.06408)
• Keywords: language model, llm, rag
• Abstract: Large language models' (LLMs) abilities are drawn from their pretraining data, and model development begins with data curation. However, decisions around what data is retained or removed during this initial stage is under-scrutinized. In our work, we ground web text, which is a popular pretraining data source, to its social and geographic contexts. We create a new dataset of 10.3 million self-descriptions of website creators, and extract information about who they are and where they are from: their topical interests, social roles, and geographic affiliations. Then, we conduct the first study investigating how ten "quality" and English language identification (langID) filters affect webpages that vary along these social dimensions. Our experiments illuminate a range of implicit preferences in data curation: we show that some quality classifiers act like topical domain filters, and langID can overlook English content from some regions of the world. Overall, we hope that our work will encourage a new line of research on pretraining data curation practices and its social implications.
• 摘要：大型语言模型 (LLM) 的能力来自其预训练数据，模型开发从数据管理开始。然而，关于在这个初始阶段保留或删除哪些数据的决策尚未得到充分审查。在我们的工作中，我们将网络文本（一种流行的预训练数据源）与其社会和地理背景联系起来。我们创建了一个包含 1030 万个网站创建者自我描述的新数据集，并提取有关他们是谁、来自哪里的信息：他们的主题兴趣、社会角色和地理归属。然后，我们进行了第一项研究，调查十个“质量”和英语语言识别 (langID) 过滤器如何影响沿这些社交维度变化的网页。我们的实验阐明了数据管理中的一系列隐式偏好：我们表明一些质量分类器就像主题域过滤器一样，而 langID 可以忽略来自世界某些地区的英语内容。总的来说，我们希望我们的工作能够鼓励对预训练数据管理实践及其社会影响进行新的研究。

Title: Mission: Impossible Language Models

• Authors: Julie Kallini, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, Christopher Potts
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06416
• Pdf URL: https://arxiv.org/pdf/2401.06416
• Copy Paste: [[2401.06416]] Mission: Impossible Language Models(https://arxiv.org/abs/2401.06416)
• Keywords: language model, gpt, llm
• Abstract: Chomsky and others have very directly claimed that large language models (LLMs) are equally capable of learning languages that are possible and impossible for humans to learn. However, there is very little published experimental evidence to support such a claim. Here, we develop a set of synthetic impossible languages of differing complexity, each designed by systematically altering English data with unnatural word orders and grammar rules. These languages lie on an impossibility continuum: at one end are languages that are inherently impossible, such as random and irreversible shuffles of English words, and on the other, languages that may not be intuitively impossible but are often considered so in linguistics, particularly those with rules based on counting word positions. We report on a wide range of evaluations to assess the capacity of GPT-2 small models to learn these uncontroversially impossible languages, and crucially, we perform these assessments at various stages throughout training to compare the learning process for each language. Our core finding is that GPT-2 struggles to learn impossible languages when compared to English as a control, challenging the core claim. More importantly, we hope our approach opens up a productive line of inquiry in which different LLM architectures are tested on a variety of impossible languages in an effort to learn more about how LLMs can be used as tools for these cognitive and typological investigations.
• 摘要：乔姆斯基和其他人非常直接地声称，大型语言模型（LLM）同样能够学习人类可能学习和不可能学习的语言。然而，很少有公开的实验证据支持这种说法。在这里，我们开发了一组不同复杂度的合成不可能语言，每种语言都是通过使用不自然的词序和语法规则系统地改变英语数据而设计的。这些语言处于不可能的连续体上：一方面是本质上不可能的语言，例如英语单词的随机和不可逆的洗牌，另一方面，直觉上可能并非不可能的语言，但在语言学中通常被认为是不可能的，特别是那些基于计算单词位置的规则。我们报告了广泛的评估，以评估 GPT-2 小模型学习这些毫无争议的不可能语言的能力，最重要的是，我们在整个训练的不同阶段进行这些评估，以比较每种语言的学习过程。我们的核心发现是，与英语作为对照相比，GPT-2 很难学习不可能的语言，这挑战了核心主张。更重要的是，我们希望我们的方法能够开辟一条富有成效的探究路线，其中不同的法学硕士架构在各种不可能的语言上进行测试，以努力了解更多关于如何将法学硕士用作这些认知和类型学调查的工具。

Title: From Automation to Augmentation: Large Language Models Elevating Essay Scoring Landscape

• Authors: Changrong Xiao, Wenxing Ma, Sean Xin Xu, Kunpeng Zhang, Yufang Wang, Qi Fu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06431
• Pdf URL: https://arxiv.org/pdf/2401.06431
• Copy Paste: [[2401.06431]] From Automation to Augmentation: Large Language Models Elevating Essay Scoring Landscape(https://arxiv.org/abs/2401.06431)
• Keywords: language model, gpt, llm
• Abstract: Receiving immediate and personalized feedback is crucial for second-language learners, and Automated Essay Scoring (AES) systems are a vital resource when human instructors are unavailable. This study investigates the effectiveness of Large Language Models (LLMs), specifically GPT-4 and fine-tuned GPT-3.5, as tools for AES. Our comprehensive set of experiments, conducted on both public and private datasets, highlights the remarkable advantages of LLM-based AES systems. They include superior accuracy, consistency, generalizability, and interpretability, with fine-tuned GPT-3.5 surpassing traditional grading models. Additionally, we undertake LLM-assisted human evaluation experiments involving both novice and expert graders. One pivotal discovery is that LLMs not only automate the grading process but also enhance the performance of human graders. Novice graders when provided with feedback generated by LLMs, achieve a level of accuracy on par with experts, while experts become more efficient and maintain greater consistency in their assessments. These results underscore the potential of LLMs in educational technology, paving the way for effective collaboration between humans and AI, ultimately leading to transformative learning experiences through AI-generated feedback.
• 摘要：接收即时和个性化的反馈对于第二语言学习者来说至关重要，而自动作文评分 (AES) 系统是当人类教师无法使用时的重要资源。本研究调查了大型语言模型 (LLM)（特别是 GPT-4 和微调 GPT-3.5）作为 AES 工具的有效性。我们在公共和私人数据集上进行的一系列综合实验凸显了基于 LLM 的 AES 系统的显着优势。它们包括卓越的准确性、一致性、普遍性和可解释性，经过微调的 GPT-3.5 超越了传统的评分模型。此外，我们还开展法学硕士辅助的人类评估实验，涉及新手和专家评分者。一项关键的发现是，法学硕士不仅可以自动化评分过程，还可以提高人类评分者的表现。新手评分者在获得法学硕士生成的反馈后，可以达到与专家相当的准确性水平，而专家则变得更加高效，并在评估中保持更大的一致性。这些结果强调了法学硕士在教育技术方面的潜力，为人类与人工智能之间的有效合作铺平了道路，最终通过人工智能生成的反馈带来变革性的学习体验。

Title: Heterogeneous Low-Rank Approximation for Federated Fine-tuning of On-Device Foundation Models

• Authors: Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, Gauri Joshi
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2401.06432
• Pdf URL: https://arxiv.org/pdf/2401.06432
• Copy Paste: [[2401.06432]] Heterogeneous Low-Rank Approximation for Federated Fine-tuning of On-Device Foundation Models(https://arxiv.org/abs/2401.06432)
• Keywords: lora
• Abstract: Large foundation models (FMs) adapt surprisingly well to specific domains or tasks with fine-tuning. Federated learning (FL) further enables private FM fine-tuning using the local data on devices. However, the standard FMs' large size poses challenges for resource-constrained and heterogeneous devices. To address this, we consider FMs with reduced parameter sizes, referred to as on-device FMs (ODFMs). While ODFMs allow on-device inference, computational constraints still hinder efficient federated fine-tuning. We propose a parameter-efficient federated fine-tuning method for ODFMs using heterogeneous low-rank approximations (LoRAs) that addresses system and data heterogeneity. We show that homogeneous LoRA ranks face a trade-off between overfitting and slow convergence, and propose HetLoRA, which employs heterogeneous ranks across clients and eliminates the shortcomings of homogeneous HetLoRA. By applying rank self-pruning locally and sparsity-weighted aggregation at the server, we combine the advantages of high and low-rank LoRAs, which achieves improved convergence speed and final performance compared to homogeneous LoRA. Furthermore, it offers enhanced computation efficiency compared to full fine-tuning, making it suitable for heterogeneous devices while preserving data privacy.
• 摘要：大型基础模型 (FM) 通过微调可以很好地适应特定领域或任务。联邦学习 (FL) 进一步支持使用设备上的本地数据进行私有 FM 微调。然而，标准FM的大尺寸给资源受限和异构设备带来了挑战。为了解决这个问题，我们考虑减小参数大小的 FM，称为设备上 FM (ODFM)。虽然 ODFM 允许设备上推理，但计算限制仍然阻碍有效的联合微调。我们提出了一种使用异构低秩近似 (LoRA) 的 ODFM 参数高效联合微调方法，可解决系统和数据的异构性。我们证明同质 LoRA 排名面临过度拟合和收敛缓慢之间的权衡，并提出 HetLoRA，它在客户端之间采用异构排名并消除同质 HetLoRA 的缺点。通过在本地应用秩自剪枝和在服务器上的稀疏加权聚合，我们结合了高秩和低秩 LoRA 的优点，与同构 LoRA 相比，实现了更高的收敛速度和最终性能。此外，与完全微调相比，它提供了更高的计算效率，使其适用于异构设备，同时保护数据隐私。

Title: Improving Graph Convolutional Networks with Transformer Layer in social-based items recommendation

• Authors: Thi Linh Hoang, Tuan Dung Pham, Viet Cuong Ta
• Subjects: cs.LG, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2401.06436
• Pdf URL: https://arxiv.org/pdf/2401.06436
• Copy Paste: [[2401.06436]] Improving Graph Convolutional Networks with Transformer Layer in social-based items recommendation(https://arxiv.org/abs/2401.06436)
• Keywords: code
• Abstract: In this work, we have proposed an approach for improving the GCN for predicting ratings in social networks. Our model is expanded from the standard model with several layers of transformer architecture. The main focus of the paper is on the encoder architecture for node embedding in the network. Using the embedding layer from the graph-based convolution layer, the attention mechanism could rearrange the feature space to get a more efficient embedding for the downstream task. The experiments showed that our proposed architecture achieves better performance than GCN on the traditional link prediction task.
• 摘要：在这项工作中，我们提出了一种改进 GCN 来预测社交网络评分的方法。我们的模型是从标准模型扩展而来的，具有多层变压器架构。本文的主要重点是用于网络中嵌入节点的编码器架构。使用基于图的卷积层的嵌入层，注意力机制可以重新排列特征空间，以获得下游任务更有效的嵌入。实验表明，我们提出的架构在传统的链接预测任务上取得了比 GCN 更好的性能。

Title: BOK-VQA: Bilingual Outside Knowledge-based Visual Question Answering via Graph Representation Pretraining

• Authors: Minjun Kim, Seungwoo Song, Youhan Lee, Haneol Jang, Kyungtae Lim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06443
• Pdf URL: https://arxiv.org/pdf/2401.06443
• Copy Paste: [[2401.06443]] BOK-VQA: Bilingual Outside Knowledge-based Visual Question Answering via Graph Representation Pretraining(https://arxiv.org/abs/2401.06443)
• Keywords: gpt
• Abstract: The current research direction in generative models, such as the recently developed GPT4, aims to find relevant knowledge information for multimodal and multilingual inputs to provide answers. Under these research circumstances, the demand for multilingual evaluation of visual question answering (VQA) tasks, a representative task of multimodal systems, has increased. Accordingly, we propose a bilingual outside-knowledge VQA (BOK-VQA) dataset in this study that can be extended to multilingualism. The proposed data include 17K images, 17K question-answer pairs for both Korean and English and 280K instances of knowledge information related to question-answer content. We also present a framework that can effectively inject knowledge information into a VQA system by pretraining the knowledge information of BOK-VQA data in the form of graph embeddings. Finally, through in-depth analysis, we demonstrated the actual effect of the knowledge information contained in the constructed training data on VQA.
• 摘要：目前生成模型的研究方向，例如最近开发的GPT4，旨在为多模态、多语言输入寻找相关知识信息，以提供答案。在这些研究背景下，对多模态系统的代表性任务视觉问答（VQA）任务的多语言评估的需求不断增加。因此，我们在本研究中提出了一个可以扩展到多语言的双语外部知识 VQA（BOK-VQA）数据集。所提出的数据包括 17K 图像、17K 韩语和英语问答对以及 280K 与问答内容相关的知识信息实例。我们还提出了一个框架，可以通过以图嵌入的形式预训练 BOK-VQA 数据的知识信息，有效地将知识信息注入到 VQA 系统中。最后，通过深入分析，论证了构建的训练数据中包含的知识信息对VQA的实际效果。

Title: Sanity Checks Revisited: An Exploration to Repair the Model Parameter Randomisation Test

• Authors: Anna Hedström, Leander Weber, Sebastian Lapuschkin, Marina MC Höhne
• Subjects: cs.AI, cs.LG, stat.ME
• Abstract URL: https://arxiv.org/abs/2401.06465
• Pdf URL: https://arxiv.org/pdf/2401.06465
• Copy Paste: [[2401.06465]] Sanity Checks Revisited: An Exploration to Repair the Model Parameter Randomisation Test(https://arxiv.org/abs/2401.06465)
• Keywords: lora
• Abstract: The Model Parameter Randomisation Test (MPRT) is widely acknowledged in the eXplainable Artificial Intelligence (XAI) community for its well-motivated evaluative principle: that the explanation function should be sensitive to changes in the parameters of the model function. However, recent works have identified several methodological caveats for the empirical interpretation of MPRT. To address these caveats, we introduce two adaptations to the original MPRT -- Smooth MPRT and Efficient MPRT, where the former minimises the impact that noise has on the evaluation results through sampling and the latter circumvents the need for biased similarity measurements by re-interpreting the test through the explanation's rise in complexity, after full parameter randomisation. Our experimental results demonstrate that these proposed variants lead to improved metric reliability, thus enabling a more trustworthy application of XAI methods.
• 摘要：模型参数随机化测试（MPRT）因其动机良好的评估原则而在可解释人工智能（XAI）社区中得到广泛认可：解释函数应该对模型函数参数的变化敏感。然而，最近的研究发现了 MPRT 实证解释的一些方法论警告。为了解决这些问题，我们对原始 MPRT 进行了两种改进——平滑 MPRT 和高效 MPRT，前者通过采样最大限度地减少噪声对评估结果的影响，后者通过重新解释来避免有偏差的相似性测量的需要在完全参数随机化之后，通过解释复杂性的增加进行测试。我们的实验结果表明，这些提出的变体可以提高度量可靠性，从而使 XAI 方法的应用更加值得信赖。

Title: PersianMind: A Cross-Lingual Persian-English Large Language Model

• Authors: Pedram Rostami, Ali Salemi, Mohammad Javad Dousti
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06466
• Pdf URL: https://arxiv.org/pdf/2401.06466
• Copy Paste: [[2401.06466]] PersianMind: A Cross-Lingual Persian-English Large Language Model(https://arxiv.org/abs/2401.06466)
• Keywords: language model, gpt
• Abstract: Large language models demonstrate remarkable proficiency in various linguistic tasks and have extensive knowledge across various domains. Although they perform best in English, their ability in other languages is notable too. In contrast, open-source models, such as LLaMa, are primarily trained on English datasets, resulting in poor performance in non-English languages. In this paper, we introduce PersianMind, an open-source bilingual large language model which demonstrates comparable performance to closed-source GPT-3.5-turbo in the Persian language. By expanding LLaMa2's vocabulary with 10,000 Persian tokens and training it on a dataset comprising nearly 2 billion Persian tokens, we show that our approach preserves the model's English knowledge and employs transfer learning to excel at transferring task knowledge from one language to another.
• 摘要：大型语言模型在各种语言任务中表现出卓越的熟练程度，并在各个领域拥有广泛的知识。尽管他们在英语方面表现最好，但他们在其他语言方面的能力也很引人注目。相比之下，LLaMa 等开源模型主要在英语数据集上进行训练，导致在非英语语言中表现不佳。在本文中，我们介绍了 PersianMind，这是一种开源双语大语言模型，它在波斯语中表现出与闭源 GPT-3.5-turbo 相当的性能。通过使用 10,000 个波斯语标记扩展 LLaMa2 的词汇量，并在包含近 20 亿个波斯语标记的数据集上对其进行训练，我们表明我们的方法保留了模型的英语知识，并采用迁移学习来擅长将任务知识从一种语言迁移到另一种语言。

Title: Adapting Large Language Models for Document-Level Machine Translation

• Authors: Minghao Wu, Thuy-Trang Vu, Lizhen Qu, George Foster, Gholamreza Haffari
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06468
• Pdf URL: https://arxiv.org/pdf/2401.06468
• Copy Paste: [[2401.06468]] Adapting Large Language Models for Document-Level Machine Translation(https://arxiv.org/abs/2401.06468)
• Keywords: language model, gpt, llm, prompt
• Abstract: Large language models (LLMs) have made significant strides in various natural language processing (NLP) tasks. Recent research shows that the moderately-sized LLMs often outperform their larger counterparts after task-specific fine-tuning. In this work, we delve into the process of adapting LLMs to specialize in document-level machine translation (DocMT) for a specific language pair. Firstly, we explore how prompt strategies affect downstream translation performance. Then, we conduct extensive experiments with two fine-tuning methods, three LLM backbones, and 18 translation tasks across nine language pairs. Our findings indicate that in some cases, these specialized models even surpass GPT-4 in translation performance, while they still significantly suffer from the off-target translation issue in others, even if they are exclusively fine-tuned on bilingual parallel documents. Furthermore, we provide an in-depth analysis of these LLMs tailored for DocMT, exploring aspects such as translation errors, the scaling law of parallel documents, out-of-domain generalization, and the impact of zero-shot crosslingual transfer. The findings of this research not only shed light on the strengths and limitations of LLM-based DocMT models but also provide a foundation for future research in DocMT.
• 摘要：大型语言模型 (LLM) 在各种自然语言处理 (NLP) 任务中取得了重大进展。最近的研究表明，在针对特定任务进行微调后，中等规模的法学硕士通常比规模较大的法学硕士表现更好。在这项工作中，我们深入研究了使法学硕士适应特定语言对的文档级机器翻译（DocMT）的过程。首先，我们探讨即时策略如何影响下游翻译性能。然后，我们使用两种微调方法、三个 LLM 主干和跨 9 个语言对的 18 个翻译任务进行了广泛的实验。我们的研究结果表明，在某些情况下，这些专门模型的翻译性能甚至超过了 GPT-4，而在其他模型中，即使它们专门针对双语并行文档进行了微调，但它们仍然严重受到脱靶翻译问题的影响。此外，我们还对这些为 DocMT 量身定制的法学硕士进行了深入分析，探讨了翻译错误、并行文档的缩放规律、域外泛化以及零样本跨语言迁移的影响等方面。这项研究的结果不仅揭示了基于 LLM 的 DocMT 模型的优点和局限性，而且为 DocMT 的未来研究奠定了基础。

Title: Batch-ICL: Effective, Efficient, and Order-Agnostic In-Context Learning

• Authors: Kaiyi Zhang, Ang Lv, Yuhan Chen, Hansen Ha, Tao Xu, Rui Yan
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06469
• Pdf URL: https://arxiv.org/pdf/2401.06469
• Copy Paste: [[2401.06469]] Batch-ICL: Effective, Efficient, and Order-Agnostic In-Context Learning(https://arxiv.org/abs/2401.06469)
• Keywords: llm
• Abstract: In this paper, by treating in-context learning (ICL) as a meta-optimization process, we explain why LLMs are sensitive to the order of ICL examples. This understanding leads us to the development of Batch-ICL, an effective, efficient, and order-agnostic inference algorithm for ICL. Differing from the standard N-shot learning approach, Batch-ICL employs $N$ separate 1-shot forward computations and aggregates the resulting meta-gradients. These aggregated meta-gradients are then applied to a zero-shot learning to generate the final prediction. This batch processing approach renders the LLM agnostic to the order of ICL examples. Through extensive experiments and analysis, we demonstrate that Batch-ICL consistently outperforms most permutations of example sequences. In some cases, it even exceeds the performance of the optimal order for standard ICL, all while reducing the computational resources required. Furthermore, we develop a novel variant of Batch-ICL featuring multiple "epochs" of meta-optimization. This variant implicitly explores permutations of ICL examples, further enhancing ICL performance.
• 摘要：在本文中，通过将上下文学习（ICL）视为元优化过程，我们解释了为什么法学硕士对 ICL 示例的顺序敏感。这种理解引导我们开发了 Batch-ICL，这是一种有效、高效且与顺序无关的 ICL 推理算法。与标准的 N-shot 学习方法不同，Batch-ICL 采用 $N$ 单独的 1-shot 前向计算并聚合所得的元梯度。然后将这些聚合的元梯度应用于零样本学习以生成最终预测。这种批处理方法使 LLM 与 ICL 示例的顺序无关。通过大量的实验和分析，我们证明 Batch-ICL 始终优于示例序列的大多数排列。在某些情况下，它甚至超过了标准 ICL 的最佳性能，同时减少了所需的计算资源。此外，我们开发了 Batch-ICL 的一种新变体，具有元优化的多个“时期”。该变体隐含地探索了 ICL 示例的排列，进一步增强了 ICL 性能。

Title: Kun: Answer Polishment for Chinese Self-Alignment with Instruction Back-Translation

• Authors: Tianyu Zheng, Shuyue Guo, Xingwei Qu, Jiawei Guo, Weixu Zhang, Xinrun Du, Chenghua Lin, Wenhao Huang, Wenhu Chen, Jie Fu, Ge Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06477
• Pdf URL: https://arxiv.org/pdf/2401.06477
• Copy Paste: [[2401.06477]] Kun: Answer Polishment for Chinese Self-Alignment with Instruction Back-Translation(https://arxiv.org/abs/2401.06477)
• Keywords: language model, llm, code, rag
• Abstract: In this paper, we introduce Kun, a novel approach for creating high-quality instruction-tuning datasets for large language models (LLMs) without relying on manual annotations. Adapting a self-training algorithm based on instruction back-translation and answer polishment, Kun leverages unlabelled data from diverse sources such as Wudao, Wanjuan, and SkyPile to generate a substantial dataset of over a million Chinese instructional data points. This approach significantly deviates from traditional methods by using a self-curation process to refine and select the most effective instruction-output pairs. Our experiments with the 6B-parameter Yi model across various benchmarks demonstrate Kun's robustness and scalability. Our method's core contributions lie in its algorithmic advancement, which enhances data retention and clarity, and its innovative data generation approach that substantially reduces the reliance on costly and time-consuming manual annotations. This methodology presents a scalable and efficient solution for improving the instruction-following capabilities of LLMs, with significant implications for their application across diverse fields. The code and dataset can be found at https://github.com/Zheng0428/COIG-Kun
• 摘要：在本文中，我们介绍了 Kun，一种无需依赖手动注释即可为大型语言模型 (LLM) 创建高质量指令调优数据集的新颖方法。 Kun 采用基于指令反向翻译和答案完善的自训练算法，利用来自五道、万卷和 SkyPile 等不同来源的未标记数据，生成包含超过 100 万个中文教学数据点的大量数据集。这种方法与传统方法有很大不同，它使用自我管理过程来细化和选择最有效的指令输出对。我们在各种基准上对 6B 参数 Yi 模型进行的实验证明了 Kun 的稳健性和可扩展性。我们方法的核心贡献在于其算法的进步，增强了数据保留和清晰度，以及其创新的数据生成方法，大大减少了对昂贵且耗时的手动注释的依赖。该方法提供了一种可扩展且高效的解决方案，用于提高法学硕士的指令遵循能力，对其在不同领域的应用具有重大影响。代码和数据集可以在https://github.com/Zheng0428/COIG-Kun找到

Title: An investigation of structures responsible for gender bias in BERT and DistilBERT

• Authors: Thibaud Leteno, Antoine Gourru, Charlotte Laclau, Christophe Gravier
• Subjects: cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06495
• Pdf URL: https://arxiv.org/pdf/2401.06495
• Copy Paste: [[2401.06495]] An investigation of structures responsible for gender bias in BERT and DistilBERT(https://arxiv.org/abs/2401.06495)
• Keywords: language model, lora, code
• Abstract: In recent years, large Transformer-based Pre-trained Language Models (PLM) have changed the Natural Language Processing (NLP) landscape, by pushing the performance boundaries of the state-of-the-art on a wide variety of tasks. However, this performance gain goes along with an increase in complexity, and as a result, the size of such models (up to billions of parameters) represents a constraint for their deployment on embedded devices or short-inference time tasks. To cope with this situation, compressed models emerged (e.g. DistilBERT), democratizing their usage in a growing number of applications that impact our daily lives. A crucial issue is the fairness of the predictions made by both PLMs and their distilled counterparts. In this paper, we propose an empirical exploration of this problem by formalizing two questions: (1) Can we identify the neural mechanism(s) responsible for gender bias in BERT (and by extension DistilBERT)? (2) Does distillation tend to accentuate or mitigate gender bias (e.g. is DistilBERT more prone to gender bias than its uncompressed version, BERT)? Our findings are the following: (I) one cannot identify a specific layer that produces bias; (II) every attention head uniformly encodes bias; except in the context of underrepresented classes with a high imbalance of the sensitive attribute; (III) this subset of heads is different as we re-fine tune the network; (IV) bias is more homogeneously produced by the heads in the distilled model.
• 摘要：近年来，基于 Transformer 的大型预训练语言模型 (PLM) 突破了各种任务上最先进技术的性能界限，改变了自然语言处理 (NLP) 的格局。然而，这种性能提升伴随着复杂性的增加，因此，此类模型的大小（高达数十亿个参数）对其在嵌入式设备或短推理时间任务上的部署构成了限制。为了应对这种情况，压缩模型出现了（例如 DistilBERT），使它们在越来越多影响我们日常生活的应用程序中民主化。一个关键问题是 PLM 及其精炼同行做出的预测的公平性。在本文中，我们通过形式化两个问题提出了对该问题的实证探索：（1）我们能否识别出 BERT（以及扩展的 DistilBERT）中导致性别偏见的神经机制？ (2) 蒸馏是否会加剧或减轻性别偏见（例如，DistilBERT 是否比其未压缩版本 BERT 更容易出现性别偏见）？我们的发现如下：（一）无法识别产生偏差的特定层； (II) 每个注意力头统一编码偏差；除非在敏感属性高度不平衡的代表性不足的类别的情况下； (III) 当我们重新微调网络时，这个头部子集会有所不同； (IV) 蒸馏模型中的头部产生的偏差更加均匀。

Title: AntEval: Quantitatively Evaluating Informativeness and Expressiveness of Agent Social Interactions

• Authors: Yuanzhi Liang, Linchao Zhu, Yi Yang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06509
• Pdf URL: https://arxiv.org/pdf/2401.06509
• Copy Paste: [[2401.06509]] AntEval: Quantitatively Evaluating Informativeness and Expressiveness of Agent Social Interactions(https://arxiv.org/abs/2401.06509)
• Keywords: language model, llm, lora, rag, agent
• Abstract: While Large Language Models (LLMs) based agents have successfully mimicked human behaviors in various scenarios, the realm of complex, multi-character social interactions within extended contexts remains underexplored. The challenge is compounded by privacy concerns, making it difficult to capture and utilize intricate real-life interactions. More importantly, the absence of quantitative evaluation methods hampers the pursuit of high-quality agent interactions, often leading to interactions that are limited in informativeness and expressiveness, characterized by superficial small talk without clear intentions. In this work, we leverage the rules of Tabletop Role-Playing Games (TRPG) to create an environment conducive to complex, context-rich interactions, emphasizing informativeness and expressiveness. This virtual setting alleviates privacy concerns and motivates agents to engage in meaningful, high-quality interactions as part of their in-game objectives. To assess these interactions, we introduce the Agent interaction Evaluation framework (AntEval), targeting the qualitative evaluation of interaction informativeness and expressiveness. Specifically, we propose two novel evaluation metrics: Information Exchanging Precision (IEP) and Interaction Expressiveness Gap (IEG). These metrics are designed to assess interactions in scenarios focused on information exchange and intention expression, respectively. Our experimental results demonstrate the effectiveness of these metrics in evaluating interaction quality. Notably, we identify significant areas for improvement in LLMs regarding social interactions, as highlighted by our metrics. We believe AntEval will guide further exploration in complex agent interactions, bringing them closer to emulating real human behavior and enhancing their integration and utility in real-world applications.
• 摘要：虽然基于大型语言模型 (LLM) 的智能体已成功模仿各种场景中的人类行为，但扩展上下文中复杂、多角色社交互动的领域仍未得到充分探索。由于隐私问题，这一挑战变得更加复杂，使得捕捉和利用复杂的现实生活交互变得困难。更重要的是，缺乏定量评估方法阻碍了对高质量智能体交互的追求，往往导致交互的信息量和表达能力受到限制，表现为肤浅的闲聊，没有明确的意图。在这项工作中，我们利用桌面角色扮演游戏（TRPG）的规则来创建一个有利于复杂、上下文丰富的交互的环境，强调信息性和表现力。这种虚拟设置减轻了隐私问题，并激励代理进行有意义的、高质量的交互，作为其游戏内目标的一部分。为了评估这些交互，我们引入了代理交互评估框架（AntEval），针对交互信息性和表达性的定性评估。具体来说，我们提出了两个新颖的评估指标：信息交换精度（IEP）和交互表现力差距（IEG）。这些指标旨在评估分别侧重于信息交换和意图表达的场景中的交互。我们的实验结果证明了这些指标在评估交互质量方面的有效性。值得注意的是，正如我们的指标所强调的那样，我们确定了法学硕士在社交互动方面需要改进的重要领域。我们相信 AntEval 将指导对复杂代理交互的进一步探索，使它们更接近于模拟真实的人类行为，并增强它们在实际应用中的集成和实用性。

Title: INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning

• Authors: Yutao Zhu, Peitian Zhang, Chenghao Zhang, Yifei Chen, Binyu Xie, Zhicheng Dou, Zheng Liu, Ji-Rong Wen
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2401.06532
• Pdf URL: https://arxiv.org/pdf/2401.06532
• Copy Paste: [[2401.06532]] INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning(https://arxiv.org/abs/2401.06532)
• Keywords: language model, llm, prompt
• Abstract: Large language models (LLMs) have demonstrated impressive capabilities in various natural language processing tasks. Despite this, their application to information retrieval (IR) tasks is still challenging due to the infrequent occurrence of many IR-specific concepts in natural language. While prompt-based methods can provide task descriptions to LLMs, they often fall short in facilitating comprehensive understanding and execution of IR tasks, thereby limiting LLMs' applicability. To address this gap, in this work, we explore the potential of instruction tuning to enhance LLMs' proficiency in IR tasks. We introduce a novel instruction tuning dataset, INTERS, encompassing 21 tasks across three fundamental IR categories: query understanding, document understanding, and query-document relationship understanding. The data are derived from 43 distinct datasets with manually written templates. Our empirical results reveal that INTERS significantly boosts the performance of various publicly available LLMs, such as LLaMA, Mistral, and Phi, in search-related tasks. Furthermore, we conduct a comprehensive analysis to ascertain the effects of base model selection, instruction design, volume of instructions, and task variety on performance. We make our dataset and the models fine-tuned on it publicly accessible at https://github.com/DaoD/INTERS.
• 摘要：大型语言模型 (LLM) 在各种自然语言处理任务中表现出了令人印象深刻的能力。尽管如此，由于许多 IR 特定概念在自然语言中很少出现，因此它们在信息检索 (IR) 任务中的应用仍然具有挑战性。虽然基于提示的方法可以为法学硕士提供任务描述，但它们往往无法促进IR任务的全面理解和执行，从而限制了法学硕士的适用性。为了解决这一差距，在这项工作中，我们探索了指令调整的潜力，以提高法学硕士在 IR 任务中的熟练程度。我们引入了一个新颖的指令调优数据集 INTERS，包含跨三个基本 IR 类别的 21 项任务：查询理解、文档理解和查询-文档关系理解。这些数据源自 43 个不同的数据集，并带有手动编写的模板。我们的实证结果表明，INTERS 显着提高了各种公开的 LLM（例如 LLaMA、Mistral 和 Phi）在搜索相关任务中的表现。此外，我们进行了全面的分析，以确定基本模型选择、指令设计、指令量和任务多样性对性能的影响。我们将数据集和在其上微调的模型公开访问 https://github.com/DaoD/INTERS。

Title: Treatment-Aware Hyperbolic Representation Learning for Causal Effect Estimation with Social Networks

• Authors: Ziqiang Cui, Xing Tang, Yang Qiao, Bowei He, Liang Chen, Xiuqiang He, Chen Ma
• Subjects: cs.LG, cs.AI, cs.SI, stat.ME
• Abstract URL: https://arxiv.org/abs/2401.06557
• Pdf URL: https://arxiv.org/pdf/2401.06557
• Copy Paste: [[2401.06557]] Treatment-Aware Hyperbolic Representation Learning for Causal Effect Estimation with Social Networks(https://arxiv.org/abs/2401.06557)
• Keywords: code
• Abstract: Estimating the individual treatment effect (ITE) from observational data is a crucial research topic that holds significant value across multiple domains. How to identify hidden confounders poses a key challenge in ITE estimation. Recent studies have incorporated the structural information of social networks to tackle this challenge, achieving notable advancements. However, these methods utilize graph neural networks to learn the representation of hidden confounders in Euclidean space, disregarding two critical issues: (1) the social networks often exhibit a scalefree structure, while Euclidean embeddings suffer from high distortion when used to embed such graphs, and (2) each ego-centric network within a social network manifests a treatment-related characteristic, implying significant patterns of hidden confounders. To address these issues, we propose a novel method called Treatment-Aware Hyperbolic Representation Learning (TAHyper). Firstly, TAHyper employs the hyperbolic space to encode the social networks, thereby effectively reducing the distortion of confounder representation caused by Euclidean embeddings. Secondly, we design a treatment-aware relationship identification module that enhances the representation of hidden confounders by identifying whether an individual and her neighbors receive the same treatment. Extensive experiments on two benchmark datasets are conducted to demonstrate the superiority of our method.
• 摘要：根据观察数据估计个体治疗效果（ITE）是一个重要的研究课题，在多个领域都具有重要价值。如何识别隐藏的混杂因素是 ITE 估算中的一个关键挑战。最近的研究结合了社交网络的结构信息来应对这一挑战，取得了显着的进展。然而，这些方法利用图神经网络来学习欧几里德空间中隐藏混杂因素的表示，忽略了两个关键问题：（1）社交网络通常表现出无标度结构，而欧几里德嵌入在用于嵌入此类图时会遭受高失真， （2）社交网络中的每个以自我为中心的网络都表现出与治疗相关的特征，这意味着隐藏的混杂因素的显着模式。为了解决这些问题，我们提出了一种称为治疗感知双曲表示学习（TAHyper）的新方法。首先，TAHyper利用双曲空间对社交网络进行编码，从而有效地减少了欧几里德嵌入造成的混杂表示的失真。其次，我们设计了一个治疗感知关系识别模块，通过识别个人及其邻居是否接受相同的治疗来增强隐藏混杂因素的表示。对两个基准数据集进行了广泛的实验，以证明我们方法的优越性。

Title: Intention Analysis Prompting Makes Large Language Models A Good Jailbreak Defender

• Authors: Yuqi Zhang, Liang Ding, Lefei Zhang, Dacheng Tao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06561
• Pdf URL: https://arxiv.org/pdf/2401.06561
• Copy Paste: [[2401.06561]] Intention Analysis Prompting Makes Large Language Models A Good Jailbreak Defender(https://arxiv.org/abs/2401.06561)
• Keywords: language model, gpt, llm, prompt, code, chat, rag
• Abstract: Aligning large language models (LLMs) with human values, particularly in the face of stealthy and complex jailbreaks, presents a formidable challenge. In this study, we present a simple yet highly effective defense strategy, i.e., Intention Analysis Prompting (IAPrompt). The principle behind is to trigger LLMs' inherent self-correct and improve ability through a two-stage process: 1) essential intention analysis, and 2) policy-aligned response. Notably, IAPrompt is an inference-only method, thus could enhance the safety of LLMs without compromising their helpfulness. Extensive experiments on SAP200 and DAN benchmarks across Vicuna, ChatGLM, MPT, DeepSeek, and GPT-3.5 show that IAPrompt could consistently and significantly reduce the harmfulness in response (averagely -46.5% attack success rate) and maintain the general helpfulness. Further analyses present some insights into how our method works. To facilitate reproducibility, We release our code and scripts at: https://github.com/alphadl/SafeLLM_with_IntentionAnalysis
• 摘要：让大型语言模型 (LLM) 与人类价值观保持一致，尤其是在面对秘密且复杂的越狱时，这是一项艰巨的挑战。在本研究中，我们提出了一种简单但高效的防御策略，即意图分析提示（IAPrompt）。其背后的原理是通过两个阶段的过程来触发LLM固有的自我纠正和提高能力：1）本质意图分析，2）政策一致的响应。值得注意的是，IAPrompt 是一种仅推理的方法，因此可以增强法学硕士的安全性，而不会影响其有用性。在 Vicuna、ChatGLM、MPT、DeepSeek 和 GPT-3.5 上对 SAP200 和 DAN 基准进行的广泛实验表明，IAPrompt 可以持续显着降低响应的危害性（平均 -46.5% 的攻击成功率）并保持总体有用性。进一步的分析对我们的方法如何工作提供了一些见解。为了促进可重复性，我们在以下位置发布了代码和脚本：https://github.com/alphadl/SafeLLM_with_IntentionAnalysis

Title: Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translation

• Authors: Xu Huang, Zhirui Zhang, Xiang Geng, Yichao Du, Jiajun Chen, Shujian Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06568
• Pdf URL: https://arxiv.org/pdf/2401.06568
• Copy Paste: [[2401.06568]] Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translation(https://arxiv.org/abs/2401.06568)
• Keywords: language model, llm, prompt, rag
• Abstract: Large Language Models (LLMs) have achieved remarkable results in the machine translation evaluation task, yet there remains a gap in knowledge regarding how they utilize the provided data to conduct evaluations. This study aims to explore how LLMs leverage source and reference information in evaluating translations, with the ultimate goal of better understanding the working mechanism of LLMs. To this end, we design the controlled experiments across various input modes and model types, and employ both coarse-grained and fine-grained prompts to discern the utility of source versus reference information. Surprisingly, we find that reference information significantly enhances the evaluation accuracy, while source information sometimes is counterproductive, indicating a lack of cross-lingual capability when using LLMs to evaluate translations. We further conduct a meta-evaluation for translation error detection of LLMs, observing a similar phenomenon. These findings also suggest a potential research direction for LLMs that fully exploits the cross-lingual capability of LLMs to achieve better performance in machine translation evaluation tasks.
• 摘要：大型语言模型（LLM）在机器翻译评估任务中取得了显着的成果，但在如何利用所提供的数据进行评估方面仍然存在知识空白。本研究旨在探讨法学硕士如何利用源信息和参考信息来评估翻译，最终目标是更好地理解法学硕士的工作机制。为此，我们设计了跨各种输入模式和模型类型的受控实验，并采用粗粒度和细粒度提示来辨别源信息与参考信息的效用。令人惊讶的是，我们发现参考信息显着提高了评估准确性，而源信息有时会适得其反，表明使用法学硕士评估翻译时缺乏跨语言能力。我们进一步对法学硕士的翻译错误检测进行了元评估，观察到类似的现象。这些发现还为法学硕士提出了一个潜在的研究方向，即充分利用法学硕士的跨语言能力，以在机器翻译评估任务中取得更好的表现。

Title: Mapping Transformer Leveraged Embeddings for Cross-Lingual Document Representation

• Authors: Tsegaye Misikir Tashu, Eduard-Raul Kontos, Matthia Sabatelli, Matias Valdenegro-Toro
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06583
• Pdf URL: https://arxiv.org/pdf/2401.06583
• Copy Paste: [[2401.06583]] Mapping Transformer Leveraged Embeddings for Cross-Lingual Document Representation(https://arxiv.org/abs/2401.06583)
• Keywords: rag
• Abstract: Recommendation systems, for documents, have become tools to find relevant content on the Web. However, these systems have limitations when it comes to recommending documents in languages different from the query language, which means they might overlook resources in non-native languages. This research focuses on representing documents across languages by using Transformer Leveraged Document Representations (TLDRs) that are mapped to a cross-lingual domain. Four multilingual pre-trained transformer models (mBERT, mT5 XLM RoBERTa, ErnieM) were evaluated using three mapping methods across 20 language pairs representing combinations of five selected languages of the European Union. Metrics like Mate Retrieval Rate and Reciprocal Rank were used to measure the effectiveness of mapped TLDRs compared to non-mapped ones. The results highlight the power of cross-lingual representations achieved through pre-trained transformers and mapping approaches suggesting a promising direction for expanding beyond language connections, between two specific languages.
• 摘要：文档推荐系统已成为在网络上查找相关内容的工具。然而，这些系统在推荐与查询语言不同的语言的文档时存在局限性，这意味着它们可能会忽略非母语的资源。这项研究的重点是通过使用映射到跨语言域的 Transformer Leverages Document Representations (TLDR) 来表示跨语言的文档。使用代表欧盟五种选定语言组合的 20 个语言对的三种映射方法对四种多语言预训练 Transformer 模型（mBERT、mT5 XLM RoBERTa、ErnieM）进行了评估。诸如配对检索率和倒数排名之类的指标用于衡量映射 TLDR 与非映射 TLDR 相比的有效性。结果强调了通过预先训练的转换器和映射方法实现的跨语言表示的力量，这表明了在两种特定语言之间超越语言连接的有希望的方向。

Title: Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation

• Authors: Seongyun Lee, Seungone Kim, Sue Hyun Park, Geewook Kim, Minjoon Seo
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06591
• Pdf URL: https://arxiv.org/pdf/2401.06591
• Copy Paste: [[2401.06591]] Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation(https://arxiv.org/abs/2401.06591)
• Keywords: language model, gpt, code
• Abstract: Assessing long-form responses generated by Vision-Language Models (VLMs) is challenging. It not only requires checking whether the VLM follows the given instruction but also verifying whether the text output is properly grounded on the given image. Inspired by the recent approach of evaluating LMs with LMs, in this work, we propose to evaluate VLMs with VLMs. For this purpose, we present a new feedback dataset called the Perception Collection, encompassing 15K customized score rubrics that users might care about during assessment. Using the Perception Collection, we train Prometheus-Vision, the first open-source VLM evaluator model that can understand the user-defined score criteria during evaluation. Prometheus-Vision shows the highest Pearson correlation with human evaluators and GPT-4V among open-source models, showing its effectiveness for transparent and accessible evaluation of VLMs. We open-source our code, dataset, and model at https://github.com/kaistAI/prometheus-vision
• 摘要：评估视觉语言模型 (VLM) 生成的长格式响应具有挑战性。它不仅需要检查VLM是否遵循给定的指令，还需要验证文本输出是否正确地基于给定的图像。受到最近用 LM 评估 LM 的方法的启发，在这项工作中，我们建议用 VLM 评估 VLM。为此，我们提出了一个名为 Perception Collection 的新反馈数据集，其中包含用户在评估过程中可能关心的 15K 个定制评分标准。使用 Perception Collection，我们训练 Prometheus-Vision，这是第一个开源 VLM 评估器模型，可以在评估过程中理解用户定义的评分标准。 Prometheus-Vision 在开源模型中显示出与人类评估者和 GPT-4V 最高的 Pearson 相关性，显示了其对 VLM 透明且可访问的评估的有效性。我们在 https://github.com/kaistAI/prometheus-vision 开源我们的代码、数据集和模型

Title: Every Node is Different: Dynamically Fusing Self-Supervised Tasks for Attributed Graph Clustering

• Authors: Pengfei Zhu, Qian Wang, Yu Wang, Jialu Li, Qinghua Hu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06595
• Pdf URL: https://arxiv.org/pdf/2401.06595
• Copy Paste: [[2401.06595]] Every Node is Different: Dynamically Fusing Self-Supervised Tasks for Attributed Graph Clustering(https://arxiv.org/abs/2401.06595)
• Keywords: code
• Abstract: Attributed graph clustering is an unsupervised task that partitions nodes into different groups. Self-supervised learning (SSL) shows great potential in handling this task, and some recent studies simultaneously learn multiple SSL tasks to further boost performance. Currently, different SSL tasks are assigned the same set of weights for all graph nodes. However, we observe that some graph nodes whose neighbors are in different groups require significantly different emphases on SSL tasks. In this paper, we propose to dynamically learn the weights of SSL tasks for different nodes and fuse the embeddings learned from different SSL tasks to boost performance. We design an innovative graph clustering approach, namely Dynamically Fusing Self-Supervised Learning (DyFSS). Specifically, DyFSS fuses features extracted from diverse SSL tasks using distinct weights derived from a gating network. To effectively learn the gating network, we design a dual-level self-supervised strategy that incorporates pseudo labels and the graph structure. Extensive experiments on five datasets show that DyFSS outperforms the state-of-the-art multi-task SSL methods by up to 8.66% on the accuracy metric. The code of DyFSS is available at: https://github.com/q086/DyFSS.
• 摘要：属性图聚类是一种无监督任务，它将节点划分为不同的组。自监督学习 (SSL) 在处理这项任务方面显示出巨大的潜力，最近的一些研究同时学习多个 SSL 任务以进一步提高性能。目前，不同的 SSL 任务为所有图节点分配相同的权重集。然而，我们观察到，一些邻居位于不同组的图节点对 SSL 任务的侧重点要求明显不同。在本文中，我们建议动态学习不同节点的 SSL 任务的权重，并融合从不同 SSL 任务中学习到的嵌入以提高性能。我们设计了一种创新的图聚类方法，即动态融合自监督学习（DyFSS）。具体来说，DyFSS 使用源自门控网络的不同权重融合从不同 SSL 任务中提取的特征。为了有效地学习门网络，我们设计了一种结合伪标签和图结构的双层自监督策略。对五个数据集的大量实验表明，DyFSS 在准确度指标上优于最先进的多任务 SSL 方法高达 8.66%。 DyFSS的代码位于：https://github.com/q086/DyFSS。

Title: Mutual Enhancement of Large Language and Reinforcement Learning Models through Bi-Directional Feedback Mechanisms: A Case Study

• Authors: Shangding Gu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06603
• Pdf URL: https://arxiv.org/pdf/2401.06603
• Copy Paste: [[2401.06603]] Mutual Enhancement of Large Language and Reinforcement Learning Models through Bi-Directional Feedback Mechanisms: A Case Study(https://arxiv.org/abs/2401.06603)
• Keywords: language model, llm, lora, agent
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities for reinforcement learning (RL) models, such as planning and reasoning capabilities. However, the problems of LLMs and RL model collaboration still need to be solved. In this study, we employ a teacher-student learning framework to tackle these problems, specifically by offering feedback for LLMs using RL models and providing high-level information for RL models with LLMs in a cooperative multi-agent setting. Within this framework, the LLM acts as a teacher, while the RL model acts as a student. The two agents cooperatively assist each other through a process of recursive help, such as "I help you help I help." The LLM agent supplies abstract information to the RL agent, enabling efficient exploration and policy improvement. In turn, the RL agent offers feedback to the LLM agent, providing valuable, real-time information that helps generate more useful tokens. This bi-directional feedback loop promotes optimization, exploration, and mutual improvement for both agents, enabling them to accomplish increasingly challenging tasks. Remarkably, we propose a practical algorithm to address the problem and conduct empirical experiments to evaluate the effectiveness of our method.
• 摘要：大型语言模型 (LLM) 已经展示了强化学习 (RL) 模型的卓越功能，例如规划和推理能力。然而，LLM 和 RL 模型协作的问题仍然需要解决。在这项研究中，我们采用师生学习框架来解决这些问题，特别是通过为使用强化学习模型的法学硕士提供反馈，并在合作的多智能体环境中为法学硕士的强化学习模型提供高级信息。在这个框架内，LLM 充当老师，而 RL 模型充当学生。两个代理通过递归帮助的过程相互协作，例如“我帮助你帮助我帮助”。 LLM 代理向 RL 代理提供抽象信息，从而实现高效的探索和策略改进。反过来，RL 代理向 LLM 代理提供反馈，提供有价值的实时信息，帮助生成更有用的代币。这种双向反馈循环促进两个代理的优化、探索和相互改进，使他们能够完成越来越具有挑战性的任务。值得注意的是，我们提出了一种实用的算法来解决该问题，并进行实证实验来评估我们方法的有效性。

Title: Identifying Policy Gradient Subspaces

• Authors: Jan Schneider, Pierre Schumacher, Simon Guist, Le Chen, Daniel Häufle, Bernhard Schölkopf, Dieter Büchler
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06604
• Pdf URL: https://arxiv.org/pdf/2401.06604
• Copy Paste: [[2401.06604]] Identifying Policy Gradient Subspaces(https://arxiv.org/abs/2401.06604)
• Keywords: lora, rag
• Abstract: Policy gradient methods hold great potential for solving complex continuous control tasks. Still, their training efficiency can be improved by exploiting structure within the optimization problem. Recent work indicates that supervised learning can be accelerated by leveraging the fact that gradients lie in a low-dimensional and slowly-changing subspace. In this paper, we conduct a thorough evaluation of this phenomenon for two popular deep policy gradient methods on various simulated benchmark tasks. Our results demonstrate the existence of such gradient subspaces despite the continuously changing data distribution inherent to reinforcement learning. These findings reveal promising directions for future work on more efficient reinforcement learning, e.g., through improving parameter-space exploration or enabling second-order optimization.
• 摘要：策略梯度方法在解决复杂的连续控制任务方面具有巨大的潜力。尽管如此，他们的训练效率仍然可以通过利用优化问题中的结构来提高。最近的工作表明，可以通过利用梯度位于低维且缓慢变化的子空间这一事实来加速监督学习。在本文中，我们对两种流行的深度策略梯度方法在各种模拟基准任务上对这种现象进行了彻底的评估。我们的结果证明了这种梯度子空间的存在，尽管强化学习固有的数据分布不断变化。这些发现揭示了未来更有效的强化学习工作的有希望的方向，例如，通过改进参数空间探索或实现二阶优化。

Title: TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models

• Authors: Yihong Liu, Chunlan Ma, Haotian Ye, Hinrich Schütze
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06620
• Pdf URL: https://arxiv.org/pdf/2401.06620
• Copy Paste: [[2401.06620]] TransliCo: A Contrastive Learning Framework to Address the Script Barrier in Multilingual Pretrained Language Models(https://arxiv.org/abs/2401.06620)
• Keywords: language model, code
• Abstract: There are 293 scripts representing over 7,000 languages in the written form. Due to various reasons, many closely related languages use different scripts, which poses difficulty for multilingual pretrained language models (mPLMs) in learning crosslingual knowledge through lexical overlap. As a result, mPLMs present a script barrier: representations from different scripts are located in different subspaces, which is a strong indicator of why crosslingual transfer involving languages of different scripts shows sub-optimal performance. To address this problem, we propose a simple framework TransliCo that contains Transliteration Contrastive Modeling (TCM) to fine-tune an mPLM by contrasting sentences in its training data and their transliterations in a unified script (Latn, in our case), which ensures uniformity in the representation space for different scripts. Using Glot500-m, an mPLM pretrained on over 500 languages, as our source model, we find-tune it on a small portion (5\%) of its training data, and refer to the resulting model as Furina. We show that Furina not only better aligns representations from distinct scripts but also outperforms the original Glot500-m on various crosslingual transfer tasks. Additionally, we achieve consistent improvement in a case study on the Indic group where the languages are highly related but use different scripts. We make our code and models publicly available.
• 摘要：有 293 种文字，代表 7,000 多种书面语言。由于各种原因，许多密切相关的语言使用不同的文字，这给多语言预训练语言模型（mPLM）通过词汇重叠学习跨语言知识带来了困难。因此，mPLM 存在脚本障碍：不同脚本的表示位于不同的子空间中，这有力地说明了为什么涉及不同脚本的语言的跨语言传输表现出次优性能。为了解决这个问题，我们提出了一个简单的框架 TransliCo，其中包含音译对比建模 (TCM)，通过对比训练数据中的句子及其统一脚本（在我们的例子中为 Latn）中的音译来微调 mPLM，从而确保一致性在不同脚本的表示空间中。使用 Glot500-m（一种针对 500 多种语言进行预训练的 mPLM）作为我们的源模型，我们对其训练数据的一小部分 (5\%) 进行查找调优，并将生成的模型称为 Furina。我们表明，Furina 不仅可以更好地对齐不同脚本的表示，而且在各种跨语言迁移任务上也优于原始的 Glot500-m。此外，我们在印度语群体的案例研究中取得了持续的改进，该群体的语言高度相关，但使用不同的文字。我们公开我们的代码和模型。

Title: OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models

• Authors: Shuai Wang, Liang Ding, Li Shen, Yong Luo, Bo Du, Dacheng Tao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06628
• Pdf URL: https://arxiv.org/pdf/2401.06628
• Copy Paste: [[2401.06628]] OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models(https://arxiv.org/abs/2401.06628)
• Keywords: language model, gpt, llm, code, chat
• Abstract: Advancing automated programming necessitates robust and comprehensive code generation benchmarks, yet current evaluation frameworks largely neglect object-oriented programming (OOP) in favor of functional programming (FP), e.g., HumanEval and MBPP. To address this, our study introduces a pioneering OOP-focused benchmark, featuring 431 Python programs that encompass essential OOP concepts and features like classes and encapsulation methods. We propose a novel evaluation metric, pass@o, tailored for OOP, enhancing traditional pass@k measures. Our evaluation of 23 leading large language models (LLMs), including both general and code-specialized models, reveals three key insights: 1) pass@o offers a more relevant and comprehensive assessment for OOP code generation; 2) Despite excelling in FP, code-specialized LLMs like WizardCoder lag in OOP compared to models like ChatGPT; 3) The poor performance of all advanced LLMs on our OOP benchmark highlights a critical need for improvements in this field. Our benchmark and scripts are publicly released at: https://github.com/alphadl/OOP-eval.
• 摘要：推进自动化编程需要强大而全面的代码生成基准，但当前的评估框架在很大程度上忽略了面向对象编程（OOP），而倾向于函数式编程（FP），例如 HumanEval 和 MBPP。为了解决这个问题，我们的研究引入了一个以 OOP 为中心的开创性基准测试，其中包含 431 个 Python 程序，这些程序包含基本的 OOP 概念和功能，例如类和封装方法。我们提出了一种新颖的评估指标 pass@o，专为 OOP 量身定制，增强了传统的 pass@k 衡量标准。我们对 23 个领先的大型语言模型 (LLM)（包括通用模型和代码专用模型）的评估揭示了三个关键见解：1) pass@o 为 OOP 代码生成提供了更相关和更全面的评估； 2) 尽管在 FP 方面表现出色，但与 ChatGPT 等模型相比，WizardCoder 这样的代码专业法学硕士在 OOP 方面落后； 3) 所有高级法学硕士在我们的 OOP 基准测试中表现不佳，凸显了该领域迫切需要改进。我们的基准测试和脚本公开发布于：https://github.com/alphadl/OOP-eval。

Title: Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistently

• Authors: Kanishka Misra, Allyson Ettinger, Kyle Mahowald
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06640
• Pdf URL: https://arxiv.org/pdf/2401.06640
• Copy Paste: [[2401.06640]] Experimental Contexts Can Facilitate Robust Semantic Property Inference in Language Models, but Inconsistently(https://arxiv.org/abs/2401.06640)
• Keywords: language model
• Abstract: Recent zero-shot evaluations have highlighted important limitations in the abilities of language models (LMs) to perform meaning extraction. However, it is now well known that LMs can demonstrate radical improvements in the presence of experimental contexts such as in-context examples and instructions. How well does this translate to previously studied meaning-sensitive tasks? We present a case-study on the extent to which experimental contexts can improve LMs' robustness in performing property inheritance -- predicting semantic properties of novel concepts, a task that they have been previously shown to fail on. Upon carefully controlling the nature of the in-context examples and the instructions, our work reveals that they can indeed lead to non-trivial property inheritance behavior in LMs. However, this ability is inconsistent: with a minimal reformulation of the task, some LMs were found to pick up on shallow, non-semantic heuristics from their inputs, suggesting that the computational principles of semantic property inference are yet to be mastered by LMs.
• 摘要：最近的零样本评估凸显了语言模型（LM）执行意义提取的能力的重要局限性。然而，现在众所周知，语言模型可以在实验上下文（例如上下文示例和说明）的存在下展示出根本性的改进。这对于之前研究的意义敏感任务的转化效果如何？我们提出了一个案例研究，说明实验上下文可以在多大程度上提高语言模型在执行属性继承方面的鲁棒性——预测新概念的语义属性，而之前的研究表明它们无法完成这项任务。在仔细控制上下文示例和指令的性质后，我们的工作表明它们确实可以导致 LM 中的非平凡属性继承行为。然而，这种能力是不一致的：通过对任务进行最小程度的重新表述，我们发现一些语言模型从其输入中获取了浅层的、非语义启发式的信息，这表明语言模型尚未掌握语义属性推断的计算原理。

Title: Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentation

• Authors: Jan Cegin, Branislav Pecher, Jakub Simko, Ivan Srba, Maria Bielikova, Peter Brusilovsky
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06643
• Pdf URL: https://arxiv.org/pdf/2401.06643
• Copy Paste: [[2401.06643]] Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentation(https://arxiv.org/abs/2401.06643)
• Keywords: language model, llm, prompt
• Abstract: The latest generative large language models (LLMs) have found their application in data augmentation tasks, where small numbers of text samples are LLM-paraphrased and then used to fine-tune the model. However, more research is needed to assess how different prompts, seed data selection strategies, filtering methods, or model settings affect the quality of paraphrased data (and downstream models). In this study, we investigate three text diversity incentive methods well established in crowdsourcing: taboo words, hints by previous outlier solutions, and chaining on previous outlier solutions. Using these incentive methods as part of instructions to LLMs augmenting text datasets, we measure their effects on generated texts' lexical diversity and downstream model performance. We compare the effects over 5 different LLMs and 6 datasets. We show that diversity is most increased by taboo words, while downstream model performance is highest when previously created paraphrases are used as hints.
• 摘要：最新的生成式大语言模型 (LLM) 已在数据增强任务中得到应用，其中少量文本样本经过 LLM 释义，然后用于微调模型。然而，需要更多的研究来评估不同的提示、种子数据选择策略、过滤方法或模型设置如何影响释义数据（和下游模型）的质量。在本研究中，我们研究了众包中成熟的三种文本多样性激励方法：禁忌词、先前异常值解决方案的提示以及先前异常值解决方案的链接。使用这些激励方法作为法学硕士增强文本数据集的说明的一部分，我们测量它们对生成文本的词汇多样性和下游模型性能的影响。我们比较了 5 个不同的法学硕士和 6 个数据集的效果。我们表明，禁忌词可以最大程度地增加多样性，而当使用先前创建的释义作为提示时，下游模型的性能最高。

Title: SeizNet: An AI-enabled Implantable Sensor Network System for Seizure Prediction

• Authors: Ali Saeizadeh, Douglas Schonholtz, Daniel Uvaydov, Raffaele Guida, Emrecan Demirors, Pedram Johari, Jorge M. Jimenez, Joseph S. Neimat, Tommaso Melodia
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2401.06644
• Pdf URL: https://arxiv.org/pdf/2401.06644
• Copy Paste: [[2401.06644]] SeizNet: An AI-enabled Implantable Sensor Network System for Seizure Prediction(https://arxiv.org/abs/2401.06644)
• Keywords: rag
• Abstract: In this paper, we introduce SeizNet, a closed-loop system for predicting epileptic seizures through the use of Deep Learning (DL) method and implantable sensor networks. While pharmacological treatment is effective for some epilepsy patients (with ~65M people affected worldwide), one out of three suffer from drug-resistant epilepsy. To alleviate the impact of seizure, predictive systems have been developed that can notify such patients of an impending seizure, allowing them to take precautionary measures. SeizNet leverages DL techniques and combines data from multiple recordings, specifically intracranial electroencephalogram (iEEG) and electrocardiogram (ECG) sensors, that can significantly improve the specificity of seizure prediction while preserving very high levels of sensitivity. SeizNet DL algorithms are designed for efficient real-time execution at the edge, minimizing data privacy concerns, data transmission overhead, and power inefficiencies associated with cloud-based solutions. Our results indicate that SeizNet outperforms traditional single-modality and non-personalized prediction systems in all metrics, achieving up to 99% accuracy in predicting seizure, offering a promising new avenue in refractory epilepsy treatment.
• 摘要：在本文中，我们介绍了 SeizNet，这是一个通过使用深度学习（DL）方法和植入式传感器网络来预测癫痫发作的闭环系统。虽然药物治疗对某些癫痫患者有效（全世界约有 6500 万人受到影响），但三分之一的人患有耐药性癫痫。为了减轻癫痫发作的影响，已经开发了预测系统，可以通知此类患者即将发生癫痫发作，从而使他们能够采取预防措施。 SeizNet 利用深度学习技术并结合来自多个记录的数据，特别是颅内脑电图 (iEEG) 和心电图 (ECG) 传感器的数据，可以显着提高癫痫预测的特异性，同时保持非常高的灵敏度。 SeizNet DL 算法专为在边缘高效实时执行而设计，最大限度地减少与基于云的解决方案相关的数据隐私问题、数据传输开销和电源效率低下。我们的结果表明，SeizNet 在所有指标上都优于传统的单一模式和非个性化预测系统，预测癫痫发作的准确率高达 99%，为难治性癫痫治疗提供了一条有前景的新途径。

Title: WisdoM: Improving Multimodal Sentiment Analysis by Fusing Contextual World Knowledge

• Authors: Wenbin Wang, Liang Ding, Li Shen, Yong Luo, Han Hu, Dacheng Tao
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06659
• Pdf URL: https://arxiv.org/pdf/2401.06659
• Copy Paste: [[2401.06659]] WisdoM: Improving Multimodal Sentiment Analysis by Fusing Contextual World Knowledge(https://arxiv.org/abs/2401.06659)
• Keywords: language model, code, rag
• Abstract: Sentiment analysis is rapidly advancing by utilizing various data modalities (e.g., text, image). However, most previous works relied on superficial information, neglecting the incorporation of contextual world knowledge (e.g., background information derived from but beyond the given image and text pairs) and thereby restricting their ability to achieve better multimodal sentiment analysis. In this paper, we proposed a plug-in framework named WisdoM, designed to leverage contextual world knowledge induced from the large vision-language models (LVLMs) for enhanced multimodal sentiment analysis. WisdoM utilizes a LVLM to comprehensively analyze both images and corresponding sentences, simultaneously generating pertinent context. To reduce the noise in the context, we also introduce a training-free Contextual Fusion mechanism. Experimental results across diverse granularities of multimodal sentiment analysis tasks consistently demonstrate that our approach has substantial improvements (brings an average +1.89 F1 score among five advanced methods) over several state-of-the-art methods. Code will be released.
• 摘要：通过利用各种数据模式（例如文本、图像），情感分析正在迅速发展。然而，以前的大多数工作依赖于表面信息，忽略了上下文世界知识的结合（例如，源自但超出给定图像和文本对的背景信息），从而限制了它们实现更好的多模态情感分析的能力。在本文中，我们提出了一个名为 WisdoM 的插件框架，旨在利用从大型视觉语言模型（LVLM）导出的上下文世界知识来增强多模式情感分析。 WisdoM 利用 LVLM 全面分析图像和相应的句子，同时生成相关上下文。为了减少上下文中的噪声，我们还引入了免训练的上下文融合机制。多模态情感分析任务的不同粒度的实验结果一致表明，我们的方法比几种最先进的方法有显着的改进（五种高级方法中平均 F1 分数为 +1.89）。代码将被发布。

Title: Don't Rank, Combine! Combining Machine Translation Hypotheses Using Quality Estimation

• Authors: Giorgos Vernikos, Andrei Popescu-Belis
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06688
• Pdf URL: https://arxiv.org/pdf/2401.06688
• Copy Paste: [[2401.06688]] Don't Rank, Combine! Combining Machine Translation Hypotheses Using Quality Estimation(https://arxiv.org/abs/2401.06688)
• Keywords: language model, llm, rag
• Abstract: Neural machine translation systems estimate probabilities of target sentences given source sentences, yet these estimates may not align with human preferences. This work introduces QE-fusion, a method utilizing a quality estimation metric (QE) that better correlates with human judgments to synthesize improved translations. QE-fusion leverages a candidate pool sampled from a model, combining spans from different candidates using QE metrics such as CometKiwi. We compare QE-fusion against beam search and recent reranking techniques, such as Minimum Bayes Risk decoding or QE-reranking. Our method consistently improves translation quality in terms of COMET and BLEURT scores when applied to large language models (LLMs) used for translation (PolyLM, XGLM, Llama2, and Mistral) and to multilingual translation models (NLLB), over five language pairs. Notably, QE-fusion exhibits larger improvements for LLMs due to their ability to generate diverse outputs. We demonstrate that our approach generates novel translations in over half of the cases and consistently outperforms other methods across varying numbers of candidates (5-200). Furthermore, we empirically establish that QE-fusion scales linearly with the number of candidates in the pool. QE-fusion proves effective in enhancing LLM-based translation without the need for costly retraining of LLMs.
• 摘要：神经机器翻译系统估计给定源句子的目标句子的概率，但这些估计可能与人类的偏好不一致。这项工作引入了 QE-fusion，这是一种利用质量估计指标 (QE) 的方法，可以更好地与人类判断相关联，以综合改进的翻译。 QE-fusion 利用从模型中采样的候选池，使用 CometKiwi 等 QE 指标组合来自不同候选的跨度。我们将 QE 融合与波束搜索和最近的重新排序技术（例如最小贝叶斯风险解码或 QE 重新排序）进行比较。当应用于用于翻译的大型语言模型 (LLM)（PolyLM、XGLM、Llama2 和 Mistral）以及超过五种语言对的多语言翻译模型 (NLLB) 时，我们的方法在 COMET 和 BLEURT 分数方面持续提高翻译质量。值得注意的是，QE-fusion 对 LLM 表现出更大的改进，因为它们能够产生不同的输出。我们证明，我们的方法在超过一半的案例中生成了新颖的翻译，并且在不同数量的候选者（5-20​​0）中始终优于其他方法。此外，我们凭经验确定 QE 融合与池中候选者的数量呈线性关系。事实证明，QE-fusion 可以有效增强基于 LLM 的翻译，而无需对 LLM 进行昂贵的再培训。

Title: An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models

• Authors: Gantavya Bhatt, Yifang Chen, Arnav M. Das, Jifan Zhang, Sang T. Truong, Stephen Mussmann, Yinglun Zhu, Jeffrey Bilmes, Simon S. Du, Kevin Jamieson, Jordan T. Ash, Robert D. Nowak
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06692
• Pdf URL: https://arxiv.org/pdf/2401.06692
• Copy Paste: [[2401.06692]] An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models(https://arxiv.org/abs/2401.06692)
• Keywords: language model, llm
• Abstract: Supervised finetuning (SFT) on instruction datasets has played a crucial role in achieving the remarkable zero-shot generalization capabilities observed in modern large language models (LLMs). However, the annotation efforts required to produce high quality responses for instructions are becoming prohibitively expensive, especially as the number of tasks spanned by instruction datasets continues to increase. Active learning is effective in identifying useful subsets of samples to annotate from an unlabeled pool, but its high computational cost remains a barrier to its widespread applicability in the context of LLMs. To mitigate the annotation cost of SFT and circumvent the computational bottlenecks of active learning, we propose using experimental design. Experimental design techniques select the most informative samples to label, and typically maximize some notion of uncertainty and/or diversity. In our work, we implement a framework that evaluates several existing and novel experimental design techniques and find that these methods consistently yield significant gains in label efficiency with little computational overhead. On generative tasks, our methods achieve the same generalization performance with only $50\%$ of annotation cost required by random sampling.
• 摘要：指令数据集上的监督微调（SFT）在实现现代大型语言模型（LLM）中观察到的卓越的零样本泛化能力方面发挥了至关重要的作用。然而，为指令生成高质量响应所需的注释工作变得异常昂贵，特别是随着指令数据集跨越的任务数量不断增加。主动学习可以有效地从未标记的池中识别有用的样本子集进行注释，但其高昂的计算成本仍然是其在法学硕士背景下广泛适用的障碍。为了减轻 SFT 的注释成本并规避主动学习的计算瓶颈，我们建议使用实验设计。实验设计技术选择信息最丰富的样本进行标记，并且通常最大化一些不确定性和/或多样性的概念。在我们的工作中，我们实现了一个框架，该框架评估了几种现有的和新颖的实验设计技术，并发现这些方法始终能够以很少的计算开销显着提高标签效率。在生成任务上，我们的方法只需随机抽样所需的 50\%$ 注释成本即可实现相同的泛化性能。

Title: Multi-Candidate Speculative Decoding

• Authors: Sen Yang, Shujian Huang, Xinyu Dai, Jiajun Chen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06706
• Pdf URL: https://arxiv.org/pdf/2401.06706
• Copy Paste: [[2401.06706]] Multi-Candidate Speculative Decoding(https://arxiv.org/abs/2401.06706)
• Keywords: language model
• Abstract: Large language models have shown impressive capabilities across a variety of NLP tasks, yet their generating text autoregressively is time-consuming. One way to speed them up is speculative decoding, which generates candidate segments (a sequence of tokens) from a fast draft model that is then verified in parallel by the target model. However, the acceptance rate of candidate tokens receives limitations from several factors, such as the model, the dataset, and the decoding setup. This paper proposes sampling multiple candidates from a draft model and then organising them in batches for verification. We design algorithms for efficient multi-candidate verification while maintaining the distribution of the target model. Our approach shows significant improvements in acceptance rates on multiple datasets and models, consistently outperforming standard speculative decoding.
• 摘要：大型语言模型在各种 NLP 任务中表现出了令人印象深刻的能力，但它们以自回归方式生成文本非常耗时。加速它们的一种方法是推测解码，它从快速草稿模型生成候选片段（令牌序列），然后由目标模型并行验证。然而，候选令牌的接受率受到模型、数据集和解码设置等多个因素的限制。本文建议从草稿模型中采样多个候选者，然后将它们分批组织以进行验证。我们设计了有效的多候选验证算法，同时保持目标模型的分布。我们的方法显示了多个数据集和模型的接受率显着提高，始终优于标准推测解码。

Title: Reliability Analysis of Psychological Concept Extraction and Classification in User-penned Text

• Authors: Muskan Garg, MSVPJ Sathvik, Amrit Chadha, Shaina Raza, Sunghwan Sohn
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06709
• Pdf URL: https://arxiv.org/pdf/2401.06709
• Copy Paste: [[2401.06709]] Reliability Analysis of Psychological Concept Extraction and Classification in User-penned Text(https://arxiv.org/abs/2401.06709)
• Keywords: language model
• Abstract: The social NLP research community witness a recent surge in the computational advancements of mental health analysis to build responsible AI models for a complex interplay between language use and self-perception. Such responsible AI models aid in quantifying the psychological concepts from user-penned texts on social media. On thinking beyond the low-level (classification) task, we advance the existing binary classification dataset, towards a higher-level task of reliability analysis through the lens of explanations, posing it as one of the safety measures. We annotate the LoST dataset to capture nuanced textual cues that suggest the presence of low self-esteem in the posts of Reddit users. We further state that the NLP models developed for determining the presence of low self-esteem, focus more on three types of textual cues: (i) Trigger: words that triggers mental disturbance, (ii) LoST indicators: text indicators emphasizing low self-esteem, and (iii) Consequences: words describing the consequences of mental disturbance. We implement existing classifiers to examine the attention mechanism in pre-trained language models (PLMs) for a domain-specific psychology-grounded task. Our findings suggest the need of shifting the focus of PLMs from Trigger and Consequences to a more comprehensive explanation, emphasizing LoST indicators while determining low self-esteem in Reddit posts.
• 摘要：社会 NLP 研究界见证了心理健康分析的计算进步最近激增，为语言使用和自我感知之间的复杂相互作用构建了负责任的人工智能模型。这种负责任的人工智能模型有助于量化社交媒体上用户撰写的文本中的心理概念。在超越低级（分类）任务的思考中，我们将现有的二元分类数据集推进到通过解释的角度进行可靠性分析的更高级别的任务，将其作为安全措施之一。我们对 LoST 数据集进行注释，以捕获微妙的文本线索，这些线索表明 Reddit 用户的帖子中存在低自尊。我们进一步指出，为确定低自尊的存在而开发的 NLP 模型，更多地关注三种类型的文本提示：（i）触发因素：引发精神障碍的单词，（ii）LoST 指标：强调低自尊的文本指标。 (iii) 后果：描述精神障碍后果的词语。我们实现现有的分类器来检查预训练语言模型（PLM）中的注意力机制，以完成特定领域的心理学任务。我们的研究结果表明，需要将 PLM 的重点从触发因素和后果转向更全面的解释，强调 LoST 指标，同时确定 Reddit 帖子中的低自尊。

Title: Few-Shot Detection of Machine-Generated Text using Style Representations

• Authors: Rafael Rivera Soto, Kailin Koch, Aleem Khan, Barry Chen, Marcus Bishop, Nicholas Andrews
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06712
• Pdf URL: https://arxiv.org/pdf/2401.06712
• Copy Paste: [[2401.06712]] Few-Shot Detection of Machine-Generated Text using Style Representations(https://arxiv.org/abs/2401.06712)
• Keywords: language model, gpt, chat, rag
• Abstract: The advent of instruction-tuned language models that convincingly mimic human writing poses a significant risk of abuse. For example, such models could be used for plagiarism, disinformation, spam, or phishing. However, such abuse may be counteracted with the ability to detect whether a piece of text was composed by a language model rather than a human. Some previous approaches to this problem have relied on supervised methods trained on corpora of confirmed human and machine-written documents. Unfortunately, model under-specification poses an unavoidable challenge for neural network-based detectors, making them brittle in the face of data shifts, such as the release of further language models producing still more fluent text than the models used to train the detectors. Other previous approaches require access to the models that may have generated a document in question at inference or detection time, which is often impractical. In light of these challenges, we pursue a fundamentally different approach not relying on samples from language models of concern at training time. Instead, we propose to leverage representations of writing style estimated from human-authored text. Indeed, we find that features effective at distinguishing among human authors are also effective at distinguishing human from machine authors, including state of the art large language models like Llama 2, ChatGPT, and GPT-4. Furthermore, given a handful of examples composed by each of several specific language models of interest, our approach affords the ability to predict which model generated a given document.
• 摘要：令人信服地模仿人类书写的指令调整语言模型的出现带来了巨大的滥用风险。例如，此类模型可用于剽窃、虚假信息、垃圾邮件或网络钓鱼。然而，这种滥用可以通过检测一段文本是否由语言模型而不是人类编写的能力来抵消。以前解决这个问题的一些方法依赖于在已确认的人类和机器编写文档的语料库上训练的监督方法。不幸的是，模型规格不足给基于神经网络的检测器带来了不可避免的挑战，使它们在面对数据变化时变得脆弱，例如发布进一步的语言模型，产生比用于训练检测器的模型更流畅的文本。先前的其他方法需要访问可能在推理或检测时生成相关文档的模型，这通常是不切实际的。鉴于这些挑战，我们追求一种根本不同的方法，不依赖于训练时关注的语言模型中的样本。相反，我们建议利用从人类创作的文本中估计的写作风格的表示。事实上，我们发现能够有效区分人类作者的特征也能有效区分人类和机器作者，包括 Llama 2、ChatGPT 和 GPT-4 等最先进的大型语言模型。此外，考虑到由感兴趣的几种特定语言模型中的每一个组成的少数示例，我们的方法提供了预测哪个模型生成给定文档的能力。

Title: Deep Manifold Graph Auto-Encoder for Attributed Graph Embedding

• Authors: Bozhen Hu, Zelin Zang, Jun Xia, Lirong Wu, Cheng Tan, Stan Z. Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06727
• Pdf URL: https://arxiv.org/pdf/2401.06727
• Copy Paste: [[2401.06727]] Deep Manifold Graph Auto-Encoder for Attributed Graph Embedding(https://arxiv.org/abs/2401.06727)
• Keywords: code
• Abstract: Representing graph data in a low-dimensional space for subsequent tasks is the purpose of attributed graph embedding. Most existing neural network approaches learn latent representations by minimizing reconstruction errors. Rare work considers the data distribution and the topological structure of latent codes simultaneously, which often results in inferior embeddings in real-world graph data. This paper proposes a novel Deep Manifold (Variational) Graph Auto-Encoder (DMVGAE/DMGAE) method for attributed graph data to improve the stability and quality of learned representations to tackle the crowding problem. The node-to-node geodesic similarity is preserved between the original and latent space under a pre-defined distribution. The proposed method surpasses state-of-the-art baseline algorithms by a significant margin on different downstream tasks across popular datasets, which validates our solutions. We promise to release the code after acceptance.
• 摘要：在低维空间中表示图数据以供后续任务使用是属性图嵌入的目的。大多数现有的神经网络方法通过最小化重建误差来学习潜在表示。很少有工作同时考虑数据分布和潜在代码的拓扑结构，这通常会导致现实世界图数据中的嵌入质量较差。本文针对属性图数据提出了一种新颖的深度流形（变分）图自动编码器（DMVGAE/DMGAE）方法，以提高学习表示的稳定性和质量，从而解决拥挤问题。在预定义的分布下，原始空间和潜在空间之间保留了节点到节点的测地线相似性。所提出的方法在流行数据集的不同下游任务上明显超越了最先进的基线算法，这验证了我们的解决方案。我们承诺在接受后发布代码。

Title: Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty

• Authors: Kaitlyn Zhou, Jena D. Hwang, Xiang Ren, Maarten Sap
• Subjects: cs.CL, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2401.06730
• Pdf URL: https://arxiv.org/pdf/2401.06730
• Copy Paste: [[2401.06730]] Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty(https://arxiv.org/abs/2401.06730)
• Keywords: language model, prompt, rag
• Abstract: As natural language becomes the default interface for human-AI interaction, there is a critical need for LMs to appropriately communicate uncertainties in downstream applications. In this work, we investigate how LMs incorporate confidence about their responses via natural language and how downstream users behave in response to LM-articulated uncertainties. We examine publicly deployed models and find that LMs are unable to express uncertainties when answering questions even when they produce incorrect responses. LMs can be explicitly prompted to express confidences, but tend to be overconfident, resulting in high error rates (on average 47%) among confident responses. We test the risks of LM overconfidence by running human experiments and show that users rely heavily on LM generations, whether or not they are marked by certainty. Lastly, we investigate the preference-annotated datasets used in RLHF alignment and find that humans have a bias against texts with uncertainty. Our work highlights a new set of safety harms facing human-LM interactions and proposes design recommendations and mitigating strategies moving forward.
• 摘要：随着自然语言成为人类与人工智能交互的默认界面，语言模型迫切需要在下游应用程序中适当地传达不确定性。在这项工作中，我们研究了 LM 如何通过自然语言整合对其响应的信心，以及下游用户如何响应 LM 表达的不确定性。我们检查了公开部署的模型，发现语言模型在回答问题时无法表达不确定性，即使它们给出了错误的答案。可以明确提示 LM 表达信心，但往往过于自信，导致自信响应中的错误率很高（平均 47%）。我们通过人体实验来测试 LM 过度自信的风险，并表明用户严重依赖 LM 世代，无论它们是否具有确定性。最后，我们研究了 RLHF 对齐中使用的偏好注释数据集，发现人类对不确定的文本存在偏见。我们的工作强调了人类与LM交互面临的一系列新的安全危害，并提出了未来的设计建议和缓解策略。

Title: Using Natural Language Inference to Improve Persona Extraction from Dialogue in a New Domain

• Authors: Alexandra DeLucia, Mengjie Zhao, Yoshinori Maeda, Makoto Yoda, Keiichi Yamada, Hiromi Wakaki
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.06742
• Pdf URL: https://arxiv.org/pdf/2401.06742
• Copy Paste: [[2401.06742]] Using Natural Language Inference to Improve Persona Extraction from Dialogue in a New Domain(https://arxiv.org/abs/2401.06742)
• Keywords: chat, agent
• Abstract: While valuable datasets such as PersonaChat provide a foundation for training persona-grounded dialogue agents, they lack diversity in conversational and narrative settings, primarily existing in the "real" world. To develop dialogue agents with unique personas, models are trained to converse given a specific persona, but hand-crafting these persona can be time-consuming, thus methods exist to automatically extract persona information from existing character-specific dialogue. However, these persona-extraction models are also trained on datasets derived from PersonaChat and struggle to provide high-quality persona information from conversational settings that do not take place in the real world, such as the fantasy-focused dataset, LIGHT. Creating new data to train models on a specific setting is human-intensive, thus prohibitively expensive. To address both these issues, we introduce a natural language inference method for post-hoc adapting a trained persona extraction model to a new setting. We draw inspiration from the literature of dialog natural language inference (NLI), and devise NLI-reranking methods to extract structured persona information from dialogue. Compared to existing persona extraction models, our method returns higher-quality extracted persona and requires less human annotation.
• 摘要：虽然 PersonaChat 等有价值的数据集为训练基于角色的对话代理提供了基础，但它们在对话和叙事环境中缺乏多样性，主要存在于“现实”世界中。为了开发具有独特角色的对话代理，需要训练模型来与给定的特定角色进行对话，但手工制作这些角色可能非常耗时，因此存在从现有的特定角色对话中自动提取角色信息的方法。然而，这些角色提取模型也在源自 PersonaChat 的数据集上进行训练，并且很难从现实世界中不发生的对话设置中提供高质量的角色信息，例如以幻想为中心的数据集 LIGHT。创建新数据来在特定环境下训练模型需要大量人力，因此成本高昂。为了解决这两个问题，我们引入了一种自然语言推理方法，用于事后将经过训练的角色提取模型适应新的设置。我们从对话自然语言推理（NLI）文献中汲取灵感，并设计了 NLI 重排序方法来从对话中提取结构化人物信息。与现有的角色提取模型相比，我们的方法返回更高质量的提取角色并且需要更少的人工注释。

Title: The Unreasonable Effectiveness of Easy Training Data for Hard Tasks

• Authors: Peter Hase, Mohit Bansal, Peter Clark, Sarah Wiegreffe
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.06751
• Pdf URL: https://arxiv.org/pdf/2401.06751
• Copy Paste: [[2401.06751]] The Unreasonable Effectiveness of Easy Training Data for Hard Tasks(https://arxiv.org/abs/2401.06751)
• Keywords: language model, lora, code
• Abstract: How can we train models to perform well on hard test data when hard training data is by definition difficult to label correctly? This question has been termed the scalable oversight problem and has drawn increasing attention as language models have continually improved. In this paper, we present the surprising conclusion that current language models often generalize relatively well from easy to hard data, even performing as well as "oracle" models trained on hard data. We demonstrate this kind of easy-to-hard generalization using simple training methods like in-context learning, linear classifier heads, and QLoRA for seven different measures of datapoint hardness, including six empirically diverse human hardness measures (like grade level) and one model-based measure (loss-based). Furthermore, we show that even if one cares most about model performance on hard data, it can be better to collect and train on easy data rather than hard data, since hard data is generally noisier and costlier to collect. Our experiments use open models up to 70b in size and four publicly available question-answering datasets with questions ranging in difficulty from 3rd grade science questions to college level STEM questions and general-knowledge trivia. We conclude that easy-to-hard generalization in LMs is surprisingly strong for the tasks studied, suggesting the scalable oversight problem may be easier than previously thought. Our code is available at https://github.com/allenai/easy-to-hard-generalization
• 摘要：当硬训练数据根据定义很难正确标记时，我们如何训练模型在硬测试数据上表现良好？这个问题被称为可​​扩展的监督问题，并且随着语言模型的不断改进而引起越来越多的关注。在本文中，我们提出了一个令人惊讶的结论，即当前的语言模型通常能够相对较好地从简单数据到硬数据进行泛化，甚至表现得与在硬数据上训练的“oracle”模型一样好。我们使用简单的训练方法（例如上下文学习、线性分类器头和 QLoRA）来演示这种从易到难的泛化，用于七种不同的数据点硬度测量，包括六种经验上不同的人类硬度测量（如年级水平）和一个模型基于测量（基于损失）。此外，我们表明，即使人们最关心硬数据上的模型性能，收集和训练简单数据也比硬数据更好，因为硬数据通常噪声更大，收集成本更高。我们的实验使用大小高达 70b 的开放模型和四个公开可用的问答数据集，问题的难度范围从三年级科学问题到大学水平的 STEM 问题和一般知识琐事。我们得出的结论是，语言模型中从易到难的泛化对于所研究的任务来说出人意料地强大，这表明可扩展的监督问题可能比之前想象的更容易。我们的代码位于 https://github.com/allenai/easy-to-hard-generalization

Title: APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decoding

• Authors: Mingdao Liu, Aohan Zeng, Bowen Wang, Peng Zhang, Jie Tang, Yuxiao Dong
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06761
• Pdf URL: https://arxiv.org/pdf/2401.06761
• Copy Paste: [[2401.06761]] APAR: LLMs Can Do Auto-Parallel Auto-Regressive Decoding(https://arxiv.org/abs/2401.06761)
• Keywords: language model, llm
• Abstract: The massive adoption of large language models (LLMs) demands efficient deployment strategies. However, the auto-regressive decoding process, which is fundamental to how most LLMs generate text, poses challenges to achieve efficient serving. In this work, we introduce a parallel auto-regressive generation method. By instruct-tuning on general domain data that contains hierarchical structures, we enable LLMs to independently plan their generation process and perform auto-parallel auto-regressive (APAR) generation, significantly reducing the number of generation steps. APAR alone can achieve up to 2x speed-up, and when combined with speculative decoding, the speed-up can reach up to 4x. In addition, APAR reduces the key-value cache consumption and attention computation during generation. This leads to a throughput increase of 20-70% and a latency reduce of 20-35% in high-throughput scenarios, compared to state-of-the-art serving frameworks.
• 摘要：大型语言模型 (LLM) 的大规模采用需要高效的部署策略。然而，自回归解码过程是大多数法学硕士生成文本的基础，它对实现高效服务提出了挑战。在这项工作中，我们介绍了一种并行自回归生成方法。通过对包含层次结构的通用领域数据进行指令调整，我们使法学硕士能够独立规划其生成过程并执行自动并行自回归（APAR）生成，从而显着减少生成步骤的数量。单独使用 APAR 即可实现高达 2 倍的加速，与推测解码结合使用时，加速可高达 4 倍。此外，APAR 减少了生成过程中的键值缓存消耗和注意力计算。与最先进的服务框架相比，在高吞吐量场景中，吞吐量增加了 20-70%，延迟减少了 20-35%。

Title: Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements

• Authors: Anton Voronov, Lena Wolf, Max Ryabinin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06766
• Pdf URL: https://arxiv.org/pdf/2401.06766
• Copy Paste: [[2401.06766]] Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements(https://arxiv.org/abs/2401.06766)
• Keywords: language model, prompt, rag
• Abstract: Large language models demonstrate a remarkable capability for learning to solve new tasks from a few examples. The prompt template, or the way the input examples are formatted to obtain the prompt, is an important yet often overlooked aspect of in-context learning. In this work, we conduct a comprehensive study of the template format's influence on the in-context learning performance. We evaluate the impact of the prompt template across models (from 770M to 70B parameters) and 4 standard classification datasets. We show that a poor choice of the template can reduce the performance of the strongest models and inference methods to a random guess level. More importantly, the best templates do not transfer between different setups and even between models of the same family. Our findings show that the currently prevalent approach to evaluation, which ignores template selection, may give misleading results due to different templates in different works. As a first step towards mitigating this issue, we propose Template Ensembles that aggregate model predictions across several templates. This simple test-time augmentation boosts average performance while being robust to the choice of random set of templates.
• 摘要：大型语言模型展示了从几个例子中学习解决新任务的卓越能力。提示模板，或者输入示例的格式化方式以获得提示，是上下文学习的一个重要但经常被忽视的方面。在这项工作中，我们对模板格式对情境学习性能的影响进行了全面的研究。我们评估了提示模板跨模型（从 770M 到 70B 参数）和 4 个标准分类数据集的影响。我们表明，模板选择不当可能会将最强模型和推理方法的性能降低到随机猜测水平。更重要的是，最好的模板不会在不同的设置之间转移，甚至不会在同一系列的型号之间转移。我们的研究结果表明，目前流行的评估方法忽视了模板的选择，可能会因不同作品中的模板不同而给出误导性的结果。作为缓解这个问题的第一步，我们提出了模板集成，它可以聚合多个模板的模型预测。这种简单的测试时间增强提高了平均性能，同时对随机模板集的选择具有鲁棒性。

Title: Machine Translation Models are Zero-Shot Detectors of Translation Direction

• Authors: Michelle Wastl, Jannis Vamvas, Rico Sennrich
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.06769
• Pdf URL: https://arxiv.org/pdf/2401.06769
• Copy Paste: [[2401.06769]] Machine Translation Models are Zero-Shot Detectors of Translation Direction(https://arxiv.org/abs/2401.06769)
• Keywords: code
• Abstract: Detecting the translation direction of parallel text has applications for machine translation training and evaluation, but also has forensic applications such as resolving plagiarism or forgery allegations. In this work, we explore an unsupervised approach to translation direction detection based on the simple hypothesis that $p(\text{translation}|\text{original})>p(\text{original}|\text{translation})$, motivated by the well-known simplification effect in translationese or machine-translationese. In experiments with massively multilingual machine translation models across 20 translation directions, we confirm the effectiveness of the approach for high-resource language pairs, achieving document-level accuracies of 82-96% for NMT-produced translations, and 60-81% for human translations, depending on the model used. Code and demo are available at https://github.com/ZurichNLP/translation-direction-detection
• 摘要：检测并行文本的翻译方向不仅可用于机器翻译训练和评估，还具有法医应用，例如解决抄袭或伪造指控。在这项工作中，我们基于简单的假设 $p(\text{translation}|\text{original})>p(\text{original}|\text{translation})$ 探索一种无监督的平移方向检测方法，受到翻译语或机器翻译语中众所周知的简化效应的启发。在跨 20 个翻译方向的大规模多语言机器翻译模型的实验中，我们证实了该方法对于高资源语言对的有效性，NMT 生成的翻译的文档级准确率达到 82-96%，人类翻译的文档级准确率达到 60-81%翻译，取决于所使用的模型。代码和演示可在 https://github.com/ZurichNLP/translation-direction-detection 获取