2024-01-12

diffusion

Title: From Pampas to Pixels: Fine-Tuning Diffusion Models for Ga\'ucho Heritage. (arXiv:2401.05520v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05520
• Code URL: null
• Copy Paste: [[2401.05520]] From Pampas to Pixels: Fine-Tuning Diffusion Models for Ga\'ucho Heritage(http://arxiv.org/abs/2401.05520)
• Summary:

  Generative AI has become pervasive in society, witnessing significant advancements in various domains. Particularly in the realm of Text-to-Image (TTI) models, Latent Diffusion Models (LDMs), showcase remarkable capabilities in generating visual content based on textual prompts. This paper addresses the potential of LDMs in representing local cultural concepts, historical figures, and endangered species. In this study, we use the cultural heritage of Rio Grande do Sul (RS), Brazil, as an illustrative case. Our objective is to contribute to the broader understanding of how generative models can help to capture and preserve the cultural and historical identity of regions. The paper outlines the methodology, including subject selection, dataset creation, and the fine-tuning process. The results showcase the images generated, alongside the challenges and feasibility of each concept. In conclusion, this work shows the power of these models to represent and preserve unique aspects of diverse regions and communities.

• 摘要：

  生成式人工智能已在社会中普及，并在各个领域取得了重大进展。特别是在文本到图像 (TTI) 模型领域，潜在扩散模型 (LDM) 展示了基于文本提示生成视觉内容的卓越能力。本文探讨了 LDM 在代表当地文化概念、历史人物和濒危物种方面的潜力。在本研究中，我们以巴西南里奥格兰德州（RS）的文化遗产为例。我们的目标是帮助人们更广泛地理解生成模型如何帮助捕捉和保护地区的文化和历史特征。本文概述了该方法，包括主题选择、数据集创建和微调过程。结果展示了生成的图像，以及每个概念的挑战和可行性。总之，这项工作展示了这些模型代表和保护不同地区和社区独特方面的力量。

Title: Diffusion Priors for Dynamic View Synthesis from Monocular Videos. (arXiv:2401.05583v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05583
• Code URL: null
• Copy Paste: [[2401.05583]] Diffusion Priors for Dynamic View Synthesis from Monocular Videos(http://arxiv.org/abs/2401.05583)
• Summary:

  Dynamic novel view synthesis aims to capture the temporal evolution of visual content within videos. Existing methods struggle to distinguishing between motion and structure, particularly in scenarios where camera poses are either unknown or constrained compared to object motion. Furthermore, with information solely from reference images, it is extremely challenging to hallucinate unseen regions that are occluded or partially observed in the given videos. To address these issues, we first finetune a pretrained RGB-D diffusion model on the video frames using a customization technique. Subsequently, we distill the knowledge from the finetuned model to a 4D representations encompassing both dynamic and static Neural Radiance Fields (NeRF) components. The proposed pipeline achieves geometric consistency while preserving the scene identity. We perform thorough experiments to evaluate the efficacy of the proposed method qualitatively and quantitatively. Our results demonstrate the robustness and utility of our approach in challenging cases, further advancing dynamic novel view synthesis.

• 摘要：

  动态新颖视图合成旨在捕捉视频中视觉内容的时间演变。现有的方法很难区分运动和结构，特别是在相机姿势与物体运动相比未知或受到限制的情况下。此外，仅利用参考图像的信息，对给定视频中被遮挡或部分观察到的看不见的区域产生幻觉是极具挑战性的。为了解决这些问题，我们首先使用定制技术在视频帧上微调预训练的 RGB-D 扩散模型。随后，我们将微调模型中的知识提炼为包含动态和静态神经辐射场 (NeRF) 组件的 4D 表示。所提出的管道在保留场景身份的同时实现了几何一致性。我们进行了彻底的实验，以定性和定量地评估所提出方法的有效性。我们的结果证明了我们的方法在具有挑战性的情况下的稳健性和实用性，进一步推进了动态新颖的视图合成。

Title: Object-Centric Diffusion for Efficient Video Editing. (arXiv:2401.05735v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05735
• Code URL: null
• Copy Paste: [[2401.05735]] Object-Centric Diffusion for Efficient Video Editing(http://arxiv.org/abs/2401.05735)
• Summary:

  Diffusion-based video editing have reached impressive quality and can transform either the global style, local structure, and attributes of given video inputs, following textual edit prompts. However, such solutions typically incur heavy memory and computational costs to generate temporally-coherent frames, either in the form of diffusion inversion and/or cross-frame attention. In this paper, we conduct an analysis of such inefficiencies, and suggest simple yet effective modifications that allow significant speed-ups whilst maintaining quality. Moreover, we introduce Object-Centric Diffusion, coined as OCD, to further reduce latency by allocating computations more towards foreground edited regions that are arguably more important for perceptual quality. We achieve this by two novel proposals: i) Object-Centric Sampling, decoupling the diffusion steps spent on salient regions or background, allocating most of the model capacity to the former, and ii) Object-Centric 3D Token Merging, which reduces cost of cross-frame attention by fusing redundant tokens in unimportant background regions. Both techniques are readily applicable to a given video editing model \textit{without} retraining, and can drastically reduce its memory and computational cost. We evaluate our proposals on inversion-based and control-signal-based editing pipelines, and show a latency reduction up to 10x for a comparable synthesis quality.

• 摘要：

  基于扩散的视频编辑已经达到了令人印象深刻的质量，并且可以按照文本编辑提示转换给定视频输入的全局样式、局部结构和属性。然而，这样的解决方案通常会产生大量的内存和计算成本来生成时间相干的帧，无论是采用扩散反转和/或跨帧注意的形式。在本文中，我们对这种低效率进行了分析，并提出了简单而有效的修改建议，可以在保持质量的同时显着提高速度。此外，我们引入了以对象为中心的扩散（被称为 OCD），通过将计算更多地分配给前台编辑区域来进一步减少延迟，这对于感知质量来说可能更重要。我们通过两个新颖的提议来实现这一目标：i) 以对象为中心的采样，解耦在显着区域或背景上花费的扩散步骤，将大部分模型容量分配给前者，以及 ii) 以对象为中心的 3D 令牌合并，这降低了成本通过在不重要的背景区域融合冗余标记来实现跨帧注意力。这两种技术都很容易适用于给定的视频编辑模型\textit{无需}重新训练，并且可以大大减少其内存和计算成本。我们评估了我们关于基于反转和基于控制信号的编辑管道的建议，结果表明，在同等合成质量的情况下，延迟减少了高达 10 倍。

Title: EraseDiff: Erasing Data Influence in Diffusion Models. (arXiv:2401.05779v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05779
• Code URL: null
• Copy Paste: [[2401.05779]] EraseDiff: Erasing Data Influence in Diffusion Models(http://arxiv.org/abs/2401.05779)
• Summary:

  In response to data protection regulations and the ``right to be forgotten'', in this work, we introduce an unlearning algorithm for diffusion models. Our algorithm equips a diffusion model with a mechanism to mitigate the concerns related to data memorization. To achieve this, we formulate the unlearning problem as a bi-level optimization problem, wherein the outer objective is to preserve the utility of the diffusion model on the remaining data. The inner objective aims to scrub the information associated with forgetting data by deviating the learnable generative process from the ground-truth denoising procedure. To solve the resulting bi-level problem, we adopt a first-order method, having superior practical performance while being vigilant about the diffusion process and solving a bi-level problem therein. Empirically, we demonstrate that our algorithm can preserve the model utility, effectiveness, and efficiency while removing across two widely-used diffusion models and in both conditional and unconditional image generation scenarios. In our experiments, we demonstrate the unlearning of classes, attributes, and even a race from face and object datasets such as UTKFace, CelebA, CelebA-HQ, and CIFAR10.

• 摘要：

  为了响应数据保护法规和“被遗忘权”，在这项工作中，我们引入了一种扩散模型的遗忘算法。我们的算法为扩散模型配备了一种机制，以减轻与数据记忆相关的问题。为了实现这一目标，我们将遗忘问题表述为双层优化问题，其中外部目标是保留扩散模型对剩余数据的效用。内部目标旨在通过使可学习的生成过程偏离真实的去噪过程来清除与遗忘数据相关的信息。为了解决由此产生的双层问题，我们采用一阶方法，具有优越的实用性能，同时警惕扩散过程并解决其中的双层问题。根据经验，我们证明我们的算法可以保留模型的实用性、有效性和效率，同时在两种广泛使用的扩散模型以及条件和无条件图像生成场景中进行删除。在我们的实验中，我们演示了如何从 UTKFace、CelebA、CelebA-HQ 和 CIFAR10 等人脸和对象数据集中忘记类别、属性甚至种族。

Title: HiCAST: Highly Customized Arbitrary Style Transfer with Adapter Enhanced Diffusion Models. (arXiv:2401.05870v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05870
• Code URL: null
• Copy Paste: [[2401.05870]] HiCAST: Highly Customized Arbitrary Style Transfer with Adapter Enhanced Diffusion Models(http://arxiv.org/abs/2401.05870)
• Summary:

  The goal of Arbitrary Style Transfer (AST) is injecting the artistic features of a style reference into a given image/video. Existing methods usually focus on pursuing the balance between style and content, whereas ignoring the significant demand for flexible and customized stylization results and thereby limiting their practical application. To address this critical issue, a novel AST approach namely HiCAST is proposed, which is capable of explicitly customizing the stylization results according to various source of semantic clues. In the specific, our model is constructed based on Latent Diffusion Model (LDM) and elaborately designed to absorb content and style instance as conditions of LDM. It is characterized by introducing of \textit{Style Adapter}, which allows user to flexibly manipulate the output results by aligning multi-level style information and intrinsic knowledge in LDM. Lastly, we further extend our model to perform video AST. A novel learning objective is leveraged for video diffusion model training, which significantly improve cross-frame temporal consistency in the premise of maintaining stylization strength. Qualitative and quantitative comparisons as well as comprehensive user studies demonstrate that our HiCAST outperforms the existing SoTA methods in generating visually plausible stylization results.

• 摘要：

  任意风格迁移（AST）的目标是将风格参考的艺术特征注入给定的图像/视频中。现有的方法通常注重追求风格和内容之间的平衡，而忽略了对灵活和定制的风格化结果的巨大需求，从而限制了其实际应用。为了解决这个关键问题，提出了一种新的 AST 方法，即 HiCAST，它能够根据各种语义线索来源显式定制样式化结果。具体来说，我们的模型是基于潜在扩散模型（LDM）构建的，并精心设计以吸收内容和风格实例作为LDM的条件。它的特点是引入了\textit{Style Adapter}，允许用户通过对齐LDM中的多级样式信息和内在知识来灵活地操纵输出结果。最后，我们进一步扩展我们的模型来执行视频 AST。利用新颖的学习目标进行视频扩散模型训练，在保持风格化强度的前提下显着提高跨帧时间一致性。定性和定量比较以及全面的用户研究表明，我们的 HiCAST 在生成视觉上合理的风格化结果方面优于现有的 SoTA 方法。

Title: Efficient Image Deblurring Networks based on Diffusion Models. (arXiv:2401.05907v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05907
• Code URL: https://github.com/bnm6900030/swintormer
• Copy Paste: [[2401.05907]] Efficient Image Deblurring Networks based on Diffusion Models(http://arxiv.org/abs/2401.05907)
• Summary:

  This article introduces a sliding window model for defocus deblurring that achieves the best performance to date with extremely low memory usage. Named Swintormer, the method utilizes a diffusion model to generate latent prior features that assist in restoring more detailed images. It also extends the sliding window strategy to specialized Transformer blocks for efficient inference. Additionally, we have further optimized Multiply-Accumulate operations (Macs). Compared to the currently top-performing GRL method, our Swintormer model drastically reduces computational complexity from 140.35 GMACs to 8.02 GMacs, while also improving the Signal-to-Noise Ratio (SNR) for defocus deblurring from 27.04 dB to 27.07 dB. This new method allows for the processing of higher resolution images on devices with limited memory, significantly expanding potential application scenarios. The article concludes with an ablation study that provides an in-depth analysis of the impact of each network module on final performance. The source code and model will be available at the following website: https://github.com/bnm6900030/swintormer.

• 摘要：

  本文介绍了一种用于散焦去模糊的滑动窗口模型，该模型以极低的内存使用量实现了迄今为止的最佳性能。该方法名为 Swintormer，利用扩散模型生成潜在的先验特征，有助于恢复更详细的图像。它还将滑动窗口策略扩展到专门的 Transformer 块，以实现高效推理。此外，我们还进一步优化了乘法累加运算 (Mac)。与目前表现最好的 GRL 方法相比，我们的 Swintormer 模型将计算复杂度从 140.35 GMAC 大幅降低到 8.02 GMac，同时还将散焦去模糊的信噪比 (SNR) 从 27.04 dB 提高到 27.07 dB。这种新方法允许在内存有限的设备上处理更高分辨率的图像，显​​着扩展了潜在的应用场景。本文最后进行了消融研究，深入分析了每个网络模块对最终性能的影响。源代码和模型可在以下网站获取：https://github.com/bnm6900030/swintormer。

Title: E$^{2}$GAN: Efficient Training of Efficient GANs for Image-to-Image Translation. (arXiv:2401.06127v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.06127
• Code URL: null
• Copy Paste: [[2401.06127]] E$^{2}$GAN: Efficient Training of Efficient GANs for Image-to-Image Translation(http://arxiv.org/abs/2401.06127)
• Summary:

  One highly promising direction for enabling flexible real-time on-device image editing is utilizing data distillation by leveraging large-scale text-to-image diffusion models, such as Stable Diffusion, to generate paired datasets used for training generative adversarial networks (GANs). This approach notably alleviates the stringent requirements typically imposed by high-end commercial GPUs for performing image editing with diffusion models. However, unlike text-to-image diffusion models, each distilled GAN is specialized for a specific image editing task, necessitating costly training efforts to obtain models for various concepts. In this work, we introduce and address a novel research direction: can the process of distilling GANs from diffusion models be made significantly more efficient? To achieve this goal, we propose a series of innovative techniques. First, we construct a base GAN model with generalized features, adaptable to different concepts through fine-tuning, eliminating the need for training from scratch. Second, we identify crucial layers within the base GAN model and employ Low-Rank Adaptation (LoRA) with a simple yet effective rank search process, rather than fine-tuning the entire base model. Third, we investigate the minimal amount of data necessary for fine-tuning, further reducing the overall training time. Extensive experiments show that we can efficiently empower GANs with the ability to perform real-time high-quality image editing on mobile devices with remarkable reduced training cost and storage for each concept.

• 摘要：

  实现灵活的实时设备图像编辑的一个非常有前途的方向是通过利用大规模文本到图像扩散模型（例如稳定扩散）来利用数据蒸馏来生成用于训练生成对抗网络的配对数据集（GAN）。这种方法显着缓解了高端商用 GPU 通常对使用扩散模型执行图像编辑提出的严格要求。然而，与文本到图像的扩散模型不同，每个精炼的 GAN 专门用于特定的图像编辑任务，需要昂贵的训练工作才能获得各种概念的模型。在这项工作中，我们介绍并提出了一个新的研究方向：从扩散模型中提取 GAN 的过程是否可以显着提高效率？为了实现这一目标，我们提出了一系列创新技术。首先，我们构建一个具有通用特征的基础 GAN 模型，通过微调适应不同的概念，从而无需从头开始训练。其次，我们确定基本 GAN 模型中的关键层，并通过简单而有效的排名搜索过程采用低秩适应 (LoRA)，而不是微调整个基本模型。第三，我们研究微调所需的最少量数据，进一步减少总体训练时间。大量实验表明，我们可以有效地赋予 GAN 在移动设备上执行实时高质量图像编辑的能力，并显着降低每个概念的训练成本和存储成本。

self-supervised

Title: Siamese Networks with Soft Labels for Unsupervised Lesion Detection and Patch Pretraining on Screening Mammograms. (arXiv:2401.05570v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05570
• Code URL: null
• Copy Paste: [[2401.05570]] Siamese Networks with Soft Labels for Unsupervised Lesion Detection and Patch Pretraining on Screening Mammograms(http://arxiv.org/abs/2401.05570)
• Summary:

  Self-supervised learning has become a popular way to pretrain a deep learning model and then transfer it to perform downstream tasks. However, most of these methods are developed on large-scale image datasets that contain natural objects with clear textures, outlines, and distinct color contrasts. It remains uncertain whether these methods are equally effective for medical imaging, where the regions of interest often blend subtly and indistinctly with the surrounding tissues. In this study, we propose an alternative method that uses contralateral mammograms to train a neural network to encode similar embeddings when a pair contains both normal images and different embeddings when a pair contains normal and abnormal images. Our approach leverages the natural symmetry of human body as weak labels to learn to distinguish abnormal lesions from background tissues in a fully unsupervised manner. Our findings suggest that it's feasible by incorporating soft labels derived from the Euclidean distances between the embeddings of the image pairs into the Siamese network loss. Our method demonstrates superior performance in mammogram patch classification compared to existing self-supervised learning methods. This approach not only leverages a vast amount of image data effectively but also minimizes reliance on costly labels, a significant advantage particularly in the field of medical imaging.

• 摘要：

  自监督学习已成为预训练深度学习模型然后将其转移以执行下游任务的流行方式。然而，这些方法大多数都是在大规模图像数据集上开发的，这些数据集包含具有清晰纹理、轮廓和明显颜色对比度的自然物体。目前还不确定这些方法对于医学成像是否同样有效，因为感兴趣的区域通常与周围组织巧妙且模糊地融合在一起。在本研究中，我们提出了一种替代方法，即使用对侧乳房 X 光照片来训练神经网络，当一对包含正常图像时编码相似的嵌入，当一对包含正常和异常图像时编码不同的嵌入。我们的方法利用人体的自然对称性作为弱标签，以完全无人监督的方式学习区分异常病变与背景组织。我们的研究结果表明，通过将从图像对嵌入之间的欧几里得距离导出的软标签合并到连体网络损失中是可行的。与现有的自我监督学习方法相比，我们的方法在乳房 X 光照片斑块分类方面表现出优越的性能。这种方法不仅有效地利用大量图像数据，而且最大限度地减少了对昂贵标签的依赖，这是一个显着的优势，尤其是在医学成像领域。

Title: HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition. (arXiv:2401.05698v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05698
• Code URL: https://github.com/sunlicai/hicmae
• Copy Paste: [[2401.05698]] HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition(http://arxiv.org/abs/2401.05698)
• Summary:

  Audio-Visual Emotion Recognition (AVER) has garnered increasing attention in recent years for its critical role in creating emotion-ware intelligent machines. Previous efforts in this area are dominated by the supervised learning paradigm. Despite significant progress, supervised learning is meeting its bottleneck due to the longstanding data scarcity issue in AVER. Motivated by recent advances in self-supervised learning, we propose Hierarchical Contrastive Masked Autoencoder (HiCMAE), a novel self-supervised framework that leverages large-scale self-supervised pre-training on vast unlabeled audio-visual data to promote the advancement of AVER. Following prior arts in self-supervised audio-visual representation learning, HiCMAE adopts two primary forms of self-supervision for pre-training, namely masked data modeling and contrastive learning. Unlike them which focus exclusively on top-layer representations while neglecting explicit guidance of intermediate layers, HiCMAE develops a three-pronged strategy to foster hierarchical audio-visual feature learning and improve the overall quality of learned representations. To verify the effectiveness of HiCMAE, we conduct extensive experiments on 9 datasets covering both categorical and dimensional AVER tasks. Experimental results show that our method significantly outperforms state-of-the-art supervised and self-supervised audio-visual methods, which indicates that HiCMAE is a powerful audio-visual emotion representation learner. Codes and models will be publicly available at https://github.com/sunlicai/HiCMAE.

• 摘要：

  近年来，视听情感识别 (AVER) 因其在创建情感感知智能机器方面的关键作用而受到越来越多的关注。此前该领域的工作主要以监督学习范式为主。尽管取得了重大进展，但由于 AVER 长期存在的数据稀缺问题，监督学习正在遇到瓶颈。受自监督学习最新进展的推动，我们提出了分层对比掩模自动编码器（HiCMAE），这是一种新颖的自监督框架，利用对大量未标记的视听数据进行大规模自监督预训练来促进 AVER 的进步。遵循自监督视听表示学习的现有技术，HiCMAE 采用两种主要的自监督形式进行预训练，即掩码数据建模和对比学习。与只关注顶层表示而忽略中间层的显式指导不同，HiCMAE 开发了一种三管齐下的策略来促进分层视听特征学习并提高学习表示的整体质量。为了验证 HiCMAE 的有效性，我们对涵盖分类和维度 AVER 任务的 9 个数据集进行了广泛的实验。实验结果表明，我们的方法显着优于最先进的监督和自监督视听方法，这表明 HiCMAE 是一种强大的视听情感表示学习器。代码和模型将在 https://github.com/sunlicai/HiCMAE 上公开提供。

Title: Enhancing Essay Scoring with Adversarial Weights Perturbation and Metric-specific AttentionPooling. (arXiv:2401.05433v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05433
• Code URL: null
• Copy Paste: [[2401.05433]] Enhancing Essay Scoring with Adversarial Weights Perturbation and Metric-specific AttentionPooling(http://arxiv.org/abs/2401.05433)
• Summary:

  The objective of this study is to improve automated feedback tools designed for English Language Learners (ELLs) through the utilization of data science techniques encompassing machine learning, natural language processing, and educational data analytics. Automated essay scoring (AES) research has made strides in evaluating written essays, but it often overlooks the specific needs of English Language Learners (ELLs) in language development. This study explores the application of BERT-related techniques to enhance the assessment of ELLs' writing proficiency within AES.

  To address the specific needs of ELLs, we propose the use of DeBERTa, a state-of-the-art neural language model, for improving automated feedback tools. DeBERTa, pretrained on large text corpora using self-supervised learning, learns universal language representations adaptable to various natural language understanding tasks. The model incorporates several innovative techniques, including adversarial training through Adversarial Weights Perturbation (AWP) and Metric-specific AttentionPooling (6 kinds of AP) for each label in the competition.

  The primary focus of this research is to investigate the impact of hyperparameters, particularly the adversarial learning rate, on the performance of the model. By fine-tuning the hyperparameter tuning process, including the influence of 6AP and AWP, the resulting models can provide more accurate evaluations of language proficiency and support tailored learning tasks for ELLs. This work has the potential to significantly benefit ELLs by improving their English language proficiency and facilitating their educational journey.

• 摘要：

  这项研究的目的是通过利用机器学习、自然语言处理和教育数据分析等数据科学技术来改进为英语学习者 (ELL) 设计的自动反馈工具。自动论文评分 (AES) 研究在评估书面论文方面取得了长足的进步，但它经常忽视英语学习者 (ELL) 在语言发展方面的具体需求。本研究探讨了 BERT 相关技术的应用，以增强 AES 中 ELL 写作能力的评估。

  为了满足 ELL 的特定需求，我们建议使用最先进的神经语言模型 DeBERTa 来改进自动反馈工具。 DeBERTa 使用自我监督学习在大型文本语料库上进行预训练，学习适用于各种自然语言理解任务的通用语言表示。该模型融合了多项创新技术，包括通过对抗性权重扰动（AWP）进行对抗性训练，以及针对竞赛中每个标签的特定指标注意力池（6种AP）。

  这项研究的主要重点是研究超参数（特别是对抗性学习率）对模型性能的影响。通过微调超参数调整过程，包括 6AP 和 AWP 的影响，生成的模型可以提供更准确的语言能力评估，并支持 ELL 的定制学习任务。这项工作有可能通过提高 ELL 的英语语言能力并促进他们的教育旅程而使他们受益匪浅。

Title: Learning Performance-Oriented Control Barrier Functions Under Complex Safety Constraints and Limited Actuation. (arXiv:2401.05629v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05629
• Code URL: null
• Copy Paste: [[2401.05629]] Learning Performance-Oriented Control Barrier Functions Under Complex Safety Constraints and Limited Actuation(http://arxiv.org/abs/2401.05629)
• Summary:

  Control Barrier Functions (CBFs) provide an elegant framework for designing safety filters for nonlinear control systems by constraining their trajectories to an invariant subset of a prespecified safe set. However, the task of finding a CBF that concurrently maximizes the volume of the resulting control invariant set while accommodating complex safety constraints, particularly in high relative degree systems with actuation constraints, continues to pose a substantial challenge. In this work, we propose a novel self-supervised learning framework that holistically addresses these hurdles. Given a Boolean composition of multiple state constraints that define the safe set, our approach starts with building a single continuously differentiable function whose 0-superlevel set provides an inner approximation of the safe set. We then use this function together with a smooth neural network to parameterize the CBF candidate. Finally, we design a training loss function based on a Hamilton-Jacobi partial differential equation to train the CBF while enlarging the volume of the induced control invariant set. We demonstrate the effectiveness of our approach via numerical experiments.

• 摘要：

  控制屏障函数 (CBF) 通过将非线性控制系统的轨迹限制为预先指定的安全集的不变子集，为设计非线性控制系统的安全滤波器提供了一个优雅的框架。然而，找到一个CBF，同时最大化所得控制不变集的体积，同时适应复杂的安全约束，特别是在具有致动约束的高相对度系统中，仍然是一个巨大的挑战。在这项工作中，我们提出了一种新颖的自我监督学习框架，可以全面解决这些障碍。给定定义安全集的多个状态约束的布尔组合，我们的方法首先构建一个连续可微函数，其 0 超级别集提供安全集的内部近似。然后，我们将该函数与平滑神经网络一起使用来参数化 CBF 候选者。最后，我们设计了基于 Hamilton-Jacobi 偏微分方程的训练损失函数来训练 CBF，同时扩大诱导控制不变集的体积。我们通过数值实验证明了我们方法的有效性。

foundation model

Title: Surgical-DINO: Adapter Learning of Foundation Model for Depth Estimation in Endoscopic Surgery. (arXiv:2401.06013v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.06013
• Code URL: https://github.com/beileicui/surgicaldino
• Copy Paste: [[2401.06013]] Surgical-DINO: Adapter Learning of Foundation Model for Depth Estimation in Endoscopic Surgery(http://arxiv.org/abs/2401.06013)
• Summary:

  Purpose: Depth estimation in robotic surgery is vital in 3D reconstruction, surgical navigation and augmented reality visualization. Although the foundation model exhibits outstanding performance in many vision tasks, including depth estimation (e.g., DINOv2), recent works observed its limitations in medical and surgical domain-specific applications. This work presents a low-ranked adaptation (LoRA) of the foundation model for surgical depth estimation. Methods: We design a foundation model-based depth estimation method, referred to as Surgical-DINO, a low-rank adaptation of the DINOv2 for depth estimation in endoscopic surgery. We build LoRA layers and integrate them into DINO to adapt with surgery-specific domain knowledge instead of conventional fine-tuning. During training, we freeze the DINO image encoder, which shows excellent visual representation capacity, and only optimize the LoRA layers and depth decoder to integrate features from the surgical scene. Results: Our model is extensively validated on a MICCAI challenge dataset of SCARED, which is collected from da Vinci Xi endoscope surgery. We empirically show that Surgical-DINO significantly outperforms all the state-of-the-art models in endoscopic depth estimation tasks. The analysis with ablation studies has shown evidence of the remarkable effect of our LoRA layers and adaptation. Conclusion: Surgical-DINO shed some light on the successful adaptation of the foundation models into the surgical domain for depth estimation. There is clear evidence in the results that zero-shot prediction on pre-trained weights in computer vision datasets or naive fine-tuning is not sufficient to use the foundation model in the surgical domain directly. Code is available at https://github.com/BeileiCui/SurgicalDINO.

• 摘要：

  目的：机器人手术中的深度估计对于 3D 重建、手术导航和增强现实可视化至关重要。尽管基础模型在许多视觉任务中表现出出色的性能，包括深度估计（例如 DINOv2），但最近的工作观察到其在医疗和外科领域特定应用中的局限性。这项工作提出了用于手术深度估计的基础模型的低阶适应（LoRA）。方法：我们设计了一种基于基础模型的深度估计方法，称为 Surgical-DINO，是 DINOv2 的低阶改编，用于内窥镜手术中的深度估计。我们构建 LoRA 层并将其集成到 DINO 中，以适应手术特定的领域知识，而不是传统的微调。在训练过程中，我们冻结了 DINO 图像编码器，该编码器显示出出色的视觉表示能力，并且仅优化 LoRA 层和深度解码器以集成手术场景的特征。结果：我们的模型在 SCARED 的 MICCAI 挑战数据集上得到了广泛验证，该数据集是从达芬奇 Xi 内窥镜手术中收集的。我们的经验表明，在内窥镜深度估计任务中，Surgical-DINO 显着优于所有最先进的模型。消融研究的分析证明了 LoRA 层和适应的显着效果。结论：Surgical-DINO 为将基础模型成功应用于手术领域进行深度估计提供了一些启示。结果中有明确的证据表明，对计算机视觉数据集中预训练权重的零样本预测或朴素微调不足以直接在外科领域使用基础模型。代码可在 https://github.com/BeileiCui/SurgicalDINO 获取。

Title: VI-PANN: Harnessing Transfer Learning and Uncertainty-Aware Variational Inference for Improved Generalization in Audio Pattern Recognition. (arXiv:2401.05531v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05531
• Code URL: null
• Copy Paste: [[2401.05531]] VI-PANN: Harnessing Transfer Learning and Uncertainty-Aware Variational Inference for Improved Generalization in Audio Pattern Recognition(http://arxiv.org/abs/2401.05531)
• Summary:

  Transfer learning (TL) is an increasingly popular approach to training deep learning (DL) models that leverages the knowledge gained by training a foundation model on diverse, large-scale datasets for use on downstream tasks where less domain- or task-specific data is available. The literature is rich with TL techniques and applications; however, the bulk of the research makes use of deterministic DL models which are often uncalibrated and lack the ability to communicate a measure of epistemic (model) uncertainty in prediction. Unlike their deterministic counterparts, Bayesian DL (BDL) models are often well-calibrated, provide access to epistemic uncertainty for a prediction, and are capable of achieving competitive predictive performance. In this study, we propose variational inference pre-trained audio neural networks (VI-PANNs). VI-PANNs are a variational inference variant of the popular ResNet-54 architecture which are pre-trained on AudioSet, a large-scale audio event detection dataset. We evaluate the quality of the resulting uncertainty when transferring knowledge from VI-PANNs to other downstream acoustic classification tasks using the ESC-50, UrbanSound8K, and DCASE2013 datasets. We demonstrate, for the first time, that it is possible to transfer calibrated uncertainty information along with knowledge from upstream tasks to enhance a model's capability to perform downstream tasks.

• 摘要：

  迁移学习（TL）是一种越来越流行的训练深度学习（DL）模型的方法，它利用在多样化的大规模数据集上训练基础模型所获得的知识，用于域或任务较少的下游任务。具体数据可查。文献中有丰富的 TL 技术和应用；然而，大部分研究都使用确定性深度学习模型，这些模型通常未经校准，并且缺乏在预测中传达认知（模型）不确定性度量的能力。与确定性模型不同，贝叶斯深度学习 (BDL) 模型通常经过良好校准，可以获取预测的认知不确定性，并且能够实现有竞争力的预测性能。在本研究中，我们提出了变分推理预训练音频神经网络（VI-PANN）。 VI-PANN 是流行的 ResNet-54 架构的变分推理变体，它在大规模音频事件检测数据集 AudioSet 上进行了预训练。当使用 ESC-50、UrbanSound8K 和 DCASE2013 数据集将知识从 VI-PANN 转移到其他下游声学分类任务时，我们评估了由此产生的不确定性的质量。我们首次证明，可以将校准的不确定性信息与来自上游任务的知识一起传输，以增强模型执行下游任务的能力。

generative

Title: Evaluating Data Augmentation Techniques for Coffee Leaf Disease Classification. (arXiv:2401.05768v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05768
• Code URL: null
• Copy Paste: [[2401.05768]] Evaluating Data Augmentation Techniques for Coffee Leaf Disease Classification(http://arxiv.org/abs/2401.05768)
• Summary:

  The detection and classification of diseases in Robusta coffee leaves are essential to ensure that plants are healthy and the crop yield is kept high. However, this job requires extensive botanical knowledge and much wasted time. Therefore, this task and others similar to it have been extensively researched subjects in image classification. Regarding leaf disease classification, most approaches have used the more popular PlantVillage dataset while completely disregarding other datasets, like the Robusta Coffee Leaf (RoCoLe) dataset. As the RoCoLe dataset is imbalanced and does not have many samples, fine-tuning of pre-trained models and multiple augmentation techniques need to be used. The current paper uses the RoCoLe dataset and approaches based on deep learning for classifying coffee leaf diseases from images, incorporating the pix2pix model for segmentation and cycle-generative adversarial network (CycleGAN) for augmentation. Our study demonstrates the effectiveness of Transformer-based models, online augmentations, and CycleGAN augmentation in improving leaf disease classification. While synthetic data has limitations, it complements real data, enhancing model performance. These findings contribute to developing robust techniques for plant disease detection and classification.

• 摘要：

  罗布斯塔咖啡叶病害的检测和分类对于确保植物健康和保持作物高产至关重要。然而，这项工作需要广泛的植物知识并且浪费大量时间。因此，该任务和其他类似任务已成为图像分类领域广泛研究的课题。关于叶病分类，大多数方法都使用更流行的 PlantVillage 数据集，而完全忽略其他数据集，例如 Robusta Coffee Leaf (RoCoLe) 数据集。由于RoCoLe数据集不平衡且样本不多，需要使用预训练模型的微调和多种增强技术。当前的论文使用 RoCoLe 数据集和基于深度学习的方法对图像中的咖啡叶病进行分类，并结合用于分割的 pix2pix 模型和用于增强的循环生成对抗网络 (CycleGAN)。我们的研究证明了基于 Transformer 的模型、在线增强和 CycleGAN 增强在改善叶病分类方面的有效性。虽然合成数据有局限性，但它补充了真实数据，提高了模型性能。这些发现有助于开发强大的植物病害检测和分类技术。

Title: An attempt to generate new bridge types from latent space of PixelCNN. (arXiv:2401.05964v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05964
• Code URL: https://github.com/QQ583304953/Bridge-PixelCNN
• Copy Paste: [[2401.05964]] An attempt to generate new bridge types from latent space of PixelCNN(http://arxiv.org/abs/2401.05964)
• Summary:

  Try to generate new bridge types using generative artificial intelligence technology. Using symmetric structured image dataset of three-span beam bridge, arch bridge, cable-stayed bridge and suspension bridge , based on Python programming language, TensorFlow and Keras deep learning platform framework , PixelCNN is constructed and trained. The model can capture the statistical structure of the images and calculate the probability distribution of the next pixel when the previous pixels are given. From the obtained latent space sampling, new bridge types different from the training dataset can be generated. PixelCNN can organically combine different structural components on the basis of human original bridge types, creating new bridge types that have a certain degree of human original ability. Autoregressive models cannot understand the meaning of the sequence, while multimodal models combine regression and autoregressive models to understand the sequence. Multimodal models should be the way to achieve artificial general intelligence in the future.

• 摘要：

  尝试使用生成人工智能技术生成新的桥梁类型。利用三跨梁桥、拱桥、斜拉桥、悬索桥的对称结构化图像数据集，基于Python编程语言、TensorFlow和Keras深度学习平台框架，构建并训练PixelCNN。该模型可以捕获图像的统计结构，并在给定先前像素的情况下计算下一个像素的概率分布。根据获得的潜在空间采样，可以生成与训练数据集不同的新桥梁类型。 PixelCNN可以在人类原创桥梁类型的基础上有机地组合不同的结构组件，创造出具有一定人类原创能力的新桥梁类型。自回归模型无法理解序列的含义，而多模态模型则结合回归和自回归模型来理解序列。多模态模型应该是未来实现通用人工智能的途径。

Title: GE-AdvGAN: Improving the transferability of adversarial samples by gradient editing-based adversarial generative model. (arXiv:2401.06031v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.06031
• Code URL: https://github.com/lmbtough/ge-advgan
• Copy Paste: [[2401.06031]] GE-AdvGAN: Improving the transferability of adversarial samples by gradient editing-based adversarial generative model(http://arxiv.org/abs/2401.06031)
• Summary:

  Adversarial generative models, such as Generative Adversarial Networks (GANs), are widely applied for generating various types of data, i.e., images, text, and audio. Accordingly, its promising performance has led to the GAN-based adversarial attack methods in the white-box and black-box attack scenarios. The importance of transferable black-box attacks lies in their ability to be effective across different models and settings, more closely aligning with real-world applications. However, it remains challenging to retain the performance in terms of transferable adversarial examples for such methods. Meanwhile, we observe that some enhanced gradient-based transferable adversarial attack algorithms require prolonged time for adversarial sample generation. Thus, in this work, we propose a novel algorithm named GE-AdvGAN to enhance the transferability of adversarial samples whilst improving the algorithm's efficiency. The main approach is via optimising the training process of the generator parameters. With the functional and characteristic similarity analysis, we introduce a novel gradient editing (GE) mechanism and verify its feasibility in generating transferable samples on various models. Moreover, by exploring the frequency domain information to determine the gradient editing direction, GE-AdvGAN can generate highly transferable adversarial samples while minimizing the execution time in comparison to the state-of-the-art transferable adversarial attack algorithms. The performance of GE-AdvGAN is comprehensively evaluated by large-scale experiments on different datasets, which results demonstrate the superiority of our algorithm. The code for our algorithm is available at: https://github.com/LMBTough/GE-advGAN

• 摘要：

  对抗生成模型，例如生成对抗网络（GAN），广泛应用于生成各种类型的数据，即图像、文本和音频。因此，其令人鼓舞的性能催生了白盒和黑盒攻击场景中基于 GAN 的对抗攻击方法。可转移黑盒攻击的重要性在于它们能够在不同的模型和设置中发挥作用，与现实世界的应用程序更加紧密地结合。然而，保持此类方法在可转移对抗样本方面的性能仍然具有挑战性。同时，我们观察到一些增强的基于梯度的可转移对抗攻击算法需要更长的时间来生成对抗样本。因此，在这项工作中，我们提出了一种名为 GE-AdvGAN 的新算法，以增强对抗样本的可转移性，同时提高算法的效率。主要方法是通过优化生成器参数的训练过程。通过功能和特征相似性分析，我们引入了一种新颖的梯度编辑（GE）机制，并验证了其在各种模型上生成可转移样本的可行性。此外，通过探索频域信息来确定梯度编辑方向，GE-AdvGAN 可以生成高度可转移的对抗样本，同时与最先进的可转移对抗攻击算法相比，最大限度地减少执行时间。通过在不同数据集上的大规模实验对GE-AdvGAN的性能进行了综合评估，结果证明了我们算法的优越性。我们算法的代码位于：https://github.com/LMBTough/GE-advGAN

Title: RAVEN: Rethinking Adversarial Video Generation with Efficient Tri-plane Networks. (arXiv:2401.06035v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.06035
• Code URL: null
• Copy Paste: [[2401.06035]] RAVEN: Rethinking Adversarial Video Generation with Efficient Tri-plane Networks(http://arxiv.org/abs/2401.06035)
• Summary:

  We present a novel unconditional video generative model designed to address long-term spatial and temporal dependencies. To capture these dependencies, our approach incorporates a hybrid explicit-implicit tri-plane representation inspired by 3D-aware generative frameworks developed for three-dimensional object representation and employs a singular latent code to model an entire video sequence. Individual video frames are then synthesized from an intermediate tri-plane representation, which itself is derived from the primary latent code. This novel strategy reduces computational complexity by a factor of $2$ as measured in FLOPs. Consequently, our approach facilitates the efficient and temporally coherent generation of videos. Moreover, our joint frame modeling approach, in contrast to autoregressive methods, mitigates the generation of visual artifacts. We further enhance the model's capabilities by integrating an optical flow-based module within our Generative Adversarial Network (GAN) based generator architecture, thereby compensating for the constraints imposed by a smaller generator size. As a result, our model is capable of synthesizing high-fidelity video clips at a resolution of $256\times256$ pixels, with durations extending to more than $5$ seconds at a frame rate of 30 fps. The efficacy and versatility of our approach are empirically validated through qualitative and quantitative assessments across three different datasets comprising both synthetic and real video clips.

• 摘要：

  我们提出了一种新颖的无条件视频生成模型，旨在解决长期的空间和时间依赖性。为了捕获这些依赖性，我们的方法采用了混合显式-隐式三平面表示，其灵感来自于为三维对象表示而开发的 3D 感知生成框架，并采用单一潜在代码来建模整个视频序列。然后从中间三平面表示合成各个视频帧，该中间三平面表示本身是从主要潜在代码导出的。这种新颖的策略将计算复杂性降低了 2 美元（以 FLOP 为单位）。因此，我们的方法有助于高效且时间连贯地生成视频。此外，与自回归方法相比，我们的联合帧建模方法可以减少视觉伪影的产生。我们通过将基于光流的模块集成到基于生成对抗网络（GAN）的生成器架构中，进一步增强了模型的功能，从而补偿了较小生成器尺寸所带来的限制。因此，我们的模型能够以 256\times256$ 像素的分辨率合成高保真视频剪辑，持续时间在 30 fps 的帧速率下延长到超过 5$ 秒。我们的方法的有效性和多功能性通过对三个不同数据集（包括合成视频剪辑和真实视频剪辑）的定性和定量评估进行了实证验证。

Title: Designing Heterogeneous LLM Agents for Financial Sentiment Analysis. (arXiv:2401.05799v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05799
• Code URL: null
• Copy Paste: [[2401.05799]] Designing Heterogeneous LLM Agents for Financial Sentiment Analysis(http://arxiv.org/abs/2401.05799)
• Summary:

  Large language models (LLMs) have drastically changed the possible ways to design intelligent systems, shifting the focuses from massive data acquisition and new modeling training to human alignment and strategical elicitation of the full potential of existing pre-trained models. This paradigm shift, however, is not fully realized in financial sentiment analysis (FSA), due to the discriminative nature of this task and a lack of prescriptive knowledge of how to leverage generative models in such a context. This study investigates the effectiveness of the new paradigm, i.e., using LLMs without fine-tuning for FSA. Rooted in Minsky's theory of mind and emotions, a design framework with heterogeneous LLM agents is proposed. The framework instantiates specialized agents using prior domain knowledge of the types of FSA errors and reasons on the aggregated agent discussions. Comprehensive evaluation on FSA datasets show that the framework yields better accuracies, especially when the discussions are substantial. This study contributes to the design foundations and paves new avenues for LLMs-based FSA. Implications on business and management are also discussed.

• 摘要：

  大型语言模型 (LLM) 极大地改变了设计智能系统的可能方式，将重点从海量数据采集和新的建模训练转移到人类调整和战略性激发现有预训练模型的全部潜力。然而，由于这项任务的歧视性以及缺乏如何在这种背景下利用生成模型的规范性知识，这种范式转变在金融情绪分析（FSA）中并未完全实现。本研究调查了新范式的有效性，即使用法学硕士而不对 FSA 进行微调。植根于明斯基的心灵和情感理论，提出了一种具有异构 LLM 代理的设计框架。该框架使用 FSA 错误类型的先验领域知识以及聚合代理讨论的原因来实例化专用代理。对 FSA 数据集的综合评估表明，该框架具有更好的准确性，特别是在讨论大量时。这项研究为基于法学硕士的 FSA 奠定了设计基础并铺平了新的途径。还讨论了对业务和管理的影响。

Title: Tuning LLMs with Contrastive Alignment Instructions for Machine Translation in Unseen, Low-resource Languages. (arXiv:2401.05811v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05811
• Code URL: null
• Copy Paste: [[2401.05811]] Tuning LLMs with Contrastive Alignment Instructions for Machine Translation in Unseen, Low-resource Languages(http://arxiv.org/abs/2401.05811)
• Summary:

  This article introduces contrastive alignment instructions (AlignInstruct) to address two challenges in machine translation (MT) on large language models (LLMs). One is the expansion of supported languages to previously unseen ones. The second relates to the lack of data in low-resource languages. Model fine-tuning through MT instructions (MTInstruct) is a straightforward approach to the first challenge. However, MTInstruct is limited by weak cross-lingual signals inherent in the second challenge. AlignInstruct emphasizes cross-lingual supervision via a cross-lingual discriminator built using statistical word alignments. Our results based on fine-tuning the BLOOMZ models (1b1, 3b, and 7b1) in up to 24 unseen languages showed that: (1) LLMs can effectively translate unseen languages using MTInstruct; (2) AlignInstruct led to consistent improvements in translation quality across 48 translation directions involving English; (3) Discriminator-based instructions outperformed their generative counterparts as cross-lingual instructions; (4) AlignInstruct improved performance in 30 zero-shot directions.

• 摘要：

  本文介绍了对比对齐指令 (AlignInstruct)，以解决大型语言模型 (LLM) 上机器翻译 (MT) 的两个挑战。一是将支持的语言扩展到以前未见过的语言。第二个与缺乏资源语言的数据有关。通过 MT 指令 (MTInstruct) 进行模型微调是应对第一个挑战的简单方法。然而，MTInstruct 受到第二个挑战中固有的微弱跨语言信号的限制。 AlignInstruct 强调通过使用统计单词对齐构建的跨语言鉴别器进行跨语言监督。我们基于最多 24 种未见过的语言对 BLOOMZ 模型（1b1、3b 和 7b1）进行微调的结果表明：（1）法学硕士可以使用 MTInstruct 有效翻译未见的语言； (2) AlignInstruct 导致 48 个涉及英语的翻译方向的翻译质量持续提高； (3) 作为跨语言指令，基于判别器的指令优于生成指令； (4) AlignInstruct 改进了 30 个零样本方向的性能。

Title: Generative Deduplication For Socia Media Data Selection. (arXiv:2401.05883v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05883
• Code URL: null
• Copy Paste: [[2401.05883]] Generative Deduplication For Socia Media Data Selection(http://arxiv.org/abs/2401.05883)
• Summary:

  Social media data is plagued by the redundancy problem caused by its noisy nature, leading to increased training time and model bias. To address this issue, we propose a novel approach called generative duplication. It aims to remove duplicate text from noisy social media data and mitigate model bias. By doing so, it can improve social media language understanding performance and save training time. Extensive experiments demonstrate that the proposed generative deduplication can effectively reduce training samples while improving performance. This evidence suggests the effectiveness of generative deduplication and its importance in social media language understanding.

• 摘要：

  社交媒体数据因其噪声性质而受到冗余问题的困扰，导致训练时间增加和模型偏差。为了解决这个问题，我们提出了一种称为生成复制的新方法。它的目的是从嘈杂的社交媒体数据中删除重复的文本并减轻模型偏差。通过这样做，它可以提高社交媒体语言理解性能并节省培训时间。大量实验表明，所提出的生成重复数据删除可以有效减少训练样本，同时提高性能。这一证据表明生成式重复数据删除的有效性及其在社交媒体语言理解中的重要性。

Title: EpilepsyLLM: Domain-Specific Large Language Model Fine-tuned with Epilepsy Medical Knowledge. (arXiv:2401.05908v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05908
• Code URL: null
• Copy Paste: [[2401.05908]] EpilepsyLLM: Domain-Specific Large Language Model Fine-tuned with Epilepsy Medical Knowledge(http://arxiv.org/abs/2401.05908)
• Summary:

  With large training datasets and massive amounts of computing sources, large language models (LLMs) achieve remarkable performance in comprehensive and generative ability. Based on those powerful LLMs, the model fine-tuned with domain-specific datasets posseses more specialized knowledge and thus is more practical like medical LLMs. However, the existing fine-tuned medical LLMs are limited to general medical knowledge with English language. For disease-specific problems, the model's response is inaccurate and sometimes even completely irrelevant, especially when using a language other than English. In this work, we focus on the particular disease of Epilepsy with Japanese language and introduce a customized LLM termed as EpilepsyLLM. Our model is trained from the pre-trained LLM by fine-tuning technique using datasets from the epilepsy domain. The datasets contain knowledge of basic information about disease, common treatment methods and drugs, and important notes in life and work. The experimental results demonstrate that EpilepsyLLM can provide more reliable and specialized medical knowledge responses.

• 摘要：

  凭借庞大的训练数据集和海量的计算源，大型语言模型（LLM）在综合能力和生成能力方面表现出色。基于这些强大的LLM，用特定领域的数据集进行微调的模型拥有更专业的知识，因此比医学LLM更实用。然而，现有的微调医学法学硕士仅限于英语语言的一般医学知识。对于特定疾病的问题，模型的响应不准确，有时甚至完全无关，尤其是在使用英语以外的语言时。在这项工作中，我们专注于日语中的癫痫这一特殊疾病，并引入了名为 EpilepsyLLM 的定制法学硕士。我们的模型是通过使用癫痫领域的数据集进行微调技术，从预训练的法学硕士进行训练的。数据集包含疾病的基本信息、常用治疗方法和药物的知识以及生活和工作中的重要注意事项。实验结果表明EpilepsyLLM可以提供更可靠、更专业的医学知识应答。

Title: Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint. (arXiv:2401.06081v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.06081
• Code URL: null
• Copy Paste: [[2401.06081]] Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint(http://arxiv.org/abs/2401.06081)
• Summary:

  Reinforcement learning (RL) has been widely used in training large language models~(LLMs) for preventing unexpected outputs, \eg reducing harmfulness and errors. However, existing RL methods mostly adopt the instance-level reward, which is unable to provide fine-grained supervision for complex reasoning tasks, and can not focus on the few key tokens that lead to the incorrectness. To address it, we propose a new RL method named \textbf{RLMEC} that incorporates a generative model as the reward model, which is trained by the erroneous solution rewriting task under the minimum editing constraint, and can produce token-level rewards for RL training. Based on the generative reward model, we design the token-level RL objective for training and an imitation-based regularization for stabilizing RL process. And the both objectives focus on the learning of the key tokens for the erroneous solution, reducing the effect of other unimportant tokens. The experiment results on mathematical tasks and question-answering tasks have demonstrated the effectiveness of our approach. Our code and data are available at \url{https://github.com/RUCAIBox/RLMEC}.

• 摘要：

  强化学习（RL）已广泛用于训练大型语言模型〜（LLM）以防止意外输出，例如减少危害和错误。然而，现有的强化学习方法大多采用实例级奖励，无法为复杂的推理任务提供细粒度的监督，也无法关注导致错误的少数关键标记。为了解决这个问题，我们提出了一种名为 \textbf{RLMEC} 的新 RL 方法，该方法将生成模型作为奖励模型，在最小编辑约束下通过错误解重写任务进行训练，并且可以为 RL 产生 token 级奖励训练。基于生成奖励模型，我们设计了用于训练的代币级强化学习目标，以及用于稳定强化学习过程的基于模仿的正则化。这两个目标都集中在学习错误解决方案的关键标记，减少其他不重要标记的影响。数学任务和问答任务的实验结果证明了我们方法的有效性。我们的代码和数据可在 \url{https://github.com/RUCAIBox/RLMEC} 获取。

Title: Autocompletion of Chief Complaints in the Electronic Health Records using Large Language Models. (arXiv:2401.06088v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.06088
• Code URL: null
• Copy Paste: [[2401.06088]] Autocompletion of Chief Complaints in the Electronic Health Records using Large Language Models(http://arxiv.org/abs/2401.06088)
• Summary:

  The Chief Complaint (CC) is a crucial component of a patient's medical record as it describes the main reason or concern for seeking medical care. It provides critical information for healthcare providers to make informed decisions about patient care. However, documenting CCs can be time-consuming for healthcare providers, especially in busy emergency departments. To address this issue, an autocompletion tool that suggests accurate and well-formatted phrases or sentences for clinical notes can be a valuable resource for triage nurses. In this study, we utilized text generation techniques to develop machine learning models using CC data. In our proposed work, we train a Long Short-Term Memory (LSTM) model and fine-tune three different variants of Biomedical Generative Pretrained Transformers (BioGPT), namely microsoft/biogpt, microsoft/BioGPT-Large, and microsoft/BioGPT-Large-PubMedQA. Additionally, we tune a prompt by incorporating exemplar CC sentences, utilizing the OpenAI API of GPT-4. We evaluate the models' performance based on the perplexity score, modified BERTScore, and cosine similarity score. The results show that BioGPT-Large exhibits superior performance compared to the other models. It consistently achieves a remarkably low perplexity score of 1.65 when generating CC, whereas the baseline LSTM model achieves the best perplexity score of 170. Further, we evaluate and assess the proposed models' performance and the outcome of GPT-4.0. Our study demonstrates that utilizing LLMs such as BioGPT, leads to the development of an effective autocompletion tool for generating CC documentation in healthcare settings.

• 摘要：

  主诉 (CC) 是患者医疗记录的重要组成部分，因为它描述了寻求医疗护理的主要原因或担忧。它为医疗保健提供者提供关键信息，以做出有关患者护理的明智决策。然而，对于医疗保健提供者来说，记录 CC 可能非常耗时，尤其是在繁忙的急诊科。为了解决这个问题，自动完成工具可以为临床记录提供准确且格式良好的短语或句子，这对于分诊护士来说可能是宝贵的资源。在本研究中，我们利用文本生成技术来开发使用 CC 数据的机器学习模型。在我们提出的工作中，我们训练了一个长短期记忆（LSTM）模型，并对生物医学生成预训练变压器（BioGPT）的三种不同变体进行了微调，即 microsoft/biogpt、microsoft/BioGPT-Large 和 microsoft/BioGPT-Large -PubMedQA。此外，我们还利用 GPT-4 的 OpenAI API，通过合并示例 CC 句子来调整提示。我们根据困惑度得分、修改后的 BERTScore 和余弦相似度得分来评估模型的性能。结果表明，与其他模型相比，BioGPT-Large 表现出优越的性能。在生成 CC 时，它始终实现了 1.65 的非常低的困惑度分数，而基线 LSTM 模型实现了 170 的最佳困惑度分数。此外，我们评估和评估了所提出的模型的性能和 GPT-4.0 的结果。我们的研究表明，利用 BioGPT 等法学硕士，可以开发出一种有效的自动完成工具，用于在医疗保健环境中生成 CC 文档。

Title: An Augmented Surprise-guided Sequential Learning Framework for Predicting the Melt Pool Geometry. (arXiv:2401.05579v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05579
• Code URL: null
• Copy Paste: [[2401.05579]] An Augmented Surprise-guided Sequential Learning Framework for Predicting the Melt Pool Geometry(http://arxiv.org/abs/2401.05579)
• Summary:

  Metal Additive Manufacturing (MAM) has reshaped the manufacturing industry, offering benefits like intricate design, minimal waste, rapid prototyping, material versatility, and customized solutions. However, its full industry adoption faces hurdles, particularly in achieving consistent product quality. A crucial aspect for MAM's success is understanding the relationship between process parameters and melt pool characteristics. Integrating Artificial Intelligence (AI) into MAM is essential. Traditional machine learning (ML) methods, while effective, depend on large datasets to capture complex relationships, a significant challenge in MAM due to the extensive time and resources required for dataset creation. Our study introduces a novel surprise-guided sequential learning framework, SurpriseAF-BO, signaling a significant shift in MAM. This framework uses an iterative, adaptive learning process, modeling the dynamics between process parameters and melt pool characteristics with limited data, a key benefit in MAM's cyber manufacturing context. Compared to traditional ML models, our sequential learning method shows enhanced predictive accuracy for melt pool dimensions. Further improving our approach, we integrated a Conditional Tabular Generative Adversarial Network (CTGAN) into our framework, forming the CT-SurpriseAF-BO. This produces synthetic data resembling real experimental data, improving learning effectiveness. This enhancement boosts predictive precision without requiring additional physical experiments. Our study demonstrates the power of advanced data-driven techniques in cyber manufacturing and the substantial impact of sequential AI and ML, particularly in overcoming MAM's traditional challenges.

• 摘要：

  金属增材制造 (MAM) 重塑了制造业，提供了复杂设计、最少浪费、快速原型制作、材料多功能性和定制解决方案等优势。然而，其在整个行业的采用面临着障碍，特别是在实现一致的产品质量方面。 MAM 成功的一个关键因素是了解工艺参数和熔池特性之间的关系。将人工智能 (AI) 集成到 MAM 中至关重要。传统的机器学习 (ML) 方法虽然有效，但依赖于大型数据集来捕获复杂的关系，由于创建数据集需要大量的时间和资源，这对 MAM 来说是一个重大挑战。我们的研究引入了一种新颖的惊喜引导顺序学习框架 SurpriseAF-BO，标志着 MAM 的重大转变。该框架采用迭代、自适应学习过程，利用有限的数据对工艺参数和熔池特性之间的动态进行建模，这是 MAM 网络制造环境中的一个关键优势。与传统的机器学习模型相比，我们的顺序学习方法显示出熔池尺寸的预测准确性更高。进一步改进我们的方法，我们将条件表格生成对抗网络（CTGAN）集成到我们的框架中，形成 CT-SurpriseAF-BO。这会产生类似于真实实验数据的合成数据，从而提高学习效率。这一增强功能提高了预测精度，无需额外的物理实验。我们的研究展示了先进数据驱动技术在网络制造中的力量以及顺序人工智能和机器学习的重大影响，特别是在克服 MAM 的传统挑战方面。

anomaly

Title: Video Anomaly Detection and Explanation via Large Language Models. (arXiv:2401.05702v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2401.05702
• Code URL: null
• Copy Paste: [[2401.05702]] Video Anomaly Detection and Explanation via Large Language Models(http://arxiv.org/abs/2401.05702)
• Summary:

  Video Anomaly Detection (VAD) aims to localize abnormal events on the timeline of long-range surveillance videos. Anomaly-scoring-based methods have been prevailing for years but suffer from the high complexity of thresholding and low explanability of detection results. In this paper, we conduct pioneer research on equipping video-based large language models (VLLMs) in the framework of VAD, making the VAD model free from thresholds and able to explain the reasons for the detected anomalies. We introduce a novel network module Long-Term Context (LTC) to mitigate the incapability of VLLMs in long-range context modeling. We design a three-phase training method to improve the efficiency of fine-tuning VLLMs by substantially minimizing the requirements for VAD data and lowering the costs of annotating instruction-tuning data. Our trained model achieves the top performance on the anomaly videos of the UCF-Crime and TAD benchmarks, with the AUC improvements of +3.86\% and +4.96\%, respectively. More impressively, our approach can provide textual explanations for detected anomalies.

• 摘要：

  视频异常检测（VAD）旨在定位远程监控视频时间轴上的异常事件。基于异常评分的方法已经流行多年，但存在阈值复杂度高和检测结果可解释性低的问题。在本文中，我们在VAD框架中装备基于视频的大语言模型（VLLM）进行了开创性的研究，使VAD模型不受阈值限制，并且能够解释检测到的异常的原因。我们引入了一种新颖的网络模块长期上下文（LTC）来缓解 VLLM 在远程上下文建模中的无能。我们设计了一种三阶段训练方法，通过大幅减少对 VAD 数据的需求并降低注释指令调优数据的成本来提高微调 VLLM 的效率。我们训练的模型在 UCF-Crime 和 TAD 基准的异常视频上实现了最佳性能，AUC 分别提高了 +3.86\% 和 +4.96\%。更令人印象深刻的是，我们的方法可以为检测到的异常提供文本解释。

Title: Root Cause Analysis on Energy Efficiency with Transfer Entropy Flow. (arXiv:2401.05664v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05664
• Code URL: null
• Copy Paste: [[2401.05664]] Root Cause Analysis on Energy Efficiency with Transfer Entropy Flow(http://arxiv.org/abs/2401.05664)
• Summary:

  Energy efficiency is a big concern in industrial sectors. Finding the root cause of anomaly state of energy efficiency can help to improve energy efficiency of industrial systems and therefore save energy cost. In this research, we propose to use transfer entropy (TE) for root cause analysis on energy efficiency of industrial systems. A method, called TE flow, is proposed in that a TE flow from physical measurements of each subsystem to the energy efficiency indicator along timeline is considered as causal strength for diagnosing root cause of anomaly states of energy efficiency of a system. The copula entropy-based nonparametric TE estimator is used in the proposed method. We conducted experiments on real data collected from a compressing air system to verify the proposed method. Experimental results show that the TE flow method successfully identified the root cause of the energy (in)efficiency of the system.

• 摘要：

  能源效率是工业部门的一个大问题。找到能源效率异常状态的根本原因，有助于提高工业系统的能源效率，从而节省能源成本。在这项研究中，我们建议使用转移熵（TE）来分析工业系统能源效率的根本原因。提出了一种称为 TE 流的方法，该方法将沿时间线从每个子系统的物理测量到能效指标的 TE 流视为因果强度，用于诊断系统能效异常状态的根本原因。该方法使用基于 copula 熵的非参数 TE 估计器。我们对从压缩空气系统收集的真实数据进行了实验，以验证所提出的方法。实验结果表明，TE流方法成功识别了系统能效（低效）的根本原因。

Title: Graph Spatiotemporal Process for Multivariate Time Series Anomaly Detection with Missing Values. (arXiv:2401.05800v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05800
• Code URL: null
• Copy Paste: [[2401.05800]] Graph Spatiotemporal Process for Multivariate Time Series Anomaly Detection with Missing Values(http://arxiv.org/abs/2401.05800)
• Summary:

  The detection of anomalies in multivariate time series data is crucial for various practical applications, including smart power grids, traffic flow forecasting, and industrial process control. However, real-world time series data is usually not well-structured, posting significant challenges to existing approaches: (1) The existence of missing values in multivariate time series data along variable and time dimensions hinders the effective modeling of interwoven spatial and temporal dependencies, resulting in important patterns being overlooked during model training; (2) Anomaly scoring with irregularly-sampled observations is less explored, making it difficult to use existing detectors for multivariate series without fully-observed values. In this work, we introduce a novel framework called GST-Pro, which utilizes a graph spatiotemporal process and anomaly scorer to tackle the aforementioned challenges in detecting anomalies on irregularly-sampled multivariate time series. Our approach comprises two main components. First, we propose a graph spatiotemporal process based on neural controlled differential equations. This process enables effective modeling of multivariate time series from both spatial and temporal perspectives, even when the data contains missing values. Second, we present a novel distribution-based anomaly scoring mechanism that alleviates the reliance on complete uniform observations. By analyzing the predictions of the graph spatiotemporal process, our approach allows anomalies to be easily detected. Our experimental results show that the GST-Pro method can effectively detect anomalies in time series data and outperforms state-of-the-art methods, regardless of whether there are missing values present in the data. Our code is available: https://github.com/huankoh/GST-Pro.

• 摘要：

  多元时间序列数据中的异常检测对于各种实际应用至关重要，包括智能电网、交通流量预测和工业过程控制。然而，现实世界的时间序列数据通常结构不佳，这对现有方法提出了重大挑战：（1）多元时间序列数据在变量和时间维度上存在缺失值，阻碍了交织的空间和时间依赖性的有效建模，导致模型训练过程中重要的模式被忽略； (2) 对不规则采样观测值的异常评分研究较少，这使得在没有完全观测值的情况下很难将现有的检测器用于多元序列。在这项工作中，我们介绍了一种名为 GST-Pro 的新颖框架，它利用图时空过程和异常评分器来解决上述在不规则采样的多元时间序列上检测异常的挑战。我们的方法包括两个主要部分。首先，我们提出了一种基于神经控制微分方程的图时空过程。即使数据包含缺失值，此过程也可以从空间和时间角度对多元时间序列进行有效建模。其次，我们提出了一种新颖的基于分布的异常评分机制，减轻了对完全统一观察的依赖。通过分析图时空过程的预测，我们的方法可以轻松检测到异常情况。我们的实验结果表明，无论数据中是否存在缺失值，GST-Pro 方法都可以有效地检测时间序列数据中的异常，并且优于最先进的方法。我们的代码可用：https://github.com/huankoh/GST-Pro。

in-context

Title: POMP: Probability-driven Meta-graph Prompter for LLMs in Low-resource Unsupervised Neural Machine Translation. (arXiv:2401.05596v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05596
• Code URL: null
• Copy Paste: [[2401.05596]] POMP: Probability-driven Meta-graph Prompter for LLMs in Low-resource Unsupervised Neural Machine Translation(http://arxiv.org/abs/2401.05596)
• Summary:

  Low-resource languages (LRLs) face challenges in supervised neural machine translation due to limited parallel data, prompting research into unsupervised methods. Unsupervised neural machine translation (UNMT) methods, including back-translation, transfer learning, and pivot-based translation, offer practical solutions for LRL translation, but they are hindered by issues like synthetic data noise, language bias, and error propagation, which can potentially be mitigated by Large Language Models (LLMs). LLMs have advanced NMT with in-context learning (ICL) and supervised fine-tuning methods, but insufficient training data results in poor performance in LRLs. We argue that LLMs can mitigate the linguistic noise with auxiliary languages to improve translations in LRLs. In this paper, we propose Probability-driven Meta-graph Prompter (POMP), a novel approach employing a dynamic, sampling-based graph of multiple auxiliary languages to enhance LLMs' translation capabilities for LRLs. POMP involves constructing a directed acyclic meta-graph for each source language, from which we dynamically sample multiple paths to prompt LLMs to mitigate the linguistic noise and improve translations during training. We use the BLEURT metric to evaluate the translations and back-propagate rewards, estimated by scores, to update the probabilities of auxiliary languages in the paths. Our experiments show significant improvements in the translation quality of three LRLs, demonstrating the effectiveness of our approach.

• 摘要：

  由于并行数据有限，低资源语言 (LRL) 在监督神经机器翻译方面面临挑战，这促使人们对无监督方法进行研究。无监督神经机器翻译 (UNMT) 方法，包括反向翻译、迁移学习和基于枢轴的翻译，为 LRL 翻译提供了实用的解决方案，但它们受到合成数据噪声、语言偏差和错误传播等问题的阻碍，这些问题可能会导致大型语言模型 (LLM) 可能会缓解这一问题。 LLM 拥有带有上下文学习 (ICL) 和监督微调方法的先进 NMT，但训练数据不足会导致 LRL 表现不佳。我们认为法学硕士可以通过辅助语言减轻语言噪音，从而改善法学硕士的翻译。在本文中，我们提出了概率驱动的元图提示器（POMP），这是一种采用动态、基于采样的多种辅助语言图来增强法学硕士对 LRL 的翻译能力的新颖方法。 POMP 涉及为每种源语言构建一个有向非循环元图，我们从中动态采样多个路径，以提示法学硕士在训练期间减轻语言噪音并改进翻译。我们使用 BLEURT 指标来评估翻译和反向传播奖励（通过分数估计），以更新路径中辅助语言的概率。我们的实验表明三个 LRL 的翻译质量显着提高，证明了我们方法的有效性。

Title: Probing Structured Semantics Understanding and Generation of Language Models via Question Answering. (arXiv:2401.05777v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05777
• Code URL: null
• Copy Paste: [[2401.05777]] Probing Structured Semantics Understanding and Generation of Language Models via Question Answering(http://arxiv.org/abs/2401.05777)
• Summary:

  Recent advancement in the capabilities of large language models (LLMs) has triggered a new surge in LLMs' evaluation. Most recent evaluation works tends to evaluate the comprehensive ability of LLMs over series of tasks. However, the deep structure understanding of natural language is rarely explored. In this work, we examine the ability of LLMs to deal with structured semantics on the tasks of question answering with the help of the human-constructed formal language. Specifically, we implement the inter-conversion of natural and formal language through in-context learning of LLMs to verify their ability to understand and generate the structured logical forms. Extensive experiments with models of different sizes and in different formal languages show that today's state-of-the-art LLMs' understanding of the logical forms can approach human level overall, but there still are plenty of room in generating correct logical forms, which suggest that it is more effective to use LLMs to generate more natural language training data to reinforce a small model than directly answering questions with LLMs. Moreover, our results also indicate that models exhibit considerable sensitivity to different formal languages. In general, the formal language with the lower the formalization level, i.e. the more similar it is to natural language, is more LLMs-friendly.

• 摘要：

  最近大型语言模型 (LLM) 功能的进步引发了 LLM 评估的新一轮激增。最近的评估工作倾向于评估法学硕士在一系列任务上的综合能力。然而，自然语言的深层结构理解却很少被探索。在这项工作中，我们研究了法学硕士在人类构建的形式语言的帮助下处理问答任务中的结构化语义的能力。具体来说，我们通过法学硕士的情境学习实现自然语言和形式语言的相互转换，以验证他们理解和生成结构化逻辑形式的能力。对不同规模和不同形式语言的模型进行的大量实验表明，当今最先进的法学硕士对逻辑形式的理解总体上可以接近人类水平，但在生成正确的逻辑形式方面仍然有很大的空间，这表明使用法学硕士生成更多自然语言训练数据来强化小型模型比直接使用法学硕士回答问题更有效。此外，我们的结果还表明模型对不同的形式语言表现出相当的敏感性。一般来说，形式化程度越低的形式语言，即与自然语言越相似，就越适合法学硕士。

Title: Universal Vulnerabilities in Large Language Models: In-context Learning Backdoor Attacks. (arXiv:2401.05949v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2401.05949
• Code URL: https://github.com/shuaizhao95/iclattack
• Copy Paste: [[2401.05949]] Universal Vulnerabilities in Large Language Models: In-context Learning Backdoor Attacks(http://arxiv.org/abs/2401.05949)
• Summary:

  In-context learning, a paradigm bridging the gap between pre-training and fine-tuning, has demonstrated high efficacy in several NLP tasks, especially in few-shot settings. Unlike traditional fine-tuning methods, in-context learning adapts pre-trained models to unseen tasks without updating any parameters. Despite being widely applied, in-context learning is vulnerable to malicious attacks. In this work, we raise security concerns regarding this paradigm. Our studies demonstrate that an attacker can manipulate the behavior of large language models by poisoning the demonstration context, without the need for fine-tuning the model. Specifically, we have designed a new backdoor attack method, named ICLAttack, to target large language models based on in-context learning. Our method encompasses two types of attacks: poisoning demonstration examples and poisoning prompts, which can make models behave in accordance with predefined intentions. ICLAttack does not require additional fine-tuning to implant a backdoor, thus preserving the model's generality. Furthermore, the poisoned examples are correctly labeled, enhancing the natural stealth of our attack method. Extensive experimental results across several language models, ranging in size from 1.3B to 40B parameters, demonstrate the effectiveness of our attack method, exemplified by a high average attack success rate of 95.0% across the three datasets on OPT models. Our findings highlight the vulnerabilities of language models, and we hope this work will raise awareness of the possible security threats associated with in-context learning.

• 摘要：

  上下文学习是一种弥合预训练和微调之间差距的范式，已在多项 NLP 任务中表现出高效能，尤其是在少量样本设置中。与传统的微调方法不同，上下文学习使预先训练的模型适应未见过的任务，而无需更新任何参数。尽管应用广泛，但情境学习很容易受到恶意攻击。在这项工作中，我们提出了有关此范例的安全问题。我们的研究表明，攻击者可以通过毒害演示上下文来操纵大型语言模型的行为，而无需对模型进行微调。具体来说，我们设计了一种新的后门攻击方法，名为 ICLAtack，针对基于上下文学习的大型语言模型。我们的方法包含两种类型的攻击：中毒演示示例和中毒提示，这可以使模型按照预定义的意图行事。 ICLAttack 不需要额外的微调来植入后门，从而保留了模型的通用性。此外，中毒的例子被正确标记，增强了我们的攻击方法的自然隐蔽性。跨多种语言模型的广泛实验结果（参数大小从 1.3B 到 40B 不等）证明了我们的攻击方法的有效性，OPT 模型上的三个数据集的平均攻击成功率高达 95.0%。我们的研究结果强调了语言模型的漏洞，我们希望这项工作能够提高人们对与上下文学习相关的可能安全威胁的认识。

Title: Learning Cognitive Maps from Transformer Representations for Efficient Planning in Partially Observed Environments. (arXiv:2401.05946v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2401.05946
• Code URL: null
• Copy Paste: [[2401.05946]] Learning Cognitive Maps from Transformer Representations for Efficient Planning in Partially Observed Environments(http://arxiv.org/abs/2401.05946)
• Summary:

  Despite their stellar performance on a wide range of tasks, including in-context tasks only revealed during inference, vanilla transformers and variants trained for next-token predictions (a) do not learn an explicit world model of their environment which can be flexibly queried and (b) cannot be used for planning or navigation. In this paper, we consider partially observed environments (POEs), where an agent receives perceptually aliased observations as it navigates, which makes path planning hard. We introduce a transformer with (multiple) discrete bottleneck(s), TDB, whose latent codes learn a compressed representation of the history of observations and actions. After training a TDB to predict the future observation(s) given the history, we extract interpretable cognitive maps of the environment from its active bottleneck(s) indices. These maps are then paired with an external solver to solve (constrained) path planning problems. First, we show that a TDB trained on POEs (a) retains the near perfect predictive performance of a vanilla transformer or an LSTM while (b) solving shortest path problems exponentially faster. Second, a TDB extracts interpretable representations from text datasets, while reaching higher in-context accuracy than vanilla sequence models. Finally, in new POEs, a TDB (a) reaches near-perfect in-context accuracy, (b) learns accurate in-context cognitive maps (c) solves in-context path planning problems.

• 摘要：

  尽管它们在广泛的任务上表现出色，包括仅在推理过程中揭示的上下文任务，但为下一个标记预测训练的普通变压器和变体（a）没有学习其环境的明确的世界模型，这可以是灵活查询，(b) 不能用于规划或导航。在本文中，我们考虑部分观察环境（POE），其中代理在导航时接收到感知混叠的观察结果，这使得路径规划变得困难。我们引入了一个具有（多个）离散瓶颈的变压器 TDB，其潜在代码学习观察和动作历史的压缩表示。在训练 TDB 来预测给定历史的未来观察之后，我们从其活动瓶颈索引中提取环境的可解释认知图。然后将这些地图与外部求解器配对以解决（受限）路径规划问题。首先，我们表明，在 POE 上训练的 TDB (a) 保留了普通 Transformer 或 LSTM 近乎完美的预测性能，同时 (b) 解决最短路径问题的速度呈指数级增长。其次，TDB 从文本数据集中提取可解释的表示，同时达到比普通序列模型更高的上下文准确性。最后，在新的 POE 中，TDB (a) 达到近乎完美的上下文准确性，(b) 学习准确的上下文认知图，(c) 解决上下文路径规划问题。