2025-01-15

Title: BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations

• Authors: Weixi Feng, Chao Liu, Sifei Liu, William Yang Wang, Arash Vahdat, Weili Nie
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.07647
• Pdf URL: https://arxiv.org/pdf/2501.07647
• Copy Paste: [[2501.07647]] BlobGEN-Vid: Compositional Text-to-Video Generation with Blob Video Representations(https://arxiv.org/abs/2501.07647)
• Keywords: generation
• Abstract: Existing video generation models struggle to follow complex text prompts and synthesize multiple objects, raising the need for additional grounding input for improved controllability. In this work, we propose to decompose videos into visual primitives - blob video representation, a general representation for controllable video generation. Based on blob conditions, we develop a blob-grounded video diffusion model named BlobGEN-Vid that allows users to control object motions and fine-grained object appearance. In particular, we introduce a masked 3D attention module that effectively improves regional consistency across frames. In addition, we introduce a learnable module to interpolate text embeddings so that users can control semantics in specific frames and obtain smooth object transitions. We show that our framework is model-agnostic and build BlobGEN-Vid based on both U-Net and DiT-based video diffusion models. Extensive experimental results show that BlobGEN-Vid achieves superior zero-shot video generation ability and state-of-the-art layout controllability on multiple benchmarks. When combined with an LLM for layout planning, our framework even outperforms proprietary text-to-video generators in terms of compositional accuracy.
• 摘要：现有的视频生成模型难以遵循复杂的文本提示并合成多个对象，因此需要额外的基础输入来提高可控性。在这项工作中，我们建议将视频分解为视觉基元 - blob 视频表示，这是可控视频生成的通用表示。基于 blob 条件，我们开发了一个基于 blob 的视频扩散模型，名为 BlobGEN-Vid，允许用户控制对象运动和细粒度对象外观。具体来说，我们引入了一个带掩码的 3D 注意模块，可有效提高跨帧的区域一致性。此外，我们引入了一个可学习的模块来插入文本嵌入，以便用户可以控制特定帧中的语义并获得平滑的对象过渡。我们表明我们的框架与模型无关，并基于 U-Net 和基于 DiT 的视频扩散模型构建 BlobGEN-Vid。大量实验结果表明，BlobGEN-Vid 在多个基准上实现了卓越的零样本视频生成能力和最先进的布局可控性。当与 LLM 结合进行布局规划时，我们的框架在构图准确性方面甚至优于专有的文本到视频生成器。

Title: Dataset Distillation as Pushforward Optimal Quantization

• Authors: Hong Ye Tan, Emma Slade
• Subjects: cs.LG, cs.CV, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.07681
• Pdf URL: https://arxiv.org/pdf/2501.07681
• Copy Paste: [[2501.07681]] Dataset Distillation as Pushforward Optimal Quantization(https://arxiv.org/abs/2501.07681)
• Keywords: generative
• Abstract: Dataset distillation aims to find a synthetic training set such that training on the synthetic data achieves similar performance to training on real data, with orders of magnitude less computational requirements. Existing methods can be broadly categorized as either bi-level optimization problems that have neural network training heuristics as the lower level problem, or disentangled methods that bypass the bi-level optimization by matching distributions of data. The latter method has the major advantages of speed and scalability in terms of size of both training and distilled datasets. We demonstrate that when equipped with an encoder-decoder structure, the empirically successful disentangled methods can be reformulated as an optimal quantization problem, where a finite set of points is found to approximate the underlying probability measure by minimizing the expected projection distance. In particular, we link existing disentangled dataset distillation methods to the classical optimal quantization and Wasserstein barycenter problems, demonstrating consistency of distilled datasets for diffusion-based generative priors. We propose a simple extension of the state-of-the-art data distillation method D4M, achieving better performance on the ImageNet-1K dataset with trivial additional computation, and state-of-the-art performance in higher image-per-class settings.
• 摘要：数据集提炼旨在找到一个合成训练集，使得对合成数据的训练能够达到与对真实数据的训练类似的性能，而计算要求要低几个数量级。现有的方法可以大致分为两类：一类是将神经网络训练启发式方法作为低级问题的双层优化问题；一类是通过匹配数据分布来绕过双层优化的解缠结方法。后一种方法在训练和提炼数据集的大小方面具有速度和可扩展性的主要优势。我们证明，当配备编码器-解码器结构时，经验上成功的解缠结方法可以重新表述为最佳量化问题，其中通过最小化预期投影距离可以找到一组有限的点来近似底层概率测度。具体而言，我们将现有的解缠结数据集提炼方法与经典的最佳量化和 Wasserstein 重心问题联系起来，证明了基于扩散的生成先验的提炼数据集的一致性。我们提出了一种最先进的数据蒸馏方法 D4M 的简单扩展，通过少量额外计算在 ImageNet-1K 数据集上实现更好的性能，并在更高的每类图像设置中实现最先进的性能。

Title: C2PD: Continuity-Constrained Pixelwise Deformation for Guided Depth Super-Resolution

• Authors: Jiahui Kang, Qing Cai, Runqing Tan, Yimei Liu, Zhi Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07688
• Pdf URL: https://arxiv.org/pdf/2501.07688
• Copy Paste: [[2501.07688]] C2PD: Continuity-Constrained Pixelwise Deformation for Guided Depth Super-Resolution(https://arxiv.org/abs/2501.07688)
• Keywords: super-resolution
• Abstract: Guided depth super-resolution (GDSR) has demonstrated impressive performance across a wide range of domains, with numerous methods being proposed. However, existing methods often treat depth maps as images, where shading values are computed discretely, making them struggle to effectively restore the continuity inherent in the depth map. In this paper, we propose a novel approach that maximizes the utilization of spatial characteristics in depth, coupled with human abstract perception of real-world substance, by transforming the GDSR issue into deformation of a roughcast with ideal plasticity, which can be deformed by force like a continuous object. Specifically, we firstly designed a cross-modal operation, Continuity-constrained Asymmetrical Pixelwise Operation (CAPO), which can mimic the process of deforming an isovolumetrically flexible object through external forces. Utilizing CAPO as the fundamental component, we develop the Pixelwise Cross Gradient Deformation (PCGD), which is capable of emulating operations on ideal plastic objects (without volume constraint). Notably, our approach demonstrates state-of-the-art performance across four widely adopted benchmarks for GDSR, with significant advantages in large-scale tasks and generalizability.
• 摘要：引导深度超分辨率 (GDSR) 在众多领域都表现出色，并提出了许多方法。然而，现有方法通常将深度图视为图像，其中的阴影值是离散计算的，这使得它们难以有效恢复深度图固有的连续性。在本文中，我们提出了一种新方法，该方法最大限度地利用深度的空间特征，结合人类对现实世界物质的抽象感知，将 GDSR 问题转化为具有理想塑性的毛坯的变形，毛坯可以像连续物体一样被力变形。具体来说，我们首先设计了一个跨模态操作，即连续性约束的非对称像素操作 (CAPO)，它可以模拟通过外力使等体积柔性物体变形的过程。利用 CAPO 作为基本组件，我们开发了像素级交叉梯度变形 (PCGD)，它能够模拟对理想塑性物体（无体积约束）的操作。值得注意的是，我们的方法在 GDSR 的四个广泛采用的基准中展示了最先进的性能，并且在大规模任务和通用性方面具有显著的优势。

Title: Pedestrian Trajectory Prediction Based on Social Interactions Learning With Random Weights

• Authors: Jiajia Xie, Sheng Zhang, Beihao Xia, Zhu Xiao, Hongbo Jiang, Siwang Zhou, Zheng Qin, Hongyang Chen
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2501.07711
• Pdf URL: https://arxiv.org/pdf/2501.07711
• Copy Paste: [[2501.07711]] Pedestrian Trajectory Prediction Based on Social Interactions Learning With Random Weights(https://arxiv.org/abs/2501.07711)
• Keywords: generative
• Abstract: Pedestrian trajectory prediction is a critical technology in the evolution of self-driving cars toward complete artificial intelligence. Over recent years, focusing on the trajectories of pedestrians to model their social interactions has surged with great interest in more accurate trajectory predictions. However, existing methods for modeling pedestrian social interactions rely on pre-defined rules, struggling to capture non-explicit social interactions. In this work, we propose a novel framework named DTGAN, which extends the application of Generative Adversarial Networks (GANs) to graph sequence data, with the primary objective of automatically capturing implicit social interactions and achieving precise predictions of pedestrian trajectory. DTGAN innovatively incorporates random weights within each graph to eliminate the need for pre-defined interaction rules. We further enhance the performance of DTGAN by exploring diverse task loss functions during adversarial training, which yields improvements of 16.7\% and 39.3\% on metrics ADE and FDE, respectively. The effectiveness and accuracy of our framework are verified on two public datasets. The experimental results show that our proposed DTGAN achieves superior performance and is well able to understand pedestrians' intentions.
• 摘要：行人轨迹预测是自动驾驶汽车向完全人工智能发展的关键技术。近年来，人们越来越关注行人的轨迹以模拟他们的社交互动，对更准确的轨迹预测产生了浓厚的兴趣。然而，现有的行人社交互动建模方法依赖于预定义的规则，难以捕捉非显式的社交互动。在这项工作中，我们提出了一个名为 DTGAN 的新框架，它将生成对抗网络 (GAN) 的应用扩展到图序列数据，主要目标是自动捕捉隐式社交互动并实现对行人轨迹的精确预测。DTGAN 创新地在每个图中加入了随机权重，从而无需预定义的交互规则。我们通过在对抗训练期间探索不同的任务损失函数来进一步提高 DTGAN 的性能，在 ADE 和 FDE 指标上分别实现了 16.7% 和 39.3% 的改进。我们的框架的有效性和准确性在两个公共数据集上得到了验证。实验结果表明，我们提出的 DTGAN 取得了优异的性能，并且能够很好地理解行人的意图。

Title: Democratizing Text-to-Image Masked Generative Models with Compact Text-Aware One-Dimensional Tokens

• Authors: Dongwon Kim, Ju He, Qihang Yu, Chenglin Yang, Xiaohui Shen, Suha Kwak, Liang-Chieh Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07730
• Pdf URL: https://arxiv.org/pdf/2501.07730
• Copy Paste: [[2501.07730]] Democratizing Text-to-Image Masked Generative Models with Compact Text-Aware One-Dimensional Tokens(https://arxiv.org/abs/2501.07730)
• Keywords: generative
• Abstract: Image tokenizers form the foundation of modern text-to-image generative models but are notoriously difficult to train. Furthermore, most existing text-to-image models rely on large-scale, high-quality private datasets, making them challenging to replicate. In this work, we introduce Text-Aware Transformer-based 1-Dimensional Tokenizer (TA-TiTok), an efficient and powerful image tokenizer that can utilize either discrete or continuous 1-dimensional tokens. TA-TiTok uniquely integrates textual information during the tokenizer decoding stage (i.e., de-tokenization), accelerating convergence and enhancing performance. TA-TiTok also benefits from a simplified, yet effective, one-stage training process, eliminating the need for the complex two-stage distillation used in previous 1-dimensional tokenizers. This design allows for seamless scalability to large datasets. Building on this, we introduce a family of text-to-image Masked Generative Models (MaskGen), trained exclusively on open data while achieving comparable performance to models trained on private data. We aim to release both the efficient, strong TA-TiTok tokenizers and the open-data, open-weight MaskGen models to promote broader access and democratize the field of text-to-image masked generative models.
• 摘要：图像标记器是现代文本转图像生成模型的基础，但训练起来却非常困难。此外，大多数现有的文本转图像模型都依赖于大规模、高质量的私有数据集，因此很难复制。在这项工作中，我们引入了基于 Text-Aware Transformer 的一维标记器 (TA-TiTok)，这是一种高效且功能强大的图像标记器，可以使用离散或连续的一维标记。TA-TiTok 在标记器解码阶段（即去标记化）独特地集成了文本信息，从而加速了收敛并提高了性能。TA-TiTok 还受益于简化但有效的单阶段训练过程，从而无需以前的一维标记器中使用的复杂的两阶段提炼。这种设计允许无缝扩展到大型数据集。在此基础上，我们推出了一系列文本转图像蒙版生成模型 (MaskGen)，这些模型专门在开放数据上进行训练，同时实现了与在私有数据上训练的模型相当的性能。我们的目标是发布高效、强大的 TA-TiTok 分词器和开放数据、开放权重的 MaskGen 模型，以促进更广泛的访问并使文本转图像蒙版生成模型领域民主化。

Title: Symmetry-Aware Generative Modeling through Learned Canonicalization

• Authors: Kusha Sareen, Daniel Levy, Arnab Kumar Mondal, Sékou-Oumar Kaba, Tara Akhound-Sadegh, Siamak Ravanbakhsh
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.07773
• Pdf URL: https://arxiv.org/pdf/2501.07773
• Copy Paste: [[2501.07773]] Symmetry-Aware Generative Modeling through Learned Canonicalization(https://arxiv.org/abs/2501.07773)
• Keywords: generative
• Abstract: Generative modeling of symmetric densities has a range of applications in AI for science, from drug discovery to physics simulations. The existing generative modeling paradigm for invariant densities combines an invariant prior with an equivariant generative process. However, we observe that this technique is not necessary and has several drawbacks resulting from the limitations of equivariant networks. Instead, we propose to model a learned slice of the density so that only one representative element per orbit is learned. To accomplish this, we learn a group-equivariant canonicalization network that maps training samples to a canonical pose and train a non-equivariant generative model over these canonicalized samples. We implement this idea in the context of diffusion models. Our preliminary experimental results on molecular modeling are promising, demonstrating improved sample quality and faster inference time.
• 摘要：对称密度的生成建模在科学人工智能领域有广泛的应用，从药物发现到物理模拟。现有的不变密度生成建模范式将不变先验与等变生成过程相结合。然而，我们观察到这种技术不是必需的，并且由于等变网络的局限性而存在一些缺点。相反，我们建议对密度的学习片段进行建模，以便每个轨道只学习一个代表性元素。为了实现这一点，我们学习了一个组等变规范化网络，该网络将训练样本映射到规范姿势，并在这些规范化样本上训练一个非等变生成模型。我们在扩散模型的背景下实现了这个想法。我们在分子建模方面的初步实验结果很有希望，表明样本质量有所提高，推理时间更快。

Title: Linearly Convergent Mixup Learning

• Authors: Gakuto Obi, Ayato Saito, Yuto Sasaki, Tsuyoshi Kato
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.07794
• Pdf URL: https://arxiv.org/pdf/2501.07794
• Copy Paste: [[2501.07794]] Linearly Convergent Mixup Learning(https://arxiv.org/abs/2501.07794)
• Keywords: generation
• Abstract: Learning in the reproducing kernel Hilbert space (RKHS) such as the support vector machine has been recognized as a promising technique. It continues to be highly effective and competitive in numerous prediction tasks, particularly in settings where there is a shortage of training data or computational limitations exist. These methods are especially valued for their ability to work with small datasets and their interpretability. To address the issue of limited training data, mixup data augmentation, widely used in deep learning, has remained challenging to apply to learning in RKHS due to the generation of intermediate class labels. Although gradient descent methods handle these labels effectively, dual optimization approaches are typically not directly applicable. In this study, we present two novel algorithms that extend to a broader range of binary classification models. Unlike gradient-based approaches, our algorithms do not require hyperparameters like learning rates, simplifying their implementation and optimization. Both the number of iterations to converge and the computational cost per iteration scale linearly with respect to the dataset size. The numerical experiments demonstrate that our algorithms achieve faster convergence to the optimal solution compared to gradient descent approaches, and that mixup data augmentation consistently improves the predictive performance across various loss functions.
• 摘要：在再生核希尔伯特空间 (RKHS) 中学习（例如支持向量机）已被公认为一种有前途的技术。它在许多预测任务中仍然非常有效且具有竞争力，特别是在训练数据短缺或存在计算限制的情况下。这些方法因其处理小数据集的能力和可解释性而特别受到重视。为了解决训练数据有限的问题，广泛用于深度学习的混合数据增强由于中间类标签的生成而难以应用于 RKHS 中的学习。虽然梯度下降法可以有效地处理这些标签，但对偶优化方法通常不直接适用。在本研究中，我们提出了两种扩展到更广泛的二元分类模型的新算法。与基于梯度的方法不同，我们的算法不需要学习率等超参数，从而简化了它们的实现和优化。收敛的迭代次数和每次迭代的计算成本都与数据集大小成线性比例。数值实验表明，与梯度下降方法相比，我们的算法能够更快地收敛到最优解，并且混合数据增强可以持续提高各种损失函数的预测性能。

Title: Prediction Interval Construction Method for Electricity Prices

• Authors: Xin Lu
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2501.07827
• Pdf URL: https://arxiv.org/pdf/2501.07827
• Copy Paste: [[2501.07827]] Prediction Interval Construction Method for Electricity Prices(https://arxiv.org/abs/2501.07827)
• Keywords: generative
• Abstract: Accurate prediction of electricity prices plays an essential role in the electricity market. To reflect the uncertainty of electricity prices, price intervals are predicted. This paper proposes a novel prediction interval construction method. A conditional generative adversarial network is first presented to generate electricity price scenarios, with which the prediction intervals can be constructed. Then, different generated scenarios are stacked to obtain the probability densities, which can be applied to accurately reflect the uncertainty of electricity prices. Furthermore, a reinforced prediction mechanism based on the volatility level of weather factors is introduced to address the spikes or volatile prices. A case study is conducted to verify the effectiveness of the proposed novel prediction interval construction method. The method can also provide the probability density of each price scenario within the prediction interval and has the superiority to address the volatile prices and price spikes with a reinforced prediction mechanism.
• 摘要：准确预测电价在电力市场中起着至关重要的作用。为了反映电价的不确定性，需要对价格区间进行预测。本文提出了一种新的预测区间构建方法。首先提出一个条件生成对抗网络来生成电价场景，并利用该网络构建预测区间。然后，将生成的不同场景叠加以获得概率密度，从而可以准确反映电价的不确定性。此外，引入一种基于天气因素波动程度的强化预测机制来应对尖峰或波动的价格。通过案例研究验证了所提出的新预测区间构建方法的有效性。该方法还可以提供预测区间内每种价格情景的概率密度，并且具有使用强化预测机制应对波动价格和价格尖峰的优势。

Title: State-of-the-Art Transformer Models for Image Super-Resolution: Techniques, Challenges, and Applications

• Authors: Debasish Dutta, Deepjyoti Chetia, Neeharika Sonowal, Sanjib Kr Kalita
• Subjects: cs.CV, cs.AI, cs.ET, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2501.07855
• Pdf URL: https://arxiv.org/pdf/2501.07855
• Copy Paste: [[2501.07855]] State-of-the-Art Transformer Models for Image Super-Resolution: Techniques, Challenges, and Applications(https://arxiv.org/abs/2501.07855)
• Keywords: super-resolution
• Abstract: Image Super-Resolution (SR) aims to recover a high-resolution image from its low-resolution counterpart, which has been affected by a specific degradation process. This is achieved by enhancing detail and visual quality. Recent advancements in transformer-based methods have remolded image super-resolution by enabling high-quality reconstructions surpassing previous deep-learning approaches like CNN and GAN-based. This effectively addresses the limitations of previous methods, such as limited receptive fields, poor global context capture, and challenges in high-frequency detail recovery. Additionally, the paper reviews recent trends and advancements in transformer-based SR models, exploring various innovative techniques and architectures that combine transformers with traditional networks to balance global and local contexts. These neoteric methods are critically analyzed, revealing promising yet unexplored gaps and potential directions for future research. Several visualizations of models and techniques are included to foster a holistic understanding of recent trends. This work seeks to offer a structured roadmap for researchers at the forefront of deep learning, specifically exploring the impact of transformers on super-resolution techniques.
• 摘要：图像超分辨率 (SR) 旨在从受特定退化过程影响的低分辨率图像中恢复高分辨率图像。这是通过增强细节和视觉质量来实现的。基于 Transformer 的方法的最新进展重塑了图像超分辨率，实现了超越 CNN 和 GAN 等以前深度学习方法的高质量重建。这有效地解决了以前方法的局限性，例如有限的接受域、较差的全局上下文捕获以及高频细节恢复中的挑战。此外，本文回顾了基于 Transformer 的 SR 模型的最新趋势和进展，探索了将 Transformer 与传统网络相结合以平衡全局和局部上下文的各种创新技术和架构。对这些新方法进行了批判性分析，揭示了有希望但尚未探索的差距和未来研究的潜在方向。其中包括几种模型和技术的可视化，以促进对最新趋势的整体理解。这项工作旨在为深度学习前沿的研究人员提供结构化的路线图，特别是探索 Transformer 对超分辨率技术的影响。

Title: Make-A-Character 2: Animatable 3D Character Generation From a Single Image

• Authors: Lin Liu, Yutong Wang, Jiahao Chen, Jianfang Li, Tangli Xue, Longlong Li, Jianqiang Ren, Liefeng Bo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07870
• Pdf URL: https://arxiv.org/pdf/2501.07870
• Copy Paste: [[2501.07870]] Make-A-Character 2: Animatable 3D Character Generation From a Single Image(https://arxiv.org/abs/2501.07870)
• Keywords: generation
• Abstract: This report introduces Make-A-Character 2, an advanced system for generating high-quality 3D characters from single portrait photographs, ideal for game development and digital human applications. Make-A-Character 2 builds upon its predecessor by incorporating several significant improvements for image-based head generation. We utilize the IC-Light method to correct non-ideal illumination in input photos and apply neural network-based color correction to harmonize skin tones between the photos and game engine renders. We also employ the Hierarchical Representation Network to capture high-frequency facial structures and conduct adaptive skeleton calibration for accurate and expressive facial animations. The entire image-to-3D-character generation process takes less than 2 minutes. Furthermore, we leverage transformer architecture to generate co-speech facial and gesture actions, enabling real-time conversation with the generated character. These technologies have been integrated into our conversational AI avatar products.
• 摘要：本报告介绍了 Make-A-Character 2，这是一种先进的系统，可从单张肖像照片生成高质量的 3D 角色，非常适合游戏开发和数字人应用。Make-A-Character 2 在其前身的基础上，结合了基于图像的头部生成的几项重大改进。我们利用 IC-Light 方法来校正输入照片中的非理想照明，并应用基于神经网络的色彩校正来协调照片和游戏引擎渲染之间的肤色。我们还采用分层表示网络来捕捉高频面部结构并进行自适应骨架校准，以实现准确而富有表现力的面部动画。整个图像到 3D 角色的生成过程不到 2 分钟。此外，我们利用 Transformer 架构来生成同声面部和手势动作，从而实现与生成的角色的实时对话。这些技术已集成到我们的对话式 AI 头像产品中。

Title: Demographic Variability in Face Image Quality Measures

• Authors: Wassim Kabbani, Kiran Raja, Raghavendra Ramachandra, Christoph Busch
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07898
• Pdf URL: https://arxiv.org/pdf/2501.07898
• Copy Paste: [[2501.07898]] Demographic Variability in Face Image Quality Measures(https://arxiv.org/abs/2501.07898)
• Keywords: quality assessment
• Abstract: Face image quality assessment (FIQA) algorithms are being integrated into online identity management applications. These applications allow users to upload a face image as part of their document issuance process, where the image is then run through a quality assessment process to make sure it meets the quality and compliance requirements. Concerns about demographic bias have been raised about biometric systems, given the societal implications this may cause. It is therefore important that demographic variability in FIQA algorithms is assessed such that mitigation measures can be created. In this work, we study the demographic variability of all face image quality measures included in the ISO/IEC 29794-5 international standard across three demographic variables: age, gender, and skin tone. The results are rather promising and show no clear bias toward any specific demographic group for most measures. Only two quality measures are found to have considerable variations in their outcomes for different groups on the skin tone variable.
• 摘要：人脸图像质量评估 (FIQA) 算法正在被整合到在线身份管理应用程序中。这些应用程序允许用户在文件签发过程中上传人脸图像，然后对图像进行质量评估，以确保其符合质量和合规性要求。考虑到这可能会对社会造成影响，人们对生物识别系统的人口统计学偏见提出了担忧。因此，重要的是评估 FIQA 算法中的人口统计学变异性，以便制定缓解措施。在这项工作中，我们研究了 ISO/IEC 29794-5 国际标准中包含的所有人脸图像质量指标在三个人口统计学变量（年龄、性别和肤色）中的人口统计学变异性。结果相当令人鼓舞，并且大多数指标都没有明显偏向任何特定的人口统计学群体。只有两个质量指标在肤色变量上对不同群体的结果存在相当大的差异。

Title: Cloud Removal With PolSAR-Optical Data Fusion Using A Two-Flow Residual Network

• Authors: Yuxi Wang, Wenjuan Zhang, Bing Zhang
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.07901
• Pdf URL: https://arxiv.org/pdf/2501.07901
• Copy Paste: [[2501.07901]] Cloud Removal With PolSAR-Optical Data Fusion Using A Two-Flow Residual Network(https://arxiv.org/abs/2501.07901)
• Keywords: restoration
• Abstract: Optical remote sensing images play a crucial role in the observation of the Earth's surface. However, obtaining complete optical remote sensing images is challenging due to cloud cover. Reconstructing cloud-free optical images has become a major task in recent years. This paper presents a two-flow Polarimetric Synthetic Aperture Radar (PolSAR)-Optical data fusion cloud removal algorithm (PODF-CR), which achieves the reconstruction of missing optical images. PODF-CR consists of an encoding module and a decoding module. The encoding module includes two parallel branches that extract PolSAR image features and optical image features. To address speckle noise in PolSAR images, we introduce dynamic filters in the PolSAR branch for image denoising. To better facilitate the fusion between multimodal optical images and PolSAR images, we propose fusion blocks based on cross-skip connections to enable interaction of multimodal data information. The obtained fusion features are refined through an attention mechanism to provide better conditions for the subsequent decoding of the fused images. In the decoding module, multi-scale convolution is introduced to obtain multi-scale information. Additionally, to better utilize comprehensive scattering information and polarization characteristics to assist in the restoration of optical images, we use a dataset for cloud restoration called OPT-BCFSAR-PFSAR, which includes backscatter coefficient feature images and polarization feature images obtained from PoLSAR data and optical images. Experimental results demonstrate that this method outperforms existing methods in both qualitative and quantitative evaluations.
• 摘要：光学遥感图像在地球表面观测中起着至关重要的作用。然而，由于云层覆盖，获取完整的光学遥感图像具有挑战性。重建无云的光学图像成为近年来的一项主要任务。本文提出了一种双流极化合成孔径雷达（PolSAR）-光学数据融合去云算法（PODF-CR），实现了缺失光学图像的重建。PODF-CR由编码模块和解码模块组成。编码模块包括两个并行的分支，分别提取PolSAR图像特征和光学图像特征。为了解决PolSAR图像中的斑点噪声，我们在PolSAR分支中引入动态滤波器进行图像去噪。为了更好地促进多模态光学图像和PolSAR图像的融合，我们提出了基于交叉跳跃连接的融合块，以实现多模态数据信息的交互。通过注意机制对获得的融合特征进行细化，为后续融合图像的解码提供更好的条件。在解码模块中，引入多尺度卷积来获得多尺度信息。此外，为了更好地利用综合的散射信息和偏振特征辅助光学图像复原，本文采用了OPT-BCFSAR-PFSAR云层复原数据集，该数据集包括基于PoLSAR数据和光学图像得到的后向散射系数特征图像和偏振特征图像。实验结果表明，该方法在定性和定量评估上均优于现有方法。

Title: VENOM: Text-driven Unrestricted Adversarial Example Generation with Diffusion Models

• Authors: Hui Kuurila-Zhang, Haoyu Chen, Guoying Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.07922
• Pdf URL: https://arxiv.org/pdf/2501.07922
• Copy Paste: [[2501.07922]] VENOM: Text-driven Unrestricted Adversarial Example Generation with Diffusion Models(https://arxiv.org/abs/2501.07922)
• Keywords: generation, generative
• Abstract: Adversarial attacks have proven effective in deceiving machine learning models by subtly altering input images, motivating extensive research in recent years. Traditional methods constrain perturbations within $l_p$-norm bounds, but advancements in Unrestricted Adversarial Examples (UAEs) allow for more complex, generative-model-based manipulations. Diffusion models now lead UAE generation due to superior stability and image quality over GANs. However, existing diffusion-based UAE methods are limited to using reference images and face challenges in generating Natural Adversarial Examples (NAEs) directly from random noise, often producing uncontrolled or distorted outputs. In this work, we introduce VENOM, the first text-driven framework for high-quality unrestricted adversarial examples generation through diffusion models. VENOM unifies image content generation and adversarial synthesis into a single reverse diffusion process, enabling high-fidelity adversarial examples without sacrificing attack success rate (ASR). To stabilize this process, we incorporate an adaptive adversarial guidance strategy with momentum, ensuring that the generated adversarial examples $x^*$ align with the distribution $p(x)$ of natural images. Extensive experiments demonstrate that VENOM achieves superior ASR and image quality compared to prior methods, marking a significant advancement in adversarial example generation and providing insights into model vulnerabilities for improved defense development.
• 摘要：对抗性攻击已被证明能够通过巧妙地改变输入图像来欺骗机器学习模型，这激发了近年来的广泛研究。传统方法将扰动限制在 $l_p$ 范数范围内，但无限制对抗性示例 (UAE) 的进步允许进行更复杂的基于生成模型的操作。扩散模型现在由于比 GAN 具有更好的稳定性和图像质量而领先于 UAE 生成。然而，现有的基于扩散的 UAE 方法仅限于使用参考图像，并且在直接从随机噪声生成自然对抗性示例 (NAE) 时面临挑战，通常会产生不受控制或扭曲的输出。在这项工作中，我们介绍了 VENOM，这是第一个通过扩散模型生成高质量无限制对抗性示例的文本驱动框架。VENOM 将图像内容生成和对抗性合成统一为单个反向扩散过程，从而无需牺牲攻击成功率 (ASR) 即可实现高保真对抗性示例。为了稳定这一过程，我们结合了具有动量的自适应对抗引导策略，确保生成的对抗示例 $x^*$ 与自然图像的分布 $p(x)$ 一致。大量实验表明，与之前的方法相比，VENOM 实现了卓越的 ASR 和图像质量，标志着对抗示例生成取得了重大进步，并提供了对模型漏洞的洞察，从而改进了防御开发。

Title: Maximizing Uncertainty for Federated learning via Bayesian Optimisation-based Model Poisoning

• Authors: Marios Aristodemou, Xiaolan Liu, Yuan Wang, Konstantinos G. Kyriakopoulos, Sangarapillai Lambotharan, Qingsong Wei
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.08002
• Pdf URL: https://arxiv.org/pdf/2501.08002
• Copy Paste: [[2501.08002]] Maximizing Uncertainty for Federated learning via Bayesian Optimisation-based Model Poisoning(https://arxiv.org/abs/2501.08002)
• Keywords: generative
• Abstract: As we transition from Narrow Artificial Intelligence towards Artificial Super Intelligence, users are increasingly concerned about their privacy and the trustworthiness of machine learning (ML) technology. A common denominator for the metrics of trustworthiness is the quantification of uncertainty inherent in DL algorithms, and specifically in the model parameters, input data, and model predictions. One of the common approaches to address privacy-related issues in DL is to adopt distributed learning such as federated learning (FL), where private raw data is not shared among users. Despite the privacy-preserving mechanisms in FL, it still faces challenges in trustworthiness. Specifically, the malicious users, during training, can systematically create malicious model parameters to compromise the models predictive and generative capabilities, resulting in high uncertainty about their reliability. To demonstrate malicious behaviour, we propose a novel model poisoning attack method named Delphi which aims to maximise the uncertainty of the global model output. We achieve this by taking advantage of the relationship between the uncertainty and the model parameters of the first hidden layer of the local model. Delphi employs two types of optimisation , Bayesian Optimisation and Least Squares Trust Region, to search for the optimal poisoned model parameters, named as Delphi-BO and Delphi-LSTR. We quantify the uncertainty using the KL Divergence to minimise the distance of the predictive probability distribution towards an uncertain distribution of model output. Furthermore, we establish a mathematical proof for the attack effectiveness demonstrated in FL. Numerical results demonstrate that Delphi-BO induces a higher amount of uncertainty than Delphi-LSTR highlighting vulnerability of FL systems to model poisoning attacks.
• 摘要：随着我们从狭义人工智能向超级人工智能过渡，用户越来越关注他们的隐私和机器学习 (ML) 技术的可信度。可信度指标的共同点是量化深度学习算法固有的不确定性，特别是模型参数、输入数据和模型预测中的不确定性。解决深度学习隐私相关问题的常用方法之一是采用分布式学习，例如联邦学习 (FL)，其中用户之间不共享私人原始数据。尽管联邦学习具有隐私保护机制，但它在可信度方面仍然面临挑战。具体而言，恶意用户可以在训练期间系统地创建恶意模型参数来破坏模型的预测和生成能力，从而对其可靠性产生高度不确定性。为了展示恶意行为，我们提出了一种名为 Delphi 的新型模型中毒攻击方法，旨在最大化全局模型输出的不确定性。我们通过利用局部模型第一个隐藏层的不确定性与模型参数之间的关系来实现这一点。 Delphi 采用两种优化方法，即贝叶斯优化和最小二乘信任区域，来搜索最佳的中毒模型参数，称为 Delphi-BO 和 Delphi-LSTR。我们使用 KL 散度来量化不确定性，以最小化预测概率分布与模型输出不确定分布之间的距离。此外，我们为 FL 中展示的攻击有效性建立了数学证明。数值结果表明，Delphi-BO 比 Delphi-LSTR 引起的不确定性更高，突显了 FL 系统易受模型中毒攻击的弱点。

Title: An AI-driven framework for rapid and localized optimizations of urban open spaces

• Authors: Pegah Eshraghi, Arman Nikkhah Dehnavi, Maedeh Mirdamadi, Riccardo Talami, Zahra-Sadat Zomorodian
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2501.08019
• Pdf URL: https://arxiv.org/pdf/2501.08019
• Copy Paste: [[2501.08019]] An AI-driven framework for rapid and localized optimizations of urban open spaces(https://arxiv.org/abs/2501.08019)
• Keywords: generation
• Abstract: As urbanization accelerates, open spaces are increasingly recognized for their role in enhancing sustainability and well-being, yet they remain underexplored compared to built spaces. This study introduces an AI-driven framework that integrates machine learning models (MLMs) and explainable AI techniques to optimize Sky View Factor (SVF) and visibility, key spatial metrics influencing thermal comfort and perceived safety in urban spaces. Unlike global optimization methods, which are computationally intensive and impractical for localized adjustments, this framework supports incremental design improvements with lower computational costs and greater flexibility. The framework employs SHapley Adaptive Explanations (SHAP) to analyze feature importance and Counterfactual Explanations (CFXs) to propose minimal design changes. Simulations tested five MLMs, identifying XGBoost as the most accurate, with building width, park area, and heights of surrounding buildings as critical for SVF, and distances from southern buildings as key for visibility. Compared to Genetic Algorithms, which required approximately 15/30 minutes across 3/4 generations to converge, the tested CFX approach achieved optimized results in 1 minute with a 5% RMSE error, demonstrating significantly faster performance and suitability for scalable retrofitting strategies. This interpretable and computationally efficient framework advances urban performance optimization, providing data-driven insights and practical retrofitting solutions for enhancing usability and environmental quality across diverse urban contexts.
• 摘要：随着城市化进程的加快，开放空间在增强可持续性和福祉方面的作用越来越受到认可，但与建筑空间相比，开放空间仍未得到充分开发。本研究引入了一个人工智能驱动的框架，该框架集成了机器学习模型 (MLM) 和可解释的人工智能技术，以优化天空视野系数 (SVF) 和可见性，这是​​影响城市空间热舒适度和感知安全性的关键空间指标。与计算密集型且不适合局部调整的全局优化方法不同，该框架支持增量设计改进，计算成本更低，灵活性更高。该框架采用 SHapley 自适应解释 (SHAP) 来分析特征重要性，并采用反事实解释 (CFX) 来提出最小的设计变更。模拟测试了五个 MLM，确定 XGBoost 最准确，其中建筑宽度、公园面积和周围建筑的高度对 SVF 至关重要，与南部建筑的距离是可见性的关键。与需要 3/4 代约 15/30 分钟才能收敛的遗传算法相比，经过测试的 CFX 方法在 1 分钟内以 5% 的 RMSE 误差实现了优化结果，表明其性能显著提高，并且适用于可扩展的改造策略。这种可解释且计算效率高的框架推动了城市性能优化，提供了数据驱动的见解和实用的改造解决方案，以提高不同城市环境中的可用性和环境质量。

Title: Enhanced SPS Velocity-adaptive Scheme: Access Fariness in 5G NR V2I Networks

• Authors: Xiao Xu, Qiong Wu, Pingyi Fan, Kezhi Wang
• Subjects: cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2501.08037
• Pdf URL: https://arxiv.org/pdf/2501.08037
• Copy Paste: [[2501.08037]] Enhanced SPS Velocity-adaptive Scheme: Access Fariness in 5G NR V2I Networks(https://arxiv.org/abs/2501.08037)
• Keywords: generation
• Abstract: Vehicle-to-Infrastructure (V2I) technology enables information exchange between vehicles and road infrastructure. Specifically, when a vehicle approaches a roadside unit (RSU), it can exchange information with the RSU to obtain accurate data that assists in driving. With the release of the 3rd Generation Partnership Project (3GPP) Release 16, which includes the 5G New Radio (NR) Vehicle-to-Everything (V2X) standards, vehicles typically adopt mode-2 communication using sensing-based semi-persistent scheduling (SPS) for resource allocation. In this approach, vehicles identify candidate resources within a selection window and exclude ineligible resources based on information from a sensing window. However, vehicles often drive at different speeds, resulting in varying amounts of data transmission with RSUs as they pass by, which leads to unfair access. Therefore, it is essential to design an access scheme that accounts for different vehicle speeds to achieve fair access across the network. This paper formulates an optimization problem for vehicular networks and proposes a multi-objective optimization scheme to address it by adjusting the selection window in the SPS mechanism of 5G NR V2I mode-2. Simulation results demonstrate the effectiveness of the proposed scheme
• 摘要：车对基础设施 (V2I) 技术可实现车辆与道路基础设施之间的信息交换。具体而言，当车辆接近路边单元 (RSU) 时，它可以与 RSU 交换信息以获取辅助驾驶的准确数据。随着第三代合作伙伴计划 (3GPP) Release 16 的发布，其中包括 5G 新无线电 (NR) 车对万物 (V2X) 标准，车辆通常采用基于感知的半持久调度 (SPS) 进行资源分配的模式 2 通信。在这种方法中，车辆在选择窗口内识别候选资源，并根据来自感知窗口的信息排除不合格的资源。然而，车辆通常以不同的速度行驶，导致它们经过时与 RSU 的数据传输量不同，从而导致不公平的访问。因此，设计一个考虑到不同车辆速度的访问方案以实现整个网络的公平访问至关重要。本文提出了车载网络优化问题，并提出了一种多目标优化方案，通过调整 5G NR V2I 模式 2 的 SPS 机制中的选择窗口来解决该问题。仿真结果证明了所提方案的有效性

Title: Skeleton and Font Generation Network for Zero-shot Chinese Character Generation

• Authors: Mobai Xue, Jun Du, Zhenrong Zhang, Jiefeng Ma, Qikai Chang, Pengfei Hu, Jianshu Zhang, Yu Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.08062
• Pdf URL: https://arxiv.org/pdf/2501.08062
• Copy Paste: [[2501.08062]] Skeleton and Font Generation Network for Zero-shot Chinese Character Generation(https://arxiv.org/abs/2501.08062)
• Keywords: generation
• Abstract: Automatic font generation remains a challenging research issue, primarily due to the vast number of Chinese characters, each with unique and intricate structures. Our investigation of previous studies reveals inherent bias capable of causing structural changes in characters. Specifically, when generating a Chinese character similar to, but different from, those in the training samples, the bias is prone to either correcting or ignoring these subtle variations. To address this concern, we propose a novel Skeleton and Font Generation Network (SFGN) to achieve a more robust Chinese character font generation. Our approach includes a skeleton builder and font generator. The skeleton builder synthesizes content features using low-resource text input, enabling our technique to realize font generation independently of content image inputs. Unlike previous font generation methods that treat font style as a global embedding, we introduce a font generator to align content and style features on the radical level, which is a brand-new perspective for font generation. Except for common characters, we also conduct experiments on misspelled characters, a substantial portion of which slightly differs from the common ones. Our approach visually demonstrates the efficacy of generated images and outperforms current state-of-the-art font generation methods. Moreover, we believe that misspelled character generation have significant pedagogical implications and verify such supposition through experiments. We used generated misspelled characters as data augmentation in Chinese character error correction tasks, simulating the scenario where students learn handwritten Chinese characters with the help of misspelled characters. The significantly improved performance of error correction tasks demonstrates the effectiveness of our proposed approach and the value of misspelled character generation.
• 摘要：自动字体生成仍然是一个具有挑战性的研究问题，主要是因为汉字数量庞大，每个汉字都有独特而复杂的结构。我们对先前研究的调查揭示了固有的偏见，这种偏见可能导致字符的结构变化。具体来说，当生成与训练样本中相似但不同的汉字时，偏见很容易纠正或忽略这些细微的变化。为了解决这个问题，我们提出了一种新颖的骨架和字体生成网络 (SFGN)，以实现更强大的汉字字体生成。我们的方法包括骨架构建器和字体生成器。骨架构建器使用低资源文本输入合成内容特征，使我们的技术能够独立于内容图像输入实现字体生成。与以前的将字体样式视为全局嵌入的字体生成方法不同，我们引入了一个字体生成器来在部首级别上对齐内容和样式特征，这是字体生成的全新视角。除了常用字符外，我们还对拼写错误的字符进行了实验，其中很大一部分与常用字符略有不同。我们的方法直观地展示了生成图像的有效性，并且优于目前最先进的字体生成方法。此外，我们认为拼写错误字符生成具有重要的教学意义，并通过实验验证了这一假设。我们将生成的拼写错误字符用作汉字纠错任务中的数据增强，模拟学生借助拼写错误字符学习手写汉字的场景。纠错任务性能的显著提高证明了我们提出的方法的有效性和拼写错误字符生成的价值。

Title: Evaluating Human Perception of Novel View Synthesis: Subjective Quality Assessment of Gaussian Splatting and NeRF in Dynamic Scenes

• Authors: Yuhang Zhang, Joshua Maraval, Zhengyu Zhang, Nicolas Ramin, Shishun Tian, Lu Zhang
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2501.08072
• Pdf URL: https://arxiv.org/pdf/2501.08072
• Copy Paste: [[2501.08072]] Evaluating Human Perception of Novel View Synthesis: Subjective Quality Assessment of Gaussian Splatting and NeRF in Dynamic Scenes(https://arxiv.org/abs/2501.08072)
• Keywords: quality assessment
• Abstract: Gaussian Splatting (GS) and Neural Radiance Fields (NeRF) are two groundbreaking technologies that have revolutionized the field of Novel View Synthesis (NVS), enabling immersive photorealistic rendering and user experiences by synthesizing multiple viewpoints from a set of images of sparse views. The potential applications of NVS, such as high-quality virtual and augmented reality, detailed 3D modeling, and realistic medical organ imaging, underscore the importance of quality assessment of NVS methods from the perspective of human perception. Although some previous studies have explored subjective quality assessments for NVS technology, they still face several challenges, especially in NVS methods selection, scenario coverage, and evaluation methodology. To address these challenges, we conducted two subjective experiments for the quality assessment of NVS technologies containing both GS-based and NeRF-based methods, focusing on dynamic and real-world scenes. This study covers 360°, front-facing, and single-viewpoint videos while providing a richer and greater number of real scenes. Meanwhile, it's the first time to explore the impact of NVS methods in dynamic scenes with moving objects. The two types of subjective experiments help to fully comprehend the influences of different viewing paths from a human perception perspective and pave the way for future development of full-reference and no-reference quality metrics. In addition, we established a comprehensive benchmark of various state-of-the-art objective metrics on the proposed database, highlighting that existing methods still struggle to accurately capture subjective quality. The results give us some insights into the limitations of existing NVS methods and may promote the development of new NVS methods.
• 摘要：高斯散射 (GS) 和神经辐射场 (NeRF) 是两项突破性的技术，它们彻底改变了新视图合成 (NVS) 领域，通过从一组稀疏视图图像中合成多个视点，实现了沉浸式的真实感渲染和用户体验。NVS 的潜在应用，例如高质量的虚拟和增强现实、详细的 3D 建模和逼真的医学器官成像，凸显了从人类感知的角度对 NVS 方法进行质量评估的重要性。尽管之前的一些研究已经探索了 NVS 技术的主观质量评估，但它们仍然面临一些挑战，特别是在 NVS 方法选择、场景覆盖和评估方法方面。为了应对这些挑战，我们对包含基于 GS 和基于 NeRF 的方法的 NVS 技术进行了两项主观质量评估实验，重点关注动态和真实场景。本研究涵盖 360°、正面和单视点视频，同时提供更丰富和更多的真实场景。同时，这是首次探索 NVS 方法在具有移动物体的动态场景中的影响。两类主观实验有助于从人类感知的角度充分理解不同观看路径的影响，并为未来开发全参考和无参考质量指标铺平道路。此外，我们在所提出的数据库上建立了各种最新客观指标的综合基准，强调现有方法仍然难以准确捕捉主观质量。结果让我们对现有 NVS 方法的局限性有了一些了解，并可能促进新 NVS 方法的开发。

Title: Audio-visual Deepfake Detection With Local Temporal Inconsistencies

• Authors: Marcella Astrid, Enjie Ghorbel, Djamila Aouada
• Subjects: cs.CV, cs.CR, cs.MM, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2501.08137
• Pdf URL: https://arxiv.org/pdf/2501.08137
• Copy Paste: [[2501.08137]] Audio-visual Deepfake Detection With Local Temporal Inconsistencies(https://arxiv.org/abs/2501.08137)
• Keywords: generation
• Abstract: This paper proposes an audio-visual deepfake detection approach that aims to capture fine-grained temporal inconsistencies between audio and visual modalities. To achieve this, both architectural and data synthesis strategies are introduced. From an architectural perspective, a temporal distance map, coupled with an attention mechanism, is designed to capture these inconsistencies while minimizing the impact of irrelevant temporal subsequences. Moreover, we explore novel pseudo-fake generation techniques to synthesize local inconsistencies. Our approach is evaluated against state-of-the-art methods using the DFDC and FakeAVCeleb datasets, demonstrating its effectiveness in detecting audio-visual deepfakes.
• 摘要：本文提出了一种音频-视频深度伪造检测方法，旨在捕捉音频和视觉模态之间的细粒度时间不一致性。为了实现这一目标，我们引入了架构和数据合成策略。从架构的角度来看，时间距离图与注意力机制相结合，旨在捕捉这些不一致性，同时最大限度地减少不相关时间子序列的影响。此外，我们探索了新颖的伪造生成技术来合成局部不一致性。我们的方法与最先进的方法进行了比较，使用 DFDC 和 FakeAVCeleb 数据集，证明了其在检测音频-视频深度伪造方面的有效性。

Title: Bootstrapping Corner Cases: High-Resolution Inpainting for Safety Critical Detect and Avoid for Automated Flying

• Authors: Jonathan Lyhs, Lars Hinneburg, Michael Fischer, Florian Ölsner, Stefan Milz, Jeremy Tschirner, Patrick Mäder
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.08142
• Pdf URL: https://arxiv.org/pdf/2501.08142
• Copy Paste: [[2501.08142]] Bootstrapping Corner Cases: High-Resolution Inpainting for Safety Critical Detect and Avoid for Automated Flying(https://arxiv.org/abs/2501.08142)
• Keywords: generation, generative
• Abstract: Modern machine learning techniques have shown tremendous potential, especially for object detection on camera images. For this reason, they are also used to enable safety-critical automated processes such as autonomous drone flights. We present a study on object detection for Detect and Avoid, a safety critical function for drones that detects air traffic during automated flights for safety reasons. An ill-posed problem is the generation of good and especially large data sets, since detection itself is the corner case. Most models suffer from limited ground truth in raw data, \eg recorded air traffic or frontal flight with a small aircraft. It often leads to poor and critical detection rates. We overcome this problem by using inpainting methods to bootstrap the dataset such that it explicitly contains the corner cases of the raw data. We provide an overview of inpainting methods and generative models and present an example pipeline given a small annotated dataset. We validate our method by generating a high-resolution dataset, which we make publicly available and present it to an independent object detector that was fully trained on real data.
• 摘要：现代机器学习技术已显示出巨大的潜力，尤其是在相机图像上的物体检测方面。因此，它们还用于实现安全关键的自动化过程，例如无人机自主飞行。我们介绍了一项关于“检测和避免”物体检测的研究，这是无人机的安全关键功能，出于安全原因在自动飞行期间检测空中交通。一个不适定的问题是生成好的、特别是大的数据集，因为检测本身是一个极端情况。大多数模型都受到原始数据中地面真实性有限的困扰，例如记录的空中交通或小型飞机的正面飞行。这通常会导致检测率低下和临界。我们通过使用修复方法来引导数据集来克服这个问题，使其明确包含原始数据的极端情况。我们概述了修复方法和生成模型，并给出了一个给定小型注释数据集的示例管道。我们通过生成高分辨率数据集来验证我们的方法，我们将该数据集公开，并将其呈现给一个完全在真实数据上训练过的独立物体检测器。

Title: D$^2$-DPM: Dual Denoising for Quantized Diffusion Probabilistic Models

• Authors: Qian Zeng, Jie Song, Han Zheng, Hao Jiang, Mingli Song
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.08180
• Pdf URL: https://arxiv.org/pdf/2501.08180
• Copy Paste: [[2501.08180]] D$^2$-DPM: Dual Denoising for Quantized Diffusion Probabilistic Models(https://arxiv.org/abs/2501.08180)
• Keywords: generation
• Abstract: Diffusion models have achieved cutting-edge performance in image generation. However, their lengthy denoising process and computationally intensive score estimation network impede their scalability in low-latency and resource-constrained scenarios. Post-training quantization (PTQ) compresses and accelerates diffusion models without retraining, but it inevitably introduces additional quantization noise, resulting in mean and variance deviations. In this work, we propose D2-DPM, a dual denoising mechanism aimed at precisely mitigating the adverse effects of quantization noise on the noise estimation network. Specifically, we first unravel the impact of quantization noise on the sampling equation into two components: the mean deviation and the variance deviation. The mean deviation alters the drift coefficient of the sampling equation, influencing the trajectory trend, while the variance deviation magnifies the diffusion coefficient, impacting the convergence of the sampling trajectory. The proposed D2-DPM is thus devised to denoise the quantization noise at each time step, and then denoise the noisy sample through the inverse diffusion iterations. Experimental results demonstrate that D2-DPM achieves superior generation quality, yielding a 1.42 lower FID than the full-precision model while achieving 3.99x compression and 11.67x bit-operation acceleration.
• 摘要：扩散模型在图像生成中取得了尖端的性能。然而，它们冗长的去噪过程和计算密集的分数估计网络阻碍了它们在低延迟和资源受限场景中的可扩展性。训练后量化（PTQ）无需重新训练即可压缩和加速扩散模型，但它不可避免地会引入额外的量化噪声，导致均值和方差偏差。在本文中，我们提出了 D2-DPM，一种双重去噪机制，旨在精确减轻量化噪声对噪声估计网络的不利影响。具体而言，我们首先将量化噪声对采样方程的影响分解为两个部分：均值偏差和方差偏差。均值偏差改变了采样方程的漂移系数，影响轨迹趋势，而方差偏差放大了扩散系数，影响采样轨迹的收敛。因此，我们设计了 D2-DPM 在每个时间步骤中对量化噪声进行去噪，然后通过逆扩散迭代对噪声样本进行去噪。实验结果表明，D2-DPM 实现了卓越的生成质量，与全精度模型相比，FID 降低了 1.42，同时实现了 3.99 倍的压缩率和 11.67 倍的位操作加速。

Title: Investigating Energy Efficiency and Performance Trade-offs in LLM Inference Across Tasks and DVFS Settings

• Authors: Paul Joe Maliakel, Shashikant Ilager, Ivona Brandic
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.08219
• Pdf URL: https://arxiv.org/pdf/2501.08219
• Copy Paste: [[2501.08219]] Investigating Energy Efficiency and Performance Trade-offs in LLM Inference Across Tasks and DVFS Settings(https://arxiv.org/abs/2501.08219)
• Keywords: generation
• Abstract: Large language models (LLMs) have shown significant improvements in many natural language processing (NLP) tasks, accelerating their rapid adoption across many industries. These models are resource-intensive, requiring extensive computational resources both during training and inference, leading to increased energy consumption and negative environmental impact. As their adoption accelerates, the sustainability of LLMs has become a critical issue, necessitating strategies to optimize their runtime efficiency without compromising performance. Hence, it is imperative to identify the parameters that significantly influence the performance and energy efficiency of LLMs. To that end, in this work, we investigate the effect of important parameters on the performance and energy efficiency of LLMs during inference and examine their trade-offs. First, we analyze how different types of models with varying numbers of parameters and architectures perform on tasks like text generation, question answering, and summarization by benchmarking LLMs such as Falcon-7B, Mistral-7B-v0.1, T5-3B, GPT-2, GPT-J-6B, and GPT-Neo-2.7B. Second, we study input and output sequence characteristics such as sequence length concerning energy consumption, performance, and throughput. Finally, we explore the impact of hardware-based power-saving techniques, i.e., Dynamic Voltage Frequency Scaling (DVFS), on the models' latency and energy efficiency. Our extensive benchmarking and statistical analysis reveal many interesting findings, uncovering how specific optimizations can reduce energy consumption while maintaining throughput and accuracy. This study provides actionable insights for researchers and practitioners to design energy-efficient LLM inference systems.
• 摘要：大型语言模型 (LLM) 在许多自然语言处理 (NLP) 任务中表现出显著的改进，加速了它们在许多行业的快速应用。这些模型是资源密集型的，在训练和推理过程中都需要大量的计算资源，从而导致能源消耗增加并对环境产生负面影响。随着它们的采用加速，LLM 的可持续性已成为一个关键问题，需要制定策略来优化其运行时效率而不影响性能。因此，必须确定对 LLM 的性能和能源效率有重大影响的参数。为此，在这项工作中，我们研究了重要参数对 LLM 在推理过程中的性能和能源效率的影响，并研究了它们的权衡。首先，我们通过对 Falcon-7B、Mistral-7B-v0.1、T5-3B、GPT-2、GPT-J-6B 和 GPT-Neo-2.7B 等 LLM 进行基准测试，分析具有不同数量参数和架构的不同类型模型在文本生成、问答和摘要等任务上的表现。其次，我们研究输入和输出序列特征，例如序列长度与能耗、性能和吞吐量的关系。最后，我们探讨基于硬件的节能技术（即动态电压频率调节 (DVFS)）对模型延迟和能效的影响。我们进行了广泛的基准测试和统计分析，发现了许多有趣的发现，揭示了特定的优化如何降低能耗，同时保持吞吐量和准确性。这项研究为研究人员和从业者提供了可行的见解，以设计节能的 LLM 推理系统。

Title: FramePainter: Endowing Interactive Image Editing with Video Diffusion Priors

• Authors: Yabo Zhang, Xinpeng Zhou, Yihan Zeng, Hang Xu, Hui Li, Wangmeng Zuo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.08225
• Pdf URL: https://arxiv.org/pdf/2501.08225
• Copy Paste: [[2501.08225]] FramePainter: Endowing Interactive Image Editing with Video Diffusion Priors(https://arxiv.org/abs/2501.08225)
• Keywords: generation
• Abstract: Interactive image editing allows users to modify images through visual interaction operations such as drawing, clicking, and dragging. Existing methods construct such supervision signals from videos, as they capture how objects change with various physical interactions. However, these models are usually built upon text-to-image diffusion models, so necessitate (i) massive training samples and (ii) an additional reference encoder to learn real-world dynamics and visual consistency. In this paper, we reformulate this task as an image-to-video generation problem, so that inherit powerful video diffusion priors to reduce training costs and ensure temporal consistency. Specifically, we introduce FramePainter as an efficient instantiation of this formulation. Initialized with Stable Video Diffusion, it only uses a lightweight sparse control encoder to inject editing signals. Considering the limitations of temporal attention in handling large motion between two frames, we further propose matching attention to enlarge the receptive field while encouraging dense correspondence between edited and source image tokens. We highlight the effectiveness and efficiency of FramePainter across various of editing signals: it domainantly outperforms previous state-of-the-art methods with far less training data, achieving highly seamless and coherent editing of images, \eg, automatically adjust the reflection of the cup. Moreover, FramePainter also exhibits exceptional generalization in scenarios not present in real-world videos, \eg, transform the clownfish into shark-like shape. Our code will be available at this https URL.
• 摘要：交互式图像编辑允许用户通过视觉交互操作（例如绘制、单击和拖动）来修改图像。现有方法从视频中构建此类监督信号，因为它们可以捕捉物体如何随着各种物理交互而变化。然而，这些模型通常建立在文本到图像的扩散模型之上，因此需要 (i) 大量训练样本和 (ii) 额外的参考编码器来学习现实世界的动态和视觉一致性。在本文中，我们将这项任务重新表述为图像到视频的生成问题，以便继承强大的视频扩散先验来降低训练成本并确保时间一致性。具体来说，我们引入 FramePainter 作为此公式的有效实例。它使用稳定视频扩散进行初始化，仅使用轻量级稀疏控制编码器来注入编辑信号。考虑到时间注意力在处理两帧之间的大运动方面的局限性，我们进一步提出匹配注意力以扩大感受野，同时鼓励编辑图像标记和源图像标记之间的密集对应。我们重点介绍了 FramePainter 在各种编辑信号中的有效性和效率：它以少得多的训练数据全面超越了之前最先进的方法，实现了高度无缝和连贯的图像编辑，例如，自动调整杯子的反射。此外，FramePainter 还在现实世界视频中不存在的场景中表现出卓越的泛化能力，例如，将小丑鱼变成鲨鱼形状。我们的代码将在此 https URL 上提供。

Title: Towards an End-to-End (E2E) Adversarial Learning and Application in the Physical World

• Authors: Dudi Biton, Jacob Shams, Koda Satoru, Asaf Shabtai, Yuval Elovici, Ben Nassi
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2501.08258
• Pdf URL: https://arxiv.org/pdf/2501.08258
• Copy Paste: [[2501.08258]] Towards an End-to-End (E2E) Adversarial Learning and Application in the Physical World(https://arxiv.org/abs/2501.08258)
• Keywords: generation
• Abstract: The traditional learning process of patch-based adversarial attacks, conducted in the digital domain and then applied in the physical domain (e.g., via printed stickers), may suffer from reduced performance due to adversarial patches' limited transferability from the digital domain to the physical domain. Given that previous studies have considered using projectors to apply adversarial attacks, we raise the following question: can adversarial learning (i.e., patch generation) be performed entirely in the physical domain with a projector? In this work, we propose the Physical-domain Adversarial Patch Learning Augmentation (PAPLA) framework, a novel end-to-end (E2E) framework that converts adversarial learning from the digital domain to the physical domain using a projector. We evaluate PAPLA across multiple scenarios, including controlled laboratory settings and realistic outdoor environments, demonstrating its ability to ensure attack success compared to conventional digital learning-physical application (DL-PA) methods. We also analyze the impact of environmental factors, such as projection surface color, projector strength, ambient light, distance, and angle of the target object relative to the camera, on the effectiveness of projected patches. Finally, we demonstrate the feasibility of the attack against a parked car and a stop sign in a real-world outdoor environment. Our results show that under specific conditions, E2E adversarial learning in the physical domain eliminates the transferability issue and ensures evasion by object detectors. Finally, we provide insights into the challenges and opportunities of applying adversarial learning in the physical domain and explain where such an approach is more effective than using a sticker.
• 摘要：传统的基于补丁的对抗性攻击学习过程是在数字域中进行，然后应用于物理域（例如，通过印刷贴纸），由于对抗性补丁从数字域到物理域的可转移性有限，因此性能可能会下降。鉴于先前的研究已经考虑使用投影仪来应用对抗性攻击，我们提出以下问题：对抗性学习（即补丁生成）是否可以完全在物理域中使用投影仪进行？在这项工作中，我们提出了物理域对抗性补丁学习增强 (PAPLA) 框架，这是一种新颖的端到端 (E2E) 框架，它使用投影仪将对抗性学习从数字域转换到物理域。我们在多种场景中评估了 PAPLA，包括受控的实验室环境和逼真的户外环境，证明了与传统的数字学习-物理应用 (DL-PA) 方法相比，它能够确保攻击成功。我们还分析了环境因素（例如投影表面颜色、投影仪强度、环境光、距离以及目标物体相对于相机的角度）对投影补丁效果的影响。最后，我们证明了在真实户外环境中对停放的汽车和停车标志进行攻击的可行性。我们的结果表明，在特定条件下，物理域中的 E2E 对抗性学习消除了可迁移性问题并确保了物体检测器的规避。最后，我们深入了解了在物理域中应用对抗性学习的挑战和机遇，并解释了这种方法在哪些方面比使用贴纸更有效。

Title: SmartEraser: Remove Anything from Images using Masked-Region Guidance

• Authors: Longtao Jiang, Zhendong Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Lei Shi, Dong Chen, Houqiang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.08279
• Pdf URL: https://arxiv.org/pdf/2501.08279
• Copy Paste: [[2501.08279]] SmartEraser: Remove Anything from Images using Masked-Region Guidance(https://arxiv.org/abs/2501.08279)
• Keywords: generation
• Abstract: Object removal has so far been dominated by the mask-and-inpaint paradigm, where the masked region is excluded from the input, leaving models relying on unmasked areas to inpaint the missing region. However, this approach lacks contextual information for the masked area, often resulting in unstable performance. In this work, we introduce SmartEraser, built with a new removing paradigm called Masked-Region Guidance. This paradigm retains the masked region in the input, using it as guidance for the removal process. It offers several distinct advantages: (a) it guides the model to accurately identify the object to be removed, preventing its regeneration in the output; (b) since the user mask often extends beyond the object itself, it aids in preserving the surrounding context in the final result. Leveraging this new paradigm, we present Syn4Removal, a large-scale object removal dataset, where instance segmentation data is used to copy and paste objects onto images as removal targets, with the original images serving as ground truths. Experimental results demonstrate that SmartEraser significantly outperforms existing methods, achieving superior performance in object removal, especially in complex scenes with intricate compositions.
• 摘要：到目前为止，物体移除一直以掩码和修复范式为主，其中掩码区域被排除在输入之外，而模型则依靠未掩码区域来修复缺失区域。然而，这种方法缺乏掩码区域的上下文信息，通常会导致性能不稳定。在这项工作中，我们引入了 SmartEraser，它采用一种名为“掩码区域指导”的新移除范式构建。该范式在输入中保留掩码区域，并将其用作移除过程的指导。它提供了几个明显的优势：(a) 它引导模型准确识别要移除的物体，防止其在输出中重新生成；(b) 由于用户掩码通常超出物体本身，因此它有助于在最终结果中保留周围的上下文。利用这种新范式，我们提出了 Syn4Removal，这是一个大规模物体移除数据集，其中实例分割数据用于将物体复制并粘贴到图像上作为移除目标，原始图像作为基本事实。实验结果表明，SmartEraser 的性能明显优于现有方法，在物体移除方面取得了优异的性能，尤其是在具有复杂构图的复杂场景中。

Title: LayerAnimate: Layer-specific Control for Animation

• Authors: Yuxue Yang, Lue Fan, Zuzen Lin, Feng Wang, Zhaoxiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.08295
• Pdf URL: https://arxiv.org/pdf/2501.08295
• Copy Paste: [[2501.08295]] LayerAnimate: Layer-specific Control for Animation(https://arxiv.org/abs/2501.08295)
• Keywords: generation
• Abstract: Animated video separates foreground and background elements into layers, with distinct processes for sketching, refining, coloring, and in-betweening. Existing video generation methods typically treat animation as a monolithic data domain, lacking fine-grained control over individual layers. In this paper, we introduce LayerAnimate, a novel architectural approach that enhances fine-grained control over individual animation layers within a video diffusion model, allowing users to independently manipulate foreground and background elements in distinct layers. To address the challenge of limited layer-specific data, we propose a data curation pipeline that features automated element segmentation, motion-state hierarchical merging, and motion coherence refinement. Through quantitative and qualitative comparisons, and user study, we demonstrate that LayerAnimate outperforms current methods in terms of animation quality, control precision, and usability, making it an ideal tool for both professional animators and amateur enthusiasts. This framework opens up new possibilities for layer-specific animation applications and creative flexibility. Our code is available at this https URL.
• 摘要：动画视频将前景和背景元素分成多个层，并采用不同的流程进行草图绘制、细化、着色和中间处理。现有的视频生成方法通常将动画视为一个整体数据域，缺乏对各个层的细粒度控制。在本文中，我们介绍了 LayerAnimate，这是一种新颖的架构方法，可增强视频扩散模型中对各个动画层的细粒度控制，允许用户独立操作不同层中的前景和背景元素。为了应对特定层数据有限的挑战，我们提出了一种数据管理流程，该流程具有自动元素分割、运动状态分层合并和运动连贯性细化功能。通过定量和定性比较以及用户研究，我们证明 LayerAnimate 在动画质量、控制精度和可用性方面优于当前方法，使其成为专业动画师和业余爱好者的理想工具。该框架为特定层的动画应用和创作灵活性开辟了新的可能性。我们的代码可在此 https URL 上找到。

Title: Diffusion Adversarial Post-Training for One-Step Video Generation

• Authors: Shanchuan Lin, Xin Xia, Yuxi Ren, Ceyuan Yang, Xuefeng Xiao, Lu Jiang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.08316
• Pdf URL: https://arxiv.org/pdf/2501.08316
• Copy Paste: [[2501.08316]] Diffusion Adversarial Post-Training for One-Step Video Generation(https://arxiv.org/abs/2501.08316)
• Keywords: generation
• Abstract: The diffusion models are widely used for image and video generation, but their iterative generation process is slow and expansive. While existing distillation approaches have demonstrated the potential for one-step generation in the image domain, they still suffer from significant quality degradation. In this work, we propose Adversarial Post-Training (APT) against real data following diffusion pre-training for one-step video generation. To improve the training stability and quality, we introduce several improvements to the model architecture and training procedures, along with an approximated R1 regularization objective. Empirically, our experiments show that our adversarial post-trained model, Seaweed-APT, can generate 2-second, 1280x720, 24fps videos in real time using a single forward evaluation step. Additionally, our model is capable of generating 1024px images in a single step, achieving quality comparable to state-of-the-art methods.
• 摘要：扩散模型广泛用于图像和视频生成，但其迭代生成过程缓慢且耗时。虽然现有的蒸馏方法已经证明了在图像域中一步生成的潜力，但它们仍然遭受严重的质量下降。在这项工作中，我们提出了针对真实数据的对抗性后训练 (APT)，在扩散预训练之后进行一步视频生成。为了提高训练稳定性和质量，我们对模型架构和训练程序进行了多项改进，并提出了近似的 R1 正则化目标。从经验上讲，我们的实验表明，我们的对抗性后训练模型 Seaweed-APT 可以使用单个前向评估步骤实时生成 2 秒、1280x720、24fps 的视频。此外，我们的模型能够一步生成 1024px 图像，实现与最先进方法相当的质量。

Title: GameFactory: Creating New Games with Generative Interactive Videos

• Authors: Jiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, Xihui Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.08325
• Pdf URL: https://arxiv.org/pdf/2501.08325
• Copy Paste: [[2501.08325]] GameFactory: Creating New Games with Generative Interactive Videos(https://arxiv.org/abs/2501.08325)
• Keywords: generation, generative
• Abstract: Generative game engines have the potential to revolutionize game development by autonomously creating new content and reducing manual workload. However, existing video-based game generation methods fail to address the critical challenge of scene generalization, limiting their applicability to existing games with fixed styles and scenes. In this paper, we present GameFactory, a framework focused on exploring scene generalization in game video generation. To enable the creation of entirely new and diverse games, we leverage pre-trained video diffusion models trained on open-domain video data. To bridge the domain gap between open-domain priors and small-scale game dataset, we propose a multi-phase training strategy that decouples game style learning from action control, preserving open-domain generalization while achieving action controllability. Using Minecraft as our data source, we release GF-Minecraft, a high-quality and diversity action-annotated video dataset for research. Furthermore, we extend our framework to enable autoregressive action-controllable game video generation, allowing the production of unlimited-length interactive game videos. Experimental results demonstrate that GameFactory effectively generates open-domain, diverse, and action-controllable game videos, representing a significant step forward in AI-driven game generation. Our dataset and project page are publicly available at \url{this https URL}.
• 摘要：生成式游戏引擎有可能通过自主创建新内容和减少手动工作量来彻底改变游戏开发。然而，现有的基于视频的游戏生成方法未能解决场景泛化的关键挑战，限制了它们对具有固定风格和场景的现有游戏的适用性。在本文中，我们介绍了 GameFactory，这是一个专注于探索游戏视频生成中的场景泛化的框架。为了能够创建全新的多样化游戏，我们利用在开放域视频数据上训练的预训练视频扩散模型。为了弥合开放域先验和小规模游戏数据集之间的领域差距，我们提出了一种多阶段训练策略，将游戏风格学习与动作控制分离，在实现动作可控性的同时保留开放域泛化。使用 Minecraft 作为我们的数据源，我们发布了 GF-Minecraft，这是一个高质量且多样化的动作注释视频数据集，用于研究。此外，我们扩展了我们的框架以实现自回归动作可控游戏视频生成，从而可以制作无限长度的交互式游戏视频。实验结果表明，GameFactory 可以有效生成开放域、多样化且可控制动作的游戏视频，代表了人工智能驱动的游戏生成领域向前迈出的重要一步。我们的数据集和项目页面可在 \url{此 https URL} 上公开获取。

Title: Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise

• Authors: Ryan Burgert, Yuancheng Xu, Wenqi Xian, Oliver Pilarski, Pascal Clausen, Mingming He, Li Ma, Yitong Deng, Lingxiao Li, Mohsen Mousavi, Michael Ryoo, Paul Debevec, Ning Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.08331
• Pdf URL: https://arxiv.org/pdf/2501.08331
• Copy Paste: [[2501.08331]] Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise(https://arxiv.org/abs/2501.08331)
• Keywords: generative
• Abstract: Generative modeling aims to transform random noise into structured outputs. In this work, we enhance video diffusion models by allowing motion control via structured latent noise sampling. This is achieved by just a change in data: we pre-process training videos to yield structured noise. Consequently, our method is agnostic to diffusion model design, requiring no changes to model architectures or training pipelines. Specifically, we propose a novel noise warping algorithm, fast enough to run in real time, that replaces random temporal Gaussianity with correlated warped noise derived from optical flow fields, while preserving the spatial Gaussianity. The efficiency of our algorithm enables us to fine-tune modern video diffusion base models using warped noise with minimal overhead, and provide a one-stop solution for a wide range of user-friendly motion control: local object motion control, global camera movement control, and motion transfer. The harmonization between temporal coherence and spatial Gaussianity in our warped noise leads to effective motion control while maintaining per-frame pixel quality. Extensive experiments and user studies demonstrate the advantages of our method, making it a robust and scalable approach for controlling motion in video diffusion models. Video results are available on our webpage: this https URL source code and model checkpoints are available on GitHub: this https URL.
• 摘要：生成建模旨在将随机噪声转换为结构化输出。在这项工作中，我们通过允许通过结构化潜在噪声采样进行运动控制来增强视频扩散模型。这只需更改数据即可实现：我们对训练视频进行预处理以产生结构化噪声。因此，我们的方法与扩散模型设计无关，不需要更改模型架构或训练管道。具体来说，我们提出了一种新颖的噪声扭曲算法，该算法足够快以实时运行，它用来自光流场的相关扭曲噪声取代随机时间高斯性，同时保留空间高斯性。我们算法的效率使我们能够以最小的开销使用扭曲噪声微调现代视频扩散基础模型，并为广泛的用户友好型运动控制提供一站式解决方案：局部物体运动控制、全局相机运动控制和运动传递。我们的扭曲噪声中时间相干性和空间高斯性之间的协调可实现有效的运动控制，同时保持每帧像素质量。大量实验和用户研究证明了我们方法的优势，使其成为一种用于控制视频扩散模型中的运动的稳健且可扩展的方法。视频结果可在我们的网页上找到：此 https URL 源代码和模型检查点可在 GitHub 上找到：此 https URL。