2024-12-19

Title: CompactFlowNet: Efficient Real-time Optical Flow Estimation on Mobile Devices

• Authors: Andrei Znobishchev, Valerii Filev, Oleg Kudashev, Nikita Orlov, Humphrey Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13273
• Pdf URL: https://arxiv.org/pdf/2412.13273
• Copy Paste: [[2412.13273]] CompactFlowNet: Efficient Real-time Optical Flow Estimation on Mobile Devices(https://arxiv.org/abs/2412.13273)
• Keywords: restoration, generation
• Abstract: We present CompactFlowNet, the first real-time mobile neural network for optical flow prediction, which involves determining the displacement of each pixel in an initial frame relative to the corresponding pixel in a subsequent frame. Optical flow serves as a fundamental building block for various video-related tasks, such as video restoration, motion estimation, video stabilization, object tracking, action recognition, and video generation. While current state-of-the-art methods prioritize accuracy, they often overlook constraints regarding speed and memory usage. Existing light models typically focus on reducing size but still exhibit high latency, compromise significantly on quality, or are optimized for high-performance GPUs, resulting in sub-optimal performance on mobile devices. This study aims to develop a mobile-optimized optical flow model by proposing a novel mobile device-compatible architecture, as well as enhancements to the training pipeline, which optimize the model for reduced weight, low memory utilization, and increased speed while maintaining minimal error. Our approach demonstrates superior or comparable performance to the state-of-the-art lightweight models on the challenging KITTI and Sintel benchmarks. Furthermore, it attains a significantly accelerated inference speed, thereby yielding real-time operational efficiency on the iPhone 8, while surpassing real-time performance levels on more advanced mobile devices.
• 摘要：我们提出了 CompactFlowNet，这是第一个用于光流预测的实时移动神经网络，它涉及确定初始帧中每个像素相对于后续帧中相应像素的位移。光流是各种视频相关任务的基本构建块，例如视频恢复、运动估计、视频稳定、对象跟踪、动作识别和视频生成。虽然当前最先进的方法优先考虑准确性，但它们往往忽略了速度和内存使用方面的限制。现有的轻量级模型通常专注于减小尺寸，但仍然表现出高延迟、质量大幅下降或针对高性能 GPU 进行了优化，导致移动设备上的性能不佳。本研究旨在通过提出一种新颖的移动设备兼容架构以及对训练管道的增强来开发移动优化的光流模型，从而优化模型以减轻重量、降低内存利用率并提高速度，同时保持最小错误。我们的方法在具有挑战性的 KITTI 和 Sintel 基准上表现出优于或可与最先进的轻量级模型相媲美的性能。此外，其推理速度也显著加快，从而在 iPhone 8 上实现实时运行效率，同时超越更先进的移动设备上的实时性能水平。

Title: Posterior Mean Matching: Generative Modeling through Online Bayesian Inference

• Authors: Sebastian Salazar, Michal Kucer, Yixin Wang, Emily Casleton, David Blei
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.13286
• Pdf URL: https://arxiv.org/pdf/2412.13286
• Copy Paste: [[2412.13286]] Posterior Mean Matching: Generative Modeling through Online Bayesian Inference(https://arxiv.org/abs/2412.13286)
• Keywords: generation, generative
• Abstract: This paper introduces posterior mean matching (PMM), a new method for generative modeling that is grounded in Bayesian inference. PMM uses conjugate pairs of distributions to model complex data of various modalities like images and text, offering a flexible alternative to existing methods like diffusion models. PMM models iteratively refine noisy approximations of the target distribution using updates from online Bayesian inference. PMM is flexible because its mechanics are based on general Bayesian models. We demonstrate this flexibility by developing specialized examples: a generative PMM model of real-valued data using the Normal-Normal model, a generative PMM model of count data using a Gamma-Poisson model, and a generative PMM model of discrete data using a Dirichlet-Categorical model. For the Normal-Normal PMM model, we establish a direct connection to diffusion models by showing that its continuous-time formulation converges to a stochastic differential equation (SDE). Additionally, for the Gamma-Poisson PMM, we derive a novel SDE driven by a Cox process, which is a significant departure from traditional Brownian motion-based generative models. PMMs achieve performance that is competitive with generative models for language modeling and image generation.
• 摘要：本文介绍了后验均值匹配 (PMM)，这是一种基于贝叶斯推理的生成建模新方法。PMM 使用共轭分布对来对各种模态（如图像和文本）的复杂数据进行建模，为现有方法（如扩散模型）提供了一种灵活的替代方案。PMM 模型使用在线贝叶斯推理的更新迭代地细化目标分布的噪声近似值。PMM 非常灵活，因为它的机制基于一般的贝叶斯模型。我们通过开发专门的示例来展示这种灵活性：使用正态-正态模型的实值数据生成 PMM 模型、使用伽马-泊松模型的计数数据生成 PMM 模型以及使用狄利克雷-分类模型的离散数据生成 PMM 模型。对于正态-正态 PMM 模型，我们通过展示其连续时间公式收敛到随机微分方程 (SDE) 来建立与扩散模型的直接联系。此外，对于 Gamma-Poisson PMM，我们推导出一种由 Cox 过程驱动的新型 SDE，这与传统的基于布朗运动的生成模型有很大不同。PMM 的性能可与语言建模和图像生成的生成模型相媲美。

Title: Wind Speed Forecasting Based on Data Decomposition and Deep Learning Models: A Case Study of a Wind Farm in Saudi Arabia

• Authors: Yasmeen Aldossary, Nabil Hewahi, Abdulla Alasaadi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13356
• Pdf URL: https://arxiv.org/pdf/2412.13356
• Copy Paste: [[2412.13356]] Wind Speed Forecasting Based on Data Decomposition and Deep Learning Models: A Case Study of a Wind Farm in Saudi Arabia(https://arxiv.org/abs/2412.13356)
• Keywords: generation
• Abstract: With industrial and technological development and the increasing demand for electric power, wind energy has gradually become the fastest-growing and most environmentally friendly new energy source. Nevertheless, wind power generation is always accompanied by uncertainty due to the wind speed's volatility. Wind speed forecasting (WSF) is essential for power grids' dispatch, stability, and controllability, and its accuracy is crucial to effectively using wind resources. Therefore, this study proposes a novel WSF framework for stationary data based on a hybrid decomposition method and the Bidirectional Long Short-term Memory (BiLSTM) to achieve high forecasting accuracy for the Dumat Al-Jandal wind farm in Al-Jouf, Saudi Arabia. The hybrid decomposition method combines the Wavelet Packet Decomposition (WPD) and the Seasonal Adjustment Method (SAM). The SAM method eliminates the seasonal component of the decomposed subseries generated by WPD to reduce forecasting complexity. The BiLSTM is applied to forecast all the deseasonalized decomposed subseries. Five years of hourly wind speed observations acquired from a location in the Al-Jouf region were used to prove the effectiveness of the proposed model. The comparative experimental results, including 27 other models, demonstrated the proposed model's superiority in single and multiple WSF with an overall average mean absolute error of 0.176549, root mean square error of 0.247069, and R-squared error of 0.985987.
• 摘要：随着工业和技术的发展以及对电力需求的不断增加，风能逐渐成为增长最快、最环保的新能源。然而，由于风速的波动性，风力发电总是伴随着不确定性。风速预测（WSF）对于电网的调度、稳定性和可控性至关重要，其准确性对于有效利用风能资源至关重要。因此，本研究提出了一种基于混合分解方法和双向长短期记忆（BiLSTM）的新型平稳数据WSF框架，以实现对沙特阿拉伯Al-Jouf的Dumat Al-Jandal风电场的高精度预测。混合分解方法结合了小波包分解（WPD）和季节调整方法（SAM）。SAM方法消除了WPD生成的分解子序列的季节性成分，以降低预测复杂度。BiLSTM用于预测所有去季节化的分解子序列。利用焦夫地区某地5年的逐时风速观测数据验证了所提模型的有效性，对比试验结果（包括27个其他模型）表明，所提模型在单次和多次风速预测中均表现出优越性，总体平均绝对误差为0.176549，均方根误差为0.247069，R平方误差为0.985987。

Title: Marigold-DC: Zero-Shot Monocular Depth Completion with Guided Diffusion

• Authors: Massimiliano Viola, Kevin Qu, Nando Metzger, Bingxin Ke, Alexander Becker, Konrad Schindler, Anton Obukhov
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13389
• Pdf URL: https://arxiv.org/pdf/2412.13389
• Copy Paste: [[2412.13389]] Marigold-DC: Zero-Shot Monocular Depth Completion with Guided Diffusion(https://arxiv.org/abs/2412.13389)
• Keywords: generation
• Abstract: Depth completion upgrades sparse depth measurements into dense depth maps guided by a conventional image. Existing methods for this highly ill-posed task operate in tightly constrained settings and tend to struggle when applied to images outside the training domain or when the available depth measurements are sparse, irregularly distributed, or of varying density. Inspired by recent advances in monocular depth estimation, we reframe depth completion as an image-conditional depth map generation guided by sparse measurements. Our method, Marigold-DC, builds on a pretrained latent diffusion model for monocular depth estimation and injects the depth observations as test-time guidance via an optimization scheme that runs in tandem with the iterative inference of denoising diffusion. The method exhibits excellent zero-shot generalization across a diverse range of environments and handles even extremely sparse guidance effectively. Our results suggest that contemporary monocular depth priors greatly robustify depth completion: it may be better to view the task as recovering dense depth from (dense) image pixels, guided by sparse depth; rather than as inpainting (sparse) depth, guided by an image. Project website: this https URL
• 摘要：深度补全将稀疏深度测量升级为由传统图像引导的密集深度图。现有的针对这种高度不适定任务的方法在严格约束的设置下运行，当应用于训练域之外的图像或可用的深度测量稀疏、分布不规则或密度各异时往往会遇到困难。受到单目深度估计最新进展的启发，我们将深度补全重新定义为由稀疏测量引导的图像条件深度图生成。我们的方法 Marigold-DC 建立在用于单目深度估计的预训练潜在扩散模型之上，并通过与去噪扩散的迭代推理协同运行的优化方案将深度观测作为测试时指导注入。该方法在各种环境中都表现出出色的零样本泛化能力，甚至可以有效处理极其稀疏的指导。我们的结果表明，当代的单目深度先验极大地增强了深度补全：最好将任务视为在稀疏深度的引导下从（密集）图像像素中恢复密集深度；而不是通过图像引导的修复（稀疏）深度。项目网站：此 https URL

Title: MMHMR: Generative Masked Modeling for Hand Mesh Recovery

• Authors: Muhammad Usama Saleem, Ekkasit Pinyoanuntapong, Mayur Jagdishbhai Patel, Hongfei Xue, Ahmed Helmy, Srijan Das, Pu Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13393
• Pdf URL: https://arxiv.org/pdf/2412.13393
• Copy Paste: [[2412.13393]] MMHMR: Generative Masked Modeling for Hand Mesh Recovery(https://arxiv.org/abs/2412.13393)
• Keywords: generative
• Abstract: Reconstructing a 3D hand mesh from a single RGB image is challenging due to complex articulations, self-occlusions, and depth ambiguities. Traditional discriminative methods, which learn a deterministic mapping from a 2D image to a single 3D mesh, often struggle with the inherent ambiguities in 2D-to-3D mapping. To address this challenge, we propose MMHMR, a novel generative masked model for hand mesh recovery that synthesizes plausible 3D hand meshes by learning and sampling from the probabilistic distribution of the ambiguous 2D-to-3D mapping process. MMHMR consists of two key components: (1) a VQ-MANO, which encodes 3D hand articulations as discrete pose tokens in a latent space, and (2) a Context-Guided Masked Transformer that randomly masks out pose tokens and learns their joint distribution, conditioned on corrupted token sequences, image context, and 2D pose cues. This learned distribution facilitates confidence-guided sampling during inference, producing mesh reconstructions with low uncertainty and high precision. Extensive evaluations on benchmark and real-world datasets demonstrate that MMHMR achieves state-of-the-art accuracy, robustness, and realism in 3D hand mesh reconstruction. Project website: this https URL
• 摘要：由于复杂的关节、自遮挡和深度模糊性，从单个 RGB 图像重建 3D 手部网格具有挑战性。传统的判别方法学习从 2D 图像到单个 3D 网格的确定性映射，通常难以应对 2D 到 3D 映射中固有的模糊性。为了应对这一挑战，我们提出了 MMHMR，这是一种用于手部网格恢复的新型生成式掩蔽模型，它通过从模糊的 2D 到 3D 映射过程的概率分布中学习和采样来合成合理的 3D 手部网格。MMHMR 由两个关键组件组成：(1) VQ-MANO，它将 3D 手部关节编码为潜在空间中的离散姿势标记，以及 (2) 上下文引导的掩蔽变换器，它随机掩蔽姿势标记并学习它们的联合分布，以损坏的标记序列、图像上下文和 2D 姿势线索为条件。这种学习分布有助于在推理过程中进行置信度引导采样，从而产生具有低不确定性和高精度的网格重建。对基准和真实世界数据集的广泛评估表明，MMHMR 在 3D 手部网格重建中实现了最先进的准确性、稳健性和真实性。项目网站：此 https URL

Title: Zero-Shot Low Light Image Enhancement with Diffusion Prior

• Authors: Joshua Cho, Sara Aghajanzadeh, Zhen Zhu, D. A. Forsyth
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13401
• Pdf URL: https://arxiv.org/pdf/2412.13401
• Copy Paste: [[2412.13401]] Zero-Shot Low Light Image Enhancement with Diffusion Prior(https://arxiv.org/abs/2412.13401)
• Keywords: generative
• Abstract: Balancing aesthetic quality with fidelity when enhancing images from challenging, degraded sources is a core objective in computational photography. In this paper, we address low light image enhancement (LLIE), a task in which dark images often contain limited visible information. Diffusion models, known for their powerful image enhancement capacities, are a natural choice for this problem. However, their deep generative priors can also lead to hallucinations, introducing non-existent elements or substantially altering the visual semantics of the original scene. In this work, we introduce a novel zero-shot method for controlling and refining the generative behavior of diffusion models for dark-to-light image conversion tasks. Our method demonstrates superior performance over existing state-of-the-art methods in the task of low-light image enhancement, as evidenced by both quantitative metrics and qualitative analysis.
• 摘要：在增强来自具有挑战性的劣化源的图像时，平衡美学质量和保真度是计算摄影的核心目标。在本文中，我们讨论了低光图像增强 (LLIE)，在这项任务中，暗图像通常包含有限的可见信息。以强大的图像增强能力而闻名的扩散模型是解决这一问题的自然选择。然而，它们的深度生成先验也会导致幻觉，引入不存在的元素或大幅改变原始场景的视觉语义。在这项工作中，我们引入了一种新颖的零样本方法来控制和改进扩散模型在暗到亮图像转换任务中的生成行为。定量指标和定性分析都证明了我们的方法在低光图像增强任务中比现有的最先进方法具有更优异的性能。

Title: DarkIR: Robust Low-Light Image Restoration

• Authors: Daniel Feijoo, Juan C. Benito, Alvaro Garcia, Marcos V. Conde
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.13443
• Pdf URL: https://arxiv.org/pdf/2412.13443
• Copy Paste: [[2412.13443]] DarkIR: Robust Low-Light Image Restoration(https://arxiv.org/abs/2412.13443)
• Keywords: restoration
• Abstract: Photography during night or in dark conditions typically suffers from noise, low light and blurring issues due to the dim environment and the common use of long exposure. Although Deblurring and Low-light Image Enhancement (LLIE) are related under these conditions, most approaches in image restoration solve these tasks separately. In this paper, we present an efficient and robust neural network for multi-task low-light image restoration. Instead of following the current tendency of Transformer-based models, we propose new attention mechanisms to enhance the receptive field of efficient CNNs. Our method reduces the computational costs in terms of parameters and MAC operations compared to previous methods. Our model, DarkIR, achieves new state-of-the-art results on the popular LOLBlur, LOLv2 and Real-LOLBlur datasets, being able to generalize on real-world night and dark images. Code and models at this https URL
• 摘要：由于环境昏暗和长时间曝光的普遍使用，夜间或黑暗条件下的摄影通常会受到噪音、弱光和模糊问题的影响。尽管在这些条件下去模糊和低光图像增强 (LLIE) 是相关的，但图像恢复中的大多数方法都是单独解决这些任务的。在本文中，我们提出了一种高效且稳健的神经网络，用于多任务低光图像恢复。我们没有遵循基于 Transformer 的模型的当前趋势，而是提出了新的注意力机制来增强高效 CNN 的感受野。与以前的方法相比，我们的方法降低了参数和 MAC 操作方面的计算成本。我们的模型 DarkIR 在流行的 LOLBlur、LOLv2 和 Real-LOLBlur 数据集上取得了新的最佳结果，能够在现实世界的夜晚和黑暗图像上推广。代码和模型在此 https URL

Title: FlexPose: Pose Distribution Adaptation with Limited Guidance

• Authors: Zixiao Wang, Junwu Weng, Mengyuan Liu, Bei Yu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13463
• Pdf URL: https://arxiv.org/pdf/2412.13463
• Copy Paste: [[2412.13463]] FlexPose: Pose Distribution Adaptation with Limited Guidance(https://arxiv.org/abs/2412.13463)
• Keywords: generative
• Abstract: Numerous well-annotated human key-point datasets are publicly available to date. However, annotating human poses for newly collected images is still a costly and time-consuming progress. Pose distributions from different datasets share similar pose hinge-structure priors with different geometric transformations, such as pivot orientation, joint rotation, and bone length ratio. The difference between Pose distributions is essentially the difference between the transformation distributions. Inspired by this fact, we propose a method to calibrate a pre-trained pose generator in which the pose prior has already been learned to an adapted one following a new pose distribution. We treat the representation of human pose joint coordinates as skeleton image and transfer a pre-trained pose annotation generator with only a few annotation guidance. By fine-tuning a limited number of linear layers that closely related to the pose transformation, the adapted generator is able to produce any number of pose annotations that are similar to the target poses. We evaluate our proposed method, FlexPose, on several cross-dataset settings both qualitatively and quantitatively, which demonstrates that our approach achieves state-of-the-art performance compared to the existing generative-model-based transfer learning methods when given limited annotation guidance.
• 摘要：迄今为止，已有大量经过良好注释的人体关键点数据集可供公开使用。然而，为新收集的图像注释人体姿势仍然是一个昂贵且耗时的过程。来自不同数据集的姿势分布共享相似的姿势铰链结构先验，但具有不同的几何变换，例如枢轴方向、关节旋转和骨骼长度比。姿势分布之间的差异本质上是变换分布之间的差异。受此事实的启发，我们提出了一种方法来校准预训练的姿势生成器，其中已经将姿势先验学习为遵循新姿势分布的自适应姿势生成器。我们将人体姿势关节坐标的表示视为骨架图像，并仅使用少量注释指导来传输预训练的姿势注释生成器。通过微调与姿势变换密切相关的有限数量的线性层，自适应生成器能够生成与目标姿势相似的任意数量的姿势注释。我们从定性和定量两个方面在多个跨数据集设置上对我们提出的方法 FlexPose 进行了评估，这表明，在有限的注释指导下，与现有的基于生成模型的迁移学习方法相比，我们的方法实现了最先进的性能。

Title: Real-time One-Step Diffusion-based Expressive Portrait Videos Generation

• Authors: Hanzhong Guo, Hongwei Yi, Daquan Zhou, Alexander William Bergman, Michael Lingelbach, Yizhou Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13479
• Pdf URL: https://arxiv.org/pdf/2412.13479
• Copy Paste: [[2412.13479]] Real-time One-Step Diffusion-based Expressive Portrait Videos Generation(https://arxiv.org/abs/2412.13479)
• Keywords: generation
• Abstract: Latent diffusion models have made great strides in generating expressive portrait videos with accurate lip-sync and natural motion from a single reference image and audio input. However, these models are far from real-time, often requiring many sampling steps that take minutes to generate even one second of video-significantly limiting practical use. We introduce OSA-LCM (One-Step Avatar Latent Consistency Model), paving the way for real-time diffusion-based avatars. Our method achieves comparable video quality to existing methods but requires only one sampling step, making it more than 10x faster. To accomplish this, we propose a novel avatar discriminator design that guides lip-audio consistency and motion expressiveness to enhance video quality in limited sampling steps. Additionally, we employ a second-stage training architecture using an editing fine-tuned method (EFT), transforming video generation into an editing task during training to effectively address the temporal gap challenge in single-step generation. Experiments demonstrate that OSA-LCM outperforms existing open-source portrait video generation models while operating more efficiently with a single sampling step.
• 摘要：潜在扩散模型在从单个参考图像和音频输入生成具有准确口型同步和自然动作的富有表现力的肖像视频方面取得了巨大进步。然而，这些模型远非实时，通常需要许多采样步骤，甚至需要几分钟才能生成一秒钟的视频 - 严重限制了实际使用。我们引入了 OSA-LCM（一步化身潜在一致性模型），为基于实时扩散的化身铺平了道路。我们的方法实现了与现有方法相当的视频质量，但只需要一个采样步骤，使其速度提高了 10 倍以上。为了实现这一点，我们提出了一种新颖的化身鉴别器设计，它可以引导唇音一致性和运动表现力，以在有限的采样步骤中提高视频质量。此外，我们采用了使用编辑微调方法 (EFT) 的第二阶段训练架构，在训练期间将视频生成转变为编辑任务，以有效解决单步生成中的时间间隔挑战。实验表明，OSA-LCM 优于现有的开源肖像视频生成模型，同时通过单个采样步骤更高效地运行。

Title: T$^3$-S2S: Training-free Triplet Tuning for Sketch to Scene Generation

• Authors: Zhenhong Sun, Yifu Wang, Yonhon Ng, Yunfei Duan, Daoyi Dong, Hongdong Li, Pan Ji
• Subjects: cs.CV, cs.CL, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.13486
• Pdf URL: https://arxiv.org/pdf/2412.13486
• Copy Paste: [[2412.13486]] T$^3$-S2S: Training-free Triplet Tuning for Sketch to Scene Generation(https://arxiv.org/abs/2412.13486)
• Keywords: generation, generative
• Abstract: Scene generation is crucial to many computer graphics applications. Recent advances in generative AI have streamlined sketch-to-image workflows, easing the workload for artists and designers in creating scene concept art. However, these methods often struggle for complex scenes with multiple detailed objects, sometimes missing small or uncommon instances. In this paper, we propose a Training-free Triplet Tuning for Sketch-to-Scene (T3-S2S) generation after reviewing the entire cross-attention mechanism. This scheme revitalizes the existing ControlNet model, enabling effective handling of multi-instance generations, involving prompt balance, characteristics prominence, and dense tuning. Specifically, this approach enhances keyword representation via the prompt balance module, reducing the risk of missing critical instances. It also includes a characteristics prominence module that highlights TopK indices in each channel, ensuring essential features are better represented based on token sketches. Additionally, it employs dense tuning to refine contour details in the attention map, compensating for instance-related regions. Experiments validate that our triplet tuning approach substantially improves the performance of existing sketch-to-image models. It consistently generates detailed, multi-instance 2D images, closely adhering to the input prompts and enhancing visual quality in complex multi-instance scenes. Code is available at this https URL.
• 摘要：场景生成对于许多计算机图形应用至关重要。生成式人工智能的最新进展简化了从草图到图像的工作流程，减轻了艺术家和设计师在创作场景概念艺术方面的工作量。然而，这些方法通常难以处理具有多个详细对象的复杂场景，有时会遗漏小的或不常见的实例。在本文中，我们在回顾了整个交叉注意机制后，提出了一种无需训练的三重调优草图到场景 (T3-S2S) 生成方法。该方案使现有的 ControlNet 模型焕发活力，能够有效处理多实例生成，包括提示平衡、特征突出和密集调优。具体而言，这种方法通过提示平衡模块增强了关键字表示，降低了遗漏关键实例的风险。它还包括一个特征突出模块，该模块突出显示每个通道中的 TopK 索引，确保基于标记草图更好地表示基本特征。此外，它采用密集调优来细化注意力图中的轮廓细节，补偿与实例相关的区域。实验证明，我们的三重调优方法显著提高了现有草图转图像模型的性能。它可以一致地生成详细的多实例 2D 图像，紧密遵循输入提示并增强复杂多实例场景中的视觉质量。代码可在此 https URL 上获取。

Title: Learning Causal Transition Matrix for Instance-dependent Label Noise

• Authors: Jiahui Li, Tai-Wei Chang, Kun Kuang, Ximing Li, Long Chen, Jun Zhou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13516
• Pdf URL: https://arxiv.org/pdf/2412.13516
• Copy Paste: [[2412.13516]] Learning Causal Transition Matrix for Instance-dependent Label Noise(https://arxiv.org/abs/2412.13516)
• Keywords: generation
• Abstract: Noisy labels are both inevitable and problematic in machine learning methods, as they negatively impact models' generalization ability by causing overfitting. In the context of learning with noise, the transition matrix plays a crucial role in the design of statistically consistent algorithms. However, the transition matrix is often considered unidentifiable. One strand of methods typically addresses this problem by assuming that the transition matrix is instance-independent; that is, the probability of mislabeling a particular instance is not influenced by its characteristics or attributes. This assumption is clearly invalid in complex real-world scenarios. To better understand the transition relationship and relax this assumption, we propose to study the data generation process of noisy labels from a causal perspective. We discover that an unobservable latent variable can affect either the instance itself, the label annotation procedure, or both, which complicates the identification of the transition matrix. To address various scenarios, we have unified these observations within a new causal graph. In this graph, the input instance is divided into a noise-resistant component and a noise-sensitive component based on whether they are affected by the latent variable. These two components contribute to identifying the ``causal transition matrix'', which approximates the true transition matrix with theoretical guarantee. In line with this, we have designed a novel training framework that explicitly models this causal relationship and, as a result, achieves a more accurate model for inferring the clean label.
• 摘要：在机器学习方法中，噪声标签既不可避免又成问题，因为它们会导致过度拟合，从而对模型的泛化能力产生负面影响。在噪声学习的背景下，转换矩阵在统计一致算法的设计中起着至关重要的作用。然而，转换矩阵通常被认为是不可识别的。一种方法通常通过假设转换矩阵与实例无关来解决这个问题；也就是说，错误标记特定实例的概率不受其特征或属性的影响。这个假设在复杂的现实场景中显然是无效的。为了更好地理解转换关系并放宽这一假设，我们建议从因果角度研究噪声标签的数据生成过程。我们发现不可观察的潜在变量可以影响实例本身、标签注释过程或两者，这使转换矩阵的识别变得复杂。为了应对各种情况，我们在一个新的因果图中统一了这些观察结果。在这个图中，输入实例根据它们是否受潜在变量的影响被分为抗噪声组件和噪声敏感组件。这两个组成部分有助于识别“因果转换矩阵”，该矩阵在理论上可以近似真实转换矩阵。为此，我们设计了一个新颖的训练框架，明确地模拟这种因果关系，从而实现更准确的模型来推断清洁标签。

Title: Hybrid Data-Free Knowledge Distillation

• Authors: Jialiang Tang, Shuo Chen, Chen Gong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13525
• Pdf URL: https://arxiv.org/pdf/2412.13525
• Copy Paste: [[2412.13525]] Hybrid Data-Free Knowledge Distillation(https://arxiv.org/abs/2412.13525)
• Keywords: generation, generative
• Abstract: Data-free knowledge distillation aims to learn a compact student network from a pre-trained large teacher network without using the original training data of the teacher network. Existing collection-based and generation-based methods train student networks by collecting massive real examples and generating synthetic examples, respectively. However, they inevitably become weak in practical scenarios due to the difficulties in gathering or emulating sufficient real-world data. To solve this problem, we propose a novel method called \textbf{H}ybr\textbf{i}d \textbf{D}ata-\textbf{F}ree \textbf{D}istillation (HiDFD), which leverages only a small amount of collected data as well as generates sufficient examples for training student networks. Our HiDFD comprises two primary modules, \textit{i.e.}, the teacher-guided generation and student distillation. The teacher-guided generation module guides a Generative Adversarial Network (GAN) by the teacher network to produce high-quality synthetic examples from very few real-world collected examples. Specifically, we design a feature integration mechanism to prevent the GAN from overfitting and facilitate the reliable representation learning from the teacher network. Meanwhile, we drive a category frequency smoothing technique via the teacher network to balance the generative training of each category. In the student distillation module, we explore a data inflation strategy to properly utilize a blend of real and synthetic data to train the student network via a classifier-sharing-based feature alignment technique. Intensive experiments across multiple benchmarks demonstrate that our HiDFD can achieve state-of-the-art performance using 120 times less collected data than existing methods. Code is available at this https URL.
• 摘要：无数据知识提炼旨在从预先训练的大型教师网络中学习紧凑的学生网络，而不使用教师网络的原始训练数据。现有的基于集合和基于生成的方法分别通过收集大量真实示例和生成合成示例来训练学生网络。然而，由于难以收集或模拟足够的真实世界数据，它们在实际场景中不可避免地会变得薄弱。为了解决这个问题，我们提出了一种名为 \textbf{H}ybr\textbf{i}d \textbf{D}ata-\textbf{F}ree \textbf{D}istillation (HiDFD) 的新方法，它只利用少量收集的数据并生成足够的示例来训练学生网络。我们的 HiDFD 包含两个主要模块，\textit{i.e.}，即教师指导的生成和学生提炼。教师指导的生成模块通过教师网络指导生成对抗网络 (GAN) 从极少的真实世界收集示例中生成高质量的合成示例。具体来说，我们设计了一种特征集成机制，以防止 GAN 过度拟合并促进从教师网络进行可靠的表示学习。同时，我们通过教师网络驱动类别频率平滑技术来平衡每个类别的生成训练。在学生蒸馏模块中，我们探索了一种数据膨胀策略，以正确利用真实数据和合成数据的混合来通过基于分类器共享的特征对齐技术训练学生网络。在多个基准测试中开展的密集实验表明，我们的 HiDFD 可以使用比现有方法少 120 倍的收集数据实现最先进的性能。代码可在此 https URL 上找到。

Title: Information-Theoretic Generative Clustering of Documents

• Authors: Xin Du, Kumiko Tanaka-Ishii
• Subjects: cs.LG, cs.CL, cs.IR, cs.IT
• Abstract URL: https://arxiv.org/abs/2412.13534
• Pdf URL: https://arxiv.org/pdf/2412.13534
• Copy Paste: [[2412.13534]] Information-Theoretic Generative Clustering of Documents(https://arxiv.org/abs/2412.13534)
• Keywords: generative
• Abstract: We present {\em generative clustering} (GC) for clustering a set of documents, $\mathrm{X}$, by using texts $\mathrm{Y}$ generated by large language models (LLMs) instead of by clustering the original documents $\mathrm{X}$. Because LLMs provide probability distributions, the similarity between two documents can be rigorously defined in an information-theoretic manner by the KL divergence. We also propose a natural, novel clustering algorithm by using importance sampling. We show that GC achieves the state-of-the-art performance, outperforming any previous clustering method often by a large margin. Furthermore, we show an application to generative document retrieval in which documents are indexed via hierarchical clustering and our method improves the retrieval accuracy.
• 摘要：我们提出了生成聚类 (GC)，通过使用由大型语言模型 (LLM) 生成的文本 $\mathrm{Y}$ 来对一组文档 $\mathrm{X}$ 进行聚类，而不是通过对原始文档 $\mathrm{X}$ 进行聚类。由于 LLM 提供概率分布，因此可以通过 KL 散度以信息论方式严格定义两个文档之间的相似性。我们还提出了一种使用重要性抽样的自然、新颖的聚类算法。我们表明 GC 实现了最先进的性能，通常比任何以前的聚类方法都好很多。此外，我们展示了生成文档检索的一个应用，其中通过层次聚类对文档进行索引，我们的方法提高了检索准确性。

Title: CA-Edit: Causality-Aware Condition Adapter for High-Fidelity Local Facial Attribute Editing

• Authors: Xiaole Xian, Xilin He, Zenghao Niu, Junliang Zhang, Weicheng Xie, Siyang Song, Zitong Yu, Linlin Shen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13565
• Pdf URL: https://arxiv.org/pdf/2412.13565
• Copy Paste: [[2412.13565]] CA-Edit: Causality-Aware Condition Adapter for High-Fidelity Local Facial Attribute Editing(https://arxiv.org/abs/2412.13565)
• Keywords: generation
• Abstract: For efficient and high-fidelity local facial attribute editing, most existing editing methods either require additional fine-tuning for different editing effects or tend to affect beyond the editing regions. Alternatively, inpainting methods can edit the target image region while preserving external areas. However, current inpainting methods still suffer from the generation misalignment with facial attributes description and the loss of facial skin details. To address these challenges, (i) a novel data utilization strategy is introduced to construct datasets consisting of attribute-text-image triples from a data-driven perspective, (ii) a Causality-Aware Condition Adapter is proposed to enhance the contextual causality modeling of specific details, which encodes the skin details from the original image while preventing conflicts between these cues and textual conditions. In addition, a Skin Transition Frequency Guidance technique is introduced for the local modeling of contextual causality via sampling guidance driven by low-frequency alignment. Extensive quantitative and qualitative experiments demonstrate the effectiveness of our method in boosting both fidelity and editability for localized attribute editing. The code is available at this https URL.
• 摘要：为了实现高效、高保真的局部人脸属性编辑，大多数现有编辑方法要么需要额外微调以实现不同的编辑效果，要么往往会影响编辑区域之外的区域。或者，修复方法可以编辑目标图像区域，同时保留外部区域。然而，当前的修复方法仍然存在与人脸属性描述的生成不一致以及人脸皮肤细节丢失的问题。为了应对这些挑战，(i) 引入了一种新颖的数据利用策略，从数据驱动的角度构建由属性-文本-图像三元组组成的数据集，(ii) 提出了一种因果关系感知条件适配器来增强特定细节的上下文因果关系建模，它对原始图像中的皮肤细节进行编码，同时防止这些线索与文本条件发生冲突。此外，引入了一种皮肤转换频率引导技术，通过低频对齐驱动的采样引导对上下文因果关系进行局部建模。大量定量和定性实验证明了我们的方法在提高局部属性编辑的保真度和可编辑性方面的有效性。代码可在此 https URL 上获取。

Title: Bridge then Begin Anew: Generating Target-relevant Intermediate Model for Source-free Visual Emotion Adaptation

• Authors: Jiankun Zhu, Sicheng Zhao, Jing Jiang, Wenbo Tang, Zhaopan Xu, Tingting Han, Pengfei Xu, Hongxun Yao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.13577
• Pdf URL: https://arxiv.org/pdf/2412.13577
• Copy Paste: [[2412.13577]] Bridge then Begin Anew: Generating Target-relevant Intermediate Model for Source-free Visual Emotion Adaptation(https://arxiv.org/abs/2412.13577)
• Keywords: generation
• Abstract: Visual emotion recognition (VER), which aims at understanding humans' emotional reactions toward different visual stimuli, has attracted increasing attention. Given the subjective and ambiguous characteristics of emotion, annotating a reliable large-scale dataset is hard. For reducing reliance on data labeling, domain adaptation offers an alternative solution by adapting models trained on labeled source data to unlabeled target data. Conventional domain adaptation methods require access to source data. However, due to privacy concerns, source emotional data may be inaccessible. To address this issue, we propose an unexplored task: source-free domain adaptation (SFDA) for VER, which does not have access to source data during the adaptation process. To achieve this, we propose a novel framework termed Bridge then Begin Anew (BBA), which consists of two steps: domain-bridged model generation (DMG) and target-related model adaptation (TMA). First, the DMG bridges cross-domain gaps by generating an intermediate model, avoiding direct alignment between two VER datasets with significant differences. Then, the TMA begins training the target model anew to fit the target structure, avoiding the influence of source-specific knowledge. Extensive experiments are conducted on six SFDA settings for VER. The results demonstrate the effectiveness of BBA, which achieves remarkable performance gains compared with state-of-the-art SFDA methods and outperforms representative unsupervised domain adaptation approaches.
• 摘要：视觉情绪识别 (VER) 旨在理解人类对不同视觉刺激的情绪反应，引起了越来越多的关注。鉴于情绪的主观性和模糊性，注释可靠的大规模数据集非常困难。为了减少对数据标记的依赖，领域自适应提供了一种替代解决方案，即将在标记的源数据上训练的模型调整到未标记的目标数据。传统的领域自适应方法需要访问源数据。然而，出于隐私方面的考虑，源情绪数据可能无法访问。为了解决这个问题，我们提出了一项尚未探索的任务：VER 的无源领域自适应 (SFDA)，它在自适应过程中无法访问源数据。为了实现这一点，我们提出了一个称为 Bridge then Begin Anew (BBA) 的新框架，它包括两个步骤：领域桥接模型生成 (DMG) 和目标相关模型自适应 (TMA)。首先，DMG 通过生成中间模型来弥合跨域差距，避免两个具有显著差异的 VER 数据集之间的直接对齐。然后，TMA 开始重新训练目标模型以适应目标结构，避免源特定知识的影响。对 VER 的六种 SFDA 设置进行了广泛的实验。结果证明了 BBA 的有效性，与最先进的 SFDA 方法相比，它实现了显着的性能提升，并且优于代表性的无监督领域自适应方法。

Title: Unlocking the Potential of Weakly Labeled Data: A Co-Evolutionary Learning Framework for Abnormality Detection and Report Generation

• Authors: Jinghan Sun, Dong Wei, Zhe Xu, Donghuan Lu, Hong Liu, Hong Wang, Sotirios A. Tsaftaris, Steven McDonagh, Yefeng Zheng, Liansheng Wang
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13599
• Pdf URL: https://arxiv.org/pdf/2412.13599
• Copy Paste: [[2412.13599]] Unlocking the Potential of Weakly Labeled Data: A Co-Evolutionary Learning Framework for Abnormality Detection and Report Generation(https://arxiv.org/abs/2412.13599)
• Keywords: generation
• Abstract: Anatomical abnormality detection and report generation of chest X-ray (CXR) are two essential tasks in clinical practice. The former aims at localizing and characterizing cardiopulmonary radiological findings in CXRs, while the latter summarizes the findings in a detailed report for further diagnosis and treatment. Existing methods often focused on either task separately, ignoring their correlation. This work proposes a co-evolutionary abnormality detection and report generation (CoE-DG) framework. The framework utilizes both fully labeled (with bounding box annotations and clinical reports) and weakly labeled (with reports only) data to achieve mutual promotion between the abnormality detection and report generation tasks. Specifically, we introduce a bi-directional information interaction strategy with generator-guided information propagation (GIP) and detector-guided information propagation (DIP). For semi-supervised abnormality detection, GIP takes the informative feature extracted by the generator as an auxiliary input to the detector and uses the generator's prediction to refine the detector's pseudo labels. We further propose an intra-image-modal self-adaptive non-maximum suppression module (SA-NMS). This module dynamically rectifies pseudo detection labels generated by the teacher detection model with high-confidence predictions by the this http URL, for report generation, DIP takes the abnormalities' categories and locations predicted by the detector as input and guidance for the generator to improve the generated reports.
• 摘要：胸部X光（CXR）的解剖异常检测和报告生成是临床实践中的两项基本任务。前者旨在定位和表征 CXR 中的心肺放射学发现，而后者则在详细报告中总结发现，以便进一步诊断和治疗。现有方法通常分别关注任一任务，而忽略它们之间的相关性。这项工作提出了一个共同进化的异常检测和报告生成 (CoE-DG) 框架。该框架利用完全标记（带有边界框注释和临床报告）和弱标记（仅带有报告）数据来实现异常检测和报告生成任务之间的相互促进。具体而言，我们引入了一种双向信息交互策略，即生成器引导的信息传播 (GIP) 和检测器引导的信息传播 (DIP)。对于半监督异常检测，GIP 将生成器提取的信息特征作为检测器的辅助输入，并使用生成器的预测来细化检测器的伪标签。我们进一步提出了一个图像内模态自适应非最大抑制模块（SA-NMS）。该模块通过此 http URL 的高置信度预测动态纠正教师检测模型生成的伪检测标签，对于报告生成，DIP 将检测器预测的异常类别和位置作为输入和生成器的指导，以改进生成的报告。

Title: Sign-IDD: Iconicity Disentangled Diffusion for Sign Language Production

• Authors: Shengeng Tang, Jiayi He, Dan Guo, Yanyan Wei, Feng Li, Richang Hong
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.13609
• Pdf URL: https://arxiv.org/pdf/2412.13609
• Copy Paste: [[2412.13609]] Sign-IDD: Iconicity Disentangled Diffusion for Sign Language Production(https://arxiv.org/abs/2412.13609)
• Keywords: generation
• Abstract: Sign Language Production (SLP) aims to generate semantically consistent sign videos from textual statements, where the conversion from textual glosses to sign poses (G2P) is a crucial step. Existing G2P methods typically treat sign poses as discrete three-dimensional coordinates and directly fit them, which overlooks the relative positional relationships among joints. To this end, we provide a new perspective, constraining joint associations and gesture details by modeling the limb bones to improve the accuracy and naturalness of the generated poses. In this work, we propose a pioneering iconicity disentangled diffusion framework, termed Sign-IDD, specifically designed for SLP. Sign-IDD incorporates a novel Iconicity Disentanglement (ID) module to bridge the gap between relative positions among joints. The ID module disentangles the conventional 3D joint representation into a 4D bone representation, comprising the 3D spatial direction vector and 1D spatial distance vector between adjacent joints. Additionally, an Attribute Controllable Diffusion (ACD) module is introduced to further constrain joint associations, in which the attribute separation layer aims to separate the bone direction and length attributes, and the attribute control layer is designed to guide the pose generation by leveraging the above attributes. The ACD module utilizes the gloss embeddings as semantic conditions and finally generates sign poses from noise embeddings. Extensive experiments on PHOENIX14T and USTC-CSL datasets validate the effectiveness of our method. The code is available at: this https URL.
• 摘要：手语制作 (SLP) 旨在从文本语句生成语义一致的手势视频，其中从文本注释到手势姿势 (G2P) 的转换是关键步骤。现有的 G2P 方法通常将手势姿势视为离散的三维坐标并直接拟合它们，这忽略了关节之间的相对位置关系。为此，我们提供了一个新的视角，通过对肢体骨骼进行建模来约束关节关联和手势细节，以提高生成的姿势的准确性和自然度。在这项工作中，我们提出了一种专为 SLP 设计的开创性图像性解缠扩散框架，称为 Sign-IDD。Sign-IDD 结合了一种新颖的图像性解缠 (ID) 模块来弥合关节之间相对位置之间的差距。ID 模块将传统的 3D 关节表示解缠为 4D 骨骼表示，包括相邻关节之间的 3D 空间方向向量和 1D 空间距离向量。此外，引入了属性可控扩散 (ACD) 模块来进一步约束关节关联，其中属性分离层旨在分离骨骼方向和长度属性，属性控制层旨在利用上述属性来指导姿势生成。ACD 模块利用光泽嵌入作为语义条件，最终从噪声嵌入生成手势姿势。在 PHOENIX14T 和 USTC-CSL 数据集上进行的大量实验验证了我们方法的有效性。代码可在以下网址获取：此 https URL。

Title: TAUDiff: Improving statistical downscaling for extreme weather events using generative diffusion models

• Authors: Rahul Sundar, Nishant Parashar, Antoine Blanchard, Boyko Dodov
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13627
• Pdf URL: https://arxiv.org/pdf/2412.13627
• Copy Paste: [[2412.13627]] TAUDiff: Improving statistical downscaling for extreme weather events using generative diffusion models(https://arxiv.org/abs/2412.13627)
• Keywords: generative
• Abstract: Deterministic regression-based downscaling models for climate variables often suffer from spectral bias, which can be mitigated by generative models like diffusion models. To enable efficient and reliable simulation of extreme weather events, it is crucial to achieve rapid turnaround, dynamical consistency, and accurate spatio-temporal spectral recovery. We propose an efficient correction diffusion model, TAUDiff, that combines a deterministic spatio-temporal model for mean field downscaling with a smaller generative diffusion model for recovering the fine-scale stochastic features. We demonstrate the efficacy of this approach on downscaling atmospheric wind velocity fields obtained from coarse GCM simulations. Our approach can not only ensure quicker simulation of extreme events but also reduce overall carbon footprint due to low inference times.
• 摘要：基于确定性回归的气候变量降尺度模型经常受到光谱偏差的影响，而生成模型（如扩散模型）可以缓解这种偏差。为了高效可靠地模拟极端天气事件，实现快速周转、动态一致性和准确的时空光谱恢复至关重要。我们提出了一种有效的校正扩散模型 TAUDiff，它将确定性时空模型与较小的生成扩散模型相结合，用于平均场降尺度，以恢复细尺度随机特征。我们证明了这种方法在降尺度大气风速场方面的有效性，该大气风速场是从粗 GCM 模拟中获得的。我们的方法不仅可以确保更快地模拟极端事件，还可以由于推理时间短而减少总体碳足迹。

Title: Self-control: A Better Conditional Mechanism for Masked Autoregressive Model

• Authors: Qiaoying Qu, Shiyu Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13635
• Pdf URL: https://arxiv.org/pdf/2412.13635
• Copy Paste: [[2412.13635]] Self-control: A Better Conditional Mechanism for Masked Autoregressive Model(https://arxiv.org/abs/2412.13635)
• Keywords: generation, generative
• Abstract: Autoregressive conditional image generation algorithms are capable of generating photorealistic images that are consistent with given textual or image conditions, and have great potential for a wide range of applications. Nevertheless, the majority of popular autoregressive image generation methods rely heavily on vector quantization, and the inherent discrete characteristic of codebook presents a considerable challenge to achieving high-quality image generation. To address this limitation, this paper introduces a novel conditional introduction network for continuous masked autoregressive models. The proposed self-control network serves to mitigate the negative impact of vector quantization on the quality of the generated images, while simultaneously enhancing the conditional control during the generation process. In particular, the self-control network is constructed upon a continuous mask autoregressive generative model, which incorporates multimodal conditional information, including text and images, into a unified autoregressive sequence in a serial manner. Through a self-attention mechanism, the network is capable of generating images that are controllable based on specific conditions. The self-control network discards the conventional cross-attention-based conditional fusion mechanism and effectively unifies the conditional and generative information within the same space, thereby facilitating more seamless learning and fusion of multimodal features.
• 摘要：自回归条件图像生成算法能够生成符合给定文本或图像条件的逼真图像，具有广泛的应用潜力。然而，大多数流行的自回归图像生成方法严重依赖于矢量量化，而码本固有的离散特性对实现高质量的图像生成提出了相当大的挑战。针对这一限制，本文引入了一种新的连续掩模自回归模型条件引入网络。所提出的自控制网络可以减轻矢量量化对生成图像质量的负面影响，同时增强生成过程中的条件控制。具体而言，自控制网络构建在连续掩模自回归生成模型之上，将包括文本和图像在内的多模态条件信息以串行方式纳入统一的自回归序列，通过自注意力机制，网络能够生成基于特定条件可控的图像。自控网络摒弃了传统的基于交叉注意的条件融合机制，有效地将条件信息和生成信息统一在同一空间内，从而促进多模态特征的更无缝的学习和融合。

Title: RelationField: Relate Anything in Radiance Fields

• Authors: Sebastian Koch, Johanna Wald, Mirco Colosi, Narunas Vaskevicius, Pedro Hermosilla, Federico Tombari, Timo Ropinski
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13652
• Pdf URL: https://arxiv.org/pdf/2412.13652
• Copy Paste: [[2412.13652]] RelationField: Relate Anything in Radiance Fields(https://arxiv.org/abs/2412.13652)
• Keywords: generation
• Abstract: Neural radiance fields are an emerging 3D scene representation and recently even been extended to learn features for scene understanding by distilling open-vocabulary features from vision-language models. However, current method primarily focus on object-centric representations, supporting object segmentation or detection, while understanding semantic relationships between objects remains largely unexplored. To address this gap, we propose RelationField, the first method to extract inter-object relationships directly from neural radiance fields. RelationField represents relationships between objects as pairs of rays within a neural radiance field, effectively extending its formulation to include implicit relationship queries. To teach RelationField complex, open-vocabulary relationships, relationship knowledge is distilled from multi-modal LLMs. To evaluate RelationField, we solve open-vocabulary 3D scene graph generation tasks and relationship-guided instance segmentation, achieving state-of-the-art performance in both tasks. See the project website at this https URL.
• 摘要：神经辐射场是一种新兴的 3D 场景表示，最近甚至通过从视觉语言模型中提取开放词汇特征，扩展到学习场景理解的特征。然而，当前的方法主要侧重于以对象为中心的表示，支持对象分割或检测，而理解对象之间的语义关系仍未得到充分探索。为了解决这一差距，我们提出了 RelationField，这是第一种直接从神经辐射场中提取对象间关系的方法。RelationField 将对象之间的关系表示为神经辐射场内的射线对，有效地扩展了其公式以包含隐式关系查询。为了教授 RelationField 复杂的开放词汇关系，关系知识是从多模态 LLM 中提取出来的。为了评估 RelationField，我们解决了开放词汇 3D 场景图生成任务和关系引导的实例分割，在两个任务中都取得了最先进的性能。请访问此 https URL 上的项目网站。

Title: GLCF: A Global-Local Multimodal Coherence Analysis Framework for Talking Face Generation Detection

• Authors: Xiaocan Chen, Qilin Yin, Jiarui Liu, Wei Lu, Xiangyang Luo, Jiantao Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13656
• Pdf URL: https://arxiv.org/pdf/2412.13656
• Copy Paste: [[2412.13656]] GLCF: A Global-Local Multimodal Coherence Analysis Framework for Talking Face Generation Detection(https://arxiv.org/abs/2412.13656)
• Keywords: generation
• Abstract: Talking face generation (TFG) allows for producing lifelike talking videos of any character using only facial images and accompanying text. Abuse of this technology could pose significant risks to society, creating the urgent need for research into corresponding detection methods. However, research in this field has been hindered by the lack of public datasets. In this paper, we construct the first large-scale multi-scenario talking face dataset (MSTF), which contains 22 audio and video forgery techniques, filling the gap of datasets in this field. The dataset covers 11 generation scenarios and more than 20 semantic scenarios, closer to the practical application scenario of TFG. Besides, we also propose a TFG detection framework, which leverages the analysis of both global and local coherence in the multimodal content of TFG videos. Therefore, a region-focused smoothness detection module (RSFDM) and a discrepancy capture-time frame aggregation module (DCTAM) are introduced to evaluate the global temporal coherence of TFG videos, aggregating multi-grained spatial information. Additionally, a visual-audio fusion module (V-AFM) is designed to evaluate audiovisual coherence within a localized temporal perspective. Comprehensive experiments demonstrate the reasonableness and challenges of our datasets, while also indicating the superiority of our proposed method compared to the state-of-the-art deepfake detection approaches.
• 摘要：说话人脸生成 (TFG) 允许仅使用人脸图像和附带的文字来制作任何角色的逼真说话视频。滥用该技术可能对社会造成重大风险，因此迫切需要研究相应的检测方法。然而，由于缺乏公开数据集，该领域的研究受到了阻碍。在本文中，我们构建了第一个大规模多场景说话人脸数据集 (MSTF)，其中包含 22 种音频和视频伪造技术，填补了该领域数据集的空白。该数据集涵盖 11 个生成场景和 20 多个语义场景，更接近 TFG 的实际应用场景。此外，我们还提出了一个 TFG 检测框架，该框架利用对 TFG 视频多模态内容中的全局和局部相干性的分析。因此，引入了以区域为中心的平滑度检测模块 (RSFDM) 和差异捕获时间帧聚合模块 (DCTAM) 来评估 TFG 视频的全局时间相干性，聚合多粒度的空间信息。此外，我们还设计了一个视听融合模块 (V-AFM) 来评估局部时间视角内的视听连贯性。全面的实验证明了我们数据集的合理性和挑战性，同时也表明了我们提出的方法与最先进的深度伪造检测方法相比的优越性。

Title: MMO-IG: Multi-Class and Multi-Scale Object Image Generation for Remote Sensing

• Authors: Chuang Yang, Bingxuan Zhao, Qing Zhou, Qi Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13684
• Pdf URL: https://arxiv.org/pdf/2412.13684
• Copy Paste: [[2412.13684]] MMO-IG: Multi-Class and Multi-Scale Object Image Generation for Remote Sensing(https://arxiv.org/abs/2412.13684)
• Keywords: generation, generative
• Abstract: The rapid advancement of deep generative models (DGMs) has significantly advanced research in computer vision, providing a cost-effective alternative to acquiring vast quantities of expensive imagery. However, existing methods predominantly focus on synthesizing remote sensing (RS) images aligned with real images in a global layout view, which limits their applicability in RS image object detection (RSIOD) research. To address these challenges, we propose a multi-class and multi-scale object image generator based on DGMs, termed MMO-IG, designed to generate RS images with supervised object labels from global and local aspects simultaneously. Specifically, from the local view, MMO-IG encodes various RS instances using an iso-spacing instance map (ISIM). During the generation process, it decodes each instance region with iso-spacing value in ISIM-corresponding to both background and foreground instances-to produce RS images through the denoising process of diffusion models. Considering the complex interdependencies among MMOs, we construct a spatial-cross dependency knowledge graph (SCDKG). This ensures a realistic and reliable multidirectional distribution among MMOs for region embedding, thereby reducing the discrepancy between source and target domains. Besides, we propose a structured object distribution instruction (SODI) to guide the generation of synthesized RS image content from a global aspect with SCDKG-based ISIM together. Extensive experimental results demonstrate that our MMO-IG exhibits superior generation capabilities for RS images with dense MMO-supervised labels, and RS detectors pre-trained with MMO-IG show excellent performance on real-world datasets.
• 摘要：深度生成模型 (DGM) 的快速发展极大地推动了计算机视觉研究，为获取大量昂贵图像提供了一种经济高效的替代方案。然而，现有方法主要侧重于在全局布局视图中合成与真实图像对齐的遥感 (RS) 图像，这限制了它们在 RS 图像对象检测 (RSIOD) 研究中的应用。为了应对这些挑战，我们提出了一种基于 DGM 的多类多尺度对象图像生成器，称为 MMO-IG，旨在同时从全局和局部方面生成具有监督对象标签的 RS 图像。具体来说，从局部视图来看，MMO-IG 使用等间距实例图 (ISIM) 对各种 RS 实例进行编码。在生成过程中，它解码 ISIM 中具有等间距值的每个实例区域（对应于背景和前景实例），通过扩散模型的去噪过程生成 RS 图像。考虑到 MMO 之间的复杂相互依赖关系，我们构建了一个空间交叉依赖知识图 (SCDKG)。这确保了区域嵌入的 MMO 之间具有真实可靠的多向分布，从而减少了源域和目标域之间的差异。此外，我们提出了一种结构化对象分布指令 (SODI)，以从全局角度指导与基于 SCDKG 的 ISIM 一起生成合成的 RS 图像内容。大量实验结果表明，我们的 MMO-IG 对具有密集 MMO 监督标签的 RS 图像表现出卓越的生成能力，并且使用 MMO-IG 预训练的 RS 检测器在真实世界数据集上表现出色。

Title: Mitigating Adversarial Attacks in LLMs through Defensive Suffix Generation

• Authors: Minkyoung Kim, Yunha Kim, Hyeram Seo, Heejung Choi, Jiye Han, Gaeun Kee, Soyoung Ko, HyoJe Jung, Byeolhee Kim, Young-Hak Kim, Sanghyun Park, Tae Joon Jun
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.13705
• Pdf URL: https://arxiv.org/pdf/2412.13705
• Copy Paste: [[2412.13705]] Mitigating Adversarial Attacks in LLMs through Defensive Suffix Generation(https://arxiv.org/abs/2412.13705)
• Keywords: generation
• Abstract: Large language models (LLMs) have exhibited outstanding performance in natural language processing tasks. However, these models remain susceptible to adversarial attacks in which slight input perturbations can lead to harmful or misleading outputs. A gradient-based defensive suffix generation algorithm is designed to bolster the robustness of LLMs. By appending carefully optimized defensive suffixes to input prompts, the algorithm mitigates adversarial influences while preserving the models' utility. To enhance adversarial understanding, a novel total loss function ($L_{\text{total}}$) combining defensive loss ($L_{\text{def}}$) and adversarial loss ($L_{\text{adv}}$) generates defensive suffixes more effectively. Experimental evaluations conducted on open-source LLMs such as Gemma-7B, mistral-7B, Llama2-7B, and Llama2-13B show that the proposed method reduces attack success rates (ASR) by an average of 11\% compared to models without defensive suffixes. Additionally, the perplexity score of Gemma-7B decreased from 6.57 to 3.93 when applying the defensive suffix generated by openELM-270M. Furthermore, TruthfulQA evaluations demonstrate consistent improvements with Truthfulness scores increasing by up to 10\% across tested configurations. This approach significantly enhances the security of LLMs in critical applications without requiring extensive retraining.
• 摘要：大型语言模型 (LLM) 在自然语言处理任务中表现出色。然而，这些模型仍然容易受到对抗性攻击，其中轻微的输入扰动都可能导致有害或误导性的输出。基于梯度的防御性后缀生成算法旨在增强 LLM 的鲁棒性。通过将精心优化的防御性后缀附加到输入提示，该算法可以减轻对抗性影响，同时保留模型的实用性。为了增强对抗性理解，一种新颖的总损失函数 ($L_{\text{total}}$) 结合了防御性损失 ($L_{\text{def}}$) 和对抗性损失 ($L_{\text{adv}}$)，可以更有效地生成防御性后缀。在 Gemma-7B、mistral-7B、Llama2-7B 和 Llama2-13B 等开源 LLM 上进行的实验评估表明，与没有防御性后缀的模型相比，所提出的方法将攻击成功率 (ASR) 平均降低了 11%。此外，当应用 openELM-270M 生成的防御性后缀时，Gemma-7B 的困惑度得分从 6.57 降低到 3.93。此外，TruthfulQA 评估显示出持续的改进，在测试配置中，真实性得分提高了高达 10%。这种方法显著提高了关键应用中 LLM 的安全性，而无需进行大量的再培训。

Title: Text2Relight: Creative Portrait Relighting with Text Guidance

• Authors: Junuk Cha, Mengwei Ren, Krishna Kumar Singh, He Zhang, Yannick Hold-Geoffroy, Seunghyun Yoon, HyunJoon Jung, Jae Shin Yoon, Seungryul Baek
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13734
• Pdf URL: https://arxiv.org/pdf/2412.13734
• Copy Paste: [[2412.13734]] Text2Relight: Creative Portrait Relighting with Text Guidance(https://arxiv.org/abs/2412.13734)
• Keywords: generation, generative
• Abstract: We present a lighting-aware image editing pipeline that, given a portrait image and a text prompt, performs single image relighting. Our model modifies the lighting and color of both the foreground and background to align with the provided text description. The unbounded nature in creativeness of a text allows us to describe the lighting of a scene with any sensory features including temperature, emotion, smell, time, and so on. However, the modeling of such mapping between the unbounded text and lighting is extremely challenging due to the lack of dataset where there exists no scalable data that provides large pairs of text and relighting, and therefore, current text-driven image editing models does not generalize to lighting-specific use cases. We overcome this problem by introducing a novel data synthesis pipeline: First, diverse and creative text prompts that describe the scenes with various lighting are automatically generated under a crafted hierarchy using a large language model (*e.g.,* ChatGPT). A text-guided image generation model creates a lighting image that best matches the text. As a condition of the lighting images, we perform image-based relighting for both foreground and background using a single portrait image or a set of OLAT (One-Light-at-A-Time) images captured from lightstage system. Particularly for the background relighting, we represent the lighting image as a set of point lights and transfer them to other background images. A generative diffusion model learns the synthesized large-scale data with auxiliary task augmentation (*e.g.,* portrait delighting and light positioning) to correlate the latent text and lighting distribution for text-guided portrait relighting.
• 摘要：我们提出了一种照明感知图像编辑管道，给定一张肖像图像和一个文本提示，该管道执行单张图像重新照明。我们的模型修改了前景和背景的照明和颜色，以与提供的文本描述保持一致。文本的无限创造力使我们能够描述具有任何感官特征的场景的照明，包括温度、情感、气味、时间等。然而，由于缺乏数据集，对无界文本和照明之间的这种映射进行建模极具挑战性，因为不存在提供大量文本和重新照明对的可扩展数据，因此，当前的文本驱动图像编辑模型不能推广到特定于照明的用例。我们通过引入一种新颖的数据合成管道来克服这个问题：首先，使用大型语言模型（例如 ChatGPT）在精心设计的层次结构下自动生成描述具有各种照明场景的多样化和创造性的文本提示。文本引导的图像生成模型会创建与文本最匹配的照明图像。作为照明图像的条件，我们使用单张肖像图像或从 lightstage 系统捕获的一组 OLAT（一次一光）图像对前景和背景执行基于图像的重新照明。特别是对于背景重新照明，我们将照明图像表示为一组点光并将它们传输到其他背景图像。生成扩散模型使用辅助任务增强（例如，肖像愉悦和光定位）学习合成的大规模数据，以关联潜在文本和照明分布，以进行文本引导的肖像重新照明。

Title: Object Style Diffusion for Generalized Object Detection in Urban Scene

• Authors: Hao Li, Xiangyuan Yang, Mengzhu Wang, Long Lan, Ke Liang, Xinwang Liu, Kenli Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13815
• Pdf URL: https://arxiv.org/pdf/2412.13815
• Copy Paste: [[2412.13815]] Object Style Diffusion for Generalized Object Detection in Urban Scene(https://arxiv.org/abs/2412.13815)
• Keywords: generation
• Abstract: Object detection is a critical task in computer vision, with applications in various domains such as autonomous driving and urban scene monitoring. However, deep learning-based approaches often demand large volumes of annotated data, which are costly and difficult to acquire, particularly in complex and unpredictable real-world environments. This dependency significantly hampers the generalization capability of existing object detection techniques. To address this issue, we introduce a novel single-domain object detection generalization method, named GoDiff, which leverages a pre-trained model to enhance generalization in unseen domains. Central to our approach is the Pseudo Target Data Generation (PTDG) module, which employs a latent diffusion model to generate pseudo-target domain data that preserves source domain characteristics while introducing stylistic variations. By integrating this pseudo data with source domain data, we diversify the training dataset. Furthermore, we introduce a cross-style instance normalization technique to blend style features from different domains generated by the PTDG module, thereby increasing the detector's robustness. Experimental results demonstrate that our method not only enhances the generalization ability of existing detectors but also functions as a plug-and-play enhancement for other single-domain generalization methods, achieving state-of-the-art performance in autonomous driving scenarios.
• 摘要：物体检测是计算机视觉中的一项关键任务，可应用于自动驾驶和城市场景监控等各个领域。然而，基于深度学习的方法通常需要大量带注释的数据，这些数据成本高昂且难以获取，尤其是在复杂且不可预测的现实环境中。这种依赖性严重阻碍了现有物体检测技术的泛化能力。为了解决这个问题，我们引入了一种新颖的单域物体检测泛化方法，名为 GoDiff，它利用预先训练的模型来增强在看不见的域中的泛化能力。我们方法的核心是伪目标数据生成 (PTDG) 模块，它采用潜在扩散模型来生成伪目标域数据，在引入风格变化的同时保留源域特征。通过将这些伪数据与源域数据集成，我们使训练数据集多样化。此外，我们引入了一种跨风格实例规范化技术来混合 PTDG 模块生成的不同域的风格特征，从而提高检测器的鲁棒性。实验结果表明，我们的方法不仅增强了现有检测器的泛化能力，而且可以作为其他单域泛化方法的即插即用增强，在自动驾驶场景中实现最先进的性能。

Title: LLaVA-UHD v2: an MLLM Integrating High-Resolution Feature Pyramid via Hierarchical Window Transformer

• Authors: Yipeng Zhang, Yifan Liu, Zonghao Guo, Yidan Zhang, Xuesong Yang, Chi Chen, Jun Song, Bo Zheng, Yuan Yao, Zhiyuan Liu, Tat-Seng Chua, Maosong Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.13871
• Pdf URL: https://arxiv.org/pdf/2412.13871
• Copy Paste: [[2412.13871]] LLaVA-UHD v2: an MLLM Integrating High-Resolution Feature Pyramid via Hierarchical Window Transformer(https://arxiv.org/abs/2412.13871)
• Keywords: generation
• Abstract: In multimodal large language models (MLLMs), vision transformers (ViTs) are widely employed for visual encoding. However, their performance in solving universal MLLM tasks is not satisfactory. We attribute it to a lack of information from diverse visual levels, impeding alignment with the various semantic granularity required for language generation. To address this issue, we present LLaVA-UHD v2, an advanced MLLM centered around a Hierarchical window transformer that enables capturing diverse visual granularity by constructing and integrating a high-resolution feature pyramid. As a vision-language projector, Hiwin transformer comprises two primary modules: (i) an inverse feature pyramid, constructed by a ViT-derived feature up-sampling process utilizing high-frequency details from an image pyramid, and (ii) hierarchical window attention, focusing on a set of key sampling features within cross-scale windows to condense multi-level feature maps. Extensive experiments demonstrate that LLaVA-UHD v2 achieves superior performance over existing MLLMs on popular benchmarks. Notably, our design brings an average boost of 3.7% across 14 benchmarks compared with the baseline method, 9.3% on DocVQA for instance. We make all the data, model checkpoint, and code publicly available to facilitate future research.
• 摘要：在多模态大型语言模型 (MLLM) 中，视觉变换器 (ViT) 被广泛用于视觉编码。然而，它们在解决通用 MLLM 任务中的表现并不令人满意。我们将其归因于缺乏来自不同视觉层次的信息，阻碍了与语言生成所需的各种语义粒度的对齐。为了解决这个问题，我们提出了 LLaVA-UHD v2，这是一种以分层窗口变换器为中心的高级 MLLM，它通过构建和集成高分辨率特征金字塔来捕获不同的视觉粒度。作为视觉语言投影仪，Hiwin Transformer 包含两个主要模块：(i) 逆特征金字塔，由 ViT 衍生的特征上采样过程构建，利用图像金字塔中的高频细节，以及 (ii) 分层窗口注意，专注于跨尺度窗口内的一组关键采样特征以压缩多级特征图。大量实验表明，LLaVA-UHD v2 在流行的基准测试中比现有的 MLLM 取得了更好的性能。值得注意的是，与基线方法相比，我们的设计在 14 个基准测试中平均提升了 3.7%，例如在 DocVQA 上提升了 9.3%。我们公开了所有数据、模型检查点和代码，以方便未来的研究。

Title: RAG for Effective Supply Chain Security Questionnaire Automation

• Authors: Zaynab Batool Reza, Abdul Rafay Syed, Omer Iqbal, Ethel Mensah, Qian Liu, Maxx Richard Rahman, Wolfgang Maass
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.13988
• Pdf URL: https://arxiv.org/pdf/2412.13988
• Copy Paste: [[2412.13988]] RAG for Effective Supply Chain Security Questionnaire Automation(https://arxiv.org/abs/2412.13988)
• Keywords: generation
• Abstract: In an era where digital security is crucial, efficient processing of security-related inquiries through supply chain security questionnaires is imperative. This paper introduces a novel approach using Natural Language Processing (NLP) and Retrieval-Augmented Generation (RAG) to automate these responses. We developed QuestSecure, a system that interprets diverse document formats and generates precise responses by integrating large language models (LLMs) with an advanced retrieval system. Our experiments show that QuestSecure significantly improves response accuracy and operational efficiency. By employing advanced NLP techniques and tailored retrieval mechanisms, the system consistently produces contextually relevant and semantically rich responses, reducing cognitive load on security teams and minimizing potential errors. This research offers promising avenues for automating complex security management tasks, enhancing organizational security processes.
• 摘要：在数字安全至关重要的时代，通过供应链安全问卷高效处理与安全相关的查询势在必行。本文介绍了一种使用自然语言处理 (NLP) 和检索增强生成 (RAG) 来自动化这些响应的新方法。我们开发了 QuestSecure，该系统通过将大型语言模型 (LLM) 与高级检索系统相结合，可以解释各种文档格式并生成精确的响应。我们的实验表明，QuestSecure 显著提高了响应准确性和运营效率。通过采用先进的 NLP 技术和量身定制的检索机制，该系统可以持续生成与上下文相关且语义丰富的响应，从而减少安全团队的认知负担并最大限度地减少潜在错误。这项研究为自动化复杂的安全管理任务、增强组织安全流程提供了有希望的途径。

Title: Real-Time Position-Aware View Synthesis from Single-View Input

• Authors: Manu Gond, Emin Zerman, Sebastian Knorr, Mårten Sjöström
• Subjects: cs.CV, cs.GR, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.14005
• Pdf URL: https://arxiv.org/pdf/2412.14005
• Copy Paste: [[2412.14005]] Real-Time Position-Aware View Synthesis from Single-View Input(https://arxiv.org/abs/2412.14005)
• Keywords: generation
• Abstract: Recent advancements in view synthesis have significantly enhanced immersive experiences across various computer graphics and multimedia applications, including telepresence, and entertainment. By enabling the generation of new perspectives from a single input view, view synthesis allows users to better perceive and interact with their environment. However, many state-of-the-art methods, while achieving high visual quality, face limitations in real-time performance, which makes them less suitable for live applications where low latency is critical. In this paper, we present a lightweight, position-aware network designed for real-time view synthesis from a single input image and a target camera pose. The proposed framework consists of a Position Aware Embedding, modeled with a multi-layer perceptron, which efficiently maps positional information from the target pose to generate high dimensional feature maps. These feature maps, along with the input image, are fed into a Rendering Network that merges features from dual encoder branches to resolve both high level semantics and low level details, producing a realistic new view of the scene. Experimental results demonstrate that our method achieves superior efficiency and visual quality compared to existing approaches, particularly in handling complex translational movements without explicit geometric operations like warping. This work marks a step toward enabling real-time view synthesis from a single image for live and interactive applications.
• 摘要：视图合成的最新进展显著增强了各种计算机图形和多媒体应用（包括远程呈现和娱乐）的沉浸式体验。通过从单个输入视图生成新视角，视图合成使用户能够更好地感知和与周围环境交互。然而，许多最先进的方法虽然实现了高视觉质量，但在实时性能方面却面临限制，这使得它们不太适合低延迟至关重要的实时应用。在本文中，我们提出了一个轻量级的位置感知网络，旨在从单个输入图像和目标相机姿势进行实时视图合成。所提出的框架由一个位置感知嵌入组成，用多层感知器建模，它有效地映射来自目标姿势的位置信息以生成高维特征图。这些特征图与输入图像一起被输入到渲染网络中，该网络合并来自双编码器分支的特征以解析高级语义和低级细节，从而产生逼真的场景新视图。实验结果表明，与现有方法相比，我们的方法实现了卓越的效率和视觉质量，特别是在处理复杂的平移运动时，无需进行扭曲等显式几何操作。这项工作标志着我们朝着从单个图像实现实时视图合成（用于实时和交互式应用）迈出了一步。

Title: Prompting Depth Anything for 4K Resolution Accurate Metric Depth Estimation

• Authors: Haotong Lin, Sida Peng, Jingxiao Chen, Songyou Peng, Jiaming Sun, Minghuan Liu, Hujun Bao, Jiashi Feng, Xiaowei Zhou, Bingyi Kang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.14015
• Pdf URL: https://arxiv.org/pdf/2412.14015
• Copy Paste: [[2412.14015]] Prompting Depth Anything for 4K Resolution Accurate Metric Depth Estimation(https://arxiv.org/abs/2412.14015)
• Keywords: generation
• Abstract: Prompts play a critical role in unleashing the power of language and vision foundation models for specific tasks. For the first time, we introduce prompting into depth foundation models, creating a new paradigm for metric depth estimation termed Prompt Depth Anything. Specifically, we use a low-cost LiDAR as the prompt to guide the Depth Anything model for accurate metric depth output, achieving up to 4K resolution. Our approach centers on a concise prompt fusion design that integrates the LiDAR at multiple scales within the depth decoder. To address training challenges posed by limited datasets containing both LiDAR depth and precise GT depth, we propose a scalable data pipeline that includes synthetic data LiDAR simulation and real data pseudo GT depth generation. Our approach sets new state-of-the-arts on the ARKitScenes and ScanNet++ datasets and benefits downstream applications, including 3D reconstruction and generalized robotic grasping.
• 摘要：提示在释放语言和视觉基础模型在特定任务中的威力方面发挥着关键作用。我们首次将提示引入深度基础模型，为度量深度估计创建了一种新范式，称为“Prompt Depth Anything”。具体来说，我们使用低成本的 LiDAR 作为提示，引导 Depth Anything 模型实现准确的度量深度输出，分辨率高达 4K。我们的方法以简洁的提示融合设计为中心，该设计在深度解码器内集成了多个尺度的 LiDAR。为了解决由包含 LiDAR 深度和精确 GT 深度的有限数据集所带来的训练挑战，我们提出了一种可扩展的数据管道，其中包括合成数据 LiDAR 模拟和真实数据伪 GT 深度生成。我们的方法在 ARKitScenes 和 ScanNet++ 数据集上树立了新的领先地位，并使下游应用受益，包括 3D 重建和广义机器人抓取。

Title: SurgSora: Decoupled RGBD-Flow Diffusion Model for Controllable Surgical Video Generation

• Authors: Tong Chen, Shuya Yang, Junyi Wang, Long Bai, Hongliang Ren, Luping Zhou
• Subjects: cs.CV, cs.AI, cs.MM, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.14018
• Pdf URL: https://arxiv.org/pdf/2412.14018
• Copy Paste: [[2412.14018]] SurgSora: Decoupled RGBD-Flow Diffusion Model for Controllable Surgical Video Generation(https://arxiv.org/abs/2412.14018)
• Keywords: generation
• Abstract: Medical video generation has transformative potential for enhancing surgical understanding and pathology insights through precise and controllable visual representations. However, current models face limitations in controllability and authenticity. To bridge this gap, we propose SurgSora, a motion-controllable surgical video generation framework that uses a single input frame and user-controllable motion cues. SurgSora consists of three key modules: the Dual Semantic Injector (DSI), which extracts object-relevant RGB and depth features from the input frame and integrates them with segmentation cues to capture detailed spatial features of complex anatomical structures; the Decoupled Flow Mapper (DFM), which fuses optical flow with semantic-RGB-D features at multiple scales to enhance temporal understanding and object spatial dynamics; and the Trajectory Controller (TC), which allows users to specify motion directions and estimates sparse optical flow, guiding the video generation process. The fused features are used as conditions for a frozen Stable Diffusion model to produce realistic, temporally coherent surgical videos. Extensive evaluations demonstrate that SurgSora outperforms state-of-the-art methods in controllability and authenticity, showing its potential to advance surgical video generation for medical education, training, and research.
• 摘要：医学视频生成具有变革性的潜力，可通过精确且可控的视觉表现来增强对外科手术的理解和病理学的洞察。然而，目前的模型在可控性和真实性方面存在局限性。为了弥补这一差距，我们提出了 SurgSora，这是一个使用单个输入帧和用户可控运动提示的运动可控外科视频生成框架。SurgSora 由三个关键模块组成：双语义注入器 (DSI)，它从输入帧中提取与对象相关的 RGB 和深度特征，并将它们与分割提示相结合，以捕捉复杂解剖结构的详细空间特征；解耦流映射器 (DFM)，它将光流与多个尺度的语义 RGB-D 特征融合，以增强时间理解和对象空间动态；轨迹控制器 (TC)，它允许用户指定运动方向并估计稀疏光流，从而指导视频生成过程。融合的特征用作冻结稳定扩散模型的条件，以生成逼真、时间连贯的外科手术视频。大量评估表明，SurgSora 在可控性和真实性方面优于最先进的方法，展现出其在推动医学教育、培训和研究手术视频生成方面的潜力。

Title: A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future

• Authors: Shilin Sun, Wenbin An, Feng Tian, Fang Nan, Qidong Liu, Jun Liu, Nazaraf Shah, Ping Chen
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.14056
• Pdf URL: https://arxiv.org/pdf/2412.14056
• Copy Paste: [[2412.14056]] A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future(https://arxiv.org/abs/2412.14056)
• Keywords: generative
• Abstract: Artificial intelligence (AI) has rapidly developed through advancements in computational power and the growth of massive datasets. However, this progress has also heightened challenges in interpreting the "black-box" nature of AI models. To address these concerns, eXplainable AI (XAI) has emerged with a focus on transparency and interpretability to enhance human understanding and trust in AI decision-making processes. In the context of multimodal data fusion and complex reasoning scenarios, the proposal of Multimodal eXplainable AI (MXAI) integrates multiple modalities for prediction and explanation tasks. Meanwhile, the advent of Large Language Models (LLMs) has led to remarkable breakthroughs in natural language processing, yet their complexity has further exacerbated the issue of MXAI. To gain key insights into the development of MXAI methods and provide crucial guidance for building more transparent, fair, and trustworthy AI systems, we review the MXAI methods from a historical perspective and categorize them across four eras: traditional machine learning, deep learning, discriminative foundation models, and generative LLMs. We also review evaluation metrics and datasets used in MXAI research, concluding with a discussion of future challenges and directions. A project related to this review has been created at this https URL.
• 摘要：人工智能 (AI) 通过计算能力的进步和海量数据集的增长而迅速发展。然而，这一进步也增加了解释 AI 模型“黑箱”性质的挑战。为了解决这些问题，可解释人工智能 (XAI) 应运而生，其重点是透明度和可解释性，以增强人类对 AI 决策过程的理解和信任。在多模态数据融合和复杂推理场景的背景下，多模态可解释人工智能 (MXAI) 的提出将多种模态集成在一起，用于预测和解释任务。同时，大型语言模型 (LLM) 的出现带来了自然语言处理的重大突破，但它们的复杂性进一步加剧了 MXAI 的问题。为了深入了解 MXAI 方法的发展并为构建更透明、公平和值得信赖的 AI 系统提供重要指导，我们从历史角度回顾了 MXAI 方法，并将它们分为四个时代：传统机器学习、深度学习、判别基础模型和生成 LLM。我们还回顾了 MXAI 研究中使用的评估指标和数据集，最后讨论了未来的挑战和方向。此 https URL 上已创建了一个与此回顾相关的项目。

Title: Future Research Avenues for Artificial Intelligence in Digital Gaming: An Exploratory Report

• Authors: Markus Dablander
• Subjects: cs.LG, cs.AI, cs.HC
• Abstract URL: https://arxiv.org/abs/2412.14085
• Pdf URL: https://arxiv.org/pdf/2412.14085
• Copy Paste: [[2412.14085]] Future Research Avenues for Artificial Intelligence in Digital Gaming: An Exploratory Report(https://arxiv.org/abs/2412.14085)
• Keywords: generation, generative
• Abstract: Video games are a natural and synergistic application domain for artificial intelligence (AI) systems, offering both the potential to enhance player experience and immersion, as well as providing valuable benchmarks and virtual environments to advance AI technologies in general. This report presents a high-level overview of five promising research pathways for applying state-of-the-art AI methods, particularly deep learning, to digital gaming within the context of the current research landscape. The objective of this work is to outline a curated, non-exhaustive list of encouraging research directions at the intersection of AI and video games that may serve to inspire more rigorous and comprehensive research efforts in the future. We discuss (i) investigating large language models as core engines for game agent modelling, (ii) using neural cellular automata for procedural game content generation, (iii) accelerating computationally expensive in-game simulations via deep surrogate modelling, (iv) leveraging self-supervised learning to obtain useful video game state embeddings, and (v) training generative models of interactive worlds using unlabelled video data. We also briefly address current technical challenges associated with the integration of advanced deep learning systems into video game development, and indicate key areas where further progress is likely to be beneficial.
• 摘要：电子游戏是人工智能 (AI) 系统的一个自然而协同的应用领域，既有潜力增强玩家体验和沉浸感，又能提供宝贵的基准和虚拟环境来推动整个 AI 技术的发展。本报告概述了在当前研究背景下将最先进的 AI 方法（尤其是深度学习）应用于数字游戏的五种有前途的研究途径。这项工作的目的是概述一份精选的、非详尽的清单，列出 AI 与电子游戏交叉领域的令人鼓舞的研究方向，这些方向可能有助于激发未来更严格、更全面的研究工作。我们讨论了 (i) 研究大型语言模型作为游戏代理建模的核心引擎，(ii) 使用神经细胞自动机进行程序性游戏内容生成，(iii) 通过深度代理建模加速计算成本高昂的游戏内模拟，(iv) 利用自监督学习获得有用的视频游戏状态嵌入，以及 (v) 使用未标记的视频数据训练交互式世界的生成模型。我们还简要介绍了将先进的深度学习系统集成到视频游戏开发中面临的当前技术挑战，并指出了可能取得进一步进展的关键领域。

Title: Event-based Photometric Bundle Adjustment

• Authors: Shuang Guo, Guillermo Gallego
• Subjects: cs.CV, cs.RO, eess.SP, math.OC
• Abstract URL: https://arxiv.org/abs/2412.14111
• Pdf URL: https://arxiv.org/pdf/2412.14111
• Copy Paste: [[2412.14111]] Event-based Photometric Bundle Adjustment(https://arxiv.org/abs/2412.14111)
• Keywords: generation
• Abstract: We tackle the problem of bundle adjustment (i.e., simultaneous refinement of camera poses and scene map) for a purely rotating event camera. Starting from first principles, we formulate the problem as a classical non-linear least squares optimization. The photometric error is defined using the event generation model directly in the camera rotations and the semi-dense scene brightness that triggers the events. We leverage the sparsity of event data to design a tractable Levenberg-Marquardt solver that handles the very large number of variables involved. To the best of our knowledge, our method, which we call Event-based Photometric Bundle Adjustment (EPBA), is the first event-only photometric bundle adjustment method that works on the brightness map directly and exploits the space-time characteristics of event data, without having to convert events into image-like representations. Comprehensive experiments on both synthetic and real-world datasets demonstrate EPBA's effectiveness in decreasing the photometric error (by up to 90%), yielding results of unparalleled quality. The refined maps reveal details that were hidden using prior state-of-the-art rotation-only estimation methods. The experiments on modern high-resolution event cameras show the applicability of EPBA to panoramic imaging in various scenarios (without map initialization, at multiple resolutions, and in combination with other methods, such as IMU dead reckoning or previous event-based rotation estimation methods). We make the source code publicly available. this https URL
• 摘要：我们解决了纯旋转事件相机的捆绑调整问题（即同时优化相机姿势和场景地图）。从第一原理开始，我们将问题表述为经典的非线性最小二乘优化。光度误差是使用事件生成模型直接在相机旋转和触发事件的半密集场景亮度中定义的。我们利用事件数据的稀疏性来设计一个可处理的 Levenberg-Marquardt 求解器，以处理所涉及的大量变量。据我们所知，我们的方法称为基于事件的光度捆绑调整 (EPBA)，是第一个仅针对事件的光度捆绑调整方法，它直接作用于亮度图并利用事件数据的时空特性，而无需将事件转换为类似图像的表示。在合成和真实世界数据集上进行的全面实验证明了 EPBA 在降低光度误差（高达 90%）方面的有效性，并产生了无与伦比的质量结果。经过改进的地图揭示了使用先前最先进的仅旋转估计方法所隐藏的细节。在现代高分辨率事件相机上进行的实验表明，EPBA 适用于各种场景中的全景成像（无需地图初始化、多种分辨率以及与其他方法结合使用，例如 IMU 航位推算或先前基于事件的旋转估计方法）。我们公开了源代码。此 https URL

Title: MCMat: Multiview-Consistent and Physically Accurate PBR Material Generation

• Authors: Shenhao Zhu, Lingteng Qiu, Xiaodong Gu, Zhengyi Zhao, Chao Xu, Yuxiao He, Zhe Li, Xiaoguang Han, Yao Yao, Xun Cao, Siyu Zhu, Weihao Yuan, Zilong Dong, Hao Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.14148
• Pdf URL: https://arxiv.org/pdf/2412.14148
• Copy Paste: [[2412.14148]] MCMat: Multiview-Consistent and Physically Accurate PBR Material Generation(https://arxiv.org/abs/2412.14148)
• Keywords: generation
• Abstract: Existing 2D methods utilize UNet-based diffusion models to generate multi-view physically-based rendering (PBR) maps but struggle with multi-view inconsistency, while some 3D methods directly generate UV maps, encountering generalization issues due to the limited 3D data. To address these problems, we propose a two-stage approach, including multi-view generation and UV materials refinement. In the generation stage, we adopt a Diffusion Transformer (DiT) model to generate PBR materials, where both the specially designed multi-branch DiT and reference-based DiT blocks adopt a global attention mechanism to promote feature interaction and fusion between different views, thereby improving multi-view consistency. In addition, we adopt a PBR-based diffusion loss to ensure that the generated materials align with realistic physical principles. In the refinement stage, we propose a material-refined DiT that performs inpainting in empty areas and enhances details in UV space. Except for the normal condition, this refinement also takes the material map from the generation stage as an additional condition to reduce the learning difficulty and improve generalization. Extensive experiments show that our method achieves state-of-the-art performance in texturing 3D objects with PBR materials and provides significant advantages for graphics relighting applications. Project Page: this https URL
• 摘要：现有的二维方法利用基于 UNet 的扩散模型生成多视图基于物理的渲染 (PBR) 图，但存在多视图不一致的问题；而一些三维方法直接生成 UV 图，由于三维数据有限，存在泛化问题。为了解决这些问题，我们提出了一种两阶段方法，包括多视图生成和 UV 材质细化。在生成阶段，我们采用扩散变换器 (DiT) 模型来生成 PBR 材质，其中专门设计的多分支 DiT 和基于参考的 DiT 块都采用全局注意机制来促进不同视图之间的特征交互和融合，从而提高多视图一致性。此外，我们采用基于 PBR 的扩散损失来确保生成的材质符合现实的物理原理。在细化阶段，我们提出了一种材质细化的 DiT，它在空白区域进行修复并增强 UV 空间中的细节。除了正常条件外，此细化还将生成阶段的材质图作为附加条件，以降低学习难度并提高泛化能力。大量实验表明，我们的方法在使用 PBR 材质对 3D 对象进行纹理处理时实现了最先进的性能，并为图形重新照明应用提供了显着优势。项目页面：此 https URL

Title: AKiRa: Augmentation Kit on Rays for optical video generation

• Authors: Xi Wang, Robin Courant, Marc Christie, Vicky Kalogeiton
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.14158
• Pdf URL: https://arxiv.org/pdf/2412.14158
• Copy Paste: [[2412.14158]] AKiRa: Augmentation Kit on Rays for optical video generation(https://arxiv.org/abs/2412.14158)
• Keywords: generation
• Abstract: Recent advances in text-conditioned video diffusion have greatly improved video quality. However, these methods offer limited or sometimes no control to users on camera aspects, including dynamic camera motion, zoom, distorted lens and focus shifts. These motion and optical aspects are crucial for adding controllability and cinematic elements to generation frameworks, ultimately resulting in visual content that draws focus, enhances mood, and guides emotions according to filmmakers' controls. In this paper, we aim to close the gap between controllable video generation and camera optics. To achieve this, we propose AKiRa (Augmentation Kit on Rays), a novel augmentation framework that builds and trains a camera adapter with a complex camera model over an existing video generation backbone. It enables fine-tuned control over camera motion as well as complex optical parameters (focal length, distortion, aperture) to achieve cinematic effects such as zoom, fisheye effect, and bokeh. Extensive experiments demonstrate AKiRa's effectiveness in combining and composing camera optics while outperforming all state-of-the-art methods. This work sets a new landmark in controlled and optically enhanced video generation, paving the way for future optical video generation methods.
• 摘要：文本条件视频传播的最新进展极大地提高了视频质量。然而，这些方法对用户在相机方面的控制有限，有时甚至没有控制，包括动态相机运动、变焦、镜头扭曲和焦点偏移。这些运动和光学方面对于向生成框架添加可控性和电影元素至关重要，最终产生能够根据电影制作人的控制吸引焦点、增强情绪和引导情绪的视觉内容。在本文中，我们旨在缩小可控视频生成和相机光学之间的差距。为了实现这一目标，我们提出了 AKiRa（光线增强套件），这是一种新颖的增强框架，它在现有的视频生成主干上构建和训练具有复杂相机模型的相机适配器。它可以对相机运动以及复杂的光学参数（焦距、失真、光圈）进行微调控制，以实现电影效果，例如变焦、鱼眼效果和散景。大量实验证明了 AKiRa 在组合和合成相机光学方面的有效性，同时优于所有最先进的方法。这项工作为受控和光学增强视频生成树立了新的里程碑，为未来的光学视频生成方法铺平了道路。

Title: MetaMorph: Multimodal Understanding and Generation via Instruction Tuning

• Authors: Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, Zhuang Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.14164
• Pdf URL: https://arxiv.org/pdf/2412.14164
• Copy Paste: [[2412.14164]] MetaMorph: Multimodal Understanding and Generation via Instruction Tuning(https://arxiv.org/abs/2412.14164)
• Keywords: generation
• Abstract: In this work, we propose Visual-Predictive Instruction Tuning (VPiT) - a simple and effective extension to visual instruction tuning that enables a pretrained LLM to quickly morph into an unified autoregressive model capable of generating both text and visual tokens. VPiT teaches an LLM to predict discrete text tokens and continuous visual tokens from any input sequence of image and text data curated in an instruction-following format. Our empirical investigation reveals several intriguing properties of VPiT: (1) visual generation ability emerges as a natural byproduct of improved visual understanding, and can be unlocked efficiently with a small amount of generation data; (2) while we find understanding and generation to be mutually beneficial, understanding data contributes to both capabilities more effectively than generation data. Building upon these findings, we train our MetaMorph model and achieve competitive performance on both visual understanding and generation. In visual generation, MetaMorph can leverage the world knowledge and reasoning abilities gained from LLM pretraining, and overcome common failure modes exhibited by other generation models. Our results suggest that LLMs may have strong "prior" vision capabilities that can be efficiently adapted to both visual understanding and generation with a relatively simple instruction tuning process.
• 摘要：在这项工作中，我们提出了视觉预测指令调整 (VPiT) - 一种简单有效的视觉指令调整扩展，使预训练的 LLM 能够快速转变为能够生成文本和视觉标记的统一自回归模型。VPiT 教导 LLM 从以指令跟随格式整理的任何图像和文本数据输入序列中预测离散文本标记和连续视觉标记。我们的实证调查揭示了 VPiT 的几个有趣特性：(1) 视觉生成能力是改进视觉理解的自然副产品，并且可以通过少量生成数据有效解锁；(2) 虽然我们发现理解和生成是互惠互利的，但理解数据比生成数据更有效地促进这两种能力。基于这些发现，我们训练了 MetaMorph 模型并在视觉理解和生成方面取得了有竞争力的表现。在视觉生成中，MetaMorph 可以利用从 LLM 预训练中获得的世界知识和推理能力，并克服其他生成模型表现出的常见故障模式。我们的结果表明，LLM 可能具有强大的“先前”视觉能力，可以通过相对简单的指令调整过程有效地适应视觉理解和生成。

Title: MegaSynth: Scaling Up 3D Scene Reconstruction with Synthesized Data

• Authors: Hanwen Jiang, Zexiang Xu, Desai Xie, Ziwen Chen, Haian Jin, Fujun Luan, Zhixin Shu, Kai Zhang, Sai Bi, Xin Sun, Jiuxiang Gu, Qixing Huang, Georgios Pavlakos, Hao Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.14166
• Pdf URL: https://arxiv.org/pdf/2412.14166
• Copy Paste: [[2412.14166]] MegaSynth: Scaling Up 3D Scene Reconstruction with Synthesized Data(https://arxiv.org/abs/2412.14166)
• Keywords: generation
• Abstract: We propose scaling up 3D scene reconstruction by training with synthesized data. At the core of our work is MegaSynth, a procedurally generated 3D dataset comprising 700K scenes - over 50 times larger than the prior real dataset DL3DV - dramatically scaling the training data. To enable scalable data generation, our key idea is eliminating semantic information, removing the need to model complex semantic priors such as object affordances and scene composition. Instead, we model scenes with basic spatial structures and geometry primitives, offering scalability. Besides, we control data complexity to facilitate training while loosely aligning it with real-world data distribution to benefit real-world generalization. We explore training LRMs with both MegaSynth and available real data. Experiment results show that joint training or pre-training with MegaSynth improves reconstruction quality by 1.2 to 1.8 dB PSNR across diverse image domains. Moreover, models trained solely on MegaSynth perform comparably to those trained on real data, underscoring the low-level nature of 3D reconstruction. Additionally, we provide an in-depth analysis of MegaSynth's properties for enhancing model capability, training stability, and generalization.
• 摘要：我们建议通过使用合成数据进行训练来扩大 3D 场景重建的规模。我们工作的核心是 MegaSynth，这是一个程序生成的 3D 数据集，包含 700K 个场景 - 比之前的真实数据集 DL3DV 大 50 多倍 - 大大扩展了训练数据。为了实现可扩展的数据生成，我们的关键思想是消除语义信息，消除对复杂语义先验（例如对象可供性和场景构成）进行建模的需要。相反，我们使用基本的空间结构和几何图元对场景进行建模，从而提供可扩展性。此外，我们控制数据复杂性以促进训练，同时将其与真实数据分布松散地对齐，以有利于真实世界的泛化。我们探索使用 MegaSynth 和可用的真实数据训练 LRM。实验结果表明，使用 MegaSynth 进行联合训练或预训练可将不同图像域的重建质量提高 1.2 到 1.8 dB PSNR。此外，仅在 MegaSynth 上训练的模型与在真实数据上训练的模型表现相当，这突显了 3D 重建的低级性质。此外，我们还对 MegaSynth 的特性进行了深入分析，以增强模型能力、训练稳定性和泛化能力。

Title: VideoDPO: Omni-Preference Alignment for Video Diffusion Generation

• Authors: Runtao Liu, Haoyu Wu, Zheng Ziqiang, Chen Wei, Yingqing He, Renjie Pi, Qifeng Chen
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.14167
• Pdf URL: https://arxiv.org/pdf/2412.14167
• Copy Paste: [[2412.14167]] VideoDPO: Omni-Preference Alignment for Video Diffusion Generation(https://arxiv.org/abs/2412.14167)
• Keywords: generation, generative
• Abstract: Recent progress in generative diffusion models has greatly advanced text-to-video generation. While text-to-video models trained on large-scale, diverse datasets can produce varied outputs, these generations often deviate from user preferences, highlighting the need for preference alignment on pre-trained models. Although Direct Preference Optimization (DPO) has demonstrated significant improvements in language and image generation, we pioneer its adaptation to video diffusion models and propose a VideoDPO pipeline by making several key adjustments. Unlike previous image alignment methods that focus solely on either (i) visual quality or (ii) semantic alignment between text and videos, we comprehensively consider both dimensions and construct a preference score accordingly, which we term the OmniScore. We design a pipeline to automatically collect preference pair data based on the proposed OmniScore and discover that re-weighting these pairs based on the score significantly impacts overall preference alignment. Our experiments demonstrate substantial improvements in both visual quality and semantic alignment, ensuring that no preference aspect is neglected. Code and data will be shared at this https URL.
• 摘要：生成式传播模型的最新进展极大地推动了文本转视频的生成。虽然在大规模、多样化数据集上训练的文本转视频模型可以产生不同的输出，但这些生成结果往往会偏离用户偏好，这凸显了对预训练模型进行偏好对齐的必要性。尽管直接偏好优化 (DPO) 在语言和图像生成方面表现出了显著的改进，但我们率先将其应用于视频传播模型，并通过进行几项关键调整提出了 VideoDPO 管道。与以前的图像对齐方法不同，这些方法只关注 (i) 视觉质量或 (ii) 文本和视频之间的语义对齐，我们综合考虑了这两个维度并相应地构建了一个偏好分数，我们称之为 OmniScore。我们设计了一个管道，根据提议的 OmniScore 自动收集偏好对数据，并发现根据分数重新加权这些对会显著影响整体偏好对齐。我们的实验表明，视觉质量和语义对齐都有了显著的改善，确保不会忽略任何偏好方面。代码和数据将在此 https URL 上共享。

Title: FashionComposer: Compositional Fashion Image Generation

• Authors: Sihui Ji, Yiyang Wang, Xi Chen, Xiaogang Xu, Hao Luo, Hengshuang Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.14168
• Pdf URL: https://arxiv.org/pdf/2412.14168
• Copy Paste: [[2412.14168]] FashionComposer: Compositional Fashion Image Generation(https://arxiv.org/abs/2412.14168)
• Keywords: generation
• Abstract: We present FashionComposer for compositional fashion image generation. Unlike previous methods, FashionComposer is highly flexible. It takes multi-modal input (i.e., text prompt, parametric human model, garment image, and face image) and supports personalizing the appearance, pose, and figure of the human and assigning multiple garments in one pass. To achieve this, we first develop a universal framework capable of handling diverse input modalities. We construct scaled training data to enhance the model's robust compositional capabilities. To accommodate multiple reference images (garments and faces) seamlessly, we organize these references in a single image as an "asset library" and employ a reference UNet to extract appearance features. To inject the appearance features into the correct pixels in the generated result, we propose subject-binding attention. It binds the appearance features from different "assets" with the corresponding text features. In this way, the model could understand each asset according to their semantics, supporting arbitrary numbers and types of reference images. As a comprehensive solution, FashionComposer also supports many other applications like human album generation, diverse virtual try-on tasks, etc.
• 摘要：我们提出了 FashionComposer 用于合成时尚图像生成。与以前的方法不同，FashionComposer 具有高度灵活性。它接受多模态输入（即文本提示、参数人体模型、服装图像和面部图像），并支持个性化人体的外观、姿势和身材，并一次性分配多件服装。为了实现这一点，我们首先开发了一个能够处理多种输入模态的通用框架。我们构建了缩放的训练数据来增强模型的强大合成能力。为了无缝地容纳多个参考图像（服装和面部），我们将这些参考组织在单个图像中作为“资产库”，并使用参考 UNet 来提取外观特征。为了将外观特征注入生成结果中的正确像素，我们提出了主题绑定注意。它将来自不同“资产”的外观特征与相应的文本特征绑定在一起。通过这种方式，模型可以根据每个资产的语义来理解它们，支持任意数量和类型的参考图像。作为一个全面的解决方案，FashionComposer 还支持许多其他应用程序，如人体相册生成、各种虚拟试穿任务等。

Title: Autoregressive Video Generation without Vector Quantization

• Authors: Haoge Deng, Ting Pan, Haiwen Diao, Zhengxiong Luo, Yufeng Cui, Huchuan Lu, Shiguang Shan, Yonggang Qi, Xinlong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.14169
• Pdf URL: https://arxiv.org/pdf/2412.14169
• Copy Paste: [[2412.14169]] Autoregressive Video Generation without Vector Quantization(https://arxiv.org/abs/2412.14169)
• Keywords: generation
• Abstract: This paper presents a novel approach that enables autoregressive video generation with high efficiency. We propose to reformulate the video generation problem as a non-quantized autoregressive modeling of temporal frame-by-frame prediction and spatial set-by-set prediction. Unlike raster-scan prediction in prior autoregressive models or joint distribution modeling of fixed-length tokens in diffusion models, our approach maintains the causal property of GPT-style models for flexible in-context capabilities, while leveraging bidirectional modeling within individual frames for efficiency. With the proposed approach, we train a novel video autoregressive model without vector quantization, termed NOVA. Our results demonstrate that NOVA surpasses prior autoregressive video models in data efficiency, inference speed, visual fidelity, and video fluency, even with a much smaller model capacity, i.e., 0.6B parameters. NOVA also outperforms state-of-the-art image diffusion models in text-to-image generation tasks, with a significantly lower training cost. Additionally, NOVA generalizes well across extended video durations and enables diverse zero-shot applications in one unified model. Code and models are publicly available at this https URL.
• 摘要：本文介绍了一种能够高效生成自回归视频的新方法。我们建议将视频生成问题重新表述为时间逐帧预测和空间逐集预测的非量化自回归建模。与先前自回归模型中的光栅扫描预测或扩散模型中固定长度标记的联合分布建模不同，我们的方法保留了 GPT 风格模型的因果属性，以实现灵活的上下文功能，同时利用单个帧内的双向建模来提高效率。利用所提出的方法，我们训练了一种无需矢量量化的新型视频自回归模型，称为 NOVA。我们的结果表明，即使模型容量小得多（即 0.6B 参数），NOVA 在数据效率、推理速度、视觉保真度和视频流畅度方面也超越了先前的自回归视频模型。NOVA 在文本到图像生成任务中的表现也优于最先进的图像扩散模型，并且训练成本显著降低。此外，NOVA 可以很好地适用于较长的视频时长，并可以在一个统一的模型中实现多种零样本应用。代码和模型可在此 https URL 上公开获取。

Title: E-CAR: Efficient Continuous Autoregressive Image Generation via Multistage Modeling

• Authors: Zhihang Yuan, Yuzhang Shang, Hanling Zhang, Tongcheng Fang, Rui Xie, Bingxin Xu, Yan Yan, Shengen Yan, Guohao Dai, Yu Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.14170
• Pdf URL: https://arxiv.org/pdf/2412.14170
• Copy Paste: [[2412.14170]] E-CAR: Efficient Continuous Autoregressive Image Generation via Multistage Modeling(https://arxiv.org/abs/2412.14170)
• Keywords: generation
• Abstract: Recent advances in autoregressive (AR) models with continuous tokens for image generation show promising results by eliminating the need for discrete tokenization. However, these models face efficiency challenges due to their sequential token generation nature and reliance on computationally intensive diffusion-based sampling. We present ECAR (Efficient Continuous Auto-Regressive Image Generation via Multistage Modeling), an approach that addresses these limitations through two intertwined innovations: (1) a stage-wise continuous token generation strategy that reduces computational complexity and provides progressively refined token maps as hierarchical conditions, and (2) a multistage flow-based distribution modeling method that transforms only partial-denoised distributions at each stage comparing to complete denoising in normal diffusion models. Holistically, ECAR operates by generating tokens at increasing resolutions while simultaneously denoising the image at each stage. This design not only reduces token-to-image transformation cost by a factor of the stage number but also enables parallel processing at the token level. Our approach not only enhances computational efficiency but also aligns naturally with image generation principles by operating in continuous token space and following a hierarchical generation process from coarse to fine details. Experimental results demonstrate that ECAR achieves comparable image quality to DiT Peebles & Xie [2023] while requiring 10$\times$ FLOPs reduction and 5$\times$ speedup to generate a 256$\times$256 image.
• 摘要：用于图像生成的连续标记的自回归 (AR) 模型的最新进展通过消除离散标记化的需求显示出了良好的效果。然而，这些模型由于其顺序标记生成性质和对计算密集型扩散采样的依赖而面临效率挑战。我们提出了 ECAR（通过多阶段建模实现高效的连续自回归图像生成），这种方法通过两个相互交织的创新解决了这些限制：（1）一种分阶段的连续标记生成策略，可降低计算复杂性并提供逐步细化的标记图作为分层条件，以及（2）一种基于多阶段流的分布建模方法，与正常扩散模型中的完全去噪相比，它在每个阶段仅转换部分去噪的分布。从整体上讲，ECAR 通过以越来越高的分辨率生成标记来运行，同时在每个阶段对图像进行去噪。这种设计不仅可以将标记到图像的转换成本降低一个阶段数，还可以在标记级别实现并行处理。我们的方法不仅提高了计算效率，而且通过在连续标记空间中操作并遵循从粗到细细节的分层生成过程，自然地与图像生成原理保持一致。实验结果表明，ECAR 实现了与 DiT Peebles & Xie [2023] 相当的图像质量，同时需要减少 10$\times$ FLOPs 和加速 5$\times$ 来生成 256$\times$256 的图像。

Title: AniDoc: Animation Creation Made Easier

• Authors: Yihao Meng, Hao Ouyang, Hanlin Wang, Qiuyu Wang, Wen Wang, Ka Leong Cheng, Zhiheng Liu, Yujun Shen, Huamin Qu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.14173
• Pdf URL: https://arxiv.org/pdf/2412.14173
• Copy Paste: [[2412.14173]] AniDoc: Animation Creation Made Easier(https://arxiv.org/abs/2412.14173)
• Keywords: generative
• Abstract: The production of 2D animation follows an industry-standard workflow, encompassing four essential stages: character design, keyframe animation, in-betweening, and coloring. Our research focuses on reducing the labor costs in the above process by harnessing the potential of increasingly powerful generative AI. Using video diffusion models as the foundation, AniDoc emerges as a video line art colorization tool, which automatically converts sketch sequences into colored animations following the reference character specification. Our model exploits correspondence matching as an explicit guidance, yielding strong robustness to the variations (e.g., posture) between the reference character and each line art frame. In addition, our model could even automate the in-betweening process, such that users can easily create a temporally consistent animation by simply providing a character image as well as the start and end sketches. Our code is available at: this https URL.
• 摘要：2D 动画的制作遵循行业标准的工作流程，包括四个基本阶段：角色设计、关键帧动画、中间画和着色。我们的研究重点是通过利用日益强大的生成式 AI 的潜力来降低上述过程中的劳动力成本。以视频扩散模型为基础，AniDoc 成为视频线条艺术着色工具，它根据参考角色规范自动将草图序列转换为彩色动画。我们的模型利用对应匹配作为明确指导，对参考角色和每个线条艺术帧之间的变化（例如姿势）具有很强的鲁棒性。此外，我们的模型甚至可以自动化中间画过程，这样用户只需提供角色图像以及开始和结束草图即可轻松创建时间一致的动画。我们的代码可在以下网址获得：此 https URL。