2024-12-23

Title: Next Patch Prediction for Autoregressive Visual Generation

• Authors: Yatian Pang, Peng Jin, Shuo Yang, Bin Lin, Bin Zhu, Zhenyu Tang, Liuhan Chen, Francis E. H. Tay, Ser-Nam Lim, Harry Yang, Li Yuan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15321
• Pdf URL: https://arxiv.org/pdf/2412.15321
• Copy Paste: [[2412.15321]] Next Patch Prediction for Autoregressive Visual Generation(https://arxiv.org/abs/2412.15321)
• Keywords: generation
• Abstract: Autoregressive models, built based on the Next Token Prediction (NTP) paradigm, show great potential in developing a unified framework that integrates both language and vision tasks. In this work, we rethink the NTP for autoregressive image generation and propose a novel Next Patch Prediction (NPP) paradigm. Our key idea is to group and aggregate image tokens into patch tokens containing high information density. With patch tokens as a shorter input sequence, the autoregressive model is trained to predict the next patch, thereby significantly reducing the computational cost. We further propose a multi-scale coarse-to-fine patch grouping strategy that exploits the natural hierarchical property of image data. Experiments on a diverse range of models (100M-1.4B parameters) demonstrate that the next patch prediction paradigm could reduce the training cost to around 0.6 times while improving image generation quality by up to 1.0 FID score on the ImageNet benchmark. We highlight that our method retains the original autoregressive model architecture without introducing additional trainable parameters or specifically designing a custom image tokenizer, thus ensuring flexibility and seamless adaptation to various autoregressive models for visual generation.
• 摘要：基于下一个标记预测 (NTP) 范式构建的自回归模型在开发集成语言和视觉任务的统一框架方面表现出巨大潜力。在这项工作中，我们重新思考了用于自回归图像生成的 NTP，并提出了一种新颖的下一个补丁预测 (NPP) 范式。我们的主要思想是将图像标记分组并聚合为包含高信息密度的补丁标记。使用补丁标记作为较短的输入序列，训练自回归模型来预测下一个补丁，从而显著降低计算成本。我们进一步提出了一种多尺度从粗到细的补丁分组策略，该策略利用图像数据的自然层次结构属性。在各种模型（100M-1.4B 参数）上进行的实验表明，下一个补丁预测范式可以将训练成本降低到大约 0.6 倍，同时在 ImageNet 基准上将图像生成质量提高高达 1.0 FID 分数。我们强调，我们的方法保留了原始的自回归模型架构，而无需引入额外的可训练参数或专门设计自定义图像标记器，从而确保了灵活性和对各种自回归模型的无缝适应以进行视觉生成。

Title: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis

• Authors: Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, Yuki Mitsufuji
• Subjects: cs.CV, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.15322
• Pdf URL: https://arxiv.org/pdf/2412.15322
• Copy Paste: [[2412.15322]] Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis(https://arxiv.org/abs/2412.15322)
• Keywords: generation
• Abstract: We propose to synthesize high-quality and synchronized audio, given video and optional text conditions, using a novel multimodal joint training framework MMAudio. In contrast to single-modality training conditioned on (limited) video data only, MMAudio is jointly trained with larger-scale, readily available text-audio data to learn to generate semantically aligned high-quality audio samples. Additionally, we improve audio-visual synchrony with a conditional synchronization module that aligns video conditions with audio latents at the frame level. Trained with a flow matching objective, MMAudio achieves new video-to-audio state-of-the-art among public models in terms of audio quality, semantic alignment, and audio-visual synchronization, while having a low inference time (1.23s to generate an 8s clip) and just 157M parameters. MMAudio also achieves surprisingly competitive performance in text-to-audio generation, showing that joint training does not hinder single-modality performance. Code and demo are available at: this https URL
• 摘要：我们建议使用一种新颖的多模态联合训练框架 MMAudio，在给定视频和可选文本条件的情况下合成高质量同步音频。与仅以（有限）视频数据为条件的单模态训练相比，MMAudio 与更大规模、现成的文本音频数据联合训练，以学习生成语义对齐的高质量音频样本。此外，我们使用条件同步模块改进了视听同步，该模块在帧级别将视频条件与音频潜在条件对齐。使用流匹配目标进行训练后，MMAudio 在音频质量、语义对齐和视听同步方面在公共模型中实现了新的视频到音频最新水平，同时具有较低的推理时间（生成 8 秒剪辑需要 1.23 秒）和仅 157M 个参数。MMAudio 在文本到音频生成方面也实现了令人惊讶的竞争性能，表明联合训练不会妨碍单模态性能。代码和演示可在以下位置获得：此 https URL

Title: Efficient Fine-Tuning and Concept Suppression for Pruned Diffusion Models

• Authors: Reza Shirkavand, Peiran Yu, Shangqian Gao, Gowthami Somepalli, Tom Goldstein, Heng Huang
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15341
• Pdf URL: https://arxiv.org/pdf/2412.15341
• Copy Paste: [[2412.15341]] Efficient Fine-Tuning and Concept Suppression for Pruned Diffusion Models(https://arxiv.org/abs/2412.15341)
• Keywords: generation, generative
• Abstract: Recent advances in diffusion generative models have yielded remarkable progress. While the quality of generated content continues to improve, these models have grown considerably in size and complexity. This increasing computational burden poses significant challenges, particularly in resource-constrained deployment scenarios such as mobile devices. The combination of model pruning and knowledge distillation has emerged as a promising solution to reduce computational demands while preserving generation quality. However, this technique inadvertently propagates undesirable behaviors, including the generation of copyrighted content and unsafe concepts, even when such instances are absent from the fine-tuning dataset. In this paper, we propose a novel bilevel optimization framework for pruned diffusion models that consolidates the fine-tuning and unlearning processes into a unified phase. Our approach maintains the principal advantages of distillation-namely, efficient convergence and style transfer capabilities-while selectively suppressing the generation of unwanted content. This plug-in framework is compatible with various pruning and concept unlearning methods, facilitating efficient, safe deployment of diffusion models in controlled environments.
• 摘要：传播生成模型的最新进展取得了显著进展。虽然生成内容的质量不断提高，但这些模型的规模和复杂性却大幅增加。这种不断增加的计算负担带来了重大挑战，尤其是在资源受限的部署场景（例如移动设备）中。模型修剪和知识蒸馏的结合已成为一种有前途的解决方案，可以在保持生成质量的同时减少计算需求。然而，这种技术会无意中传播不良行为，包括生成受版权保护的内容和不安全的概念，即使微调数据集中没有此类实例。在本文中，我们提出了一种用于修剪传播模型的新型双层优化框架，将微调和反学习过程整合为一个统一的阶段。我们的方法保留了蒸馏的主要优势（即高效的收敛和风格转换能力），同时有选择地抑制了不需要的内容的生成。该插件框架与各种修剪和概念反学习方法兼容，有助于在受控环境中高效、安全地部署传播模型。

Title: Large Language Models on Small Resource-Constrained Systems: Performance Characterization, Analysis and Trade-offs

• Authors: Liam Seymour, Basar Kutukcu, Sabur Baidya
• Subjects: cs.LG, cs.CC
• Abstract URL: https://arxiv.org/abs/2412.15352
• Pdf URL: https://arxiv.org/pdf/2412.15352
• Copy Paste: [[2412.15352]] Large Language Models on Small Resource-Constrained Systems: Performance Characterization, Analysis and Trade-offs(https://arxiv.org/abs/2412.15352)
• Keywords: generation, generative
• Abstract: Generative AI like the Large Language Models (LLMs) has become more available for the general consumer in recent years. Publicly available services, e.g., ChatGPT, perform token generation on networked cloud server hardware, effectively removing the hardware entry cost for end users. However, the reliance on network access for these services, privacy and security risks involved, and sometimes the needs of the application make it necessary to run LLMs locally on edge devices. A significant amount of research has been done on optimization of LLMs and other transformer-based models on non-networked, resource-constrained devices, but they typically target older hardware. Our research intends to provide a 'baseline' characterization of more recent commercially available embedded hardware for LLMs, and to provide a simple utility to facilitate batch testing LLMs on recent Jetson hardware. We focus on the latest line of NVIDIA Jetson devices (Jetson Orin), and a set of publicly available LLMs (Pythia) ranging between 70 million and 1.4 billion parameters. Through detailed experimental evaluation with varying software and hardware parameters, we showcase trade-off spaces and optimization choices. Additionally, we design our testing structure to facilitate further research that involves performing batch LLM testing on Jetson hardware.
• 摘要：近年来，大型语言模型 (LLM) 等生成式人工智能已越来越为普通消费者所接受。ChatGPT 等公开服务在联网云服务器硬件上执行令牌生成，有效地消除了最终用户的硬件入门成本。然而，这些服务对网络访问的依赖、所涉及的隐私和安全风险以及有时应用程序的需求使得必须在边缘设备上本地运行 LLM。已经对 LLM 和其他基于变压器的模型在非联网、资源受限的设备上进行优化进行了大量研究，但它们通常针对较旧的硬件。我们的研究旨在为 LLM 提供较新的商用嵌入式硬件的“基线”特性，并提供一个简单的实用程序，以方便在最近的 Jetson 硬件上批量测试 LLM。我们专注于最新系列的 NVIDIA Jetson 设备 (Jetson Orin) 和一组公开可用的 LLM (Pythia)，参数范围在 7000 万到 14 亿之间。通过对不同软件和硬件参数进行详细的实验评估，我们展示了权衡空间和优化选择。此外，我们设计了测试结构，以促进涉及在 Jetson 硬件上执行批量 LLM 测试的进一步研究。

Title: Spatiotemporally Coherent Probabilistic Generation of Weather from Climate

• Authors: Jonathan Schmidt, Luca Schmidt, Felix Strnad, Nicole Ludwig, Philipp Hennig
• Subjects: cs.LG, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2412.15361
• Pdf URL: https://arxiv.org/pdf/2412.15361
• Copy Paste: [[2412.15361]] Spatiotemporally Coherent Probabilistic Generation of Weather from Climate(https://arxiv.org/abs/2412.15361)
• Keywords: generation, generative
• Abstract: Local climate information is crucial for impact assessment and decision-making, yet coarse global climate simulations cannot capture small-scale phenomena. Current statistical downscaling methods infer these phenomena as temporally decoupled spatial patches. However, to preserve physical properties, estimating spatio-temporally coherent high-resolution weather dynamics for multiple variables across long time horizons is crucial. We present a novel generative approach that uses a score-based diffusion model trained on high-resolution reanalysis data to capture the statistical properties of local weather dynamics. After training, we condition on coarse climate model data to generate weather patterns consistent with the aggregate information. As this inference task is inherently uncertain, we leverage the probabilistic nature of diffusion models and sample multiple trajectories. We evaluate our approach with high-resolution reanalysis information before applying it to the climate model downscaling task. We then demonstrate that the model generates spatially and temporally coherent weather dynamics that align with global climate output.
• 摘要：当地气候信息对于影响评估和决策至关重要，但粗略的全球气候模拟无法捕捉小规模现象。当前的统计降尺度方法将这些现象推断为时间上解耦的空间斑块。然而，为了保持物理特性，估计长时间范围内多个变量的时空相干高分辨率天气动态至关重要。我们提出了一种新颖的生成方法，该方法使用基于高分辨率再分析数据训练的基于分数的扩散模型来捕捉当地天气动态的统计特性。训练后，我们以粗略的气候模型数据为条件，生成与总体信息一致的天气模式。由于这个推理任务本质上是不确定的，我们利用扩散模型的概率性质并采样多个轨迹。在将我们的方法应用于气候模型降尺度任务之前，我们会使用高分辨率再分析信息对其进行评估。然后，我们证明该模型生成与全球气候输出一致的空间和时间相干的天气动态。

Title: Time Will Tell: Timing Side Channels via Output Token Count in Large Language Models

• Authors: Tianchen Zhang, Gururaj Saileshwar, David Lie
• Subjects: cs.LG, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.15431
• Pdf URL: https://arxiv.org/pdf/2412.15431
• Copy Paste: [[2412.15431]] Time Will Tell: Timing Side Channels via Output Token Count in Large Language Models(https://arxiv.org/abs/2412.15431)
• Keywords: generation
• Abstract: This paper demonstrates a new side-channel that enables an adversary to extract sensitive information about inference inputs in large language models (LLMs) based on the number of output tokens in the LLM response. We construct attacks using this side-channel in two common LLM tasks: recovering the target language in machine translation tasks and recovering the output class in classification tasks. In addition, due to the auto-regressive generation mechanism in LLMs, an adversary can recover the output token count reliably using a timing channel, even over the network against a popular closed-source commercial LLM. Our experiments show that an adversary can learn the output language in translation tasks with more than 75% precision across three different models (Tower, M2M100, MBart50). Using this side-channel, we also show the input class in text classification tasks can be leaked out with more than 70% precision from open-source LLMs like Llama-3.1, Llama-3.2, Gemma2, and production models like GPT-4o. Finally, we propose tokenizer-, system-, and prompt-based mitigations against the output token count side-channel.
• 摘要：本文演示了一种新的侧通道，使攻击者能够根据 LLM 响应中的输出标记数量提取有关大型语言模型 (LLM) 中推理输入的敏感信息。我们在两个常见的 LLM 任务中使用此侧通道构建攻击：在机器翻译任务中恢复目标语言，在分类任务中恢复输出类。此外，由于 LLM 中的自回归生成机制，攻击者可以使用定时通道可靠地恢复输出标记计数，甚至通过网络对抗流行的闭源商业 LLM。我们的实验表明，攻击者可以在三个不同的模型（Tower、M2M100、MBart50）中以超过 75% 的精度学习翻译任务中的输出语言。使用这种侧通道，我们还展示了文本分类任务中的输入类可以从开源 LLM（如 Llama-3.1、Llama-3.2、Gemma2）和生产模型（如 GPT-4o）中以超过 70% 的精度泄露出来。最后，我们提出了基于标记器、系统和提示的针对输出标记计数侧通道的缓解措施。

Title: LiHi-GS: LiDAR-Supervised Gaussian Splatting for Highway Driving Scene Reconstruction

• Authors: Pou-Chun Kung, Xianling Zhang, Katherine A. Skinner, Nikita Jaipuria
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.15447
• Pdf URL: https://arxiv.org/pdf/2412.15447
• Copy Paste: [[2412.15447]] LiHi-GS: LiDAR-Supervised Gaussian Splatting for Highway Driving Scene Reconstruction(https://arxiv.org/abs/2412.15447)
• Keywords: generation
• Abstract: Photorealistic 3D scene reconstruction plays an important role in autonomous driving, enabling the generation of novel data from existing datasets to simulate safety-critical scenarios and expand training data without additional acquisition costs. Gaussian Splatting (GS) facilitates real-time, photorealistic rendering with an explicit 3D Gaussian representation of the scene, providing faster processing and more intuitive scene editing than the implicit Neural Radiance Fields (NeRFs). While extensive GS research has yielded promising advancements in autonomous driving applications, they overlook two critical aspects: First, existing methods mainly focus on low-speed and feature-rich urban scenes and ignore the fact that highway scenarios play a significant role in autonomous driving. Second, while LiDARs are commonplace in autonomous driving platforms, existing methods learn primarily from images and use LiDAR only for initial estimates or without precise sensor modeling, thus missing out on leveraging the rich depth information LiDAR offers and limiting the ability to synthesize LiDAR data. In this paper, we propose a novel GS method for dynamic scene synthesis and editing with improved scene reconstruction through LiDAR supervision and support for LiDAR rendering. Unlike prior works that are tested mostly on urban datasets, to the best of our knowledge, we are the first to focus on the more challenging and highly relevant highway scenes for autonomous driving, with sparse sensor views and monotone backgrounds.
• 摘要：逼真的 3D 场景重建在自动驾驶中起着重要作用，它可以从现有数据集生成新数据，以模拟安全关键场景并扩展训练数据，而无需额外的采集成本。高斯分层 (GS) 通过显式 3D 高斯表示场景实现实时、逼真的渲染，比隐式神经辐射场 (NeRF) 提供更快的处理速度和更直观的场景编辑。虽然广泛的 GS 研究在自动驾驶应用方面取得了有希望的进展，但它们忽略了两个关键方面：首先，现有方法主要关注低速和功能丰富的城市场景，而忽略了高速公路场景在自动驾驶中发挥重要作用的事实。其次，虽然激光雷达在自动驾驶平台中很常见，但现有方法主要从图像中学习，仅将激光雷达用于初步估计或没有精确的传感器建模，因此无法利用激光雷达提供的丰富深度信息，并限制了合成激光雷达数据的能力。在本文中，我们提出了一种新颖的 GS 方法，用于动态场景合成和编辑，通过 LiDAR 监督和对 LiDAR 渲染的支持改进了场景重建。与之前主要在城市数据集上测试的研究不同，据我们所知，我们是第一个专注于更具挑战性且与自动驾驶高度相关的高速公路场景的研究，这些场景具有稀疏的传感器视图和单调的背景。

Title: GCA-3D: Towards Generalized and Consistent Domain Adaptation of 3D Generators

• Authors: Hengjia Li, Yang Liu, Yibo Zhao, Haoran Cheng, Yang Yang, Linxuan Xia, Zekai Luo, Qibo Qiu, Boxi Wu, Tu Zheng, Zheng Yang, Deng Cai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15491
• Pdf URL: https://arxiv.org/pdf/2412.15491
• Copy Paste: [[2412.15491]] GCA-3D: Towards Generalized and Consistent Domain Adaptation of 3D Generators(https://arxiv.org/abs/2412.15491)
• Keywords: generation, generative
• Abstract: Recently, 3D generative domain adaptation has emerged to adapt the pre-trained generator to other domains without collecting massive datasets and camera pose distributions. Typically, they leverage large-scale pre-trained text-to-image diffusion models to synthesize images for the target domain and then fine-tune the 3D model. However, they suffer from the tedious pipeline of data generation, which inevitably introduces pose bias between the source domain and synthetic dataset. Furthermore, they are not generalized to support one-shot image-guided domain adaptation, which is more challenging due to the more severe pose bias and additional identity bias introduced by the single image reference. To address these issues, we propose GCA-3D, a generalized and consistent 3D domain adaptation method without the intricate pipeline of data generation. Different from previous pipeline methods, we introduce multi-modal depth-aware score distillation sampling loss to efficiently adapt 3D generative models in a non-adversarial manner. This multi-modal loss enables GCA-3D in both text prompt and one-shot image prompt adaptation. Besides, it leverages per-instance depth maps from the volume rendering module to mitigate the overfitting problem and retain the diversity of results. To enhance the pose and identity consistency, we further propose a hierarchical spatial consistency loss to align the spatial structure between the generated images in the source and target domain. Experiments demonstrate that GCA-3D outperforms previous methods in terms of efficiency, generalization, pose accuracy, and identity consistency.
• 摘要：最近，出现了 3D 生成域自适应，用于将预训练的生成器适配到其他域，而无需收集大量数据集和相机姿势分布。通常，它们利用大规模预训练的文本到图像扩散模型来合成目标域的图像，然后微调 3D 模型。然而，它们遭受繁琐的数据生成流程的困扰，这不可避免地会在源域和合成数据集之间引入姿势偏差。此外，它们不能推广到支持一次性图像引导的域自适应，这更具挑战性，因为单个图像参考引入了更严重的姿势偏差和额外的身份偏差。为了解决这些问题，我们提出了 GCA-3D，这是一种通用且一致的 3D 域自适应方法，没有复杂的数据生成流程。与以前的流程方法不同，我们引入了多模态深度感知分数蒸馏采样损失，以非对抗方式有效地适应 3D 生成模型。这种多模态损失使 GCA-3D 能够进行文本提示和一次性图像提示自适应。此外，它利用体积渲染模块中的每个实例深度图来缓解过度拟合问题并保留结果的多样性。为了增强姿势和身份一致性，我们进一步提出了分层空间一致性损失，以对齐源域和目标域中生成的图像之间的空间结构。实验表明，GCA-3D 在效率、泛化、姿势准确性和身份一致性方面优于以前的方法。

Title: Stylish and Functional: Guided Interpolation Subject to Physical Constraints

• Authors: Yan-Ying Chen, Nikos Arechiga, Chenyang Yuan, Matthew Hong, Matt Klenk, Charlene Wu
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15507
• Pdf URL: https://arxiv.org/pdf/2412.15507
• Copy Paste: [[2412.15507]] Stylish and Functional: Guided Interpolation Subject to Physical Constraints(https://arxiv.org/abs/2412.15507)
• Keywords: generation, generative
• Abstract: Generative AI is revolutionizing engineering design practices by enabling rapid prototyping and manipulation of designs. One example of design manipulation involves taking two reference design images and using them as prompts to generate a design image that combines aspects of both. Real engineering designs have physical constraints and functional requirements in addition to aesthetic design considerations. Internet-scale foundation models commonly used for image generation, however, are unable to take these physical constraints and functional requirements into consideration as part of the generation process. We consider the problem of generating a design inspired by two input designs, and propose a zero-shot framework toward enforcing physical, functional requirements over the generation process by leveraging a pretrained diffusion model as the backbone. As a case study, we consider the example of rotational symmetry in generation of wheel designs. Automotive wheels are required to be rotationally symmetric for physical stability. We formulate the requirement of rotational symmetry by the use of a symmetrizer, and we use this symmetrizer to guide the diffusion process towards symmetric wheel generations. Our experimental results find that the proposed approach makes generated interpolations with higher realism than methods in related work, as evaluated by Fréchet inception distance (FID). We also find that our approach generates designs that more closely satisfy physical and functional requirements than generating without the symmetry guidance.
• 摘要：生成式 AI 正在通过实现快速原型设计和设计操作来彻底改变工程设计实践。设计操作的一个例子是获取两个参考设计图像，并将它们用作提示来生成结合两者各方面的设计图像。除了美学设计考虑之外，真正的工程设计还有物理约束和功能要求。然而，通常用于图像生成的互联网规模基础模型无法将这些物理约束和功能要求作为生成过程的一部分考虑在内。我们考虑了生成受两个输入设计启发的设计的问题，并提出了一个零样本框架，通过利用预训练的扩散模型作为骨干，在生成过程中强制执行物理和功能要求。作为一个案例研究，我们考虑了车轮设计生成中的旋转对称性示例。汽车车轮需要旋转对称才能实现物理稳定性。我们使用对称化器来制定旋转对称的要求，并使用此对称化器引导扩散过程实现对称车轮生成。我们的实验结果发现，通过 Fréchet 初始距离 (FID) 评估，所提出的方法生成的插值比相关工作中的方法更真实。我们还发现，与没有对称性指导的生成相比，我们的方法生成的设计更能满足物理和功能要求。

Title: ChangeDiff: A Multi-Temporal Change Detection Data Generator with Flexible Text Prompts via Diffusion Model

• Authors: Qi Zang, Jiayi Yang, Shuang Wang, Dong Zhao, Wenjun Yi, Zhun Zhong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15541
• Pdf URL: https://arxiv.org/pdf/2412.15541
• Copy Paste: [[2412.15541]] ChangeDiff: A Multi-Temporal Change Detection Data Generator with Flexible Text Prompts via Diffusion Model(https://arxiv.org/abs/2412.15541)
• Keywords: generative
• Abstract: Data-driven deep learning models have enabled tremendous progress in change detection (CD) with the support of pixel-level annotations. However, collecting diverse data and manually annotating them is costly, laborious, and knowledge-intensive. Existing generative methods for CD data synthesis show competitive potential in addressing this issue but still face the following limitations: 1) difficulty in flexibly controlling change events, 2) dependence on additional data to train the data generators, 3) focus on specific change detection tasks. To this end, this paper focuses on the semantic CD (SCD) task and develops a multi-temporal SCD data generator ChangeDiff by exploring powerful diffusion models. ChangeDiff innovatively generates change data in two steps: first, it uses text prompts and a text-to-layout (T2L) model to create continuous layouts, and then it employs layout-to-image (L2I) to convert these layouts into images. Specifically, we propose multi-class distribution-guided text prompts (MCDG-TP), allowing for layouts to be generated flexibly through controllable classes and their corresponding ratios. Subsequently, to generalize the T2L model to the proposed MCDG-TP, a class distribution refinement loss is further designed as training supervision. %For the former, a multi-classdistribution-guided text prompt (MCDG-TP) is proposed to complement via controllable classes and ratios. To generalize the text-to-image diffusion model to the proposed MCDG-TP, a class distribution refinement loss is designed as training supervision. For the latter, MCDG-TP in three modes is proposed to synthesize new layout masks from various texts. Our generated data shows significant progress in temporal continuity, spatial diversity, and quality realism, empowering change detectors with accuracy and transferability. The code is available at this https URL
• 摘要：数据驱动的深度学习模型在像素级注释的支持下推动了变化检测 (CD) 的巨大进步。然而，收集多样化的数据并手动注释它们成本高昂、费力且需要大量知识。现有的 CD 数据合成生成方法在解决这一问题上表现出竞争潜力，但仍然面临以下限制：1）难以灵活控制变化事件，2）依赖额外数据来训练数据生成器，3）专注于特定的变化检测任务。为此，本文专注于语义 CD (SCD) 任务，并通过探索强大的扩散模型开发了一个多时间 SCD 数据生成器 ChangeDiff。ChangeDiff 创新地分两步生成变化数据：首先，它使用文本提示和文本到布局 (T2L) 模型创建连续布局，然后使用布局到图像 (L2I) 将这些布局转换为图像。具体来说，我们提出了多类分布引导的文本提示（MCDG-TP），允许通过可控的类别及其相应的比率灵活地生成布局。随后，为了将 T2L 模型推广到所提出的 MCDG-TP，进一步设计了类分布细化损失作为训练监督。%对于前者，提出了一种多类分布引导的文本提示（MCDG-TP）通过可控的类别和比率进行补充。为了将文本到图像扩散模型推广到所提出的 MCDG-TP，设计了类分布细化损失作为训练监督。对于后者，提出了三种模式的 MCDG-TP 来从各种文本中合成新的布局蒙版。我们生成的数据在时间连续性、空间多样性和质量真实感方面取得了显着进步，为变化检测器提供了准确性和可转移性。代码可在此 https URL 上获得

Title: DefFiller: Mask-Conditioned Diffusion for Salient Steel Surface Defect Generation

• Authors: Yichun Tai, Zhenzhen Huang, Tao Peng, Zhijiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15570
• Pdf URL: https://arxiv.org/pdf/2412.15570
• Copy Paste: [[2412.15570]] DefFiller: Mask-Conditioned Diffusion for Salient Steel Surface Defect Generation(https://arxiv.org/abs/2412.15570)
• Keywords: generation, generative
• Abstract: Current saliency-based defect detection methods show promise in industrial settings, but the unpredictability of defects in steel production environments complicates dataset creation, hampering model performance. Existing data augmentation approaches using generative models often require pixel-level annotations, which are time-consuming and resource-intensive. To address this, we introduce DefFiller, a mask-conditioned defect generation method that leverages a layout-to-image diffusion model. DefFiller generates defect samples paired with mask conditions, eliminating the need for pixel-level annotations and enabling direct use in model training. We also develop an evaluation framework to assess the quality of generated samples and their impact on detection performance. Experimental results on the SD-Saliency-900 dataset demonstrate that DefFiller produces high-quality defect images that accurately match the provided mask conditions, significantly enhancing the performance of saliency-based defect detection models trained on the augmented dataset.
• 摘要：目前基于显著性的缺陷检测方法在工业环境中前景光明，但钢铁生产环境中缺陷的不可预测性使数据集创建变得复杂，从而妨碍了模型性能。使用生成模型的现有数据增强方法通常需要像素级注释，这既耗时又耗资源。为了解决这个问题，我们引入了 DefFiller，这是一种利用布局到图像扩散模型的掩模条件缺陷生成方法。DefFiller 生成与掩模条件配对的缺陷样本，从而无需像素级注释并可直接用于模型训练。我们还开发了一个评估框架来评估生成样本的质量及其对检测性能的影响。在 SD-Saliency-900 数据集上的实验结果表明，DefFiller 生成的高质量缺陷图像与提供的掩模条件准确匹配，显著提高了在增强数据集上训练的基于显著性的缺陷检测模型的性能。

Title: A Deep Probabilistic Framework for Continuous Time Dynamic Graph Generation

• Authors: Ryien Hosseini, Filippo Simini, Venkatram Vishwanath, Henry Hoffmann
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15582
• Pdf URL: https://arxiv.org/pdf/2412.15582
• Copy Paste: [[2412.15582]] A Deep Probabilistic Framework for Continuous Time Dynamic Graph Generation(https://arxiv.org/abs/2412.15582)
• Keywords: generation, generative
• Abstract: Recent advancements in graph representation learning have shifted attention towards dynamic graphs, which exhibit evolving topologies and features over time. The increased use of such graphs creates a paramount need for generative models suitable for applications such as data augmentation, obfuscation, and anomaly detection. However, there are few generative techniques that handle continuously changing temporal graph data; existing work largely relies on augmenting static graphs with additional temporal information to model dynamic interactions between nodes. In this work, we propose a fundamentally different approach: We instead directly model interactions as a joint probability of an edge forming between two nodes at a given time. This allows us to autoregressively generate new synthetic dynamic graphs in a largely assumption free, scalable, and inductive manner. We formalize this approach as DG-Gen, a generative framework for continuous time dynamic graphs, and demonstrate its effectiveness over five datasets. Our experiments demonstrate that DG-Gen not only generates higher fidelity graphs compared to traditional methods but also significantly advances link prediction tasks.
• 摘要：图表征学习的最新进展已将注意力转向动态图，动态图会随着时间的推移展现出不断发展的拓扑和特征。此类图的使用越来越多，因此迫切需要适用于数据增强、混淆和异常检测等应用的生成模型。然而，很少有生成技术能够处理不断变化的时间图数据；现有工作主要依赖于使用额外的时间信息增强静态图来模拟节点之间的动态交互。在这项工作中，我们提出了一种完全不同的方法：我们直接将交互建模为给定时间两个节点之间形成边的联合概率。这使我们能够以基本无假设、可扩展和归纳的方式自回归生成新的合成动态图。我们将这种方法形式化为 DG-Gen，这是一种连续时间动态图的生成框架，并在五个数据集上证明了其有效性。我们的实验表明，与传统方法相比，DG-Gen 不仅可以生成保真度更高的图，而且还能显著推进链接预测任务。

Title: CustomTTT: Motion and Appearance Customized Video Generation via Test-Time Training

• Authors: Xiuli Bi, Jian Lu, Bo Liu, Xiaodong Cun, Yong Zhang, Weisheng Li, Bin Xiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15646
• Pdf URL: https://arxiv.org/pdf/2412.15646
• Copy Paste: [[2412.15646]] CustomTTT: Motion and Appearance Customized Video Generation via Test-Time Training(https://arxiv.org/abs/2412.15646)
• Keywords: generation
• Abstract: Benefiting from large-scale pre-training of text-video pairs, current text-to-video (T2V) diffusion models can generate high-quality videos from the text description. Besides, given some reference images or videos, the parameter-efficient fine-tuning method, i.e. LoRA, can generate high-quality customized concepts, e.g., the specific subject or the motions from a reference video. However, combining the trained multiple concepts from different references into a single network shows obvious artifacts. To this end, we propose CustomTTT, where we can joint custom the appearance and the motion of the given video easily. In detail, we first analyze the prompt influence in the current video diffusion model and find the LoRAs are only needed for the specific layers for appearance and motion customization. Besides, since each LoRA is trained individually, we propose a novel test-time training technique to update parameters after combination utilizing the trained customized models. We conduct detailed experiments to verify the effectiveness of the proposed methods. Our method outperforms several state-of-the-art works in both qualitative and quantitative evaluations.
• 摘要：得益于对文本-视频对的大规模预训练，当前的文本到视频 (T2V) 传播模型可以根据文本描述生成高质量的视频。此外，给定一些参考图像或视频，参数高效的微调方法 LoRA 可以生成高质量的定制概念，例如参考视频中的特定主题或动作。但是，将来自不同参考的训练后的多个概念组合成一个网络会出现明显的伪影。为此，我们提出了 CustomTTT，我们可以轻松地将给定视频的外观和运动结合起来。具体来说，我们首先分析了当前视频传播模型中的即时影响，发现仅特定层需要 LoRA 来进行外观和运动定制。此外，由于每个 LoRA 都是单独训练的，我们提出了一种新颖的测试时间训练技术，利用训练后的定制模型在组合后更新参数。我们进行了详细的实验来验证所提出方法的有效性。我们的方法在定性和定量评估中都优于几项最先进的工作。

Title: Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolution

• Authors: Wentao Tan, Qiong Cao, Yibing Zhan, Chao Xue, Changxing Ding
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15650
• Pdf URL: https://arxiv.org/pdf/2412.15650
• Copy Paste: [[2412.15650]] Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolution(https://arxiv.org/abs/2412.15650)
• Keywords: generation
• Abstract: Human preference alignment can greatly enhance Multimodal Large Language Models (MLLMs), but collecting high-quality preference data is costly. A promising solution is the self-evolution strategy, where models are iteratively trained on data they generate. However, current techniques still rely on human- or GPT-annotated data and sometimes require additional models or ground truth answers. To address these issues, we propose a novel multimodal self-evolution framework that enables the model to autonomously generate high-quality questions and answers using only unannotated images. First, we implement an image-driven self-questioning mechanism, allowing the model to create and evaluate questions based on image content, regenerating them if they are irrelevant or unanswerable. This sets a strong foundation for answer generation. Second, we introduce an answer self-enhancement technique, starting with image captioning to improve answer quality. We also use corrupted images to generate rejected answers, forming distinct preference pairs for optimization. Finally, we incorporate an image content alignment loss function alongside Direct Preference Optimization (DPO) loss to reduce hallucinations, ensuring the model focuses on image content. Experiments show that our framework performs competitively with methods using external information, offering a more efficient and scalable approach to MLLMs.
• 摘要：人类偏好对齐可以极大地增强多模态大型语言模型 (MLLM)，但收集高质量的偏好数据成本高昂。一种有前途的解决方案是自我进化策略，其中模型在其生成的数据上进行迭代训练。然而，当前的技术仍然依赖于人类或 GPT 注释的数据，有时需要额外的模型或基本事实答案。为了解决这些问题，我们提出了一个新颖的多模态自进化框架，使模型能够仅使用未注释的图像自主生成高质量的问题和答案。首先，我们实现了一个图像驱动的自我提问机制，允许模型根据图像内容创建和评估问题，如果问题不相关或无法回答，则重新生成问题。这为答案生成奠定了坚实的基础。其次，我们引入了一种答案自我增强技术，从图像字幕开始提高答案质量。我们还使用损坏的图像来生成被拒绝的答案，形成不同的偏好对以进行优化。最后，我们将图像内容对齐损失函数与直接偏好优化 (DPO) 损失结合起来，以减少幻觉，确保模型专注于图像内容。实验表明，我们的框架与使用外部信息的方法相比具有竞争力，为 MLLM 提供了一种更高效、更可扩展的方法。

Title: Synthetic Tabular Data Generation for Imbalanced Classification: The Surprising Effectiveness of an Overlap Class

• Authors: Annie D'souza, Swetha M, Sunita Sarawagi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.15657
• Pdf URL: https://arxiv.org/pdf/2412.15657
• Copy Paste: [[2412.15657]] Synthetic Tabular Data Generation for Imbalanced Classification: The Surprising Effectiveness of an Overlap Class(https://arxiv.org/abs/2412.15657)
• Keywords: generation, generative
• Abstract: Handling imbalance in class distribution when building a classifier over tabular data has been a problem of long-standing interest. One popular approach is augmenting the training dataset with synthetically generated data. While classical augmentation techniques were limited to linear interpolation of existing minority class examples, recently higher capacity deep generative models are providing greater promise. However, handling of imbalance in class distribution when building a deep generative model is also a challenging problem, that has not been studied as extensively as imbalanced classifier model training. We show that state-of-the-art deep generative models yield significantly lower-quality minority examples than majority examples. %In this paper, we start with the observation that imbalanced data training of generative models trained imbalanced dataset which under-represent the minority class. We propose a novel technique of converting the binary class labels to ternary class labels by introducing a class for the region where minority and majority distributions overlap. We show that just this pre-processing of the training set, significantly improves the quality of data generated spanning several state-of-the-art diffusion and GAN-based models. While training the classifier using synthetic data, we remove the overlap class from the training data and justify the reasons behind the enhanced accuracy. We perform extensive experiments on four real-life datasets, five different classifiers, and five generative models demonstrating that our method enhances not only the synthesizer performance of state-of-the-art models but also the classifier performance.
• 摘要：在表格数据上构建分类器时处理类别分布不平衡一直是一个长期关注的问题。一种流行的方法是用合成生成的数据扩充训练数据集。虽然经典的增强技术仅限于现有少数类示例的线性插值，但最近容量更大的深度生成模型提供了更大的希望。然而，在构建深度生成模型时处理类别分布不平衡也是一个具有挑战性的问题，它还没有像不平衡分类器模型训练那样得到广泛的研究。我们表明，最先进的深度生成模型产生的少数类示例质量明显低于多数类示例。%在本文中，我们首先观察到，生成模型的不平衡数据训练训练了不平衡数据集，这些数据集对少数类的代表性不足。我们提出了一种将二元类标签转换为三元类标签的新技术，方法是为少数和多数分布重叠的区域引入一个类。我们表明，仅对训练集进行这种预处理，就可以显著提高跨多个最先进的扩散和基于 GAN 的模型生成的数据的质量。在使用合成数据训练分类器时，我们从训练数据中删除了重叠类，并解释了准确率提高的原因。我们对四个真实数据集、五个不同的分类器和五个生成模型进行了广泛的实验，结果表明，我们的方法不仅提高了最先进模型的合成器性能，还提高了分类器性能。

Title: Learning Group Interactions and Semantic Intentions for Multi-Object Trajectory Prediction

• Authors: Mengshi Qi, Yuxin Yang, Huadong Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15673
• Pdf URL: https://arxiv.org/pdf/2412.15673
• Copy Paste: [[2412.15673]] Learning Group Interactions and Semantic Intentions for Multi-Object Trajectory Prediction(https://arxiv.org/abs/2412.15673)
• Keywords: generation
• Abstract: Effective modeling of group interactions and dynamic semantic intentions is crucial for forecasting behaviors like trajectories or movements. In complex scenarios like sports, agents' trajectories are influenced by group interactions and intentions, including team strategies and opponent actions. To this end, we propose a novel diffusion-based trajectory prediction framework that integrates group-level interactions into a conditional diffusion model, enabling the generation of diverse trajectories aligned with specific group activity. To capture dynamic semantic intentions, we frame group interaction prediction as a cooperative game, using Banzhaf interaction to model cooperation trends. We then fuse semantic intentions with enhanced agent embeddings, which are refined through both global and local aggregation. Furthermore, we expand the NBA SportVU dataset by adding human annotations of team-level tactics for trajectory and tactic prediction tasks. Extensive experiments on three widely-adopted datasets demonstrate that our model outperforms state-of-the-art methods. Our source code and data are available at this https URL.
• 摘要：有效地对群体互动和动态语义意图进行建模对于预测轨迹或动作等行为至关重要。在体育运动等复杂场景中，代理的轨迹受到群体互动和意图的影响，包括团队策略和对手行动。为此，我们提出了一种基于扩散的新型轨迹预测框架，将群体级互动整合到条件扩散模型中，从而能够生成与特定群体活动一致的多样化轨迹。为了捕捉动态语义意图，我们将群体互动预测构建为合作游戏，使用 Banzhaf 互动来模拟合作趋势。然后，我们将语义意图与增强的代理嵌入融合，并通过全局和局部聚合对其进行细化。此外，我们通过为轨迹和战术预测任务添加团队级战术的人工注释来扩展 NBA SportVU 数据集。在三个广泛采用的数据集上进行的大量实验表明，我们的模型优于最先进的方法。我们的源代码和数据可在此 https URL 上获得。

Title: PersonaMagic: Stage-Regulated High-Fidelity Face Customization with Tandem Equilibrium

• Authors: Xinzhe Li, Jiahui Zhan, Shengfeng He, Yangyang Xu, Junyu Dong, Huaidong Zhang, Yong Du
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15674
• Pdf URL: https://arxiv.org/pdf/2412.15674
• Copy Paste: [[2412.15674]] PersonaMagic: Stage-Regulated High-Fidelity Face Customization with Tandem Equilibrium(https://arxiv.org/abs/2412.15674)
• Keywords: generation, generative
• Abstract: Personalized image generation has made significant strides in adapting content to novel concepts. However, a persistent challenge remains: balancing the accurate reconstruction of unseen concepts with the need for editability according to the prompt, especially when dealing with the complex nuances of facial features. In this study, we delve into the temporal dynamics of the text-to-image conditioning process, emphasizing the crucial role of stage partitioning in introducing new concepts. We present PersonaMagic, a stage-regulated generative technique designed for high-fidelity face customization. Using a simple MLP network, our method learns a series of embeddings within a specific timestep interval to capture face concepts. Additionally, we develop a Tandem Equilibrium mechanism that adjusts self-attention responses in the text encoder, balancing text description and identity preservation, improving both areas. Extensive experiments confirm the superiority of PersonaMagic over state-of-the-art methods in both qualitative and quantitative evaluations. Moreover, its robustness and flexibility are validated in non-facial domains, and it can also serve as a valuable plug-in for enhancing the performance of pretrained personalization models.
• 摘要：个性化图像生成在将内容适应新概念方面取得了重大进展。然而，仍然存在一个持续的挑战：平衡对看不见的概念的准确重建与根据提示进行编辑的需求，特别是在处理面部特征的复杂细微差别时。在本研究中，我们深入研究了文本到图像调节过程的时间动态，强调了阶段划分在引入新概念方面的关键作用。我们提出了 PersonaMagic，这是一种专为高保真面部定制而设计的阶段调节生成技术。使用一个简单的 MLP 网络，我们的方法在特定的时间步长间隔内学习一系列嵌入以捕捉面部概念。此外，我们开发了一种串联平衡机制，可以调整文本编码器中的自注意力响应，平衡文本描述和身份保存，从而改善这两个领域。大量实验证实了 PersonaMagic 在定性和定量评估方面都优于最先进的方法。此外，它的稳健性和灵活性在非面部领域得到验证，并且它还可以作为增强预训练个性化模型性能的有价值的插件。

Title: AI-generated Image Quality Assessment in Visual Communication

• Authors: Yu Tian, Yixuan Li, Baoliang Chen, Hanwei Zhu, Shiqi Wang, Sam Kwong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15677
• Pdf URL: https://arxiv.org/pdf/2412.15677
• Copy Paste: [[2412.15677]] AI-generated Image Quality Assessment in Visual Communication(https://arxiv.org/abs/2412.15677)
• Keywords: quality assessment
• Abstract: Assessing the quality of artificial intelligence-generated images (AIGIs) plays a crucial role in their application in real-world scenarios. However, traditional image quality assessment (IQA) algorithms primarily focus on low-level visual perception, while existing IQA works on AIGIs overemphasize the generated content itself, neglecting its effectiveness in real-world applications. To bridge this gap, we propose AIGI-VC, a quality assessment database for AI-Generated Images in Visual Communication, which studies the communicability of AIGIs in the advertising field from the perspectives of information clarity and emotional interaction. The dataset consists of 2,500 images spanning 14 advertisement topics and 8 emotion types. It provides coarse-grained human preference annotations and fine-grained preference descriptions, benchmarking the abilities of IQA methods in preference prediction, interpretation, and reasoning. We conduct an empirical study of existing representative IQA methods and large multi-modal models on the AIGI-VC dataset, uncovering their strengths and weaknesses.
• 摘要：评估人工智能生成的图像 (AIGI) 的质量对其在实际场景中的应用起着至关重要的作用。然而，传统的图像质量评估 (IQA) 算法主要关注低级视觉感知，而现有的 AIGI IQA 研究过分强调生成内容本身，忽视了其在实际应用中的有效性。为了弥补这一差距，我们提出了 AIGI-VC，一个视觉传播中人工智能生成的图像的质量评估数据库，它从信息清晰度和情感互动的角度研究 AIGI 在广告领域的可传播性。该数据集包含 2,500 幅图像，涵盖 14 个广告主题和 8 种情感类型。它提供粗粒度的人类偏好注释和细粒度的偏好描述，对 IQA 方法在偏好预测、解释和推理方面的能力进行了基准测试。我们在 AIGI-VC 数据集上对现有的代表性 IQA 方法和大型多模态模型进行了实证研究，揭示了它们的优缺点。

Title: DOLLAR: Few-Step Video Generation via Distillation and Latent Reward Optimization

• Authors: Zihan Ding, Chi Jin, Difan Liu, Haitian Zheng, Krishna Kumar Singh, Qiang Zhang, Yan Kang, Zhe Lin, Yuchen Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15689
• Pdf URL: https://arxiv.org/pdf/2412.15689
• Copy Paste: [[2412.15689]] DOLLAR: Few-Step Video Generation via Distillation and Latent Reward Optimization(https://arxiv.org/abs/2412.15689)
• Keywords: generation
• Abstract: Diffusion probabilistic models have shown significant progress in video generation; however, their computational efficiency is limited by the large number of sampling steps required. Reducing sampling steps often compromises video quality or generation diversity. In this work, we introduce a distillation method that combines variational score distillation and consistency distillation to achieve few-step video generation, maintaining both high quality and diversity. We also propose a latent reward model fine-tuning approach to further enhance video generation performance according to any specified reward metric. This approach reduces memory usage and does not require the reward to be differentiable. Our method demonstrates state-of-the-art performance in few-step generation for 10-second videos (128 frames at 12 FPS). The distilled student model achieves a score of 82.57 on VBench, surpassing the teacher model as well as baseline models Gen-3, T2V-Turbo, and Kling. One-step distillation accelerates the teacher model's diffusion sampling by up to 278.6 times, enabling near real-time generation. Human evaluations further validate the superior performance of our 4-step student models compared to teacher model using 50-step DDIM sampling.
• 摘要：扩散概率模型在视频生成方面取得了重大进展；然而，它们的计算效率受到所需大量采样步骤的限制。减少采样步骤通常会损害视频质量或生成多样性。在这项工作中，我们引入了一种蒸馏方法，该方法结合了变分分数蒸馏和一致性蒸馏，以实现少步视频生成，同时保持高质量和多样性。我们还提出了一种潜在奖励模型微调方法，以根据任何指定的奖励指标进一步提高视频生成性能。这种方法减少了内存使用量，并且不需要奖励可区分。我们的方法在 10 秒视频（128 帧，12 FPS）的少步生成中展示了最先进的性能。蒸馏后的学生模型在 VBench 上获得了 82.57 分，超过了教师模型以及基线模型 Gen-3、T2V-Turbo 和 Kling。一步蒸馏将教师模型的扩散采样速度提高了 278.6 倍，实现了近乎实时的生成。人工评估进一步验证了我们的 4 步学生模型与使用 50 步 DDIM 采样的教师模型相比具有更优异的性能。

Title: Robustness-enhanced Myoelectric Control with GAN-based Open-set Recognition

• Authors: Cheng Wang, Ziyang Feng, Pin Zhang, Manjiang Cao, Yiming Yuan, Tengfei Chang
• Subjects: cs.CV, cs.HC, eess.SP
• Abstract URL: https://arxiv.org/abs/2412.15819
• Pdf URL: https://arxiv.org/pdf/2412.15819
• Copy Paste: [[2412.15819]] Robustness-enhanced Myoelectric Control with GAN-based Open-set Recognition(https://arxiv.org/abs/2412.15819)
• Keywords: generative
• Abstract: Electromyography (EMG) signals are widely used in human motion recognition and medical rehabilitation, yet their variability and susceptibility to noise significantly limit the reliability of myoelectric control systems. Existing recognition algorithms often fail to handle unfamiliar actions effectively, leading to system instability and errors. This paper proposes a novel framework based on Generative Adversarial Networks (GANs) to enhance the robustness and usability of myoelectric control systems by enabling open-set recognition. The method incorporates a GAN-based discriminator to identify and reject unknown actions, maintaining system stability by preventing misclassifications. Experimental evaluations on publicly available and self-collected datasets demonstrate a recognition accuracy of 97.6\% for known actions and a 23.6\% improvement in Active Error Rate (AER) after rejecting unknown actions. The proposed approach is computationally efficient and suitable for deployment on edge devices, making it practical for real-world applications.
• 摘要：肌电信号 (EMG) 广泛应用于人体动作识别和医疗康复，但其多变性和易受噪声影响严重限制了肌电控制系统的可靠性。现有的识别算法通常无法有效处理不熟悉的动作，从而导致系统不稳定和错误。本文提出了一种基于生成对抗网络 (GAN) 的新框架，通过实现开放集识别来增强肌电控制系统的鲁棒性和可用性。该方法结合了基于 GAN 的鉴别器来识别和拒绝未知动作，通过防止错误分类来保持系统稳定性。在公开可用和自收集的数据集上进行的实验评估表明，已知动作的识别准确率为 97.6%，拒绝未知动作后主动错误率 (AER) 提高了 23.6%。所提出的方法计算效率高，适合部署在边缘设备上，使其适用于实际应用。

Title: Multi-dimensional Visual Prompt Enhanced Image Restoration via Mamba-Transformer Aggregation

• Authors: Aiwen Jiang, Hourong Chen, Zhiwen Chen, Jihua Ye, Mingwen Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15845
• Pdf URL: https://arxiv.org/pdf/2412.15845
• Copy Paste: [[2412.15845]] Multi-dimensional Visual Prompt Enhanced Image Restoration via Mamba-Transformer Aggregation(https://arxiv.org/abs/2412.15845)
• Keywords: restoration
• Abstract: Recent efforts on image restoration have focused on developing "all-in-one" models that can handle different degradation types and levels within single model. However, most of mainstream Transformer-based ones confronted with dilemma between model capabilities and computation burdens, since self-attention mechanism quadratically increase in computational complexity with respect to image size, and has inadequacies in capturing long-range dependencies. Most of Mamba-related ones solely scanned feature map in spatial dimension for global modeling, failing to fully utilize information in channel dimension. To address aforementioned problems, this paper has proposed to fully utilize complementary advantages from Mamba and Transformer without sacrificing computation efficiency. Specifically, the selective scanning mechanism of Mamba is employed to focus on spatial modeling, enabling capture long-range spatial dependencies under linear complexity. The self-attention mechanism of Transformer is applied to focus on channel modeling, avoiding high computation burdens that are in quadratic growth with image's spatial dimensions. Moreover, to enrich informative prompts for effective image restoration, multi-dimensional prompt learning modules are proposed to learn prompt-flows from multi-scale encoder/decoder layers, benefiting for revealing underlying characteristic of various degradations from both spatial and channel perspectives, therefore, enhancing the capabilities of "all-in-one" model to solve various restoration tasks. Extensive experiment results on several image restoration benchmark tasks such as image denoising, dehazing, and deraining, have demonstrated that the proposed method can achieve new state-of-the-art performance, compared with many popular mainstream methods. Related source codes and pre-trained parameters will be public on github this https URL.
• 摘要：近年来，图像恢复领域的研究主要集中在开发“一体化”模型，即在单个模型中处理不同退化类型和程度的模型。然而，目前主流的基于Transformer的模型大多面临模型能力和计算负担之间的矛盾，因为自注意力机制的计算复杂度随图像尺寸呈二次方增长，且在捕捉长程依赖性方面存在不足。大多数与Mamba相关的模型仅在空间维度上扫描特征图进行全局建模，未能充分利用通道维度的信息。针对上述问题，本文提出了在不牺牲计算效率的情况下充分利用Mamba和Transformer互补优势的方法。具体而言，利用Mamba的选择性扫描机制专注于空间建模，能够在线性复杂度下捕捉长程空间依赖性。利用Transformer的自注意力机制专注于通道建模，避免了随图像空间维度二次方增长的高计算负担。此外，为了丰富有效图像恢复的信息提示，提出了多维提示学习模块来学习来自多尺度编码器/解码器层的提示流，有利于从空间和通道角度揭示各种退化的根本特征，从而增强“一体化”模型解决各种恢复任务的能力。在图像去噪、去雾和去雨等多个图像恢复基准任务上的大量实验结果表明，与许多流行的主流方法相比，所提出的方法可以实现新的最佳性能。相关源代码和预训练参数将在 github 上公开，网址为 https。

Title: Semi-Supervised Adaptation of Diffusion Models for Handwritten Text Generation

• Authors: Kai Brandenbusch
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15853
• Pdf URL: https://arxiv.org/pdf/2412.15853
• Copy Paste: [[2412.15853]] Semi-Supervised Adaptation of Diffusion Models for Handwritten Text Generation(https://arxiv.org/abs/2412.15853)
• Keywords: generation
• Abstract: The generation of images of realistic looking, readable handwritten text is a challenging task which is referred to as handwritten text generation (HTG). Given a string and examples from a writer, the goal is to synthesize an image depicting the correctly spelled word in handwriting with the calligraphic style of the desired writer. An important application of HTG is the generation of training images in order to adapt downstream models for new data sets. With their success in natural image generation, diffusion models (DMs) have become the state-of-the-art approach in HTG. In this work, we present an extension of a latent DM for HTG to enable generation of writing styles not seen during training by learning style conditioning with a masked auto encoder. Our proposed content encoder allows for different ways of conditioning the DM on textual and calligraphic features. Additionally, we employ classifier-free guidance and explore the influence on the quality of the generated training images. For adapting the model to a new unlabeled data set, we propose a semi-supervised training scheme. We evaluate our approach on the IAM-database and use the RIMES-database to examine the generation of data not seen during training achieving improvements in this particularly promising application of DMs for HTG.
• 摘要：生成逼真、可读的手写文本图像是一项具有挑战性的任务，称为手写文本生成 (HTG)。给定一个字符串和作者的示例，目标是合成一幅描绘正确拼写的手写单词的图像，并带有所需作者的书法风格。HTG 的一个重要应用是生成训练图像，以便使下游模型适应新的数据集。凭借在自然图像生成方面的成功，扩散模型 (DM) 已成为 HTG 中最先进的方法。在这项工作中，我们提出了一种用于 HTG 的潜在 DM 扩展，以便通过使用掩码自动编码器学习风格调节来生成训练期间未见过的书写风格。我们提出的内容编码器允许以不同的方式根据文本和书法特征调节 DM。此外，我们采用无分类器指导并探索对生成的训练图像质量的影响。为了使模型适应新的未标记数据集，我们提出了一种半监督训练方案。我们在 IAM 数据库上评估了我们的方法，并使用 RIMES 数据库来检查训练期间未见的数据生成，从而实现了对 HTG 的 DM 这一特别有前景的应用的改进。

Title: NeuroPump: Simultaneous Geometric and Color Rectification for Underwater Images

• Authors: Yue Guo, Haoxiang Liao, Haibin Ling, Bingyao Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.15890
• Pdf URL: https://arxiv.org/pdf/2412.15890
• Copy Paste: [[2412.15890]] NeuroPump: Simultaneous Geometric and Color Rectification for Underwater Images(https://arxiv.org/abs/2412.15890)
• Keywords: restoration
• Abstract: Underwater image restoration aims to remove geometric and color distortions due to water refraction, absorption and scattering. Previous studies focus on restoring either color or the geometry, but to our best knowledge, not both. However, in practice it may be cumbersome to address the two rectifications one-by-one. In this paper, we propose NeuroPump, a self-supervised method to simultaneously optimize and rectify underwater geometry and color as if water were pumped out. The key idea is to explicitly model refraction, absorption and scattering in Neural Radiance Field (NeRF) pipeline, such that it not only performs simultaneous geometric and color rectification, but also enables to synthesize novel views and optical effects by controlling the decoupled parameters. In addition, to address issue of lack of real paired ground truth images, we propose an underwater 360 benchmark dataset that has real paired (i.e., with and without water) images. Our method clearly outperforms other baselines both quantitatively and qualitatively.
• 摘要：水下图像修复旨在消除由于水的折射、吸收和散射造成的几何和颜色失真。以前的研究侧重于恢复颜色或几何形状，但据我们所知，并非同时恢复两者。然而，在实践中，逐一解决这两个校正问题可能很麻烦。在本文中，我们提出了 NeuroPump，这是一种自监督方法，可以同时优化和校正水下几何形状和颜色，就像水被抽出来一样。关键思想是在神经辐射场 (NeRF) 管道中明确模拟折射、吸收和散射，这样它不仅可以同时执行几何和颜色校正，而且还可以通过控制解耦参数来合成新颖的视图和光学效果。此外，为了解决缺乏真实配对地面真实图像的问题，我们提出了一个水下 360 基准数据集，其中包含真实配对（即有水和无水）图像。我们的方法在数量和质量上都明显优于其他基线。

Title: RiTTA: Modeling Event Relations in Text-to-Audio Generation

• Authors: Yuhang He, Yash Jain, Xubo Liu, Andrew Markham, Vibhav Vineet
• Subjects: cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2412.15922
• Pdf URL: https://arxiv.org/pdf/2412.15922
• Copy Paste: [[2412.15922]] RiTTA: Modeling Event Relations in Text-to-Audio Generation(https://arxiv.org/abs/2412.15922)
• Keywords: generation
• Abstract: Despite significant advancements in Text-to-Audio (TTA) generation models achieving high-fidelity audio with fine-grained context understanding, they struggle to model the relations between audio events described in the input text. However, previous TTA methods have not systematically explored audio event relation modeling, nor have they proposed frameworks to enhance this capability. In this work, we systematically study audio event relation modeling in TTA generation models. We first establish a benchmark for this task by: 1. proposing a comprehensive relation corpus covering all potential relations in real-world scenarios; 2. introducing a new audio event corpus encompassing commonly heard audios; and 3. proposing new evaluation metrics to assess audio event relation modeling from various perspectives. Furthermore, we propose a finetuning framework to enhance existing TTA models ability to model audio events relation. Code is available at: this https URL
• 摘要：尽管文本转音频 (TTA) 生成模型取得了重大进展，实现了具有细粒度上下文理解的高保真音频，但它们仍难以对输入文本中描述的音频事件之间的关系进行建模。然而，以前的 TTA 方法并没有系统地探索音频事件关系建模，也没有提出框架来增强这种能力。在这项工作中，我们系统地研究了 TTA 生成模型中的音频事件关系建模。我们首先通过以下方式为这项任务建立一个基准：1. 提出一个全面的关系语料库，涵盖现实场景中的所有潜在关系；2. 引入一个包含常见音频的新音频事件语料库；3. 提出新的评估指标，从各个角度评估音频事件关系建模。此外，我们提出了一个微调框架来增强现有 TTA 模型对音频事件关系进行建模的能力。代码可在以下网址获得：此 https URL

Title: Watertox: The Art of Simplicity in Universal Attacks A Cross-Model Framework for Robust Adversarial Generation

• Authors: Zhenghao Gao, Shengjie Xu, Meixi Chen, Fangyao Zhao
• Subjects: cs.CV, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.15924
• Pdf URL: https://arxiv.org/pdf/2412.15924
• Copy Paste: [[2412.15924]] Watertox: The Art of Simplicity in Universal Attacks A Cross-Model Framework for Robust Adversarial Generation(https://arxiv.org/abs/2412.15924)
• Keywords: generation
• Abstract: Contemporary adversarial attack methods face significant limitations in cross-model transferability and practical applicability. We present Watertox, an elegant adversarial attack framework achieving remarkable effectiveness through architectural diversity and precision-controlled perturbations. Our two-stage Fast Gradient Sign Method combines uniform baseline perturbations ($\epsilon_1 = 0.1$) with targeted enhancements ($\epsilon_2 = 0.4$). The framework leverages an ensemble of complementary architectures, from VGG to ConvNeXt, synthesizing diverse perspectives through an innovative voting mechanism. Against state-of-the-art architectures, Watertox reduces model accuracy from 70.6% to 16.0%, with zero-shot attacks achieving up to 98.8% accuracy reduction against unseen architectures. These results establish Watertox as a significant advancement in adversarial methodologies, with promising applications in visual security systems and CAPTCHA generation.
• 摘要：当代对抗性攻击方法在跨模型可转移性和实际适用性方面面临重大限制。我们提出了 Watertox，这是一个优雅的对抗性攻击框架，通过架构多样性和精确控制的扰动实现了显著的效果。我们的两阶段快速梯度符号方法将均匀基线扰动（$\epsilon_1 = 0.1$）与有针对性的增强（$\epsilon_2 = 0.4$）相结合。该框架利用从 VGG 到 ConvNeXt 的一系列互补架构，通过创新的投票机制综合了不同的观点。与最先进的架构相比，Watertox 将模型准确率从 70.6% 降低到 16.0%，而零样本攻击对看不见的架构的准确率降低了高达 98.8%。这些结果证实了 Watertox 是对抗性方法的重大进步，在视觉安全系统和 CAPTCHA 生成中具有广阔的应用前景。

Title: Reframing Image Difference Captioning with BLIP2IDC and Synthetic Augmentation

• Authors: Gautier Evennou, Antoine Chaffin, Vivien Chappelier, Ewa Kijak
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.15939
• Pdf URL: https://arxiv.org/pdf/2412.15939
• Copy Paste: [[2412.15939]] Reframing Image Difference Captioning with BLIP2IDC and Synthetic Augmentation(https://arxiv.org/abs/2412.15939)
• Keywords: generation, generative
• Abstract: The rise of the generative models quality during the past years enabled the generation of edited variations of images at an important scale. To counter the harmful effects of such technology, the Image Difference Captioning (IDC) task aims to describe the differences between two images. While this task is successfully handled for simple 3D rendered images, it struggles on real-world images. The reason is twofold: the training data-scarcity, and the difficulty to capture fine-grained differences between complex images. To address those issues, we propose in this paper a simple yet effective framework to both adapt existing image captioning models to the IDC task and augment IDC datasets. We introduce BLIP2IDC, an adaptation of BLIP2 to the IDC task at low computational cost, and show it outperforms two-streams approaches by a significant margin on real-world IDC datasets. We also propose to use synthetic augmentation to improve the performance of IDC models in an agnostic fashion. We show that our synthetic augmentation strategy provides high quality data, leading to a challenging new dataset well-suited for IDC named Syned1.
• 摘要：过去几年中，生成模型质量的提高使得能够大规模生成经过编辑的图像变体。为了抵消这种技术的有害影响，图像差异字幕 (IDC) 任务旨在描述两幅图像之间的差异。虽然此任务可以成功处理简单的 3D 渲染图像，但它在现实世界图像上却举步维艰。原因有两个：训练数据稀缺，以及难以捕捉复杂图像之间的细粒度差异。为了解决这些问题，我们在本文中提出了一个简单而有效的框架，既可以将现有的图像字幕模型调整为 IDC 任务，也可以增强 IDC 数据集。我们引入了 BLIP2IDC，这是 BLIP2 以低计算成本适应 IDC 任务的改编，并表明它在现实世界的 IDC 数据集上的表现远远优于双流方法。我们还建议使用合成增强以不可知的方式提高 IDC 模型的性能。我们表明，我们的合成增强策略提供了高质量的数据，从而产生了一个名为 Syned1 的适合 IDC 的具有挑战性的新数据集。

Title: SafeCFG: Redirecting Harmful Classifier-Free Guidance for Safe Generation

• Authors: Jiadong Pan, Hongcheng Gao, Liang Li, Zheng-Jun Zha, Qingming Huang, Jiebo Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16039
• Pdf URL: https://arxiv.org/pdf/2412.16039
• Copy Paste: [[2412.16039]] SafeCFG: Redirecting Harmful Classifier-Free Guidance for Safe Generation(https://arxiv.org/abs/2412.16039)
• Keywords: generation
• Abstract: Diffusion models (DMs) have demonstrated exceptional performance in text-to-image (T2I) tasks, leading to their widespread use. With the introduction of classifier-free guidance (CFG), the quality of images generated by DMs is improved. However, DMs can generate more harmful images by maliciously guiding the image generation process through CFG. Some safe guidance methods aim to mitigate the risk of generating harmful images but often reduce the quality of clean image generation. To address this issue, we introduce the Harmful Guidance Redirector (HGR), which redirects harmful CFG direction while preserving clean CFG direction during image generation, transforming CFG into SafeCFG and achieving high safety and quality generation. We train HGR to redirect multiple harmful CFG directions simultaneously, demonstrating its ability to eliminate various harmful elements while preserving high-quality generation. Additionally, we find that HGR can detect image harmfulness, allowing for unsupervised fine-tuning of safe diffusion models without pre-defined clean or harmful labels. Experimental results show that by incorporating HGR, images generated by diffusion models achieve both high quality and strong safety, and safe DMs trained through unsupervised methods according to the harmfulness detected by HGR also exhibit good safety performance. The codes will be publicly available.
• 摘要：扩散模型 (DM) 在文本转图像 (T2I) 任务中表现出色，因此得到了广泛应用。随着无分类器引导 (CFG) 的引入，DM 生成的图像质量得到了提高。然而，DM 可以通过 CFG 恶意引导图像生成过程，从而生成更多有害图像。一些安全引导方法旨在降低生成有害图像的风险，但往往会降低干净图像生成的质量。为了解决这个问题，我们引入了有害引导重定向器 (HGR)，它在图像生成过程中重定向有害的 CFG 方向，同时保留干净的 CFG 方向，将 CFG 转换为安全 CFG，实现高安全性和高质量的生成。我们训练 HGR 同时重定向多个有害的 CFG 方向，展示了其消除各种有害元素同时保留高质量生成的能力。此外，我们发现 HGR 可以检测图像有害性，允许对安全扩散模型进行无监督微调，而无需预定义的干净或有害标签。实验结果表明，通过引入 HGR，由扩散模型生成的图像既具有高质量，又具有很强的安全性，而根据 HGR 检测到的危害性通过无监督方法训练的安全 DM 也表现出良好的安全性能。代码将公开。

Title: Differentially Private Federated Learning of Diffusion Models for Synthetic Tabular Data Generation

• Authors: Timur Sattarov, Marco Schreyer, Damian Borth
• Subjects: cs.LG, q-fin.ST
• Abstract URL: https://arxiv.org/abs/2412.16083
• Pdf URL: https://arxiv.org/pdf/2412.16083
• Copy Paste: [[2412.16083]] Differentially Private Federated Learning of Diffusion Models for Synthetic Tabular Data Generation(https://arxiv.org/abs/2412.16083)
• Keywords: generation
• Abstract: The increasing demand for privacy-preserving data analytics in finance necessitates solutions for synthetic data generation that rigorously uphold privacy standards. We introduce DP-Fed-FinDiff framework, a novel integration of Differential Privacy, Federated Learning and Denoising Diffusion Probabilistic Models designed to generate high-fidelity synthetic tabular data. This framework ensures compliance with stringent privacy regulations while maintaining data utility. We demonstrate the effectiveness of DP-Fed-FinDiff on multiple real-world financial datasets, achieving significant improvements in privacy guarantees without compromising data quality. Our empirical evaluations reveal the optimal trade-offs between privacy budgets, client configurations, and federated optimization strategies. The results affirm the potential of DP-Fed-FinDiff to enable secure data sharing and robust analytics in highly regulated domains, paving the way for further advances in federated learning and privacy-preserving data synthesis.
• 摘要：金融领域对隐私保护数据分析的需求日益增长，这需要严格遵守隐私标准的合成数据生成解决方案。我们推出了 DP-Fed-FinDiff 框架，这是差分隐私、联邦学习和去噪扩散概率模型的全新集成，旨在生成高保真合成表格数据。该框架确保遵守严格的隐私法规，同时保持数据实用性。我们在多个现实世界的金融数据集上证明了 DP-Fed-FinDiff 的有效性，在不影响数据质量的情况下显着改善了隐私保障。我们的实证评估揭示了隐私预算、客户端配置和联邦优化策略之间的最佳权衡。结果肯定了 DP-Fed-FinDiff 在高度监管的领域实现安全数据共享和强大分析的潜力，为联邦学习和隐私保护数据合成的进一步发展铺平了道路。

Title: Explainable AI for Multivariate Time Series Pattern Exploration: Latent Space Visual Analytics with Time Fusion Transformer and Variational Autoencoders in Power Grid Event Diagnosis

• Authors: Haowen Xu, Ali Boyaci, Jianming Lian, Aaron Wilson
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16098
• Pdf URL: https://arxiv.org/pdf/2412.16098
• Copy Paste: [[2412.16098]] Explainable AI for Multivariate Time Series Pattern Exploration: Latent Space Visual Analytics with Time Fusion Transformer and Variational Autoencoders in Power Grid Event Diagnosis(https://arxiv.org/abs/2412.16098)
• Keywords: generative
• Abstract: Detecting and analyzing complex patterns in multivariate time-series data is crucial for decision-making in urban and environmental system operations. However, challenges arise from the high dimensionality, intricate complexity, and interconnected nature of complex patterns, which hinder the understanding of their underlying physical processes. Existing AI methods often face limitations in interpretability, computational efficiency, and scalability, reducing their applicability in real-world scenarios. This paper proposes a novel visual analytics framework that integrates two generative AI models, Time Fusion Transformer (TFT) and Variational Autoencoders (VAEs), to reduce complex patterns into lower-dimensional latent spaces and visualize them in 2D using dimensionality reduction techniques such as PCA, t-SNE, and UMAP with DBSCAN. These visualizations, presented through coordinated and interactive views and tailored glyphs, enable intuitive exploration of complex multivariate temporal patterns, identifying patterns' similarities and uncover their potential correlations for a better interpretability of the AI outputs. The framework is demonstrated through a case study on power grid signal data, where it identifies multi-label grid event signatures, including faults and anomalies with diverse root causes. Additionally, novel metrics and visualizations are introduced to validate the models and evaluate the performance, efficiency, and consistency of latent maps generated by TFT and VAE under different configurations. These analyses provide actionable insights for model parameter tuning and reliability improvements. Comparative results highlight that TFT achieves shorter run times and superior scalability to diverse time-series data shapes compared to VAE. This work advances fault diagnosis in multivariate time series, fostering explainable AI to support critical system operations.
• 摘要：检测和分析多变量时间序列数据中的复杂模式对于城市和环境系统运营中的决策至关重要。然而，复杂模式的高维性、错综复杂和相互关联的性质带来了挑战，阻碍了对其底层物理过程的理解。现有的人工智能方法往往在可解释性、计算效率和可扩展性方面受到限制，从而降低了它们在现实场景中的适用性。本文提出了一种新颖的可视化分析框架，该框架集成了两种生成式人工智能模型，即时间融合变换器 (TFT) 和变分自动编码器 (VAE)，以将复杂模式简化为低维潜在空间，并使用降维技术（如 PCA、t-SNE 和带有 DBSCAN 的 UMAP）在 2D 中可视化它们。这些可视化通过协调和交互式视图以及定制的字形呈现，可以直观地探索复杂的多变量时间模式，识别模式的相似性并发现它们的潜在相关性，从而更好地解释人工智能输出。该框架通过电网信号数据的案例研究进行了展示，该框架识别多标签电网事件特征，包括具有不同根本原因的故障和异常。此外，还引入了新颖的指标和可视化来验证模型并评估 TFT 和 VAE 在不同配置下生成的潜在图的性能、效率和一致性。这些分析为模型参数调整和可靠性改进提供了可行的见解。比较结果表明，与 VAE 相比，TFT 实现了更短的运行时间和对各种时间序列数据形状的出色可扩展性。这项工作推动了多变量时间序列中的故障诊断，促进了可解释的 AI 来支持关键系统操作。

Title: CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up

• Authors: Songhua Liu, Zhenxiong Tan, Xinchao Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16112
• Pdf URL: https://arxiv.org/pdf/2412.16112
• Copy Paste: [[2412.16112]] CLEAR: Conv-Like Linearization Revs Pre-Trained Diffusion Transformers Up(https://arxiv.org/abs/2412.16112)
• Keywords: generation
• Abstract: Diffusion Transformers (DiT) have become a leading architecture in image generation. However, the quadratic complexity of attention mechanisms, which are responsible for modeling token-wise relationships, results in significant latency when generating high-resolution images. To address this issue, we aim at a linear attention mechanism in this paper that reduces the complexity of pre-trained DiTs to linear. We begin our exploration with a comprehensive summary of existing efficient attention mechanisms and identify four key factors crucial for successful linearization of pre-trained DiTs: locality, formulation consistency, high-rank attention maps, and feature integrity. Based on these insights, we introduce a convolution-like local attention strategy termed CLEAR, which limits feature interactions to a local window around each query token, and thus achieves linear complexity. Our experiments indicate that, by fine-tuning the attention layer on merely 10K self-generated samples for 10K iterations, we can effectively transfer knowledge from a pre-trained DiT to a student model with linear complexity, yielding results comparable to the teacher model. Simultaneously, it reduces attention computations by 99.5% and accelerates generation by 6.3 times for generating 8K-resolution images. Furthermore, we investigate favorable properties in the distilled attention layers, such as zero-shot generalization cross various models and plugins, and improved support for multi-GPU parallel inference. Models and codes are available here: this https URL.
• 摘要：扩散变换器 (DiT) 已成为图像生成领域的领先架构。然而，负责对 token 关系进行建模的注意力机制的二次复杂度会导致生成高分辨率图像时出现显著的延迟。为了解决这个问题，我们在本文中提出了一种线性注意力机制，将预训练 DiT 的复杂度降低到线性。我们从对现有有效注意力机制的全面总结开始探索，并确定了成功线性化预训练 DiT 的四个关键因素：局部性、公式一致性、高阶注意力图和特征完整性。基于这些见解，我们引入了一种类似卷积的局部注意力策略，称为 CLEAR，它将特征交互限制在每个查询 token 周围的局部窗口中，从而实现线性复杂度。我们的实验表明，通过仅在 10K 个自生成样本上对注意力层进行 10K 次迭代微调，我们可以有效地将知识从预训练的 DiT 转移到具有线性复杂度的学生模型，产生与教师模型相当的结果。同时，它可将注意力计算量减少 99.5%，并将生成速度提高 6.3 倍，以生成 8K 分辨率图像。此外，我们还研究了蒸馏注意力层中的有利属性，例如跨各种模型和插件的零样本泛化，以及对多 GPU 并行推理的改进支持。模型和代码可在此处获取：此 https URL。

Title: MotiF: Making Text Count in Image Animation with Motion Focal Loss

• Authors: Shijie Wang, Samaneh Azadi, Rohit Girdhar, Saketh Rambhatla, Chen Sun, Xi Yin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.16153
• Pdf URL: https://arxiv.org/pdf/2412.16153
• Copy Paste: [[2412.16153]] MotiF: Making Text Count in Image Animation with Motion Focal Loss(https://arxiv.org/abs/2412.16153)
• Keywords: generation
• Abstract: Text-Image-to-Video (TI2V) generation aims to generate a video from an image following a text description, which is also referred to as text-guided image animation. Most existing methods struggle to generate videos that align well with the text prompts, particularly when motion is specified. To overcome this limitation, we introduce MotiF, a simple yet effective approach that directs the model's learning to the regions with more motion, thereby improving the text alignment and motion generation. We use optical flow to generate a motion heatmap and weight the loss according to the intensity of the motion. This modified objective leads to noticeable improvements and complements existing methods that utilize motion priors as model inputs. Additionally, due to the lack of a diverse benchmark for evaluating TI2V generation, we propose TI2V Bench, a dataset consists of 320 image-text pairs for robust evaluation. We present a human evaluation protocol that asks the annotators to select an overall preference between two videos followed by their justifications. Through a comprehensive evaluation on TI2V Bench, MotiF outperforms nine open-sourced models, achieving an average preference of 72%. The TI2V Bench is released in this https URL.
• 摘要：文本图像到视频 (TI2V) 生成旨在根据文本描述从图像生成视频，这也称为文本引导的图像动画。大多数现有方法都难以生成与文本提示很好地一致的视频，尤其是在指定运动时。为了克服这一限制，我们引入了 MotiF，这是一种简单而有效的方法，它将模型的学习引导到运动较多的区域，从而改善文本对齐和运动生成。我们使用光流生成运动热图并根据运动强度加权损失。这个修改后的目标带来了显着的改进，并补充了利用运动先验作为模型输入的现有方法。此外，由于缺乏用于评估 TI2V 生成的多样化基准，我们提出了 TI2V Bench，这是一个由 320 个图像文本对组成的数据集，用于进行稳健评估。我们提出了一种人工评估协议，要求注释者在两个视频之间选择一个总体偏好，然后给出理由。经过对 TI2V Bench 的综合评估，MotiF 的表现优于九个开源模型，平均偏好度达到 72%。TI2V Bench 发布在此 https URL 中。

Title: Can Generative Video Models Help Pose Estimation?

• Authors: Ruojin Cai, Jason Y. Zhang, Philipp Henzler, Zhengqi Li, Noah Snavely, Ricardo Martin-Brualla
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.16155
• Pdf URL: https://arxiv.org/pdf/2412.16155
• Copy Paste: [[2412.16155]] Can Generative Video Models Help Pose Estimation?(https://arxiv.org/abs/2412.16155)
• Keywords: generative
• Abstract: Pairwise pose estimation from images with little or no overlap is an open challenge in computer vision. Existing methods, even those trained on large-scale datasets, struggle in these scenarios due to the lack of identifiable correspondences or visual overlap. Inspired by the human ability to infer spatial relationships from diverse scenes, we propose a novel approach, InterPose, that leverages the rich priors encoded within pre-trained generative video models. We propose to use a video model to hallucinate intermediate frames between two input images, effectively creating a dense, visual transition, which significantly simplifies the problem of pose estimation. Since current video models can still produce implausible motion or inconsistent geometry, we introduce a self-consistency score that evaluates the consistency of pose predictions from sampled videos. We demonstrate that our approach generalizes among three state-of-the-art video models and show consistent improvements over the state-of-the-art DUSt3R on four diverse datasets encompassing indoor, outdoor, and object-centric scenes. Our findings suggest a promising avenue for improving pose estimation models by leveraging large generative models trained on vast amounts of video data, which is more readily available than 3D data. See our project page for results: this https URL.
• 摘要：从重叠很少或没有重叠的图像中进行成对姿势估计是计算机视觉领域的一项开放性挑战。现有的方法，即使是那些在大型数据集上训练的方法，在这些情况下也会因为缺乏可识别的对应关系或视觉重叠而陷入困境。受人类从不同场景推断空间关系的能力的启发，我们提出了一种新方法 InterPose，该方法利用了预先训练的生成视频模型中编码的丰富先验。我们建议使用视频模型来幻化两个输入图像之间的中间帧，有效地创建密集的视觉过渡，从而大大简化姿势估计问题。由于当前的视频模型仍然会产生不合理的运动或不一致的几何形状，我们引入了一个自洽性分数来评估从采样视频中得出的姿势预测的一致性。我们证明了我们的方法在三种最先进的视频模型中具有泛化性，并且在四个不同的数据集上显示出比最先进的 DUSt3R 一致的改进，这些数据集涵盖了室内、室外和以物体为中心的场景。我们的研究结果表明，通过利用在大量视频数据上训练的大型生成模型，可以改善姿势估计模型，而这些视频数据比 3D 数据更容易获得。请参阅我们的项目页面了解结果：此 https URL。

Title: Personalized Representation from Personalized Generation

• Authors: Shobhita Sundaram, Julia Chae, Yonglong Tian, Sara Beery, Phillip Isola
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.16156
• Pdf URL: https://arxiv.org/pdf/2412.16156
• Copy Paste: [[2412.16156]] Personalized Representation from Personalized Generation(https://arxiv.org/abs/2412.16156)
• Keywords: generation
• Abstract: Modern vision models excel at general purpose downstream tasks. It is unclear, however, how they may be used for personalized vision tasks, which are both fine-grained and data-scarce. Recent works have successfully applied synthetic data to general-purpose representation learning, while advances in T2I diffusion models have enabled the generation of personalized images from just a few real examples. Here, we explore a potential connection between these ideas, and formalize the challenge of using personalized synthetic data to learn personalized representations, which encode knowledge about an object of interest and may be flexibly applied to any downstream task relating to the target object. We introduce an evaluation suite for this challenge, including reformulations of two existing datasets and a novel dataset explicitly constructed for this purpose, and propose a contrastive learning approach that makes creative use of image generators. We show that our method improves personalized representation learning for diverse downstream tasks, from recognition to segmentation, and analyze characteristics of image generation approaches that are key to this gain.
• 摘要：现代视觉模型擅长通用下游任务。然而，目前尚不清楚如何将它们用于个性化视觉任务，这些任务既细粒度又数据稀缺。最近的研究已成功地将合成数据应用于通用表示学习，而 T2I 扩散模型的进步使得仅从几个真实示例中生成个性化图像成为可能。在这里，我们探索了这些想法之间的潜在联系，并将使用个性化合成数据学习个性化表示的挑战形式化，这些表示编码了有关感兴趣对象的知识，可以灵活地应用于与目标对象相关的任何下游任务。我们为这一挑战引入了一个评估套件，包括两个现有数据集的重新表述和一个为此目的专门构建的新数据集，并提出了一种创造性地使用图像生成器的对比学习方法。我们表明，我们的方法改进了从识别到分割等各种下游任务的个性化表示学习，并分析了实现这一目标的关键图像生成方法的特征。