2024-12-11

Title: Enhancing LLMs for Physics Problem-Solving using Reinforcement Learning with Human-AI Feedback

• Authors: Avinash Anand, Kritarth Prasad, Chhavi Kirtani, Ashwin R Nair, Mohit Gupta, Saloni Garg, Anurag Gautam, Snehal Buldeo, Rajiv Ratn Shah
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06827
• Pdf URL: https://arxiv.org/pdf/2412.06827
• Copy Paste: [[2412.06827]] Enhancing LLMs for Physics Problem-Solving using Reinforcement Learning with Human-AI Feedback(https://arxiv.org/abs/2412.06827)
• Keywords: generation
• Abstract: Large Language Models (LLMs) have demonstrated strong capabilities in text-based tasks but struggle with the complex reasoning required for physics problems, particularly in advanced arithmetic and conceptual understanding. While some research has explored ways to enhance LLMs in physics education using techniques such as prompt engineering and Retrieval Augmentation Generation (RAG), not enough effort has been made in addressing their limitations in physics reasoning. This paper presents a novel approach to improving LLM performance on physics questions using Reinforcement Learning with Human and Artificial Intelligence Feedback (RLHAIF). We evaluate several reinforcement learning methods, including Proximal Policy Optimization (PPO), Direct Preference Optimization (DPO), and Remax optimization. These methods are chosen to investigate RL policy performance with different settings on the PhyQA dataset, which includes challenging physics problems from high school textbooks. Our RLHAIF model, tested on leading LLMs like LLaMA2 and Mistral, achieved superior results, notably with the MISTRAL-PPO model, demonstrating marked improvements in reasoning and accuracy. It achieved high scores, with a 58.67 METEOR score and a 0.74 Reasoning score, making it a strong example for future physics reasoning research in this area.
• 摘要：大型语言模型 (LLM) 在基于文本的任务中表现出强大的能力，但在物理问题所需的复杂推理方面却举步维艰，尤其是在高级算术和概念理解方面。虽然一些研究已经探索了使用诸如快速工程和检索增强生成 (RAG) 等技术来增强 LLM 在物理教育中的表现的方法，但在解决其在物理推理方面的局限性方面所做的努力还不够。本文介绍了一种使用带有人类和人工智能反馈的强化学习 (RLHAIF) 来提高 LLM 在物理问题上的表现的新方法。我们评估了几种强化学习方法，包括近端策略优化 (PPO)、直接偏好优化 (DPO) 和 Remax 优化。选择这些方法来研究 PhyQA 数据集上具有不同设置的 RL 策略性能，其中包括高中教科书中的具有挑战性的物理问题。我们的 RLHAIF 模型在 LLaMA2 和 Mistral 等领先的 LLM 上进行了测试，取得了优异的结果，尤其是 MISTRAL-PPO 模型，在推理和准确性方面表现出显着的改进。它取得了高分，METEOR 得分为 58.67，推理得分为 0.74，为未来该领域物理推理研究树立了典范。

Title: GL-Fusion: Rethinking the Combination of Graph Neural Network and Large Language model

• Authors: Haotong Yang, Xiyuan Wang, Qian Tao, Shuxian Hu, Zhouchen Lin, Muhan Zhang
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.06849
• Pdf URL: https://arxiv.org/pdf/2412.06849
• Copy Paste: [[2412.06849]] GL-Fusion: Rethinking the Combination of Graph Neural Network and Large Language model(https://arxiv.org/abs/2412.06849)
• Keywords: generation
• Abstract: Recent research on integrating Large Language Models (LLMs) with Graph Neural Networks (GNNs) typically follows two approaches: LLM-centered models, which convert graph data into tokens for LLM processing, and GNN-centered models, which use LLMs to encode text features into node and edge representations for GNN input. LLM-centered models often struggle to capture graph structures effectively, while GNN-centered models compress variable-length textual data into fixed-size vectors, limiting their ability to understand complex semantics. Additionally, GNN-centered approaches require converting tasks into a uniform, manually-designed format, restricting them to classification tasks and preventing language output. To address these limitations, we introduce a new architecture that deeply integrates GNN with LLM, featuring three key innovations: (1) Structure-Aware Transformers, which incorporate GNN's message-passing capabilities directly into LLM's transformer layers, allowing simultaneous processing of textual and structural information and generating outputs from both GNN and LLM; (2) Graph-Text Cross-Attention, which processes full, uncompressed text from graph nodes and edges, ensuring complete semantic integration; and (3) GNN-LLM Twin Predictor, enabling LLM's flexible autoregressive generation alongside GNN's scalable one-pass prediction. GL-Fusion achieves outstand performance on various tasks. Notably, it achieves state-of-the-art performance on OGBN-Arxiv and OGBG-Code2.
• 摘要：近期关于将大型语言模型 (LLM) 与图神经网络 (GNN) 相结合的研究通常遵循两种方法：以 LLM 为中心的模型，将图数据转换为标记以供 LLM 处理；以 GNN 为中心的模型，使用 LLM 将文本特征编码为节点和边表示以供 GNN 输入。以 LLM 为中心的模型通常难以有效地捕获图结构，而以 GNN 为中心的模型将可变长度的文本数据压缩为固定大小的向量，从而限制了它们理解复杂语义的能力。此外，以 GNN 为中心的方法需要将任务转换为统一的、手动设计的格式，将其限制为分类任务并阻止语言输出。为了解决这些限制，我们引入了一种将 GNN 与 LLM 深度集成的新架构，具有三大关键创新：(1) 结构感知 Transformers，它将 GNN 的消息传递功能直接整合到 LLM 的 Transformer 层中，允许同时处理文本和结构信息并从 GNN 和 LLM 生成输出； (2) Graph-Text Cross-Attention，处理来自图节点和边缘的完整、未压缩的文本，确保完整的语义集成；(3) GNN-LLM Twin Predictor，实现 LLM 灵活的自回归生成以及 GNN 可扩展的一次性预测。GL-Fusion 在各种任务上都取得了出色的表现。值得注意的是，它在 OGBN-Arxiv 和 OGBG-Code2 上实现了最先进的性能。

Title: Comb Tensor Networks vs. Matrix Product States: Enhanced Efficiency in High-Dimensional Spaces

• Authors: Danylo Kolesnyk, Yelyzaveta Vodovozova
• Subjects: cs.LG, quant-ph
• Abstract URL: https://arxiv.org/abs/2412.06857
• Pdf URL: https://arxiv.org/pdf/2412.06857
• Copy Paste: [[2412.06857]] Comb Tensor Networks vs. Matrix Product States: Enhanced Efficiency in High-Dimensional Spaces(https://arxiv.org/abs/2412.06857)
• Keywords: generative
• Abstract: Modern approaches to generative modeling of continuous data using tensor networks incorporate compression layers to capture the most meaningful features of high-dimensional inputs. These methods, however, rely on traditional Matrix Product States (MPS) architectures. Here, we demonstrate that beyond a certain threshold in data and bond dimensions, a comb-shaped tensor network architecture can yield more efficient contractions than a standard MPS. This finding suggests that for continuous and high-dimensional data distributions, transitioning from MPS to a comb tensor network representation can substantially reduce computational overhead while maintaining accuracy.
• 摘要：使用张量网络对连续数据进行生成建模的现代方法结合了压缩层，以捕获高维输入中最有意义的特征。然而，这些方法依赖于传统的矩阵乘积状态 (MPS) 架构。在这里，我们证明，在数据和键维度超过某个阈值时，梳状张量网络架构可以比标准 MPS 产生更有效的收缩。这一发现表明，对于连续和高维数据分布，从 MPS 过渡到梳状张量网络表示可以大大减少计算开销，同时保持准确性。

Title: Generating floorplans for various building functionalities via latent diffusion model

• Authors: Mohamed R. Ibrahim, Josef Musil, Irene Gallou
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06859
• Pdf URL: https://arxiv.org/pdf/2412.06859
• Copy Paste: [[2412.06859]] Generating floorplans for various building functionalities via latent diffusion model(https://arxiv.org/abs/2412.06859)
• Keywords: generative
• Abstract: In the domain of architectural design, the foundational essence of creativity and human intelligence lies in the mastery of solving floorplans, a skill demanding distinctive expertise and years of experience. Traditionally, the architectural design process of creating floorplans often requires substantial manual labour and architectural expertise. Even when relying on parametric design approaches, the process is limited based on the designer's ability to build a complex set of parameters to iteratively explore design alternatives. As a result, these approaches hinder creativity and limit discovery of an optimal solution. Here, we present a generative latent diffusion model that learns to generate floorplans for various building types based on building footprints and design briefs. The introduced model learns from the complexity of the inter-connections between diverse building types and the mutations of architectural designs. By harnessing the power of latent diffusion models, this research surpasses conventional limitations in the design process. The model's ability to learn from diverse building types means that it cannot only replicate existing designs but also produce entirely new configurations that fuse design elements in unexpected ways. This innovation introduces a new dimension of creativity into architectural design, allowing architects, urban planners and even individuals without specialised expertise to explore uncharted territories of form and function with speed and cost-effectiveness.
• 摘要：在建筑设计领域，创造力和人类智慧的根本本质在于掌握解决平面图的技能，这是一项需要独特专业知识和多年经验的技能。传统上，创建平面图的建筑设计过程通常需要大量的体力劳动和建筑专业知识。即使依靠参数化设计方法，该过程也受到设计师构建一组复杂参数以迭代探索设计替代方案的能力的限制。因此，这些方法阻碍了创造力并限制了最佳解决方案的发现。在这里，我们提出了一个生成潜在扩散模型，该模型学习根据建筑足迹和设计简介为各种建筑类型生成平面图。引入的模型从不同建筑类型之间的相互联系的复杂性和建筑设计的变异中学习。通过利用潜在扩散模型的力量，这项研究超越了设计过程中的传统限制。该模型从不同建筑类型中学习的能力意味着它不仅可以复制现有设计，还可以产生全新的配置，以意想不到的方式融合设计元素。这项创新为建筑设计带来了新的创意维度，使建筑师、城市规划师甚至没有专业知识的个人能够快速且经济高效地探索形式和功能的未知领域。

Title: SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanations

• Authors: Zhaorun Chen, Francesco Pinto, Minzhou Pan, Bo Li
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.06878
• Pdf URL: https://arxiv.org/pdf/2412.06878
• Copy Paste: [[2412.06878]] SafeWatch: An Efficient Safety-Policy Following Video Guardrail Model with Transparent Explanations(https://arxiv.org/abs/2412.06878)
• Keywords: generation, generative
• Abstract: With the rise of generative AI and rapid growth of high-quality video generation, video guardrails have become more crucial than ever to ensure safety and security across platforms. Current video guardrails, however, are either overly simplistic, relying on pure classification models trained on simple policies with limited unsafe categories, which lack detailed explanations, or prompting multimodal large language models (MLLMs) with long safety guidelines, which are inefficient and impractical for guardrailing real-world content. To bridge this gap, we propose SafeWatch, an efficient MLLM-based video guardrail model designed to follow customized safety policies and provide multi-label video guardrail outputs with content-specific explanations in a zero-shot manner. In particular, unlike traditional MLLM-based guardrails that encode all safety policies autoregressively, causing inefficiency and bias, SafeWatch uniquely encodes each policy chunk in parallel and eliminates their position bias such that all policies are attended simultaneously with equal importance. In addition, to improve efficiency and accuracy, SafeWatch incorporates a policy-aware visual token pruning algorithm that adaptively selects the most relevant video tokens for each policy, discarding noisy or irrelevant information. This allows for more focused, policy-compliant guardrail with significantly reduced computational overhead. Considering the limitations of existing video guardrail benchmarks, we propose SafeWatch-Bench, a large-scale video guardrail benchmark comprising over 2M videos spanning six safety categories which covers over 30 tasks to ensure a comprehensive coverage of all potential safety scenarios. SafeWatch outperforms SOTA by 28.2% on SafeWatch-Bench, 13.6% on benchmarks, cuts costs by 10%, and delivers top-tier explanations validated by LLM and human reviews.
• 摘要：随着生成式人工智能的兴起和高质量视频生成的快速增长，视频护栏对于确保跨平台的安全比以往任何时候都更加重要。然而，当前的视频护栏要么过于简单，依赖于在具有有限不安全类别的简单策略上训练的纯分类模型，缺乏详细的解释，要么促使使用具有冗长安全指南的多模态大型语言模型 (MLLM)，这对于保护现实世界的内容既低效又不切实际。为了弥补这一差距，我们提出了 SafeWatch，这是一种基于 MLLM 的高效视频护栏模型，旨在遵循定制的安全策略并以零样本方式提供具有内容特定解释的多标签视频护栏输出。特别是，与传统的基于 MLLM 的护栏不同，传统的基于 MLLM 的护栏会对所有安全策略进行自回归编码，从而导致效率低下和偏差，而 SafeWatch 会以独特的方式并行编码每个策略块并消除它们的位置偏差，从而让所有策略都以同等重要性同时得到关注。此外，为了提高效率和准确性，SafeWatch 采用了策略感知的视觉标记修剪算法，该算法可以自适应地为每项策略选择最相关的视频标记，丢弃嘈杂或不相关的信息。这样可以实现更集中、更符合策略的护栏，同时显著降低计算开销。考虑到现有视频护栏基准的局限性，我们提出了 SafeWatch-Bench，这是一个大规模视频护栏基准，包含超过 200 万个视频，涵盖六个安全类别，涵盖 30 多项任务，以确保全面覆盖所有潜在的安全场景。SafeWatch 在 SafeWatch-Bench 上的表现比 SOTA 高出 28.2%，在基准上高出 13.6%，成本降低了 10%，并提供了经过 LLM 和人工审核验证的顶级解释。

Title: Enhancing operational wind downscaling capabilities over Canada: Application of a Conditional Wasserstein GAN methodology

• Authors: Jorge Guevara, Victor Nascimento, Johannes Schmude, Daniel Salles, Simon Corbeil-Létourneau, Madalina Surcel, Dominique Brunet
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.06958
• Pdf URL: https://arxiv.org/pdf/2412.06958
• Copy Paste: [[2412.06958]] Enhancing operational wind downscaling capabilities over Canada: Application of a Conditional Wasserstein GAN methodology(https://arxiv.org/abs/2412.06958)
• Keywords: generative
• Abstract: Wind downscaling is essential for improving the spatial resolution of weather forecasts, particularly in operational Numerical Weather Prediction (NWP). This study advances wind downscaling by extending the DownGAN framework introduced by Annau et al.,to operational datasets from the Global Deterministic Prediction System (GDPS) and High-Resolution Deterministic Prediction System (HRDPS), covering the entire Canadian domain. We enhance the model by incorporating high-resolution static covariates, such as HRDPS-derived topography, into a Conditional Wasserstein Generative Adversarial Network with Gradient Penalty, implemented using a UNET-based generator. Following the DownGAN framework, our methodology integrates low-resolution GDPS forecasts (15 km, 10-day horizon) and high-resolution HRDPS forecasts (2.5 km, 48-hour horizon) with Frequency Separation techniques adapted from computer vision. Through robust training and inference over the Canadian region, we demonstrate the operational scalability of our approach, achieving significant improvements in wind downscaling accuracy. Statistical validation highlights reductions in root mean square error (RMSE) and log spectral distance (LSD) metrics compared to the original DownGAN. High-resolution conditioning covariates and Frequency Separation strategies prove instrumental in enhancing model performance. This work underscores the potential for extending high-resolution wind forecasts beyond the 48-hour horizon, bridging the gap to the 10-day low resolution global forecast window.
• 摘要：风降尺度对于提高天气预报的空间分辨率至关重要，特别是在业务数值天气预报 (NWP) 中。本研究通过将 Annau 等人引入的 DownGAN 框架扩展到全球确定性预测系统 (GDPS) 和高分辨率确定性预测系统 (HRDPS) 的业务数据集来推进风降尺度，覆盖整个加拿大领域。我们通过将高分辨率静态协变量（例如 HRDPS 衍生的地形）合并到使用基于 UNET 的生成器实现的具有梯度惩罚的条件 Wasserstein 生成对抗网络中来增强模型。遵循 DownGAN 框架，我们的方法将低分辨率 GDPS 预测（15 公里，10 天视野）和高分辨率 HRDPS 预测（2.5 公里，48 小时视野）与改编自计算机视觉的频率分离技术相结合。通过在加拿大地区进行强大的训练和推理，我们展示了我们的方法的操作可扩展性，显著提高了风速降尺度精度。统计验证突出了与原始 DownGAN 相比，均方根误差 (RMSE) 和对数谱距离 (LSD) 指标的降低。高分辨率条件协变量和频率分离策略被证明有助于提高模型性能。这项工作强调了将高分辨率风速预报扩展到 48 小时范围之外的潜力，从而弥补了 10 天低分辨率全球预报窗口的差距。

Title: Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy, Research, and Practice

• Authors: A. Feder Cooper, Christopher A. Choquette-Choo, Miranda Bogen, Matthew Jagielski, Katja Filippova, Ken Ziyu Liu, Alexandra Chouldechova, Jamie Hayes, Yangsibo Huang, Niloofar Mireshghallah, Ilia Shumailov, Eleni Triantafillou, Peter Kairouz, Nicole Mitchell, Percy Liang, Daniel E. Ho, Yejin Choi, Sanmi Koyejo, Fernando Delgado, James Grimmelmann, Vitaly Shmatikov, Christopher De Sa, Solon Barocas, Amy Cyphert, Mark Lemley, danah boyd, Jennifer Wortman Vaughan, Miles Brundage, David Bau, Seth Neel, Abigail Z. Jacobs, Andreas Terzis, Hanna Wallach, Nicolas Papernot, Katherine Lee
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2412.06966
• Pdf URL: https://arxiv.org/pdf/2412.06966
• Copy Paste: [[2412.06966]] Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy, Research, and Practice(https://arxiv.org/abs/2412.06966)
• Keywords: generation, generative
• Abstract: We articulate fundamental mismatches between technical methods for machine unlearning in Generative AI, and documented aspirations for broader impact that these methods could have for law and policy. These aspirations are both numerous and varied, motivated by issues that pertain to privacy, copyright, safety, and more. For example, unlearning is often invoked as a solution for removing the effects of targeted information from a generative-AI model's parameters, e.g., a particular individual's personal data or in-copyright expression of Spiderman that was included in the model's training data. Unlearning is also proposed as a way to prevent a model from generating targeted types of information in its outputs, e.g., generations that closely resemble a particular individual's data or reflect the concept of "Spiderman." Both of these goals--the targeted removal of information from a model and the targeted suppression of information from a model's outputs--present various technical and substantive challenges. We provide a framework for thinking rigorously about these challenges, which enables us to be clear about why unlearning is not a general-purpose solution for circumscribing generative-AI model behavior in service of broader positive impact. We aim for conceptual clarity and to encourage more thoughtful communication among machine learning (ML), law, and policy experts who seek to develop and apply technical methods for compliance with policy objectives.
• 摘要：我们阐明了生成式人工智能中机器反学习的技术方法与这些方法可能对法律和政策产生更广泛影响的记录愿望之间的根本性不匹配。这些愿望既多种多样，又各不相同，其动机是与隐私、版权、安全等有关的问题。例如，反学习通常被用作从生成式人工智能模型的参数中消除目标信息影响的解决方案，例如，特定个人的个人数据或包含在模型训练数据中的蜘蛛侠的版权表达。反学习还被提议作为一种防止模型在其输出中生成目标类型信息的方法，例如，与特定个人的数据非常相似或反映“蜘蛛侠”概念的生成。这两个目标——有针对性地从模型中删除信息和有针对性地从模型的输出中抑制信息——都带来了各种技术和实质性挑战。我们提供了一个框架来严格思考这些挑战，这使我们能够清楚地了解为什么“忘却学习”不是限制生成式人工智能模型行为以产生更广泛的积极影响的通用解决方案。我们的目标是概念清晰，并鼓励机器学习 (ML)、法律和政策专家之间进行更周到的沟通，这些专家寻求开发和应用技术方法来遵守政策目标。

Title: Edge-SD-SR: Low Latency and Parameter Efficient On-device Super-Resolution with Stable Diffusion via Bidirectional Conditioning

• Authors: Mehdi Noroozi, Isma Hadji, Victor Escorcia, Anestis Zaganidis, Brais Martinez, Georgios Tzimiropoulos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.06978
• Pdf URL: https://arxiv.org/pdf/2412.06978
• Copy Paste: [[2412.06978]] Edge-SD-SR: Low Latency and Parameter Efficient On-device Super-Resolution with Stable Diffusion via Bidirectional Conditioning(https://arxiv.org/abs/2412.06978)
• Keywords: super-resolution
• Abstract: There has been immense progress recently in the visual quality of Stable Diffusion-based Super Resolution (SD-SR). However, deploying large diffusion models on computationally restricted devices such as mobile phones remains impractical due to the large model size and high latency. This is compounded for SR as it often operates at high res (e.g. 4Kx3K). In this work, we introduce Edge-SD-SR, the first parameter efficient and low latency diffusion model for image super-resolution. Edge-SD-SR consists of ~169M parameters, including UNet, encoder and decoder, and has a complexity of only ~142 GFLOPs. To maintain a high visual quality on such low compute budget, we introduce a number of training strategies: (i) A novel conditioning mechanism on the low resolution input, coined bidirectional conditioning, which tailors the SD model for the SR task. (ii) Joint training of the UNet and encoder, while decoupling the encodings of the HR and LR images and using a dedicated schedule. (iii) Finetuning the decoder using the UNet's output to directly tailor the decoder to the latents obtained at inference time. Edge-SD-SR runs efficiently on device, e.g. it can upscale a 128x128 patch to 512x512 in 38 msec while running on a Samsung S24 DSP, and of a 512x512 to 2048x2048 (requiring 25 model evaluations) in just ~1.1 sec. Furthermore, we show that Edge-SD-SR matches or even outperforms state-of-the-art SR approaches on the most established SR benchmarks.
• 摘要：最近，基于稳定扩散的超分辨率 (SD-SR) 的视觉质量取得了巨大进步。然而，由于模型尺寸大且延迟高，在计算受限的设备（如移动电话）上部署大型扩散模型仍然不切实际。这对于 SR 来说更加复杂，因为它通常以高分辨率（例如 4Kx3K）运行。在这项工作中，我们引入了 Edge-SD-SR，这是第一个用于图像超分辨率的参数高效且低延迟的扩散模型。Edge-SD-SR 包含约 169M 个参数，包括 UNet、编码器和解码器，复杂度仅为约 142 GFLOP。为了在如此低的计算预算下保持高视觉质量，我们引入了许多训练策略：(i) 一种针对低分辨率输入的新型调节机制，称为双向调节，它为 SR 任务定制 SD 模型。(ii) UNet 和编码器的联合训练，同时解耦 HR 和 LR 图像的编码并使用专用时间表。 (iii) 使用 UNet 的输出对解码器进行微调，以直接根据推理时获得的潜在信息定制解码器。Edge-SD-SR 在设备上高效运行，例如，在 Samsung S24 DSP 上运行时，它可以在 38 毫秒内将 128x128 补丁升级到 512x512，并在约 1.1 秒内将 512x512 升级到 2048x2048（需要 25 次模型评估）。此外，我们表明 Edge-SD-SR 在最成熟的 SR 基准测试中与最先进的 SR 方法相媲美甚至优于最先进的 SR 方法。

Title: TAE: A Model-Constrained Tikhonov Autoencoder Approach for Forward and Inverse Problems

• Authors: Hai V. Nguyen, Tan Bui-Thanh
• Subjects: cs.LG, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2412.07010
• Pdf URL: https://arxiv.org/pdf/2412.07010
• Copy Paste: [[2412.07010]] TAE: A Model-Constrained Tikhonov Autoencoder Approach for Forward and Inverse Problems(https://arxiv.org/abs/2412.07010)
• Keywords: generative
• Abstract: Efficient real-time solvers for forward and inverse problems are essential in engineering and science applications. Machine learning surrogate models have emerged as promising alternatives to traditional methods, offering substantially reduced computational time. Nevertheless, these models typically demand extensive training datasets to achieve robust generalization across diverse scenarios. While physics-based approaches can partially mitigate this data dependency and ensure physics-interpretable solutions, addressing scarce data regimes remains a challenge. Both purely data-driven and physics-based machine learning approaches demonstrate severe overfitting issues when trained with insufficient data. We propose a novel Tikhonov autoencoder model-constrained framework, called TAE, capable of learning both forward and inverse surrogate models using a single arbitrary observation sample. We develop comprehensive theoretical foundations including forward and inverse inference error bounds for the proposed approach for linear cases. For comparative analysis, we derive equivalent formulations for pure data-driven and model-constrained approach counterparts. At the heart of our approach is a data randomization strategy, which functions as a generative mechanism for exploring the training data space, enabling effective training of both forward and inverse surrogate models from a single observation, while regularizing the learning process. We validate our approach through extensive numerical experiments on two challenging inverse problems: 2D heat conductivity inversion and initial condition reconstruction for time-dependent 2D Navier-Stokes equations. Results demonstrate that TAE achieves accuracy comparable to traditional Tikhonov solvers and numerical forward solvers for both inverse and forward problems, respectively, while delivering orders of magnitude computational speedups.
• 摘要：在工程和科学应用中，高效的正向和逆向问题实时求解器至关重要。机器学习替代模型已成为传统方法的有前途的替代方案，大大缩短了计算时间。然而，这些模型通常需要大量的训练数据集才能在不同场景中实现稳健的泛化。虽然基于物理的方法可以部分缓解这种数据依赖性并确保物理可解释的解决方案，但解决稀缺数据机制仍然是一个挑战。纯数据驱动和基于物理的机器学习方法在数据不足的情况下训练时都会出现严重的过度拟合问题。我们提出了一种新颖的 Tikhonov 自动编码器模型约束框架，称为 TAE，它能够使用单个任意观察样本学习正向和逆向替代模型。我们为所提出的方法开发了全面的理论基础，包括线性情况下的正向和逆向推理误差界限。为了进行比较分析，我们推导出纯数据驱动和模型约束方法的等效公式。我们方法的核心是数据随机化策略，它充当探索训练数据空间的生成机制，能够从单个观察中有效训练正向和逆向替代模型，同时规范学习过程。我们通过对两个具有挑战性的逆问题进行大量数值实验来验证我们的方法：2D 热导率反演和时间相关 2D Navier-Stokes 方程的初始条件重建。结果表明，TAE 分别在逆问题和正问题上实现了与传统 Tikhonov 求解器和数值正向求解器相当的精度，同时实现了数量级的计算速度提升。

Title: ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models

• Authors: Jieyu Zhang, Le Xue, Linxin Song, Jun Wang, Weikai Huang, Manli Shu, An Yan, Zixian Ma, Juan Carlos Niebles, silvio savarese, Caiming Xiong, Zeyuan Chen, Ranjay Krishna, Ran Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07012
• Pdf URL: https://arxiv.org/pdf/2412.07012
• Copy Paste: [[2412.07012]] ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models(https://arxiv.org/abs/2412.07012)
• Keywords: generation
• Abstract: With the rise of multimodal applications, instruction data has become critical for training multimodal language models capable of understanding complex image-based queries. Existing practices rely on powerful but costly large language models (LLMs) or multimodal language models (MLMs) to produce instruction data. These are often prone to hallucinations, licensing issues and the generation process is often hard to scale and interpret. In this work, we present a programmatic approach that employs scene graphs as symbolic representations of images and human-written programs to systematically synthesize vision-centric instruction data. Our approach ensures the interpretability and controllability of the data generation process and scales efficiently while maintaining factual accuracy. By implementing a suite of 24 single-image, 14 multi-image instruction generators, and a scene graph generation pipeline, we build a scalable, cost-effective system: ProVision which produces diverse question-answer pairs concerning objects, attributes, relations, depth, etc., for any given image. Applied to Visual Genome and DataComp datasets, we generate over 10 million instruction data points, ProVision-10M, and leverage them in both pretraining and instruction tuning stages of MLMs. When adopted in the instruction tuning stage, our single-image instruction data yields up to a 7% improvement on the 2D split and 8% on the 3D split of CVBench, along with a 3% increase in performance on QBench2, RealWorldQA, and MMMU. Our multi-image instruction data leads to an 8% improvement on Mantis-Eval. Incorporation of our data in both pre-training and fine-tuning stages of xGen-MM-4B leads to an averaged improvement of 1.6% across 11 benchmarks.
• 摘要：随着多模态应用的兴起，指令数据对于训练能够理解复杂图像查询的多模态语言模型至关重要。现有的实践依赖于功能强大但成本高昂的大型语言模型 (LLM) 或多模态语言模型 (MLM) 来生成指令数据。这些模型通常容易产生幻觉、许可问题，并且生成过程通常难以扩展和解释。在这项工作中，我们提出了一种编程方法，该方法使用场景图作为图像和人工编写的程序的符号表示来系统地合成以视觉为中心的指令数据。我们的方法确保了数据生成过程的可解释性和可控性，并在保持事实准确性的同时有效扩展。通过实施一套 24 个单图像、14 个多图像指令生成器和一个场景图生成管道，我们构建了一个可扩展、经济高效的系统：ProVision，它可以为任何给定的图像生成有关对象、属性、关系、深度等的各种问答对。应用于 Visual Genome 和 DataComp 数据集后，我们生成了超过 1000 万个指令数据点 ProVision-10M，并在 MLM 的预训练和指令调整阶段中利用它们。在指令调整阶段采用时，我们的单图像指令数据在 CVBench 的 2D 分割上可提高 7%，在 3D 分割上可提高 8%，同时在 QBench2、RealWorldQA 和 MMMU 上的性能可提高 3%。我们的多图像指令数据在 Mantis-Eval 上可提高 8%。在 xGen-MM-4B 的预训练和微调阶段结合我们的数据，可在 11 个基准测试中平均提高 1.6%。

Title: GenAI4UQ: A Software for Inverse Uncertainty Quantification Using Conditional Generative Models

• Authors: Ming Fan, Zezhong Zhang, Dan Lu, Guannan Zhang
• Subjects: cs.LG, physics.geo-ph
• Abstract URL: https://arxiv.org/abs/2412.07026
• Pdf URL: https://arxiv.org/pdf/2412.07026
• Copy Paste: [[2412.07026]] GenAI4UQ: A Software for Inverse Uncertainty Quantification Using Conditional Generative Models(https://arxiv.org/abs/2412.07026)
• Keywords: generation, generative
• Abstract: We introduce GenAI4UQ, a software package for inverse uncertainty quantification in model calibration, parameter estimation, and ensemble forecasting in scientific applications. GenAI4UQ leverages a generative artificial intelligence (AI) based conditional modeling framework to address the limitations of traditional inverse modeling techniques, such as Markov Chain Monte Carlo methods. By replacing computationally intensive iterative processes with a direct, learned mapping, GenAI4UQ enables efficient calibration of model input parameters and generation of output predictions directly from observations. The software's design allows for rapid ensemble forecasting with robust uncertainty quantification, while maintaining high computational and storage efficiency. GenAI4UQ simplifies the model training process through built-in auto-tuning of hyperparameters, making it accessible to users with varying levels of expertise. Its conditional generative framework ensures versatility, enabling applicability across a wide range of scientific domains. At its core, GenAI4UQ transforms the paradigm of inverse modeling by providing a fast, reliable, and user-friendly solution. It empowers researchers and practitioners to quickly estimate parameter distributions and generate model predictions for new observations, facilitating efficient decision-making and advancing the state of uncertainty quantification in computational modeling. (The code and data are available at this https URL).
• 摘要：我们推出了 GenAI4UQ，这是一个用于科学应用中模型校准、参数估计和集合预报中的逆不确定性量化的软件包。GenAI4UQ 利用基于生成人工智能 (AI) 的条件建模框架来解决传统逆建模技术（如马尔可夫链蒙特卡罗方法）的局限性。通过用直接学习映射取代计算密集型迭代过程，GenAI4UQ 能够高效校准模型输入参数并直接从观测值生成输出预测。该软件的设计允许快速集合预报和稳健的不确定性量化，同时保持高计算和存储效率。GenAI4UQ 通过内置的超参数自动调整简化了模型训练过程，使不同专业水平的用户都可以使用它。它的条件生成框架确保了多功能性，使其适用于广泛的科学领域。从本质上讲，GenAI4UQ 通过提供快速、可靠且用户友好的解决方案改变了逆建模的范式。它使研究人员和从业人员能够快速估计参数分布并为新观测生成模型预测，从而促进高效决策并推进计算建模中不确定性量化的状态。（代码和数据可在此 https URL 上获得）。

Title: Data Augmentation with Variational Autoencoder for Imbalanced Dataset

• Authors: Samuel Stocksieker, Denys Pommeret, Arthur Charpentier
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07039
• Pdf URL: https://arxiv.org/pdf/2412.07039
• Copy Paste: [[2412.07039]] Data Augmentation with Variational Autoencoder for Imbalanced Dataset(https://arxiv.org/abs/2412.07039)
• Keywords: generation
• Abstract: Learning from an imbalanced distribution presents a major challenge in predictive modeling, as it generally leads to a reduction in the performance of standard algorithms. Various approaches exist to address this issue, but many of them concern classification problems, with a limited focus on regression. In this paper, we introduce a novel method aimed at enhancing learning on tabular data in the Imbalanced Regression (IR) framework, which remains a significant problem. We propose to use variational autoencoders (VAE) which are known as a powerful tool for synthetic data generation, offering an interesting approach to modeling and capturing latent representations of complex distributions. However, VAEs can be inefficient when dealing with IR. Therefore, we develop a novel approach for generating data, combining VAE with a smoothed bootstrap, specifically designed to address the challenges of IR. We numerically investigate the scope of this method by comparing it against its competitors on simulations and datasets known for IR.
• 摘要：从不平衡分布中学习是预测建模中的一大挑战，因为它通常会导致标准算法性能下降。存在各种方法来解决此问题，但其中许多方法涉及分类问题，而对回归的关注有限。在本文中，我们介绍了一种新方法，旨在增强不平衡回归 (IR) 框架中表格数据的学习，这仍然是一个重大问题。我们建议使用变分自动编码器 (VAE)，它被称为合成数据生成的强大工具，为建模和捕获复杂分布的潜在表示提供了一种有趣的方法。然而，VAE 在处理 IR 时效率低下。因此，我们开发了一种新的数据生成方法，将 VAE 与平滑引导程序相结合，专门用于解决 IR 的挑战。我们通过在模拟和已知的 IR 数据集上将其与竞争对手进行比较，从数字上研究了该方法的范围。

Title: Creative Portraiture: Exploring Creative Adversarial Networks and Conditional Creative Adversarial Networks

• Authors: Sebastian Hereu, Qianfei Hu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07091
• Pdf URL: https://arxiv.org/pdf/2412.07091
• Copy Paste: [[2412.07091]] Creative Portraiture: Exploring Creative Adversarial Networks and Conditional Creative Adversarial Networks(https://arxiv.org/abs/2412.07091)
• Keywords: generative
• Abstract: Convolutional neural networks (CNNs) have been combined with generative adversarial networks (GANs) to create deep convolutional generative adversarial networks (DCGANs) with great success. DCGANs have been used for generating images and videos from creative domains such as fashion design and painting. A common critique of the use of DCGANs in creative applications is that they are limited in their ability to generate creative products because the generator simply learns to copy the training distribution. We explore an extension of DCGANs, creative adversarial networks (CANs). Using CANs, we generate novel, creative portraits, using the WikiArt dataset to train the network. Moreover, we introduce our extension of CANs, conditional creative adversarial networks (CCANs), and demonstrate their potential to generate creative portraits conditioned on a style label. We argue that generating products that are conditioned, or inspired, on a style label closely emulates real creative processes in which humans produce imaginative work that is still rooted in previous styles.
• 摘要：卷积神经网络 (CNN) 与生成对抗网络 (GAN) 相结合，创建了深度卷积生成对抗网络 (DCGAN)，并取得了巨大成功。DCGAN 已用于生成时装设计和绘画等创意领域的图像和视频。对 DCGAN 在创意应用中的使用的一个常见批评是，它们在生成创意产品方面的能力有限，因为生成器只是学习复制训练分布。我们探索了 DCGAN 的扩展，即创意对抗网络 (CAN)。使用 CAN，我们生成新颖的创意肖像，并使用 WikiArt 数据集来训练网络。此外，我们介绍了 CAN 的扩展，即条件创意对抗网络 (CCAN)，并展示了它们生成以风格标签为条件的创意肖像的潜力。我们认为，生成以风格标签为条件或灵感的产品非常接近真实的创意过程，在这种过程中，人类会创作出仍然植根于先前风格的富有想象力的作品。

Title: TT-MPD: Test Time Model Pruning and Distillation

• Authors: Haihang Wu, Wei Wang, Tamasha Malepathirana, Sachith Seneviratne, Denny Oetomo, Saman Halgamuge
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07114
• Pdf URL: https://arxiv.org/pdf/2412.07114
• Copy Paste: [[2412.07114]] TT-MPD: Test Time Model Pruning and Distillation(https://arxiv.org/abs/2412.07114)
• Keywords: generation
• Abstract: Pruning can be an effective method of compressing large pre-trained models for inference speed acceleration. Previous pruning approaches rely on access to the original training dataset for both pruning and subsequent fine-tuning. However, access to the training data can be limited due to concerns such as data privacy and commercial confidentiality. Furthermore, with covariate shift (disparities between test and training data distributions), pruning and finetuning with training datasets can hinder the generalization of the pruned model to test data. To address these issues, pruning and finetuning the model with test time samples becomes essential. However, test-time model pruning and fine-tuning incur additional computation costs and slow down the model's prediction speed, thus posing efficiency issues. Existing pruning methods are not efficient enough for test time model pruning setting, since finetuning the pruned model is needed to evaluate the importance of removable components. To address this, we propose two variables to approximate the fine-tuned accuracy. We then introduce an efficient pruning method that considers the approximated finetuned accuracy and potential inference latency saving. To enhance fine-tuning efficiency, we propose an efficient knowledge distillation method that only needs to generate pseudo labels for a small set of finetuning samples one time, thereby reducing the expensive pseudo-label generation cost. Experimental results demonstrate that our method achieves a comparable or superior tradeoff between test accuracy and inference latency, with a 32% relative reduction in pruning and finetuning time compared to the best existing method.
• 摘要：修剪是一种压缩大型预训练模型以加速推理速度的有效方法。以前的修剪方法依赖于对原始训练数据集的访问，以进行修剪和随后的微调。然而，由于数据隐私和商业机密等问题，对训练数据的访问可能会受到限制。此外，由于协变量偏移（测试数据和训练数据分布之间的差异），使用训练数据集进行修剪和微调可能会阻碍修剪后的模型推广到测试数据。为了解决这些问题，使用测试时间样本对模型进行修剪和微调变得至关重要。然而，测试时间模型修剪和微调会产生额外的计算成本并降低模型的预测速度，从而带来效率问题。现有的修剪方法对于测试时间模型修剪设置来说不够有效，因为需要微调修剪后的模型来评估可拆卸组件的重要性。为了解决这个问题，我们提出了两个变量来近似微调后的准确度。然后，我们介绍了一种有效的修剪方法，该方法考虑了近似的微调准确率和潜在的推理延迟节省。为了提高微调效率，我们提出了一种有效的知识蒸馏方法，该方法只需要为少量微调样本生成一次伪标签，从而降低了昂贵的伪标签生成成本。实验结果表明，我们的方法在测试准确率和推理延迟之间实现了相当或更好的权衡，与现有最佳方法相比，修剪和微调时间相对减少了 32%。

Title: A Review of Human Emotion Synthesis Based on Generative Technology

• Authors: Fei Ma, Yukan Li, Yifan Xie, Ying He, Yi Zhang, Hongwei Ren, Zhou Liu, Wei Yao, Fuji Ren, Fei Richard Yu, Shiguang Ni
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07116
• Pdf URL: https://arxiv.org/pdf/2412.07116
• Copy Paste: [[2412.07116]] A Review of Human Emotion Synthesis Based on Generative Technology(https://arxiv.org/abs/2412.07116)
• Keywords: generative
• Abstract: Human emotion synthesis is a crucial aspect of affective computing. It involves using computational methods to mimic and convey human emotions through various modalities, with the goal of enabling more natural and effective human-computer interactions. Recent advancements in generative models, such as Autoencoders, Generative Adversarial Networks, Diffusion Models, Large Language Models, and Sequence-to-Sequence Models, have significantly contributed to the development of this field. However, there is a notable lack of comprehensive reviews in this field. To address this problem, this paper aims to address this gap by providing a thorough and systematic overview of recent advancements in human emotion synthesis based on generative models. Specifically, this review will first present the review methodology, the emotion models involved, the mathematical principles of generative models, and the datasets used. Then, the review covers the application of different generative models to emotion synthesis based on a variety of modalities, including facial images, speech, and text. It also examines mainstream evaluation metrics. Additionally, the review presents some major findings and suggests future research directions, providing a comprehensive understanding of the role of generative technology in the nuanced domain of emotion synthesis.
• 摘要：人类情感合成是情感计算的一个重要方面。它涉及使用计算方法通过各种模态模仿和传达人类情感，目的是实现更自然、更有效的人机交互。自动编码器、生成对抗网络、扩散模型、大型语言模型和序列到序列模型等生成模型的最新进展为该领域的发展做出了重大贡献。然而，该领域明显缺乏全面的评论。为了解决这个问题，本文旨在通过提供基于生成模型的人类情感合成最新进展的全面和系统概述来弥补这一空白。具体来说，本综述将首先介绍综述方法、所涉及的情感模型、生成模型的数学原理以及所使用的数据集。然后，本综述涵盖了不同生成模型在基于各种模态（包括面部图像、语音和文本）的情感合成中的应用。它还研究了主流评估指标。此外，该评论提出了一些主要发现并提出了未来的研究方向，全面了解生成技术在情感合成细微领域中的作用。

Title: Bridging the Gap for Test-Time Multimodal Sentiment Analysis

• Authors: Zirun Guo, Tao Jin, Wenlong Xu, Wang Lin, Yangyang Wu
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2412.07121
• Pdf URL: https://arxiv.org/pdf/2412.07121
• Copy Paste: [[2412.07121]] Bridging the Gap for Test-Time Multimodal Sentiment Analysis(https://arxiv.org/abs/2412.07121)
• Keywords: generation
• Abstract: Multimodal sentiment analysis (MSA) is an emerging research topic that aims to understand and recognize human sentiment or emotions through multiple modalities. However, in real-world dynamic scenarios, the distribution of target data is always changing and different from the source data used to train the model, which leads to performance degradation. Common adaptation methods usually need source data, which could pose privacy issues or storage overheads. Therefore, test-time adaptation (TTA) methods are introduced to improve the performance of the model at inference time. Existing TTA methods are always based on probabilistic models and unimodal learning, and thus can not be applied to MSA which is often considered as a multimodal regression task. In this paper, we propose two strategies: Contrastive Adaptation and Stable Pseudo-label generation (CASP) for test-time adaptation for multimodal sentiment analysis. The two strategies deal with the distribution shifts for MSA by enforcing consistency and minimizing empirical risk, respectively. Extensive experiments show that CASP brings significant and consistent improvements to the performance of the model across various distribution shift settings and with different backbones, demonstrating its effectiveness and versatility. Our codes are available at this https URL.
• 摘要：多模态情绪分析 (MSA) 是一个新兴的研究课题，旨在通过多种模态理解和识别人类的情绪或情感。然而，在现实世界的动态场景中，目标数据的分布总是在变化，并且与用于训练模型的源数据不同，这会导致性能下降。常见的自适应方法通常需要源数据，这可能会带来隐私问题或存储开销。因此，引入测试时自适应 (TTA) 方法来提高模型在推理时的性能。现有的 TTA 方法始终基于概率模型和单模态学习，因此不能应用于通常被视为多模态回归任务的 MSA。在本文中，我们提出了两种策略：对比自适应和稳定伪标签生成 (CASP) 用于多模态情绪分析的测试时自适应。这两种策略分别通过强制一致性和最小化经验风险来处理 MSA 的分布变化。大量实验表明，CASP 在不同分布偏移设置和不同主干上为模型的性能带来了显著且一致的改进，证明了其有效性和多功能性。我们的代码可在此 https URL 上找到。

Title: Deep Learning-Enhanced Preconditioning for Efficient Conjugate Gradient Solvers in Large-Scale PDE Systems

• Authors: Rui Li, Song Wang, Chen Wang
• Subjects: cs.LG, cs.AI, math.NA
• Abstract URL: https://arxiv.org/abs/2412.07127
• Pdf URL: https://arxiv.org/pdf/2412.07127
• Copy Paste: [[2412.07127]] Deep Learning-Enhanced Preconditioning for Efficient Conjugate Gradient Solvers in Large-Scale PDE Systems(https://arxiv.org/abs/2412.07127)
• Keywords: generation
• Abstract: Preconditioning techniques are crucial for enhancing the efficiency of solving large-scale linear equation systems that arise from partial differential equation (PDE) discretization. These techniques, such as Incomplete Cholesky factorization (IC) and data-driven neural network methods, accelerate the convergence of iterative solvers like Conjugate Gradient (CG) by approximating the original matrices. This paper introduces a novel approach that integrates Graph Neural Network (GNN) with traditional IC, addressing the shortcomings of direct generation methods based on GNN and achieving significant improvements in computational efficiency and scalability. Experimental results demonstrate an average reduction in iteration counts by 24.8% compared to IC and a two-order-of-magnitude increase in training scale compared to previous methods. A three-dimensional static structural analysis utilizing finite element methods was validated on training sparse matrices of up to 5 million dimensions and inference scales of up to 10 million. Furthermore, the approach demon-strates robust generalization capabilities across scales, facilitating the effective acceleration of CG solvers for large-scale linear equations using small-scale data on modest hardware. The method's robustness and scalability make it a practical solution for computational science.
• 摘要：预处理技术对于提高求解由偏微分方程 (PDE) 离散化引起的大规模线性方程组的效率至关重要。这些技术，例如不完全乔莱斯基分解 (IC) 和数据驱动的神经网络方法，通过近似原始矩阵来加速共轭梯度 (CG) 等迭代求解器的收敛。本文介绍了一种将图神经网络 (GNN) 与传统 IC 相结合的新方法，解决了基于 GNN 的直接生成方法的缺点，并显著提高了计算效率和可扩展性。实验结果表明，与 IC 相比，迭代次数平均减少了 24.8%，与以前的方法相比，训练规模增加了两个数量级。利用有限元方法的三维静态结构分析在高达 500 万维的训练稀疏矩阵和高达 1000 万的推理规模上得到了验证。此外，该方法还展示了跨尺度的稳健泛化能力，有助于在适度硬件上使用小规模数据有效加速大规模线性方程的 CG 求解器。该方法的稳健性和可扩展性使其成为计算科学的实用解决方案。

Title: FIRE: Robust Detection of Diffusion-Generated Images via Frequency-Guided Reconstruction Error

• Authors: Beilin Chu, Xuan Xu, Xin Wang, Yufei Zhang, Weike You, Linna Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07140
• Pdf URL: https://arxiv.org/pdf/2412.07140
• Copy Paste: [[2412.07140]] FIRE: Robust Detection of Diffusion-Generated Images via Frequency-Guided Reconstruction Error(https://arxiv.org/abs/2412.07140)
• Keywords: generation
• Abstract: The rapid advancement of diffusion models has significantly improved high-quality image generation, making generated content increasingly challenging to distinguish from real images and raising concerns about potential misuse. In this paper, we observe that diffusion models struggle to accurately reconstruct mid-band frequency information in real images, suggesting the limitation could serve as a cue for detecting diffusion model generated images. Motivated by this observation, we propose a novel method called Frequency-guided Reconstruction Error (FIRE), which, to the best of our knowledge, is the first to investigate the influence of frequency decomposition on reconstruction error. FIRE assesses the variation in reconstruction error before and after the frequency decomposition, offering a robust method for identifying diffusion model generated images. Extensive experiments show that FIRE generalizes effectively to unseen diffusion models and maintains robustness against diverse perturbations.
• 摘要：扩散模型的快速发展显著提高了高质量图像生成的速度，使得生成的内容越来越难以与真实图像区分开来，也引发了对潜在滥用的担忧。在本文中，我们观察到扩散模型难以准确地重建真实图像中的中频信息，这表明这种限制可以作为检测扩散模型生成图像的线索。受这一观察的启发，我们提出了一种称为频率引导重建误差 (FIRE) 的新方法，据我们所知，这是第一个研究频率分解对重建误差影响的方法。FIRE 评估频率分解前后重建误差的变化，为识别扩散模型生成的图像提供了一种稳健的方法。大量实验表明，FIRE 可以有效地推广到看不见的扩散模型，并对各种扰动保持稳健性。

Title: Integrating MedCLIP and Cross-Modal Fusion for Automatic Radiology Report Generation

• Authors: Qianhao Han, Junyi Liu, Zengchang Qin, Zheng Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07141
• Pdf URL: https://arxiv.org/pdf/2412.07141
• Copy Paste: [[2412.07141]] Integrating MedCLIP and Cross-Modal Fusion for Automatic Radiology Report Generation(https://arxiv.org/abs/2412.07141)
• Keywords: generation
• Abstract: Automating radiology report generation can significantly reduce the workload of radiologists and enhance the accuracy, consistency, and efficiency of clinical this http URL propose a novel cross-modal framework that uses MedCLIP as both a vision extractor and a retrieval mechanism to improve the process of medical report this http URL extracting retrieved report features and image features through an attention-based extract module, and integrating them with a fusion module, our method improves the coherence and clinical relevance of generated this http URL results on the widely used IU-Xray dataset demonstrate the effectiveness of our approach, showing improvements over commonly used methods in both report quality and this http URL, ablation studies provide further validation of the framework, highlighting the importance of accurate report retrieval and feature integration in generating comprehensive medical reports.
• 摘要：自动生成放射学报告可以显著减少放射科医生的工作量，提高临床报告的准确性、一致性和效率，提出一种新颖的跨模态框架，使用 MedCLIP 作为视觉提取器和检索机制，以改进医疗报告的生成过程，通过基于注意的提取模块提取检索到的报告特征和图像特征，并将它们与融合模块集成，我们的方法提高了生成的结果的连贯性和临床相关性，在广泛使用的 IU-Xray 数据集上证明了我们方法的有效性，与常用方法相比，报告质量和消融研究提供了对框架的进一步验证，强调了准确的报告检索和特征集成在生成全面的医疗报告中的重要性。

Title: RAP-SR: RestorAtion Prior Enhancement in Diffusion Models for Realistic Image Super-Resolution

• Authors: Jiangang Wang, Qingnan Fan, Jinwei Chen, Hong Gu, Feng Huang, Wenqi Ren
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07149
• Pdf URL: https://arxiv.org/pdf/2412.07149
• Copy Paste: [[2412.07149]] RAP-SR: RestorAtion Prior Enhancement in Diffusion Models for Realistic Image Super-Resolution(https://arxiv.org/abs/2412.07149)
• Keywords: restoration, super-resolution, generative
• Abstract: Benefiting from their powerful generative capabilities, pretrained diffusion models have garnered significant attention for real-world image super-resolution (Real-SR). Existing diffusion-based SR approaches typically utilize semantic information from degraded images and restoration prompts to activate prior for producing realistic high-resolution images. However, general-purpose pretrained diffusion models, not designed for restoration tasks, often have suboptimal prior, and manually defined prompts may fail to fully exploit the generated potential. To address these limitations, we introduce RAP-SR, a novel restoration prior enhancement approach in pretrained diffusion models for Real-SR. First, we develop the High-Fidelity Aesthetic Image Dataset (HFAID), curated through a Quality-Driven Aesthetic Image Selection Pipeline (QDAISP). Our dataset not only surpasses existing ones in fidelity but also excels in aesthetic quality. Second, we propose the Restoration Priors Enhancement Framework, which includes Restoration Priors Refinement (RPR) and Restoration-Oriented Prompt Optimization (ROPO) modules. RPR refines the restoration prior using the HFAID, while ROPO optimizes the unique restoration identifier, improving the quality of the resulting images. RAP-SR effectively bridges the gap between general-purpose models and the demands of Real-SR by enhancing restoration prior. Leveraging the plug-and-play nature of RAP-SR, our approach can be seamlessly integrated into existing diffusion-based SR methods, boosting their performance. Extensive experiments demonstrate its broad applicability and state-of-the-art results. Codes and datasets will be available upon acceptance.
• 摘要：得益于强大的生成能力，预训练扩散模型在真实世界图像超分辨率 (Real-SR) 中引起了广泛关注。现有的基于扩散的 SR 方法通常利用来自退化图像和恢复提示的语义信息来激活先验，以生成逼真的高分辨率图像。然而，通用的预训练扩散模型并非为恢复任务而设计，通常具有次优先验，而手动定义的提示可能无法充分利用生成的潜力。为了解决这些限制，我们在 Real-SR 的预训练扩散模型中引入了 RAP-SR，这是一种新颖的恢复先验增强方法。首先，我们开发了高保真美学图像数据集 (HFAID)，该数据集通过质量驱动的美学图像选择管道 (QDAISP) 进行策划。我们的数据集不仅在保真度方面超越现有数据集，而且在美学质量方面也表现出色。其次，我们提出了恢复先验增强框架，其中包括恢复先验细化 (RPR) 和面向恢复的快速优化 (ROPO) 模块。RPR 使用 HFAID 细化恢复先验，而 ROPO 优化唯一恢复标识符，从而提高生成图像的质量。RAP-SR 通过增强恢复先验有效地弥合了通用模型与 Real-SR 需求之间的差距。利用 RAP-SR 的即插即用特性，我们的方法可以无缝集成到现有的基于扩散的 SR 方法中，从而提高其性能。大量实验证明了其广泛的适用性和最先进的结果。代码和数据集将在接受后提供。

Title: Hero-SR: One-Step Diffusion for Super-Resolution with Human Perception Priors

• Authors: Jiangang Wang, Qingnan Fan, Qi Zhang, Haigen Liu, Yuhang Yu, Jinwei Chen, Wenqi Ren
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07152
• Pdf URL: https://arxiv.org/pdf/2412.07152
• Copy Paste: [[2412.07152]] Hero-SR: One-Step Diffusion for Super-Resolution with Human Perception Priors(https://arxiv.org/abs/2412.07152)
• Keywords: super-resolution
• Abstract: Owing to the robust priors of diffusion models, recent approaches have shown promise in addressing real-world super-resolution (Real-SR). However, achieving semantic consistency and perceptual naturalness to meet human perception demands remains difficult, especially under conditions of heavy degradation and varied input complexities. To tackle this, we propose Hero-SR, a one-step diffusion-based SR framework explicitly designed with human perception priors. Hero-SR consists of two novel modules: the Dynamic Time-Step Module (DTSM), which adaptively selects optimal diffusion steps for flexibly meeting human perceptual standards, and the Open-World Multi-modality Supervision (OWMS), which integrates guidance from both image and text domains through CLIP to improve semantic consistency and perceptual naturalness. Through these modules, Hero-SR generates high-resolution images that not only preserve intricate details but also reflect human perceptual preferences. Extensive experiments validate that Hero-SR achieves state-of-the-art performance in Real-SR. The code will be publicly available upon paper acceptance.
• 摘要：由于扩散模型具有稳健的先验，最近的方法在解决现实世界超分辨率 (Real-SR) 方面显示出良好的前景。然而，实现语义一致性和感知自然性以满足人类感知需求仍然很困难，尤其是在严重退化和输入复杂度变化的情况下。为了解决这个问题，我们提出了 Hero-SR，这是一个基于扩散的一步式 SR 框架，专门设计了人类感知先验。Hero-SR 由两个新模块组成：动态时间步长模块 (DTSM)，它自适应地选择最佳扩散步骤以灵活地满足人类感知标准，以及开放世界多模态监督 (OWMS​​)，它通过 CLIP 集成来自图像和文本域的指导以提高语义一致性和感知自然性。通过这些模块，Hero-SR 可以生成高分辨率图像，这些图像不仅可以保留复杂的细节，还可以反映人类的感知偏好。大量实验验证了 Hero-SR 在 Real-SR 中实现了最先进的性能。代码将在论文被接受后公开。

Title: Motion-aware Contrastive Learning for Temporal Panoptic Scene Graph Generation

• Authors: Thong Thanh Nguyen, Xiaobao Wu, Yi Bin, Cong-Duy T Nguyen, See-Kiong Ng, Anh Tuan Luu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07160
• Pdf URL: https://arxiv.org/pdf/2412.07160
• Copy Paste: [[2412.07160]] Motion-aware Contrastive Learning for Temporal Panoptic Scene Graph Generation(https://arxiv.org/abs/2412.07160)
• Keywords: generation
• Abstract: To equip artificial intelligence with a comprehensive understanding towards a temporal world, video and 4D panoptic scene graph generation abstracts visual data into nodes to represent entities and edges to capture temporal relations. Existing methods encode entity masks tracked across temporal dimensions (mask tubes), then predict their relations with temporal pooling operation, which does not fully utilize the motion indicative of the entities' relation. To overcome this limitation, we introduce a contrastive representation learning framework that focuses on motion pattern for temporal scene graph generation. Firstly, our framework encourages the model to learn close representations for mask tubes of similar subject-relation-object triplets. Secondly, we seek to push apart mask tubes from their temporally shuffled versions. Moreover, we also learn distant representations for mask tubes belonging to the same video but different triplets. Extensive experiments show that our motion-aware contrastive framework significantly improves state-of-the-art methods on both video and 4D datasets.
• 摘要：为了让人工智能全面理解时间世界，视频和 4D 全景场景图生成将视觉数据抽象为节点以表示实体和边缘以捕获时间关系。现有方法对跨时间维度跟踪的实体掩码（掩码管）进行编码，然后使用时间池化操作预测它们的关系，这并没有充分利用指示实体关系的运动。为了克服这一限制，我们引入了一个对比表示学习框架，该框架专注于时间场景图生成的运动模式。首先，我们的框架鼓励模型学习类似主体-关系-客体三元组的掩码管的近似表示。其次，我们试图将掩码管与其时间打乱的版本分开。此外，我们还学习属于同一视频但不同三元组的掩码管的远距离表示。大量实验表明，我们的运动感知对比框架显着提高了视频和 4D 数据集上的最新方法。

Title: Compositional Zero-Shot Learning with Contextualized Cues and Adaptive Contrastive Training

• Authors: Yun Li, Zhe Liu, Lina Yao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07161
• Pdf URL: https://arxiv.org/pdf/2412.07161
• Copy Paste: [[2412.07161]] Compositional Zero-Shot Learning with Contextualized Cues and Adaptive Contrastive Training(https://arxiv.org/abs/2412.07161)
• Keywords: generation
• Abstract: Compositional Zero-Shot Learning (CZSL) aims to recognize unseen combinations of seen attributes and objects. Current CLIP-based methods in CZSL, despite their advancements, often fail to effectively understand and link the attributes and objects due to inherent limitations in CLIP's pretraining mechanisms. To address these shortcomings, this paper introduces a novel framework, Understanding and Linking Attributes and Objects (ULAO) in CZSL, which comprises two innovative modules. The Understanding Attributes and Objects (UAO) module improves primitive understanding by sequential primitive prediction and leveraging recognized objects as contextual hints for attribute classification. Concurrently, the Linking Attributes and Objects (LAO) module improves the attribute-object linkage understanding through a new contrastive learning strategy that incorporates tailored hard negative generation and adaptive loss adjustments. We demonstrate our model's superiority by showcasing its state-of-the-art performance across three benchmark datasets in both Closed-World (CW) and Open-World (OW) scenarios.
• 摘要：组合零样本学习 (CZSL) 旨在识别可见属性和对象的未见组合。尽管 CZSL 中基于 CLIP 的当前方法取得了进步，但由于 CLIP 预训练机制的固有局限性，它们往往无法有效地理解和链接属性和对象。为了解决这些缺点，本文介绍了一个新框架，即 CZSL 中的理解和链接属性和对象 (ULAO)，它包含两个创新模块。理解属性和对象 (UAO) 模块通过顺序原始预测并利用已识别的对象作为属性分类的上下文提示来提高原始理解。同时，链接属性和对象 (LAO) 模块通过一种新的对比学习策略来提高属性对象链接理解，该策略结合了量身定制的硬负生成和自适应损失调整。我们通过展示模型在封闭世界 (CW) 和开放世界 (OW) 场景中的三个基准数据集上的最新性能来证明我们的模型的优越性。

Title: A Step towards Automated and Generalizable Tactile Map Generation using Generative Adversarial Networks

• Authors: David G Hobson, Majid Komeili
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07191
• Pdf URL: https://arxiv.org/pdf/2412.07191
• Copy Paste: [[2412.07191]] A Step towards Automated and Generalizable Tactile Map Generation using Generative Adversarial Networks(https://arxiv.org/abs/2412.07191)
• Keywords: generation, generative
• Abstract: Blindness and visual impairments affect many people worldwide. For help with navigation, people with visual impairments often rely on tactile maps that utilize raised surfaces and edges to convey information through touch. Although these maps are helpful, they are often not widely available and current tools to automate their production have similar limitations including only working at certain scales, for particular world regions, or adhering to specific tactile map standards. To address these shortcomings, we train a proof-of-concept model as a first step towards applying computer vision techniques to help automate the generation of tactile maps. We create a first-of-its-kind tactile maps dataset of street-views from Google Maps spanning 6500 locations and including different tactile line- and area-like features. Generative adversarial network (GAN) models trained on a single zoom successfully identify key map elements, remove extraneous ones, and perform inpainting with median F1 and intersection-over-union (IoU) scores of better than 0.97 across all features. Models trained on two zooms experience only minor drops in performance, and generalize well both to unseen map scales and world regions. Finally, we discuss future directions towards a full implementation of a tactile map solution that builds on our results.
• 摘要：失明和视力障碍影响着全世界的许多人。为了帮助导航，视力障碍人士通常依靠触觉地图，这种地图利用凸起的表面和边缘通过触摸传达信息。虽然这些地图很有用，但它们通常并不广泛可用，并且当前用于自动化制作这些地图的工具也有类似的限制，包括仅在特定比例下工作、针对特定的世界区域或遵守特定的触觉地图标准。为了解决这些缺点，我们训练了一个概念验证模型，作为应用计算机视觉技术帮助自动生成触觉地图的第一步。我们创建了一个独一无二的触觉地图数据集，其中包含来自 Google 地图的街景，涵盖 6500 个地点，包括不同的触觉线和区域特征。在单次缩放上训练的生成对抗网络 (GAN) 模型成功识别关键地图元素，删除无关元素，并在所有特征上以高于 0.97 的中值 F1 和交并比 (IoU) 得分执行修复。在两次缩放上训练的模型性能仅略有下降，并且能够很好地推广到未见过的地图比例和世界区域。最后，我们讨论了基于我们的结果全面实施触觉地图解决方案的未来方向。

Title: A Progressive Image Restoration Network for High-order Degradation Imaging in Remote Sensing

• Authors: Yujie Feng, Yin Yang, Xiaohong Fan, Zhengpeng Zhang, Lijing Bu, Jianping Zhang
• Subjects: cs.CV, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.07195
• Pdf URL: https://arxiv.org/pdf/2412.07195
• Copy Paste: [[2412.07195]] A Progressive Image Restoration Network for High-order Degradation Imaging in Remote Sensing(https://arxiv.org/abs/2412.07195)
• Keywords: restoration, super-resolution
• Abstract: Recently, deep learning methods have gained remarkable achievements in the field of image restoration for remote sensing (RS). However, most existing RS image restoration methods focus mainly on conventional first-order degradation models, which may not effectively capture the imaging mechanisms of remote sensing images. Furthermore, many RS image restoration approaches that use deep learning are often criticized for their lacks of architecture transparency and model interpretability. To address these problems, we propose a novel progressive restoration network for high-order degradation imaging (HDI-PRNet), to progressively restore different image degradation. HDI-PRNet is developed based on the theoretical framework of degradation imaging, offering the benefit of mathematical interpretability within the unfolding network. The framework is composed of three main components: a module for image denoising that relies on proximal mapping prior learning, a module for image deblurring that integrates Neumann series expansion with dual-domain degradation learning, and a module for super-resolution. Extensive experiments demonstrate that our method achieves superior performance on both synthetic and real remote sensing images.
• 摘要：最近，深度学习方法在遥感图像恢复领域取得了显著的成就。然而，现有的大多数遥感图像恢复方法主要集中在传统的一阶退化模型上，这些模型可能无法有效地捕捉遥感图像的成像机制。此外，许多使用深度学习的遥感图像恢复方法经常因缺乏架构透明度和模型可解释性而受到批评。为了解决这些问题，我们提出了一种用于高阶退化成像的新型渐进式恢复网络（HDI-PRNet），以逐步恢复不同的图像退化。HDI-PRNet 是基于退化成像的理论框架开发的，在展开网络中具有数学可解释性的优势。该框架由三个主要组件组成：一个依赖于近端映射先验学习的图像去噪模块、一个将诺伊曼级数展开与双域退化学习相结合的图像去模糊模块和一个超分辨率模块。大量实验表明，我们的方法在合成和真实遥感图像上都取得了优异的性能。

Title: Fine-grained Text to Image Synthesis

• Authors: Xu Ouyang, Ying Chen, Kaiyue Zhu, Gady Agam
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07196
• Pdf URL: https://arxiv.org/pdf/2412.07196
• Copy Paste: [[2412.07196]] Fine-grained Text to Image Synthesis(https://arxiv.org/abs/2412.07196)
• Keywords: generative
• Abstract: Fine-grained text to image synthesis involves generating images from texts that belong to different categories. In contrast to general text to image synthesis, in fine-grained synthesis there is high similarity between images of different subclasses, and there may be linguistic discrepancy among texts describing the same image. Recent Generative Adversarial Networks (GAN), such as the Recurrent Affine Transformation (RAT) GAN model, are able to synthesize clear and realistic images from texts. However, GAN models ignore fine-grained level information. In this paper we propose an approach that incorporates an auxiliary classifier in the discriminator and a contrastive learning method to improve the accuracy of fine-grained details in images synthesized by RAT GAN. The auxiliary classifier helps the discriminator classify the class of images, and helps the generator synthesize more accurate fine-grained images. The contrastive learning method minimizes the similarity between images from different subclasses and maximizes the similarity between images from the same subclass. We evaluate on several state-of-the-art methods on the commonly used CUB-200-2011 bird dataset and Oxford-102 flower dataset, and demonstrated superior performance.
• 摘要：细粒度文本到图像合成涉及从属于不同类别的文本生成图像。与一般的文本到图像合成相比，在细粒度合成中，不同子类的图像之间具有很高的相似度，并且描述同一图像的文本之间可能存在语言差异。最近的生成对抗网络（GAN），例如循环仿射变换（RAT）GAN 模型，能够从文本合成清晰逼真的图像。然而，GAN 模型忽略了细粒度​​级别的信息。在本文中，我们提出了一种在鉴别器中结合辅助分类器和对比学习方法的方法，以提高 RAT GAN 合成图像中细粒度细节的准确性。辅助分类器帮助鉴别器对图像类别进行分类，并帮助生成器合成更准确的细粒度图像。对比学习方法最小化来自不同子类的图像之间的相似性并最大化来自同一子类的图像之间的相似性。我们对常用的 CUB-200-2011 鸟类数据集和 Oxford-102 花卉数据集上的几种最先进的方法进行了评估，并证明了卓越的性能。

Title: MPSI: Mamba enhancement model for pixel-wise sequential interaction Image Super-Resolution

• Authors: Yuchun He, Yuhan He
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.07222
• Pdf URL: https://arxiv.org/pdf/2412.07222
• Copy Paste: [[2412.07222]] MPSI: Mamba enhancement model for pixel-wise sequential interaction Image Super-Resolution(https://arxiv.org/abs/2412.07222)
• Keywords: super-resolution
• Abstract: Single image super-resolution (SR) has long posed a challenge in the field of computer vision. While the advent of deep learning has led to the emergence of numerous methods aimed at tackling this persistent issue, the current methodologies still encounter challenges in modeling long sequence information, leading to limitations in effectively capturing the global pixel interactions. To tackle this challenge and achieve superior SR outcomes, we propose the Mamba pixel-wise sequential interaction network (MPSI), aimed at enhancing the establishment of long-range connections of information, particularly focusing on pixel-wise sequential interaction. We propose the Channel-Mamba Block (CMB) to capture comprehensive pixel interaction information by effectively modeling long sequence information. Moreover, in the existing SR methodologies, there persists the issue of the neglect of features extracted by preceding layers, leading to the loss of valuable feature information. While certain existing models strive to preserve these features, they frequently encounter difficulty in establishing connections across all layers. To overcome this limitation, MPSI introduces the Mamba channel recursion module (MCRM), which maximizes the retention of valuable feature information from early layers, thereby facilitating the acquisition of pixel sequence interaction information from multiple-level layers. Through extensive experimentation, we demonstrate that MPSI outperforms existing super-resolution methods in terms of image reconstruction results, attaining state-of-the-art performance.
• 摘要：单幅图像超分辨率 (SR) 长期以来一直是计算机视觉领域的一大挑战。虽然深度学习的出现导致了许多旨在解决这一长期问题的方法的出现，但当前的方法在建模长序列信息方面仍然面临挑战，导致无法有效捕获全局像素交互。为了应对这一挑战并获得卓越的 SR 结果，我们提出了 Mamba 像素顺序交互网络 (MPSI)，旨在增强信息的长距离连接建立，尤其关注像素顺序交互。我们提出了 Channel-Mamba Block (CMB)，通过有效地建模长序列信息来捕获全面的像素交互信息。此外，在现有的 SR 方法中，仍然存在忽略前几层提取的特征的问题，从而导致宝贵的特征信息丢失。虽然某些现有模型努力保留这些特征，但它们经常难以在所有层之间建立连接。为了突破这一限制，MPSI 引入了 Mamba 通道递归模块 (MCRM)，最大化地保留了前几层的宝贵特征信息，从而有利于从多层获取像素序列交互信息。通过大量实验，我们证明了 MPSI 在图像重建结果方面优于现有的超分辨率方法，达到了最佳性能。

Title: EchoIR: Advancing Image Restoration with Echo Upsampling and Bi-Level Optimization

• Authors: Yuhan He, Yuchun He
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2412.07225
• Pdf URL: https://arxiv.org/pdf/2412.07225
• Copy Paste: [[2412.07225]] EchoIR: Advancing Image Restoration with Echo Upsampling and Bi-Level Optimization(https://arxiv.org/abs/2412.07225)
• Keywords: restoration
• Abstract: Image restoration represents a fundamental challenge in low-level vision, focusing on reconstructing high-quality images from their degraded counterparts. With the rapid advancement of deep learning technologies, transformer-based methods with pyramid structures have advanced the field by capturing long-range cross-scale spatial interaction. Despite its popularity, the degradation of essential features during the upsampling process notably compromised the restoration performance, resulting in suboptimal reconstruction outcomes. We introduce the EchoIR, an UNet-like image restoration network with a bilateral learnable upsampling mechanism to bridge this gap. Specifically, we proposed the Echo-Upsampler that optimizes the upsampling process by learning from the bilateral intermediate features of U-Net, the "Echo", aiming for a more refined restoration by minimizing the degradation during upsampling. In pursuit of modeling a hierarchical model of image restoration and upsampling tasks, we propose the Approximated Sequential Bi-level Optimization (AS-BLO), an advanced bi-level optimization model establishing a relationship between upsampling learning and image restoration tasks. Extensive experiments against the state-of-the-art (SOTA) methods demonstrate the proposed EchoIR surpasses the existing methods, achieving SOTA performance in image restoration tasks.
• 摘要：图像恢复是低级视觉领域的一项基本挑战，重点是从退化的图像中重建高质量图像。随着深度学习技术的快速发展，基于变换器的金字塔结构方法通过捕捉长距离跨尺度空间交互推动了该领域的发展。尽管它很受欢迎，但在上采样过程中，基本特征的退化显著损害了恢复性能，导致重建结果不理想。我们引入了 EchoIR，这是一个类似 UNet 的图像恢复网络，具有双边可学习的上采样机制来弥补这一差距。具体来说，我们提出了 Echo-Upsampler，它通过从 U-Net 的双边中间特征“Echo”中学习来优化上采样过程，旨在通过最大限度地减少上采样过程中的退化来实现更精细的恢复。为了建立图像恢复和上采样任务的分层模型，我们提出了近似顺序双层优化 (AS-BLO)，这是一种先进的双层优化模型，建立了上采样学习和图像恢复任务之间的关系。与最先进 (SOTA) 方法进行的大量实验表明，所提出的 EchoIR 超越了现有方法，在图像恢复任务中实现了 SOTA 性能。

Title: Moderating the Generalization of Score-based Generative Model

• Authors: Wan Jiang, He Wang, Xin Zhang, Dan Guo, Zhaoxin Fan, Yunfeng Diao, Richang Hong
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07229
• Pdf URL: https://arxiv.org/pdf/2412.07229
• Copy Paste: [[2412.07229]] Moderating the Generalization of Score-based Generative Model(https://arxiv.org/abs/2412.07229)
• Keywords: generation, generative
• Abstract: Score-based Generative Models (SGMs) have demonstrated remarkable generalization abilities, e.g. generating unseen, but natural data. However, the greater the generalization power, the more likely the unintended generalization, and the more dangerous the abuse. Research on moderated generalization in SGMs remains limited. To fill this gap, we first examine the current 'gold standard' in Machine Unlearning (MU), i.e., re-training the model after removing the undesirable training data, and find it does not work in SGMs. Further analysis of score functions reveals that the MU 'gold standard' does not alter the original score function, which explains its ineffectiveness. Based on this insight, we propose the first Moderated Score-based Generative Model (MSGM), which introduces a novel score adjustment strategy that redirects the score function away from undesirable data during the continuous-time stochastic differential equation process. Extensive experimental results demonstrate that MSGM significantly reduces the likelihood of generating undesirable content while preserving high visual quality for normal image generation. Albeit designed for SGMs, MSGM is a general and flexible MU framework that is compatible with diverse diffusion architectures (SGM and DDPM) and training strategies (re-training and fine-tuning), and enables zero-shot transfer of the pre-trained models to downstream tasks, e.g. image inpainting and reconstruction. The code will be shared upon acceptance.
• 摘要：基于分数的生成模型 (SGM) 已展现出卓越的泛化能力，例如生成看不见的但自然的数据。然而，泛化能力越强，越容易出现非预期的泛化，滥用的危险性也越大。对 SGM 中适度泛化的研究仍然有限。为了填补这一空白，我们首先研究了机器反学习 (MU) 中的当前“黄金标准”，即在删除不良训练数据后重新训练模型，发现它在 SGM 中不起作用。对分数函数的进一步分析表明，MU“黄金标准”不会改变原始分数函数，这解释了其无效性。基于这一见解，我们提出了第一个适度的基于分数的生成模型 (MSGM)，该模型引入了一种新颖的分数调整策略，可在连续时间随机微分方程过程中将分数函数从不良数据中重定向出来。大量实验结果表明，MSGM 显著降低了生成不良内容的可能性，同时保持了正常图像生成的高视觉质量。尽管 MSGM 是为 SGM 设计的，但它是一个通用且灵活的 MU 框架，可与各种扩散架构（SGM 和 DDPM）和训练策略（重新训练和微调）兼容，并支持将预训练模型零样本迁移到下游任务，例如图像修复和重建。代码将在接受后共享。

Title: ArtFormer: Controllable Generation of Diverse 3D Articulated Objects

• Authors: Jiayi Su, Youhe Feng, Zheng Li, Jinhua Song, Yangfan He, Botao Ren, Botian Xu
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2412.07237
• Pdf URL: https://arxiv.org/pdf/2412.07237
• Copy Paste: [[2412.07237]] ArtFormer: Controllable Generation of Diverse 3D Articulated Objects(https://arxiv.org/abs/2412.07237)
• Keywords: generation
• Abstract: This paper presents a novel framework for modeling and conditional generation of 3D articulated objects. Troubled by flexibility-quality tradeoffs, existing methods are often limited to using predefined structures or retrieving shapes from static datasets. To address these challenges, we parameterize an articulated object as a tree of tokens and employ a transformer to generate both the object's high-level geometry code and its kinematic relations. Subsequently, each sub-part's geometry is further decoded using a signed-distance-function (SDF) shape prior, facilitating the synthesis of high-quality 3D shapes. Our approach enables the generation of diverse objects with high-quality geometry and varying number of parts. Comprehensive experiments on conditional generation from text descriptions demonstrate the effectiveness and flexibility of our method.
• 摘要：本文介绍了一种用于建模和条件生成 3D 铰接式物体的新型框架。受制于灵活性和质量之间的权衡，现有方法通常仅限于使用预定义结构或从静态数据集中检索形状。为了应对这些挑战，我们将铰接式物体参数化为一个标记树，并使用转换器生成该物体的高级几何代码及其运动关系。随后，使用有符号距离函数 (SDF) 形状先验进一步解码每个子部分的几何形状，从而促进高质量 3D 形状的合成。我们的方法能够生成具有高质量几何形状和不同数量部件的各种物体。对文本描述进行条件生成的综合实验证明了我们方法的有效性和灵活性。

Title: Buster: Incorporating Backdoor Attacks into Text Encoder to Mitigate NSFW Content Generation

• Authors: Xin Zhao, Xiaojun Chen, Yuexin Xuan, Zhendong Zhao
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07249
• Pdf URL: https://arxiv.org/pdf/2412.07249
• Copy Paste: [[2412.07249]] Buster: Incorporating Backdoor Attacks into Text Encoder to Mitigate NSFW Content Generation(https://arxiv.org/abs/2412.07249)
• Keywords: generation
• Abstract: In the digital age, the proliferation of deep learning models has led to significant concerns about the generation of Not Safe for Work (NSFW) content. Existing defense methods primarily involve model fine-tuning and post-hoc content moderation. However, these approaches often lack scalability in eliminating harmful content, degrade the quality of benign image generation, or incur high inference costs. To tackle these challenges, we propose an innovative framework called \textbf{Buster}, which injects backdoor attacks into the text encoder to prevent NSFW content generation. Specifically, Buster leverages deep semantic information rather than explicit prompts as triggers, redirecting NSFW prompts towards targeted benign prompts. This approach demonstrates exceptional resilience and scalability in mitigating NSFW content. Remarkably, Buster fine-tunes the text encoder of Text-to-Image models within just five minutes, showcasing high efficiency. Our extensive experiments reveal that Buster outperforms all other baselines, achieving superior NSFW content removal rate while preserving the quality of harmless images.
• 摘要：在数字时代，深度学习模型的激增引发了人们对“不适合工作 (NSFW)”内容生成的严重担忧。现有的防御方法主要涉及模型微调和事后内容审核。然而，这些方法在消除有害内容方面往往缺乏可扩展性，降低了良性图像生成的质量，或者产生了高昂的推理成本。为了应对这些挑战，我们提出了一个名为 \textbf{Buster} 的创新框架，它将后门攻击注入文本编码器以防止生成 NSFW 内容。具体来说，Buster 利用深度语义信息而不是显式提示作为触发器，将 NSFW 提示重定向到有针对性的良性提示。这种方法在缓解 NSFW 内容方面表现出卓越的弹性和可扩展性。值得注意的是，Buster 只需五分钟即可微调文本转图像模型的文本编码器，展现出高效率。我们广泛的实验表明，Buster 的表现优于所有其他基线，在保持无害图像质量的同时实现了卓越的 NSFW 内容删除率。

Title: CapGen:An Environment-Adaptive Generator of Adversarial Patches

• Authors: Chaoqun Li, Zhuodong Liu, Huanqian Yan, Hang Su
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07253
• Pdf URL: https://arxiv.org/pdf/2412.07253
• Copy Paste: [[2412.07253]] CapGen:An Environment-Adaptive Generator of Adversarial Patches(https://arxiv.org/abs/2412.07253)
• Keywords: generation
• Abstract: Adversarial patches, often used to provide physical stealth protection for critical assets and assess perception algorithm robustness, usually neglect the need for visual harmony with the background environment, making them easily noticeable. Moreover, existing methods primarily concentrate on improving attack performance, disregarding the intricate dynamics of adversarial patch elements. In this work, we introduce the Camouflaged Adversarial Pattern Generator (CAPGen), a novel approach that leverages specific base colors from the surrounding environment to produce patches that seamlessly blend with their background for superior visual stealthiness while maintaining robust adversarial performance. We delve into the influence of both patterns (i.e., color-agnostic texture information) and colors on the effectiveness of attacks facilitated by patches, discovering that patterns exert a more pronounced effect on performance than colors. Based on these findings, we propose a rapid generation strategy for adversarial patches. This involves updating the colors of high-performance adversarial patches to align with those of the new environment, ensuring visual stealthiness without compromising adversarial impact. This paper is the first to comprehensively examine the roles played by patterns and colors in the context of adversarial patches.
• 摘要：对抗补丁通常用于为关键资产提供物理隐身保护并评估感知算法的稳健性，通常忽略了与背景环境的视觉和谐性，因此很容易被注意到。此外，现有方法主要集中在提高攻击性能上，而忽略了对抗补丁元素的复杂动态。在这项工作中，我们引入了伪装对抗模式生成器 (CAPGen)，这是一种新颖的方法，它利用周围环境中的特定基色来生成与背景无缝融合的补丁，以实现卓越的视觉隐身性，同时保持强大的对抗性能。我们深入研究了模式（即与颜色无关的纹理信息）和颜色对补丁促进的攻击有效性的影响，发现模式对性能的影响比颜色更明显。基于这些发现，我们提出了一种对抗补丁的快速生成策略。这涉及更新高性能对抗补丁的颜色以与新环境的颜色保持一致，确保视觉隐身性而不损害对抗影响。本文首次全面研究了对抗性补丁背景下图案和颜色所起的作用。

Title: A Generative Victim Model for Segmentation

• Authors: Aixuan Li, Jing Zhang, Jiawei Shi, Yiran Zhong, Yuchao Dai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07274
• Pdf URL: https://arxiv.org/pdf/2412.07274
• Copy Paste: [[2412.07274]] A Generative Victim Model for Segmentation(https://arxiv.org/abs/2412.07274)
• Keywords: generation, generative
• Abstract: We find that the well-trained victim models (VMs), against which the attacks are generated, serve as fundamental prerequisites for adversarial attacks, i.e. a segmentation VM is needed to generate attacks for segmentation. In this context, the victim model is assumed to be robust to achieve effective adversarial perturbation generation. Instead of focusing on improving the robustness of the task-specific victim models, we shift our attention to image generation. From an image generation perspective, we derive a novel VM for segmentation, aiming to generate adversarial perturbations for segmentation tasks without requiring models explicitly designed for image segmentation. Our approach to adversarial attack generation diverges from conventional white-box or black-box attacks, offering a fresh outlook on adversarial attack strategies. Experiments show that our attack method is able to generate effective adversarial attacks with good transferability.
• 摘要：我们发现，针对其生成攻击的训练有素的受害者模型（VM）是进行对抗性攻击的基本先决条件，即需要分割 VM 来生成针对分割的攻击。在这种情况下，假设受害者模型具有鲁棒性，以实现有效的对抗性扰动生成。我们不再专注于提高特定于任务的受害者模型的鲁棒性，而是将注意力转移到图像生成上。从图像生成的角度来看，我们推导出一种用于分割的新型 VM，旨在为分割任务生成对抗性扰动，而无需明确为图像分割设计的模型。我们的对抗性攻击生成方法不同于传统的白盒或黑盒攻击，为对抗性攻击策略提供了全新的视角。实验表明，我们的攻击方法能够生成具有良好可转移性的有效对抗性攻击。

Title: Backdoor Attacks against No-Reference Image Quality Assessment Models via A Scalable Trigger

• Authors: Yi Yu, Song Xia, Xun Lin, Wenhan Yang, Shijian Lu, Yap-peng Tan, Alex Kot
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2412.07277
• Pdf URL: https://arxiv.org/pdf/2412.07277
• Copy Paste: [[2412.07277]] Backdoor Attacks against No-Reference Image Quality Assessment Models via A Scalable Trigger(https://arxiv.org/abs/2412.07277)
• Keywords: quality assessment
• Abstract: No-Reference Image Quality Assessment (NR-IQA), responsible for assessing the quality of a single input image without using any reference, plays a critical role in evaluating and optimizing computer vision systems, e.g., low-light enhancement. Recent research indicates that NR-IQA models are susceptible to adversarial attacks, which can significantly alter predicted scores with visually imperceptible perturbations. Despite revealing vulnerabilities, these attack methods have limitations, including high computational demands, untargeted manipulation, limited practical utility in white-box scenarios, and reduced effectiveness in black-box scenarios. To address these challenges, we shift our focus to another significant threat and present a novel poisoning-based backdoor attack against NR-IQA (BAIQA), allowing the attacker to manipulate the IQA model's output to any desired target value by simply adjusting a scaling coefficient $\alpha$ for the trigger. We propose to inject the trigger in the discrete cosine transform (DCT) domain to improve the local invariance of the trigger for countering trigger diminishment in NR-IQA models due to widely adopted data augmentations. Furthermore, the universal adversarial perturbations (UAP) in the DCT space are designed as the trigger, to increase IQA model susceptibility to manipulation and improve attack effectiveness. In addition to the heuristic method for poison-label BAIQA (P-BAIQA), we explore the design of clean-label BAIQA (C-BAIQA), focusing on $\alpha$ sampling and image data refinement, driven by theoretical insights we reveal. Extensive experiments on diverse datasets and various NR-IQA models demonstrate the effectiveness of our attacks. Code will be released at this https URL.
• 摘要：无参考图像质量评估 (NR-IQA) 负责在不使用任何参考的情况下评估单个输入图像的质量，在评估和优化计算机视觉系统（例如低光增强）中起着关键作用。最近的研究表明，NR-IQA 模型容易受到对抗性攻击，这种攻击可以通过视觉上不可察觉的扰动显著改变预测分数。尽管暴露了漏洞，但这些攻击方法存在局限性，包括高计算需求、无针对性的操纵、白盒场景中的实际效用有限以及黑盒场景中的有效性降低。为了应对这些挑战，我们将重点转移到另一个重大威胁上，并提出了一种针对 NR-IQA (BAIQA) 的新型基于中毒的后门攻击，允许攻击者通过简单地调整触发器的缩放系数 $\alpha$ 将 IQA 模型的输出操纵为任何所需的目标值。我们建议在离散余弦变换 (DCT) 域中注入触发器，以提高触发器的局部不变性，以抵消由于广泛采用的数据增强而导致的 NR-IQA 模型中的触发器减少。此外，DCT 空间中的通用对抗扰动 (UAP) 被设计为触发器，以增加 IQA 模型对操纵的敏感性并提高攻击效果。除了毒标​​签 BAIQA (P-BAIQA) 的启发式方法外，我们还探索了清洁标签 BAIQA (C-BAIQA) 的设计，重点关注 $\alpha$ 采样和图像数据细化，由我们揭示的理论见解驱动。对各种数据集和各种 NR-IQA 模型进行的大量实验证明了我们攻击的有效性。代码将在此 https URL 上发布。

Title: CoMA: Compositional Human Motion Generation with Multi-modal Agents

• Authors: Shanlin Sun, Gabriel De Araujo, Jiaqi Xu, Shenghan Zhou, Hanwen Zhang, Ziheng Huang, Chenyu You, Xiaohui Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07320
• Pdf URL: https://arxiv.org/pdf/2412.07320
• Copy Paste: [[2412.07320]] CoMA: Compositional Human Motion Generation with Multi-modal Agents(https://arxiv.org/abs/2412.07320)
• Keywords: generation
• Abstract: 3D human motion generation has seen substantial advancement in recent years. While state-of-the-art approaches have improved performance significantly, they still struggle with complex and detailed motions unseen in training data, largely due to the scarcity of motion datasets and the prohibitive cost of generating new training examples. To address these challenges, we introduce CoMA, an agent-based solution for complex human motion generation, editing, and comprehension. CoMA leverages multiple collaborative agents powered by large language and vision models, alongside a mask transformer-based motion generator featuring body part-specific encoders and codebooks for fine-grained control. Our framework enables generation of both short and long motion sequences with detailed instructions, text-guided motion editing, and self-correction for improved quality. Evaluations on the HumanML3D dataset demonstrate competitive performance against state-of-the-art methods. Additionally, we create a set of context-rich, compositional, and long text prompts, where user studies show our method significantly outperforms existing approaches.
• 摘要：近年来，3D 人体运动生成取得了长足进步。虽然最先进的方法已显著提高了性能，但它们仍然难以处理训练数据中未见的复杂和详细运动，这主要是由于运动数据集的稀缺以及生成新训练示例的成本过高。为了应对这些挑战，我们推出了 CoMA，这是一种基于代理的复杂人体运动生成、编辑和理解解决方案。CoMA 利用由大型语言和视觉模型驱动的多个协作代理，以及基于掩码转换器的运动生成器，该生成器具有针对身体部位的编码器和码本，可进行细粒度控制。我们的框架能够生成短运动序列和长运动序列，并提供详细说明、文本引导的运动编辑和自我校正以提高质量。在 HumanML3D 数据集上的评估表明，与最先进的方法相比，我们具有竞争力。此外，我们创建了一组上下文丰富、构图丰富的长文本提示，用户研究表明我们的方法明显优于现有方法。

Title: ConceptSearch: Towards Efficient Program Search Using LLMs for Abstraction and Reasoning Corpus (ARC)

• Authors: Kartik Singhal, Gautam Shroff
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07322
• Pdf URL: https://arxiv.org/pdf/2412.07322
• Copy Paste: [[2412.07322]] ConceptSearch: Towards Efficient Program Search Using LLMs for Abstraction and Reasoning Corpus (ARC)(https://arxiv.org/abs/2412.07322)
• Keywords: generation
• Abstract: The Abstraction and Reasoning Corpus (ARC) poses a significant challenge to artificial intelligence, demanding broad generalization and few-shot learning capabilities that remain elusive for current deep learning methods, including large language models (LLMs). While LLMs excel in program synthesis, their direct application to ARC yields limited success. To address this, we introduce ConceptSearch, a novel function-search algorithm that leverages LLMs for program generation and employs a concept-based scoring method to guide the search efficiently. Unlike simplistic pixel-based metrics like Hamming distance, ConceptSearch evaluates programs on their ability to capture the underlying transformation concept reflected in the input-output examples. We explore three scoring functions: Hamming distance, a CNN-based scoring function, and an LLM-based natural language scoring function. Experimental results demonstrate the effectiveness of ConceptSearch, achieving a significant performance improvement over direct prompting with GPT-4. Moreover, our novel concept-based scoring exhibits up to 30% greater efficiency compared to Hamming distance, measured in terms of the number of iterations required to reach the correct solution. These findings highlight the potential of LLM-driven program search when integrated with concept-based guidance for tackling challenging generalization problems like ARC. Code: this https URL
• 摘要：抽象和推理语料库 (ARC) 对人工智能提出了重大挑战，要求广泛的泛化和少量学习能力，而这对于当前的深度学习方法（包括大型语言模型 (LLM)）来说仍然难以实现。虽然 LLM 在程序合成方面表现出色，但将其直接应用于 ARC 取得的成功有限。为了解决这个问题，我们引入了 ConceptSearch，这是一种新颖的函数搜索算法，它利用 LLM 进行程序生成，并采用基于概念的评分方法来有效地指导搜索。与汉明距离等简单的基于像素的指标不同，ConceptSearch 根据程序捕捉输入输出示例中反映的底层转换概念的能力来评估程序。我们探索了三个评分函数：汉明距离、基于 CNN 的评分函数和基于 LLM 的自然语言评分函数。实验结果证明了 ConceptSearch 的有效性，与使用 GPT-4 的直接提示相比，实现了显着的性能提升。此外，与汉明距离相比，我们新颖的基于概念的评分表现出高达 30% 的效率，以达到正确解决方案所需的迭代次数来衡量。这些发现凸显了 LLM 驱动的程序搜索与基于概念的指导相结合时解决 ARC 等具有挑战性的泛化问题的潜力。代码：此 https URL

Title: Fusion Embedding for Pose-Guided Person Image Synthesis with Diffusion Model

• Authors: Donghwna Lee, Kyungha Min, Kirok Kim, Seyoung Jeong, Jiwoo Jeong, Wooju Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07333
• Pdf URL: https://arxiv.org/pdf/2412.07333
• Copy Paste: [[2412.07333]] Fusion Embedding for Pose-Guided Person Image Synthesis with Diffusion Model(https://arxiv.org/abs/2412.07333)
• Keywords: generation, generative
• Abstract: Pose-Guided Person Image Synthesis (PGPIS) aims to synthesize high-quality person images corresponding to target poses while preserving the appearance of the source image. Recently, PGPIS methods that use diffusion models have achieved competitive performance. Most approaches involve extracting representations of the target pose and source image and learning their relationships in the generative model's training process. This approach makes it difficult to learn the semantic relationships between the input and target images and complicates the model structure needed to enhance generation results. To address these issues, we propose Fusion embedding for PGPIS using a Diffusion Model (FPDM). Inspired by the successful application of pre-trained CLIP models in text-to-image diffusion models, our method consists of two stages. The first stage involves training the fusion embedding of the source image and target pose to align with the target image's embedding. In the second stage, the generative model uses this fusion embedding as a condition to generate the target image. We applied the proposed method to the benchmark datasets DeepFashion and RWTH-PHOENIX-Weather 2014T, and conducted both quantitative and qualitative evaluations, demonstrating state-of-the-art (SOTA) performance. An ablation study of the model structure showed that even a model using only the second stage achieved performance close to the other PGPIS SOTA models. The code is available at this https URL.
• 摘要：姿势引导人物图像合成 (PGPIS) 旨在合成与目标姿势相对应的高质量人物图像，同时保留源图像的外观。最近，使用扩散模型的 PGPIS 方法已经取得了有竞争力的表现。大多数方法涉及提取目标姿势和源图像的表示，并在生成模型的训练过程中学习它们的关系。这种方法使得学习输入和目标图像之间的语义关系变得困难，并使增强生成结果所需的模型结构复杂化。为了解决这些问题，我们提出了使用扩散模型 (FPDM)​​ 的 PGPIS 融合嵌入。受预训练 CLIP 模型在文本到图像扩散模型中的成功应用的启发，我们的方法包括两个阶段。第一阶段涉及训练源图像和目标姿势的融合嵌入以与目标图像的嵌入对齐。在第二阶段，生成模型使用此融合嵌入作为生成目标图像的条件。我们将所提出的方法应用于基准数据集 DeepFashion 和 RWTH-PHOENIX-Weather 2014T，并进行了定量和定性评估，展示了最先进的 (SOTA) 性能。对模型结构的消融研究表明，即使仅使用第二阶段的模型也能实现接近其他 PGPIS SOTA 模型的性能。代码可在此 https URL 上找到。

Title: StoryWeaver: A Unified World Model for Knowledge-Enhanced Story Character Customization

• Authors: Jinlu Zhang, Jiji Tang, Rongsheng Zhang, Tangjie Lv, Xiaoshuai Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07375
• Pdf URL: https://arxiv.org/pdf/2412.07375
• Copy Paste: [[2412.07375]] StoryWeaver: A Unified World Model for Knowledge-Enhanced Story Character Customization(https://arxiv.org/abs/2412.07375)
• Keywords: generation
• Abstract: Story visualization has gained increasing attention in artificial intelligence. However, existing methods still struggle with maintaining a balance between character identity preservation and text-semantics alignment, largely due to a lack of detailed semantic modeling of the story scene. To tackle this challenge, we propose a novel knowledge graph, namely Character Graph (\textbf{CG}), which comprehensively represents various story-related knowledge, including the characters, the attributes related to characters, and the relationship between characters. We then introduce StoryWeaver, an image generator that achieve Customization via Character Graph (\textbf{C-CG}), capable of consistent story visualization with rich text semantics. To further improve the multi-character generation performance, we incorporate knowledge-enhanced spatial guidance (\textbf{KE-SG}) into StoryWeaver to precisely inject character semantics into generation. To validate the effectiveness of our proposed method, extensive experiments are conducted using a new benchmark called TBC-Bench. The experiments confirm that our StoryWeaver excels not only in creating vivid visual story plots but also in accurately conveying character identities across various scenarios with considerable storage efficiency, \emph{e.g.}, achieving an average increase of +9.03\% DINO-I and +13.44\% CLIP-T. Furthermore, ablation experiments are conducted to verify the superiority of the proposed module. Codes and datasets are released at this https URL.
• 摘要：故事可视化在人工智能领域受到越来越多的关注。然而，现有的方法仍然难以在人物身份保存和文本语义对齐之间保持平衡，这主要是因为缺乏对故事场景的详细语义建模。为了应对这一挑战，我们提出了一种新颖的知识图谱，即角色图谱（\textbf{CG}），它全面表示了与故事相关的各种知识，包括人物、与人物相关的属性以及人物之间的关系。然后，我们介绍了 StoryWeaver，这是一个通过角色图谱（\textbf{C-CG}）实现自定义的图像生成器，能够实现具有丰富文本语义的故事可视化。为了进一步提高多角色生成的性能，我们将知识增强空间引导（\textbf{KE-SG}）融入 StoryWeaver，以将人物语义精确地注入生成中。为了验证我们提出的方法的有效性，我们使用一个名为 TBC-Bench 的新基准进行了大量实验。实验证实，我们的 StoryWeaver 不仅擅长创建生动的视觉故事情节，而且还擅长在各种场景中准确传达角色身份，并且具有相当高的存储效率，例如，DINO-I 平均提升了 +9.03% ，CLIP-T 平均提升了 +13.44%。此外，还进行了消融实验以验证所提模块的优越性。代码和数据集在此 https URL 上发布。

Title: LOGen: Toward Lidar Object Generation by Point Diffusion

• Authors: Ellington Kirby, Mickael Chen, Renaud Marlet, Nermin Samet
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07385
• Pdf URL: https://arxiv.org/pdf/2412.07385
• Copy Paste: [[2412.07385]] LOGen: Toward Lidar Object Generation by Point Diffusion(https://arxiv.org/abs/2412.07385)
• Keywords: generation
• Abstract: A common strategy to improve lidar segmentation results on rare semantic classes consists of pasting objects from one lidar scene into another. While this augments the quantity of instances seen at training time and varies their context, the instances fundamentally remain the same. In this work, we explore how to enhance instance diversity using a lidar object generator. We introduce a novel diffusion-based method to produce lidar point clouds of dataset objects, including reflectance, and with an extensive control of the generation via conditioning information. Our experiments on nuScenes show the quality of our object generations measured with new 3D metrics developed to suit lidar objects.
• 摘要：一种常见的改进激光雷达在稀有语义类别上的分割结果的策略是将物体从一个激光雷达场景粘贴到另一个场景。虽然这增加了训练时看到的实例数量并改变了它们的上下文，但实例基本上保持不变。在这项工作中，我们探索如何使用激光雷达对象生成器增强实例多样性。我们引入了一种基于扩散的新型方法来生成数据集对象的激光雷达点云，包括反射率，并通过调节信息对生成进行广泛的控制。我们在 nuScenes 上的实验表明，我们用为适合激光雷达对象而开发的新 3D 指标来衡量对象生成的质量。

Title: BENet: A Cross-domain Robust Network for Detecting Face Forgeries via Bias Expansion and Latent-space Attention

• Authors: Weihua Liu, Jianhua Qiu, Said Boumaraf, Chaochao lin, Pan liyuan, Lin Li, Mohammed Bennamoun, Naoufel Werghi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2412.07431
• Pdf URL: https://arxiv.org/pdf/2412.07431
• Copy Paste: [[2412.07431]] BENet: A Cross-domain Robust Network for Detecting Face Forgeries via Bias Expansion and Latent-space Attention(https://arxiv.org/abs/2412.07431)
• Keywords: generation
• Abstract: In response to the growing threat of deepfake technology, we introduce BENet, a Cross-Domain Robust Bias Expansion Network. BENet enhances the detection of fake faces by addressing limitations in current detectors related to variations across different types of fake face generation techniques, where ``cross-domain" refers to the diverse range of these deepfakes, each considered a separate domain. BENet's core feature is a bias expansion module based on autoencoders. This module maintains genuine facial features while enhancing differences in fake reconstructions, creating a reliable bias for detecting fake faces across various deepfake domains. We also introduce a Latent-Space Attention (LSA) module to capture inconsistencies related to fake faces at different scales, ensuring robust defense against advanced deepfake techniques. The enriched LSA feature maps are multiplied with the expanded bias to create a versatile feature space optimized for subtle forgeries detection. To improve its ability to detect fake faces from unknown sources, BENet integrates a cross-domain detector module that enhances recognition accuracy by verifying the facial domain during inference. We train our network end-to-end with a novel bias expansion loss, adopted for the first time, in face forgery detection. Extensive experiments covering both intra and cross-dataset demonstrate BENet's superiority over current state-of-the-art solutions.
• 摘要：为了应对日益严重的深度伪造技术威胁，我们推出了跨域稳健偏差扩展网络 BENet。BENet 通过解决当前检测器与不同类型的假脸生成技术之间的差异相关的局限性来增强对假脸的检测，其中“跨域”是指这些深度伪造的多样性，每个都被视为一个单独的域。BENet 的核心功能是基于自动编码器的偏差扩展模块。该模块保留了真实的面部特征，同时增强了假重建的差异，为检测各种深度伪造域中的假脸创建了可靠的偏差。我们还引入了一个潜在空间注意 (LSA) 模块来捕获与不同尺度的假脸相关的不一致性，确保对高级深度伪造技术的强大防御。丰富的 LSA 特征图与扩展的偏差相乘，以创建一个针对微妙伪造检测优化的多功能特征空间。为了提高其检测来自未知来源的假脸的能力，BENet 集成了一个跨域检测器模块，通过在推理过程中验证面部域来提高识别准确性。我们采用一种新颖的偏差扩展损失对网络进行端到端训练，这是首次在人脸伪造检测中采用。大量涵盖内部和跨数据集的实验证明了 BENet 优于当前最先进的解决方案。

Title: FireFlow: Fast Inversion of Rectified Flow for Image Semantic Editing

• Authors: Yingying Deng, Xiangyu He, Changwang Mei, Peisong Wang, Fan Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07517
• Pdf URL: https://arxiv.org/pdf/2412.07517
• Copy Paste: [[2412.07517]] FireFlow: Fast Inversion of Rectified Flow for Image Semantic Editing(https://arxiv.org/abs/2412.07517)
• Keywords: generation
• Abstract: Though Rectified Flows (ReFlows) with distillation offers a promising way for fast sampling, its fast inversion transforms images back to structured noise for recovery and following editing remains unsolved. This paper introduces FireFlow, a simple yet effective zero-shot approach that inherits the startling capacity of ReFlow-based models (such as FLUX) in generation while extending its capabilities to accurate inversion and editing in $8$ steps. We first demonstrate that a carefully designed numerical solver is pivotal for ReFlow inversion, enabling accurate inversion and reconstruction with the precision of a second-order solver while maintaining the practical efficiency of a first-order Euler method. This solver achieves a $3\times$ runtime speedup compared to state-of-the-art ReFlow inversion and editing techniques, while delivering smaller reconstruction errors and superior editing results in a training-free mode. The code is available at $\href{this https URL}{this URL}$.
• 摘要：虽然带有蒸馏的整流流 (ReFlows) 为快速采样提供了一种有前途的方法，但其快速反演将图像转换回结构化噪声以进行恢复，并且随后的编辑仍未解决。本文介绍了 FireFlow，这是一种简单而有效的零样本方法，它继承了基于 ReFlow 的模型（例如 FLUX）在生成方面的惊人能力，同时将其功能扩展到 8 步的精确反演和编辑。我们首先证明精心设计的数值求解器对于 ReFlow 反演至关重要，它能够以二阶求解器的精度进行精确的反演和重建，同时保持一阶欧拉方法的实际效率。与最先进的 ReFlow 反演和编辑技术相比，该求解器实现了 3 倍的运行时间加速，同时在无训练模式下提供更小的重建误差和出色的编辑结果。代码可在 $\href{this https URL}{this URL}$ 获得。

Title: Hallucination Elimination and Semantic Enhancement Framework for Vision-Language Models in Traffic Scenarios

• Authors: Jiaqi Fan, Jianhua Wu, Hongqing Chu, Quanbo Ge, Bingzhao Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07518
• Pdf URL: https://arxiv.org/pdf/2412.07518
• Copy Paste: [[2412.07518]] Hallucination Elimination and Semantic Enhancement Framework for Vision-Language Models in Traffic Scenarios(https://arxiv.org/abs/2412.07518)
• Keywords: generation
• Abstract: Large vision-language models (LVLMs) have demonstrated remarkable capabilities in multimodal understanding and generation tasks. However, these models occasionally generate hallucinatory texts, resulting in descriptions that seem reasonable but do not correspond to the image. This phenomenon can lead to wrong driving decisions of the autonomous driving system. To address this challenge, this paper proposes HCOENet, a plug-and-play chain-of-thought correction method designed to eliminate object hallucinations and generate enhanced descriptions for critical objects overlooked in the initial response. Specifically, HCOENet employs a cross-checking mechanism to filter entities and directly extracts critical objects from the given image, enriching the descriptive text. Experimental results on the POPE benchmark demonstrate that HCOENet improves the F1-score of the Mini-InternVL-4B and mPLUG-Owl3 models by 12.58% and 4.28%, respectively. Additionally, qualitative results using images collected in open campus scene further highlight the practical applicability of the proposed method. Compared with the GPT-4o model, HCOENet achieves comparable descriptive performance while significantly reducing costs. Finally, two novel semantic understanding datasets, CODA_desc and nuScenes_desc, are created for traffic scenarios to support future research. The codes and datasets are publicly available at this https URL.
• 摘要：大型视觉语言模型 (LVLM) 在多模态理解和生成任务中表现出了卓越的能力。然而，这些模型偶尔会生成幻觉文本，导致描述看似合理但与图像不符。这种现象可能导致自动驾驶系统做出错误的驾驶决策。为了应对这一挑战，本文提出了 HCOENet，这是一种即插即用的思路链校正方法，旨在消除对象幻觉并为初始响应中忽略的关键对象生成增强的描述。具体而言，HCOENet 采用交叉检查机制来过滤实体并直接从给定图像中提取关键对象，从而丰富描述性文本。在 POPE 基准上的实验结果表明，HCOENet 分别将 Mini-InternVL-4B 和 mPLUG-Owl3 模型的 F1 分数提高了 12.58% 和 4.28%。此外，使用在开放校园场景中收集的图像的定性结果进一步突出了所提方法的实际适用性。与 GPT-4o 模型相比，HCOENet 实现了相当的描述性能，同时显著降低了成本。最后，针对交通场景创建了两个新的语义理解数据集 CODA_desc 和 nuScenes_desc，以支持未来的研究。代码和数据集在此 https URL 上公开提供。

Title: Paired Wasserstein Autoencoders for Conditional Sampling

• Authors: Moritz Piening, Matthias Chung
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2412.07586
• Pdf URL: https://arxiv.org/pdf/2412.07586
• Copy Paste: [[2412.07586]] Paired Wasserstein Autoencoders for Conditional Sampling(https://arxiv.org/abs/2412.07586)
• Keywords: generative
• Abstract: Wasserstein distances greatly influenced and coined various types of generative neural network models. Wasserstein autoencoders are particularly notable for their mathematical simplicity and straight-forward implementation. However, their adaptation to the conditional case displays theoretical difficulties. As a remedy, we propose the use of two paired autoencoders. Under the assumption of an optimal autoencoder pair, we leverage the pairwise independence condition of our prescribed Gaussian latent distribution to overcome this theoretical hurdle. We conduct several experiments to showcase the practical applicability of the resulting paired Wasserstein autoencoders. Here, we consider imaging tasks and enable conditional sampling for denoising, inpainting, and unsupervised image translation. Moreover, we connect our image translation model to the Monge map behind Wasserstein-2 distances.
• 摘要：Wasserstein 距离极大地影响并创造了各种类型的生成神经网络模型。Wasserstein 自动编码器因其数学简单性和直接实现而尤为引人注目。然而，它们对条件情况的适应显示出理论上的困难。作为一种补救措施，我们建议使用两个配对的自动编码器。在最佳自动编码器对的假设下，我们利用我们规定的高斯潜在分布的成对独立条件来克服这一理论障碍。我们进行了几次实验来展示由此产生的配对 Wasserstein 自动编码器的实际适用性。在这里，我们考虑成像任务并启用条件采样以进行去噪、修复和无监督图像转换。此外，我们将图像转换模型连接到 Wasserstein-2 距离背后的 Monge 映射。

Title: DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation

• Authors: Jianzong Wu, Chao Tang, Jingbo Wang, Yanhong Zeng, Xiangtai Li, Yunhai Tong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07589
• Pdf URL: https://arxiv.org/pdf/2412.07589
• Copy Paste: [[2412.07589]] DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation(https://arxiv.org/abs/2412.07589)
• Keywords: generation
• Abstract: Story visualization, the task of creating visual narratives from textual descriptions, has seen progress with text-to-image generation models. However, these models often lack effective control over character appearances and interactions, particularly in multi-character scenes. To address these limitations, we propose a new task: \textbf{customized manga generation} and introduce \textbf{DiffSensei}, an innovative framework specifically designed for generating manga with dynamic multi-character control. DiffSensei integrates a diffusion-based image generator with a multimodal large language model (MLLM) that acts as a text-compatible identity adapter. Our approach employs masked cross-attention to seamlessly incorporate character features, enabling precise layout control without direct pixel transfer. Additionally, the MLLM-based adapter adjusts character features to align with panel-specific text cues, allowing flexible adjustments in character expressions, poses, and actions. We also introduce \textbf{MangaZero}, a large-scale dataset tailored to this task, containing 43,264 manga pages and 427,147 annotated panels, supporting the visualization of varied character interactions and movements across sequential frames. Extensive experiments demonstrate that DiffSensei outperforms existing models, marking a significant advancement in manga generation by enabling text-adaptable character customization. The project page is this https URL.
• 摘要：故事可视化是从文本描述创建视觉叙事的任务，文本到图像生成模型已取得进展。然而，这些模型通常缺乏对角色外观和互动的有效控制，尤其是在多角色场景中。为了解决这些限制，我们提出了一项新任务：\textbf{定制漫画生成}，并引入了 \textbf{DiffSensei}，这是一个专门为生成具有动态多角色控制的漫画而设计的创新框架。DiffSensei 将基于扩散的图像生成器与充当文本兼容身份适配器的多模态大型语言模型 (MLLM) 集成在一起。我们的方法采用掩蔽交叉注意来无缝整合角色特征，无需直接像素传输即可实现精确的布局控制。此外，基于 MLLM 的适配器会调整角色特征以与面板特定的文本提示保持一致，从而可以灵活调整角色表情、姿势和动作。我们还引入了 \textbf{MangaZero}，这是一个专为此任务量身定制的大型数据集，包含 43,264 页漫画和 427,147 个带注释的面板，支持可视化连续帧中各种角色的交互和动作。大量实验表明，DiffSensei 优于现有模型，通过实现文本自适应角色定制，标志着漫画生成取得了重大进步。项目页面是这个 https URL。

Title: RFL: Simplifying Chemical Structure Recognition with Ring-Free Language

• Authors: Qikai Chang, Mingjun Chen, Changpeng Pi, Pengfei Hu, Zhenrong Zhang, Jiefeng Ma, Jun Du, Baocai Yin, Jinshui Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07594
• Pdf URL: https://arxiv.org/pdf/2412.07594
• Copy Paste: [[2412.07594]] RFL: Simplifying Chemical Structure Recognition with Ring-Free Language(https://arxiv.org/abs/2412.07594)
• Keywords: generation
• Abstract: The primary objective of Optical Chemical Structure Recognition is to identify chemical structure images into corresponding markup sequences. However, the complex two-dimensional structures of molecules, particularly those with rings and multiple branches, present significant challenges for current end-to-end methods to learn one-dimensional markup directly. To overcome this limitation, we propose a novel Ring-Free Language (RFL), which utilizes a divide-and-conquer strategy to describe chemical structures in a hierarchical form. RFL allows complex molecular structures to be decomposed into multiple parts, ensuring both uniqueness and conciseness while enhancing readability. This approach significantly reduces the learning difficulty for recognition models. Leveraging RFL, we propose a universal Molecular Skeleton Decoder (MSD), which comprises a skeleton generation module that progressively predicts the molecular skeleton and individual rings, along with a branch classification module for predicting branch information. Experimental results demonstrate that the proposed RFL and MSD can be applied to various mainstream methods, achieving superior performance compared to state-of-the-art approaches in both printed and handwritten scenarios. The code is available at this https URL.
• 摘要：光学化学结构识别的主要目标是将化学结构图像识别为相应的标记序列。然而，分子的复杂二维结构，特别是那些具有环和多个分支的分子，对当前端到端方法直接学习一维标记提出了重大挑战。为了克服这一限制，我们提出了一种新颖的无环语言 (RFL)，它利用分而治之的策略以分层形式描述化学结构。RFL 允许将复杂的分子结构分解为多个部分，确保唯一性和简洁性，同时提高可读性。这种方法大大降低了识别模型的学习难度。利用 RFL，我们提出了一种通用的分子骨架解码器 (MSD)，它包括一个逐步预测分子骨架和单个环的骨架生成模块，以及一个用于预测分支信息的分支分类模块。实验结果表明，所提出的 RFL 和 MSD 可以应用于各种主流方法，在印刷和手写场景中均能实现优于最新方法的性能。代码可在此 https URL 上获取。

Title: OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations

• Authors: Linke Ouyang, Yuan Qu, Hongbin Zhou, Jiawei Zhu, Rui Zhang, Qunshu Lin, Bin Wang, Zhiyuan Zhao, Man Jiang, Xiaomeng Zhao, Jin Shi, Fan Wu, Pei Chu, Minghao Liu, Zhenxiang Li, Chao Xu, Bo Zhang, Botian Shi, Zhongying Tu, Conghui He
• Subjects: cs.CV, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2412.07626
• Pdf URL: https://arxiv.org/pdf/2412.07626
• Copy Paste: [[2412.07626]] OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotations(https://arxiv.org/abs/2412.07626)
• Keywords: generation
• Abstract: Document content extraction is crucial in computer vision, especially for meeting the high-quality data needs of large language models (LLMs) and retrieval-augmented generation (RAG) technologies. However, current document parsing methods suffer from significant limitations in terms of diversity and comprehensive evaluation. To address these challenges, we introduce OmniDocBench, a novel multi-source benchmark designed to advance automated document content extraction. OmniDocBench includes a meticulously curated and annotated high-quality evaluation dataset comprising nine diverse document types, such as academic papers, textbooks, slides, among others. Our benchmark provides a flexible and comprehensive evaluation framework with 19 layout category labels and 14 attribute labels, enabling multi-level assessments across entire datasets, individual modules, or specific data types. Using OmniDocBench, we perform an exhaustive comparative analysis of existing modular pipelines and multimodal end-to-end methods, highlighting their limitations in handling document diversity and ensuring fair evaluation. OmniDocBench establishes a robust, diverse, and fair evaluation standard for the document content extraction field, offering crucial insights for future advancements and fostering the development of document parsing technologies. The codes and dataset is available in this https URL.
• 摘要：文档内容提取在计算机视觉中至关重要，尤其是为了满足大型语言模型 (LLM) 和检索增强生成 (RAG) 技术的高质量数据需求。然而，当前的文档解析方法在多样性和综合评估方面存在很大局限性。为了应对这些挑战，我们推出了 OmniDocBench，这是一种新颖的多源基准，旨在推进自动文档内容提取。OmniDocBench 包含一个精心策划和注释的高质量评估数据集，包含九种不同的文档类型，例如学术论文、教科书、幻灯片等。我们的基准提供了一个灵活而全面的评估框架，具有 19 个布局类别标签和 14 个属性标签，可跨整个数据集、单个模块或特定数据类型进行多级评估。使用 OmniDocBench，我们对现有的模块化管道和多模式端到端方法进行了详尽的比较分析，强调了它们在处理文档多样性和确保公平评估方面的局限性。 OmniDocBench 为文档内容提取领域建立了一个强大、多样化且公平的评估标准，为未来的发展提供了重要见解，并促进了文档解析技术的发展。代码和数据集可在此 https URL 中找到。

Title: Sampling from Boltzmann densities with physics informed low-rank formats

• Authors: Paul Hagemann, Janina Schütte, David Sommer, Martin Eigel, Gabriele Steidl
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2412.07637
• Pdf URL: https://arxiv.org/pdf/2412.07637
• Copy Paste: [[2412.07637]] Sampling from Boltzmann densities with physics informed low-rank formats(https://arxiv.org/abs/2412.07637)
• Keywords: generation
• Abstract: Our method proposes the efficient generation of samples from an unnormalized Boltzmann density by solving the underlying continuity equation in the low-rank tensor train (TT) format. It is based on the annealing path commonly used in MCMC literature, which is given by the linear interpolation in the space of energies. Inspired by Sequential Monte Carlo, we alternate between deterministic time steps from the TT representation of the flow field and stochastic steps, which include Langevin and resampling steps. These adjust the relative weights of the different modes of the target distribution and anneal to the correct path distribution. We showcase the efficiency of our method on multiple numerical examples.
• 摘要：我们的方法提出通过求解低秩张量序列 (TT) 格式的底层连续性方程，从非归一化玻尔兹曼密度中高效生成样本。它基于 MCMC 文献中常用的退火路径，该路径由能量空间中的线性插值给出。受序贯蒙特卡罗的启发，我们在流场的 TT 表示中的确定性时间步骤和随机步骤（包括朗之万和重采样步骤）之间交替。这些调整目标分布不同模式的相对权重并退火到正确的路径分布。我们在多个数值示例上展示了我们方法的效率。

Title: Proc-GS: Procedural Building Generation for City Assembly with 3D Gaussians

• Authors: Yixuan Li, Xingjian Ran, Linning Xu, Tao Lu, Mulin Yu, Zhenzhi Wang, Yuanbo Xiangli, Dahua Lin, Bo Dai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07660
• Pdf URL: https://arxiv.org/pdf/2412.07660
• Copy Paste: [[2412.07660]] Proc-GS: Procedural Building Generation for City Assembly with 3D Gaussians(https://arxiv.org/abs/2412.07660)
• Keywords: generation, generative
• Abstract: Buildings are primary components of cities, often featuring repeated elements such as windows and doors. Traditional 3D building asset creation is labor-intensive and requires specialized skills to develop design rules. Recent generative models for building creation often overlook these patterns, leading to low visual fidelity and limited scalability. Drawing inspiration from procedural modeling techniques used in the gaming and visual effects industry, our method, Proc-GS, integrates procedural code into the 3D Gaussian Splatting (3D-GS) framework, leveraging their advantages in high-fidelity rendering and efficient asset management from both worlds. By manipulating procedural code, we can streamline this process and generate an infinite variety of buildings. This integration significantly reduces model size by utilizing shared foundational assets, enabling scalable generation with precise control over building assembly. We showcase the potential for expansive cityscape generation while maintaining high rendering fidelity and precise control on both real and synthetic cases.
• 摘要：建筑是城市的主要组成部分，通常具有重复的元素，例如窗户和门。传统的 3D 建筑资产创建是劳动密集型的，需要专业技能来制定设计规则。最近的建筑创建生成模型经常忽略这些模式，导致视觉保真度低和可扩展性有限。从游戏和视觉效果行业使用的程序建模技术中汲取灵感，我们的方法 Proc-GS 将程序代码集成到 3D Gaussian Splatting (3D-GS) 框架中，利用它们在高保真渲染和高效资产管理方面的优势。通过操纵程序代码，我们可以简化这个过程并生成无限多样的建筑。这种集成通过利用共享的基础资产显着减小了模型大小，实现了可扩展的生成并精确控制建筑物组装。我们展示了扩展城市景观生成的潜力，同时在真实和合成情况下保持高渲染保真度和精确控制。

Title: FiVA: Fine-grained Visual Attribute Dataset for Text-to-Image Diffusion Models

• Authors: Tong Wu, Yinghao Xu, Ryan Po, Mengchen Zhang, Guandao Yang, Jiaqi Wang, Ziwei Liu, Dahua Lin, Gordon Wetzstein
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07674
• Pdf URL: https://arxiv.org/pdf/2412.07674
• Copy Paste: [[2412.07674]] FiVA: Fine-grained Visual Attribute Dataset for Text-to-Image Diffusion Models(https://arxiv.org/abs/2412.07674)
• Keywords: generation
• Abstract: Recent advances in text-to-image generation have enabled the creation of high-quality images with diverse applications. However, accurately describing desired visual attributes can be challenging, especially for non-experts in art and photography. An intuitive solution involves adopting favorable attributes from the source images. Current methods attempt to distill identity and style from source images. However, "style" is a broad concept that includes texture, color, and artistic elements, but does not cover other important attributes such as lighting and dynamics. Additionally, a simplified "style" adaptation prevents combining multiple attributes from different sources into one generated image. In this work, we formulate a more effective approach to decompose the aesthetics of a picture into specific visual attributes, allowing users to apply characteristics such as lighting, texture, and dynamics from different images. To achieve this goal, we constructed the first fine-grained visual attributes dataset (FiVA) to the best of our knowledge. This FiVA dataset features a well-organized taxonomy for visual attributes and includes around 1 M high-quality generated images with visual attribute annotations. Leveraging this dataset, we propose a fine-grained visual attribute adaptation framework (FiVA-Adapter), which decouples and adapts visual attributes from one or more source images into a generated one. This approach enhances user-friendly customization, allowing users to selectively apply desired attributes to create images that meet their unique preferences and specific content requirements.
• 摘要：文本到图像生成的最新进展使得创建具有多种应用的高质量图像成为可能。然而，准确描述所需的视觉属性可能具有挑战性，尤其是对于艺术和摄影方面的非专家而言。一种直观的解决方案涉及从源图像中采用有利的属性。当前的方法试图从源图像中提取身份和风格。然而，“风格”是一个广泛的概念，包括纹理、颜色和艺术元素，但不包括其他重要属性，如照明和动态。此外，简化的“风格”适应性会阻止将来自不同来源的多个属性组合成一张生成的图像。在这项工作中，我们制定了一种更有效的方法，将图片的美学分解为特定的视觉属性，允许用户应用来自不同图像的照明、纹理和动态等特征。为了实现这一目标，我们尽我们所知构建了第一个细粒度的视觉属性数据集 (FiVA)。这个 FiVA 数据集具有组织良好的视觉属性分类法，包括大约 100 万张带有视觉属性注释的高质量生成图像。利用此数据集，我们提出了一个细粒度视觉属性自适应框架 (FiVA-Adapter)，它将一个或多个源图像中的视觉属性解耦并调整为生成的图像。这种方法增强了用户友好的自定义功能，允许用户有选择地应用所需的属性来创建满足其独特偏好和特定内容要求的图像。

Title: Privacy-Preserving Customer Support: A Framework for Secure and Scalable Interactions

• Authors: Anant Prakash Awasthi, Chandraketu Singh, Rakshit Varma, Sanchit Sharma
• Subjects: cs.LG, cs.CR, stat.AP
• Abstract URL: https://arxiv.org/abs/2412.07687
• Pdf URL: https://arxiv.org/pdf/2412.07687
• Copy Paste: [[2412.07687]] Privacy-Preserving Customer Support: A Framework for Secure and Scalable Interactions(https://arxiv.org/abs/2412.07687)
• Keywords: generation
• Abstract: The growing reliance on artificial intelligence (AI) in customer support has significantly improved operational efficiency and user experience. However, traditional machine learning (ML) approaches, which require extensive local training on sensitive datasets, pose substantial privacy risks and compliance challenges with regulations like the General Data Protection Regulation (GDPR) and California Consumer Privacy Act (CCPA). Existing privacy-preserving techniques, such as anonymization, differential privacy, and federated learning, address some concerns but face limitations in utility, scalability, and complexity. This paper introduces the Privacy-Preserving Zero-Shot Learning (PP-ZSL) framework, a novel approach leveraging large language models (LLMs) in a zero-shot learning mode. Unlike conventional ML methods, PP-ZSL eliminates the need for local training on sensitive data by utilizing pre-trained LLMs to generate responses directly. The framework incorporates real-time data anonymization to redact or mask sensitive information, retrieval-augmented generation (RAG) for domain-specific query resolution, and robust post-processing to ensure compliance with regulatory standards. This combination reduces privacy risks, simplifies compliance, and enhances scalability and operational efficiency. Empirical analysis demonstrates that the PP-ZSL framework provides accurate, privacy-compliant responses while significantly lowering the costs and complexities of deploying AI-driven customer support systems. The study highlights potential applications across industries, including financial services, healthcare, e-commerce, legal support, telecommunications, and government services. By addressing the dual challenges of privacy and performance, this framework establishes a foundation for secure, efficient, and regulatory-compliant AI applications in customer interactions.
• 摘要：客户支持对人工智能 (AI) 的日益依赖显著提高了运营效率和用户体验。然而，传统的机器学习 (ML) 方法需要对敏感数据集进行大量的本地训练，这带来了巨大的隐私风险，并面临着《通用数据保护条例》(GDPR) 和《加州消费者隐私法案》(CCPA) 等法规的合规挑战。现有的隐私保护技术，如匿名化、差异隐私和联邦学习，解决了一些问题，但在实用性、可扩展性和复杂性方面受到限制。本文介绍了隐私保护零样本学习 (PP-ZSL) 框架，这是一种在零样本学习模式下利用大型语言模型 (LLM) 的新方法。与传统的 ML 方法不同，PP-ZSL 通过利用预先训练的 LLM 直接生成响应，消除了对敏感数据进行本地训练的需要。该框架结合了实时数据匿名化来编辑或屏蔽敏感信息、检索增强生成 (RAG) 用于特定领域的查询解析，以及强大的后处理以确保符合监管标准。这种组合降低了隐私风险，简化了合规性，并提高了可扩展性和运营效率。实证分析表明，PP-ZSL 框架提供了准确、符合隐私要求的响应，同时显著降低了部署 AI 驱动的客户支持系统的成本和复杂性。该研究重点介绍了跨行业的潜在应用，包括金融服务、医疗保健、电子商务、法律支持、电信和政府服务。通过解决隐私和性能的双重挑战，该框架为客户互动中安全、高效且符合法规要求的 AI 应用奠定了基础。

Title: SimVS: Simulating World Inconsistencies for Robust View Synthesis

• Authors: Alex Trevithick, Roni Paiss, Philipp Henzler, Dor Verbin, Rundi Wu, Hadi Alzayer, Ruiqi Gao, Ben Poole, Jonathan T. Barron, Aleksander Holynski, Ravi Ramamoorthi, Pratul P. Srinivasan
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07696
• Pdf URL: https://arxiv.org/pdf/2412.07696
• Copy Paste: [[2412.07696]] SimVS: Simulating World Inconsistencies for Robust View Synthesis(https://arxiv.org/abs/2412.07696)
• Keywords: generative
• Abstract: Novel-view synthesis techniques achieve impressive results for static scenes but struggle when faced with the inconsistencies inherent to casual capture settings: varying illumination, scene motion, and other unintended effects that are difficult to model explicitly. We present an approach for leveraging generative video models to simulate the inconsistencies in the world that can occur during capture. We use this process, along with existing multi-view datasets, to create synthetic data for training a multi-view harmonization network that is able to reconcile inconsistent observations into a consistent 3D scene. We demonstrate that our world-simulation strategy significantly outperforms traditional augmentation methods in handling real-world scene variations, thereby enabling highly accurate static 3D reconstructions in the presence of a variety of challenging inconsistencies. Project page: this https URL
• 摘要：新颖的视图合成技术在静态场景中取得了令人印象深刻的结果，但在面对随意捕捉设置固有的不一致性时却举步维艰：变化的照明、场景运动和其他难以明确建模的意外效果。我们提出了一种利用生成视频模型来模拟捕捉过程中可能发生的世界不一致性的方法。我们使用此过程以及现有的多视图数据集来创建合成数据，以训练能够将不一致的观察结果协调为一致的 3D 场景的多视图协调网络。我们证明，我们的世界模拟策略在处理现实世界场景变化方面明显优于传统的增强方法，从而能够在存在各种具有挑战性的不一致性的情况下实现高精度的静态 3D 重建。项目页面：此 https URL

Title: ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformer

• Authors: Jinyi Hu, Shengding Hu, Yuxuan Song, Yufei Huang, Mingxuan Wang, Hao Zhou, Zhiyuan Liu, Wei-Ying Ma, Maosong Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07720
• Pdf URL: https://arxiv.org/pdf/2412.07720
• Copy Paste: [[2412.07720]] ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformer(https://arxiv.org/abs/2412.07720)
• Keywords: generation
• Abstract: The recent surge of interest in comprehensive multimodal models has necessitated the unification of diverse modalities. However, the unification suffers from disparate methodologies. Continuous visual generation necessitates the full-sequence diffusion-based approach, despite its divergence from the autoregressive modeling in the text domain. We posit that autoregressive modeling, i.e., predicting the future based on past deterministic experience, remains crucial in developing both a visual generation model and a potential unified multimodal model. In this paper, we explore an interpolation between the autoregressive modeling and full-parameters diffusion to model visual information. At its core, we present ACDiT, an Autoregressive blockwise Conditional Diffusion Transformer, where the block size of diffusion, i.e., the size of autoregressive units, can be flexibly adjusted to interpolate between token-wise autoregression and full-sequence diffusion. ACDiT is easy to implement, as simple as creating a Skip-Causal Attention Mask (SCAM) during training. During inference, the process iterates between diffusion denoising and autoregressive decoding that can make full use of KV-Cache. We verify the effectiveness of ACDiT on image and video generation tasks. We also demonstrate that benefitted from autoregressive modeling, ACDiT can be seamlessly used in visual understanding tasks despite being trained on the diffusion objective. The analysis of the trade-off between autoregressive modeling and diffusion demonstrates the potential of ACDiT to be used in long-horizon visual generation tasks. These strengths make it promising as the backbone of future unified models.
• 摘要：最近，人们对综合多模态模型的兴趣高涨，这使得多种模态必须统一。然而，统一受到不同方法论的困扰。连续视觉生成需要基于全序列扩散的方法，尽管它与文本领域的自回归建模有所不同。我们认为，自回归建模，即根据过去确定性经验预测未来，对于开发视觉生成模型和潜在的统一多模态模型仍然至关重要。在本文中，我们探索了在自回归建模和全参数扩散之间进行插值以对视觉信息进行建模。在其核心，我们提出了 ACDiT，一种自回归块条件扩散变压器，其中扩散的块大小，即自回归单元的大小，可以灵活调整以在逐个标记的自回归和全序列扩散之间进行插值。 ACDiT 易于实现，就像在训练期间创建 Skip-Causal Attention Mask (SCAM) 一样简单。在推理过程中，该过程在扩散去噪和自回归解码之间迭代，可以充分利用 KV-Cache。我们在图像和视频生成任务上验证了 ACDiT 的有效性。我们还证明，得益于自回归建模，尽管在扩散目标上进行了训练，ACDiT 仍可以无缝地用于视觉理解任务。对自回归建模和扩散之间权衡的分析表明，ACDiT 有潜力用于长期视觉生成任务。这些优势使其有望成为未来统一模型的骨干。

Title: ObjCtrl-2.5D: Training-free Object Control with Camera Poses

• Authors: Zhouxia Wang, Yushi Lan, Shangchen Zhou, Chen Change Loy
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07721
• Pdf URL: https://arxiv.org/pdf/2412.07721
• Copy Paste: [[2412.07721]] ObjCtrl-2.5D: Training-free Object Control with Camera Poses(https://arxiv.org/abs/2412.07721)
• Keywords: generation
• Abstract: This study aims to achieve more precise and versatile object control in image-to-video (I2V) generation. Current methods typically represent the spatial movement of target objects with 2D trajectories, which often fail to capture user intention and frequently produce unnatural results. To enhance control, we present ObjCtrl-2.5D, a training-free object control approach that uses a 3D trajectory, extended from a 2D trajectory with depth information, as a control signal. By modeling object movement as camera movement, ObjCtrl-2.5D represents the 3D trajectory as a sequence of camera poses, enabling object motion control using an existing camera motion control I2V generation model (CMC-I2V) without training. To adapt the CMC-I2V model originally designed for global motion control to handle local object motion, we introduce a module to isolate the target object from the background, enabling independent local control. In addition, we devise an effective way to achieve more accurate object control by sharing low-frequency warped latent within the object's region across frames. Extensive experiments demonstrate that ObjCtrl-2.5D significantly improves object control accuracy compared to training-free methods and offers more diverse control capabilities than training-based approaches using 2D trajectories, enabling complex effects like object rotation. Code and results are available at this https URL.
• 摘要：本研究旨在实现图像到视频 (I2V) 生成中更精确、更通用的对象控制。当前的方法通常用 2D 轨迹表示目标对象的空间运动，这通常无法捕捉用户意图并经常产生不自然的结果。为了增强控制，我们提出了 ObjCtrl-2.5D，这是一种无需训练的对象控制方法，它使用从具有深度信息的 2D 轨迹扩展而来的 3D 轨迹作为控制信号。通过将对象运动建模为相机运动，ObjCtrl-2.5D 将 3D 轨迹表示为一系列相机姿势，从而可以使用现有的相机运动控制 I2V 生成模型 (CMC-I2V) 进行对象运动控制而无需训练。为了使最初为全局运动控制设计的 CMC-I2V 模型适应处理局部对象运动，我们引入了一个模块来将目标对象与背景隔离，从而实现独立的局部控制。此外，我们设计了一种有效的方法，通过在帧之间共享对象区域内的低频扭曲潜伏信号来实现更精确的对象控制。大量实验表明，与无需训练的方法相比，ObjCtrl-2.5D 显著提高了物体控制的准确性，并且比使用 2D 轨迹的基于训练的方法提供了更多样化的控制能力，从而实现了物体旋转等复杂效果。代码和结果可在此 https URL 上找到。

Title: STIV: Scalable Text and Image Conditioned Video Generation

• Authors: Zongyu Lin, Wei Liu, Chen Chen, Jiasen Lu, Wenze Hu, Tsu-Jui Fu, Jesse Allardice, Zhengfeng Lai, Liangchen Song, Bowen Zhang, Cha Chen, Yiran Fei, Yifan Jiang, Lezhi Li, Yizhou Sun, Kai-Wei Chang, Yinfei Yang
• Subjects: cs.CV, cs.AI, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2412.07730
• Pdf URL: https://arxiv.org/pdf/2412.07730
• Copy Paste: [[2412.07730]] STIV: Scalable Text and Image Conditioned Video Generation(https://arxiv.org/abs/2412.07730)
• Keywords: generation
• Abstract: The field of video generation has made remarkable advancements, yet there remains a pressing need for a clear, systematic recipe that can guide the development of robust and scalable models. In this work, we present a comprehensive study that systematically explores the interplay of model architectures, training recipes, and data curation strategies, culminating in a simple and scalable text-image-conditioned video generation method, named STIV. Our framework integrates image condition into a Diffusion Transformer (DiT) through frame replacement, while incorporating text conditioning via a joint image-text conditional classifier-free guidance. This design enables STIV to perform both text-to-video (T2V) and text-image-to-video (TI2V) tasks simultaneously. Additionally, STIV can be easily extended to various applications, such as video prediction, frame interpolation, multi-view generation, and long video generation, etc. With comprehensive ablation studies on T2I, T2V, and TI2V, STIV demonstrate strong performance, despite its simple design. An 8.7B model with 512 resolution achieves 83.1 on VBench T2V, surpassing both leading open and closed-source models like CogVideoX-5B, Pika, Kling, and Gen-3. The same-sized model also achieves a state-of-the-art result of 90.1 on VBench I2V task at 512 resolution. By providing a transparent and extensible recipe for building cutting-edge video generation models, we aim to empower future research and accelerate progress toward more versatile and reliable video generation solutions.
• 摘要：视频生成领域取得了显著的进步，但仍然迫切需要一个清晰、系统的方法来指导开发稳健且可扩展的模型。在这项工作中，我们提出了一项全面的研究，系统地探索了模型架构、训练方法和数据管理策略之间的相互作用，最终形成了一种简单且可扩展的文本图像条件视频生成方法，名为 STIV。我们的框架通过帧替换将图像条件集成到扩散变换器 (DiT) 中，同时通过联合图像文本条件无分类器指导结合文本条件。这种设计使 STIV 能够同时执行文本到视频 (T2V) 和文本图像到视频 (TI2V) 任务。此外，STIV 可以轻松扩展到各种应用，例如视频预测、帧插值、多视图生成和长视频生成等。通过对 T2I、T2V 和 TI2V 的全面消融研究，STIV 尽管设计简单，但仍表现出强大的性能。具有 512 分辨率的 8.7B 模型在 VBench T2V 上达到 83.1，超越了 CogVideoX-5B、Pika、Kling 和 Gen-3 等领先的开源和闭源模型。相同大小的模型在 512 分辨率的 VBench I2V 任务上也取得了 90.1 的最佳结果。通过提供构建尖端视频生成模型的透明且可扩展的方法，我们旨在为未来研究提供支持，并加速向更通用、更可靠的视频生成解决方案迈进。

Title: StyleMaster: Stylize Your Video with Artistic Generation and Translation

• Authors: Zixuan Ye, Huijuan Huang, Xintao Wang, Pengfei Wan, Di Zhang, Wenhan Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07744
• Pdf URL: https://arxiv.org/pdf/2412.07744
• Copy Paste: [[2412.07744]] StyleMaster: Stylize Your Video with Artistic Generation and Translation(https://arxiv.org/abs/2412.07744)
• Keywords: generation
• Abstract: Style control has been popular in video generation models. Existing methods often generate videos far from the given style, cause content leakage, and struggle to transfer one video to the desired style. Our first observation is that the style extraction stage matters, whereas existing methods emphasize global style but ignore local textures. In order to bring texture features while preventing content leakage, we filter content-related patches while retaining style ones based on prompt-patch similarity; for global style extraction, we generate a paired style dataset through model illusion to facilitate contrastive learning, which greatly enhances the absolute style consistency. Moreover, to fill in the image-to-video gap, we train a lightweight motion adapter on still videos, which implicitly enhances stylization extent, and enables our image-trained model to be seamlessly applied to videos. Benefited from these efforts, our approach, StyleMaster, not only achieves significant improvement in both style resemblance and temporal coherence, but also can easily generalize to video style transfer with a gray tile ControlNet. Extensive experiments and visualizations demonstrate that StyleMaster significantly outperforms competitors, effectively generating high-quality stylized videos that align with textual content and closely resemble the style of reference images. Our project page is at this https URL
• 摘要：风格控制在视频生成模型中很流行。现有方法生成的视频通常远离给定的风格，导致内容泄漏，并且难以将一个视频转换为所需的风格。我们的第一个观察结果是风格提取阶段很重要，而现有方法强调全局风格但忽略了局部纹理。为了在防止内容泄漏的同时带来纹理特征，我们根据即时补丁相似性过滤内容相关补丁，同时保留风格补丁；对于全局风格提取，我们通过模型错觉生成配对风格数据集以促进对比学习，这大大提高了绝对风格一致性。此外，为了填补图像到视频的差距，我们在静态视频上训练了一个轻量级运动适配器，这隐式地增强了风格化程度，并使我们的图像训练模型能够无缝应用于视频。得益于这些努力，我们的方法 StyleMaster 不仅在风格相似性和时间连贯性方面取得了显着的改进，而且还可以轻松地推广到使用灰度图块 ControlNet 的视频风格转换。大量实验和可视化表明，StyleMaster 的表现明显优于竞争对手，能够有效生成与文本内容一致且与参考图像风格非常相似的高质量风格化视频。我们的项目页面位于此 https URL

Title: Multi-Shot Character Consistency for Text-to-Video Generation

• Authors: Yuval Atzmon, Rinon Gal, Yoad Tewel, Yoni Kasten, Gal Chechik
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07750
• Pdf URL: https://arxiv.org/pdf/2412.07750
• Copy Paste: [[2412.07750]] Multi-Shot Character Consistency for Text-to-Video Generation(https://arxiv.org/abs/2412.07750)
• Keywords: generation
• Abstract: Text-to-video models have made significant strides in generating short video clips from textual descriptions. Yet, a significant challenge remains: generating several video shots of the same characters, preserving their identity without hurting video quality, dynamics, and responsiveness to text prompts. We present Video Storyboarding, a training-free method to enable pretrained text-to-video models to generate multiple shots with consistent characters, by sharing features between them. Our key insight is that self-attention query features (Q) encode both motion and identity. This creates a hard-to-avoid trade-off between preserving character identity and making videos dynamic, when features are shared. To address this issue, we introduce a novel query injection strategy that balances identity preservation and natural motion retention. This approach improves upon naive consistency techniques applied to videos, which often struggle to maintain this delicate equilibrium. Our experiments demonstrate significant improvements in character consistency across scenes while maintaining high-quality motion and text alignment. These results offer insights into critical stages of video generation and the interplay of structure and motion in video diffusion models.
• 摘要：文本转视频模型在根据文本描述生成短视频片段方面取得了重大进展。然而，仍然存在一个重大挑战：生成相同角色的多个视频镜头，在不损害视频质量、动态和对文本提示的响应能力的情况下保留他们的身份。我们提出了视频故事板，这是一种无需训练的方法，通过在它们之间共享特征，使预训练的文本转视频模型能够生成具有一致角色的多个镜头。我们的主要见解是，自注意力查询特征 (Q) 同时编码运动和身份。当共享特征时，这在保留角色身份和使视频动态化之间产生了难以避免的权衡。为了解决这个问题，我们引入了一种新颖的查询注入策略，以平衡身份保留和自然运动保留。这种方法改进了应用于视频的简单一致性技术，这些技术通常难以保持这种微妙的平衡。我们的实验表明，在保持高质量运动和文本对齐的同时，场景间的角色一致性得到了显着改善。这些结果为视频生成的关键阶段以及视频扩散模型中结构和运动的相互作用提供了见解。

Title: PortraitTalk: Towards Customizable One-Shot Audio-to-Talking Face Generation

• Authors: Fatemeh Nazarieh, Zhenhua Feng, Diptesh Kanojia, Muhammad Awais, Josef Kittler
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2412.07754
• Pdf URL: https://arxiv.org/pdf/2412.07754
• Copy Paste: [[2412.07754]] PortraitTalk: Towards Customizable One-Shot Audio-to-Talking Face Generation(https://arxiv.org/abs/2412.07754)
• Keywords: generation
• Abstract: Audio-driven talking face generation is a challenging task in digital communication. Despite significant progress in the area, most existing methods concentrate on audio-lip synchronization, often overlooking aspects such as visual quality, customization, and generalization that are crucial to producing realistic talking faces. To address these limitations, we introduce a novel, customizable one-shot audio-driven talking face generation framework, named PortraitTalk. Our proposed method utilizes a latent diffusion framework consisting of two main components: IdentityNet and AnimateNet. IdentityNet is designed to preserve identity features consistently across the generated video frames, while AnimateNet aims to enhance temporal coherence and motion consistency. This framework also integrates an audio input with the reference images, thereby reducing the reliance on reference-style videos prevalent in existing approaches. A key innovation of PortraitTalk is the incorporation of text prompts through decoupled cross-attention mechanisms, which significantly expands creative control over the generated videos. Through extensive experiments, including a newly developed evaluation metric, our model demonstrates superior performance over the state-of-the-art methods, setting a new standard for the generation of customizable realistic talking faces suitable for real-world applications.
• 摘要：音频驱动的说话脸生成是数字通信领域的一项挑战性任务。尽管该领域取得了重大进展，但大多数现有方法都集中在音频唇同步上，往往忽略了视觉质量、定制和泛化等方面，而这些对于生成逼真的说话脸至关重要。为了解决这些限制，我们引入了一种新颖的、可定制的一次性音频驱动说话脸生成框架，名为 PortraitTalk。我们提出的方法利用了一个潜在扩散框架，该框架由两个主要组件组成：IdentityNet 和 AnimateNet。IdentityNet 旨在在生成的视频帧中始终保持身份特征，而 AnimateNet 旨在增强时间连贯性和运动一致性。该框架还将音频输入与参考图像集成在一起，从而减少了对现有方法中普遍存在的参考式视频的依赖。PortraitTalk 的一项关键创新是通过解耦的交叉注意机制加入文本提示，这大大扩展了对生成视频的创作控制。通过大量实验，包括新开发的评估指标，我们的模型表现出优于最先进方法的性能，为生成适合实际应用的可定制逼真的说话面孔树立了新标准。

Title: 3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation

• Authors: Xiao Fu, Xian Liu, Xintao Wang, Sida Peng, Menghan Xia, Xiaoyu Shi, Ziyang Yuan, Pengfei Wan, Di Zhang, Dahua Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07759
• Pdf URL: https://arxiv.org/pdf/2412.07759
• Copy Paste: [[2412.07759]] 3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation(https://arxiv.org/abs/2412.07759)
• Keywords: generation
• Abstract: This paper aims to manipulate multi-entity 3D motions in video generation. Previous methods on controllable video generation primarily leverage 2D control signals to manipulate object motions and have achieved remarkable synthesis results. However, 2D control signals are inherently limited in expressing the 3D nature of object motions. To overcome this problem, we introduce 3DTrajMaster, a robust controller that regulates multi-entity dynamics in 3D space, given user-desired 6DoF pose (location and rotation) sequences of entities. At the core of our approach is a plug-and-play 3D-motion grounded object injector that fuses multiple input entities with their respective 3D trajectories through a gated self-attention mechanism. In addition, we exploit an injector architecture to preserve the video diffusion prior, which is crucial for generalization ability. To mitigate video quality degradation, we introduce a domain adaptor during training and employ an annealed sampling strategy during inference. To address the lack of suitable training data, we construct a 360-Motion Dataset, which first correlates collected 3D human and animal assets with GPT-generated trajectory and then captures their motion with 12 evenly-surround cameras on diverse 3D UE platforms. Extensive experiments show that 3DTrajMaster sets a new state-of-the-art in both accuracy and generalization for controlling multi-entity 3D motions. Project page: this http URL
• 摘要：本文旨在在视频生成中操纵多实体 3D 运动。以前的可控视频生成方法主要利用 2D 控制信号来操纵物体运动，并取得了显著的合成效果。然而，2D 控制信号在表达物体运动的 3D 特性方面存在固有限制。为了解决这个问题，我们引入了 3DTrajMaster，这是一个强大的控制器，可以根据用户所需的 6DoF 姿势（位置和旋转）实体序列来调节 3D 空间中的多实体动态。我们方法的核心是一个即插即用的 3D 运动接地物体注入器，它通过门控自注意力机制将多个输入实体与其各自的 3D 轨迹融合在一起。此外，我们利用注入器架构来保留视频扩散先验，这对于泛化能力至关重要。为了减轻视频质量下降，我们在训练期间引入了一个域适配器，并在推理期间采用了退火采样策略。为了解决缺乏合适训练数据的问题，我们构建了一个 360-Motion 数据集，该数据集首先将收集的 3D 人类和动物资产与 GPT 生成的轨迹相关联，然后使用不同 3D UE 平台上的 12 个均匀环绕的摄像头捕捉它们的运动。大量实验表明，3DTrajMaster 在控制多实体 3D 运动的准确性和泛化方面都达到了新的最高水平。项目页面：此 http URL

Title: SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints

• Authors: Jianhong Bai, Menghan Xia, Xintao Wang, Ziyang Yuan, Xiao Fu, Zuozhu Liu, Haoji Hu, Pengfei Wan, Di Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07760
• Pdf URL: https://arxiv.org/pdf/2412.07760
• Copy Paste: [[2412.07760]] SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints(https://arxiv.org/abs/2412.07760)
• Keywords: generation
• Abstract: Recent advancements in video diffusion models have shown exceptional abilities in simulating real-world dynamics and maintaining 3D consistency. This progress inspires us to investigate the potential of these models to ensure dynamic consistency across various viewpoints, a highly desirable feature for applications such as virtual filming. Unlike existing methods focused on multi-view generation of single objects for 4D reconstruction, our interest lies in generating open-world videos from arbitrary viewpoints, incorporating 6 DoF camera poses. To achieve this, we propose a plug-and-play module that enhances a pre-trained text-to-video model for multi-camera video generation, ensuring consistent content across different viewpoints. Specifically, we introduce a multi-view synchronization module to maintain appearance and geometry consistency across these viewpoints. Given the scarcity of high-quality training data, we design a hybrid training scheme that leverages multi-camera images and monocular videos to supplement Unreal Engine-rendered multi-camera videos. Furthermore, our method enables intriguing extensions, such as re-rendering a video from novel viewpoints. We also release a multi-view synchronized video dataset, named SynCamVideo-Dataset. Project page: this https URL.
• 摘要：视频扩散模型的最新进展已显示出在模拟现实世界动态和保持 3D 一致性方面的卓越能力。这一进展激励我们研究这些模型的潜力，以确保跨各种视点的动态一致性，这对于虚拟拍摄等应用来说是一个非常理想的功能。与专注于为 4D 重建生成单个对象的多视图现有方法不同，我们的兴趣在于从任意视点生成开放世界视频，并结合 6 DoF 相机姿势。为了实现这一点，我们提出了一个即插即用模块，该模块增强了预训练的文本到视频模型，用于多摄像机视频生成，确保跨不同视点的内容一致。具体而言，我们引入了一个多视图同步模块，以保持这些视点的外观和几何一致性。鉴于高质量训练数据的稀缺性，我们设计了一种混合训练方案，利用多摄像机图像和单目视频来补充虚幻引擎渲染的多摄像机视频。此外，我们的方法还可以实现有趣的扩展，例如从新颖的视点重新渲染视频。我们还发布了一个多视角同步视频数据集，名为 SynCamVideo-Dataset。项目页面：这个 https URL。

Title: Make-A-Texture: Fast Shape-Aware Texture Generation in 3 Seconds

• Authors: Xiaoyu Xiang, Liat Sless Gorelik, Yuchen Fan, Omri Armstrong, Forrest Iandola, Yilei Li, Ita Lifshitz, Rakesh Ranjan
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2412.07766
• Pdf URL: https://arxiv.org/pdf/2412.07766
• Copy Paste: [[2412.07766]] Make-A-Texture: Fast Shape-Aware Texture Generation in 3 Seconds(https://arxiv.org/abs/2412.07766)
• Keywords: generation
• Abstract: We present Make-A-Texture, a new framework that efficiently synthesizes high-resolution texture maps from textual prompts for given 3D geometries. Our approach progressively generates textures that are consistent across multiple viewpoints with a depth-aware inpainting diffusion model, in an optimized sequence of viewpoints determined by an automatic view selection algorithm. A significant feature of our method is its remarkable efficiency, achieving a full texture generation within an end-to-end runtime of just 3.07 seconds on a single NVIDIA H100 GPU, significantly outperforming existing methods. Such an acceleration is achieved by optimizations in the diffusion model and a specialized backprojection method. Moreover, our method reduces the artifacts in the backprojection phase, by selectively masking out non-frontal faces, and internal faces of open-surfaced objects. Experimental results demonstrate that Make-A-Texture matches or exceeds the quality of other state-of-the-art methods. Our work significantly improves the applicability and practicality of texture generation models for real-world 3D content creation, including interactive creation and text-guided texture editing.
• 摘要：我们提出了 Make-A-Texture，这是一个新框架，可从给定 3D 几何的文本提示中高效合成高分辨率纹理图。我们的方法使用深度感知修复扩散模型逐步生成跨多个视点一致的纹理，并按照由自动视图选择算法确定的优化视点序列生成纹理。我们方法的一个显著特点是其卓越的效率，在单个 NVIDIA H100 GPU 上仅需 3.07 秒即可在端到端运行时间内完成完整的纹理生成，性能显著优于现有方法。这种加速是通过扩散模型中的优化和专门的反投影方法实现的。此外，我们的方法通过选择性地遮盖非正面和开放表面物体的内部面来减少反投影阶段的伪影。实验结果表明，Make-A-Texture 达到或超过了其他最先进方法的质量。我们的工作显著提高了纹理生成模型对于现实世界 3D 内容创作的适用性和实用性，包括交互式创作和文本引导的纹理编辑。

Title: Learning Visual Generative Priors without Text

• Authors: Shuailei Ma, Kecheng Zheng, Ying Wei, Wei Wu, Fan Lu, Yifei Zhang, Chen-wei Xie, Jiapeng Zhu, Yujun Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07767
• Pdf URL: https://arxiv.org/pdf/2412.07767
• Copy Paste: [[2412.07767]] Learning Visual Generative Priors without Text(https://arxiv.org/abs/2412.07767)
• Keywords: generation, generative
• Abstract: Although text-to-image (T2I) models have recently thrived as visual generative priors, their reliance on high-quality text-image pairs makes scaling up expensive. We argue that grasping the cross-modality alignment is not a necessity for a sound visual generative prior, whose focus should be on texture modeling. Such a philosophy inspires us to study image-to-image (I2I) generation, where models can learn from in-the-wild images in a self-supervised manner. We first develop a pure vision-based training framework, Lumos, and confirm the feasibility and the scalability of learning I2I models. We then find that, as an upstream task of T2I, our I2I model serves as a more foundational visual prior and achieves on-par or better performance than existing T2I models using only 1/10 text-image pairs for fine-tuning. We further demonstrate the superiority of I2I priors over T2I priors on some text-irrelevant visual generative tasks, like image-to-3D and image-to-video.
• 摘要：尽管文本到图像 (T2I) 模型最近作为视觉生成先验蓬勃发展，但它们对高质量文本-图像对的依赖使得扩大规模的成本很高。我们认为，掌握跨模态对齐并不是健全的视觉生成先验的必要条件，其重点应该放在纹理建模上。这种理念启发我们研究图像到图像 (I2I) 生成，其中模型可以以自监督的方式从自然图像中学习。我们首先开发了一个纯视觉训练框架 Lumos，并确认了学习 I2I 模型的可行性和可扩展性。然后我们发现，作为 T2I 的上游任务，我们的 I2I 模型充当了更基础的视觉先验，并且仅使用 1/10 的文本-图像对进行微调即可实现与现有 T2I 模型相当或更好的性能。我们进一步证明了 I2I 先验在一些与文本无关的视觉生成任务（例如图像到 3D 和图像到视频）上优于 T2I 先验。

Title: BiMediX2: Bio-Medical EXpert LMM for Diverse Medical Modalities

• Authors: Sahal Shaji Mullappilly, Mohammed Irfan Kurpath, Sara Pieri, Saeed Yahya Alseiari, Shanavas Cholakkal, Khaled Aldahmani, Fahad Khan, Rao Anwer, Salman Khan, Timothy Baldwin, Hisham Cholakkal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07769
• Pdf URL: https://arxiv.org/pdf/2412.07769
• Copy Paste: [[2412.07769]] BiMediX2: Bio-Medical EXpert LMM for Diverse Medical Modalities(https://arxiv.org/abs/2412.07769)
• Keywords: generation
• Abstract: This paper introduces BiMediX2, a bilingual (Arabic-English) Bio-Medical EXpert Large Multimodal Model (LMM) with a unified architecture that integrates text and visual modalities, enabling advanced image understanding and medical applications. BiMediX2 leverages the Llama3.1 architecture and integrates text and visual capabilities to facilitate seamless interactions in both English and Arabic, supporting text-based inputs and multi-turn conversations involving medical images. The model is trained on an extensive bilingual healthcare dataset consisting of 1.6M samples of diverse medical interactions for both text and image modalities, mixed in Arabic and English. We also propose the first bilingual GPT-4o based medical LMM benchmark named BiMed-MBench. BiMediX2 is benchmarked on both text-based and image-based tasks, achieving state-of-the-art performance across several medical benchmarks. It outperforms recent state-of-the-art models in medical LLM evaluation benchmarks. Our model also sets a new benchmark in multimodal medical evaluations with over 9% improvement in English and over 20% in Arabic evaluations. Additionally, it surpasses GPT-4 by around 9% in UPHILL factual accuracy evaluations and excels in various medical Visual Question Answering, Report Generation, and Report Summarization tasks. The project page including source code and the trained model, is available at this https URL.
• 摘要：本文介绍了 BiMediX2，这是一种双语（阿拉伯语-英语）生物医学专家大型多模态模型 (LMM)，具有统一的架构，集成了文本和视觉模态，可实现高级图像理解和医学应用。BiMediX2 利用 Llama3.1 架构并集成文本和视觉功能，以促进英语和阿拉伯语的无缝交互，支持基于文本的输入和涉及医学图像的多轮对话。该模型在一个广泛的双语医疗保健数据集上进行训练，该数据集包含 160 万个针对文本和图像模态的各种医学交互样本，混合了阿拉伯语和英语。我们还提出了第一个基于双语 GPT-4o 的医学 LMM 基准，名为 BiMed-MBench。BiMediX2 在基于文本和基于图像的任务上都进行了基准测试，在多个医学基准上取得了最先进的性能。它在医学 LLM 评估基准中的表现优于最近最先进的模型。我们的模型还在多模态医学评估中树立了新的标杆，英语评估提高了 9% 以上，阿拉伯语评估提高了 20% 以上。此外，它在 UPHILL 事实准确性评估中比 GPT-4 高出约 9%，并在各种医学视觉问答、报告生成和报告摘要任务中表现出色。项目页面包括源代码和经过训练的模型，可在此 https URL 上找到。

Title: From Slow Bidirectional to Fast Causal Video Generators

• Authors: Tianwei Yin, Qiang Zhang, Richard Zhang, William T. Freeman, Fredo Durand, Eli Shechtman, Xun Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07772
• Pdf URL: https://arxiv.org/pdf/2412.07772
• Copy Paste: [[2412.07772]] From Slow Bidirectional to Fast Causal Video Generators(https://arxiv.org/abs/2412.07772)
• Keywords: generation
• Abstract: Current video diffusion models achieve impressive generation quality but struggle in interactive applications due to bidirectional attention dependencies. The generation of a single frame requires the model to process the entire sequence, including the future. We address this limitation by adapting a pretrained bidirectional diffusion transformer to a causal transformer that generates frames on-the-fly. To further reduce latency, we extend distribution matching distillation (DMD) to videos, distilling 50-step diffusion model into a 4-step generator. To enable stable and high-quality distillation, we introduce a student initialization scheme based on teacher's ODE trajectories, as well as an asymmetric distillation strategy that supervises a causal student model with a bidirectional teacher. This approach effectively mitigates error accumulation in autoregressive generation, allowing long-duration video synthesis despite training on short clips. Our model supports fast streaming generation of high quality videos at 9.4 FPS on a single GPU thanks to KV caching. Our approach also enables streaming video-to-video translation, image-to-video, and dynamic prompting in a zero-shot manner. We will release the code based on an open-source model in the future.
• 摘要：当前的视频扩散模型实现了令人印象深刻的生成质量，但由于双向注意力依赖性，在交互式应用中遇到了困难。生成单个帧需要模型处理整个序列，包括未来。我们通过将预训练的双向扩散变压器调整为即时生成帧的因果变压器来解决这一限制。为了进一步减少延迟，我们将分布匹配蒸馏 (DMD) 扩展到视频，将 50 步扩散模型蒸馏为 4 步生成器。为了实现稳定和高质量的蒸馏，我们引入了基于教师 ODE 轨迹的学生初始化方案，以及监督具有双向教师的因果学生模型的非对称蒸馏策略。这种方法有效地减轻了自回归生成中的错误积累，尽管在短片段上进行训练，但仍允许长时间的视频合成。得益于 KV 缓存，我们的模型支持在单个 GPU 上以 9.4 FPS 的速度快速流式生成高质量视频。我们的方法还可以实现零样本流式视频到视频的翻译、图像到视频和动态提示。我们将在未来基于开源模型发布代码。

Title: UniReal: Universal Image Generation and Editing via Learning Real-world Dynamics

• Authors: Xi Chen, Zhifei Zhang, He Zhang, Yuqian Zhou, Soo Ye Kim, Qing Liu, Yijun Li, Jianming Zhang, Nanxuan Zhao, Yilin Wang, Hui Ding, Zhe Lin, Hengshuang Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07774
• Pdf URL: https://arxiv.org/pdf/2412.07774
• Copy Paste: [[2412.07774]] UniReal: Universal Image Generation and Editing via Learning Real-world Dynamics(https://arxiv.org/abs/2412.07774)
• Keywords: generation
• Abstract: We introduce UniReal, a unified framework designed to address various image generation and editing tasks. Existing solutions often vary by tasks, yet share fundamental principles: preserving consistency between inputs and outputs while capturing visual variations. Inspired by recent video generation models that effectively balance consistency and variation across frames, we propose a unifying approach that treats image-level tasks as discontinuous video generation. Specifically, we treat varying numbers of input and output images as frames, enabling seamless support for tasks such as image generation, editing, customization, composition, etc. Although designed for image-level tasks, we leverage videos as a scalable source for universal supervision. UniReal learns world dynamics from large-scale videos, demonstrating advanced capability in handling shadows, reflections, pose variation, and object interaction, while also exhibiting emergent capability for novel applications.
• 摘要：我们推出了 UniReal，这是一个统一的框架，旨在解决各种图像生成和编辑任务。现有的解决方案通常因任务而异，但基本原则是相同的：在捕捉视觉变化的同时保持输入和输出之间的一致性。受最近视频生成模型的启发，这些模型有效地平衡了帧之间的一致性和变化，我们提出了一种统一的方法，将图像级任务视为不连续的视频生成。具体来说，我们将不同数量的输入和输出图像视为帧，从而无缝支持图像生成、编辑、自定义、合成等任务。虽然是为图像级任务设计的，但我们利用视频作为通用监督的可扩展来源。UniReal 从大规模视频中学习世界动态，展示了处理阴影、反射、姿势变化和物体交互的高级能力，同时也展示了用于新应用的新兴能力。

Title: Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNets

• Authors: Zhen Liu, Tim Z. Xiao, Weiyang Liu, Yoshua Bengio, Dinghuai Zhang
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2412.07775
• Pdf URL: https://arxiv.org/pdf/2412.07775
• Copy Paste: [[2412.07775]] Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNets(https://arxiv.org/abs/2412.07775)
• Keywords: generative
• Abstract: While one commonly trains large diffusion models by collecting datasets on target downstream tasks, it is often desired to align and finetune pretrained diffusion models on some reward functions that are either designed by experts or learned from small-scale datasets. Existing methods for finetuning diffusion models typically suffer from lack of diversity in generated samples, lack of prior preservation, and/or slow convergence in finetuning. Inspired by recent successes in generative flow networks (GFlowNets), a class of probabilistic models that sample with the unnormalized density of a reward function, we propose a novel GFlowNet method dubbed Nabla-GFlowNet (abbreviated as $\nabla$-GFlowNet), the first GFlowNet method that leverages the rich signal in reward gradients, together with an objective called $\nabla$-DB plus its variant residual $\nabla$-DB designed for prior-preserving diffusion alignment. We show that our proposed method achieves fast yet diversity- and prior-preserving alignment of Stable Diffusion, a large-scale text-conditioned image diffusion model, on different realistic reward functions.
• 摘要：虽然人们通常通过收集目标下游任务的数据集来训练大型扩散模型，但通常需要在某些由专家设计或从小规模数据集中学习的奖励函数上对齐和微调预训练的扩散模型。现有的微调扩散模型的方法通常存在生成样本缺乏多样性、缺乏先验保存和/或微调收敛缓慢的问题。受生成流网络 (GFlowNets) 近期成功的启发，GFlowNets 是一类使用奖励函数的非归一化密度进行采样的概率模型，我们提出了一种新颖的 GFlowNet 方法，称为 Nabla-GFlowNet（缩写为 $\nabla$-GFlowNet），这是第一个利用奖励梯度中丰富信号的 GFlowNet 方法，以及一个名为 $\nabla$-DB 的目标及其变体残差 $\nabla$-DB，旨在进行先验保存扩散对齐。我们表明，我们提出的方法在不同的实际奖励函数上实现了稳定扩散（一种大规模文本条件图像扩散模型）的快速且保持多样性和先验性的对齐。