2025-03-28

Title: Unified Multimodal Discrete Diffusion

• Authors: Alexander Swerdlow, Mihir Prabhudesai, Siddharth Gandhi, Deepak Pathak, Katerina Fragkiadaki
• Subjects: cs.CV, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2503.20853
• Pdf URL: https://arxiv.org/pdf/2503.20853
• Copy Paste: [[2503.20853]] Unified Multimodal Discrete Diffusion(https://arxiv.org/abs/2503.20853)
• Keywords: generation, generative
• Abstract: Multimodal generative models that can understand and generate across multiple modalities are dominated by autoregressive (AR) approaches, which process tokens sequentially from left to right, or top to bottom. These models jointly handle images, text, video, and audio for various tasks such as image captioning, question answering, and image generation. In this work, we explore discrete diffusion models as a unified generative formulation in the joint text and image domain, building upon their recent success in text generation. Discrete diffusion models offer several advantages over AR models, including improved control over quality versus diversity of generated samples, the ability to perform joint multimodal inpainting (across both text and image domains), and greater controllability in generation through guidance. Leveraging these benefits, we present the first Unified Multimodal Discrete Diffusion (UniDisc) model which is capable of jointly understanding and generating text and images for a variety of downstream tasks. We compare UniDisc to multimodal AR models, performing a scaling analysis and demonstrating that UniDisc outperforms them in terms of both performance and inference-time compute, enhanced controllability, editability, inpainting, and flexible trade-off between inference time and generation quality. Code and additional visualizations are available at this https URL.
• 摘要：可以理解和生成多种模式的多模式生成模型以自回归（AR）方法为主，这些方法从左到右或上或底部或底部或底部依次处理令牌。这些模型共同处理图像，文本，视频和音频，以进行各种任务，例如图像字幕，问答和图像生成。在这项工作中，我们将离散的扩散模型作为联合文本和图像域中的统一生成式公式，基于它们最近在文本生成方面的成功。离散扩散模型比AR模型具有多个优点，包括对生成样品的质量与多样性的控制，可以执行联合多模式授课的能力（跨文本和图像域），以及通过指导的生成可控性更大。利用这些好处，我们提出了第一个统一的多模式离散扩散（UNIDISC）模型，该模型能够共同理解和生成各种下游任务的文本和图像。我们将UniDISC与多模式AR模型进行比较，进行缩放分析，并证明UniDISC在性能和推理时间计算，增强的可控性，可控性，可编辑性，介绍以及推理时间和发电质量之间的灵活权衡方面优于它们。该HTTPS URL可用代码和其他可视化。

Title: VinaBench: Benchmark for Faithful and Consistent Visual Narratives

• Authors: Silin Gao, Sheryl Mathew, Li Mi, Sepideh Mamooler, Mengjie Zhao, Hiromi Wakaki, Yuki Mitsufuji, Syrielle Montariol, Antoine Bosselut
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2503.20871
• Pdf URL: https://arxiv.org/pdf/2503.20871
• Copy Paste: [[2503.20871]] VinaBench: Benchmark for Faithful and Consistent Visual Narratives(https://arxiv.org/abs/2503.20871)
• Keywords: generation, generative
• Abstract: Visual narrative generation transforms textual narratives into sequences of images illustrating the content of the text. However, generating visual narratives that are faithful to the input text and self-consistent across generated images remains an open challenge, due to the lack of knowledge constraints used for planning the stories. In this work, we propose a new benchmark, VinaBench, to address this challenge. Our benchmark annotates the underlying commonsense and discourse constraints in visual narrative samples, offering systematic scaffolds for learning the implicit strategies of visual storytelling. Based on the incorporated narrative constraints, we further propose novel metrics to closely evaluate the consistency of generated narrative images and the alignment of generations with the input textual narrative. Our results across three generative vision models demonstrate that learning with VinaBench's knowledge constraints effectively improves the faithfulness and cohesion of generated visual narratives.
• 摘要：视觉叙事生成将文本叙述转换为图像序列，说明了文本的内容。但是，由于缺乏用于计划故事的知识限制，生成忠于输入文本和自洽的视觉叙事仍然是一个挑战。在这项工作中，我们提出了一个新的基准Vinabench，以应对这一挑战。我们的基准标记了视觉叙事样本中的基本常识和话语约束，提供了系统的脚手架来学习视觉讲故事的隐式策略。基于合并的叙述约束，我们进一步提出了新颖的指标，以密切评估生成的叙事图像的一致性以及世代与输入文本叙述的一致性。我们在三种生成视觉模型中的结果表明，通过Vinabench的知识约束学习可以有效地改善产生的视觉叙事的忠诚和凝聚力。

Title: Assessing Generative Models for Structured Data

• Authors: Reilly Cannon, Nicolette M. Laird, Caesar Vazquez, Andy Lin, Amy Wagler, Tony Chiang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.20903
• Pdf URL: https://arxiv.org/pdf/2503.20903
• Copy Paste: [[2503.20903]] Assessing Generative Models for Structured Data(https://arxiv.org/abs/2503.20903)
• Keywords: generation, generative
• Abstract: Synthetic tabular data generation has emerged as a promising method to address limited data availability and privacy concerns. With the sharp increase in the performance of large language models in recent years, researchers have been interested in applying these models to the generation of tabular data. However, little is known about the quality of the generated tabular data from large language models. The predominant method for assessing the quality of synthetic tabular data is the train-synthetic-test-real approach, where the artificial examples are compared to the original by how well machine learning models, trained separately on the real and synthetic sets, perform in some downstream tasks. This method does not directly measure how closely the distribution of generated data approximates that of the original. This paper introduces rigorous methods for directly assessing synthetic tabular data against real data by looking at inter-column dependencies within the data. We find that large language models (GPT-2), both when queried via few-shot prompting and when fine-tuned, and GAN (CTGAN) models do not produce data with dependencies that mirror the original real data. Results from this study can inform future practice in synthetic data generation to improve data quality.
• 摘要：合成表格数据生成已成为一种有前途的方法，可以解决有限的数据可用性和隐私问题。随着近年来大语言模型的性能的急剧提高，研究人员对将这些模型应用于表格数据的生成感兴趣。但是，对于来自大语言模型生成的表格数据的质量知之甚少。评估合成表格数据质量的主要方法是火车合成检验的方法，其中人造示例与原始示例与原始示例通过对真实和合成集的机器学习模型的良好方式进行比较，并在某些下游任务中执行。该方法无法直接测量生成的数据的分布近似于原始数据的分布。本文通过查看数据中的列依赖项，介绍了用于直接评估合成表格数据的严格方法。我们发现，当通过几次弹药提示和微调时查询时，大型语言模型（GPT-2）和GAN（CTGAN）模型都不会产生具有反映原始真实数据的依赖关系的数据。这项研究的结果可以为未来的综合数据生成实践提供信息，以提高数据质量。

Title: Prototype Guided Backdoor Defense

• Authors: Venkat Adithya Amula, Sunayana Samavedam, Saurabh Saini, Avani Gupta, Narayanan P J
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.20925
• Pdf URL: https://arxiv.org/pdf/2503.20925
• Copy Paste: [[2503.20925]] Prototype Guided Backdoor Defense(https://arxiv.org/abs/2503.20925)
• Keywords: generation, generative
• Abstract: Deep learning models are susceptible to {\em backdoor attacks} involving malicious attackers perturbing a small subset of training data with a {\em trigger} to causes misclassifications. Various triggers have been used, including semantic triggers that are easily realizable without requiring the attacker to manipulate the image. The emergence of generative AI has eased the generation of varied poisoned samples. Robustness across types of triggers is crucial to effective defense. We propose Prototype Guided Backdoor Defense (PGBD), a robust post-hoc defense that scales across different trigger types, including previously unsolved semantic triggers. PGBD exploits displacements in the geometric spaces of activations to penalize movements toward the trigger. This is done using a novel sanitization loss of a post-hoc fine-tuning step. The geometric approach scales easily to all types of attacks. PGBD achieves better performance across all settings. We also present the first defense against a new semantic attack on celebrity face images. Project page: \hyperlink{this https URL}{this https URL}.
• 摘要：深度学习模型容易受到{\ em后门攻击}的影响，涉及恶意攻击者，扰乱了一小部分培训数据，以{\ em Trigger}导致错误分类。已经使用了各种触发器，包括易于实现的语义触发器，而无需攻击者操纵图像。生成AI的出现缓解了各种毒药的产生。跨类型触发器的鲁棒性对于有效的防御至关重要。我们提出了原型后门防御（PGBD）的原型，这是一种稳健的事后防御，跨越了不同的触发类型，包括以前未解决的语义触发器。 PGBD利用激活的几何空间中的位移，以惩罚向触发器的运动。这是使用新的卫生后微调步骤的新型消毒损失来完成的。几何方法很容易缩放到所有类型的攻击。 PGBD在所有设置中都能取得更好的性能。我们还提出了针对名人面部图像的新语义攻击的首次防御。项目页面：\超链接{此https url} {this https url}。

Title: Eyes Tell the Truth: GazeVal Highlights Shortcomings of Generative AI in Medical Imaging

• Authors: David Wong, Bin Wang, Gorkem Durak, Marouane Tliba, Akshay Chaudhari, Aladine Chetouani, Ahmet Enis Cetin, Cagdas Topel, Nicolo Gennaro, Camila Lopes Vendrami, Tugce Agirlar Trabzonlu, Amir Ali Rahsepar, Laetitia Perronne, Matthew Antalek, Onural Ozturk, Gokcan Okur, Andrew C. Gordon, Ayis Pyrros, Frank H. Miller, Amir Borhani, Hatice Savas, Eric Hart, Drew Torigian, Jayaram K. Udupa, Elizabeth Krupinski, Ulas Bagci
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.20967
• Pdf URL: https://arxiv.org/pdf/2503.20967
• Copy Paste: [[2503.20967]] Eyes Tell the Truth: GazeVal Highlights Shortcomings of Generative AI in Medical Imaging(https://arxiv.org/abs/2503.20967)
• Keywords: generative
• Abstract: The demand for high-quality synthetic data for model training and augmentation has never been greater in medical imaging. However, current evaluations predominantly rely on computational metrics that fail to align with human expert recognition. This leads to synthetic images that may appear realistic numerically but lack clinical authenticity, posing significant challenges in ensuring the reliability and effectiveness of AI-driven medical tools. To address this gap, we introduce GazeVal, a practical framework that synergizes expert eye-tracking data with direct radiological evaluations to assess the quality of synthetic medical images. GazeVal leverages gaze patterns of radiologists as they provide a deeper understanding of how experts perceive and interact with synthetic data in different tasks (i.e., diagnostic or Turing tests). Experiments with sixteen radiologists revealed that 96.6% of the generated images (by the most recent state-of-the-art AI algorithm) were identified as fake, demonstrating the limitations of generative AI in producing clinically accurate images.
• 摘要：在医学成像中，对模型训练和增强的高质量合成数据的需求从未有所更大。但是，当前的评估主要依赖于无法与人类专家认可保持一致的计算指标。这导致合成图像在数字上可能看起来很现实，但缺乏临床真实性，这在确保AI驱动的医疗工具的可靠性和有效性方面构成了重大挑战。为了解决这一差距，我们引入了Gazeval，这是一个实用的框架，可以通过直接放射学评估来协同专家追踪数据，以评估合成医学图像的质量。 Gazeval利用放射科医生的凝视模式，因为它们对专家如何感知和与不同任务（即诊断或图灵测试）中的合成数据进行更深入的了解。对十六位放射科医生进行的实验表明，有96.6％的生成图像（通过最新的最新AI算法）被确定为假货，这表明生成AI在产生临床准确的图像中的局限性。

Title: Forensic Self-Descriptions Are All You Need for Zero-Shot Detection, Open-Set Source Attribution, and Clustering of AI-generated Images

• Authors: Tai D. Nguyen, Aref Azizpour, Matthew C. Stamm
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21003
• Pdf URL: https://arxiv.org/pdf/2503.21003
• Copy Paste: [[2503.21003]] Forensic Self-Descriptions Are All You Need for Zero-Shot Detection, Open-Set Source Attribution, and Clustering of AI-generated Images(https://arxiv.org/abs/2503.21003)
• Keywords: generative
• Abstract: The emergence of advanced AI-based tools to generate realistic images poses significant challenges for forensic detection and source attribution, especially as new generative techniques appear rapidly. Traditional methods often fail to generalize to unseen generators due to reliance on features specific to known sources during training. To address this problem, we propose a novel approach that explicitly models forensic microstructures - subtle, pixel-level patterns unique to the image creation process. Using only real images in a self-supervised manner, we learn a set of diverse predictive filters to extract residuals that capture different aspects of these microstructures. By jointly modeling these residuals across multiple scales, we obtain a compact model whose parameters constitute a unique forensic self-description for each image. This self-description enables us to perform zero-shot detection of synthetic images, open-set source attribution of images, and clustering based on source without prior knowledge. Extensive experiments demonstrate that our method achieves superior accuracy and adaptability compared to competing techniques, advancing the state of the art in synthetic media forensics.
• 摘要：基于AI的先进工具的出现为逼真的图像带来了大量的挑战，即法医检测和源归因，尤其是随着新的生成技术迅速出现。由于依靠培训期间特定的已知来源功能，传统方法通常无法推广到看不见的发电机。为了解决这个问题，我们提出了一种新颖的方法，该方法明确模拟了法医微结构 - 微妙的像素级模式是图像创建过程所特有的。我们仅以自我监督的方式使用真实的图像，我们学习一组不同的预测过滤器来提取捕获这些微观结构不同方面的残差。通过在多个尺度上共同建模这些残差，我们获得了一个紧凑的模型，其参数构成了每个图像的独特法医自我描述。这种自我描述使我们能够对合成图像，图像的开放式源归因和基于源的聚类进行零摄像检测，而没有事先知识。广泛的实验表明，与竞争技术相比，我们的方法可以达到卓越的准确性和适应性，从而在合成媒体取证中提高了最新技术。

Title: Efficient Multi-Instance Generation with Janus-Pro-Dirven Prompt Parsing

• Authors: Fan Qi, Yu Duan, Changsheng Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21069
• Pdf URL: https://arxiv.org/pdf/2503.21069
• Copy Paste: [[2503.21069]] Efficient Multi-Instance Generation with Janus-Pro-Dirven Prompt Parsing(https://arxiv.org/abs/2503.21069)
• Keywords: generation
• Abstract: Recent advances in text-guided diffusion models have revolutionized conditional image generation, yet they struggle to synthesize complex scenes with multiple objects due to imprecise spatial grounding and limited scalability. We address these challenges through two key modules: 1) Janus-Pro-driven Prompt Parsing, a prompt-layout parsing module that bridges text understanding and layout generation via a compact 1B-parameter architecture, and 2) MIGLoRA, a parameter-efficient plug-in integrating Low-Rank Adaptation (LoRA) into UNet (SD1.5) and DiT (SD3) backbones. MIGLoRA is capable of preserving the base model's parameters and ensuring plug-and-play adaptability, minimizing architectural intrusion while enabling efficient fine-tuning. To support a comprehensive evaluation, we create DescripBox and DescripBox-1024, benchmarks that span diverse scenes and resolutions. The proposed method achieves state-of-the-art performance on COCO and LVIS benchmarks while maintaining parameter efficiency, demonstrating superior layout fidelity and scalability for open-world synthesis.
• 摘要：文本引导的扩散模型的最新进展彻底改变了条件形象的产生，但是由于不精确的空间接地和有限的可扩展性，它们很难与多个物体合成复杂场景。我们通过两个关键模块解决了这些挑战：1）Janus-Pro驱动的提示解析，这是一个及时的解析模块，通过紧凑的1B参数架构将文本理解和布局生成桥接，以及2）Miglora，Miglora，一个有力的参数插入插入式插件插入式插件低级适应（LORA）中的后卫（LORA）中的后卫（LORA）中的后卫（LORA）和DIT（SD1.5）和DIT（SD1.5）和DIT（sd1.5）和DIT（SD）和DIT（SD）和DIT（SD）和DIT（SD）和DIT（SD）和DIT。 Miglora能够保留基本模型的参数并确保插件的适应性，从而最大程度地减少体系结构的入侵，同时有效地进行微调。为了支持全面的评估，我们创建了描述框和descripbox-1024，即跨越各种场景和分辨率的基准测试。所提出的方法在维持参数效率的同时，在可可和LVIS基准测试方面实现了最新的性能，证明了出色的布局保真度和开放世界合成的可扩展性。

Title: One Snapshot is All You Need: A Generalized Method for mmWave Signal Generation

• Authors: Teng Huang, Han Ding, Wenxin Sun, Cui Zhao, Ge Wang, Fei Wang, Kun Zhao, Zhi Wang, Wei Xi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21122
• Pdf URL: https://arxiv.org/pdf/2503.21122
• Copy Paste: [[2503.21122]] One Snapshot is All You Need: A Generalized Method for mmWave Signal Generation(https://arxiv.org/abs/2503.21122)
• Keywords: generation
• Abstract: Wireless sensing systems, particularly those using mmWave technology, offer distinct advantages over traditional vision-based approaches, such as enhanced privacy and effectiveness in poor lighting conditions. These systems, leveraging FMCW signals, have shown success in human-centric applications like localization, gesture recognition, and so on. However, comprehensive mmWave datasets for diverse applications are scarce, often constrained by pre-processed signatures (e.g., point clouds or RA heatmaps) and inconsistent annotation formats. To overcome these limitations, we propose mmGen, a novel and generalized framework tailored for full-scene mmWave signal generation. By constructing physical signal transmission models, mmGen synthesizes human-reflected and environment-reflected mmWave signals from the constructed 3D meshes. Additionally, we incorporate methods to account for material properties, antenna gains, and multipath reflections, enhancing the realism of the synthesized signals. We conduct extensive experiments using a prototype system with commercial mmWave devices and Kinect sensors. The results show that the average similarity of Range-Angle and micro-Doppler signatures between the synthesized and real-captured signals across three different environments exceeds 0.91 and 0.89, respectively, demonstrating the effectiveness and practical applicability of mmGen.
• 摘要：无线传感系统，尤其是使用MMWave技术的系统，比传统的基于视觉的方法具有明显的优势，例如在较差的照明条件下的隐私和有效性增强。这些利用FMCW信号的系统在以人为中心的应用，诸如本地化，手势识别等等方面的方式显示了成功。但是，用于不同应用程序的综合MMWAVE数据集稀缺，通常受到预处理的签名（例如，点云或RA热图）和不一致的注释格式的限制。为了克服这些局限性，我们提出了MMGEN，这是一种针对全景MMWave信号生成的新颖而广义的框架。通过构建物理信号传输模型，MMGEN从构造的3D网格中合成了由人类反射和环境反射的MMWave信号。此外，我们结合了说明材料特性，天线增益和多径反射的方法，从而增强了合成信号的现实性。我们使用带有商用MMWave设备和Kinect传感器的原型系统进行广泛的实验。结果表明，在三种不同环境中综合和实际捕捉信号之间的范围角和微数签名的平均相似性分别超过0.91和0.89，这表明了MMGER的有效性和实际适用性。

Title: ChatAnyone: Stylized Real-time Portrait Video Generation with Hierarchical Motion Diffusion Model

• Authors: Jinwei Qi, Chaonan Ji, Sheng Xu, Peng Zhang, Bang Zhang, Liefeng Bo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21144
• Pdf URL: https://arxiv.org/pdf/2503.21144
• Copy Paste: [[2503.21144]] ChatAnyone: Stylized Real-time Portrait Video Generation with Hierarchical Motion Diffusion Model(https://arxiv.org/abs/2503.21144)
• Keywords: generation
• Abstract: Real-time interactive video-chat portraits have been increasingly recognized as the future trend, particularly due to the remarkable progress made in text and voice chat technologies. However, existing methods primarily focus on real-time generation of head movements, but struggle to produce synchronized body motions that match these head actions. Additionally, achieving fine-grained control over the speaking style and nuances of facial expressions remains a challenge. To address these limitations, we introduce a novel framework for stylized real-time portrait video generation, enabling expressive and flexible video chat that extends from talking head to upper-body interaction. Our approach consists of the following two stages. The first stage involves efficient hierarchical motion diffusion models, that take both explicit and implicit motion representations into account based on audio inputs, which can generate a diverse range of facial expressions with stylistic control and synchronization between head and body movements. The second stage aims to generate portrait video featuring upper-body movements, including hand gestures. We inject explicit hand control signals into the generator to produce more detailed hand movements, and further perform face refinement to enhance the overall realism and expressiveness of the portrait video. Additionally, our approach supports efficient and continuous generation of upper-body portrait video in maximum 512 * 768 resolution at up to 30fps on 4090 GPU, supporting interactive video-chat in real-time. Experimental results demonstrate the capability of our approach to produce portrait videos with rich expressiveness and natural upper-body movements.
• 摘要：实时互动视频奇特肖像越来越多地被认为是未来趋势，尤其是由于文本和语音聊天技术取得的显着进展。但是，现有的方法主要集中于实时发动的头部运动，但努力产生与这些头部动作相匹配的同步身体运动。此外，实现对口语风格和面部表情细微差别的细粒度控制仍然是一个挑战。为了解决这些局限性，我们介绍了一个新颖的框架，用于定型实时肖像视频生成，从而使表达和灵活的视频聊天从说话的头到上身互动延伸。我们的方法包括以下两个阶段。第一阶段涉及有效的层次运动扩散模型，这些模型同时根据音频输入来考虑明确的和隐式运动表示，这可以在头部和身体运动之间产生各种面部表达式，并具有各种面部表达式。第二阶段的目的是生成具有上身动作（包括手势）的肖像视频。我们将明确的手控制信号注入发电机，以产生更详细的手动运动，并进一步执行面部改进，以增强肖像视频的整体现实主义和表现力。此外，我们的方法还支持最多512 * 768分辨率在4090 GPU上以高达30fps的分辨率的高效，连续生成的上下体肖像视频，从而实时支持交互式视频聊天。实验结果证明了我们的方法能够以丰富的表现力和自然的上身运动来制作肖像视频。

Title: Real-Time Evaluation Models for RAG: Who Detects Hallucinations Best?

• Authors: Ashish Sardana
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2503.21157
• Pdf URL: https://arxiv.org/pdf/2503.21157
• Copy Paste: [[2503.21157]] Real-Time Evaluation Models for RAG: Who Detects Hallucinations Best?(https://arxiv.org/abs/2503.21157)
• Keywords: generation
• Abstract: This article surveys Evaluation models to automatically detect hallucinations in Retrieval-Augmented Generation (RAG), and presents a comprehensive benchmark of their performance across six RAG applications. Methods included in our study include: LLM-as-a-Judge, Prometheus, Lynx, the Hughes Hallucination Evaluation Model (HHEM), and the Trustworthy Language Model (TLM). These approaches are all reference-free, requiring no ground-truth answers/labels to catch incorrect LLM responses. Our study reveals that, across diverse RAG applications, some of these approaches consistently detect incorrect RAG responses with high precision/recall.
• 摘要：本文调查了评估模型，以自动检测到检索增强发电（RAG）中的幻觉，并在六个抹布应用中呈现其性能的全面基准。我们的研究中包括的方法包括：LLM-AS-A-Gudge，Prometheus，Lynx，Hughes幻觉评估模型（HHEM）和可信赖的语言模型（TLM）。这些方法都是无参考的，不需要基本真相的答案/标签来收集错误的LLM响应。我们的研究表明，在不同的抹布应用中，其中一些方法始终以高精度/回忆来检测不正确的破布响应。

Title: Integrating Travel Behavior Forecasting and Generative Modeling for Predicting Future Urban Mobility and Spatial Transformations

• Authors: Eugene Denteh, Andrews Danyo, Joshua Kofi Asamoah, Blessing Agyei Kyem, Twitchell Addai, Armstrong Aboah
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21158
• Pdf URL: https://arxiv.org/pdf/2503.21158
• Copy Paste: [[2503.21158]] Integrating Travel Behavior Forecasting and Generative Modeling for Predicting Future Urban Mobility and Spatial Transformations(https://arxiv.org/abs/2503.21158)
• Keywords: generative
• Abstract: Transportation planning plays a critical role in shaping urban development, economic mobility, and infrastructure sustainability. However, traditional planning methods often struggle to accurately predict long-term urban growth and transportation demands. This may sometimes result in infrastructure demolition to make room for current transportation planning demands. This study integrates a Temporal Fusion Transformer to predict travel patterns from demographic data with a Generative Adversarial Network to predict future urban settings through satellite imagery. The framework achieved a 0.76 R-square score in travel behavior prediction and generated high-fidelity satellite images with a Structural Similarity Index of 0.81. The results demonstrate that integrating predictive analytics and spatial visualization can significantly improve the decision-making process, fostering more sustainable and efficient urban development. This research highlights the importance of data-driven methodologies in modern transportation planning and presents a step toward optimizing infrastructure placement, capacity, and long-term viability.
• 摘要：运输计划在塑造城市发展，经济流动性和基础设施可持续性中起着至关重要的作用。但是，传统的计划方法通常难以准确预测长期的城市增长和运输需求。这有时可能导致基础设施拆除，以腾出当前的运输计划需求。这项研究集成了时间融合变压器，以通过卫星图像预测未来的城市环境，以预测人口统计数据的旅行模式。该框架在行进行为预测中达到了0.76 R平方的得分，并产生了具有0.81的结构相似性指数的高保真卫星图像。结果表明，整合预测分析和空间可视化可以显着改善决策过程，从而促进更可持续和有效的城市发展。这项研究强调了数据驱动方法在现代运输计划中的重要性，并提出了优化基础设施，容量和长期生存能力的一步。

Title: Model as a Game: On Numerical and Spatial Consistency for Generative Games

• Authors: Jingye Chen, Yuzhong Zhao, Yupan Huang, Lei Cui, Li Dong, Tengchao Lv, Qifeng Chen, Furu Wei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21172
• Pdf URL: https://arxiv.org/pdf/2503.21172
• Copy Paste: [[2503.21172]] Model as a Game: On Numerical and Spatial Consistency for Generative Games(https://arxiv.org/abs/2503.21172)
• Keywords: generation, generative
• Abstract: Recent advances in generative models have significantly impacted game generation. However, despite producing high-quality graphics and adequately receiving player input, existing models often fail to maintain fundamental game properties such as numerical and spatial consistency. Numerical consistency ensures gameplay mechanics correctly reflect score changes and other quantitative elements, while spatial consistency prevents jarring scene transitions, providing seamless player experiences. In this paper, we revisit the paradigm of generative games to explore what truly constitutes a Model as a Game (MaaG) with a well-developed mechanism. We begin with an empirical study on ``Traveler'', a 2D game created by an LLM featuring minimalist rules yet challenging generative models in maintaining consistency. Based on the DiT architecture, we design two specialized modules: (1) a numerical module that integrates a LogicNet to determine event triggers, with calculations processed externally as conditions for image generation; and (2) a spatial module that maintains a map of explored areas, retrieving location-specific information during generation and linking new observations to ensure continuity. Experiments across three games demonstrate that our integrated modules significantly enhance performance on consistency metrics compared to baselines, while incurring minimal time overhead during inference.
• 摘要：生成模型的最新进展极大地影响了游戏的产生。但是，尽管产生了高质量的图形并充分接收玩家的输入，但现有模型通常无法维护基本的游戏属性，例如数值和空间一致性。数值一致性可确保游戏机制正确反映分数变化和其他定量元素，而空间一致性则可以防止场景过渡，从而提供无缝的玩家体验。在本文中，我们重新审视了生成游戏的范式，以探索具有完善的机制的真正构成模型（MAAG）。我们从一项关于``旅行者''的经验研究开始，这是一款由LLM创建的2D游戏，该游戏具有简约的规则，但在保持一致性方面具有挑战性生成模型。基于DIT体系结构，我们设计了两个专门的模块：（1）一个集成逻辑网络以确定事件触发器的数值模块，并将外部处理为图像生成条件； （2）一个空间模块，该空间模块维护探索区域的地图，在发电过程中检索特定于位置的信息并链接新的观测值以确保连续性。在三场比赛中进行的实验表明，与基准相比，我们的集成模块显着提高了一致性指标的性能，同时在推理过程中会产生最小的时间开销。

Title: FakeReasoning: Towards Generalizable Forgery Detection and Reasoning

• Authors: Yueying Gao, Dongliang Chang, Bingyao Yu, Haotian Qin, Lei Chen, Kongming Liang, Zhanyu Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21210
• Pdf URL: https://arxiv.org/pdf/2503.21210
• Copy Paste: [[2503.21210]] FakeReasoning: Towards Generalizable Forgery Detection and Reasoning(https://arxiv.org/abs/2503.21210)
• Keywords: generative
• Abstract: Accurate and interpretable detection of AI-generated images is essential for mitigating risks associated with AI misuse. However, the substantial domain gap among generative models makes it challenging to develop a generalizable forgery detection model. Moreover, since every pixel in an AI-generated image is synthesized, traditional saliency-based forgery explanation methods are not well suited for this task. To address these challenges, we propose modeling AI-generated image detection and explanation as a Forgery Detection and Reasoning task (FDR-Task), leveraging vision-language models (VLMs) to provide accurate detection through structured and reliable reasoning over forgery attributes. To facilitate this task, we introduce the Multi-Modal Forgery Reasoning dataset (MMFR-Dataset), a large-scale dataset containing 100K images across 10 generative models, with 10 types of forgery reasoning annotations, enabling comprehensive evaluation of FDR-Task. Additionally, we propose FakeReasoning, a forgery detection and reasoning framework with two key components. First, Forgery-Aligned Contrastive Learning enhances VLMs' understanding of forgery-related semantics through both cross-modal and intra-modal contrastive learning between images and forgery attribute reasoning. Second, a Classification Probability Mapper bridges the optimization gap between forgery detection and language modeling by mapping the output logits of VLMs to calibrated binary classification probabilities. Experiments across multiple generative models demonstrate that FakeReasoning not only achieves robust generalization but also outperforms state-of-the-art methods on both detection and reasoning tasks.
• 摘要：对AI生成的图像的准确检测对于缓解与AI滥用相关的风险至关重要。但是，生成模型之间的巨大域间隙使得开发可推广的伪造检测模型变得具有挑战性。此外，由于合成了AI生成的图像中的每个像素，因此基于传统的显着性解释方法不适合此任务。为了应对这些挑战，我们建议将AI生成的图像检测和解释作为伪造的检测和推理任务（FDR任务）（利用视觉语言模型（VLM）（VLM），以通过伪造属性进行结构化和可靠的推理来提供准确的检测。为了促进这项任务，我们介绍了多模式伪造推理数据集（MMFR-DATASET），这是一个大规模数据集，其中包含10个生成模型的100K图像，并具有10种类型的伪造推理注释，从而实现了FDR任务的全面评估。此外，我们建议使用两个关键组成部分的伪造发现和推理框架。首先，伪造的对比度学习通过图像和伪造属性属性推理之间的跨模式和模式对比度学习，增强了VLM对与伪造的语义的理解。其次，分类概率映射器通过将VLM的输出逻辑映射到校准的二进制分类概率来弥合伪造检测和语言建模之间的优化差距。跨多个生成模型的实验表明，伪造不仅实现了强大的概括，而且在检测和推理任务上都胜过最先进的方法。

Title: GenFusion: Closing the Loop between Reconstruction and Generation via Videos

• Authors: Sibo Wu, Congrong Xu, Binbin Huang, Andreas Geiger, Anpei Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.21219
• Pdf URL: https://arxiv.org/pdf/2503.21219
• Copy Paste: [[2503.21219]] GenFusion: Closing the Loop between Reconstruction and Generation via Videos(https://arxiv.org/abs/2503.21219)
• Keywords: restoration, generation, generative
• Abstract: Recently, 3D reconstruction and generation have demonstrated impressive novel view synthesis results, achieving high fidelity and efficiency. However, a notable conditioning gap can be observed between these two fields, e.g., scalable 3D scene reconstruction often requires densely captured views, whereas 3D generation typically relies on a single or no input view, which significantly limits their applications. We found that the source of this phenomenon lies in the misalignment between 3D constraints and generative priors. To address this problem, we propose a reconstruction-driven video diffusion model that learns to condition video frames on artifact-prone RGB-D renderings. Moreover, we propose a cyclical fusion pipeline that iteratively adds restoration frames from the generative model to the training set, enabling progressive expansion and addressing the viewpoint saturation limitations seen in previous reconstruction and generation pipelines. Our evaluation, including view synthesis from sparse view and masked input, validates the effectiveness of our approach.
• 摘要：最近，3D重建和产生表现出了令人印象深刻的新观点综合结果，从而实现了高保真度和效率。但是，可以在这两个字段之间观察到一个显着的条件差距，例如，可扩展的3D场景重建通常需要密集捕获的视图，而3D一代通常依赖于单个或没有输入视图，这显着限制了其应用。我们发现这种现象的来源在于3D约束与生成先验之间的错位。为了解决这个问题，我们提出了一个由重建驱动的视频扩散模型，该模型学会了在易于人工制品的RGB-D渲染上调节视频帧。此外，我们提出了一条周期性的融合管道，它迭代地将恢复帧从生成模型添加到训练集，从而实现了渐进式扩展并解决了先前的重建和生成管道中所见的观点饱和度限制。我们的评估，包括从稀疏视图和掩盖输入的视图综合，验证了我们方法的有效性。

Title: Efficient Learning for Entropy-regularized Markov Decision Processes via Multilevel Monte Carlo

• Authors: Matthieu Meunier, Christoph Reisinger, Yufei Zhang
• Subjects: cs.LG, math.OC, math.PR, stat.ML
• Abstract URL: https://arxiv.org/abs/2503.21224
• Pdf URL: https://arxiv.org/pdf/2503.21224
• Copy Paste: [[2503.21224]] Efficient Learning for Entropy-regularized Markov Decision Processes via Multilevel Monte Carlo(https://arxiv.org/abs/2503.21224)
• Keywords: generative
• Abstract: Designing efficient learning algorithms with complexity guarantees for Markov decision processes (MDPs) with large or continuous state and action spaces remains a fundamental challenge. We address this challenge for entropy-regularized MDPs with Polish state and action spaces, assuming access to a generative model of the environment. We propose a novel family of multilevel Monte Carlo (MLMC) algorithms that integrate fixed-point iteration with MLMC techniques and a generic stochastic approximation of the Bellman operator. We quantify the precise impact of the chosen approximate Bellman operator on the accuracy of the resulting MLMC estimator. Leveraging this error analysis, we show that using a biased plain MC estimate for the Bellman operator results in quasi-polynomial sample complexity, whereas an unbiased randomized multilevel approximation of the Bellman operator achieves polynomial sample complexity in expectation. Notably, these complexity bounds are independent of the dimensions or cardinalities of the state and action spaces, distinguishing our approach from existing algorithms whose complexities scale with the sizes of these spaces. We validate these theoretical performance guarantees through numerical experiments.
• 摘要：使用较大或连续的状态和行动空间的马尔可夫决策过程（MDP）设计有效的学习算法，保证了马尔可夫决策过程（MDP）仍然是一个基本挑战。假设访问环境的生成模型，我们将针对具有波兰状态和动作空间的熵调查的MDP解决这一挑战。我们提出了一种新型的多级蒙特卡洛（MLMC）算法，该算法将固定点迭代与MLMC技术和Bellman操作员的通用随机近似结合在一起。我们量化了所选近似Bellman操作员对所得MLMC估计器的准确性的精确影响。利用此误差分析，我们表明，使用偏见的Bellman运算符的普通MC估计值会导致准多项式样品复杂性，而贝尔曼操作员的无偏随机多层次近似值实现了预期的多种方案样本复杂性。值得注意的是，这些复杂性界限独立于状态和行动空间的维度或红衣，将我们的方法与现有算法区分开，其复杂性与这些空间的大小相比。我们通过数值实验来验证这些理论绩效。

Title: DynamiCtrl: Rethinking the Basic Structure and the Role of Text for High-quality Human Image Animation

• Authors: Haoyu Zhao, Zhongang Qi, Cong Wang, Qingping Zheng, Guansong Lu, Fei Chen, Hang Xu, Zuxuan Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21246
• Pdf URL: https://arxiv.org/pdf/2503.21246
• Copy Paste: [[2503.21246]] DynamiCtrl: Rethinking the Basic Structure and the Role of Text for High-quality Human Image Animation(https://arxiv.org/abs/2503.21246)
• Keywords: generative
• Abstract: Human image animation has recently gained significant attention due to advancements in generative models. However, existing methods still face two major challenges: (1) architectural limitations, most models rely on U-Net, which underperforms compared to the MM-DiT; and (2) the neglect of textual information, which can enhance controllability. In this work, we introduce DynamiCtrl, a novel framework that not only explores different pose-guided control structures in MM-DiT, but also reemphasizes the crucial role of text in this task. Specifically, we employ a Shared VAE encoder for both reference images and driving pose videos, eliminating the need for an additional pose encoder and simplifying the overall framework. To incorporate pose features into the full attention blocks, we propose Pose-adaptive Layer Norm (PadaLN), which utilizes adaptive layer normalization to encode sparse pose features. The encoded features are directly added to the visual input, preserving the spatiotemporal consistency of the backbone while effectively introducing pose control into MM-DiT. Furthermore, within the full attention mechanism, we align textual and visual features to enhance controllability. By leveraging text, we not only enable fine-grained control over the generated content, but also, for the first time, achieve simultaneous control over both background and motion. Experimental results verify the superiority of DynamiCtrl on benchmark datasets, demonstrating its strong identity preservation, heterogeneous character driving, background controllability, and high-quality synthesis. The project page is available at this https URL.
• 摘要：由于生成模型的进步，人类形象动画最近引起了人们的重大关注。但是，现有方法仍然面临两个主要挑战：（1）建筑局限性，大多数模型都依赖于U-NET，而U-NET与MM-DIT相比表现不佳； （2）忽略文本信息，可以增强可控性。在这项工作中，我们引入了DynamicTrl，这是一个新颖的框架，不仅探讨了MM-DIT中不同姿势引导的控制结构，而且还重新强调了文本在此任务中的关键作用。具体而言，我们对参考图像和驾驶姿势视频都采用共享的VAE编码器，从而消除了对额外的姿势编码器的需求，并简化了整体框架。为了将姿势特征纳入完整的注意力块中，我们提出了姿势自适应层标准（PADALN），该标准（PADALN）利用自适应层归一化来编码稀疏姿势特征。将编码的特征直接添加到视觉输入中，从而保留了主链的时空一致性，同时有效地将姿势控制引入了MM-DIT。此外，在完整的注意机制中，我们将文本和视觉特征调整为增强可控性。通过利用文本，我们不仅可以对生成的内容进行细粒度的控制，而且首次可以同时控制背景和运动。实验结果验证了基准数据集上动态的优势，证明了其强大的身份保存，异质性角色驾驶，背景可控性和高质量合成。该项目页面可在此HTTPS URL上找到。

Title: Dual-Splitting Conformal Prediction for Multi-Step Time Series Forecasting

• Authors: Qingdi Yu, Zhiwei Cao, Ruihang Wang, Zhen Yang, Lijun Deng, Min Hu, Yong Luo, Xin Zhou
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.21251
• Pdf URL: https://arxiv.org/pdf/2503.21251
• Copy Paste: [[2503.21251]] Dual-Splitting Conformal Prediction for Multi-Step Time Series Forecasting(https://arxiv.org/abs/2503.21251)
• Keywords: generation
• Abstract: Time series forecasting is crucial for applications like resource scheduling and risk management, where multi-step predictions provide a comprehensive view of future trends. Uncertainty Quantification (UQ) is a mainstream approach for addressing forecasting uncertainties, with Conformal Prediction (CP) gaining attention due to its model-agnostic nature and statistical guarantees. However, most variants of CP are designed for single-step predictions and face challenges in multi-step scenarios, such as reliance on real-time data and limited scalability. This highlights the need for CP methods specifically tailored to multi-step forecasting. We propose the Dual-Splitting Conformal Prediction (DSCP) method, a novel CP approach designed to capture inherent dependencies within time-series data for multi-step forecasting. Experimental results on real-world datasets from four different domains demonstrate that the proposed DSCP significantly outperforms existing CP variants in terms of the Winkler Score, achieving a performance improvement of up to 23.59% compared to state-of-the-art methods. Furthermore, we deployed the DSCP approach for renewable energy generation and IT load forecasting in power management of a real-world trajectory-based application, achieving an 11.25% reduction in carbon emissions through predictive optimization of data center operations and controls.
• 摘要：时间序列预测对于诸如资源调度和风险管理等应用程序至关重要，在该应用程序中，多步预测提供了对未来趋势的全面看法。不确定性定量（UQ）是一种解决预测不确定性的主流方法，由于其模型 - 不合骨的性质和统计保证，共形预测（CP）引起了人们的注意。但是，大多数CP的变体都是为单步预测而设计的，并且在多步骤场景中面临挑战，例如依赖实时数据和有限的可伸缩性。这突出了对专门针对多步骤预测的CP方法的需求。我们提出了双分裂的共形预测（DSCP）方法，这是一种新型的CP方法，旨在捕获多步预测的时间序列数据中的固有依赖项。来自四个不同领域的现实世界数据集的实验结果表明，与最先进的方法相比，所提出的DSCP在Winkler分数方面显着优于现有的CP变体，可提高高达23.59％的性能。此外，我们采用了可再生能源生成的DSCP方法，并在基于实际轨迹的应用的电力管理中进行了预测，通过对数据中心的操作和控件的预测优化，碳排放量降低了11.25％。

Title: Vision-to-Music Generation: A Survey

• Authors: Zhaokai Wang, Chenxi Bao, Le Zhuo, Jingrui Han, Yang Yue, Yihong Tang, Victor Shea-Jay Huang, Yue Liao
• Subjects: cs.CV, cs.AI, cs.MM, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2503.21254
• Pdf URL: https://arxiv.org/pdf/2503.21254
• Copy Paste: [[2503.21254]] Vision-to-Music Generation: A Survey(https://arxiv.org/abs/2503.21254)
• Keywords: generation
• Abstract: Vision-to-music Generation, including video-to-music and image-to-music tasks, is a significant branch of multimodal artificial intelligence demonstrating vast application prospects in fields such as film scoring, short video creation, and dance music synthesis. However, compared to the rapid development of modalities like text and images, research in vision-to-music is still in its preliminary stage due to its complex internal structure and the difficulty of modeling dynamic relationships with video. Existing surveys focus on general music generation without comprehensive discussion on vision-to-music. In this paper, we systematically review the research progress in the field of vision-to-music generation. We first analyze the technical characteristics and core challenges for three input types: general videos, human movement videos, and images, as well as two output types of symbolic music and audio music. We then summarize the existing methodologies on vision-to-music generation from the architecture perspective. A detailed review of common datasets and evaluation metrics is provided. Finally, we discuss current challenges and promising directions for future research. We hope our survey can inspire further innovation in vision-to-music generation and the broader field of multimodal generation in academic research and industrial applications. To follow latest works and foster further innovation in this field, we are continuously maintaining a GitHub repository at this https URL.
• 摘要：视觉到音乐的一代，包括视频到音乐和图像到音乐任务，是多模式人工智能的重要分支，展示了电影评分，短视频创作和舞蹈音乐综合等领域的巨大应用前景。但是，与文本和图像等模式的快速发展相比，由于其复杂的内部结构以及与视频的动态关系建模，视觉到音乐的研究仍处于初步阶段。现有的调查专注于一般音乐发电，而无需全面讨论愿景到音乐。在本文中，我们系统地回顾了视觉到音乐生成领域的研究进展。我们首先分析了三种输入类型的技术特征和核心挑战：一般视频，人类运动视频和图像，以及两种符号音乐和音频音乐的输出类型。然后，我们从体系结构的角度总结了有关愿景到音乐生成的现有方法。提供了对常见数据集和评估指标的详细审查。最后，我们讨论了当前的挑战和未来研究的有希望的方向。我们希望我们的调查能够激发愿景到音乐产生的进一步创新，以及在学术研究和工业应用中更广泛的多模式生成领域。要遵循最新的作品并在该领域促进进一步的创新，我们一直在此HTTPS URL上不断维护GitHub存储库。

Title: Learn by Reasoning: Analogical Weight Generation for Few-Shot Class-Incremental Learning

• Authors: Jizhou Han, Chenhao Ding, Yuhang He, Songlin Dong, Qiang Wang, Xinyuan Gao, Yihong Gong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.21258
• Pdf URL: https://arxiv.org/pdf/2503.21258
• Copy Paste: [[2503.21258]] Learn by Reasoning: Analogical Weight Generation for Few-Shot Class-Incremental Learning(https://arxiv.org/abs/2503.21258)
• Keywords: generation, generative
• Abstract: Few-shot class-incremental Learning (FSCIL) enables models to learn new classes from limited data while retaining performance on previously learned classes. Traditional FSCIL methods often require fine-tuning parameters with limited new class data and suffer from a separation between learning new classes and utilizing old knowledge. Inspired by the analogical learning mechanisms of the human brain, we propose a novel analogical generative method. Our approach includes the Brain-Inspired Analogical Generator (BiAG), which derives new class weights from existing classes without parameter fine-tuning during incremental stages. BiAG consists of three components: Weight Self-Attention Module (WSA), Weight & Prototype Analogical Attention Module (WPAA), and Semantic Conversion Module (SCM). SCM uses Neural Collapse theory for semantic conversion, WSA supplements new class weights, and WPAA computes analogies to generate new class weights. Experiments on miniImageNet, CUB-200, and CIFAR-100 datasets demonstrate that our method achieves higher final and average accuracy compared to SOTA methods.
• 摘要：很少有类型的课堂学习（FSCIL）使模型能够从有限的数据中学习新课程，同时保留先前学习的课程的绩效。传统的FSCIL方法通常需要具有有限的新类数据的微调参数，并且学习新课程和利用旧知识之间的分离。受人脑的类似学习机制的启发，我们提出了一种新颖的类似生成方法。我们的方法包括脑启发的类似发电机（BIAG），该生成器（BIAG）从现有类中衍生出新的类权重，而没有参数阶段的参数进行微调。 BIAG由三个组成部分组成：体重自我发项模块（WSA），重量和原型类似物注意模块（WPAA）和语义转换模块（SCM）。 SCM使用神经塌陷理论进行语义转换，WSA补充了新的班级体重，WPAA计算类比来产生新的班级体重。在Miniimagenet，Cub-200和CIFAR-100数据集上进行的实验表明，与SOTA方法相比，我们的方法达到了更高的最终和平均精度。

Title: Zero-Shot Visual Concept Blending Without Text Guidance

• Authors: Hiroya Makino, Takahiro Yamaguchi, Hiroyuki Sakai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21277
• Pdf URL: https://arxiv.org/pdf/2503.21277
• Copy Paste: [[2503.21277]] Zero-Shot Visual Concept Blending Without Text Guidance(https://arxiv.org/abs/2503.21277)
• Keywords: generation
• Abstract: We propose a novel, zero-shot image generation technique called "Visual Concept Blending" that provides fine-grained control over which features from multiple reference images are transferred to a source image. If only a single reference image is available, it is difficult to isolate which specific elements should be transferred. However, using multiple reference images, the proposed approach distinguishes between common and unique features by selectively incorporating them into a generated output. By operating within a partially disentangled Contrastive Language-Image Pre-training (CLIP) embedding space (from IP-Adapter), our method enables the flexible transfer of texture, shape, motion, style, and more abstract conceptual transformations without requiring additional training or text prompts. We demonstrate its effectiveness across a diverse range of tasks, including style transfer, form metamorphosis, and conceptual transformations, showing how subtle or abstract attributes (e.g., brushstroke style, aerodynamic lines, and dynamism) can be seamlessly combined into a new image. In a user study, participants accurately recognized which features were intended to be transferred. Its simplicity, flexibility, and high-level control make Visual Concept Blending valuable for creative fields such as art, design, and content creation, where combining specific visual qualities from multiple inspirations is crucial.
• 摘要：我们提出了一种称为“视觉概念融合”的新颖，零击图像生成技术，该技术可通过将来自多个参考图像的特征转移到源图像中，提供细粒度的控制。如果只有一个参考图像，则很难分离应传输哪些特定元素。但是，使用多个参考图像，建议的方法通过选择性地将其纳入生成的输出来区分常见和唯一特征。通过在部分解开的对比语言图像预训练（剪辑）嵌入空间（来自IP-Adapter）中，我们的方法可以灵活地转移纹理，形状，形状，运动，样式和更抽象的概念转换，而无需其他培训或文本提示。我们展示了它在各种任务中的有效性，包括样式转移，形成变态和概念转换，表明如何将微妙或抽象的属性（例如，笔触风格，空气动力学线和动态主义）无缝组合到新的图像中。在一项用户研究中，参与者准确地认识到要传输哪些功能。它的简单性，灵活性和高级控制使视觉概念融合在艺术，设计和内容创建等创意领域中很有价值，在这种创意领域中，将来自多种灵感的特定视觉素质结合在一起至关重要。

Title: Tricking Retrievers with Influential Tokens: An Efficient Black-Box Corpus Poisoning Attack

• Authors: Cheng Wang, Yiwei Wang, Yujun Cai, Bryan Hooi
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2503.21315
• Pdf URL: https://arxiv.org/pdf/2503.21315
• Copy Paste: [[2503.21315]] Tricking Retrievers with Influential Tokens: An Efficient Black-Box Corpus Poisoning Attack(https://arxiv.org/abs/2503.21315)
• Keywords: generation
• Abstract: Retrieval-augmented generation (RAG) systems enhance large language models by incorporating external knowledge, addressing issues like outdated internal knowledge and hallucination. However, their reliance on external knowledge bases makes them vulnerable to corpus poisoning attacks, where adversarial passages can be injected to manipulate retrieval results. Existing methods for crafting such passages, such as random token replacement or training inversion models, are often slow and computationally expensive, requiring either access to retriever's gradients or large computational resources. To address these limitations, we propose Dynamic Importance-Guided Genetic Algorithm (DIGA), an efficient black-box method that leverages two key properties of retrievers: insensitivity to token order and bias towards influential tokens. By focusing on these characteristics, DIGA dynamically adjusts its genetic operations to generate effective adversarial passages with significantly reduced time and memory usage. Our experimental evaluation shows that DIGA achieves superior efficiency and scalability compared to existing methods, while maintaining comparable or better attack success rates across multiple datasets.
• 摘要：检索增强的生成（RAG）系统通过合并外部知识来增强大语言模型，解决过时的内部知识和幻觉等问题。但是，它们对外部知识基础的依赖使它们容易受到语料库中毒攻击的影响，在这些攻击中，可以注入对抗性段落以操纵检索结果。现有的制作此类段落的方法，例如随机令牌更换或训练倒置模型，通常很慢且计算上很昂贵，需要访问回猎犬的梯度或大量的计算资源。为了解决这些局限性，我们提出了动态重要性引导的遗传算法（DIGA），这是一种有效的黑盒方法，利用了检索器的两个关键特性：对代币顺序不敏感和对影响力令牌的偏见。通过关注这些特征，Diga动态调整其遗传操作，以产生有效的对抗通道，并大大减少时间和记忆使用。我们的实验评估表明，与现有方法相比，DIGA可实现优异的效率和可伸缩性，同时保持了多个数据集的可比或更好的攻击成功率。

Title: Scalable Expectation Estimation with Subtractive Mixture Models

• Authors: Lena Zellinger, Nicola Branchini, Víctor Elvira, Antonio Vergari
• Subjects: cs.LG, stat.CO, stat.ML
• Abstract URL: https://arxiv.org/abs/2503.21346
• Pdf URL: https://arxiv.org/pdf/2503.21346
• Copy Paste: [[2503.21346]] Scalable Expectation Estimation with Subtractive Mixture Models(https://arxiv.org/abs/2503.21346)
• Keywords: generative
• Abstract: Many Monte Carlo (MC) and importance sampling (IS) methods use mixture models (MMs) for their simplicity and ability to capture multimodal distributions. Recently, subtractive mixture models (SMMs), i.e. MMs with negative coefficients, have shown greater expressiveness and success in generative modeling. However, their negative parameters complicate sampling, requiring costly auto-regressive techniques or accept-reject algorithms that do not scale in high dimensions. In this work, we use the difference representation of SMMs to construct an unbiased IS estimator ($\Delta\text{Ex}$) that removes the need to sample from the SMM, enabling high-dimensional expectation estimation with SMMs. In our experiments, we show that $\Delta\text{Ex}$ can achieve comparable estimation quality to auto-regressive sampling while being considerably faster in MC estimation. Moreover, we conduct initial experiments with $\Delta\text{Ex}$ using hand-crafted proposals, gaining first insights into how to construct safe proposals for $\Delta\text{Ex}$.
• 摘要：许多蒙特卡洛（MC）和重要性采样（IS）方法使用混合模型（MMS）来捕获多模式分布的能力。最近，减法混合物模型（SMM），即具有负系数的MMS，在生成模型中显示出更大的表现力和成功。但是，它们的负参数使采样复杂化，需要代价高昂的自动回归技术或不扩展高维度的接受置换算法。在这项工作中，我们使用SMM的差异表示形式来构建公正的是估计器（$ \ delta \ text {ex} $），它消除了从SMM中进行采样的需求，从而可以通过SMMS进行高维期望估计。在我们的实验中，我们表明$ \ delta \ text {ex} $可以达到可比的估计质量与自动回归抽样，同时在MC估计中要快得多。此外，我们使用手工制作的建议使用$ \ delta \ text {ex} $进行初始实验，从而获得了如何为$ \ delta \ text {ex} $构建安全建议的第一见解。

Title: Diffusion Image Prior

• Authors: Hamadi Chihaoui, Paolo Favaro
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21410
• Pdf URL: https://arxiv.org/pdf/2503.21410
• Copy Paste: [[2503.21410]] Diffusion Image Prior(https://arxiv.org/abs/2503.21410)
• Keywords: restoration, super-resolution
• Abstract: Zero-shot image restoration (IR) methods based on pretrained diffusion models have recently achieved significant success. These methods typically require at least a parametric form of the degradation model. However, in real-world scenarios, the degradation may be too complex to define explicitly. To handle this general case, we introduce the Diffusion Image Prior (DIIP). We take inspiration from the Deep Image Prior (DIP)[16], since it can be used to remove artifacts without the need for an explicit degradation model. However, in contrast to DIP, we find that pretrained diffusion models offer a much stronger prior, despite being trained without knowledge from corrupted data. We show that, the optimization process in DIIP first reconstructs a clean version of the image before eventually overfitting to the degraded input, but it does so for a broader range of degradations than DIP. In light of this result, we propose a blind image restoration (IR) method based on early stopping, which does not require prior knowledge of the degradation model. We validate DIIP on various degradation-blind IR tasks, including JPEG artifact removal, waterdrop removal, denoising and super-resolution with state-of-the-art results.
• 摘要：基于预处理扩散模型的零拍图像恢复（IR）方法最近取得了显着成功。这些方法通常至少需要降解模型的参数形式。但是，在实际情况下，降解可能太复杂了，无法明确定义。为了处理这种一般情况，我们介绍了先验的扩散图像（DIIP）。我们从深度图像先验（DIP）[16]中汲取灵感，因为它可以用于删除伪像，而无需明确的退化模型。但是，与DIP相反，我们发现审计的扩散模型提供了更强的先验，尽管在没有损坏的数据的知识的情况下接受了训练。我们表明，DIIP中的优化过程首先重建了图像的干净版本，然后最终过度适合降级输入，但是它的降级范围比DIP更广泛。鉴于此结果，我们提出了基于早期停止的盲图恢复（IR）方法，该方法不需要先验了解降解模型。我们验证了各种降级IR任务的差异，包括JPEG伪影，去除水滴，去除和超级分辨率，并具有最先进的结果。

Title: Towards Generating Realistic 3D Semantic Training Data for Autonomous Driving

• Authors: Lucas Nunes, Rodrigo Marcuzzi, Jens Behley, Cyrill Stachniss
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21449
• Pdf URL: https://arxiv.org/pdf/2503.21449
• Copy Paste: [[2503.21449]] Towards Generating Realistic 3D Semantic Training Data for Autonomous Driving(https://arxiv.org/abs/2503.21449)
• Keywords: generation, generative
• Abstract: Semantic scene understanding is crucial for robotics and computer vision applications. In autonomous driving, 3D semantic segmentation plays an important role for enabling safe navigation. Despite significant advances in the field, the complexity of collecting and annotating 3D data is a bottleneck in this developments. To overcome that data annotation limitation, synthetic simulated data has been used to generate annotated data on demand. There is still however a domain gap between real and simulated data. More recently, diffusion models have been in the spotlight, enabling close-to-real data synthesis. Those generative models have been recently applied to the 3D data domain for generating scene-scale data with semantic annotations. Still, those methods either rely on image projection or decoupled models trained with different resolutions in a coarse-to-fine manner. Such intermediary representations impact the generated data quality due to errors added in those transformations. In this work, we propose a novel approach able to generate 3D semantic scene-scale data without relying on any projection or decoupled trained multi-resolution models, achieving more realistic semantic scene data generation compared to previous state-of-the-art methods. Besides improving 3D semantic scene-scale data synthesis, we thoroughly evaluate the use of the synthetic scene samples as labeled data to train a semantic segmentation network. In our experiments, we show that using the synthetic annotated data generated by our method as training data together with the real semantic segmentation labels, leads to an improvement in the semantic segmentation model performance. Our results show the potential of generated scene-scale point clouds to generate more training data to extend existing datasets, reducing the data annotation effort. Our code is available at this https URL.
• 摘要：语义场景的理解对于机器人和计算机视觉应用至关重要。在自动驾驶中，3D语义分割在实现安全导航方面起着重要作用。尽管该领域取得了重大进展，但收集和注释3D数据的复杂性是这一发展的瓶颈。为了克服数据注释限制，合成模拟数据已被用来按需生成带注释的数据。但是，实际数据和模拟数据之间仍然存在域差距。最近，扩散模型已引起人们的关注，从而实现了接近到现实的数据综合。这些生成模型最近已应用于3D数据域，以生成带有语义注释的场景规模数据。尽管如此，这些方法还是依赖于图像投影或以粗到精细的方式训练不同分辨率的脱钩模型。由于这些转换中添加的错误，这种中间表示会影响生成的数据质量。在这项工作中，我们提出了一种能够生成3D语义场景尺度数据的新方法，而无需依赖任何投影或脱钩的多分辨率模型，与以前的先进方法相比，实现了更现实的语义场景数据生成。除了改善3D语义场景尺度数据综合外，我们还彻底评估了合成场景样本作为标记数据来训练语义分割网络的使用。在我们的实验中，我们表明，使用我们的方法生成的合成注释数据作为训练数据以及实际语义分割标签，从而改善了语义分割模型性能。我们的结果表明，生成的场景规模点云的潜力生成了更多的培训数据以扩展现有数据集，从而减少了数据注释工作。我们的代码可在此HTTPS URL上找到。

Title: Invert2Restore: Zero-Shot Degradation-Blind Image Restoration

• Authors: Hamadi Chihaoui, Paolo Favaro
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21486
• Pdf URL: https://arxiv.org/pdf/2503.21486
• Copy Paste: [[2503.21486]] Invert2Restore: Zero-Shot Degradation-Blind Image Restoration(https://arxiv.org/abs/2503.21486)
• Keywords: restoration
• Abstract: Two of the main challenges of image restoration in real-world scenarios are the accurate characterization of an image prior and the precise modeling of the image degradation operator. Pre-trained diffusion models have been very successfully used as image priors in zero-shot image restoration methods. However, how to best handle the degradation operator is still an open problem. In real-world data, methods that rely on specific parametric assumptions about the degradation model often face limitations in their applicability. To address this, we introduce Invert2Restore, a zero-shot, training-free method that operates in both fully blind and partially blind settings -- requiring no prior knowledge of the degradation model or only partial knowledge of its parametric form without known parameters. Despite this, Invert2Restore achieves high-fidelity results and generalizes well across various types of image degradation. It leverages a pre-trained diffusion model as a deterministic mapping between normal samples and undistorted image samples. The key insight is that the input noise mapped by a diffusion model to a degraded image lies in a low-probability density region of the standard normal distribution. Thus, we can restore the degraded image by carefully guiding its input noise toward a higher-density region. We experimentally validate Invert2Restore across several image restoration tasks, demonstrating that it achieves state-of-the-art performance in scenarios where the degradation operator is either unknown or partially known.
• 摘要：在实际情况下，图像恢复的两个主要挑战是图像先验的准确表征和图像退化操作员的精确建模。预先训练的扩散模型已被非常成功地用作零拍图像恢复方法中的图像先验。但是，如何最好地处理降解操作员仍然是一个空旷的问题。在实际数据中，依赖于有关降解模型的特定参数假设的方法通常会在其适用性中面临限制。为了解决这个问题，我们引入了Invert2Restore，这是一种在完全盲目和部分盲目的设置中运行的零射击，无训练方法 - 不需要先验了解退化模型，或者仅需要部分了解其参数形式而没有已知参数。尽管如此，Interver2restore还是在各种类型的图像降解中都能达到高保真的结果。它利用预先训练的扩散模型作为正常样本和未变形图像样本之间的确定性映射。关键见解是，通过扩散模型映射到降级图像的输入噪声位于标准正态分布的低概率密度区域。因此，我们可以通过仔细引导其输入噪声向高密度区域来恢复降解的图像。我们在实验中验证了几个图像恢复任务中的Invert2Restore，表明它在降解操作员是未知或部分已知的情况下实现了最先进的性能。

Title: Shape Modeling of Longitudinal Medical Images: From Diffeomorphic Metric Mapping to Deep Learning

• Authors: Edwin Tay, Nazli Tümer, Amir A. Zadpoor
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21489
• Pdf URL: https://arxiv.org/pdf/2503.21489
• Copy Paste: [[2503.21489]] Shape Modeling of Longitudinal Medical Images: From Diffeomorphic Metric Mapping to Deep Learning(https://arxiv.org/abs/2503.21489)
• Keywords: generative
• Abstract: Living biological tissue is a complex system, constantly growing and changing in response to external and internal stimuli. These processes lead to remarkable and intricate changes in shape. Modeling and understanding both natural and pathological (or abnormal) changes in the shape of anatomical structures is highly relevant, with applications in diagnostic, prognostic, and therapeutic healthcare. Nevertheless, modeling the longitudinal shape change of biological tissue is a non-trivial task due to its inherent nonlinear nature. In this review, we highlight several existing methodologies and tools for modeling longitudinal shape change (i.e., spatiotemporal shape modeling). These methods range from diffeomorphic metric mapping to deep-learning based approaches (e.g., autoencoders, generative networks, recurrent neural networks, etc.). We discuss the synergistic combinations of existing technologies and potential directions for future research, underscoring key deficiencies in the current research landscape.
• 摘要：生物组织是一个复杂的系统，响应外部和内部刺激而不断增长和变化。这些过程导致形状的显着变化。建模和理解解剖结构形状的自然和病理（或异常）变化非常相关，并在诊断，预后和治疗性医疗保健中应用。然而，由于其固有的非线性性质，对生物组织的纵向形状变化进行建模是一项非平凡的任务。在这篇综述中，我们重点介绍了几种现有的方法和工具，用于建模纵向形状变化（即时空形状建模）。这些方法范围从差异度量映射到基于深度学习的方法（例如自动编码器，生成网络，经常性神经网络等）。我们讨论了现有技术的协同组合和未来研究的潜在方向，强调了当前研究局势中的关键缺陷。

Title: Uncertainty-aware Bayesian machine learning modelling of land cover classification

• Authors: Samuel Bilson, Anna Pustogvar
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21510
• Pdf URL: https://arxiv.org/pdf/2503.21510
• Copy Paste: [[2503.21510]] Uncertainty-aware Bayesian machine learning modelling of land cover classification(https://arxiv.org/abs/2503.21510)
• Keywords: generative
• Abstract: Land cover classification involves the production of land cover maps, which determine the type of land through remote sensing imagery. Over recent years, such classification is being performed by machine learning classification models, which can give highly accurate predictions on land cover per pixel using large quantities of input training data. However, such models do not currently take account of input measurement uncertainty, which is vital for traceability in metrology. In this work we propose a Bayesian classification framework using generative modelling to take account of input measurement uncertainty. We take the specific case of Bayesian quadratic discriminant analysis, and apply it to land cover datasets from Copernicus Sentinel-2 in 2020 and 2021. We benchmark the performance of the model against more popular classification models used in land cover maps such as random forests and neural networks. We find that such Bayesian models are more trustworthy, in the sense that they are more interpretable, explicitly model the input measurement uncertainty, and maintain predictive performance of class probability outputs across datasets of different years and sizes, whilst also being computationally efficient.
• 摘要：土地覆盖分类涉及生产土地覆盖地图，这些图通过遥感图像确定了土地的类型。近年来，机器学习分类模型正在执行此类分类，该模型可以使用大量的输入训练数据对每个像素的土地覆盖率进行高度准确的预测。但是，此类模型目前尚未考虑输入测量不确定性，这对于计量学的可追溯性至关重要。在这项工作中，我们提出了使用生成建模的贝叶斯分类框架，以考虑输入测量不确定性。我们采用了贝叶斯二次判别分析的具体情况，并将其应用于2020年和2021年哥白尼前后2号的土地覆盖数据集。我们基准了该模型对陆地覆盖地图中更流行的分类模型的性能，例如随机森林和神经网络。我们发现，这样的贝叶斯模型更值得信赖，从某种意义上说，它们更容易解释，明确地对输入测量不确定性进行建模，并在不同年份和尺寸的数据集中保持类概率输出的预测性能，同时也是计算效率效率的。

Title: Exploring the Energy Landscape of RBMs: Reciprocal Space Insights into Bosons, Hierarchical Learning and Symmetry Breaking

• Authors: J. Quetzalcóatl Toledo-Marin, Anindita Maiti, Geoffrey C. Fox, Roger G. Melko
• Subjects: cs.LG, cond-mat.dis-nn
• Abstract URL: https://arxiv.org/abs/2503.21536
• Pdf URL: https://arxiv.org/pdf/2503.21536
• Copy Paste: [[2503.21536]] Exploring the Energy Landscape of RBMs: Reciprocal Space Insights into Bosons, Hierarchical Learning and Symmetry Breaking(https://arxiv.org/abs/2503.21536)
• Keywords: generative
• Abstract: Deep generative models have become ubiquitous due to their ability to learn and sample from complex distributions. Despite the proliferation of various frameworks, the relationships among these models remain largely unexplored, a gap that hinders the development of a unified theory of AI learning. We address two central challenges: clarifying the connections between different deep generative models and deepening our understanding of their learning mechanisms. We focus on Restricted Boltzmann Machines (RBMs), known for their universal approximation capabilities for discrete distributions. By introducing a reciprocal space formulation, we reveal a connection between RBMs, diffusion processes, and coupled Bosons. We show that at initialization, the RBM operates at a saddle point, where the local curvature is determined by the singular values, whose distribution follows the Marcenko-Pastur law and exhibits rotational symmetry. During training, this rotational symmetry is broken due to hierarchical learning, where different degrees of freedom progressively capture features at multiple levels of abstraction. This leads to a symmetry breaking in the energy landscape, reminiscent of Landau theory. This symmetry breaking in the energy landscape is characterized by the singular values and the weight matrix eigenvector matrix. We derive the corresponding free energy in a mean-field approximation. We show that in the limit of infinite size RBM, the reciprocal variables are Gaussian distributed. Our findings indicate that in this regime, there will be some modes for which the diffusion process will not converge to the Boltzmann distribution. To illustrate our results, we trained replicas of RBMs with different hidden layer sizes using the MNIST dataset. Our findings bridge the gap between disparate generative frameworks and also shed light on the processes underpinning learning in generative models.
• 摘要：由于它们能够从复杂的分布中学习和采样，因此深层生成模型已变得无处不在。尽管各种框架扩散，但这些模型之间的关系仍然在很大程度上尚未开发，这一差距阻碍了统一的AI学习理论的发展。我们解决了两个核心挑战：阐明不同深层生成模型之间的联系并加深我们对他们的学习机制的理解。我们专注于受限制的玻尔兹曼机器（RBMS），该机器以其用于离散分布的通用近似功能而闻名。通过引入相互的空间公式，我们揭示了RBM，扩散过程和耦合玻色子之间的联系。我们表明，在初始化时，RBM在鞍点上运行，在鞍点上，局部曲率由单数值确定，其分布遵循Marcenko-Pastur定律，并展示了旋转对称性。在训练过程中，由于分层学习，这种旋转对称性被打破，其中不同程度的自由度逐渐捕获了多个抽象级别的特征。这导致能量景观中的对称性破裂，让人联想到兰道理论。能量景观中这种对称性破裂的特征是奇异值和权重矩阵特征向量矩阵。我们在平均场近似中得出相应的自由能。我们表明，在无限大小的rbm的极限中，倒数变量是高斯分布式的。我们的发现表明，在这种制度中，将有一些模式的扩散过程不会收敛到玻尔兹曼分布。为了说明结果，我们使用MNIST数据集训练了具有不同隐藏层大小的RBM的复制品。我们的发现弥合了不同生成框架之间的差距，并阐明了生成模型中学习的过程。

Title: SyncSDE: A Probabilistic Framework for Diffusion Synchronization

• Authors: Hyunjun Lee, Hyunsoo Lee, Sookwan Han
• Subjects: cs.LG, cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2503.21555
• Pdf URL: https://arxiv.org/pdf/2503.21555
• Copy Paste: [[2503.21555]] SyncSDE: A Probabilistic Framework for Diffusion Synchronization(https://arxiv.org/abs/2503.21555)
• Keywords: generation
• Abstract: There have been many attempts to leverage multiple diffusion models for collaborative generation, extending beyond the original domain. A prominent approach involves synchronizing multiple diffusion trajectories by mixing the estimated scores to artificially correlate the generation processes. However, existing methods rely on naive heuristics, such as averaging, without considering task specificity. These approaches do not clarify why such methods work and often fail when a heuristic suitable for one task is blindly applied to others. In this paper, we present a probabilistic framework for analyzing why diffusion synchronization works and reveal where heuristics should be focused - modeling correlations between multiple trajectories and adapting them to each specific task. We further identify optimal correlation models per task, achieving better results than previous approaches that apply a single heuristic across all tasks without justification.
• 摘要：有许多尝试利用多个扩散模型进行协作生成，扩展到了原始领域。一种突出的方法涉及通过将估计的分数混合以人为相关的生成过程来同步多个扩散轨迹。但是，现有的方法依赖于幼稚的启发式方法，例如平均，而无需考虑任务特异性。这些方法并不能阐明为什么这种方法可以使用，而当适合一项任务的启发式词被盲目应用于他人时，这些方法通常会失败。在本文中，我们提出了一个概率框架，用于分析为什么扩散同步起作用并揭示应集中启发式方法的何处 - 对多个轨迹之间的相关性进行建模并将其调整到每个特定任务中。我们进一步确定每个任务的最佳相关模型，比以前在没有理由的所有任务中应用单个启发式的方法获得了更好的结果。

Title: Critical Iterative Denoising: A Discrete Generative Model Applied to Graphs

• Authors: Yoann Boget, Alexandros Kalousis
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.21592
• Pdf URL: https://arxiv.org/pdf/2503.21592
• Copy Paste: [[2503.21592]] Critical Iterative Denoising: A Discrete Generative Model Applied to Graphs(https://arxiv.org/abs/2503.21592)
• Keywords: generation, generative
• Abstract: Discrete Diffusion and Flow Matching models have significantly advanced generative modeling for discrete structures, including graphs. However, the time dependencies in the noising process of these models lead to error accumulation and propagation during the backward process. This issue, particularly pronounced in mask diffusion, is a known limitation in sequence modeling and, as we demonstrate, also impacts discrete diffusion models for graphs. To address this problem, we propose a novel framework called Iterative Denoising, which simplifies discrete diffusion and circumvents the issue by assuming conditional independence across time. Additionally, we enhance our model by incorporating a Critic, which during generation selectively retains or corrupts elements in an instance based on their likelihood under the data distribution. Our empirical evaluations demonstrate that the proposed method significantly outperforms existing discrete diffusion baselines in graph generation tasks.
• 摘要：离散的扩散和流匹配模型具有包括图形在内的离散结构的明显高级生成建模。但是，这些模型的尖锐过程中的时间依赖性导致向后过程中的错误积累和传播。这个问题，尤其是在掩码扩散中明显的问题，是序列建模的已知限制，正如我们所证明的那样，还影响了图形的离散扩散模型。为了解决这个问题，我们提出了一个称为迭代denoisising的新型框架，该框架通过假设跨时间的条件独立性来简化离散扩散并避免问题。此外，我们通过纳入评论家来增强我们的模型，该评论家在世代期间根据数据分布中的可能性选择性地保留或破坏了实例中的元素。我们的经验评估表明，所提出的方法在图生成任务中的现有离散扩散基线大大优于现有的离散扩散基线。

Title: Audio-driven Gesture Generation via Deviation Feature in the Latent Space

• Authors: Jiahui Chen, Yang Huan, Runhua Shi, Chanfan Ding, Xiaoqi Mo, Siyu Xiong, Yinong He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21616
• Pdf URL: https://arxiv.org/pdf/2503.21616
• Copy Paste: [[2503.21616]] Audio-driven Gesture Generation via Deviation Feature in the Latent Space(https://arxiv.org/abs/2503.21616)
• Keywords: generation
• Abstract: Gestures are essential for enhancing co-speech communication, offering visual emphasis and complementing verbal interactions. While prior work has concentrated on point-level motion or fully supervised data-driven methods, we focus on co-speech gestures, advocating for weakly supervised learning and pixel-level motion deviations. We introduce a weakly supervised framework that learns latent representation deviations, tailored for co-speech gesture video generation. Our approach employs a diffusion model to integrate latent motion features, enabling more precise and nuanced gesture representation. By leveraging weakly supervised deviations in latent space, we effectively generate hand gestures and mouth movements, crucial for realistic video production. Experiments show our method significantly improves video quality, surpassing current state-of-the-art techniques.
• 摘要：手势对于增强共同语音交流，提供视觉重点和补充口头互动至关重要。尽管先前的工作集中在点级运动或完全监督的数据驱动方法上，但我们专注于共同语音的手势，主张弱监督的学习和像素级运动偏差。我们引入了一个弱监督的框架，该框架学习了为共同语音的手势视频生成而定制的潜在表示偏差。我们的方法采用扩散模型来整合潜在的运动特征，从而实现更精确和细微的手势表示。通过利用潜在空间中弱监督的偏差，我们有效地产生了手势和口腔动作，对于现实的视频制作至关重要。实验表明，我们的方法显着提高了视频质量，超过了当前的最新技术。

Title: Evaluating Text-to-Image Synthesis with a Conditional Fréchet Distance

• Authors: Jaywon Koo, Jefferson Hernandez, Moayed Haji-Ali, Ziyan Yang, Vicente Ordonez
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21721
• Pdf URL: https://arxiv.org/pdf/2503.21721
• Copy Paste: [[2503.21721]] Evaluating Text-to-Image Synthesis with a Conditional Fréchet Distance(https://arxiv.org/abs/2503.21721)
• Keywords: generation
• Abstract: Evaluating text-to-image synthesis is challenging due to misalignment between established metrics and human preferences. We propose cFreD, a metric based on the notion of Conditional Fréchet Distance that explicitly accounts for both visual fidelity and text-prompt alignment. Existing metrics such as Inception Score (IS), Fréchet Inception Distance (FID) and CLIPScore assess either image quality or image-text alignment but not both which limits their correlation with human preferences. Scoring models explicitly trained to replicate human preferences require constant updates and may not generalize to novel generation techniques or out-of-domain inputs. Through extensive experiments across multiple recently proposed text-to-image models and diverse prompt datasets, we demonstrate that cFreD exhibits a higher correlation with human judgments compared to statistical metrics, including metrics trained with human preferences. Our findings validate cFreD as a robust, future-proof metric for the systematic evaluation of text-to-image models, standardizing benchmarking in this rapidly evolving field. We release our evaluation toolkit and benchmark in the appendix.
• 摘要：由于既定指标和人类偏好之间的错位，评估文本对图像综合的综合是具有挑战性的。我们提出了CFRED，这是一个基于条件弗雷切特距离概念的度量，该距离明确地说明了视觉保真度和文本效果对齐。现有的指标，例如Inception评分（IS），Fréchet成立距离（FID）和夹克评估图像质量或图像文本对齐，但并非两者都限制了它们与人类偏好的相关性。经过明确训练以复制人类偏好的评分模型需要恒定的更新，并且可能不推广到新颖的生成技术或室外输入。通过在多个最近提出的文本对图像模型和各种及时数据集的广泛实验中，我们证明，与统计指标相比，CFRED与人类判断的相关性更高，包括接受人类偏好训练的指标。我们的发现将CFRED验证为对文本到图像模型的系统评估的强大，防止未来的指标，在​​这个快速发展的领域中标准化基准测试。我们在附录中发布评估工具包和基准。

Title: SparseFlex: High-Resolution and Arbitrary-Topology 3D Shape Modeling

• Authors: Xianglong He, Zi-Xin Zou, Chia-Hao Chen, Yuan-Chen Guo, Ding Liang, Chun Yuan, Wanli Ouyang, Yan-Pei Cao, Yangguang Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21732
• Pdf URL: https://arxiv.org/pdf/2503.21732
• Copy Paste: [[2503.21732]] SparseFlex: High-Resolution and Arbitrary-Topology 3D Shape Modeling(https://arxiv.org/abs/2503.21732)
• Keywords: generation
• Abstract: Creating high-fidelity 3D meshes with arbitrary topology, including open surfaces and complex interiors, remains a significant challenge. Existing implicit field methods often require costly and detail-degrading watertight conversion, while other approaches struggle with high resolutions. This paper introduces SparseFlex, a novel sparse-structured isosurface representation that enables differentiable mesh reconstruction at resolutions up to $1024^3$ directly from rendering losses. SparseFlex combines the accuracy of Flexicubes with a sparse voxel structure, focusing computation on surface-adjacent regions and efficiently handling open surfaces. Crucially, we introduce a frustum-aware sectional voxel training strategy that activates only relevant voxels during rendering, dramatically reducing memory consumption and enabling high-resolution training. This also allows, for the first time, the reconstruction of mesh interiors using only rendering supervision. Building upon this, we demonstrate a complete shape modeling pipeline by training a variational autoencoder (VAE) and a rectified flow transformer for high-quality 3D shape generation. Our experiments show state-of-the-art reconstruction accuracy, with a ~82% reduction in Chamfer Distance and a ~88% increase in F-score compared to previous methods, and demonstrate the generation of high-resolution, detailed 3D shapes with arbitrary topology. By enabling high-resolution, differentiable mesh reconstruction and generation with rendering losses, SparseFlex significantly advances the state-of-the-art in 3D shape representation and modeling.
• 摘要：创建具有任意拓扑的高保真3D网格，包括开放表面和复杂的内饰，仍然是一个重大挑战。现有的隐式现场方法通常需要昂贵且细节降级的水密转换，而其他方法则在高分辨率方面挣扎。本文介绍了SparseFlex，这是一种新型的稀疏结构等值面表示，可直接从渲染损失中直接从最高$ 1024^3 $的分辨率下进行可区分的网格重建。 SparseFlex结合了Flexibes的精度和稀疏的体素结构，将计算集中在表面粘合区域并有效地处理开放表面上。至关重要的是，我们引入了一种易感的部分体素训练策略，该策略仅在渲染过程中激活相关的体素，大大降低记忆消耗并实现高分辨率训练。这也允许首次仅使用渲染监督重建网格内饰。在此基础上，我们通过训练A自动编码器（VAE）和用于高质量3D形状生成的整流流量变压器来展示完整的形状建模管道。我们的实验显示了最新的重建精度，与以前的方法相比，倒角距离降低了约82％，F评分降低了约88％，并证明了具有任意拓扑的高分辨率，详细的3D形状的生成。通过实现高分辨率，可区分的网格重建和渲染损失的产生，SparseFlex可以显着提高3D形状表示和建模的最新时间。

Title: 3DGen-Bench: Comprehensive Benchmark Suite for 3D Generative Models

• Authors: Yuhan Zhang, Mengchen Zhang, Tong Wu, Tengfei Wang, Gordon Wetzstein, Dahua Lin, Ziwei Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21745
• Pdf URL: https://arxiv.org/pdf/2503.21745
• Copy Paste: [[2503.21745]] 3DGen-Bench: Comprehensive Benchmark Suite for 3D Generative Models(https://arxiv.org/abs/2503.21745)
• Keywords: generation, generative
• Abstract: 3D generation is experiencing rapid advancements, while the development of 3D evaluation has not kept pace. How to keep automatic evaluation equitably aligned with human perception has become a well-recognized challenge. Recent advances in the field of language and image generation have explored human preferences and showcased respectable fitting ability. However, the 3D domain still lacks such a comprehensive preference dataset over generative models. To mitigate this absence, we develop 3DGen-Arena, an integrated platform in a battle manner. Then, we carefully design diverse text and image prompts and leverage the arena platform to gather human preferences from both public users and expert annotators, resulting in a large-scale multi-dimension human preference dataset 3DGen-Bench. Using this dataset, we further train a CLIP-based scoring model, 3DGen-Score, and a MLLM-based automatic evaluator, 3DGen-Eval. These two models innovatively unify the quality evaluation of text-to-3D and image-to-3D generation, and jointly form our automated evaluation system with their respective strengths. Extensive experiments demonstrate the efficacy of our scoring model in predicting human preferences, exhibiting a superior correlation with human ranks compared to existing metrics. We believe that our 3DGen-Bench dataset and automated evaluation system will foster a more equitable evaluation in the field of 3D generation, further promoting the development of 3D generative models and their downstream applications.
• 摘要：3D世代正在经历快速的进步，而3D评估的发展尚未保持步伐。如何保持自动评估与人类看法保持一致已成为一个充分认识的挑战。语言和图像产生领域的最新进展探讨了人类的偏好并展现了可观的拟合能力。但是，与生成模型相比，3D域仍然缺乏如此全面的偏好数据集。为了减轻这种缺席，我们以战斗方式开发了3DGen-Arena，这是一个集成的平台。然后，我们仔细设计了各种文本和图像提示，并利用Arena平台从公共用户和专家注释者那里收集人类偏好，从而产生了大规模的多维人类偏好数据集3DGEN BENCE。使用此数据集，我们进一步训练了基于夹的评分模型3DGEN-SCORE和基于MLLM的自动评估器3DGEN-EVAL。这两个模型创新了文本到3D和图像到3D代的质量评估，并以各自的优势共同构成了我们的自动化评估系统。广泛的实验证明了我们评分模型在预测人类偏好方面的功效，与现有指标相比，与人类等级的相关性较高。我们认为，我们的3DGEN基础数据集和自动化评估系统将在3D生成领域促进更公平的评估，从而进一步促进了3D生成模型及其下游应用程序的开发。

Title: CTRL-O: Language-Controllable Object-Centric Visual Representation Learning

• Authors: Aniket Didolkar, Andrii Zadaianchuk, Rabiul Awal, Maximilian Seitzer, Efstratios Gavves, Aishwarya Agrawal
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.21747
• Pdf URL: https://arxiv.org/pdf/2503.21747
• Copy Paste: [[2503.21747]] CTRL-O: Language-Controllable Object-Centric Visual Representation Learning(https://arxiv.org/abs/2503.21747)
• Keywords: generation
• Abstract: Object-centric representation learning aims to decompose visual scenes into fixed-size vectors called "slots" or "object files", where each slot captures a distinct object. Current state-of-the-art object-centric models have shown remarkable success in object discovery in diverse domains, including complex real-world scenes. However, these models suffer from a key limitation: they lack controllability. Specifically, current object-centric models learn representations based on their preconceived understanding of objects, without allowing user input to guide which objects are represented. Introducing controllability into object-centric models could unlock a range of useful capabilities, such as the ability to extract instance-specific representations from a scene. In this work, we propose a novel approach for user-directed control over slot representations by conditioning slots on language descriptions. The proposed ConTRoLlable Object-centric representation learning approach, which we term CTRL-O, achieves targeted object-language binding in complex real-world scenes without requiring mask supervision. Next, we apply these controllable slot representations on two downstream vision language tasks: text-to-image generation and visual question answering. The proposed approach enables instance-specific text-to-image generation and also achieves strong performance on visual question answering.
• 摘要：以对象为中心的表示学习旨在将视觉场景分解为称为“插槽”或​​“对象文件”的固定尺寸向量，每个插槽都会捕获一个不同的对象。当前以最新的为中心对象的模型在各种域中的对象发现（包括复杂的现实世界场景）中表现出色。但是，这些模型受到关键限制：它们缺乏可控性。具体而言，当前以对象为中心的模型基于其对象的先入为主的理解来学习表示，而无需允许用户输入指导哪些对象表示。将可控性引入以对象为中心的模型可以解锁一系列有用的功能，例如从场景中提取实例特定表示的能力。在这项工作中，我们提出了一种新颖的方法，通过根据语言描述来调节插槽，以通过用户指导控制插槽表示。我们称为CTRL-O的提出的可控制对象以对象为中心的学习方法在复杂的现实世界中实现了目标对象绑定，而无需掩盖掩护。接下来，我们将这些可控的插槽表示形式应用于两个下游视觉语言任务：文本到图像生成和视觉问题回答。所提出的方法可以实现特定实例的文本到图像生成，并且在视觉问题回答上也实现了强劲的性能。

Title: LeX-Art: Rethinking Text Generation via Scalable High-Quality Data Synthesis

• Authors: Shitian Zhao, Qilong Wu, Xinyue Li, Bo Zhang, Ming Li, Qi Qin, Dongyang Liu, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Peng Gao, Bin Fu, Zhen Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21749
• Pdf URL: https://arxiv.org/pdf/2503.21749
• Copy Paste: [[2503.21749]] LeX-Art: Rethinking Text Generation via Scalable High-Quality Data Synthesis(https://arxiv.org/abs/2503.21749)
• Keywords: generation
• Abstract: We introduce LeX-Art, a comprehensive suite for high-quality text-image synthesis that systematically bridges the gap between prompt expressiveness and text rendering fidelity. Our approach follows a data-centric paradigm, constructing a high-quality data synthesis pipeline based on Deepseek-R1 to curate LeX-10K, a dataset of 10K high-resolution, aesthetically refined 1024$\times$1024 images. Beyond dataset construction, we develop LeX-Enhancer, a robust prompt enrichment model, and train two text-to-image models, LeX-FLUX and LeX-Lumina, achieving state-of-the-art text rendering performance. To systematically evaluate visual text generation, we introduce LeX-Bench, a benchmark that assesses fidelity, aesthetics, and alignment, complemented by Pairwise Normalized Edit Distance (PNED), a novel metric for robust text accuracy evaluation. Experiments demonstrate significant improvements, with LeX-Lumina achieving a 79.81% PNED gain on CreateBench, and LeX-FLUX outperforming baselines in color (+3.18%), positional (+4.45%), and font accuracy (+3.81%). Our codes, models, datasets, and demo are publicly available.
• 摘要：我们介绍了Lex-Art，这是一套高质量文本图像合成的综合套件，该套件系统地弥合了及时表达性和文本呈现忠诚度之间的差距。我们的方法遵循以数据为中心的范式，基于DeepSeek-R1构建了高质量的数据合成管道，以策划LEX-10K，这是一个10K高分辨率的数据集，美学精制的1024 $ \ times $ \ times $ 1024图像。除了数据集构造之外，我们还开发了Lex-Enhancer，这是一种强大的及时富集模型，并培训了两个文本到图像模型，即Lex-Flux和Lex-Lumina，以实现最先进的文本渲染性能。为了系统地评估视觉文本的生成，我们介绍了Lex-Bench，这是一种评估忠诚度，美学和对齐方式的基准，并由成对归一化的编辑距离（PNEND）补充，这是一种新颖的指标，可用于强大的文本准确性评估。实验显示出显着改善，Lex-Lumina在Create Bench上获得了79.81％的PNED增益，而Lex-Flux优于颜色（+3.18％），位置（+4.45％）和字体准确性（+3.81％）的颜色（+3.18％），位置（+4.45％）。我们的代码，模型，数据集和演示已公开可用。

Title: VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness

• Authors: Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng, Yu Qiao, Ziwei Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21755
• Pdf URL: https://arxiv.org/pdf/2503.21755
• Copy Paste: [[2503.21755]] VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness(https://arxiv.org/abs/2503.21755)
• Keywords: generation, generative
• Abstract: Video generation has advanced significantly, evolving from producing unrealistic outputs to generating videos that appear visually convincing and temporally coherent. To evaluate these video generative models, benchmarks such as VBench have been developed to assess their faithfulness, measuring factors like per-frame aesthetics, temporal consistency, and basic prompt adherence. However, these aspects mainly represent superficial faithfulness, which focus on whether the video appears visually convincing rather than whether it adheres to real-world principles. While recent models perform increasingly well on these metrics, they still struggle to generate videos that are not just visually plausible but fundamentally realistic. To achieve real "world models" through video generation, the next frontier lies in intrinsic faithfulness to ensure that generated videos adhere to physical laws, commonsense reasoning, anatomical correctness, and compositional integrity. Achieving this level of realism is essential for applications such as AI-assisted filmmaking and simulated world modeling. To bridge this gap, we introduce VBench-2.0, a next-generation benchmark designed to automatically evaluate video generative models for their intrinsic faithfulness. VBench-2.0 assesses five key dimensions: Human Fidelity, Controllability, Creativity, Physics, and Commonsense, each further broken down into fine-grained capabilities. Tailored for individual dimensions, our evaluation framework integrates generalists such as state-of-the-art VLMs and LLMs, and specialists, including anomaly detection methods proposed for video generation. We conduct extensive annotations to ensure alignment with human judgment. By pushing beyond superficial faithfulness toward intrinsic faithfulness, VBench-2.0 aims to set a new standard for the next generation of video generative models in pursuit of intrinsic faithfulness.
• 摘要：视频生成已经大大发展，从产生不切实际的产出到生成视觉上令人信服且具有时间连贯的视频。为了评估这些视频生成模型，已经开发了诸如VBENCH之类的基准来评估其忠诚，衡量诸如人均美学，时间一致性和基本及时依从性等因素。但是，这些方面主要代表肤浅的忠诚，这些方面的重点是视频是否在视觉上令人信服，而不是遵守现实世界的原则。尽管最近的模型在这些指标上的表现越来越好，但他们仍然很难生成视频，这些视频在视觉上是可行的，而且从根本上是现实的。为了通过视频生成来实现真实的“世界模型”，下一个边界在于内在的忠诚，以确保生成的视频遵守物理定律，常识性推理，解剖学正确性和组成完整性。实现这一水平的现实主义对于诸如AI辅助电影制作和模拟世界建模等应用至关重要。为了弥合这一差距，我们引入了VBENCH-2.0，这是一种下一代基准测试，旨在自动评估视频生成模型的内在忠诚度。 VBENCH-2.0评估了五个关键方面：人类的忠诚，可控性，创造力，物理和常识，每个都进一步分解为细粒度的能力。我们的评估框架是针对各个维度量身定制的，它整合了最先进的VLM和LLM等通才主义者，以及专家，包括为视频生成提出的异常检测方法。我们进行广泛的注释，以确保与人类判断保持一致。 VBENCH-2.0旨在为下一代视频生成模型建立新的标准，以追求固有的忠诚度。

Title: A Unified Framework for Diffusion Bridge Problems: Flow Matching and Schrödinger Matching into One

• Authors: Minyoung Kim
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2503.21756
• Pdf URL: https://arxiv.org/pdf/2503.21756
• Copy Paste: [[2503.21756]] A Unified Framework for Diffusion Bridge Problems: Flow Matching and Schrödinger Matching into One(https://arxiv.org/abs/2503.21756)
• Keywords: generation, generative
• Abstract: The bridge problem is to find an SDE (or sometimes an ODE) that bridges two given distributions. The application areas of the bridge problem are enormous, among which the recent generative modeling (e.g., conditional or unconditional image generation) is the most popular. Also the famous Schrödinger bridge problem, a widely known problem for a century, is a special instance of the bridge problem. Two most popular algorithms to tackle the bridge problems in the deep learning era are: (conditional) flow matching and iterative fitting algorithms, where the former confined to ODE solutions, and the latter specifically for the Schrödinger bridge problem. The main contribution of this article is in two folds: i) We provide concise reviews of these algorithms with technical details to some extent; ii) We propose a novel unified perspective and framework that subsumes these seemingly unrelated algorithms (and their variants) into one. In particular, we show that our unified framework can instantiate the Flow Matching (FM) algorithm, the (mini-batch) optimal transport FM algorithm, the (mini-batch) Schrödinger bridge FM algorithm, and the deep Schrödinger bridge matching (DSBM) algorithm as its special cases. We believe that this unified framework will be useful for viewing the bridge problems in a more general and flexible perspective, and in turn can help researchers and practitioners to develop new bridge algorithms in their fields.
• 摘要：桥梁问题是找到一个桥接两个给定分布的SDE（或有时是ode）。桥梁问题的应用领域是巨大的，其中最近的生成建模（例如，有条件或无条件的图像产生）是最流行的。同样，著名的Schrödinger桥问题是一个世纪以来众所周知的问题，是桥梁问题的一个特殊情况。解决深度学习时代桥梁问题的两种最受欢迎​​的算法是：（条件）流匹配和迭代拟合算法，前者仅限于Ode Solutions，而后者则专门针对Schrödinger桥梁问题。本文的主要贡献是两折：i）我们在某种程度上提供了有关这些算法的简明评论； ii）我们提出了一种新颖的统一观点和框架，将这些看似无关的算法（及其变体）归纳为一个。特别是，我们表明我们的统一框架可以实例化流量匹配（FM）算法，（Mini Batch）最佳运输FM算法，（Mini Batch）Schrödinger桥FM算法和DeepSchrödinger桥接（DSBM）AlgorithM（DSBM）AlgorithM（其特殊情况）。我们认为，这个统一的框架将有助于从更一般和灵活的角度查看桥梁问题，进而可以帮助研究人员和从业人员在其领域开发新的桥梁算法。

Title: Fwd2Bot: LVLM Visual Token Compression with Double Forward Bottleneck

• Authors: Adrian Bulat, Yassine Ouali, Georgios Tzimiropoulos
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.21757
• Pdf URL: https://arxiv.org/pdf/2503.21757
• Copy Paste: [[2503.21757]] Fwd2Bot: LVLM Visual Token Compression with Double Forward Bottleneck(https://arxiv.org/abs/2503.21757)
• Keywords: generative
• Abstract: In this work, we aim to compress the vision tokens of a Large Vision Language Model (LVLM) into a representation that is simultaneously suitable for (a) generative and (b) discriminative tasks, (c) is nearly lossless, and (d) is storage-efficient. We propose a novel compression approach, called Fwd2Bot, that uses the LVLM itself to compress the visual information in a task-agnostic manner. At the core of Fwd2bot there exists a "double-forward pass" training strategy, whereby, during the first forward pass, the LLM (of the LVLM) creates a bottleneck by condensing the visual information into a small number of summary tokens. Then, using the same LLM, the second forward pass processes the language instruction(s) alongside the summary tokens, used as a direct replacement for the image ones. The training signal is provided by two losses: an autoregressive one applied after the second pass that provides a direct optimization objective for compression, and a contrastive loss, applied after the first pass, that further boosts the representation strength, especially for discriminative tasks. The training is further enhanced by stage-specific adapters. We accompany the proposed method by an in-depth ablation study. Overall, Fwd2Bot results in highly-informative compressed representations suitable for both generative and discriminative tasks. For generative tasks, we offer a 2x higher compression rate without compromising the generative capabilities, setting a new state-of-the-art result. For discriminative tasks, we set a new state-of-the-art on image retrieval and compositionality.
• 摘要：在这项工作中，我们旨在将大型视觉语言模型（LVLM）的视觉令牌压缩为同时适合（a）生成和（b）判别任务的表示，（c）几乎是无损的，（d）是存储效率的。我们提出了一种新型的压缩方法，称为FWD2BOT，该方法使用LVLM本身以任务不合时宜的方式压缩视觉信息。在FWD2BOT的核心上，存在着一种“双向通行证”训练策略，因此，在第一个正向通行证中，LLM（LVLM的LLM）通过将视觉信息凝结成少量的摘要图表来创建瓶颈。然后，使用相同的LLM，第二个正向通行通过摘要令牌的语言指令处理，用作直接替换图像。训练信号由两个损失提供：第二次通过后应用的自回旋信号为压缩提供了直接优化的目标，并且在第一次通过后应用了对比度损失，这进一步提高了表示强度，尤其是针对歧视任务。特定于阶段的适配器进一步增强了培训。我们通过深入的消融研究伴随提出的方法。总体而言，FWD2BOT会导致高度信息的压缩表示形式，适用于生成和歧视任务。对于生成任务，我们提供的压缩率更高2倍，而不会损害生成能力，从而设定了新的最新结果。对于判别任务，我们为图像检索和组成性设置了一个新的最新作品。

Title: Lumina-Image 2.0: A Unified and Efficient Image Generative Framework

• Authors: Qi Qin, Le Zhuo, Yi Xin, Ruoyi Du, Zhen Li, Bin Fu, Yiting Lu, Jiakang Yuan, Xinyue Li, Dongyang Liu, Xiangyang Zhu, Manyuan Zhang, Will Beddow, Erwann Millon, Victor Perez, Wenhai Wang, Conghui He, Bo Zhang, Xiaohong Liu, Hongsheng Li, Yu Qiao, Chang Xu, Peng Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21758
• Pdf URL: https://arxiv.org/pdf/2503.21758
• Copy Paste: [[2503.21758]] Lumina-Image 2.0: A Unified and Efficient Image Generative Framework(https://arxiv.org/abs/2503.21758)
• Keywords: generation, generative
• Abstract: We introduce Lumina-Image 2.0, an advanced text-to-image generation framework that achieves significant progress compared to previous work, Lumina-Next. Lumina-Image 2.0 is built upon two key principles: (1) Unification - it adopts a unified architecture (Unified Next-DiT) that treats text and image tokens as a joint sequence, enabling natural cross-modal interactions and allowing seamless task expansion. Besides, since high-quality captioners can provide semantically well-aligned text-image training pairs, we introduce a unified captioning system, Unified Captioner (UniCap), specifically designed for T2I generation tasks. UniCap excels at generating comprehensive and accurate captions, accelerating convergence and enhancing prompt adherence. (2) Efficiency - to improve the efficiency of our proposed model, we develop multi-stage progressive training strategies and introduce inference acceleration techniques without compromising image quality. Extensive evaluations on academic benchmarks and public text-to-image arenas show that Lumina-Image 2.0 delivers strong performances even with only 2.6B parameters, highlighting its scalability and design efficiency. We have released our training details, code, and models at this https URL.
• 摘要：我们介绍了Lumina-image 2.0，这是一个先进的文本到图像生成框架，与以前的工作Lumina-Next相比，它取得了重大进展。 Lumina-image 2.0建立在两个关键原则上：（1）统一 - 采用统一的体系结构（统一的隔壁），将文本和图像令牌视为一个联合序列，从而实现了自然的交叉模式相互作用并允许无缝的任务扩展。此外，由于高质量的字幕符可以提供语义上良好的文本图像培训对，因此我们引入了统一的字幕系统，统一字幕仪（UNICAP），该系统专门为T2I生成任务设计。 Unicap擅长生成全面，准确的标题，加速融合并增强及时粘附。 （2）效率 - 为了提高我们提出的模型的效率，我们制定了多阶段的渐进式训练策略，并引入推理加速技术而不会损害图像质量。对学术基准和公共文本对图像竞技场的广泛评估表明，Lumina-image 2.0即使只有2.6B参数可提供强大的性能，从而突出了其可扩展性和设计效率。我们已经在此HTTPS URL上发布了培训详细信息，代码和模型。

Title: Exploring the Evolution of Physics Cognition in Video Generation: A Survey

• Authors: Minghui Lin, Xiang Wang, Yishan Wang, Shu Wang, Fengqi Dai, Pengxiang Ding, Cunxiang Wang, Zhengrong Zuo, Nong Sang, Siteng Huang, Donglin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21765
• Pdf URL: https://arxiv.org/pdf/2503.21765
• Copy Paste: [[2503.21765]] Exploring the Evolution of Physics Cognition in Video Generation: A Survey(https://arxiv.org/abs/2503.21765)
• Keywords: generation, generative
• Abstract: Recent advancements in video generation have witnessed significant progress, especially with the rapid advancement of diffusion models. Despite this, their deficiencies in physical cognition have gradually received widespread attention - generated content often violates the fundamental laws of physics, falling into the dilemma of ''visual realism but physical absurdity". Researchers began to increasingly recognize the importance of physical fidelity in video generation and attempted to integrate heuristic physical cognition such as motion representations and physical knowledge into generative systems to simulate real-world dynamic scenarios. Considering the lack of a systematic overview in this field, this survey aims to provide a comprehensive summary of architecture designs and their applications to fill this gap. Specifically, we discuss and organize the evolutionary process of physical cognition in video generation from a cognitive science perspective, while proposing a three-tier taxonomy: 1) basic schema perception for generation, 2) passive cognition of physical knowledge for generation, and 3) active cognition for world simulation, encompassing state-of-the-art methods, classical paradigms, and benchmarks. Subsequently, we emphasize the inherent key challenges in this domain and delineate potential pathways for future research, contributing to advancing the frontiers of discussion in both academia and industry. Through structured review and interdisciplinary analysis, this survey aims to provide directional guidance for developing interpretable, controllable, and physically consistent video generation paradigms, thereby propelling generative models from the stage of ''visual mimicry'' towards a new phase of ''human-like physical comprehension''.
• 摘要：视频发电的最新进展已经取得了重大进展，尤其是随着扩散模型的快速发展。尽管如此，他们在身体认知方面的缺陷逐渐受到了广泛的关注 - 产生的内容通常违反了物理的基本定律，陷入了“视觉现实主义但身体荒谬的困境”。研究人员开始越来越多地认识到身体忠诚度在视频中的重要性，并试图将动作的体现仿制为诸如动态的体验，例如，将其仿制为诸如动态的知识，以使其仿制物质的知识，例如，将其整合到运动中，例如，将其整合到运动中，以使其仿制物质化，以使其仿制物质的知识，以使其拟合体内的知识。在该领域缺乏系统概述，该调查旨在提供架构设计及其应用的全面摘要，以填补这一空白。包括最新方法，经典范式和基准。随后，我们强调了该领域中固有的主要挑战，并描述了未来研究的潜在途径，从而促进了学术界和行业的讨论前沿。通过结构化的审查和跨学科分析，该调查旨在为开发可解释，可控制和物理上一致的视频生成范式提供方向指导，从而使“视觉模仿”阶段的阶段推动“视觉模仿”阶段的阶段，向“人类样物理理解”的新阶段''。

Title: A Unified Image-Dense Annotation Generation Model for Underwater Scenes

• Authors: Hongkai Lin, Dingkang Liang, Zhenghao Qi, Xiang Bai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21771
• Pdf URL: https://arxiv.org/pdf/2503.21771
• Copy Paste: [[2503.21771]] A Unified Image-Dense Annotation Generation Model for Underwater Scenes(https://arxiv.org/abs/2503.21771)
• Keywords: generation
• Abstract: Underwater dense prediction, especially depth estimation and semantic segmentation, is crucial for gaining a comprehensive understanding of underwater scenes. Nevertheless, high-quality and large-scale underwater datasets with dense annotations remain scarce because of the complex environment and the exorbitant data collection costs. This paper proposes a unified Text-to-Image and DEnse annotation generation method (TIDE) for underwater scenes. It relies solely on text as input to simultaneously generate realistic underwater images and multiple highly consistent dense annotations. Specifically, we unify the generation of text-to-image and text-to-dense annotations within a single model. The Implicit Layout Sharing mechanism (ILS) and cross-modal interaction method called Time Adaptive Normalization (TAN) are introduced to jointly optimize the consistency between image and dense annotations. We synthesize a large-scale underwater dataset using TIDE to validate the effectiveness of our method in underwater dense prediction tasks. The results demonstrate that our method effectively improves the performance of existing underwater dense prediction models and mitigates the scarcity of underwater data with dense annotations. We hope our method can offer new perspectives on alleviating data scarcity issues in other fields. The code is available at https: //github.com/HongkLin/TIDE.
• 摘要：水下密集的预测，尤其是深度估计和语义细分，对于获得对水下场景的全面了解至关重要。然而，由于环境复杂和高昂的数据收集成本，具有致密注释的高质量和大规模的水下数据集仍然稀少。本文提出了用于水下场景的统一文本对图像和密集的注释生成方法（TIDE）。它仅依靠文本作为输入来同时生成逼真的水下图像和多个高度一致的致密注释。具体而言，我们统一单个模型中文本对图像和文本对密集注释的生成。引入了隐式布局共享机制（ILS）和跨模式相互作用方法，称为时间自适应归一化（TAN），以共同优化图像和密集注释之间的一致性。我们使用潮汐合成一个大规模的水下数据集，以验证我们方法在水下密集的预测任务中的有效性。结果表明，我们的方法有效地提高了现有的水下密集预测模型的性能，并减轻了具有致密注释的水下数据的稀缺性。我们希望我们的方法可以为减轻其他领域的数据稀缺问题提供新的观点。该代码可在https：//github.com/hongklin/tide上找到。

Title: Optimal Stepsize for Diffusion Sampling

• Authors: Jianning Pei, Han Hu, Shuyang Gu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21774
• Pdf URL: https://arxiv.org/pdf/2503.21774
• Copy Paste: [[2503.21774]] Optimal Stepsize for Diffusion Sampling(https://arxiv.org/abs/2503.21774)
• Keywords: generation
• Abstract: Diffusion models achieve remarkable generation quality but suffer from computational intensive sampling due to suboptimal step discretization. While existing works focus on optimizing denoising directions, we address the principled design of stepsize schedules. This paper proposes Optimal Stepsize Distillation, a dynamic programming framework that extracts theoretically optimal schedules by distilling knowledge from reference trajectories. By reformulating stepsize optimization as recursive error minimization, our method guarantees global discretization bounds through optimal substructure exploitation. Crucially, the distilled schedules demonstrate strong robustness across architectures, ODE solvers, and noise schedules. Experiments show 10x accelerated text-to-image generation while preserving 99.4% performance on GenEval. Our code is available at this https URL.
• 摘要：扩散模型具有显着的生成质量，但由于次优的离散化而受到计算密集型采样的损失。尽管现有作品着重于优化denoising指示，但我们介绍了步骤计划的原则性设计。本文提出了最佳的步骤蒸馏，这是一种动态编程框架，通过将知识从参考轨迹提取出来，从而提取理论上最佳的时间表。通过将步骤优化作为递归误差最小化，我们的方法通过最佳的子结构利用来保证全球离散化界限。至关重要的是，蒸馏的时间表表明了跨架构，ode求解器和噪声时间表的强大鲁棒性。实验显示10倍加速的文本到图像产生，同时保留了99.4％的遗传学性能。我们的代码可在此HTTPS URL上找到。

Title: StyleMotif: Multi-Modal Motion Stylization using Style-Content Cross Fusion

• Authors: Ziyu Guo, Young Yoon Lee, Joseph Liu, Yizhak Ben-Shabat, Victor Zordan, Mubbasir Kapadia
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.21775
• Pdf URL: https://arxiv.org/pdf/2503.21775
• Copy Paste: [[2503.21775]] StyleMotif: Multi-Modal Motion Stylization using Style-Content Cross Fusion(https://arxiv.org/abs/2503.21775)
• Keywords: generation
• Abstract: We present StyleMotif, a novel Stylized Motion Latent Diffusion model, generating motion conditioned on both content and style from multiple modalities. Unlike existing approaches that either focus on generating diverse motion content or transferring style from sequences, StyleMotif seamlessly synthesizes motion across a wide range of content while incorporating stylistic cues from multi-modal inputs, including motion, text, image, video, and audio. To achieve this, we introduce a style-content cross fusion mechanism and align a style encoder with a pre-trained multi-modal model, ensuring that the generated motion accurately captures the reference style while preserving realism. Extensive experiments demonstrate that our framework surpasses existing methods in stylized motion generation and exhibits emergent capabilities for multi-modal motion stylization, enabling more nuanced motion synthesis. Source code and pre-trained models will be released upon acceptance. Project Page: this https URL
• 摘要：我们提出了StyleMotif，这是一种新型的定型运动潜扩散模型，从多种方式产生了以内容和样式为条件的运动。与现有的方法不同，该方法要么着重于生成多样化的运动内容或从序列中转移样式，因此StyLemotif无缝地综合了各种内容的运动，同时结合了来自多模式输入的风格线索，包括运动，文本，图像，视频，视频和音频。为了实现这一目标，我们引入了一种风格的交叉融合机制，并将样式编码器与预训练的多模式模型对齐，以确保生成的运动能够准确地捕获参考样式的同时保留现实主义。广泛的实验表明，我们的框架超过了程式化运动产生的现有方法，并展示了多模式运动风格的新兴功能，从而实现了更细微的运动合成。源代码和预培训模型将在接受后发布。项目页面：此HTTPS URL

Title: VideoMage: Multi-Subject and Motion Customization of Text-to-Video Diffusion Models

• Authors: Chi-Pin Huang, Yen-Siang Wu, Hung-Kai Chung, Kai-Po Chang, Fu-En Yang, Yu-Chiang Frank Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.21781
• Pdf URL: https://arxiv.org/pdf/2503.21781
• Copy Paste: [[2503.21781]] VideoMage: Multi-Subject and Motion Customization of Text-to-Video Diffusion Models(https://arxiv.org/abs/2503.21781)
• Keywords: generation
• Abstract: Customized text-to-video generation aims to produce high-quality videos that incorporate user-specified subject identities or motion patterns. However, existing methods mainly focus on personalizing a single concept, either subject identity or motion pattern, limiting their effectiveness for multiple subjects with the desired motion patterns. To tackle this challenge, we propose a unified framework VideoMage for video customization over both multiple subjects and their interactive motions. VideoMage employs subject and motion LoRAs to capture personalized content from user-provided images and videos, along with an appearance-agnostic motion learning approach to disentangle motion patterns from visual appearance. Furthermore, we develop a spatial-temporal composition scheme to guide interactions among subjects within the desired motion patterns. Extensive experiments demonstrate that VideoMage outperforms existing methods, generating coherent, user-controlled videos with consistent subject identities and interactions.
• 摘要：定制的文本对视频生成旨在制作高质量的视频，以结合用户指定的主题身份或运动模式。但是，现有方法主要集中于个性化单个概念，即主题身份或运动模式，从而限制了其对多个受试者的有效性，并具有所需的运动模式。为了应对这一挑战，我们为多个主题及其互动动作提出了一个统一的框架视频，以定制视频定制。视频采用主题和运动洛拉斯从用户提供的图像和视频中捕获个性化的内容，以及一种外观不可思议的运动学习方法，从视觉外观中解开了运动模式。此外，我们开发了一种时空组成方案，以指导所需运动模式中受试者之间的相互作用。广泛的实验表明，视频表现优于现有方法，生成具有一致的主题身份和互动的连贯，用户控制的视频。