2025-04-14

Title: Have we unified image generation and understanding yet? An empirical study of GPT-4o's image generation ability

• Authors: Ning Li, Jingran Zhang, Justin Cui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08003
• Pdf URL: https://arxiv.org/pdf/2504.08003
• Copy Paste: [[2504.08003]] Have we unified image generation and understanding yet? An empirical study of GPT-4o's image generation ability(https://arxiv.org/abs/2504.08003)
• Keywords: generation
• Abstract: OpenAI's multimodal GPT-4o has demonstrated remarkable capabilities in image generation and editing, yet its ability to achieve world knowledge-informed semantic synthesis--seamlessly integrating domain knowledge, contextual reasoning, and instruction adherence--remains unproven. In this study, we systematically evaluate these capabilities across three critical dimensions: (1) Global Instruction Adherence, (2) Fine-Grained Editing Precision, and (3) Post-Generation Reasoning. While existing benchmarks highlight GPT-4o's strong capabilities in image generation and editing, our evaluation reveals GPT-4o's persistent limitations: the model frequently defaults to literal interpretations of instructions, inconsistently applies knowledge constraints, and struggles with conditional reasoning tasks. These findings challenge prevailing assumptions about GPT-4o's unified understanding and generation capabilities, exposing significant gaps in its dynamic knowledge integration. Our study calls for the development of more robust benchmarks and training strategies that go beyond surface-level alignment, emphasizing context-aware and reasoning-grounded multimodal generation.
• 摘要：Openai的多模式GPT-4O在图像生成和编辑中表现出了非凡的功能，但它具有实现世界知识知识的语义综合的能力 - 无需整合域知识，上下文推理和指导依从性 - 依据 - 尚未证实。在这项研究中，我们系统地评估了三个关键维度的这些功能：（1）全球指导依从性，（2）精细颗粒的编辑精度和（3）产后推理。尽管现有基准强调了GPT-4O在图像生成和编辑中的强大功能，但我们的评估揭示了GPT-4O的持续局限性：该模型经常默认用于指令的字面解释，不一致地应用知识限制，并在有条件的推理任务中挣扎。这些发现挑战了有关GPT-4O统一的理解和发电能力的普遍假设，从而在其动态知识整合中揭示了很大的差距。我们的研究要求开发更强大的基准和训练策略，这些策略超出了表面层面的一致性，强调了情境感知和推理的多模式生成。

Title: Teaching Humans Subtle Differences with DIFFusion

• Authors: Mia Chiquier, Orr Avrech, Yossi Gandelsman, Berthy Feng, Katherine Bouman, Carl Vondrick
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08046
• Pdf URL: https://arxiv.org/pdf/2504.08046
• Copy Paste: [[2504.08046]] Teaching Humans Subtle Differences with DIFFusion(https://arxiv.org/abs/2504.08046)
• Keywords: generative
• Abstract: Human expertise depends on the ability to recognize subtle visual differences, such as distinguishing diseases, species, or celestial phenomena. We propose a new method to teach novices how to differentiate between nuanced categories in specialized domains. Our method uses generative models to visualize the minimal change in features to transition between classes, i.e., counterfactuals, and performs well even in domains where data is sparse, examples are unpaired, and category boundaries are not easily explained by text. By manipulating the conditioning space of diffusion models, our proposed method DIFFusion disentangles category structure from instance identity, enabling high-fidelity synthesis even in challenging domains. Experiments across six domains show accurate transitions even with limited and unpaired examples across categories. User studies confirm that our generated counterfactuals outperform unpaired examples in teaching perceptual expertise, showing the potential of generative models for specialized visual learning.
• 摘要：人类的专业知识取决于识别细微的视觉差异的能力，例如区分疾病，物种或天体现象。我们提出了一种新的方法来教新手如何区分专用领域中细微的类别。我们的方法使用生成模型来可视化特征的最小变化，即相反事实之间的过渡到过渡，甚至在数据稀疏，示例不合格的域，并且类别边界也不容易通过文本解释。通过操纵扩散模型的调理空间，我们提出的方法扩散分离类别结构与实例身份相结构，即使在具有挑战性的域中也可以使高保真综合。跨六个领域的实验也显示出准确的过渡，即使跨类别的示例有限且未配对的示例。用户研究证实，我们生成的反事实在教学专业知识中的表现优于未配对的示例，显示了生成模型的专业视觉学习潜力。

Title: Compositional Flows for 3D Molecule and Synthesis Pathway Co-design

• Authors: Tony Shen, Seonghwan Seo, Ross Irwin, Kieran Didi, Simon Olsson, Woo Youn Kim, Martin Ester
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.08051
• Pdf URL: https://arxiv.org/pdf/2504.08051
• Copy Paste: [[2504.08051]] Compositional Flows for 3D Molecule and Synthesis Pathway Co-design(https://arxiv.org/abs/2504.08051)
• Keywords: generative
• Abstract: Many generative applications, such as synthesis-based 3D molecular design, involve constructing compositional objects with continuous features. Here, we introduce Compositional Generative Flows (CGFlow), a novel framework that extends flow matching to generate objects in compositional steps while modeling continuous states. Our key insight is that modeling compositional state transitions can be formulated as a straightforward extension of the flow matching interpolation process. We further build upon the theoretical foundations of generative flow networks (GFlowNets), enabling reward-guided sampling of compositional structures. We apply CGFlow to synthesizable drug design by jointly designing the molecule's synthetic pathway with its 3D binding pose. Our approach achieves state-of-the-art binding affinity on all 15 targets from the LIT-PCBA benchmark, and 5.8$\times$ improvement in sampling efficiency compared to 2D synthesis-based baseline. To our best knowledge, our method is also the first to achieve state of-art-performance in both Vina Dock (-9.38) and AiZynth success rate (62.2\%) on the CrossDocked benchmark.
• 摘要：许多生成应用，例如基于合成的3D分子设计，涉及构建具有连续特征的组成对象。在这里，我们引入了组成生成流（CGFLOW），这是一个新颖的框架，该框架扩展了流量匹配，以在组成步骤中生成对象，同时建模连续状态。我们的关键见解是，建模组成状态转换可以作为流动匹配插值过程的直接扩展。我们进一步建立在生成流网络（GFLOWNETS）的理论基础上，从而实现了组成结构的奖励指导采样。我们通过以3D结合姿势共同设计该分子的合成途径，将CGFLOF应用于可综合的药物设计。与基于2D合成基线相比，我们的方法对LIT-PCBA基准测试的所有15个目标的所有15个目标都达到了最新的结合亲和力，并提高了5.8 $ \ times $。据我们所知，我们的方法也是第一个在Vina Dock（-9.38）（-9.38）和Aizynth成功率（62.2 \％）上实现Art绩效状态的方法。

Title: X-DECODE: EXtreme Deblurring with Curriculum Optimization and Domain Equalization

• Authors: Sushant Gautam, Jingdao Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08072
• Pdf URL: https://arxiv.org/pdf/2504.08072
• Copy Paste: [[2504.08072]] X-DECODE: EXtreme Deblurring with Curriculum Optimization and Domain Equalization(https://arxiv.org/abs/2504.08072)
• Keywords: restoration
• Abstract: Restoring severely blurred images remains a significant challenge in computer vision, impacting applications in autonomous driving, medical imaging, and photography. This paper introduces a novel training strategy based on curriculum learning to improve the robustness of deep learning models for extreme image deblurring. Unlike conventional approaches that train on only low to moderate blur levels, our method progressively increases the difficulty by introducing images with higher blur severity over time, allowing the model to adapt incrementally. Additionally, we integrate perceptual and hinge loss during training to enhance fine detail restoration and improve training stability. We experimented with various curriculum learning strategies and explored the impact of the train-test domain gap on the deblurring performance. Experimental results on the Extreme-GoPro dataset showed that our method outperforms the next best method by 14% in SSIM, whereas experiments on the Extreme-KITTI dataset showed that our method outperforms the next best by 18% in SSIM. Ablation studies showed that a linear curriculum progression outperforms step-wise, sigmoid, and exponential progressions, while hyperparameter settings such as the training blur percentage and loss function formulation all play important roles in addressing extreme blur artifacts. Datasets and code are available at this https URL
• 摘要：恢复严重模糊的图像在计算机视觉中仍然是一个重大挑战，影响了自动驾驶，医学成像和摄影的应用。本文介绍了一种基于课程学习的新型培训策略，以提高深度学习模型的鲁棒性，以实现极端图像造影。与仅在低至中度模糊水平上训练的常规方法不同，我们的方法通过随着时间的推移引入较高的模糊严重程度的图像来逐渐增加难度，从而使模型可以逐步适应。此外，我们在训练过程中整合了感知和铰链损失，以增强细节恢复并提高训练稳定性。我们尝试了各种课程学习策略，并探讨了火车测试域间隙对脱张性能的影响。极端gopro数据集的实验结果表明，在SSIM中，我们的方法优于下一个最佳方法，而极端kitti数据集的实验表明，我们的方法在SSIM中的下一个优于下一个最佳状态。消融研究表明，线性课程的进展优于逐步的，乙状结肠和指数性进度，而诸如训练模糊百分比和损失函数等高参数设置都在解决极端模糊伪影方面起着重要作用。数据集和代码可在此HTTPS URL上找到

Title: ContrastiveGaussian: High-Fidelity 3D Generation with Contrastive Learning and Gaussian Splatting

• Authors: Junbang Liu, Enpei Huang, Dongxing Mao, Hui Zhang, Xinyuan Song, Yongxin Ni
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08100
• Pdf URL: https://arxiv.org/pdf/2504.08100
• Copy Paste: [[2504.08100]] ContrastiveGaussian: High-Fidelity 3D Generation with Contrastive Learning and Gaussian Splatting(https://arxiv.org/abs/2504.08100)
• Keywords: super-resolution, generation, generative
• Abstract: Creating 3D content from single-view images is a challenging problem that has attracted considerable attention in recent years. Current approaches typically utilize score distillation sampling (SDS) from pre-trained 2D diffusion models to generate multi-view 3D representations. Although some methods have made notable progress by balancing generation speed and model quality, their performance is often limited by the visual inconsistencies of the diffusion model outputs. In this work, we propose ContrastiveGaussian, which integrates contrastive learning into the generative process. By using a perceptual loss, we effectively differentiate between positive and negative samples, leveraging the visual inconsistencies to improve 3D generation quality. To further enhance sample differentiation and improve contrastive learning, we incorporate a super-resolution model and introduce another Quantity-Aware Triplet Loss to address varying sample distributions during training. Our experiments demonstrate that our approach achieves superior texture fidelity and improved geometric consistency.
• 摘要：从单视图像中创建3D内容是一个具有挑战性的问题，近年来引起了很大的关注。当前方法通常利用预先训练的2D扩散模型的得分蒸馏采样（SDS）来生成多视图3D表示。尽管某些方法通过平衡生成速度和模型质量取得了显着的进步，但它们的性能通常受到扩散模型输出的视觉不一致的限制。在这项工作中，我们提出了对比的高斯，将对比度学习整合到生成过程中。通过使用感知损失，我们有效地区分了正和负样本，利用视觉上的不一致来提高3D代质量。为了进一步增强样品分化并改善对比度学习，我们结合了一个超分辨率模型，并引入了另一个数量感知的三重态损失，以解决训练过程中不同样本分布的问题。我们的实验表明，我们的方法实现了优越的纹理保真度并提高了几何一致性。

Title: POEM: Precise Object-level Editing via MLLM control

• Authors: Marco Schouten, Mehmet Onurcan Kaya, Serge Belongie, Dim P. Papadopoulos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08111
• Pdf URL: https://arxiv.org/pdf/2504.08111
• Copy Paste: [[2504.08111]] POEM: Precise Object-level Editing via MLLM control(https://arxiv.org/abs/2504.08111)
• Keywords: generation
• Abstract: Diffusion models have significantly improved text-to-image generation, producing high-quality, realistic images from textual descriptions. Beyond generation, object-level image editing remains a challenging problem, requiring precise modifications while preserving visual coherence. Existing text-based instructional editing methods struggle with localized shape and layout transformations, often introducing unintended global changes. Image interaction-based approaches offer better accuracy but require manual human effort to provide precise guidance. To reduce this manual effort while maintaining a high image editing accuracy, in this paper, we propose POEM, a framework for Precise Object-level Editing using Multimodal Large Language Models (MLLMs). POEM leverages MLLMs to analyze instructional prompts and generate precise object masks before and after transformation, enabling fine-grained control without extensive user input. This structured reasoning stage guides the diffusion-based editing process, ensuring accurate object localization and transformation. To evaluate our approach, we introduce VOCEdits, a benchmark dataset based on PASCAL VOC 2012, augmented with instructional edit prompts, ground-truth transformations, and precise object masks. Experimental results show that POEM outperforms existing text-based image editing approaches in precision and reliability while reducing manual effort compared to interaction-based methods.
• 摘要：扩散模型已显着改善了文本形象的生成，从文本描述中产生了高质量的现实图像。除了产生外，对象级图像编辑仍然是一个具有挑战性的问题，需要精确的修改，同时保持视觉连贯性。现有的基于文本的教学编辑方法与本地化形状和布局转换相努力，通常会引入意想不到的全球变化。基于图像相互作用的方法提供了更好的准确性，但需要手动人工努力来提供精确的指导。为了减少这种手动努力，同时保持高图像编辑精度，在本文中，我们提出了诗歌，这是使用多模式大语言模型（MLLM）精确对象级编辑的框架。诗歌利用MLLM分析教学提示并在转换前后生成精确的对象掩码，从而无需大量的用户输入即可获得细粒度的控制。该结构化推理阶段指导基于扩散的编辑过程，以确保准确的对象定位和转换。为了评估我们的方法，我们介绍了基于Pascal VOC 2012的基准数据集介绍Vocedits，并增强了教学编辑提示，地面真相转换和精确的对象掩码。实验结果表明，与基于相互作用的方法相比，诗的精度和可靠性优于现有的基于文本的图像编辑方法，同时减少手动努力。

Title: Gen3DEval: Using vLLMs for Automatic Evaluation of Generated 3D Objects

• Authors: Shalini Maiti, Lourdes Agapito, Filippos Kokkinos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08125
• Pdf URL: https://arxiv.org/pdf/2504.08125
• Copy Paste: [[2504.08125]] Gen3DEval: Using vLLMs for Automatic Evaluation of Generated 3D Objects(https://arxiv.org/abs/2504.08125)
• Keywords: generation, quality assessment
• Abstract: Rapid advancements in text-to-3D generation require robust and scalable evaluation metrics that align closely with human judgment, a need unmet by current metrics such as PSNR and CLIP, which require ground-truth data or focus only on prompt fidelity. To address this, we introduce Gen3DEval, a novel evaluation framework that leverages vision large language models (vLLMs) specifically fine-tuned for 3D object quality assessment. Gen3DEval evaluates text fidelity, appearance, and surface quality by analyzing 3D surface normals, without requiring ground-truth comparisons, bridging the gap between automated metrics and user preferences. Compared to state-of-the-art task-agnostic models, Gen3DEval demonstrates superior performance in user-aligned evaluations, placing it as a comprehensive and accessible benchmark for future research on text-to-3D generation. The project page can be found here: \href{this https URL}{this https URL}.
• 摘要：文本到3D生成的快速进步需要与人类判断紧密相吻合的可靠和可扩展的评估指标，这是当前指标（例如PSNR和CLIP）未满足的，这些指标（例如PSNR和CLIP）需要地面可观的数据或仅关注迅速的保真度。为了解决这个问题，我们介绍了Gen3Deval，这是一个新型的评估框架，该框架利用视觉模型（VLLM）专门调整了3D对象质量评估。 Gen3Deval通过分析3D表面正常质量来评估文本保真度，外观和表面质量，而无需进行基础真相比较，从而弥合了自动指标和用户偏好之间的差距。与最先进的任务不可策划模型相比，Gen3Deval在用户一致的评估中表现出卓越的性能，将其作为对文本到3d世代的未来研究的全面且易于访问的基准。可以在此处找到项目页面：\ href {此https url} {this https url}。

Title: LoRAX: LoRA eXpandable Networks for Continual Synthetic Image Attribution

• Authors: Danielle Sullivan-Pao, Nicole Tian, Pooya Khorrami
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08149
• Pdf URL: https://arxiv.org/pdf/2504.08149
• Copy Paste: [[2504.08149]] LoRAX: LoRA eXpandable Networks for Continual Synthetic Image Attribution(https://arxiv.org/abs/2504.08149)
• Keywords: generative
• Abstract: As generative AI image technologies become more widespread and advanced, there is a growing need for strong attribution models. These models are crucial for verifying the authenticity of images and identifying the architecture of their originating generative models-key to maintaining media integrity. However, attribution models struggle to generalize to unseen models, and traditional fine-tuning methods for updating these models have shown to be impractical in real-world settings. To address these challenges, we propose LoRA eXpandable Networks (LoRAX), a parameter-efficient class incremental algorithm that adapts to novel generative image models without the need for full retraining. Our approach trains an extremely parameter-efficient feature extractor per continual learning task via Low Rank Adaptation. Each task-specific feature extractor learns distinct features while only requiring a small fraction of the parameters present in the underlying feature extractor's backbone model. Our extensive experimentation shows LoRAX outperforms or remains competitive with state-of-the-art class incremental learning algorithms on the Continual Deepfake Detection benchmark across all training scenarios and memory settings, while requiring less than 3% of the number of trainable parameters per feature extractor compared to the full-rank implementation. LoRAX code is available at: this https URL.
• 摘要：随着生成的AI图像技术变得更加广泛和高级，对强大归因模型的需求越来越大。这些模型对于验证图像的真实性和识别其发起生成模型键的结构以维持媒体完整性至关重要。但是，归因模型难以推广到看不见的模型，并且在现实世界中，更新这些模型的传统微调方法已显示不切实际。为了应对这些挑战，我们提出了Lora可扩展网络（Lorax），这是一种参数效率的类增量算法，可适应新颖的生成图像模型，而无需完全重新培训。我们的方法通过低级适应来训练每个连续学习任务的极其参数效率的提取器。每个特定于任务的特征提取器都学习不同的特征，而仅需要一小部分参数中存在的特征提取器的骨干模型中。我们的广泛实验表明，在所有训练场景和内存设置中，在连续的DeepFake检测基准上与最先进的类递增学习算法保持竞争能力，而与全层次实施相比，每个训练场景和内存设置中的持续深层检测基准都需要少于3％的可训练参数。 Lorax代码可用：此HTTPS URL。

Title: TokenMotion: Decoupled Motion Control via Token Disentanglement for Human-centric Video Generation

• Authors: Ruineng Li, Daitao Xing, Huiming Sun, Yuanzhou Ha, Jinglin Shen, Chiuman Ho
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.08181
• Pdf URL: https://arxiv.org/pdf/2504.08181
• Copy Paste: [[2504.08181]] TokenMotion: Decoupled Motion Control via Token Disentanglement for Human-centric Video Generation(https://arxiv.org/abs/2504.08181)
• Keywords: generation
• Abstract: Human-centric motion control in video generation remains a critical challenge, particularly when jointly controlling camera movements and human poses in scenarios like the iconic Grammy Glambot moment. While recent video diffusion models have made significant progress, existing approaches struggle with limited motion representations and inadequate integration of camera and human motion controls. In this work, we present TokenMotion, the first DiT-based video diffusion framework that enables fine-grained control over camera motion, human motion, and their joint interaction. We represent camera trajectories and human poses as spatio-temporal tokens to enable local control granularity. Our approach introduces a unified modeling framework utilizing a decouple-and-fuse strategy, bridged by a human-aware dynamic mask that effectively handles the spatially-and-temporally varying nature of combined motion signals. Through extensive experiments, we demonstrate TokenMotion's effectiveness across both text-to-video and image-to-video paradigms, consistently outperforming current state-of-the-art methods in human-centric motion control tasks. Our work represents a significant advancement in controllable video generation, with particular relevance for creative production applications.
• 摘要：视频生成中以人为中心的运动控制仍然是一个关键的挑战，尤其是当在标志性的格莱美格莱姆·格兰博特时刻（Grammy Glambot）等场景中共同控制相机运动和人类姿势时。尽管最近的视频扩散模型取得了重大进展，但现有的方法在有限的运动表示和相机和人类运动控制的集成方面遇到了障碍。在这项工作中，我们提出了TokenMotion，这是第一个基于DIT的视频扩散框架，可以对相机运动，人体运动及其关节相互作用进行细粒度的控制。我们代表摄像机轨迹和人类姿势是时空令牌，以实现局部控制粒度。我们的方法采用了一种统一的建模框架，利用了一种脱离和配合的策略，该策略是由人类感知的动态掩码桥接的，该掩模有效地处理了合并运动信号的空间和周期性变化的性质。通过广泛的实验，我们在文本对视频和图像到视频范式上都表明了令牌的有效性，在以人为中心的运动控制任务中始终优于当前最新方法。我们的工作代表了可控视频生成的重大进步，与创意生产应用程序特别相关。

Title: RealCam-Vid: High-resolution Video Dataset with Dynamic Scenes and Metric-scale Camera Movements

• Authors: Guangcong Zheng, Teng Li, Xianpan Zhou, Xi Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08212
• Pdf URL: https://arxiv.org/pdf/2504.08212
• Copy Paste: [[2504.08212]] RealCam-Vid: High-resolution Video Dataset with Dynamic Scenes and Metric-scale Camera Movements(https://arxiv.org/abs/2504.08212)
• Keywords: generation
• Abstract: Recent advances in camera-controllable video generation have been constrained by the reliance on static-scene datasets with relative-scale camera annotations, such as RealEstate10K. While these datasets enable basic viewpoint control, they fail to capture dynamic scene interactions and lack metric-scale geometric consistency-critical for synthesizing realistic object motions and precise camera trajectories in complex environments. To bridge this gap, we introduce the first fully open-source, high-resolution dynamic-scene dataset with metric-scale camera annotations in this https URL.
• 摘要：摄像机控制视频生成的最新进展受到对具有相对尺度相机注释（例如realestate10k）的静态场景数据集的依赖。尽管这些数据集启用了基本的观点控制，但它们无法捕获动态场景交互，并且缺乏关键的度量尺度几何一致性，以综合现实的对象运动和在复杂环境中的精确摄像机轨迹。为了弥合这一差距，我们介绍了第一个完全开源的高分辨率动态场景数据集，并在此HTTPS URL中带有度量尺度相机注释。

Title: VL-UR: Vision-Language-guided Universal Restoration of Images Degraded by Adverse Weather Conditions

• Authors: Ziyan Liu, Yuxu Lu, Huashan Yu, Dong yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08219
• Pdf URL: https://arxiv.org/pdf/2504.08219
• Copy Paste: [[2504.08219]] VL-UR: Vision-Language-guided Universal Restoration of Images Degraded by Adverse Weather Conditions(https://arxiv.org/abs/2504.08219)
• Keywords: restoration
• Abstract: Image restoration is critical for improving the quality of degraded images, which is vital for applications like autonomous driving, security surveillance, and digital content enhancement. However, existing methods are often tailored to specific degradation scenarios, limiting their adaptability to the diverse and complex challenges in real-world environments. Moreover, real-world degradations are typically non-uniform, highlighting the need for adaptive and intelligent solutions. To address these issues, we propose a novel vision-language-guided universal restoration (VL-UR) framework. VL-UR leverages a zero-shot contrastive language-image pre-training (CLIP) model to enhance image restoration by integrating visual and semantic information. A scene classifier is introduced to adapt CLIP, generating high-quality language embeddings aligned with degraded images while predicting degraded types for complex scenarios. Extensive experiments across eleven diverse degradation settings demonstrate VL-UR's state-of-the-art performance, robustness, and adaptability. This positions VL-UR as a transformative solution for modern image restoration challenges in dynamic, real-world environments.
• 摘要：图像恢复对于提高退化图像的质量至关重要，这对于诸如自主驾驶，安全监视和数字内容增强之类的应用至关重要。但是，现有方法通常是针对特定的降解场景量身定制的，将其适应性限制在现实环境中各种各样且复杂的挑战中。此外，现实世界中的退化通常是不均匀的，强调了对自适应和智能解决方案的需求。为了解决这些问题，我们提出了一个新颖的视觉引导的普遍恢复（VL-ur）框架。 VL-ur利用零拍对对比的语言图像预训练（剪辑）模型来通过整合视觉和语义信息来增强图像恢复。引入了场景分类器以调整剪辑，生成与降级图像对齐的高质量语言嵌入，同时预测复杂场景的退化类型。在11种不同的退化设置进行的广泛实验表明，VL-ur的最先进的表现，鲁棒性和适应性。这将VL-ur定位为在动态，现实世界环境中对现代形象恢复挑战的变革解决方案。

Title: CoProSketch: Controllable and Progressive Sketch Generation with Diffusion Model

• Authors: Ruohao Zhan, Yijin Li, Yisheng He, Shuo Chen, Yichen Shen, Xinyu Chen, Zilong Dong, Zhaoyang Huang, Guofeng Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.08259
• Pdf URL: https://arxiv.org/pdf/2504.08259
• Copy Paste: [[2504.08259]] CoProSketch: Controllable and Progressive Sketch Generation with Diffusion Model(https://arxiv.org/abs/2504.08259)
• Keywords: generation, generative
• Abstract: Sketches serve as fundamental blueprints in artistic creation because sketch editing is easier and more intuitive than pixel-level RGB image editing for painting artists, yet sketch generation remains unexplored despite advancements in generative models. We propose a novel framework CoProSketch, providing prominent controllability and details for sketch generation with diffusion models. A straightforward method is fine-tuning a pretrained image generation diffusion model with binarized sketch images. However, we find that the diffusion models fail to generate clear binary images, which makes the produced sketches chaotic. We thus propose to represent the sketches by unsigned distance field (UDF), which is continuous and can be easily decoded to sketches through a lightweight network. With CoProSketch, users generate a rough sketch from a bounding box and a text prompt. The rough sketch can be manually edited and fed back into the model for iterative refinement and will be decoded to a detailed sketch as the final result. Additionally, we curate the first large-scale text-sketch paired dataset as the training data. Experiments demonstrate superior semantic consistency and controllability over baselines, offering a practical solution for integrating user feedback into generative workflows.
• 摘要：草图是艺术创作中的基本蓝图，因为草图编辑比像素级RGB图像编辑更容易，更直观，但是尽管生成模型的进步，但素描生成仍未得到探索。我们提出了一个新颖的框架Coprosketch，为通过扩散模型提供了突出的可控性和细节。一种直接的方法是用二进制的草图图像微调验证的图像生成扩散模型。但是，我们发现扩散模型无法生成清晰的二进制图像，这使制作的草图混乱。因此，我们建议通过无符号距离字段（UDF）表示草图，该距离是连续的，可以通过轻量级网络轻松解码为草图。使用Coprosketch，用户可以从边界框中生成粗糙的草图和一个文本提示。可以手动编辑粗糙的草图，并将其馈回模型以进行迭代细化，并将解码为详细的草图作为最终结果。此外，我们将第一个大规模的文本缝制配对数据集作为培训数据。实验表明，与基线相比，具有优越的语义一致性和可控性，提供了将用户反馈整合到生成工作流中的实用解决方案。

Title: DreamFuse: Adaptive Image Fusion with Diffusion Transformer

• Authors: Junjia Huang, Pengxiang Yan, Jiyang Liu, Jie Wu, Zhao Wang, Yitong Wang, Liang Lin, Guanbin Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08291
• Pdf URL: https://arxiv.org/pdf/2504.08291
• Copy Paste: [[2504.08291]] DreamFuse: Adaptive Image Fusion with Diffusion Transformer(https://arxiv.org/abs/2504.08291)
• Keywords: generation
• Abstract: Image fusion seeks to seamlessly integrate foreground objects with background scenes, producing realistic and harmonious fused images. Unlike existing methods that directly insert objects into the background, adaptive and interactive fusion remains a challenging yet appealing task. It requires the foreground to adjust or interact with the background context, enabling more coherent integration. To address this, we propose an iterative human-in-the-loop data generation pipeline, which leverages limited initial data with diverse textual prompts to generate fusion datasets across various scenarios and interactions, including placement, holding, wearing, and style transfer. Building on this, we introduce DreamFuse, a novel approach based on the Diffusion Transformer (DiT) model, to generate consistent and harmonious fused images with both foreground and background information. DreamFuse employs a Positional Affine mechanism to inject the size and position of the foreground into the background, enabling effective foreground-background interaction through shared attention. Furthermore, we apply Localized Direct Preference Optimization guided by human feedback to refine DreamFuse, enhancing background consistency and foreground harmony. DreamFuse achieves harmonious fusion while generalizing to text-driven attribute editing of the fused results. Experimental results demonstrate that our method outperforms state-of-the-art approaches across multiple metrics.
• 摘要：图像Fusion试图将前景对象与背景场景无缝整合，从而产生逼真而和谐的融合图像。与直接将对象插入背景的现有方法不同，自适应和交互式融合仍然是一项具有挑战性但有吸引力的任务。它要求前景调整或与背景上下文相互作用，从而实现更连贯的集成。为了解决这个问题，我们提出了一个迭代的人类数据生成管道，该管道利用有限的初始数据和不同的文本提示提示在各种情况和互动中生成融合数据集，包括放置，保持，保持，磨损和样式转移。在此基础上，我们介绍了Dreamfuse，这是一种基于扩散变压器（DIT）模型的新颖方法，以生成具有前景和背景信息的一致且和谐的融合图像。 Dreamfuse采用位置仿射机制，将前景的大小和位置注入背景，从而通过共同的注意来实现有效的前景互动。此外，我们应用以人为反馈为指导的局部直接优化优化，以完善梦幻般的范围，增强背景一致性和前景和谐。 Dreamfuse可以实现和谐的融合，同时推广到融合结果的文本驱动属性编辑。实验结果表明，我们的方法优于多个指标的最先进方法。

Title: Generative AI for Film Creation: A Survey of Recent Advances

• Authors: Ruihan Zhang, Borou Yu, Jiajian Min, Yetong Xin, Zheng Wei, Juncheng Nemo Shi, Mingzhen Huang, Xianghao Kong, Nix Liu Xin, Shanshan Jiang, Praagya Bahuguna, Mark Chan, Khushi Hora, Lijian Yang, Yongqi Liang, Runhe Bian, Yunlei Liu, Isabela Campillo Valencia, Patricia Morales Tredinick, Ilia Kozlov, Sijia Jiang, Peiwen Huang, Na Chen, Xuanxuan Liu, Anyi Rao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08296
• Pdf URL: https://arxiv.org/pdf/2504.08296
• Copy Paste: [[2504.08296]] Generative AI for Film Creation: A Survey of Recent Advances(https://arxiv.org/abs/2504.08296)
• Keywords: generation, generative
• Abstract: Generative AI (GenAI) is transforming filmmaking, equipping artists with tools like text-to-image and image-to-video diffusion, neural radiance fields, avatar generation, and 3D synthesis. This paper examines the adoption of these technologies in filmmaking, analyzing workflows from recent AI-driven films to understand how GenAI contributes to character creation, aesthetic styling, and narration. We explore key strategies for maintaining character consistency, achieving stylistic coherence, and ensuring motion continuity. Additionally, we highlight emerging trends such as the growing use of 3D generation and the integration of real footage with AI-generated elements. Beyond technical advancements, we examine how GenAI is enabling new artistic expressions, from generating hard-to-shoot footage to dreamlike diffusion-based morphing effects, abstract visuals, and unworldly objects. We also gather artists' feedback on challenges and desired improvements, including consistency, controllability, fine-grained editing, and motion refinement. Our study provides insights into the evolving intersection of AI and filmmaking, offering a roadmap for researchers and artists navigating this rapidly expanding field.
• 摘要：Generative AI（Genai）正在改变电影制作，为艺术家提供文本对图像和图像到视频扩散，神经辐射场，头像产生和3D合成等工具。本文研究了这些技术在电影制作中的采用，分析了最近的AI驱动电影中的工作流程，以了解Genai如何贡献角色创作，美学风格和叙事。我们探讨了保持角色一致性，实现风格连贯性和确保运动连续性的关键策略。此外，我们重点介绍了新兴趋势，例如不断增长的3D生成以及将真实镜头与AI生成的元素整合在一起。除了技术进步之外，我们还研究了Genai如何启用新的艺术表达方式，从生成难以拍摄的镜头到基于梦幻的扩散效果，抽象的视觉效果和非凡的对象。我们还收集了艺术家对挑战和所需改进的反馈，包括一致性，可控性，细粒度的编辑和运动改进。我们的研究提供了有关AI和电影制作不断发展的交汇处的见解，为研究人员和艺术家提供了迅速扩展领域的路线图。

Title: EasyGenNet: An Efficient Framework for Audio-Driven Gesture Video Generation Based on Diffusion Model

• Authors: Renda Li, Xiaohua Qi, Qiang Ling, Jun Yu, Ziyi Chen, Peng Chang, Mei HanJing Xiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08344
• Pdf URL: https://arxiv.org/pdf/2504.08344
• Copy Paste: [[2504.08344]] EasyGenNet: An Efficient Framework for Audio-Driven Gesture Video Generation Based on Diffusion Model(https://arxiv.org/abs/2504.08344)
• Keywords: generation
• Abstract: Audio-driven cospeech video generation typically involves two stages: speech-to-gesture and gesture-to-video. While significant advances have been made in speech-to-gesture generation, synthesizing natural expressions and gestures remains challenging in gesture-to-video systems. In order to improve the generation effect, previous works adopted complex input and training strategies and required a large amount of data sets for pre-training, which brought inconvenience to practical applications. We propose a simple one-stage training method and a temporal inference method based on a diffusion model to synthesize realistic and continuous gesture videos without the need for additional training of temporal this http URL entire model makes use of existing pre-trained weights, and only a few thousand frames of data are needed for each character at a time to complete fine-tuning. Built upon the video generator, we introduce a new audio-to-video pipeline to synthesize co-speech videos, using 2D human skeleton as the intermediate motion representation. Our experiments show that our method outperforms existing GAN-based and diffusion-based methods.
• 摘要：音频驱动的同性恋视频的生成通常涉及两个阶段：语音到人的态度和手势到视频。尽管在语音到人的传说中已经取得了重大进展，但在手势到视频系统中，综合自然表达式和手势仍然具有挑战性。为了改善生成效应，以前的工作采用了复杂的输入和培训策略，并需要大量的数据集进行预训练，这给实际应用带来了不便。我们提出了一种基于扩散模型的简单一阶段训练方法和一种时间推理方法，以合成现实且连续的手势视频，而无需进行时间额外的时间培训。整个模型整个模型都可以使用现有的预训练的重量，并且每个字符都需要几千帧的数据来完成调查。我们在视频生成器上构建了一条新的音频到视频管道，以使用2D人类骨架作为中间运动表示，以合成共同语音视频。我们的实验表明，我们的方法的表现优于现有的基于GAN的方法和基于扩散的方法。

Title: LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMs

• Authors: Jiarui Wang, Huiyu Duan, Yu Zhao, Juntong Wang, Guangtao Zhai, Xiongkuo Min
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08358
• Pdf URL: https://arxiv.org/pdf/2504.08358
• Copy Paste: [[2504.08358]] LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMs(https://arxiv.org/abs/2504.08358)
• Keywords: generation
• Abstract: Recent breakthroughs in large multimodal models (LMMs) have significantly advanced both text-to-image (T2I) generation and image-to-text (I2T) interpretation. However, many generated images still suffer from issues related to perceptual quality and text-image alignment. Given the high cost and inefficiency of manual evaluation, an automatic metric that aligns with human preferences is desirable. To this end, we present EvalMi-50K, a comprehensive dataset and benchmark for evaluating large-multimodal image generation, which features (i) comprehensive tasks, encompassing 2,100 extensive prompts across 20 fine-grained task dimensions, and (ii) large-scale human-preference annotations, including 100K mean-opinion scores (MOSs) and 50K question-answering (QA) pairs annotated on 50,400 images generated from 24 T2I models. Based on EvalMi-50K, we propose LMM4LMM, an LMM-based metric for evaluating large multimodal T2I generation from multiple dimensions including perception, text-image correspondence, and task-specific accuracy. Extensive experimental results show that LMM4LMM achieves state-of-the-art performance on EvalMi-50K, and exhibits strong generalization ability on other AI-generated image evaluation benchmark datasets, manifesting the generality of both the EvalMi-50K dataset and LMM4LMM metric. Both EvalMi-50K and LMM4LMM will be released at this https URL.
• 摘要：在大型多模型（LMM）中，最近的突破具有显着提高文本形象（T2I）的生成和图像到文本（I2T）的解释。但是，许多生成的图像仍然遭受与知觉质量和文本图像对齐相关的问题。鉴于手动评估的高成本和效率低下，希望与人类偏好保持一致的自动指标。为此，我们提供了评估50K，这是一个全面的数据集和基准测试，用于评估大型型图像生成，其中（i）全面任务，包括20个精细的任务维度上的2,100个广泛的提示，以及（ii）大规模的人类预选注释，包括100k均值的人（包括100k），以及50k soss（MOSS），MOSS）（MOSS）（MOSS）（MOSS）（MOSS）（MOSS）（MOSS），以下（MOSS）w（MOSS）w（MOS）。从24个T2i型号产生的50,400张图像。基于Evalmi-50k，我们提出了LMM4LMM，这是一种基于LMM的度量，用于评估来自多个维度的大型多模式T2I生成，包括感知，文本图像对应关系和特定于任务的精度。广泛的实验结果表明，LMM4LMM在Evalmi-50k上实现了最先进的性能，并且在其他AI生成的图像评估基准数据集上具有强大的概括能力，这表明了Evalmi-I-50K数据集和LMM4LMM Metric的一般性。 Evalmi-50K和LMM4LMM均将在此HTTPS URL上发布。

Title: PCA-RAG: Principal Component Analysis for Efficient Retrieval-Augmented Generation

• Authors: Arman Khaledian, Amirreza Ghadiridehkordi, Nariman Khaledian
• Subjects: cs.LG, cs.AI, cs.IR, stat.ML
• Abstract URL: https://arxiv.org/abs/2504.08386
• Pdf URL: https://arxiv.org/pdf/2504.08386
• Copy Paste: [[2504.08386]] PCA-RAG: Principal Component Analysis for Efficient Retrieval-Augmented Generation(https://arxiv.org/abs/2504.08386)
• Keywords: generation
• Abstract: Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm for grounding large language models in external knowledge sources, improving the precision of agents responses. However, high-dimensional language model embeddings, often in the range of hundreds to thousands of dimensions, can present scalability challenges in terms of storage and latency, especially when processing massive financial text corpora. This paper investigates the use of Principal Component Analysis (PCA) to reduce embedding dimensionality, thereby mitigating computational bottlenecks without incurring large accuracy losses. We experiment with a real-world dataset and compare different similarity and distance metrics under both full-dimensional and PCA-compressed embeddings. Our results show that reducing vectors from 3,072 to 110 dimensions provides a sizeable (up to $60\times$) speedup in retrieval operations and a $\sim 28.6\times$ reduction in index size, with only moderate declines in correlation metrics relative to human-annotated similarity scores. These findings demonstrate that PCA-based compression offers a viable balance between retrieval fidelity and resource efficiency, essential for real-time systems such as Zanista AI's \textit{Newswitch} platform. Ultimately, our study underscores the practicality of leveraging classical dimensionality reduction techniques to scale RAG architectures for knowledge-intensive applications in finance and trading, where speed, memory efficiency, and accuracy must jointly be optimized.
• 摘要：检索增强的一代（RAG）已成为一种强大的范式，用于在外部知识源中扎根大语言模型，从而提高了代理响应的精度。但是，高维语言模型嵌入通常在数百到数千个维度范围内，可能会在存储和延迟方面提出可伸缩性挑战，尤其是在处理大量财务文本语料库时。本文研究了使用主成分分析（PCA）来降低嵌入维度，从而减轻计算瓶颈而不会产生大量准确性损失。我们尝试了一个现实世界数据集，并比较了全维和PCA压缩嵌入的不同相似性和距离指标。我们的结果表明，将矢量从3,072降低到110个维度可在检索操作中提供大量（$ 60 \ times $）的速度，$ \ sim 28.6 \ times $减少了指数大小，相对于人类通知相似性相似的相关度量相关的相关度量相对相关衡量标准的相关度量仅下降。这些发现表明，基于PCA的压缩性在检索保真度和资源效率之间提供了可行的平衡，这对于诸如Zanista AI的\ textit {NewSwitch}平台等实时系统至关重要。最终，我们的研究强调了利用经典维度降低技术的实用性，以扩展抹布体系结构，以在金融和交易中进行知识密集型应用，在这种情况下，必须共同优化速度，记忆效率和准确性。

Title: Graph Reduction with Unsupervised Learning in Column Generation: A Routing Application

• Authors: Abdo Abouelrous, Laurens Bliea, Adriana F. Gabor, Yaoxin Wu, Yingqian Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.08401
• Pdf URL: https://arxiv.org/pdf/2504.08401
• Copy Paste: [[2504.08401]] Graph Reduction with Unsupervised Learning in Column Generation: A Routing Application(https://arxiv.org/abs/2504.08401)
• Keywords: generation
• Abstract: Column Generation (CG) is a popular method dedicated to enhancing computational efficiency in large scale Combinatorial Optimization (CO) problems. It reduces the number of decision variables in a problem by solving a pricing problem. For many CO problems, the pricing problem is an Elementary Shortest Path Problem with Resource Constraints (ESPPRC). Large ESPPRC instances are difficult to solve to near-optimality. Consequently, we use a Graph neural Network (GNN) to reduces the size of the ESPPRC such that it becomes computationally tractable with standard solving techniques. Our GNN is trained by Unsupervised Learning and outputs a distribution for the arcs to be retained in the reduced PP. The reduced PP is solved by a local search that finds columns with large reduced costs and speeds up convergence. We apply our method on a set of Capacitated Vehicle Routing Problems with Time Windows and show significant improvements in convergence compared to simple reduction techniques from the literature. For a fixed computational budget, we improve the objective values by over 9\% for larger instances. We also analyze the performance of our CG algorithm and test the generalization of our method to different classes of instances than the training data.
• 摘要：列的生成（CG）是一种流行的方法，致力于提高大规模组合优化（CO）问题的计算效率。它通过解决定价问题来减少问题中的决策变量数量。对于许多CO问题，定价问题是资源约束（ESPPRC）的基本最短路径问题。大型ESPPRC实例很难解决近乎临时的事件。因此，我们使用图形神经网络（GNN）来降低ESPPRC的大小，从而通过标准求解技术可以在计算上进行计算。我们的GNN经过无监督的学习训练，并输出一个分布，以保留在减少的PP中的ARC。减少的PP通过本地搜索来解决，该搜索可以找到大量成本和加快收敛速度​​的列。与文献相比，我们将方法应用于一组电容车辆路由问题，并显示出收敛的显着改善。对于固定的计算预算，我们将目标值提高了9 \％以上。我们还分析了我们的CG算法的性能，并测试与培训数据相比，我们对不同类别实例的方法的概括。

Title: A Knowledge-guided Adversarial Defense for Resisting Malicious Visual Manipulation

• Authors: Dawei Zhou, Suzhi Gang, Decheng Liu, Tongliang Liu, Nannan Wang, Xinbo Gao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.08411
• Pdf URL: https://arxiv.org/pdf/2504.08411
• Copy Paste: [[2504.08411]] A Knowledge-guided Adversarial Defense for Resisting Malicious Visual Manipulation(https://arxiv.org/abs/2504.08411)
• Keywords: quality assessment
• Abstract: Malicious applications of visual manipulation have raised serious threats to the security and reputation of users in many fields. To alleviate these issues, adversarial noise-based defenses have been enthusiastically studied in recent years. However, ``data-only" methods tend to distort fake samples in the low-level feature space rather than the high-level semantic space, leading to limitations in resisting malicious manipulation. Frontier research has shown that integrating knowledge in deep learning can produce reliable and generalizable solutions. Inspired by these, we propose a knowledge-guided adversarial defense (KGAD) to actively force malicious manipulation models to output semantically confusing samples. Specifically, in the process of generating adversarial noise, we focus on constructing significant semantic confusions at the domain-specific knowledge level, and exploit a metric closely related to visual perception to replace the general pixel-wise metrics. The generated adversarial noise can actively interfere with the malicious manipulation model by triggering knowledge-guided and perception-related disruptions in the fake samples. To validate the effectiveness of the proposed method, we conduct qualitative and quantitative experiments on human perception and visual quality assessment. The results on two different tasks both show that our defense provides better protection compared to state-of-the-art methods and achieves great generalizability.
• 摘要：视觉操纵的恶意应用已对许多领域的用户的安全和声誉造成了严重威胁。为了减轻这些问题，近年来对基于对抗噪声的防御进行了热情研究。 However, ``data-only" methods tend to distort fake samples in the low-level feature space rather than the high-level semantic space, leading to limitations in resisting malicious manipulation. Frontier research has shown that integrating knowledge in deep learning can produce reliable and generalizable solutions. Inspired by these, we propose a knowledge-guided adversarial defense (KGAD) to actively force malicious manipulation models to output semantically confusing samples.具体来说，在生成对抗性噪声的过程中，我们着重于在特定的知识水平上构建重要的语义混乱，并利用与视觉感知密切相关的度量提出的方法，我们对人类感知和视觉质量评估进行定性和定量实验。两项不同任务的结果都表明，与最先进的方法相比，我们的防御能力提供了更好的保护，并且实现了极大的概括性。

Title: GeoTexBuild: 3D Building Model Generation from Map Footprints

• Authors: Ruizhe Wang, Junyan Yang, Qiao Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08419
• Pdf URL: https://arxiv.org/pdf/2504.08419
• Copy Paste: [[2504.08419]] GeoTexBuild: 3D Building Model Generation from Map Footprints(https://arxiv.org/abs/2504.08419)
• Keywords: generation, generative
• Abstract: We introduce GeoTexBuild, a modular generative framework for creating 3D building models from map footprints. The proposed framework employs a three-stage process comprising height map generation, geometry reconstruction, and appearance stylization, culminating in building models with intricate geometry and appearance attributes. By integrating customized ControlNet and Text2Mesh models, we explore effective methods for controlling both geometric and visual attributes during the generation process. By this, we eliminate the problem of structural variations behind a single facade photo of the existing 3D generation techniques. Experimental results at each stage validate the capability of GeoTexBuild to generate detailed and accurate building models from footprints derived from site planning or map designs. Our framework significantly reduces manual labor in modeling buildings and can offer inspiration for designers.
• 摘要：我们介绍了GeoteXbuild，这是一个模块化生成框架，用于从MAP足迹创建3D建筑模型。所提出的框架采用了一个三阶段的过程，其中包括高度图的产生，几何重建和外观风格，并在具有复杂的几何形状和外观属性的建筑模型中达到顶点。通过集成自定义的控制网和Text2Mesh模型，我们探索了在生成过程中控制几何和视觉属性的有效方法。通过这种情况，我们消除了现有3D代技术的单个立面照片背后的结构变化问题。每个阶段的实验结果验证了地理界的能力，可以从站点规划或地图设计得出的足迹中生成详细而准确的建筑模型。我们的框架大大减少了建模建模的体力劳动，并可以为设计师提供灵感。

Title: Customizing Spider Silk: Generative Models with Mechanical Property Conditioning for Protein Engineering

• Authors: Neeru Dubey, Elin Karlsson, Miguel Angel Redondo, Johan Reimegård, Anna Rising, Hedvig Kjellström
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.08437
• Pdf URL: https://arxiv.org/pdf/2504.08437
• Copy Paste: [[2504.08437]] Customizing Spider Silk: Generative Models with Mechanical Property Conditioning for Protein Engineering(https://arxiv.org/abs/2504.08437)
• Keywords: generation, generative
• Abstract: The remarkable mechanical properties of spider silk, including its tensile strength and extensibility, are primarily governed by the repetitive regions of the proteins that constitute the fiber, the major ampullate spidroins (MaSps). However, establishing correlations between mechanical characteristics and repeat sequences is challenging due to the intricate sequence-structure-function relationships of MaSps and the limited availability of annotated datasets. In this study, we present a novel computational framework for designing MaSp repeat sequences with customizable mechanical properties. To achieve this, we developed a lightweight GPT-based generative model by distilling the pre-trained ProtGPT2 protein language model. The distilled model was subjected to multilevel fine-tuning using curated subsets of the Spider Silkome dataset. Specifically, we adapt the model for MaSp repeat generation using 6,000 MaSp repeat sequences and further refine it with 572 repeats associated with experimentally determined fiber-level mechanical properties. Our model generates biologically plausible MaSp repeat regions tailored to specific mechanical properties while also predicting those properties for given sequences. Validation includes sequence-level analysis, assessing physicochemical attributes and expected distribution of key motifs as well as secondary structure compositions. A correlation study using BLAST on the Spider Silkome dataset and a test set of MaSp repeats with known mechanical properties further confirmed the predictive accuracy of the model. This framework advances the rational design of spider silk-inspired biomaterials, offering a versatile tool for engineering protein sequences with tailored mechanical attributes.
• 摘要：蜘蛛丝的显着机械性能，包括其拉伸强度和可扩展性，主要受构成纤维的蛋白质的重复区域，即纤维，主要的截肢蜘蛛蛋白（MASPS）。但是，由于MASP的复杂序列结构函数关系和带注释的数据集的可用性有限，因此建立机械特征与重复序列之间的相关性是具有挑战性的。在这项研究中，我们提出了一个新型的计算框架，用于设计具有可自定义机械性能的MASP重复序列。为了实现这一目标，我们通过提炼预先训练的Protgpt2蛋白质语言模型来开发一种基于GPT的生成模型。使用Spider Spidersome数据集的策划子集对蒸馏模型进行多级微调。具体而言，我们使用6,000 MASP重复序列适应了MASP重复生成的模型，并通过与实验确定的纤维级机械性能相关的572个重复序列进一步完善了它。我们的模型生成了针对特定机械性能的生物学上合理的MASP重复区域，同时还可以预测给定序列的这些特性。验证包括序列级分析，评估关键基序的物理化学属性和预期分布以及二级结构组成。使用BLAST在Spider Spider ssomome数据集和具有已知机械性能的MASP重复测试集上的相关研究进一步证实了该模型的预测精度。该框架推进了蜘蛛丝绸启发的生物材料的合理设计，为具有量身定制的机械属性的工程蛋白质序列提供了一种多功能工具。

Title: Muon-Accelerated Attention Distillation for Real-Time Edge Synthesis via Optimized Latent Diffusion

• Authors: Weiye Chen, Qingen Zhu, Qian Long
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08451
• Pdf URL: https://arxiv.org/pdf/2504.08451
• Copy Paste: [[2504.08451]] Muon-Accelerated Attention Distillation for Real-Time Edge Synthesis via Optimized Latent Diffusion(https://arxiv.org/abs/2504.08451)
• Keywords: generation
• Abstract: Recent advances in visual synthesis have leveraged diffusion models and attention mechanisms to achieve high-fidelity artistic style transfer and photorealistic text-to-image generation. However, real-time deployment on edge devices remains challenging due to computational and memory constraints. We propose Muon-AD, a co-designed framework that integrates the Muon optimizer with attention distillation for real-time edge synthesis. By eliminating gradient conflicts through orthogonal parameter updates and dynamic pruning, Muon-AD achieves 3.2 times faster convergence compared to Stable Diffusion-TensorRT, while maintaining synthesis quality (15% lower FID, 4% higher SSIM). Our framework reduces peak memory to 7GB on Jetson Orin and enables 24FPS real-time generation through mixed-precision quantization and curriculum learning. Extensive experiments on COCO-Stuff and ImageNet-Texture demonstrate Muon-AD's Pareto-optimal efficiency-quality trade-offs. Here, we show a 65% reduction in communication overhead during distributed training and real-time 10s/image generation on edge GPUs. These advancements pave the way for democratizing high-quality visual synthesis in resource-constrained environments.
• 摘要：视觉合成的最新进展利用了扩散模型和注意力机制，以实现高保真的艺术风格转移和逼真的逼真的文本对图像生成。但是，由于计算和内存限制，边缘设备上的实时部署仍然具有挑战性。我们提出了Muon-AD，这是一个共同设计的框架，将MUON优化器与注意力蒸馏进行实时边缘合成。通过通过正交参数更新和动态修剪消除梯度冲突，MUON-AD与稳定的扩散式 - 托管相比，收敛的速度更快3.2倍，同时保持合成质量（低15％下FID，SSSIM高4％）。我们的框架将峰值存储器降低到Jetson Orin上的7GB，并通过混合精确量化和课程学习启用24FPS实时生成。关于可可粘合和成像纹理的广泛实验表明，Muon-AD的帕累托 - 最佳效率质量折衷。在这里，我们在分布式培训和Edge GPU上的实时培训和实时10S/图像生成期间的通信开销降低了65％。这些进步为在资源受限环境中民主化高质量的视觉综合铺平了道路。

Title: Cut-and-Splat: Leveraging Gaussian Splatting for Synthetic Data Generation

• Authors: Bram Vanherle, Brent Zoomers, Jeroen Put, Frank Van Reeth, Nick Michiels
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08473
• Pdf URL: https://arxiv.org/pdf/2504.08473
• Copy Paste: [[2504.08473]] Cut-and-Splat: Leveraging Gaussian Splatting for Synthetic Data Generation(https://arxiv.org/abs/2504.08473)
• Keywords: generation
• Abstract: Generating synthetic images is a useful method for cheaply obtaining labeled data for training computer vision models. However, obtaining accurate 3D models of relevant objects is necessary, and the resulting images often have a gap in realism due to challenges in simulating lighting effects and camera artifacts. We propose using the novel view synthesis method called Gaussian Splatting to address these challenges. We have developed a synthetic data pipeline for generating high-quality context-aware instance segmentation training data for specific objects. This process is fully automated, requiring only a video of the target object. We train a Gaussian Splatting model of the target object and automatically extract the object from the video. Leveraging Gaussian Splatting, we then render the object on a random background image, and monocular depth estimation is employed to place the object in a believable pose. We introduce a novel dataset to validate our approach and show superior performance over other data generation approaches, such as Cut-and-Paste and Diffusion model-based generation.
• 摘要：生成合成图像是廉价获取用于训练计算机视觉模型的标记数据的有用方法。但是，必须获得相关对象的准确的3D模型，并且由于模拟照明效果和相机伪像的挑战，所得图像通常在现实主义方面存在差距。我们建议使用称为高斯脱落的新型视图合成方法来应对这些挑战。我们已经开发了一个合成数据管道，用于为特定对象生成高质量的上下文感知实例分割培训数据。此过程是完全自动化的，仅需要目标对象的视频。我们训练目标对象的高斯脱衣模型，并自动从视频中提取对象。利用高斯裂口，我们在随机背景图像上渲染对象，并采用单眼深度估计将对象放在可信的姿势中。我们介绍了一个新颖的数据集，以验证我们的方法并显示出优于其他数据生成方法，例如剪切和基于扩散模型的生成。

Title: Discriminator-Free Direct Preference Optimization for Video Diffusion

• Authors: Haoran Cheng, Qide Dong, Liang Peng, Zhizhou Sha, Weiguo Feng, Jinghui Xie, Zhao Song, Shilei Wen, Xiaofei He, Boxi Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08542
• Pdf URL: https://arxiv.org/pdf/2504.08542
• Copy Paste: [[2504.08542]] Discriminator-Free Direct Preference Optimization for Video Diffusion(https://arxiv.org/abs/2504.08542)
• Keywords: generation
• Abstract: Direct Preference Optimization (DPO), which aligns models with human preferences through win/lose data pairs, has achieved remarkable success in language and image generation. However, applying DPO to video diffusion models faces critical challenges: (1) Data inefficiency. Generating thousands of videos per DPO iteration incurs prohibitive costs; (2) Evaluation uncertainty. Human annotations suffer from subjective bias, and automated discriminators fail to detect subtle temporal artifacts like flickering or motion incoherence. To address these, we propose a discriminator-free video DPO framework that: (1) Uses original real videos as win cases and their edited versions (e.g., reversed, shuffled, or noise-corrupted clips) as lose cases; (2) Trains video diffusion models to distinguish and avoid artifacts introduced by editing. This approach eliminates the need for costly synthetic video comparisons, provides unambiguous quality signals, and enables unlimited training data expansion through simple editing operations. We theoretically prove the framework's effectiveness even when real videos and model-generated videos follow different distributions. Experiments on CogVideoX demonstrate the efficiency of the proposed method.
• 摘要：直接偏好优化（DPO）通过双赢数据对使模型与人类偏好保持一致，在语言和图像生成方面取得了巨大的成功。但是，将DPO应用于视频扩散模型面临着关键的挑战：（1）数据效率低下。每个DPO迭代产生数千个视频会产生过高的成本； （2）评估不确定性。人类注释遭受主观偏见的影响，自动化歧视者无法检测到闪烁或运动不连贯等微妙的时间伪像。为了解决这些问题，我们提出了一个不含歧视的视频DPO框架：（1）将原始的真实视频用作胜利案例及其编辑版本（例如，反向，改组或噪音触发的剪辑）作为丢失案例； （2）训练视频扩散模型，以区分和避免通过编辑引入的工件。这种方法消除了对昂贵的合成视频比较的需求，提供明确的质量信号，并通过简单的编辑操作启用无限培训数据扩展。从理论上讲，即使真实的视频和模型生成的视频遵循不同的分布，我们也可以证明该框架的有效性。 Cogvideox的实验证明了该方法的效率。

Title: Slicing the Gaussian Mixture Wasserstein Distance

• Authors: Moritz Piening, Robert Beinert
• Subjects: cs.LG, math.NA, stat.ML
• Abstract URL: https://arxiv.org/abs/2504.08544
• Pdf URL: https://arxiv.org/pdf/2504.08544
• Copy Paste: [[2504.08544]] Slicing the Gaussian Mixture Wasserstein Distance(https://arxiv.org/abs/2504.08544)
• Keywords: generative
• Abstract: Gaussian mixture models (GMMs) are widely used in machine learning for tasks such as clustering, classification, image reconstruction, and generative modeling. A key challenge in working with GMMs is defining a computationally efficient and geometrically meaningful metric. The mixture Wasserstein (MW) distance adapts the Wasserstein metric to GMMs and has been applied in various domains, including domain adaptation, dataset comparison, and reinforcement learning. However, its high computational cost -- arising from repeated Wasserstein distance computations involving matrix square root estimations and an expensive linear program -- limits its scalability to high-dimensional and large-scale problems. To address this, we propose multiple novel slicing-based approximations to the MW distance that significantly reduce computational complexity while preserving key optimal transport properties. From a theoretical viewpoint, we establish several weak and strong equivalences between the introduced metrics, and show the relations to the original MW distance and the well-established sliced Wasserstein distance. Furthermore, we validate the effectiveness of our approach through numerical experiments, demonstrating computational efficiency and applications in clustering, perceptual image comparison, and GMM minimization
• 摘要：高斯混合模型（GMM）被广泛用于机器学习中，用于诸如聚类，分类，图像重建和生成建模等任务。与GMM一起工作的关键挑战是定义计算上有效且几何有意义的度量。混合物WASSERSTEIN（MW）距离将Wasserstein指标适应GMM，并已应用于各种领域，包括域适应性，数据集比较和增强学习。但是，它的高计算成本是由涉及矩阵平方根估计和昂贵的线性程序的重复的Wasserstein距离计算引起的 - 将其可扩展性限制在高维和大规模问题上。为了解决这个问题，我们提出了多个基于切片的新型近似值，以显着降低计算复杂性，同时保留关键的最佳运输特性。从理论上的角度来看，我们在引入的指标之间建立了几个弱和强的等价，并显示了与原始MW距离和已公认的切片瓦斯坦距离的关系。此外，我们通过数值实验来验证方法的有效性，证明计算效率和在聚类，感知图像比较和GMM最小化中的应用

Title: Banana Ripeness Level Classification using a Simple CNN Model Trained with Real and Synthetic Datasets

• Authors: Luis Chuquimarca, Boris Vintimilla, Sergio Velastin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08568
• Pdf URL: https://arxiv.org/pdf/2504.08568
• Copy Paste: [[2504.08568]] Banana Ripeness Level Classification using a Simple CNN Model Trained with Real and Synthetic Datasets(https://arxiv.org/abs/2504.08568)
• Keywords: generation
• Abstract: The level of ripeness is essential in determining the quality of bananas. To correctly estimate banana maturity, the metrics of international marketing standards need to be considered. However, the process of assessing the maturity of bananas at an industrial level is still carried out using manual methods. The use of CNN models is an attractive tool to solve the problem, but there is a limitation regarding the availability of sufficient data to train these models reliably. On the other hand, in the state-of-the-art, existing CNN models and the available data have reported that the accuracy results are acceptable in identifying banana maturity. For this reason, this work presents the generation of a robust dataset that combines real and synthetic data for different levels of banana ripeness. In addition, it proposes a simple CNN architecture, which is trained with synthetic data and using the transfer learning technique, the model is improved to classify real data, managing to determine the level of maturity of the banana. The proposed CNN model is evaluated with several architectures, then hyper-parameter configurations are varied, and optimizers are used. The results show that the proposed CNN model reaches a high accuracy of 0.917 and a fast execution time.
• 摘要：成熟水平对于确定香蕉的质量至关重要。要正确估计香蕉的成熟度，需要考虑国际营销标准的指标。但是，仍使用手动方法进行评估在工业层面上评估香蕉成熟度的过程。 CNN模型的使用是解决问题的有吸引力的工具，但是关于足够数据可靠训练这些模型的可用性有一个限制。另一方面，在最新的现有CNN模型和可用数据中，已经报告说，在识别香蕉成熟度时，准确性结果是可以接受的。因此，这项工作介绍了一个强大的数据集的生成，该数据集结合了不同水平的香蕉成熟度的真实和合成数据。此外，它提出了一种简单的CNN体​​系结构，该体系结构是通过合成数据训练并使用转移学习技术的，改进了模型以对真实数据进行分类，并设法确定香蕉的成熟度。使用几个架构进行评估所提出的CNN模型，然后使用超参数配置，并使用优化器。结果表明，提出的CNN模型的高精度为0.917，并且执行时间很快。

Title: ZipIR: Latent Pyramid Diffusion Transformer for High-Resolution Image Restoration

• Authors: Yongsheng Yu, Haitian Zheng, Zhifei Zhang, Jianming Zhang, Yuqian Zhou, Connelly Barnes, Yuchen Liu, Wei Xiong, Zhe Lin, Jiebo Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08591
• Pdf URL: https://arxiv.org/pdf/2504.08591
• Copy Paste: [[2504.08591]] ZipIR: Latent Pyramid Diffusion Transformer for High-Resolution Image Restoration(https://arxiv.org/abs/2504.08591)
• Keywords: restoration, generative
• Abstract: Recent progress in generative models has significantly improved image restoration capabilities, particularly through powerful diffusion models that offer remarkable recovery of semantic details and local fidelity. However, deploying these models at ultra-high resolutions faces a critical trade-off between quality and efficiency due to the computational demands of long-range attention mechanisms. To address this, we introduce ZipIR, a novel framework that enhances efficiency, scalability, and long-range modeling for high-res image restoration. ZipIR employs a highly compressed latent representation that compresses image 32x, effectively reducing the number of spatial tokens, and enabling the use of high-capacity models like the Diffusion Transformer (DiT). Toward this goal, we propose a Latent Pyramid VAE (LP-VAE) design that structures the latent space into sub-bands to ease diffusion training. Trained on full images up to 2K resolution, ZipIR surpasses existing diffusion-based methods, offering unmatched speed and quality in restoring high-resolution images from severely degraded inputs.
• 摘要：生成模型的最新进展显着提高了图像恢复能力，尤其是通过强大的扩散模型，这些模型可以显着恢复语义细节和本地忠诚度。但是，由于远程注意机制的计算需求，在超高的决议中部署这些模型在质量和效率之间的重大权衡。为了解决这个问题，我们介绍了Zipir，这是一个新颖的框架，可提高效率，可扩展性和高分辨率图像恢复的远程建模。 Zipir采用高度压缩的潜在表示，可压缩图像32X，有效地减少了空间令牌的数量，并可以使用高容量模型（例如扩散变压器（DIT））。为了实现这一目标，我们提出了一种潜在的金字塔VAE（LP-VAE）设计，该设计将潜在空间构成子带中以减轻扩散训练。齐皮尔（Zipir）在最高2K分辨率的完整图像上受过培训，超过了基于扩散的方法，在恢复严重退化输入的高分辨率图像时提供了无与伦比的速度和质量。

Title: Efficient Mixture of Geographical Species for On Device Wildlife Monitoring

• Authors: Emmanuel Azuh Mensah, Joban Mand, Yueheng Ou, Min Jang, Kurtis Heimerl
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08620
• Pdf URL: https://arxiv.org/pdf/2504.08620
• Copy Paste: [[2504.08620]] Efficient Mixture of Geographical Species for On Device Wildlife Monitoring(https://arxiv.org/abs/2504.08620)
• Keywords: generation
• Abstract: Efficient on-device models have become attractive for near-sensor insight generation, of particular interest to the ecological conservation community. For this reason, deep learning researchers are proposing more approaches to develop lower compute models. However, since vision transformers are very new to the edge use case, there are still unexplored approaches, most notably conditional execution of subnetworks based on input data. In this work, we explore the training of a single species detector which uses conditional computation to bias structured sub networks in a geographically-aware manner. We propose a method for pruning the expert model per location and demonstrate conditional computation performance on two geographically distributed datasets: iNaturalist and iWildcam.
• 摘要：有效的驻留式模型已经对近传感器的见解产生具有吸引力，这对于生态保护界特别感兴趣。因此，深度学习研究人员正在提出更多开发较低计算模型的方法。但是，由于Vision Transformers在边缘用例中是非常新的，因此仍有未开发的方法，最著名的是基于输入数据的子网执行。在这项工作中，我们探索了单个物种检测器的训练，该训练使用条件计算以地理意识的方式偏向结构的子网络。我们提出了一种在每个位置修剪专家模型的方法，并在两个地理分布的数据集上展示了条件计算性能：inaturalist和iwildcam。

Title: Latent Diffusion Autoencoders: Toward Efficient and Meaningful Unsupervised Representation Learning in Medical Imaging

• Authors: Gabriele Lozupone, Alessandro Bria, Francesco Fontanella, Frederick J.A. Meijer, Claudio De Stefano, Henkjan Huisman
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08635
• Pdf URL: https://arxiv.org/pdf/2504.08635
• Copy Paste: [[2504.08635]] Latent Diffusion Autoencoders: Toward Efficient and Meaningful Unsupervised Representation Learning in Medical Imaging(https://arxiv.org/abs/2504.08635)
• Keywords: generation
• Abstract: This study presents Latent Diffusion Autoencoder (LDAE), a novel encoder-decoder diffusion-based framework for efficient and meaningful unsupervised learning in medical imaging, focusing on Alzheimer disease (AD) using brain MR from the ADNI database as a case study. Unlike conventional diffusion autoencoders operating in image space, LDAE applies the diffusion process in a compressed latent representation, improving computational efficiency and making 3D medical imaging representation learning tractable. To validate the proposed approach, we explore two key hypotheses: (i) LDAE effectively captures meaningful semantic representations on 3D brain MR associated with AD and ageing, and (ii) LDAE achieves high-quality image generation and reconstruction while being computationally efficient. Experimental results support both hypotheses: (i) linear-probe evaluations demonstrate promising diagnostic performance for AD (ROC-AUC: 90%, ACC: 84%) and age prediction (MAE: 4.1 years, RMSE: 5.2 years); (ii) the learned semantic representations enable attribute manipulation, yielding anatomically plausible modifications; (iii) semantic interpolation experiments show strong reconstruction of missing scans, with SSIM of 0.969 (MSE: 0.0019) for a 6-month gap. Even for longer gaps (24 months), the model maintains robust performance (SSIM > 0.93, MSE < 0.004), indicating an ability to capture temporal progression trends; (iv) compared to conventional diffusion autoencoders, LDAE significantly increases inference throughput (20x faster) while also enhancing reconstruction quality. These findings position LDAE as a promising framework for scalable medical imaging applications, with the potential to serve as a foundation model for medical image analysis. Code available at this https URL
• 摘要：这项研究介绍了潜在扩散自动编码器（LDAE），这是一种新型的编码器扩散框架，用于医学成像中有效且有意义的无监督学习，重点是使用ADNI数据库中的ADNI数据库中的大脑MR作为案例研究。与在图像空间中运行的常规扩散自动编码器不同，LDAE在压缩潜在表示中应用扩散过程，提高了计算效率并使3D医学成像表示学习可触及。为了验证所提出的方法，我们探讨了两个关键假设：（i）LDAE有效地捕获了与AD和衰老相关的3D脑MR上有意义的语义表示，并且（ii）LDAE实现了高质量的图像产生和重建，同时计算有效。实验结果支持这两个假设：（i）线性探针评估表明AD（ROC-AUC：90％，ACC：84％）和年龄预测（MAE：4.1岁，RMSE：5.2岁）有希望的诊断性能（90％，ACC：84％）； （ii）学习的语义表示可以使属性操纵，从而产生解剖学上合理的修改； （iii）语义插值实验显示缺失扫描的强大重建，SSIM为6个月的间隙为0.969（MSE：0.0019）。即使在较长的间隙（24个月）中，该模型仍保持稳健的性能（SSIM> 0.93，MSE <0.004），表明能够捕获时间进展趋势的能力； （iv）与常规扩散自动编码器相比，LDAE显着增加了推理吞吐量（更快20倍），同时也提高了重建质量。这些发现将LDAE定位为可扩展医学成像应用的有前途的框架，并有可能作为医学图像分析的基础模型。此https URL可用代码

Title: Training-free Guidance in Text-to-Video Generation via Multimodal Planning and Structured Noise Initialization

• Authors: Jialu Li, Shoubin Yu, Han Lin, Jaemin Cho, Jaehong Yoon, Mohit Bansal
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2504.08641
• Pdf URL: https://arxiv.org/pdf/2504.08641
• Copy Paste: [[2504.08641]] Training-free Guidance in Text-to-Video Generation via Multimodal Planning and Structured Noise Initialization(https://arxiv.org/abs/2504.08641)
• Keywords: generation
• Abstract: Recent advancements in text-to-video (T2V) diffusion models have significantly enhanced the visual quality of the generated videos. However, even recent T2V models find it challenging to follow text descriptions accurately, especially when the prompt requires accurate control of spatial layouts or object trajectories. A recent line of research uses layout guidance for T2V models that require fine-tuning or iterative manipulation of the attention map during inference time. This significantly increases the memory requirement, making it difficult to adopt a large T2V model as a backbone. To address this, we introduce Video-MSG, a training-free Guidance method for T2V generation based on Multimodal planning and Structured noise initialization. Video-MSG consists of three steps, where in the first two steps, Video-MSG creates Video Sketch, a fine-grained spatio-temporal plan for the final video, specifying background, foreground, and object trajectories, in the form of draft video frames. In the last step, Video-MSG guides a downstream T2V diffusion model with Video Sketch through noise inversion and denoising. Notably, Video-MSG does not need fine-tuning or attention manipulation with additional memory during inference time, making it easier to adopt large T2V models. Video-MSG demonstrates its effectiveness in enhancing text alignment with multiple T2V backbones (VideoCrafter2 and CogVideoX-5B) on popular T2V generation benchmarks (T2VCompBench and VBench). We provide comprehensive ablation studies about noise inversion ratio, different background generators, background object detection, and foreground object segmentation.
• 摘要：文本到视频（T2V）扩散模型的最新进展显着提高了生成的视频的视觉质量。但是，即使最近的T2V模型也发现准确遵循文本描述是一项挑战，尤其是当提示需要准确控制空间布局或对象轨迹时。最近的研究线对T2V模型使用布局指南，这些模型需要在推理期间对注意图进行微调或迭代操纵。这大大增加了内存需求，因此很难采用大型T2V模型作为骨干。为了解决这个问题，我们介绍了Video-MSG，这是一种基于多模式计划和结构化噪声初始化的T2V生成的无培训指导方法。 Video-MSG由三个步骤组成，在前两个步骤中，Video-MSG创建了视频草图，这是最终视频的细粒度时空计划，以视频框架的形式指定背景，前景和对象轨迹。在最后一步中，视频MSG通过噪声倒转和降解来指导下游T2V扩散模型，并通过视频草图进行了素描。值得注意的是，在推理时间内，Video-MSG不需要微调或注意力操纵，因此更容易采用大型T2V模型。 Video-MSG展示了其在流行T2V生成基准（T2VCompbench和VBench）上使用多个T2V主链（VideoCrafter2和Cogvideox-5b）增强文本对齐的有效性。我们提供有关噪声反转比率，不同背景发生器，背景对象检测和前景对象分割的全面消融研究。

Title: Seaweed-7B: Cost-Effective Training of Video Generation Foundation Model

• Authors: Team Seawead, Ceyuan Yang, Zhijie Lin, Yang Zhao, Shanchuan Lin, Zhibei Ma, Haoyuan Guo, Hao Chen, Lu Qi, Sen Wang, Feng Cheng, Feilong Zuo Xuejiao Zeng, Ziyan Yang, Fangyuan Kong, Zhiwu Qing, Fei Xiao, Meng Wei, Tuyen Hoang, Siyu Zhang, Peihao Zhu, Qi Zhao, Jiangqiao Yan, Liangke Gui, Sheng Bi, Jiashi Li, Yuxi Ren, Rui Wang, Huixia Li, Xuefeng Xiao, Shu Liu, Feng Ling, Heng Zhang, Houmin Wei, Huafeng Kuang, Jerry Duncan, Junda Zhang, Junru Zheng, Li Sun, Manlin Zhang, Renfei Sun, Xiaobin Zhuang, Xiaojie Li, Xin Xia, Xuyan Chi, Yanghua Peng, Yuping Wang, Yuxuan Wang, Zhongkai Zhao, Zhuo Chen, Zuquan Song, Zhenheng Yang, Jiashi Feng, Jianchao Yang, Lu Jiang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.08685
• Pdf URL: https://arxiv.org/pdf/2504.08685
• Copy Paste: [[2504.08685]] Seaweed-7B: Cost-Effective Training of Video Generation Foundation Model(https://arxiv.org/abs/2504.08685)
• Keywords: generation
• Abstract: This technical report presents a cost-efficient strategy for training a video generation foundation model. We present a mid-sized research model with approximately 7 billion parameters (7B) called Seaweed-7B trained from scratch using 665,000 H100 GPU hours. Despite being trained with moderate computational resources, Seaweed-7B demonstrates highly competitive performance compared to contemporary video generation models of much larger size. Design choices are especially crucial in a resource-constrained setting. This technical report highlights the key design decisions that enhance the performance of the medium-sized diffusion model. Empirically, we make two observations: (1) Seaweed-7B achieves performance comparable to, or even surpasses, larger models trained on substantially greater GPU resources, and (2) our model, which exhibits strong generalization ability, can be effectively adapted across a wide range of downstream applications either by lightweight fine-tuning or continue training. See the project page at this https URL
• 摘要：该技术报告提出了一种培训视频基础模型的经济高效策略。我们提出了一个中型的研究模型，使用665,000 H100 GPU小时，其大约70亿参数（7B）称为Seaweed-7B。尽管接受了适度的计算资源培训，但与更大尺寸的当代视频生成模型相比，Seaweed-7B表现出竞争激烈的性能。在资源约束设置中，设计选择尤为重要。该技术报告强调了增强中型扩散模型性能的关键设计决策。从经验上讲，我们进行了两个观察：（1）Seaweed-7b实现的性能可与较大的GPU资源相当甚至超过较大的模型，并且（2）我们的模型（表现出强大的概括能力）可以通过轻度范围的范围通过轻度Weiblweight Fine-fight-fight-fightning或继续培训来有效地调整。在此HTTPS URL上查看项目页面

Title: Hypergraph Vision Transformers: Images are More than Nodes, More than Edges

• Authors: Joshua Fixelle
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08710
• Pdf URL: https://arxiv.org/pdf/2504.08710
• Copy Paste: [[2504.08710]] Hypergraph Vision Transformers: Images are More than Nodes, More than Edges(https://arxiv.org/abs/2504.08710)
• Keywords: generation
• Abstract: Recent advancements in computer vision have highlighted the scalability of Vision Transformers (ViTs) across various tasks, yet challenges remain in balancing adaptability, computational efficiency, and the ability to model higher-order relationships. Vision Graph Neural Networks (ViGs) offer an alternative by leveraging graph-based methodologies but are hindered by the computational bottlenecks of clustering algorithms used for edge generation. To address these issues, we propose the Hypergraph Vision Transformer (HgVT), which incorporates a hierarchical bipartite hypergraph structure into the vision transformer framework to capture higher-order semantic relationships while maintaining computational efficiency. HgVT leverages population and diversity regularization for dynamic hypergraph construction without clustering, and expert edge pooling to enhance semantic extraction and facilitate graph-based image retrieval. Empirical results demonstrate that HgVT achieves strong performance on image classification and retrieval, positioning it as an efficient framework for semantic-based vision tasks.
• 摘要：计算机视觉的最新进展突出了各种任务中视觉变压器（VIT）的可扩展性，但是在平衡适应性，计算效率和建模高阶关系的能力方面仍然存在挑战。视觉图神经网络（VIGS）通过利用基于图的方法来提供替代方案，但受到用于边缘生成的聚类算法的计算瓶颈的阻碍。为了解决这些问题，我们提出了HyperGraph Vision Transformer（HGVT），该变压器（HGVT）将分层的两分性超图结构纳入视觉变压器框架中，以捕获高阶语义关系，同时保持计算效率。 HGVT利用人群和多样性正则化用于动态超图形结构而无需聚类，并且专家边缘汇总以增强语义提取并促进基于图形的图像检索。经验结果表明，HGVT在图像分类和检索上实现了强劲的性能，将其定位为基于语义的视觉任务的有效框架。

Title: Generating Fine Details of Entity Interactions

• Authors: Xinyi Gu, Jiayuan Mao
• Subjects: cs.CV, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.08714
• Pdf URL: https://arxiv.org/pdf/2504.08714
• Copy Paste: [[2504.08714]] Generating Fine Details of Entity Interactions(https://arxiv.org/abs/2504.08714)
• Keywords: generation
• Abstract: Images not only depict objects but also encapsulate rich interactions between them. However, generating faithful and high-fidelity images involving multiple entities interacting with each other, is a long-standing challenge. While pre-trained text-to-image models are trained on large-scale datasets to follow diverse text instructions, they struggle to generate accurate interactions, likely due to the scarcity of training data for uncommon object interactions. This paper introduces InterActing, an interaction-focused dataset with 1000 fine-grained prompts covering three key scenarios: (1) functional and action-based interactions, (2) compositional spatial relationships, and (3) multi-subject interactions. To address interaction generation challenges, we propose a decomposition-augmented refinement procedure. Our approach, DetailScribe, built on Stable Diffusion 3.5, leverages LLMs to decompose interactions into finer-grained concepts, uses a VLM to critique generated images, and applies targeted interventions within the diffusion process in refinement. Automatic and human evaluations show significantly improved image quality, demonstrating the potential of enhanced inference strategies. Our dataset and code are available at this https URL to facilitate future exploration of interaction-rich image generation.
• 摘要：图像不仅描绘了对象，而且还封装了它们之间的丰富相互作用。但是，产生涉及多个实体相互作用的忠实和高保真图像是一个长期的挑战。虽然在大规模数据集中对预训练的文本到图像模型进行了培训以遵循各种文本说明，但它们很难产生准确的互动，这可能是由于缺乏罕见的训练数据来实现罕见的对象交互。本文介绍了互动，这是一种以1000个细粒度提示为重点的数据集，涵盖了三个关键方案：（1）基于功能和基于动作的相互作用，（2）组成空间关系，以及（3）多主体交互。为了应对互动产生的挑战，我们提出了一个分解启动的完善程序。我们的方法，细节，建立在稳定的扩散3.5上，利用LLMS将相互作用分解为细粒度的概念，使用VLM来批评生成的图像，并在扩散过程中应用有针对性的干预措施。自动和人类评估显示出显着改善的图像质量，表明了增强推理策略的潜力。我们的数据集和代码可在此HTTPS URL上找到，以促进对互动富图像生成的未来探索。

Title: GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generation

• Authors: Tianwei Xiong, Jun Hao Liew, Zilong Huang, Jiashi Feng, Xihui Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.08736
• Pdf URL: https://arxiv.org/pdf/2504.08736
• Copy Paste: [[2504.08736]] GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generation(https://arxiv.org/abs/2504.08736)
• Keywords: generation
• Abstract: In autoregressive (AR) image generation, visual tokenizers compress images into compact discrete latent tokens, enabling efficient training of downstream autoregressive models for visual generation via next-token prediction. While scaling visual tokenizers improves image reconstruction quality, it often degrades downstream generation quality -- a challenge not adequately addressed in existing literature. To address this, we introduce GigaTok, the first approach to simultaneously improve image reconstruction, generation, and representation learning when scaling visual tokenizers. We identify the growing complexity of latent space as the key factor behind the reconstruction vs. generation dilemma. To mitigate this, we propose semantic regularization, which aligns tokenizer features with semantically consistent features from a pre-trained visual encoder. This constraint prevents excessive latent space complexity during scaling, yielding consistent improvements in both reconstruction and downstream autoregressive generation. Building on semantic regularization, we explore three key practices for scaling tokenizers:(1) using 1D tokenizers for better scalability, (2) prioritizing decoder scaling when expanding both encoder and decoder, and (3) employing entropy loss to stabilize training for billion-scale tokenizers. By scaling to $\bf{3 \space billion}$ parameters, GigaTok achieves state-of-the-art performance in reconstruction, downstream AR generation, and downstream AR representation quality.
• 摘要：在自动回归（AR）图像产生中，视觉引导者将图像压缩为紧凑的离散潜在代币，从而有效地训练下游自回归模型通过下一步的预测进行视觉生成。在扩展视觉引物器可以提高图像重建质量的同时，它通常会降低下游的生成质量 - 这是现有文献中未充分解决的挑战。为了解决这个问题，我们介绍了Gigatok，这是在缩放视觉令牌时同时改善图像重建，生成和表示学习的第一种方法。我们确定潜在空间的日益增长的复杂性是重建与产生困境的关键因素。为了减轻这种情况，我们提出了语义正则化，该语义正则化将令牌功能与预先训练的视觉编码器具有语义一致的特征对齐。该限制阻止了缩放期间过度的潜在空间复杂性，从而在重建和下游自回归产生中都能持续改善。在语义正则化的基础上，我们探讨了缩放象征器的三个关键实践：（1）使用1D令牌以更好的可伸缩性，（2）在扩展编码器和解码器时优先考虑解码器缩放，以及（3）使用熵损失来稳定培训数十亿个尺度的象征。通过扩展到$ \ bf {3 \ space亿美元} $参数，Gigatok在重建，下游AR生成和下游AR表示质量方面实现了最新的性能。