2025-04-18

Title: DMM: Building a Versatile Image Generation Model via Distillation-Based Model Merging

• Authors: Tianhui Song, Weixin Feng, Shuai Wang, Xubin Li, Tiezheng Ge, Bo Zheng, Limin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.12364
• Pdf URL: https://arxiv.org/pdf/2504.12364
• Copy Paste: [[2504.12364]] DMM: Building a Versatile Image Generation Model via Distillation-Based Model Merging(https://arxiv.org/abs/2504.12364)
• Keywords: generation
• Abstract: The success of text-to-image (T2I) generation models has spurred a proliferation of numerous model checkpoints fine-tuned from the same base model on various specialized datasets. This overwhelming specialized model production introduces new challenges for high parameter redundancy and huge storage cost, thereby necessitating the development of effective methods to consolidate and unify the capabilities of diverse powerful models into a single one. A common practice in model merging adopts static linear interpolation in the parameter space to achieve the goal of style mixing. However, it neglects the features of T2I generation task that numerous distinct models cover sundry styles which may lead to incompatibility and confusion in the merged model. To address this issue, we introduce a style-promptable image generation pipeline which can accurately generate arbitrary-style images under the control of style vectors. Based on this design, we propose the score distillation based model merging paradigm (DMM), compressing multiple models into a single versatile T2I model. Moreover, we rethink and reformulate the model merging task in the context of T2I generation, by presenting new merging goals and evaluation protocols. Our experiments demonstrate that DMM can compactly reorganize the knowledge from multiple teacher models and achieve controllable arbitrary-style generation.
• 摘要：文本对图像（T2I）生成模型的成功刺激了从各种专用数据集上同一基础模型微调的许多模型检查点的扩散。这种压倒性的专业模型生产为高参数冗余和巨大的存储成本带来了新的挑战，因此有必要开发有效的方法来整合和统一多样性强大的模型的能力。模型合并的一种常见实践在参数空间中采用静态线性插值，以实现样式混合的目标。但是，它忽略了T2I生成任务的特征，众多不同的模型涵盖了杂物样式，这可能会导致合并模型中的不兼容和混乱。为了解决此问题，我们引入了一种可供示意的图像生成管道，该管道可以在样式向量的控制下准确地生成任意风格的图像。基于此设计，我们提出了基于得分蒸馏的模型合并范式（DMM），将多个模型压缩到单个多功能T2I模型中。此外，我们通过提出新的合并目标和评估协议来重新思考和重新重新考虑T2I生成背景下的模型合并任务。我们的实验表明，DMM可以紧凑地重组来自多个教师模型的知识并实现可控制的任意形式。

Title: WORLDMEM: Long-term Consistent World Simulation with Memory

• Authors: Zeqi Xiao, Yushi Lan, Yifan Zhou, Wenqi Ouyang, Shuai Yang, Yanhong Zeng, Xingang Pan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.12369
• Pdf URL: https://arxiv.org/pdf/2504.12369
• Copy Paste: [[2504.12369]] WORLDMEM: Long-term Consistent World Simulation with Memory(https://arxiv.org/abs/2504.12369)
• Keywords: generation
• Abstract: World simulation has gained increasing popularity due to its ability to model virtual environments and predict the consequences of actions. However, the limited temporal context window often leads to failures in maintaining long-term consistency, particularly in preserving 3D spatial consistency. In this work, we present WorldMem, a framework that enhances scene generation with a memory bank consisting of memory units that store memory frames and states (e.g., poses and timestamps). By employing a memory attention mechanism that effectively extracts relevant information from these memory frames based on their states, our method is capable of accurately reconstructing previously observed scenes, even under significant viewpoint or temporal gaps. Furthermore, by incorporating timestamps into the states, our framework not only models a static world but also captures its dynamic evolution over time, enabling both perception and interaction within the simulated world. Extensive experiments in both virtual and real scenarios validate the effectiveness of our approach.
• 摘要：由于其对虚拟环境建模并预测行动后果的能力，世界模拟的流行程度越来越大。但是，有限的时间上下文窗口通常会导致保持长期一致性的失败，尤其是在保持3D空间一致性方面。在这项工作中，我们介绍了WorldMem，该框架通过由存储器单元组成的内存库增强场景的生成，该内存单元存储内存框架和状态（例如，姿势和时间戳）。通过采用一种记忆注意机制，该机制可以根据其状态有效从这些记忆框架中提取相关信息，我们的方法即使在显着的观点或时间间隙下，我们的方法也能够准确地重建先前观察到的场景。此外，通过将时间戳纳入州，我们的框架不仅建模了静态世界，而且还捕捉了随着时间的流逝的动态演变，从而在模拟世界中既可以感知和互动。在虚拟和实际场景中进行的广泛实验验证了我们方法的有效性。

Title: InstantCharacter: Personalize Any Characters with a Scalable Diffusion Transformer Framework

• Authors: Jiale Tao, Yanbing Zhang, Qixun Wang, Yiji Cheng, Haofan Wang, Xu Bai, Zhengguang Zhou, Ruihuang Li, Linqing Wang, Chunyu Wang, Qin Lin, Qinglin Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.12395
• Pdf URL: https://arxiv.org/pdf/2504.12395
• Copy Paste: [[2504.12395]] InstantCharacter: Personalize Any Characters with a Scalable Diffusion Transformer Framework(https://arxiv.org/abs/2504.12395)
• Keywords: generation
• Abstract: Current learning-based subject customization approaches, predominantly relying on U-Net architectures, suffer from limited generalization ability and compromised image quality. Meanwhile, optimization-based methods require subject-specific fine-tuning, which inevitably degrades textual controllability. To address these challenges, we propose InstantCharacter, a scalable framework for character customization built upon a foundation diffusion transformer. InstantCharacter demonstrates three fundamental advantages: first, it achieves open-domain personalization across diverse character appearances, poses, and styles while maintaining high-fidelity results. Second, the framework introduces a scalable adapter with stacked transformer encoders, which effectively processes open-domain character features and seamlessly interacts with the latent space of modern diffusion transformers. Third, to effectively train the framework, we construct a large-scale character dataset containing 10-million-level samples. The dataset is systematically organized into paired (multi-view character) and unpaired (text-image combinations) subsets. This dual-data structure enables simultaneous optimization of identity consistency and textual editability through distinct learning pathways. Qualitative experiments demonstrate the advanced capabilities of InstantCharacter in generating high-fidelity, text-controllable, and character-consistent images, setting a new benchmark for character-driven image generation. Our source code is available at this https URL.
• 摘要：当前基于学习的主题自定义方法主要依赖于U-NET体系结构，具有有限的概括能力和图像质量受损。同时，基于优化的方法需要特定于主题的微调，这不可避免地会降低文本可控性。为了应对这些挑战，我们提出了InstantCharacter，这是基础扩散变压器建立的角色自定义的可扩展框架。 InstantCharacter展示了三个基本优势：首先，它在保持高保真性结果的同时，在各种角色外观，姿势和样式上实现了开放域的个性化。其次，该框架引入了具有堆叠变压器编码器的可扩展适配器，该适配器有效地处理开放域特征，并与现代扩散变压器的潜在空间无缝相互作用。第三，为了有效地训练框架，我们构建了一个包含1000万级样本的大型字符数据集。该数据集系统地组织为配对（多视图字符）和未配对（文本图像组合）子集。这种双重数据结构可以通过不同的学习途径同时优化身份一致性和文本编辑性。定性实验证明了InstantCharacter在产生高保真，可控制的图像和角色一致的图像方面具有先进的功能，为角色驱动的图像生成设定了新的基准测试。我们的源代码可在此HTTPS URL上找到。

Title: Activated LoRA: Fine-tuned LLMs for Intrinsics

• Authors: Kristjan Greenewald, Luis Lastras, Thomas Parnell, Vraj Shah, Lucian Popa, Giulio Zizzo, Chulaka Gunasekara, Ambrish Rawat, David Cox
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.12397
• Pdf URL: https://arxiv.org/pdf/2504.12397
• Copy Paste: [[2504.12397]] Activated LoRA: Fine-tuned LLMs for Intrinsics(https://arxiv.org/abs/2504.12397)
• Keywords: generation
• Abstract: Low-Rank Adaptation (LoRA) has emerged as a highly efficient framework for finetuning the weights of large foundation models, and has become the go-to method for data-driven customization of LLMs. Despite the promise of highly customized behaviors and capabilities, switching between relevant LoRAs in a multiturn setting is highly inefficient, as the key-value (KV) cache of the entire turn history must be recomputed with the LoRA weights before generation can begin. To address this problem, we propose Activated LoRA (aLoRA), which modifies the LoRA framework to only adapt weights for the tokens in the sequence \emph{after} the aLoRA is invoked. This change crucially allows aLoRA to accept the base model's KV cache of the input string, meaning that aLoRA can be instantly activated whenever needed in a chain without recomputing the cache. This enables building what we call \emph{intrinsics}, i.e. highly specialized models invoked to perform well-defined operations on portions of an input chain or conversation that otherwise uses the base model by default. We use aLoRA to train a set of intrinsics models, demonstrating competitive accuracy with standard LoRA while achieving significant inference benefits.
• 摘要：低级适应性（LORA）已成为用于填充大型基础模型权重的高效框架，并已成为数据驱动的LLMS自定义的首选方法。尽管有高度定制的行为和能力的承诺，但在多弯设置中相关的洛拉斯之间的切换效率很低，因为在整个转弯历史记录的键值（KV）缓存必须在发电开始之前用Lora重量来重新计算。为了解决这个问题，我们提出了激活的lora（alora），该洛拉（Alora）修改了lora框架以仅在序列\ emph {后}中调整代币的权重。这种变化至关重要的是，Alora可以接受输入字符串的基本模型的KV缓存，这意味着可以在链条中需要时立即激活Alora而无需重新计算缓存。这使构建我们所谓的\ emph {interinsics}，即高度专业化的模型，以在输入链或对话的部分进行定义明确的操作，否则默认情况下使用基本模型。我们使用Alora来训练一组内在模型，证明了与标准LORA的竞争精度，同时实现了显着的推理优势。

Title: 3D-PointZshotS: Geometry-Aware 3D Point Cloud Zero-Shot Semantic Segmentation Narrowing the Visual-Semantic Gap

• Authors: Minmin Yang, Huantao Ren, Senem Velipasalar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.12442
• Pdf URL: https://arxiv.org/pdf/2504.12442
• Copy Paste: [[2504.12442]] 3D-PointZshotS: Geometry-Aware 3D Point Cloud Zero-Shot Semantic Segmentation Narrowing the Visual-Semantic Gap(https://arxiv.org/abs/2504.12442)
• Keywords: generation
• Abstract: Existing zero-shot 3D point cloud segmentation methods often struggle with limited transferability from seen classes to unseen classes and from semantic to visual space. To alleviate this, we introduce 3D-PointZshotS, a geometry-aware zero-shot segmentation framework that enhances both feature generation and alignment using latent geometric prototypes (LGPs). Specifically, we integrate LGPs into a generator via a cross-attention mechanism, enriching semantic features with fine-grained geometric details. To further enhance stability and generalization, we introduce a self-consistency loss, which enforces feature robustness against point-wise perturbations. Additionally, we re-represent visual and semantic features in a shared space, bridging the semantic-visual gap and facilitating knowledge transfer to unseen classes. Experiments on three real-world datasets, namely ScanNet, SemanticKITTI, and S3DIS, demonstrate that our method achieves superior performance over four baselines in terms of harmonic mIoU. The code is available at \href{this https URL}{Github}.
• 摘要：现有的零射击3D点云分割方法通常在从可见的类到看不见的类别以及从语义到视觉空间的有限转移性方面存在障碍。为了减轻这一点，我们介绍了3D Pointzshots，这是一种几何感知的零拍分段框架，可以使用潜在的几何原型（LGP）增强特征生成和对齐。具体而言，我们通过交叉注意机制将LGP集成到发电机中，并通过细粒的几何细节丰富了语义特征。为了进一步提高稳定性和概括，我们引入了自稳定性损失，该损失具有针对点扰动的鲁棒性。此外，我们在共享空间中重新代表视觉和语义特征，弥合语义 - 视觉差距，并促进知识转移到看不见的类别。在三个现实世界数据集的实验，即扫描仪，semantickitti和s3dis，这表明我们的方法在谐波miou方面取得了高于四个基准的卓越性能。该代码可在\ href {此https url} {github}上获得。

Title: Geometric Generality of Transformer-Based Gröbner Basis Computation

• Authors: Yuta Kambe, Yota Maeda, Tristan Vaccon
• Subjects: cs.LG, cs.SC, math.AG, stat.ML
• Abstract URL: https://arxiv.org/abs/2504.12465
• Pdf URL: https://arxiv.org/pdf/2504.12465
• Copy Paste: [[2504.12465]] Geometric Generality of Transformer-Based Gröbner Basis Computation(https://arxiv.org/abs/2504.12465)
• Keywords: generation
• Abstract: The intersection of deep learning and symbolic mathematics has seen rapid progress in recent years, exemplified by the work of Lample and Charton. They demonstrated that effective training of machine learning models for solving mathematical problems critically depends on high-quality, domain-specific datasets. In this paper, we address the computation of Gröbner basis using Transformers. While a dataset generation method tailored to Transformer-based Gröbner basis computation has previously been proposed, it lacked theoretical guarantees regarding the generality or quality of the generated datasets. In this work, we prove that datasets generated by the previously proposed algorithm are sufficiently general, enabling one to ensure that Transformers can learn a sufficiently diverse range of Gröbner bases. Moreover, we propose an extended and generalized algorithm to systematically construct datasets of ideal generators, further enhancing the training effectiveness of Transformer. Our results provide a rigorous geometric foundation for Transformers to address a mathematical problem, which is an answer to Lample and Charton's idea of training on diverse or representative inputs.
• 摘要：近年来，深度学习与象征性数学的交集取得了迅速的进步，例如Lample和Charton的工作。他们证明，对解决数学问题的机器学习模型的有效培训严重取决于高质量的特定于领域的数据集。在本文中，我们使用变压器解决了Gröbner基础的计算。虽然先前已经提出了针对基于变压器的Gröbner基础计算来量身定制的数据集生成方法，但它缺乏有关生成数据集的一般性或质量的理论保证。在这项工作中，我们证明了先前提出的算法生成的数据集足够通用，从而使人们能够确保变形金刚能够学习足够多样化的Gröbner碱基。此外，我们向系统构建理想发电机的数据集提出了一种扩展和广义的算法，从而进一步提高了变压器的训练有效性。我们的结果为变形金刚解决数学问题提供了严格的几何基础，这是对Lample和Charton对各种或代表性输入的培训的想法的答案。

Title: Prompt-Driven and Training-Free Forgetting Approach and Dataset for Large Language Models

• Authors: Zhenyu Yu, Mohd Yamani Inda Idris, Pei Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.12574
• Pdf URL: https://arxiv.org/pdf/2504.12574
• Copy Paste: [[2504.12574]] Prompt-Driven and Training-Free Forgetting Approach and Dataset for Large Language Models(https://arxiv.org/abs/2504.12574)
• Keywords: generation, generative
• Abstract: The widespread adoption of diffusion models in image generation has increased the demand for privacy-compliant unlearning. However, due to the high-dimensional nature and complex feature representations of diffusion models, achieving selective unlearning remains challenging, as existing methods struggle to remove sensitive information while preserving the consistency of non-sensitive regions. To address this, we propose an Automatic Dataset Creation Framework based on prompt-based layered editing and training-free local feature removal, constructing the ForgetMe dataset and introducing the Entangled evaluation metric. The Entangled metric quantifies unlearning effectiveness by assessing the similarity and consistency between the target and background regions and supports both paired (Entangled-D) and unpaired (Entangled-S) image data, enabling unsupervised evaluation. The ForgetMe dataset encompasses a diverse set of real and synthetic scenarios, including CUB-200-2011 (Birds), Stanford-Dogs, ImageNet, and a synthetic cat dataset. We apply LoRA fine-tuning on Stable Diffusion to achieve selective unlearning on this dataset and validate the effectiveness of both the ForgetMe dataset and the Entangled metric, establishing them as benchmarks for selective unlearning. Our work provides a scalable and adaptable solution for advancing privacy-preserving generative AI.
• 摘要：图像产生中扩散模型的广泛采用增加了对符合隐私性的未学习的需求。但是，由于扩散模型的高维质和复杂的特征表示，因此实现选择性的实现学习性仍然具有挑战性，因为现有方法难以消除敏感信息，同时保留了非敏感区域的一致性。为了解决这个问题，我们根据迅速的分层编辑和无培训的本地功能删除，提出一个自动数据集创建框架，构建忘记的数据集并介绍纠缠的评估指标。纠缠的度量标准通过评估目标区域和背景区域之间的相似性和一致性来量化学习效率，并支持配对（纠缠d）和未配对（纠缠的S）图像数据，从而实现无监督评估。忘记的数据集涵盖了各种各样的真实和合成场景，包括Cub-200-2011（鸟），斯坦福 - 狗狗，Imagenet和合成CAT数据集。我们在稳定的扩散上应用Lora微调，以在此数据集上实现选择性学习，并验证HevensMe数据集和纠缠度量的有效性，并确定它们作为选择性学习的基准。我们的工作提供了一种可扩展且适应性的解决方案，用于推进隐私保护生成AI。

Title: AdaQual-Diff: Diffusion-Based Image Restoration via Adaptive Quality Prompting

• Authors: Xin Su, Chen Wu, Yu Zhang, Chen Lyu, Zhuoran Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.12605
• Pdf URL: https://arxiv.org/pdf/2504.12605
• Copy Paste: [[2504.12605]] AdaQual-Diff: Diffusion-Based Image Restoration via Adaptive Quality Prompting(https://arxiv.org/abs/2504.12605)
• Keywords: restoration, generative, quality assessment
• Abstract: Restoring images afflicted by complex real-world degradations remains challenging, as conventional methods often fail to adapt to the unique mixture and severity of artifacts present. This stems from a reliance on indirect cues which poorly capture the true perceptual quality deficit. To address this fundamental limitation, we introduce AdaQual-Diff, a diffusion-based framework that integrates perceptual quality assessment directly into the generative restoration process. Our approach establishes a mathematical relationship between regional quality scores from DeQAScore and optimal guidance complexity, implemented through an Adaptive Quality Prompting mechanism. This mechanism systematically modulates prompt structure according to measured degradation severity: regions with lower perceptual quality receive computationally intensive, structurally complex prompts with precise restoration directives, while higher quality regions receive minimal prompts focused on preservation rather than intervention. The technical core of our method lies in the dynamic allocation of computational resources proportional to degradation severity, creating a spatially-varying guidance field that directs the diffusion process with mathematical precision. By combining this quality-guided approach with content-specific conditioning, our framework achieves fine-grained control over regional restoration intensity without requiring additional parameters or inference iterations. Experimental results demonstrate that AdaQual-Diff achieves visually superior restorations across diverse synthetic and real-world datasets.
• 摘要：恢复受复杂现实世界降解困扰的图像仍然具有挑战性，因为常规方法通常无法适应存在的独特混合物和严重性。这源于对间接提示的依赖，而间接提示则捕捉了真正的感知质量赤字。为了解决这一基本限制，我们引入了Adaqual-Diff，这是一个基于扩散的框架，将感知质量评估直接整合到生成恢复过程中。我们的方法建立了通过自适应质量提示机制实施的Deqascore的区域质量评分与最佳指导复杂性之间的数学关系。该机制根据测量的降解严重程度系统地调节了迅速的结构：感知质量较低的区域具有计算密集，结构复杂的提示，并具有精确的恢复指令，而较高质量的区域则获得最小的提示，专注于保存而不是干预。我们方法的技术核心在于计算资源与降解严重程度成比例的动态分配，创建了一个与空间变化的指导字段，该领域将扩散过程指导以数学精度为单位。通过将这种质量引导的方法与特定于内容的调理相结合，我们的框架可以对区域恢复强度进行细粒度的控制，而无需其他参数或推理迭代。实验结果表明，Adaqual-Diff在不同的合成和现实世界数据集中实现了视觉上较高的修复体。

Title: Robo-SGG: Exploiting Layout-Oriented Normalization and Restitution for Robust Scene Graph Generation

• Authors: Changsheng Lv, Mengshi Qi, Zijian Fu, Huadong Ma
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.12606
• Pdf URL: https://arxiv.org/pdf/2504.12606
• Copy Paste: [[2504.12606]] Robo-SGG: Exploiting Layout-Oriented Normalization and Restitution for Robust Scene Graph Generation(https://arxiv.org/abs/2504.12606)
• Keywords: generation
• Abstract: In this paper, we introduce a novel method named Robo-SGG, i.e., Layout-Oriented Normalization and Restitution for Robust Scene Graph Generation. Compared to the existing SGG setting, the robust scene graph generation aims to perform inference on a diverse range of corrupted images, with the core challenge being the domain shift between the clean and corrupted images. Existing SGG methods suffer from degraded performance due to compromised visual features e.g., corruption interference or occlusions. To obtain robust visual features, we exploit the layout information, which is domain-invariant, to enhance the efficacy of existing SGG methods on corrupted images. Specifically, we employ Instance Normalization(IN) to filter out the domain-specific feature and recover the unchangeable structural features, i.e., the positional and semantic relationships among objects by the proposed Layout-Oriented Restitution. Additionally, we propose a Layout-Embedded Encoder (LEE) that augments the existing object and predicate encoders within the SGG framework, enriching the robust positional and semantic features of objects and predicates. Note that our proposed Robo-SGG module is designed as a plug-and-play component, which can be easily integrated into any baseline SGG model. Extensive experiments demonstrate that by integrating the state-of-the-art method into our proposed Robo-SGG, we achieve relative improvements of 5.6%, 8.0%, and 6.5% in mR@50 for PredCls, SGCls, and SGDet tasks on the VG-C dataset, respectively, and achieve new state-of-the-art performance in corruption scene graph generation benchmark (VG-C and GQA-C). We will release our source code and model.
• 摘要：在本文中，我们介绍了一种名为Robo-SGG的新方法，即，以布局为导向的标准化和恢复稳定的场景图生成。与现有的SGG设置相比，强大的场景图生成旨在对各种损坏的图像进行推断，核心挑战是清洁图像和损坏的图像之间的域移动。现有的SGG方法由于视觉功能受损而导致性能降低，例如腐败干扰或阻塞。为了获得强大的视觉功能，我们利用了域不变的布局信息，以增强现有SGG方法在损坏的图像上的疗效。具体而言，我们采用实例归一化（in）来滤除特定于域的特征并恢复不变的结构特征，即通过提出的面向布局的恢复原状的对象之间的位置和语义关系。此外，我们提出了一个布局插入的编码器（LEE），该编码器（LEE）增加了SGG框架中的现有对象和谓词编码器，从而丰富了对象和谓词的可靠位置和语义特征。请注意，我们提出的Robo-SGG模块被设计为插件组件，可以轻松地集成到任何基线SGG模型中。广泛的实验表明，通过将最新方法集成到我们提出的Robo-SGG中，我们在VG-C数据集上的PredCL，SGCL和SGDET任务的相对改善中的MR@50中的相对改善分别为5.6％，8.0％和6.5％。我们将发布我们的源代码和模型。

Title: Packing Input Frame Context in Next-Frame Prediction Models for Video Generation

• Authors: Lvmin Zhang, Maneesh Agrawala
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.12626
• Pdf URL: https://arxiv.org/pdf/2504.12626
• Copy Paste: [[2504.12626]] Packing Input Frame Context in Next-Frame Prediction Models for Video Generation(https://arxiv.org/abs/2504.12626)
• Keywords: generation
• Abstract: We present a neural network structure, FramePack, to train next-frame (or next-frame-section) prediction models for video generation. The FramePack compresses input frames to make the transformer context length a fixed number regardless of the video length. As a result, we are able to process a large number of frames using video diffusion with computation bottleneck similar to image diffusion. This also makes the training video batch sizes significantly higher (batch sizes become comparable to image diffusion training). We also propose an anti-drifting sampling method that generates frames in inverted temporal order with early-established endpoints to avoid exposure bias (error accumulation over iterations). Finally, we show that existing video diffusion models can be finetuned with FramePack, and their visual quality may be improved because the next-frame prediction supports more balanced diffusion schedulers with less extreme flow shift timesteps.
• 摘要：我们提出了一个神经网络结构FramePack，以训练视频生成的下一框架（或下一框架）预测模型。 FramePack压缩输入帧，使变压器上下文长度成为固定数字，而不论视频长度如何。结果，我们能够使用与图像扩散相似的计算瓶颈来处理大量帧。这也使训练视频批量大小显着更高（批量大小可与图像扩散训练相当）。我们还提出了一种反灌溉采样方法，该方法以倒置的时间顺序生成框架，并具有早期建立的端点，以避免暴露偏见（迭代误差积累）。最后，我们表明现有的视频扩散模型可以用framepack进行填充，并且可以提高其视觉质量，因为下一框架预测支持更平衡的扩散调度程序，并且具有较小的极端流动时间段。

Title: Two Tasks, One Goal: Uniting Motion and Planning for Excellent End To End Autonomous Driving Performance

• Authors: Lin Liu, Ziying Song, Hongyu Pan, Lei Yang, Caiyan Jia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.12667
• Pdf URL: https://arxiv.org/pdf/2504.12667
• Copy Paste: [[2504.12667]] Two Tasks, One Goal: Uniting Motion and Planning for Excellent End To End Autonomous Driving Performance(https://arxiv.org/abs/2504.12667)
• Keywords: generation
• Abstract: End-to-end autonomous driving has made impressive progress in recent years. Former end-to-end autonomous driving approaches often decouple planning and motion tasks, treating them as separate modules. This separation overlooks the potential benefits that planning can gain from learning out-of-distribution data encountered in motion tasks. However, unifying these tasks poses significant challenges, such as constructing shared contextual representations and handling the unobservability of other vehicles' states. To address these challenges, we propose TTOG, a novel two-stage trajectory generation framework. In the first stage, a diverse set of trajectory candidates is generated, while the second stage focuses on refining these candidates through vehicle state information. To mitigate the issue of unavailable surrounding vehicle states, TTOG employs a self-vehicle data-trained state estimator, subsequently extended to other vehicles. Furthermore, we introduce ECSA (equivariant context-sharing scene adapter) to enhance the generalization of scene representations across different agents. Experimental results demonstrate that TTOG achieves state-of-the-art performance across both planning and motion tasks. Notably, on the challenging open-loop nuScenes dataset, TTOG reduces the L2 distance by 36.06\%. Furthermore, on the closed-loop Bench2Drive dataset, our approach achieves a 22\% improvement in the driving score (DS), significantly outperforming existing baselines.
• 摘要：近年来，端到端的自动驾驶取得了令人印象深刻的进步。前端到端的自动驾驶方法通常会使计划和运动任务脱致，将其视为单独的模块。这种分离忽略了计划在运动任务中遇到的分布数据外的潜在利益。但是，统一这些任务提出了重大挑战，例如构建共享的上下文表示并处理其他车辆州的不可观念。为了应对这些挑战，我们提出了TTOG，这是一种新型的两阶段轨迹生成框架。在第一阶段，产生了各种轨迹候选者，而第二阶段则着重于通过车辆状态信息来完善这些候选者。为了减轻周围车辆状态不可用的问题，TTOG采用了自动驾驶数据训练的状态估计器，随后扩展到其他车辆。此外，我们介绍了ECSA（ECSA（ECSA）（等效性上下文共享场景适配器），以增强不同代理商的场景表示的概括。实验结果表明，TTOG在计划和运动任务中都达到了最先进的表现。值得注意的是，在具有挑战性的开环Nuscenes数据集中，TTOG将L2距离降低36.06 \％。此外，在闭环台式数据集中，我们的方法在驾驶得分（DS）方面取得了22 \％的提高，大大优于现有基线。

Title: HSS-IAD: A Heterogeneous Same-Sort Industrial Anomaly Detection Dataset

• Authors: Qishan Wang, Shuyong Gao, Junjie Hu, Jiawen Yu, Xuan Tong, You Li, Wenqiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.12689
• Pdf URL: https://arxiv.org/pdf/2504.12689
• Copy Paste: [[2504.12689]] HSS-IAD: A Heterogeneous Same-Sort Industrial Anomaly Detection Dataset(https://arxiv.org/abs/2504.12689)
• Keywords: generation
• Abstract: Multi-class Unsupervised Anomaly Detection algorithms (MUAD) are receiving increasing attention due to their relatively low deployment costs and improved training efficiency. However, the real-world effectiveness of MUAD methods is questioned due to limitations in current Industrial Anomaly Detection (IAD) datasets. These datasets contain numerous classes that are unlikely to be produced by the same factory and fail to cover multiple structures or appearances. Additionally, the defects do not reflect real-world characteristics. Therefore, we introduce the Heterogeneous Same-Sort Industrial Anomaly Detection (HSS-IAD) dataset, which contains 8,580 images of metallic-like industrial parts and precise anomaly annotations. These parts exhibit variations in structure and appearance, with subtle defects that closely resemble the base materials. We also provide foreground images for synthetic anomaly generation. Finally, we evaluate popular IAD methods on this dataset under multi-class and class-separated settings, demonstrating its potential to bridge the gap between existing datasets and real factory conditions. The dataset is available at this https URL.
• 摘要：多类无监督的异常检测算法（MUAD）由于部署成本相对较低和提高培训效率而受到越来越多的关注。但是，由于当前工业异常检测（IAD）数据集的局限性，MUAD方法的现实有效性受到质疑。这些数据集包含许多不太可能由同一工厂生产的类，并且无法涵盖多个结构或外观。此外，缺陷不能反映现实世界的特征。因此，我们介绍了异质的同类工业异常检测（HSS-AIAD）数据集，其中包含8,580张金属样工业零件和精确的异常注释。这些部分在结构和外观方面表现出差异，具有与基本材料相似的细微缺陷。我们还为合成异常生成提供前景图像。最后，我们在多级和类别分离的设置下评估了该数据集上流行的IAD方法，这表明了其在现有数据集和实际工厂条件之间弥合差距的潜力。该数据集可在此HTTPS URL上找到。

Title: Unsupervised Cross-Domain 3D Human Pose Estimation via Pseudo-Label-Guided Global Transforms

• Authors: Jingjing Liu, Zhiyong Wang, Xinyu Fan, Amirhossein Dadashzadeh, Honghai Liu, Majid Mirmehdi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.12699
• Pdf URL: https://arxiv.org/pdf/2504.12699
• Copy Paste: [[2504.12699]] Unsupervised Cross-Domain 3D Human Pose Estimation via Pseudo-Label-Guided Global Transforms(https://arxiv.org/abs/2504.12699)
• Keywords: generation
• Abstract: Existing 3D human pose estimation methods often suffer in performance, when applied to cross-scenario inference, due to domain shifts in characteristics such as camera viewpoint, position, posture, and body size. Among these factors, camera viewpoints and locations {have been shown} to contribute significantly to the domain gap by influencing the global positions of human poses. To address this, we propose a novel framework that explicitly conducts global transformations between pose positions in the camera coordinate systems of source and target domains. We start with a Pseudo-Label Generation Module that is applied to the 2D poses of the target dataset to generate pseudo-3D poses. Then, a Global Transformation Module leverages a human-centered coordinate system as a novel bridging mechanism to seamlessly align the positional orientations of poses across disparate domains, ensuring consistent spatial referencing. To further enhance generalization, a Pose Augmentor is incorporated to address variations in human posture and body size. This process is iterative, allowing refined pseudo-labels to progressively improve guidance for domain adaptation. Our method is evaluated on various cross-dataset benchmarks, including Human3.6M, MPI-INF-3DHP, and 3DPW. The proposed method outperforms state-of-the-art approaches and even outperforms the target-trained model.
• 摘要：由于特征（例如摄像机的观点，位置，姿势和身体大小）的域变化，现有的3D人姿势估计方法在性能中通常会受到性能的影响。在这些因素中，摄像机的观点和位置{已显示}通过影响人类姿势的全球位置，从而为域间隙做出了重大贡献。为了解决这个问题，我们提出了一个新颖的框架，该框架明确地在源源和目标域的摄像头坐标系中进行姿势位置之间进行了全局转换。我们从应用于目标数据集的2D姿势的伪标签生成模块开始，以生成伪-3D姿势。然后，一个全局变换模块利用以人为本的坐标系统作为一种新型的桥接机制，可以无缝地对齐跨不同领域的姿势的位置取向，从而确保一致的空间参考。为了进一步增强概括，纳入了姿势增强器，以解决人类姿势和身体大小的变化。这个过程是迭代的，允许精制的伪标签逐步改善领域适应性的指导。我们的方法在各种跨数据基准测试中进行了评估，包括Human36M，MPI-INF-3DHP和3DPW。所提出的方法优于最先进的方法，甚至优于目标训练的模型。

Title: SmartFreeEdit: Mask-Free Spatial-Aware Image Editing with Complex Instruction Understanding

• Authors: Qianqian Sun, Jixiang Luo, Dell Zhang, Xuelong Li
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2504.12704
• Pdf URL: https://arxiv.org/pdf/2504.12704
• Copy Paste: [[2504.12704]] SmartFreeEdit: Mask-Free Spatial-Aware Image Editing with Complex Instruction Understanding(https://arxiv.org/abs/2504.12704)
• Keywords: generation
• Abstract: Recent advancements in image editing have utilized large-scale multimodal models to enable intuitive, natural instruction-driven interactions. However, conventional methods still face significant challenges, particularly in spatial reasoning, precise region segmentation, and maintaining semantic consistency, especially in complex scenes. To overcome these challenges, we introduce SmartFreeEdit, a novel end-to-end framework that integrates a multimodal large language model (MLLM) with a hypergraph-enhanced inpainting architecture, enabling precise, mask-free image editing guided exclusively by natural language instructions. The key innovations of SmartFreeEdit include:(1)the introduction of region aware tokens and a mask embedding paradigm that enhance the spatial understanding of complex scenes;(2) a reasoning segmentation pipeline designed to optimize the generation of editing masks based on natural language instructions;and (3) a hypergraph-augmented inpainting module that ensures the preservation of both structural integrity and semantic coherence during complex edits, overcoming the limitations of local-based image generation. Extensive experiments on the Reason-Edit benchmark demonstrate that SmartFreeEdit surpasses current state-of-the-art methods across multiple evaluation metrics, including segmentation accuracy, instruction adherence, and visual quality preservation, while addressing the issue of local information focus and improving global consistency in the edited image. Our project will be available at this https URL.
• 摘要：图像编辑的最新进展利用了大规模的多模型模型来实现直观的自然教学驱动相互作用。但是，传统方法仍然面临重大挑战，尤其是在空间推理，精确的区域细分和保持语义一致性的情况下，尤其是在复杂的场景中。为了克服这些挑战，我们介绍了SmartFreeedit，这是一个新颖的端到端框架，将多模式的大语言模型（MLLM）与超图增强的镶嵌体系结构集成在一起，从而启用精确的，无面膜的图像编辑，由自然语言说明独特地指导。 SmartFreeedIt的关键创新包括：（1）引入区域知识令牌和嵌入范式的嵌入范式，以增强对复杂场景的空间理解；（2）推理分段管道；旨在优化基于自然语言指令的编辑蒙版的生成，并确保在自然语言中的构建中，并（3）构建超级构造的模块化；编辑，克服基于本地图像的生成的局限性。关于理性基准基准的广泛实验表明，SmartFreeedIt超过了多个评估指标的当前最新方法，包括细分精度，指导依从性和视觉质量保存，同时解决了本地信息的重点问题，并提高了所编辑的图像中的全球一致性。我们的项目将在此HTTPS URL上可用。

Title: Set You Straight: Auto-Steering Denoising Trajectories to Sidestep Unwanted Concepts

• Authors: Leyang Li, Shilin Lu, Yan Ren, Adams Wai-Kin Kong
• Subjects: cs.CV, cs.AI, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.12782
• Pdf URL: https://arxiv.org/pdf/2504.12782
• Copy Paste: [[2504.12782]] Set You Straight: Auto-Steering Denoising Trajectories to Sidestep Unwanted Concepts(https://arxiv.org/abs/2504.12782)
• Keywords: generation, generative
• Abstract: Ensuring the ethical deployment of text-to-image models requires effective techniques to prevent the generation of harmful or inappropriate content. While concept erasure methods offer a promising solution, existing finetuning-based approaches suffer from notable limitations. Anchor-free methods risk disrupting sampling trajectories, leading to visual artifacts, while anchor-based methods rely on the heuristic selection of anchor concepts. To overcome these shortcomings, we introduce a finetuning framework, dubbed ANT, which Automatically guides deNoising Trajectories to avoid unwanted concepts. ANT is built on a key insight: reversing the condition direction of classifier-free guidance during mid-to-late denoising stages enables precise content modification without sacrificing early-stage structural integrity. This inspires a trajectory-aware objective that preserves the integrity of the early-stage score function field, which steers samples toward the natural image manifold, without relying on heuristic anchor concept selection. For single-concept erasure, we propose an augmentation-enhanced weight saliency map to precisely identify the critical parameters that most significantly contribute to the unwanted concept, enabling more thorough and efficient erasure. For multi-concept erasure, our objective function offers a versatile plug-and-play solution that significantly boosts performance. Extensive experiments demonstrate that ANT achieves state-of-the-art results in both single and multi-concept erasure, delivering high-quality, safe outputs without compromising the generative fidelity. Code is available at this https URL
• 摘要：确保文本对图像模型的道德部署需要有效的技术，以防止产生有害或不适当的内容。尽管概念擦除方法提供了一种有希望的解决方案，但现有的基于填充的方法受到了显着限制。无锚方法有可能破坏采样轨迹，导致视觉伪像，而基于锚的方法依靠锚定概念的启发式选择。为了克服这些缺点，我们引入了一个被称为蚂蚁的填充框架，该框架自动指导deno轨迹以避免不必要的概念。 ANT建立在关键见解的基础上：扭转中高层降级阶段中无分类器指导的条件方向，可以在不牺牲早期结构完整性的情况下进行精确的内容修改。这激发了一个轨迹感知的目标，该目标保留了早期分数功能领域的完整性，该目标将样品引导到自然图像歧管，而无需依靠启发式锚概念选择。对于单一概念擦除，我们提出了一个增强增强的重量显着性图，以精确地确定最大程度地有助于不需要的概念的关键参数，从而更彻底有效地擦除。对于多概念擦除，我们的目标函数提供了一种多功能的插件解决方案，可显着提高性能。广泛的实验表明，ANT实现最新的最新实验会导致单一和多概念擦除，从而提供高质量的安全产出，而不会损害生成的忠诚度。代码可在此HTTPS URL上找到

Title: Saliency-Aware Diffusion Reconstruction for Effective Invisible Watermark Removal

• Authors: Inzamamul Alam, Md Tanvir Islam, Simon S. Woo
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2504.12809
• Pdf URL: https://arxiv.org/pdf/2504.12809
• Copy Paste: [[2504.12809]] Saliency-Aware Diffusion Reconstruction for Effective Invisible Watermark Removal(https://arxiv.org/abs/2504.12809)
• Keywords: restoration
• Abstract: As digital content becomes increasingly ubiquitous, the need for robust watermark removal techniques has grown due to the inadequacy of existing embedding techniques, which lack robustness. This paper introduces a novel Saliency-Aware Diffusion Reconstruction (SADRE) framework for watermark elimination on the web, combining adaptive noise injection, region-specific perturbations, and advanced diffusion-based reconstruction. SADRE disrupts embedded watermarks by injecting targeted noise into latent representations guided by saliency masks although preserving essential image features. A reverse diffusion process ensures high-fidelity image restoration, leveraging adaptive noise levels determined by watermark strength. Our framework is theoretically grounded with stability guarantees and achieves robust watermark removal across diverse scenarios. Empirical evaluations on state-of-the-art (SOTA) watermarking techniques demonstrate SADRE's superiority in balancing watermark disruption and image quality. SADRE sets a new benchmark for watermark elimination, offering a flexible and reliable solution for real-world web content. Code is available on~\href{this https URL}{\textbf{this https URL}}.
• 摘要：随着数字内容变得越来越无处不在，由于缺乏稳健性的现有嵌入技术不足，对强大的水印技术的需求不足。本文介绍了一种新型的显着性扩散重建（SADRE），用于消除网络上的水印，结合自适应噪声注射，特定区域的扰动和基于先进的扩散的重建。萨德雷（Sadre）通过将目标噪声注射到由显着性面具引导的潜在表示中，尽管保留了基本的图像特征，从而破坏了嵌入式水印。反向扩散过程可确保高保真图像恢复，从而利用通过水印强度确定的自适应噪声水平。从理论上讲，我们的框架以稳定性的保证为基础，并在各种情况下实现了强大的水印。对先进（SOTA）水印技术的经验评估表明，萨德雷在平衡水印的破坏和图像质量方面具有优势。萨德雷（Sadre）为消除水印设定了一个新的基准，为现实世界中的网络内容提供了灵活且可靠的解决方案。代码可在〜\ href {此https url} {\ textbf {this https url}}上获得。

Title: TwoSquared: 4D Generation from 2D Image Pairs

• Authors: Lu Sang, Zehranaz Canfes, Dongliang Cao, Riccardo Marin, Florian Bernard, Daniel Cremers
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.12825
• Pdf URL: https://arxiv.org/pdf/2504.12825
• Copy Paste: [[2504.12825]] TwoSquared: 4D Generation from 2D Image Pairs(https://arxiv.org/abs/2504.12825)
• Keywords: generation, generative
• Abstract: Despite the astonishing progress in generative AI, 4D dynamic object generation remains an open challenge. With limited high-quality training data and heavy computing requirements, the combination of hallucinating unseen geometry together with unseen movement poses great challenges to generative models. In this work, we propose TwoSquared as a method to obtain a 4D physically plausible sequence starting from only two 2D RGB images corresponding to the beginning and end of the action. Instead of directly solving the 4D generation problem, TwoSquared decomposes the problem into two steps: 1) an image-to-3D module generation based on the existing generative model trained on high-quality 3D assets, and 2) a physically inspired deformation module to predict intermediate movements. To this end, our method does not require templates or object-class-specific prior knowledge and can take in-the-wild images as input. In our experiments, we demonstrate that TwoSquared is capable of producing texture-consistent and geometry-consistent 4D sequences only given 2D images.
• 摘要：尽管生成AI取得了惊人的进展，但4D动态对象生成仍然是一个开放的挑战。凭借有限的高质量训练数据和重型计算要求，幻觉看不见的几何形状以及看不见的运动的结合对生成模型构成了巨大挑战。在这项工作中，我们提出了TwoSquared作为一种获得4D物理上合理的序列的方法，从仅两个与动作开始和结束相对应的2D RGB图像开始。 TwoSquared不是直接解决4D代问题，而是将问题分解为两个步骤：1）基于现有的基于对高质量3D资产训练的现有生成模型的图像到3D模块生成，以及2）具有物理启发的变形模块，以预测中间运动。为此，我们的方法不需要模板或特定于对象类的先验知识，并且可以将野外图像作为输入。在我们的实验中，我们证明了TwoSquared能够仅给定2D图像产生纹理符合和几何符合的4D序列。

Title: High-Fidelity Image Inpainting with Multimodal Guided GAN Inversion

• Authors: Libo Zhang, Yongsheng Yu, Jiali Yao, Heng Fan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.12844
• Pdf URL: https://arxiv.org/pdf/2504.12844
• Copy Paste: [[2504.12844]] High-Fidelity Image Inpainting with Multimodal Guided GAN Inversion(https://arxiv.org/abs/2504.12844)
• Keywords: generative
• Abstract: Generative Adversarial Network (GAN) inversion have demonstrated excellent performance in image inpainting that aims to restore lost or damaged image texture using its unmasked content. Previous GAN inversion-based methods usually utilize well-trained GAN models as effective priors to generate the realistic regions for missing holes. Despite excellence, they ignore a hard constraint that the unmasked regions in the input and the output should be the same, resulting in a gap between GAN inversion and image inpainting and thus degrading the performance. Besides, existing GAN inversion approaches often consider a single modality of the input image, neglecting other auxiliary cues in images for improvements. Addressing these problems, we propose a novel GAN inversion approach, dubbed MMInvertFill, for image inpainting. MMInvertFill contains primarily a multimodal guided encoder with a pre-modulation and a GAN generator with F&W+ latent space. Specifically, the multimodal encoder aims to enhance the multi-scale structures with additional semantic segmentation edge texture modalities through a gated mask-aware attention module. Afterwards, a pre-modulation is presented to encode these structures into style vectors. To mitigate issues of conspicuous color discrepancy and semantic inconsistency, we introduce the F&W+ latent space to bridge the gap between GAN inversion and image inpainting. Furthermore, in order to reconstruct faithful and photorealistic images, we devise a simple yet effective Soft-update Mean Latent module to capture more diversified in-domain patterns for generating high-fidelity textures for massive corruptions. In our extensive experiments on six challenging datasets, we show that our MMInvertFill qualitatively and quantitatively outperforms other state-of-the-arts and it supports the completion of out-of-domain images effectively.
• 摘要：生成对抗网络（GAN）反演表现出出色的图像插入性能，旨在使用其未掩盖的内容恢复丢失或损坏的图像纹理。以前的基于GAN倒置的方法通常利用训练有素的GAN模型作为有效的先验，以生成缺失孔的现实区域。尽管出色，但他们忽略了一个严格的限制，即输入和输出中的未掩盖区域应相同，从而导致gan倒置和图像插入的差距，从而降低性能。此外，现有的GAN反转方法通常考虑输入图像的单一方式，从而忽略了图像中的其他辅助提示以进行改进。在解决这些问题时，我们提出了一种新型的gan倒置方法，称为mminvertfill，用于图像介入。 mminvertfill主要包含具有预调制的多模式引导编码器，并且具有F＆W+潜在空间的GAN发电机。具体而言，多模式编码器旨在通过通过封闭式面膜吸引的注意模块来增强多尺度结构，并具有其他语义分割边缘纹理方式。之后，提出了预调整，以将这些结构编码为样式向量。为了减轻明显的颜色差异和语义不一致的问题，我们介绍了F＆W+潜在空间，以弥合GAN倒置和图像插入之间的差距。此外，为了重建忠实和逼真的图像，我们设计了一个简单而有效的软上升的潜在模块，以捕获更多多元化的内域模式，以生成高保真质地，以实现大规模腐败。在我们在六个具有挑战性的数据集上进行的广泛实验，我们表明我们的Mminvertfill定性和定量表现优于其他最先进的方法，并且支持有效完成外域图像的完成。

Title: TTRD3: Texture Transfer Residual Denoising Dual Diffusion Model for Remote Sensing Image Super-Resolution

• Authors: Yide Liu, Haijiang Sun, Xiaowen Zhang, Qiaoyuan Liu, Zhouchang Chen, Chongzhuo Xiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.13026
• Pdf URL: https://arxiv.org/pdf/2504.13026
• Copy Paste: [[2504.13026]] TTRD3: Texture Transfer Residual Denoising Dual Diffusion Model for Remote Sensing Image Super-Resolution(https://arxiv.org/abs/2504.13026)
• Keywords: super-resolution, generation
• Abstract: Remote Sensing Image Super-Resolution (RSISR) reconstructs high-resolution (HR) remote sensing images from low-resolution inputs to support fine-grained ground object interpretation. Existing methods face three key challenges: (1) Difficulty in extracting multi-scale features from spatially heterogeneous RS scenes, (2) Limited prior information causing semantic inconsistency in reconstructions, and (3) Trade-off imbalance between geometric accuracy and visual quality. To address these issues, we propose the Texture Transfer Residual Denoising Dual Diffusion Model (TTRD3) with three innovations: First, a Multi-scale Feature Aggregation Block (MFAB) employing parallel heterogeneous convolutional kernels for multi-scale feature extraction. Second, a Sparse Texture Transfer Guidance (STTG) module that transfers HR texture priors from reference images of similar scenes. Third, a Residual Denoising Dual Diffusion Model (RDDM) framework combining residual diffusion for deterministic reconstruction and noise diffusion for diverse generation. Experiments on multi-source RS datasets demonstrate TTRD3's superiority over state-of-the-art methods, achieving 1.43% LPIPS improvement and 3.67% FID enhancement compared to best-performing baselines. Code/model: this https URL.
• 摘要：遥感图像超分辨率（RSISR）重建了来自低分辨率输入的高分辨率（HR）遥感图像，以支持细颗粒的地面对象解释。现有方法面临三个主要挑战：（1）难以从空间异质RS场景中提取多尺度特征，（2）有限的先前信息导致重建语义不一致，以及（3）几何精度和视觉质量之间的权衡不平衡。为了解决这些问题，我们提出了具有三个创新的纹理转移剩余双重扩散模型（TTRD3）：首先，采用平行异质卷积内核来用于多规模的多尺度提取。其次，稀疏的纹理传输指南（STTG）模块，该模块从相似场景的参考图像转移人力资源纹理先验。第三，一个残留的denoing双重扩散模型（RDDM）框架，结合了多种产生的确定性重建和噪声扩散的残留扩散。对多源RS数据集进行的实验证明了TTRD3优于最先进的方法，与表现最好的基线相比，LPIPS提高了1.43％和3.67％的FID。代码/型号：此HTTPS URL。

Title: Event-Enhanced Blurry Video Super-Resolution

• Authors: Dachun Kai, Yueyi Zhang, Jin Wang, Zeyu Xiao, Zhiwei Xiong, Xiaoyan Sun
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.13042
• Pdf URL: https://arxiv.org/pdf/2504.13042
• Copy Paste: [[2504.13042]] Event-Enhanced Blurry Video Super-Resolution(https://arxiv.org/abs/2504.13042)
• Keywords: super-resolution
• Abstract: In this paper, we tackle the task of blurry video super-resolution (BVSR), aiming to generate high-resolution (HR) videos from low-resolution (LR) and blurry inputs. Current BVSR methods often fail to restore sharp details at high resolutions, resulting in noticeable artifacts and jitter due to insufficient motion information for deconvolution and the lack of high-frequency details in LR frames. To address these challenges, we introduce event signals into BVSR and propose a novel event-enhanced network, Ev-DeblurVSR. To effectively fuse information from frames and events for feature deblurring, we introduce a reciprocal feature deblurring module that leverages motion information from intra-frame events to deblur frame features while reciprocally using global scene context from the frames to enhance event features. Furthermore, to enhance temporal consistency, we propose a hybrid deformable alignment module that fully exploits the complementary motion information from inter-frame events and optical flow to improve motion estimation in the deformable alignment process. Extensive evaluations demonstrate that Ev-DeblurVSR establishes a new state-of-the-art performance on both synthetic and real-world datasets. Notably, on real data, our method is +2.59 dB more accurate and 7.28$\times$ faster than the recent best BVSR baseline FMA-Net. Code: this https URL.
• 摘要：在本文中，我们解决了模糊视频超分辨率（BVSR）的任务，旨在从低分辨率（LR）和模糊输入中生成高分辨率（HR）视频。当前的BVSR方法通常无法在高分辨率上恢复尖锐的细节，从而导致明显的人工制品和抖动，这是由于运动信息不足，无法进行反向卷积，并且在LR框架中缺乏高频细节。为了应对这些挑战，我们将事件信号引入BVSR，并提出一个新颖的事件增强网络EV-DEBLURVSR。为了有效地融合来自特征Deblurring的框架和事件的信息，我们介绍了一个相互的功能DeBlurring模块，该模块利用了从框架内事件到DeBlur框架功能的运动信息，同时使用来自帧的全局场景上下文来增强事件功能。此外，为了提高时间一致性，我们提出了一个混合变形对齐模块，该模块完全利用互补运动信息从框架间事件和光学流量中，以改善可变形比对过程中的运动估计。广泛的评估表明，EV-DEBLURVSR在合成和现实世界数据集上建立了新的最新性能。值得注意的是，在实际数据上，我们的方法+2.59 dB更准确，比最近最佳BVSR基线FMA-NET快7.28 $ \ times $。代码：此HTTPS URL。

Title: Expert Kernel Generation Network Driven by Contextual Mapping for Hyperspectral Image Classification

• Authors: Guandong Li, Mengxia Ye
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.13045
• Pdf URL: https://arxiv.org/pdf/2504.13045
• Copy Paste: [[2504.13045]] Expert Kernel Generation Network Driven by Contextual Mapping for Hyperspectral Image Classification(https://arxiv.org/abs/2504.13045)
• Keywords: generation
• Abstract: Deep neural networks face several challenges in hyperspectral image classification, including high-dimensional data, sparse distribution of ground objects, and spectral redundancy, which often lead to classification overfitting and limited generalization capability. To more efficiently adapt to ground object distributions while extracting image features without introducing excessive parameters and skipping redundant information, this paper proposes EKGNet based on an improved 3D-DenseNet model, consisting of a context-aware mapping network and a dynamic kernel generation module. The context-aware mapping module translates global contextual information of hyperspectral inputs into instructions for combining base convolutional kernels, while the dynamic kernels are composed of K groups of base convolutions, analogous to K different types of experts specializing in fundamental patterns across various dimensions. The mapping module and dynamic kernel generation mechanism form a tightly coupled system - the former generates meaningful combination weights based on inputs, while the latter constructs an adaptive expert convolution system using these weights. This dynamic approach enables the model to focus more flexibly on key spatial structures when processing different regions, rather than relying on the fixed receptive field of a single static convolutional kernel. EKGNet enhances model representation capability through a 3D dynamic expert convolution system without increasing network depth or width. The proposed method demonstrates superior performance on IN, UP, and KSC datasets, outperforming mainstream hyperspectral image classification approaches.
• 摘要：深度神经网络在高光谱图像分类中面临几个挑战，包括高维数据，地面对象的稀疏分布和光谱冗余，这通常会导致分类过度拟合和有限的概括能力。为了更有效地适应地面对象分布，同时提取图像特征而不引入过多的参数并跳过冗余信息，本文提出了基于改进的3D-Densenet模型的EKGNET，该模型由上下文感知映射网络和动态内核生成模块组成。上下文感知的映射模块将高光谱输入的全局上下文信息转化为结合基础卷积内核的指令，而动态内核由k组组成的基本卷积组组成，类似于K的不同类型的专家，专门研究各个维度的基本模式。映射模块和动态内核生成机制形成了一个紧密耦合的系统 - 前者基于输入产生有意义的组合权重，而后者则使用这些权重构建了自适应专家卷积系统。这种动态方法使模型在处理不同区域时可以更灵活地专注于关键空间结构，而不是依靠单个静态卷积内核的固定接收场。 EKGNET通过3D动态专家卷积系统增强了模型表示能力，而不会增加网络深度或宽度。所提出的方法在IN，UP和KSC数据集上表现出了卓越的性能，优于主流高光谱图像分类方法。

Title: ArtistAuditor: Auditing Artist Style Pirate in Text-to-Image Generation Models

• Authors: Linkang Du, Zheng Zhu, Min Chen, Zhou Su, Shouling Ji, Peng Cheng, Jiming Chen, Zhikun Zhang
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.13061
• Pdf URL: https://arxiv.org/pdf/2504.13061
• Copy Paste: [[2504.13061]] ArtistAuditor: Auditing Artist Style Pirate in Text-to-Image Generation Models(https://arxiv.org/abs/2504.13061)
• Keywords: generation
• Abstract: Text-to-image models based on diffusion processes, such as DALL-E, Stable Diffusion, and Midjourney, are capable of transforming texts into detailed images and have widespread applications in art and design. As such, amateur users can easily imitate professional-level paintings by collecting an artist's work and fine-tuning the model, leading to concerns about artworks' copyright infringement. To tackle these issues, previous studies either add visually imperceptible perturbation to the artwork to change its underlying styles (perturbation-based methods) or embed post-training detectable watermarks in the artwork (watermark-based methods). However, when the artwork or the model has been published online, i.e., modification to the original artwork or model retraining is not feasible, these strategies might not be viable. To this end, we propose a novel method for data-use auditing in the text-to-image generation model. The general idea of ArtistAuditor is to identify if a suspicious model has been finetuned using the artworks of specific artists by analyzing the features related to the style. Concretely, ArtistAuditor employs a style extractor to obtain the multi-granularity style representations and treats artworks as samplings of an artist's style. Then, ArtistAuditor queries a trained discriminator to gain the auditing decisions. The experimental results on six combinations of models and datasets show that ArtistAuditor can achieve high AUC values (> 0.937). By studying ArtistAuditor's transferability and core modules, we provide valuable insights into the practical implementation. Finally, we demonstrate the effectiveness of ArtistAuditor in real-world cases by an online platform Scenario. ArtistAuditor is open-sourced at this https URL.
• 摘要：基于扩散过程的文本对图像模型，例如DALL-E，稳定的扩散和Midjourney，能够将文本转换为详细的图像，并在艺术和设计中具有广泛的应用。因此，业余用户可以通过收集艺术家的作品和微调模型来轻松模仿专业水平的绘画，从而导致对艺术品侵犯版权的担忧。为了解决这些问题，先前的研究要么在艺术品上增加了视觉上不可察觉的扰动，以改变其潜在的样式（基于扰动的方法），要么在艺术品中嵌入可检测的后水印（基于水印的方法）。但是，当艺术品或模型在线发布时，即对原始艺术品或模型再培训的修改是不可行的，这些策略可能是不可行的。为此，我们提出了一种新颖的方法，用于在文本到图像生成模型中进行数据使用审核。 Artistauditor的一般思想是通过分析与样式相关的功能来确定使用特定艺术家的艺术品是否对可疑模型进行了审核。具体而言，Artistauditor采用了一种样式的提取器来获得多层状风格的表现形式，并将艺术品视为艺术家风格的抽样。然后，艺术家审核者询问训练有素的歧视者以获得审计决定。六个模型和数据集组合的实验结果表明，AstriveAuditor可以实现高AUC值（> 0.937）。通过研究Artistauditor的可转让性和核心模块，我们为实施实施提供了宝贵的见解。最后，我们通过在线平台方案在现实世界中证明了Artistauditor在现实情况下的有效性。 Artistauditor在此HTTPS URL上开源。

Title: SkyReels-V2: Infinite-length Film Generative Model

• Authors: Guibin Chen, Dixuan Lin, Jiangping Yang, Chunze Lin, Juncheng Zhu, Mingyuan Fan, Hao Zhang, Sheng Chen, Zheng Chen, Chengchen Ma, Weiming Xiong, Wei Wang, Nuo Pang, Kang Kang, Zhiheng Xu, Yuzhe Jin, Yupeng Liang, Yubing Song, Peng Zhao, Boyuan Xu, Di Qiu, Debang Li, Zhengcong Fei, Yang Li, Yahui Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.13074
• Pdf URL: https://arxiv.org/pdf/2504.13074
• Copy Paste: [[2504.13074]] SkyReels-V2: Infinite-length Film Generative Model(https://arxiv.org/abs/2504.13074)
• Keywords: generation, generative
• Abstract: Recent advances in video generation have been driven by diffusion models and autoregressive frameworks, yet critical challenges persist in harmonizing prompt adherence, visual quality, motion dynamics, and duration: compromises in motion dynamics to enhance temporal visual quality, constrained video duration (5-10 seconds) to prioritize resolution, and inadequate shot-aware generation stemming from general-purpose MLLMs' inability to interpret cinematic grammar, such as shot composition, actor expressions, and camera motions. These intertwined limitations hinder realistic long-form synthesis and professional film-style generation. To address these limitations, we propose SkyReels-V2, an Infinite-length Film Generative Model, that synergizes Multi-modal Large Language Model (MLLM), Multi-stage Pretraining, Reinforcement Learning, and Diffusion Forcing Framework. Firstly, we design a comprehensive structural representation of video that combines the general descriptions by the Multi-modal LLM and the detailed shot language by sub-expert models. Aided with human annotation, we then train a unified Video Captioner, named SkyCaptioner-V1, to efficiently label the video data. Secondly, we establish progressive-resolution pretraining for the fundamental video generation, followed by a four-stage post-training enhancement: Initial concept-balanced Supervised Fine-Tuning (SFT) improves baseline quality; Motion-specific Reinforcement Learning (RL) training with human-annotated and synthetic distortion data addresses dynamic artifacts; Our diffusion forcing framework with non-decreasing noise schedules enables long-video synthesis in an efficient search space; Final high-quality SFT refines visual fidelity. All the code and models are available at this https URL.
• 摘要：Recent advances in video generation have been driven by diffusion models and autoregressive frameworks, yet critical challenges persist in harmonizing prompt adherence, visual quality, motion dynamics, and duration: compromises in motion dynamics to enhance temporal visual quality, constrained video duration (5-10 seconds) to prioritize resolution, and inadequate shot-aware generation stemming from general-purpose MLLMs' inability to interpret cinematic grammar, such作为射击作品，演员表达和摄像头动作。这些相互交织的局限性阻碍了现实的长格式合成和专业电影风格的一代。为了解决这些局限性，我们建议Skyreels-V2是一种无限长度的膜生成模型，该模型协同多模式大型语言模型（MLLM），多阶段预处理，加强学习和扩散强迫框架。首先，我们设计了视频的全面结构表示，该结构结合了多模式LLM的一般描述和Sub-Expert模型的详细镜头语言。然后，我们在人类注释的帮助下，然后训练名为SkyCaptioner-V1的统一视频字幕仪，以有效地标记视频数据。其次，我们为基本视频生成建立了渐进的预测，然后进行四阶段的训练后增强：初始概念均衡的监督微调（SFT）提高了基线质量；通过人类注销和合成失真数据的运动特异性增强学习（RL）培训涉及动态伪像；我们使用非降低噪声时间表的扩散强迫框架可以在有效的搜索空间中进行长时间的合成。最终高质量的SFT优化了视觉保真度。所有代码和模型均可在此HTTPS URL上找到。

Title: An All-Atom Generative Model for Designing Protein Complexes

• Authors: Ruizhe Chen, Dongyu Xue, Xiangxin Zhou, Zaixiang Zheng, Xiangxiang Zeng, Quanquan Gu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.13075
• Pdf URL: https://arxiv.org/pdf/2504.13075
• Copy Paste: [[2504.13075]] An All-Atom Generative Model for Designing Protein Complexes(https://arxiv.org/abs/2504.13075)
• Keywords: generative
• Abstract: Proteins typically exist in complexes, interacting with other proteins or biomolecules to perform their specific biological roles. Research on single-chain protein modeling has been extensively and deeply explored, with advancements seen in models like the series of ESM and AlphaFold. Despite these developments, the study and modeling of multi-chain proteins remain largely uncharted, though they are vital for understanding biological functions. Recognizing the importance of these interactions, we introduce APM (All-Atom Protein Generative Model), a model specifically designed for modeling multi-chain proteins. By integrating atom-level information and leveraging data on multi-chain proteins, APM is capable of precisely modeling inter-chain interactions and designing protein complexes with binding capabilities from scratch. It also performs folding and inverse-folding tasks for multi-chain proteins. Moreover, APM demonstrates versatility in downstream applications: it achieves enhanced performance through supervised fine-tuning (SFT) while also supporting zero-shot sampling in certain tasks, achieving state-of-the-art results. Code will be released at this https URL.
• 摘要：蛋白质通常存在于复合物中，与其他蛋白质或生物分子相互作用以执行其特定的生物学作用。对单链蛋白质建模的研究已经进行了广泛而深入的探索，在ESM和Alphafold系列之类的模型中看到了进步。尽管有这些发展，但多链蛋白的研究和建模仍在很大程度上未知，尽管它们对于理解生物学功能至关重要。认识到这些相互作用的重要性，我们引入了APM（全原子蛋白生成模型），该模型专为建模多链蛋白而设计。通过整合原子级信息并利用多链蛋白的数据，APM能够精确地对链间相互作用进行建模，并从头开始设计具有结合功能的蛋白质复合物。它还针对多链蛋白执行折叠和反折叠任务。此外，APM在下游应用程序中展示了多功能性：它通过监督的微调（SFT）实现了增强的性能，同时还支持某些任务中的零拍采样，从而实现了最新的结果。代码将在此HTTPS URL上发布。

Title: UniEdit-Flow: Unleashing Inversion and Editing in the Era of Flow Models

• Authors: Guanlong Jiao, Biqing Huang, Kuan-Chieh Wang, Renjie Liao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.13109
• Pdf URL: https://arxiv.org/pdf/2504.13109
• Copy Paste: [[2504.13109]] UniEdit-Flow: Unleashing Inversion and Editing in the Era of Flow Models(https://arxiv.org/abs/2504.13109)
• Keywords: generative
• Abstract: Flow matching models have emerged as a strong alternative to diffusion models, but existing inversion and editing methods designed for diffusion are often ineffective or inapplicable to them. The straight-line, non-crossing trajectories of flow models pose challenges for diffusion-based approaches but also open avenues for novel solutions. In this paper, we introduce a predictor-corrector-based framework for inversion and editing in flow models. First, we propose Uni-Inv, an effective inversion method designed for accurate reconstruction. Building on this, we extend the concept of delayed injection to flow models and introduce Uni-Edit, a region-aware, robust image editing approach. Our methodology is tuning-free, model-agnostic, efficient, and effective, enabling diverse edits while ensuring strong preservation of edit-irrelevant regions. Extensive experiments across various generative models demonstrate the superiority and generalizability of Uni-Inv and Uni-Edit, even under low-cost settings. Project page: this https URL
• 摘要：流量匹配模型已成为扩散模型的强大替代方法，但是现有的反转和编辑方法通常是无效或不适用的。流模型的直线，非交叉轨迹对基于扩散的方法提出了挑战，也为新颖的解决方案开放了途径。在本文中，我们引入了基于预测的基于 - 校正器的框架，用于倒流模型中的反演和编辑。首先，我们提出Uni-Inv，这是一种有效的反转方法，旨在精确重建。在此基础上，我们将延迟注入的概念扩展到流模型，并引入Uni-Edit，这是一种区域感知，可靠的图像编辑方法。我们的方法是无调，不合时宜的，高效和有效的，可实现多样化的编辑，同时确保对编辑的区域进行大力保存。各种生成模型的广泛实验也证明了即使在低成本设置下，Uni-Inv和Uni-Edit的优越性和普遍性也是如此。项目页面：此HTTPS URL

Title: Uncertainty-Aware Trajectory Prediction via Rule-Regularized Heteroscedastic Deep Classification

• Authors: Kumar Manas, Christian Schlauch, Adrian Paschke, Christian Wirth, Nadja Klein
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2504.13111
• Pdf URL: https://arxiv.org/pdf/2504.13111
• Copy Paste: [[2504.13111]] Uncertainty-Aware Trajectory Prediction via Rule-Regularized Heteroscedastic Deep Classification(https://arxiv.org/abs/2504.13111)
• Keywords: generation
• Abstract: Deep learning-based trajectory prediction models have demonstrated promising capabilities in capturing complex interactions. However, their out-of-distribution generalization remains a significant challenge, particularly due to unbalanced data and a lack of enough data and diversity to ensure robustness and calibration. To address this, we propose SHIFT (Spectral Heteroscedastic Informed Forecasting for Trajectories), a novel framework that uniquely combines well-calibrated uncertainty modeling with informative priors derived through automated rule extraction. SHIFT reformulates trajectory prediction as a classification task and employs heteroscedastic spectral-normalized Gaussian processes to effectively disentangle epistemic and aleatoric uncertainties. We learn informative priors from training labels, which are automatically generated from natural language driving rules, such as stop rules and drivability constraints, using a retrieval-augmented generation framework powered by a large language model. Extensive evaluations over the nuScenes dataset, including challenging low-data and cross-location scenarios, demonstrate that SHIFT outperforms state-of-the-art methods, achieving substantial gains in uncertainty calibration and displacement metrics. In particular, our model excels in complex scenarios, such as intersections, where uncertainty is inherently higher. Project page: this https URL.
• 摘要：基于深度学习的轨迹预测模型已经证明了捕获复杂相互作用的有希望的能力。但是，它们的分布概括仍然是一个重大挑战，尤其是由于数据不平衡以及缺乏足够的数据和多样性来确保鲁棒性和校准。为了解决这个问题，我们提出了转移（光谱异形的知情轨迹预测），这是一个新颖的框架，将唯一的框架结合了良好的不确定性建模与通过自动化规则提取的信息提供的知识分子。 Shift将轨迹预测重新定义为一项分类任务，并采用异性光谱范围化的高斯过程，以有效地消除认知和核心不确定性。我们使用由大型语言模型驱动的检索型生成框架从自然语言驱动规则（例如停止规则和可驱动性约束）中自动生成的培训标签中，这些先验会自动生成。对Nuscenes数据集的广泛评估，包括挑战低数据和交叉点方案，表明，Shings Shings的表现优于最先进的方法，从而在不确定性校准和流离失所度量方面取得了可观的增长。特别是，我们的模型在复杂的场景（例如交叉点）中擅长，其中不确定性本质上更高。项目页面：此HTTPS URL。

Title: Probing and Inducing Combinational Creativity in Vision-Language Models

• Authors: Yongqian Peng, Yuxi Ma, Mengmeng Wang, Yuxuan Wang, Yizhou Wang, Chi Zhang, Yixin Zhu, Zilong Zheng
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2504.13120
• Pdf URL: https://arxiv.org/pdf/2504.13120
• Copy Paste: [[2504.13120]] Probing and Inducing Combinational Creativity in Vision-Language Models(https://arxiv.org/abs/2504.13120)
• Keywords: generation
• Abstract: The ability to combine existing concepts into novel ideas stands as a fundamental hallmark of human intelligence. Recent advances in Vision-Language Models (VLMs) like GPT-4V and DALLE-3 have sparked debate about whether their outputs reflect combinational creativity--defined by M. A. Boden (1998) as synthesizing novel ideas through combining existing concepts--or sophisticated pattern matching of training data. Drawing inspiration from cognitive science, we investigate the combinational creativity of VLMs from the lens of concept blending. We propose the Identification-Explanation-Implication (IEI) framework, which decomposes creative processes into three levels: identifying input spaces, extracting shared attributes, and deriving novel semantic implications. To validate this framework, we curate CreativeMashup, a high-quality dataset of 666 artist-generated visual mashups annotated according to the IEI framework. Through extensive experiments, we demonstrate that in comprehension tasks, best VLMs have surpassed average human performance while falling short of expert-level understanding; in generation tasks, incorporating our IEI framework into the generation pipeline significantly enhances the creative quality of VLMs outputs. Our findings establish both a theoretical foundation for evaluating artificial creativity and practical guidelines for improving creative generation in VLMs.
• 摘要：将现有概念结合到新思想的能力是人类智力的基本标志。诸如GPT-4V和Dalle-3之类的视觉模型（VLM）的最新进展引发了有关它们的输出是否反映了M. A. Boden（1998）定义的组合创造力的争论，它是通过结合现有概念的新想法来综合培训数据的综合思想。从认知科学中汲取灵感，我们研究了VLM的概念融合镜头的组合创造力。我们提出了标识解释性（IEI）框架，该框架将创作过程分解为三个级别：识别输入空间，提取共享属性并获得新颖的语义含义。为了验证该框架，我们策划了CreativeMashup，这是根据IEI框架注释的666个艺术家生成的视觉混搭的高质量数据集。通过广泛的实验，我们证明，在理解任务中，最好的VLM超过了人类的平均绩效，而没有专家级别的理解。在一代任务中，将我们的IEI框架纳入生成管道可显着提高VLMS输出的创作质量。我们的发现既建立了评估人工创造力的理论基础，也建立了改善VLM中创造性产生的实用准则。

Title: Low-hallucination Synthetic Captions for Large-Scale Vision-Language Model Pre-training

• Authors: Xinsong Zhang, Yarong Zeng, Xinting Huang, Hu Hu, Runquan Xie, Han Hu, Zhanhui Kang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.13123
• Pdf URL: https://arxiv.org/pdf/2504.13123
• Copy Paste: [[2504.13123]] Low-hallucination Synthetic Captions for Large-Scale Vision-Language Model Pre-training(https://arxiv.org/abs/2504.13123)
• Keywords: generation
• Abstract: In recent years, the field of vision-language model pre-training has experienced rapid advancements, driven primarily by the continuous enhancement of textual capabilities in large language models. However, existing training paradigms for multimodal large language models heavily rely on high-quality image-text pairs. As models and data scales grow exponentially, the availability of such meticulously curated data has become increasingly scarce and saturated, thereby severely limiting further advancements in this domain. This study investigates scalable caption generation techniques for vision-language model pre-training and demonstrates that large-scale low-hallucination synthetic captions can serve dual purposes: 1) acting as a viable alternative to real-world data for pre-training paradigms and 2) achieving superior performance enhancement when integrated into vision-language models through empirical validation. This paper presents three key contributions: 1) a novel pipeline for generating high-quality, low-hallucination, and knowledge-rich synthetic captions. Our continuous DPO methodology yields remarkable results in reducing hallucinations. Specifically, the non-hallucination caption rate on a held-out test set increases from 48.2% to 77.9% for a 7B-size model. 2) Comprehensive empirical validation reveals that our synthetic captions confer superior pre-training advantages over their counterparts. Across 35 vision language tasks, the model trained with our data achieves a significant performance gain of at least 6.2% compared to alt-text pairs and other previous work. Meanwhile, it also offers considerable support in the text-to-image domain. With our dataset, the FID score is reduced by 17.1 on a real-world validation benchmark and 13.3 on the MSCOCO validation benchmark. 3) We will release Hunyuan-Recap100M, a low-hallucination and knowledge-intensive synthetic caption dataset.
• 摘要：近年来，视觉模型预训练领域已经经历了快速的进步，这主要是由于大语模型中文本能力的持续增强而驱动。但是，多模式大语言模型的现有训练范例在很大程度上依赖于高质量的图像文本对。随着模型和数据量表呈指数增长，这种精心策划的数据的可用性变得越来越稀缺和饱和，从而严重限制了该域中的进一步进步。这项研究研究了可扩展的字幕生成技术，用于视觉模型预训练，并证明大规模的低凝结合成字幕可以有双重目的：1）作为预训练范式的现实世界数据的可行替代方案，用于通过Empusical模型整合到视觉语言模型中，可实现较高的训练范式和2）实现卓越的性能。本文提出了三个关键的贡献：1）一种新的管道，用于产生高质量，低障碍和知识丰富的合成标题。我们的连续DPO方法在减少幻觉方面产生了显着的结果。具体而言，对于7B尺寸的模型，持有测试集的非障碍标题率从48.2％增加到77.9％。 2）全面的经验验证表明，我们的合成字幕赋予其优于对应物的优质培训优势。在35个视觉语言任务中，与Alt-Text对和其他先前的工作相比，接受我们数据训练的模型至少达到6.2％。同时，它还在文本到图像域中提供了相当大的支持。使用我们的数据集，在现实世界验证基准测试基准上，FID分数在17.1中减少了17.1，而在MSCOCO验证基准测试中，FID分数将减少13.3。 3）我们将发布Hunyuan-Recap100m，这是一个低障碍和知识密集的合成标题数据集。

Title: Science-T2I: Addressing Scientific Illusions in Image Synthesis

• Authors: Jialuo Li, Wenhao Chai, Xingyu Fu, Haiyang Xu, Saining Xie
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.13129
• Pdf URL: https://arxiv.org/pdf/2504.13129
• Copy Paste: [[2504.13129]] Science-T2I: Addressing Scientific Illusions in Image Synthesis(https://arxiv.org/abs/2504.13129)
• Keywords: generative
• Abstract: We present a novel approach to integrating scientific knowledge into generative models, enhancing their realism and consistency in image synthesis. First, we introduce Science-T2I, an expert-annotated adversarial dataset comprising adversarial 20k image pairs with 9k prompts, covering wide distinct scientific knowledge categories. Leveraging Science-T2I, we present SciScore, an end-to-end reward model that refines the assessment of generated images based on scientific knowledge, which is achieved by augmenting both the scientific comprehension and visual capabilities of pre-trained CLIP model. Additionally, based on SciScore, we propose a two-stage training framework, comprising a supervised fine-tuning phase and a masked online fine-tuning phase, to incorporate scientific knowledge into existing generative models. Through comprehensive experiments, we demonstrate the effectiveness of our framework in establishing new standards for evaluating the scientific realism of generated content. Specifically, SciScore attains performance comparable to human-level, demonstrating a 5% improvement similar to evaluations conducted by experienced human evaluators. Furthermore, by applying our proposed fine-tuning method to FLUX, we achieve a performance enhancement exceeding 50% on SciScore.
• 摘要：我们提出了一种新颖的方法，可以将科学知识整合到生成模型中，增强其现实主义和图像合成中的一致性。首先，我们介绍了Science-T2I，这是一个由专家注册的对抗数据集，其中包括带有9K提示的对抗性20K图像对，涵盖了广泛的不同科学知识类别。利用科学-T2I，我们提出了Sciscore，这是一个端到端奖励模型，它根据科学知识来完善对产生的图像的评估，这是通过增强预训练剪辑模型的科学理解和视觉能力来实现的。此外，基于Sciscore，我们提出了一个两阶段的培训框架，包括监督的微调阶段和一个掩盖的在线微调阶段，以将科学知识纳入现有的生成模型中。通过全面的实验，我们证明了框架在建立新标准以评估生成内容的科学现实主义方面的有效性。具体而言，Sciscore的表现与人类水平相当，证明了5％的进步与经验丰富的人类评估者进行的评估相似。此外，通过将我们提出的微调方法应用于通量，我们的性能提高超过了Sciscore的50％。

Title: $\texttt{Complex-Edit}$: CoT-Like Instruction Generation for Complexity-Controllable Image Editing Benchmark

• Authors: Siwei Yang, Mude Hui, Bingchen Zhao, Yuyin Zhou, Nataniel Ruiz, Cihang Xie
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.13143
• Pdf URL: https://arxiv.org/pdf/2504.13143
• Copy Paste: [[2504.13143]] $\texttt{Complex-Edit}$: CoT-Like Instruction Generation for Complexity-Controllable Image Editing Benchmark(https://arxiv.org/abs/2504.13143)
• Keywords: generation
• Abstract: We introduce $\texttt{Complex-Edit}$, a comprehensive benchmark designed to systematically evaluate instruction-based image editing models across instructions of varying complexity. To develop this benchmark, we harness GPT-4o to automatically collect a diverse set of editing instructions at scale. Our approach follows a well-structured ``Chain-of-Edit'' pipeline: we first generate individual atomic editing tasks independently and then integrate them to form cohesive, complex instructions. Additionally, we introduce a suite of metrics to assess various aspects of editing performance, along with a VLM-based auto-evaluation pipeline that supports large-scale assessments. Our benchmark yields several notable insights: 1) Open-source models significantly underperform relative to proprietary, closed-source models, with the performance gap widening as instruction complexity increases; 2) Increased instructional complexity primarily impairs the models' ability to retain key elements from the input images and to preserve the overall aesthetic quality; 3) Decomposing a complex instruction into a sequence of atomic steps, executed in a step-by-step manner, substantially degrades performance across multiple metrics; 4) A straightforward Best-of-N selection strategy improves results for both direct editing and the step-by-step sequential approach; and 5) We observe a ``curse of synthetic data'': when synthetic data is involved in model training, the edited images from such models tend to appear increasingly synthetic as the complexity of the editing instructions rises -- a phenomenon that intriguingly also manifests in the latest GPT-4o outputs.
• 摘要：我们介绍了$ \ texttt {complex-edit} $，这是一种综合基准，旨在系统地评估基于教学的图像编辑模型，跨不同复杂性的指令。为了开发此基准，我们利用GPT-4O自动按大规模收集各种编辑说明。我们的方法遵循结构良好的``编辑''管道：我们首先独立生成单独的原子编辑任务，然后整合它们以形成凝聚力，复杂的说明。此外，我们引入了一套指标，以评估编辑性能的各个方面，以及支持大规模评估的基于VLM的自动评估管道。我们的基准测试产生了几个值得注意的见解：1）相对于专有的封闭源模型，开源模型的表现明显不佳，并且随着指导复杂性的增加，性能差距的扩大； 2）提高教学复杂性主要会损害模型从输入图像中保留关键要素并保留整体美学质量的能力； 3）将复杂的指令分解为一系列原子步骤，以分步的方式执行，从而在多个指标上实质上降低了性能； 4）直接的最佳N选择策略可改善直接编辑和逐步顺序方法的结果； 5）我们观察到``综合数据的诅咒''：当合成数据参与模型训练时，此类模型的编辑图像往往越来越综合，因为编辑指令的复杂性升高 - 这种现象在最新的GPT-4O输出中也很有趣。

Title: Digital Twin Generation from Visual Data: A Survey

• Authors: Andrew Melnik, Benjamin Alt, Giang Nguyen, Artur Wilkowski, Maciej Stefańczyk, Qirui Wu, Sinan Harms, Helge Rhodin, Manolis Savva, Michael Beetz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.13159
• Pdf URL: https://arxiv.org/pdf/2504.13159
• Copy Paste: [[2504.13159]] Digital Twin Generation from Visual Data: A Survey(https://arxiv.org/abs/2504.13159)
• Keywords: generation, generative
• Abstract: This survey explores recent developments in generating digital twins from videos. Such digital twins can be used for robotics application, media content creation, or design and construction works. We analyze various approaches, including 3D Gaussian Splatting, generative in-painting, semantic segmentation, and foundation models highlighting their advantages and limitations. Additionally, we discuss challenges such as occlusions, lighting variations, and scalability, as well as potential future research directions. This survey aims to provide a comprehensive overview of state-of-the-art methodologies and their implications for real-world applications. Awesome list: this https URL
• 摘要：这项调查探讨了从视频产生数字双胞胎的最新发展。这种数字双胞胎可用于机器人技术应用程序，媒体内容创建或设计和构建工作。我们分析了各种方法，包括3D高斯分裂，生成性涂漆，语义分割和基础模型突出了它们的优势和局限性。此外，我们讨论了诸如遮挡，照明变化和可扩展性以及潜在的未来研究方向等挑战。这项调查旨在对最先进的方法及其对现实应用程序的影响进行全面概述。很棒的列表：此HTTPS URL

Title: Personalized Text-to-Image Generation with Auto-Regressive Models

• Authors: Kaiyue Sun, Xian Liu, Yao Teng, Xihui Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.13162
• Pdf URL: https://arxiv.org/pdf/2504.13162
• Copy Paste: [[2504.13162]] Personalized Text-to-Image Generation with Auto-Regressive Models(https://arxiv.org/abs/2504.13162)
• Keywords: generation
• Abstract: Personalized image synthesis has emerged as a pivotal application in text-to-image generation, enabling the creation of images featuring specific subjects in diverse contexts. While diffusion models have dominated this domain, auto-regressive models, with their unified architecture for text and image modeling, remain underexplored for personalized image generation. This paper investigates the potential of optimizing auto-regressive models for personalized image synthesis, leveraging their inherent multimodal capabilities to perform this task. We propose a two-stage training strategy that combines optimization of text embeddings and fine-tuning of transformer layers. Our experiments on the auto-regressive model demonstrate that this method achieves comparable subject fidelity and prompt following to the leading diffusion-based personalization methods. The results highlight the effectiveness of auto-regressive models in personalized image generation, offering a new direction for future research in this area.
• 摘要：个性化的图像合成已成为文本到图像生成中的关键应用，从而可以创建具有不同环境中特定主题的图像。尽管扩散模型已经统治了这个领域，但自动回归模型及其统一的文本和图像建筑架构仍未逐渐置于个性化图像生成。本文研究了优化自动回归模型的个性化图像合成的潜力，利用其固有的多模式能力来执行此任务。我们提出了一种两阶段的训练策略，该策略结合了文本嵌入的优化和变压器层的微调。我们在自动回归模型上的实验表明，该方法具有可比的主题保真度，并迅速遵循了基于领先的扩散的个性化方法。结果突出了自动退缩模型在个性化图像生成中的有效性，为该领域的未来研究提供了新的方向。

Title: Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling

• Authors: Tsung-Han Wu, Heekyung Lee, Jiaxin Ge, Joseph E. Gonzalez, Trevor Darrell, David M. Chan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.13169
• Pdf URL: https://arxiv.org/pdf/2504.13169
• Copy Paste: [[2504.13169]] Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling(https://arxiv.org/abs/2504.13169)
• Keywords: generation
• Abstract: Vision-Language Models (VLMs) excel at visual understanding but often suffer from visual hallucinations, where they generate descriptions of nonexistent objects, actions, or concepts, posing significant risks in safety-critical applications. Existing hallucination mitigation methods typically follow one of two paradigms: generation adjustment, which modifies decoding behavior to align text with visual inputs, and post-hoc verification, where external models assess and correct outputs. While effective, generation adjustment methods often rely on heuristics and lack correction mechanisms, while post-hoc verification is complicated, typically requiring multiple models and tending to reject outputs rather than refine them. In this work, we introduce REVERSE, a unified framework that integrates hallucination-aware training with on-the-fly self-verification. By leveraging a new hallucination-verification dataset containing over 1.3M semi-synthetic samples, along with a novel inference-time retrospective resampling technique, our approach enables VLMs to both detect hallucinations during generation and dynamically revise those hallucinations. Our evaluations show that REVERSE achieves state-of-the-art hallucination reduction, outperforming the best existing methods by up to 12% on CHAIR-MSCOCO and 28% on HaloQuest. Our dataset, model, and code are available at: this https URL.
• 摘要：视觉模型（VLMS）在视觉理解方面表现出色，但经常患有视觉幻觉，它们会在其中描述不存在的对象，动作或概念，并在安全至关重要的应用中带来重大风险。现有的缓解方法通常遵循两个范式之一：生成调整，它修改了解码行为以使文本与视觉输入和事后验证相结合，外部模型评估和正确的输出。虽然有效，但生成调整方法通常依赖于启发式方法和缺乏校正机制，而事后验证却很复杂，通常需要多种模型，并且倾向于拒绝输出而不是完善它们。在这项工作中，我们介绍了反向，这是一个统一的框架，将幻觉感知的培训与自动验证相结合。通过利用一个新的幻觉验证数据集，该数据集包含超过130万半合成样品，以及一种新颖的推理时间回顾性重新采样技术，我们的方法使VLMS可以在发电期间检测幻觉并动态修改这些幻觉。我们的评估表明，反向可实现最新的幻觉减少，在椅子上，最佳现有方法的表现高达12％，而haloquest的方法则优于28％。我们的数据集，模型和代码可在以下网址提供：此HTTPS URL。

Title: IMAGGarment-1: Fine-Grained Garment Generation for Controllable Fashion Design

• Authors: Fei Shen, Jian Yu, Cong Wang, Xin Jiang, Xiaoyu Du, Jinhui Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.13176
• Pdf URL: https://arxiv.org/pdf/2504.13176
• Copy Paste: [[2504.13176]] IMAGGarment-1: Fine-Grained Garment Generation for Controllable Fashion Design(https://arxiv.org/abs/2504.13176)
• Keywords: generation
• Abstract: This paper presents IMAGGarment-1, a fine-grained garment generation (FGG) framework that enables high-fidelity garment synthesis with precise control over silhouette, color, and logo placement. Unlike existing methods that are limited to single-condition inputs, IMAGGarment-1 addresses the challenges of multi-conditional controllability in personalized fashion design and digital apparel applications. Specifically, IMAGGarment-1 employs a two-stage training strategy to separately model global appearance and local details, while enabling unified and controllable generation through end-to-end inference. In the first stage, we propose a global appearance model that jointly encodes silhouette and color using a mixed attention module and a color adapter. In the second stage, we present a local enhancement model with an adaptive appearance-aware module to inject user-defined logos and spatial constraints, enabling accurate placement and visual consistency. To support this task, we release GarmentBench, a large-scale dataset comprising over 180K garment samples paired with multi-level design conditions, including sketches, color references, logo placements, and textual prompts. Extensive experiments demonstrate that our method outperforms existing baselines, achieving superior structural stability, color fidelity, and local controllability performance. The code and model are available at this https URL.
• 摘要：本文介绍了Imaggarment-1，这是一种精细的服装（FGG）框架，可实现高保真服装的合成，并精确控制轮廓，颜色和徽标放置。与仅限于单条件输入的现有方法不同，Imaggarment-1解决了个性化时尚设计和数字服装应用中多条件可控性的挑战。具体而言，Imaggarment-1采用两阶段的训练策略来分别建模全球外观和本地细节，同时通过端到端推断实现统一和可控制的生成。在第一阶段，我们提出了一个全球外观模型，该模型使用混合注意模块和颜色适配器共同编码轮廓和颜色。在第二阶段，我们提出了一个局部增强模型，该模型具有自适应外观感知的模块，以注入用户定义的徽标和空间约束，从而实现了准确的放置和视觉一致性。为了支持此任务，我们发布了GarmentBench，这是一个大规模数据集，其中包括超过180k的服装样品，并配对多层设计条件，包括草图，颜色参考，徽标放置和文本提示。广泛的实验表明，我们的方法表现优于现有基准，实现了卓越的结构稳定性，颜色保真度和局部可控性能。该代码和模型可在此HTTPS URL上找到。

Title: Aligning Constraint Generation with Design Intent in Parametric CAD

• Authors: Evan Casey, Tianyu Zhang, Shu Ishida, John Roger Thompson, Amir Khasahmadi, Joseph George Lambourne, Pradeep Kumar Jayaraman, Karl D.D. Willis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.13178
• Pdf URL: https://arxiv.org/pdf/2504.13178
• Copy Paste: [[2504.13178]] Aligning Constraint Generation with Design Intent in Parametric CAD(https://arxiv.org/abs/2504.13178)
• Keywords: generation, generative
• Abstract: We adapt alignment techniques from reasoning LLMs to the task of generating engineering sketch constraints found in computer-aided design (CAD) models. Engineering sketches consist of geometric primitives (e.g. points, lines) connected by constraints (e.g. perpendicular, tangent) that define the relationships between them. For a design to be easily editable, the constraints must effectively capture design intent, ensuring the geometry updates predictably when parameters change. Although current approaches can generate CAD designs, an open challenge remains to align model outputs with design intent, we label this problem `design alignment'. A critical first step towards aligning generative CAD models is to generate constraints which fully-constrain all geometric primitives, without over-constraining or distorting sketch geometry. Using alignment techniques to train an existing constraint generation model with feedback from a constraint solver, we are able to fully-constrain 93% of sketches compared to 34% when using a naïve supervised fine-tuning (SFT) baseline and only 8.9% without alignment. Our approach can be applied to any existing constraint generation model and sets the stage for further research bridging alignment strategies between the language and design domains.
• 摘要：我们将对齐技术从推理LLMS调整到计算机辅助设计（CAD）模型中生成工程草图约束的任务。工程草图由通过约束（例如垂直，切线）连接的几何原语（例如，点，线）组成，它们定义了它们之间的关系。为了使设计易于编辑，约束必须有效地捕获设计意图，以确保当参数发生变化时可预测的几何更新。尽管当前的方法可以生成CAD设计，但仍有一个开放的挑战仍然使模型输出与设计意图保持一致，我们将此问题标记为“设计对齐”。朝着对齐生成CAD模型的关键第一步是生成约束，这些约束完全构成了所有几何基原始人，而没有过度限制或扭曲草图几何形状。使用对准技术通过约束求解器的反馈来训练现有的约束生成模型，我们能够完全构成93％的草图，而使用幼稚的监督微调（SFT）基线，而只有8.9％的情况下，我们能够完全构成草图。我们的方法可以应用于任何现有的约束生成模型，并为语言和设计域之间的进一步研究桥接对齐策略奠定了基础。