2025-04-04

Title: Random Conditioning with Distillation for Data-Efficient Diffusion Model Compression

• Authors: Dohyun Kim, Sehwan Park, Geonhee Han, Seung Wook Kim, Paul Hongsuck Seo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.02011
• Pdf URL: https://arxiv.org/pdf/2504.02011
• Copy Paste: [[2504.02011]] Random Conditioning with Distillation for Data-Efficient Diffusion Model Compression(https://arxiv.org/abs/2504.02011)
• Keywords: generation, generative
• Abstract: Diffusion models generate high-quality images through progressive denoising but are computationally intensive due to large model sizes and repeated sampling. Knowledge distillation, which transfers knowledge from a complex teacher to a simpler student model, has been widely studied in recognition tasks, particularly for transferring concepts unseen during student training. However, its application to diffusion models remains underexplored, especially in enabling student models to generate concepts not covered by the training images. In this work, we propose Random Conditioning, a novel approach that pairs noised images with randomly selected text conditions to enable efficient, image-free knowledge distillation. By leveraging this technique, we show that the student can generate concepts unseen in the training images. When applied to conditional diffusion model distillation, our method allows the student to explore the condition space without generating condition-specific images, resulting in notable improvements in both generation quality and efficiency. This promotes resource-efficient deployment of generative diffusion models, broadening their accessibility for both research and real-world applications. Code, models, and datasets are available at this https URL .
• 摘要：扩散模型通过渐进式降解产生高质量的图像，但由于模型尺寸和重复采样而在计算上是计算密集的。知识蒸馏将知识从复杂的教师转移到更简单的学生模型，已被广泛研究以识别任务，尤其是在学生培训期间不见了的概念。但是，它在扩散模型中的应用仍未得到充满激光，尤其是在使学生模型能够生成训练图像不涵盖的概念时。在这项工作中，我们提出了随机调节，这是一种新型方法，该方法与随机选择的文本条件配对噪声图像，以实现有效的，无图像的知识蒸馏。通过利用这项技术，我们表明学生可以在培训图像中产生看不见的概念。当应用于条件扩散模型蒸馏时，我们的方法允许学生探索条件空间而无需产生特定条件的图像，从而显着提高了发电质量和效率。这促进了生成扩散模型的资源有效部署，从而扩大了其对研究和现实世界应用的可访问性。代码，模型和数据集可在此HTTPS URL上找到。

Title: Instruction-Guided Autoregressive Neural Network Parameter Generation

• Authors: Soro Bedionita, Bruno Andreis, Song Chong, Sung Ju Hwang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.02012
• Pdf URL: https://arxiv.org/pdf/2504.02012
• Copy Paste: [[2504.02012]] Instruction-Guided Autoregressive Neural Network Parameter Generation(https://arxiv.org/abs/2504.02012)
• Keywords: generation, generative
• Abstract: Learning to generate neural network parameters conditioned on task descriptions and architecture specifications is pivotal for advancing model adaptability and transfer learning. Existing methods especially those based on diffusion models suffer from limited scalability to large architectures, rigidity in handling varying network depths, and disjointed parameter generation that undermines inter-layer coherence. In this work, we propose IGPG (Instruction Guided Parameter Generation), an autoregressive framework that unifies parameter synthesis across diverse tasks and architectures. IGPG leverages a VQ-VAE and an autoregressive model to generate neural network parameters, conditioned on task instructions, dataset, and architecture details. By autoregressively generating neural network weights' tokens, IGPG ensures inter-layer coherence and enables efficient adaptation across models and datasets. Operating at the token level, IGPG effectively captures complex parameter distributions aggregated from a broad spectrum of pretrained models. Extensive experiments on multiple vision datasets demonstrate that IGPG consolidates diverse pretrained models into a single, flexible generative framework. The synthesized parameters achieve competitive or superior performance relative to state-of-the-art methods, especially in terms of scalability and efficiency when applied to large architectures. These results underscore ICPG potential as a powerful tool for pretrained weight retrieval, model selection, and rapid task-specific fine-tuning.
• 摘要：学习以任务描述和体系结构规范来生成神经网络参数是推进模型适应性和转移学习的关键。现有的方法尤其是基于扩散模型的方法，对大型架构的可扩展性有限，处理不同的网络深度的刚度以及破坏了层间相干性的分离参数生成。在这项工作中，我们提出了IGPG（指导引导参数生成），这是一个自动回归框架，统一跨不同任务和体系结构的参数合成。 IGPG利用VQ-VAE和自回归模型生成神经网络参数，以任务指令，数据集和体系结构详细信息为条件。通过自动加工生成神经网络权重的令牌，IGPG确保了层间相干性并在模型和数据集之间实现有效的适应性。 IGPG在代币级别运行，有效地捕获了从广泛的预验证模型汇总的复杂参数分布。在多个视觉数据集上进行的广泛实验表明，IGPG将各种预处理的模型合并为一个灵活的生成框架。合成的参数相对于最先进的方法具有竞争性或卓越的性能，尤其是在应用于大型体系结构时的可扩展性和效率方面。这些结果强调了ICPG的潜力作为预审预检测，模型选择和快速特定于任务的微调的强大工具。

Title: PolyG: Effective and Efficient GraphRAG with Adaptive Graph Traversal

• Authors: Renjie Liu, Haitian Jiang, Xiao Yan, Bo Tang, Jinyang Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.02112
• Pdf URL: https://arxiv.org/pdf/2504.02112
• Copy Paste: [[2504.02112]] PolyG: Effective and Efficient GraphRAG with Adaptive Graph Traversal(https://arxiv.org/abs/2504.02112)
• Keywords: generation
• Abstract: GraphRAG enhances large language models (LLMs) to generate quality answers for user questions by retrieving related facts from external knowledge graphs. Existing GraphRAG methods adopt a fixed graph traversal strategy for fact retrieval but we observe that user questions come in different types and require different graph traversal strategies. As such, existing GraphRAG methods are limited in effectiveness (i.e., quality of the generated answers) and/or efficiency (i.e., response time or the number of used tokens). In this paper, we propose to classify the questions according to a complete four-class taxonomy and adaptively select the appropriate graph traversal strategy for each type of questions. Our system PolyG is essentially a query planner for GraphRAG and can handle diverse questions with an unified interface and execution engine. Compared with SOTA GraphRAG methods, PolyG achieves an overall win rate of 75% on generation quality and a speedup up to 4x on response time.
• 摘要：GraphRag增强了大型语言模型（LLMS），从而通过从外部知识图中检索相关事实来为用户问题生成质量答案。现有的GraphRag方法采用固定的图形遍历策略进行事实检索，但我们观察到用户问题以不同的类型出现，需要不同的图形遍历策略。因此，现有的GraphRag方法的有效性有限（即生成的答案的质量）和/或效率（即响应时间或二手代币的数量）。在本文中，我们建议根据完整的四级分类法对问题进行分类，并适应每种类型的问题的适当的图形遍历策略。我们的系统Polyg本质上是GraphRag的查询计划者，可以通过统一的界面和执行引擎来处理各种问题。与SOTA GraphRag方法相比，Polyg的发电质量的总胜率为75％，响应时间的加速速度高达4倍。

Title: Less-to-More Generalization: Unlocking More Controllability by In-Context Generation

• Authors: Shaojin Wu, Mengqi Huang, Wenxu Wu, Yufeng Cheng, Fei Ding, Qian He
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2504.02160
• Pdf URL: https://arxiv.org/pdf/2504.02160
• Copy Paste: [[2504.02160]] Less-to-More Generalization: Unlocking More Controllability by In-Context Generation(https://arxiv.org/abs/2504.02160)
• Keywords: generation
• Abstract: Although subject-driven generation has been extensively explored in image generation due to its wide applications, it still has challenges in data scalability and subject expansibility. For the first challenge, moving from curating single-subject datasets to multiple-subject ones and scaling them is particularly difficult. For the second, most recent methods center on single-subject generation, making it hard to apply when dealing with multi-subject scenarios. In this study, we propose a highly-consistent data synthesis pipeline to tackle this challenge. This pipeline harnesses the intrinsic in-context generation capabilities of diffusion transformers and generates high-consistency multi-subject paired data. Additionally, we introduce UNO, which consists of progressive cross-modal alignment and universal rotary position embedding. It is a multi-image conditioned subject-to-image model iteratively trained from a text-to-image model. Extensive experiments show that our method can achieve high consistency while ensuring controllability in both single-subject and multi-subject driven generation.
• 摘要：尽管由于其广泛的应用，因此在图像生成中已经广泛探索了受试者驱动的生成，但它仍然在数据可扩展性和扩展性方面面临挑战。对于第一个挑战，从策划单人物数据集转移到多个受试者的数据集并缩放它们特别困难。在第二个单个主体生成上的第二个方法中，最新方法使得在处理多​​主体方案时很难应用。在这项研究中，我们提出了一条高度一致的数据合成管道，以应对这一挑战。该管道利用扩散变压器的固有内在生成能力，并生成高谐波多主体配对数据。此外，我们介绍了UNO，其中包括进行性跨模式比对和通用旋转位置嵌入。这是一个从文本到图像模型进行迭代训练的多图像对象模型。广泛的实验表明，我们的方法可以达到高稠度，同时确保单个受试者和多主体驱动的生成中的可控性。

Title: Foreground Focus: Enhancing Coherence and Fidelity in Camouflaged Image Generation

• Authors: Pei-Chi Chen, Yi Yao, Chan-Feng Hsu, HongXia Xie, Hung-Jen Chen, Hong-Han Shuai, Wen-Huang Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.02180
• Pdf URL: https://arxiv.org/pdf/2504.02180
• Copy Paste: [[2504.02180]] Foreground Focus: Enhancing Coherence and Fidelity in Camouflaged Image Generation(https://arxiv.org/abs/2504.02180)
• Keywords: generation
• Abstract: Camouflaged image generation is emerging as a solution to data scarcity in camouflaged vision perception, offering a cost-effective alternative to data collection and labeling. Recently, the state-of-the-art approach successfully generates camouflaged images using only foreground objects. However, it faces two critical weaknesses: 1) the background knowledge does not integrate effectively with foreground features, resulting in a lack of foreground-background coherence (e.g., color discrepancy); 2) the generation process does not prioritize the fidelity of foreground objects, which leads to distortion, particularly for small objects. To address these issues, we propose a Foreground-Aware Camouflaged Image Generation (FACIG) model. Specifically, we introduce a Foreground-Aware Feature Integration Module (FAFIM) to strengthen the integration between foreground features and background knowledge. In addition, a Foreground-Aware Denoising Loss is designed to enhance foreground reconstruction supervision. Experiments on various datasets show our method outperforms previous methods in overall camouflaged image quality and foreground fidelity.
• 摘要：伪装的图像生成正在作为解决伪装视力感知中数据稀缺的解决方案，为数据收集和标签提供了具有成本效益的替代方案。最近，最先进的方法仅使用前景对象成功地生成了伪装的图像。但是，它面临两个关键弱点：1）背景知识并未与前景特征有效整合，从而导致缺乏前景背景连贯性（例如，颜色差异）； 2）生成过程没有优先考虑前景对象的保真度，这会导致失真，尤其是对于小物体。为了解决这些问题，我们提出了一种前景感知的伪装图像生成（FOCIG）模型。具体来说，我们引入了一个前景感知的特征集成模块（FAFIM），以增强前景特征和背景知识之间的集成。此外，前景感知的剥夺损失旨在增强前景重建监督。各种数据集上的实验表明，我们的方法在整体伪装的图像质量和前景保真度上都优于先前的方法。

Title: AC-LoRA: Auto Component LoRA for Personalized Artistic Style Image Generation

• Authors: Zhipu Cui, Andong Tian, Zhi Ying, Jialiang Lu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.02231
• Pdf URL: https://arxiv.org/pdf/2504.02231
• Copy Paste: [[2504.02231]] AC-LoRA: Auto Component LoRA for Personalized Artistic Style Image Generation(https://arxiv.org/abs/2504.02231)
• Keywords: generation
• Abstract: Personalized image generation allows users to preserve styles or subjects of a provided small set of images for further image generation. With the advancement in large text-to-image models, many techniques have been developed to efficiently fine-tune those models for personalization, such as Low Rank Adaptation (LoRA). However, LoRA-based methods often face the challenge of adjusting the rank parameter to achieve satisfactory results. To address this challenge, AutoComponent-LoRA (AC-LoRA) is proposed, which is able to automatically separate the signal component and noise component of the LoRA matrices for fast and efficient personalized artistic style image generation. This method is based on Singular Value Decomposition (SVD) and dynamic heuristics to update the hyperparameters during training. Superior performance over existing methods in overcoming model underfitting or overfitting problems is demonstrated. The results were validated using FID, CLIP, DINO, and ImageReward, achieving an average of 9% improvement.
• 摘要：个性化的图像生成使用户可以保留提供的少量图像的样式或主题，以进行进一步的图像生成。随着大型文本到图像模型的进步，已经开发了许多技术来有效地微调这些模型的个性化模型，例如低级适应（Lora）。但是，基于洛拉的方法通常面临调整等级参数以获得令人满意的结果的挑战。为了应对这一挑战，提出了AutoComponent-Lora（AC-Lora），该挑战能够自动将Lora矩阵的信号组件和噪声组件分开，以快速有效地个性化的艺术风格图像生成。该方法基于奇异值分解（SVD）和动态启发式方法，以更新训练期间的超参数。在克服模型中的现有方法中，表现出色的性能不足以适应或过度拟合问题。使用FID，剪辑，恐龙和成像智能验证了结果，平均提高了9％。

Title: WonderTurbo: Generating Interactive 3D World in 0.72 Seconds

• Authors: Chaojun Ni, Xiaofeng Wang, Zheng Zhu, Weijie Wang, Haoyun Li, Guosheng Zhao, Jie Li, Wenkang Qin, Guan Huang, Wenjun Mei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.02261
• Pdf URL: https://arxiv.org/pdf/2504.02261
• Copy Paste: [[2504.02261]] WonderTurbo: Generating Interactive 3D World in 0.72 Seconds(https://arxiv.org/abs/2504.02261)
• Keywords: generation
• Abstract: Interactive 3D generation is gaining momentum and capturing extensive attention for its potential to create immersive virtual experiences. However, a critical challenge in current 3D generation technologies lies in achieving real-time interactivity. To address this issue, we introduce WonderTurbo, the first real-time interactive 3D scene generation framework capable of generating novel perspectives of 3D scenes within 0.72 seconds. Specifically, WonderTurbo accelerates both geometric and appearance modeling in 3D scene generation. In terms of geometry, we propose StepSplat, an innovative method that constructs efficient 3D geometric representations through dynamic updates, each taking only 0.26 seconds. Additionally, we design QuickDepth, a lightweight depth completion module that provides consistent depth input for StepSplat, further enhancing geometric accuracy. For appearance modeling, we develop FastPaint, a 2-steps diffusion model tailored for instant inpainting, which focuses on maintaining spatial appearance consistency. Experimental results demonstrate that WonderTurbo achieves a remarkable 15X speedup compared to baseline methods, while preserving excellent spatial consistency and delivering high-quality output.
• 摘要：互动3D代表正在获得动力，并引起广泛的关注，以创造身临其境的虚拟体验的潜力。但是，当前3D一代技术的关键挑战在于实现实时互动。为了解决这个问题，我们介绍了Wonderturbo，这是第一个实时交互式3D场景生成框架，能够在0.72秒内生成3D场景的新观点。具体而言，Wonderturbo在3D场景生成中加速了几何和外观建模。在几何形状方面，我们提出了一种创新方法，该方法通过动态更新构建有效的3D几何表示，每种方法仅需0.26秒。此外，我们设计了QuickDepth，这是一个轻巧的深度完成模块，该模块为Stepplat提供一致的深度输入，从而进一步提高了几何精度。对于外观建模，我们开发了FastPaint，这是一种针对即时涂料的2步扩散模型，该模型的重点是保持空间外观一致性。实验结果表明，与基线方法相比，Wonderturbo达到了显着的15倍加速度，同时保留了出色的空间一致性并提供了高质量的输出。

Title: Advancing Semantic Caching for LLMs with Domain-Specific Embeddings and Synthetic Data

• Authors: Waris Gill (1 and 2), Justin Cechmanek (1), Tyler Hutcherson (1), Srijith Rajamohan (1), Jen Agarwal (1), Muhammad Ali Gulzar (2), Manvinder Singh (1), Benoit Dion ((1) Redis, (2) Virginia Tech)
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2504.02268
• Pdf URL: https://arxiv.org/pdf/2504.02268
• Copy Paste: [[2504.02268]] Advancing Semantic Caching for LLMs with Domain-Specific Embeddings and Synthetic Data(https://arxiv.org/abs/2504.02268)
• Keywords: generation
• Abstract: This report investigates enhancing semantic caching effectiveness by employing specialized, fine-tuned embedding models. Semantic caching relies on embedding similarity rather than exact key matching, presenting unique challenges in balancing precision, query latency, and computational efficiency. We propose leveraging smaller, domain-specific embedding models, fine-tuned with targeted real-world and synthetically generated datasets. Our empirical evaluations demonstrate that compact embedding models fine-tuned for just one epoch on specialized datasets significantly surpass both state-of-the-art open-source and proprietary alternatives in precision and recall. Moreover, we introduce a novel synthetic data generation pipeline for the semantic cache that mitigates the challenge of limited domain-specific annotated data, further boosting embedding performance. Our approach effectively balances computational overhead and accuracy, establishing a viable and efficient strategy for practical semantic caching implementations.
• 摘要：该报告通过采用专业的，微调的嵌入模型来研究增强语义缓存效果。语义缓存依赖于嵌入相似性而不是确切的关键匹配，这在平衡精度，查询延迟和计算效率方面带来了独特的挑战。我们建议利用较小的域特异性嵌入模型，并通过目标现实世界和合成生成的数据集进行了微调。我们的经验评估表明，仅在专业数据集上的一个时代进行了精细调整的紧凑型嵌入模型在精确和召回方面显着超过了最新的开源和专有替代方案。此外，我们引入了一种新型的综合数据生成管道，以减轻有限域特异性注释数据的挑战，从而进一步增强了嵌入性能。我们的方法有效地平衡了计算开销和准确性，为实用语义缓存实现建立了可行有效的策略。

Title: Generative Classifier for Domain Generalization

• Authors: Shaocong Long, Qianyu Zhou, Xiangtai Li, Chenhao Ying, Yunhai Tong, Lizhuang Ma, Yuan Luo, Dacheng Tao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.02272
• Pdf URL: https://arxiv.org/pdf/2504.02272
• Copy Paste: [[2504.02272]] Generative Classifier for Domain Generalization(https://arxiv.org/abs/2504.02272)
• Keywords: generative
• Abstract: Domain generalization (DG) aims to improve the generalizability of computer vision models toward distribution shifts. The mainstream DG methods focus on learning domain invariance, however, such methods overlook the potential inherent in domain-specific information. While the prevailing practice of discriminative linear classifier has been tailored to domain-invariant features, it struggles when confronted with diverse domain-specific information, e.g., intra-class shifts, that exhibits multi-modality. To address these issues, we explore the theoretical implications of relying on domain invariance, revealing the crucial role of domain-specific information in mitigating the target risk for DG. Drawing from these insights, we propose Generative Classifier-driven Domain Generalization (GCDG), introducing a generative paradigm for the DG classifier based on Gaussian Mixture Models (GMMs) for each class across domains. GCDG consists of three key modules: Heterogeneity Learning Classifier~(HLC), Spurious Correlation Blocking~(SCB), and Diverse Component Balancing~(DCB). Concretely, HLC attempts to model the feature distributions and thereby capture valuable domain-specific information via GMMs. SCB identifies the neural units containing spurious correlations and perturbs them, mitigating the risk of HLC learning spurious patterns. Meanwhile, DCB ensures a balanced contribution of components in HLC, preventing the underestimation or neglect of critical components. In this way, GCDG excels in capturing the nuances of domain-specific information characterized by diverse distributions. GCDG demonstrates the potential to reduce the target risk and encourage flat minima, improving the generalizability. Extensive experiments show GCDG's comparable performance on five DG benchmarks and one face anti-spoofing dataset, seamlessly integrating into existing DG methods with consistent improvements.
• 摘要：域概括（DG）旨在提高计算机视觉模型对分配变化的概括性。但是，主流DG方法着重于学习域的不变性，但是，这种方法忽略了特定于域的信息中固有的潜在。虽然判别线性分类器的普遍实践是针对域不变特征量身定制的，但在面对各种特定领域的信息时，它会挣扎，例如表现出多模态的阶级内部变化。为了解决这些问题，我们探讨了依靠域不变性的理论含义，揭示了特定于领域特定信息在减轻DG的目标风险中的关键作用。根据这些见解，我们提出了生成分类器驱动的域的概括（GCDG），它基于跨域的每个类别的高斯混合物模型（GMM）引入了DG分类器的生成范式。 GCDG由三个关键模块组成：异质性学习分类器〜（HLC），虚假相关性阻止〜（SCB）和多样化的组件平衡〜（DCB）。具体而言，HLC试图对特征分布进行建模，从而通过GMM捕获有价值的域特异性信息。 SCB识别包含虚假相关性的神经单位并散布它们，从而减轻HLC学习伪造模式的风险。同时，DCB确保了HLC中组件的平衡贡献，以防止低估或忽略关键组件。通过这种方式，GCDG擅长捕捉以各种分布为特征的域特异性信息的细微差别。 GCDG证明了降低目标风险并鼓励平坦的最小值的潜力，从而提高了普遍性。广泛的实验表明，GCDG在五个DG基准和一个面对抗烟数据集上的可比性能，无缝地集成到现有的DG方法中，并具有一致的改进。

Title: OmniCam: Unified Multimodal Video Generation via Camera Control

• Authors: Xiaoda Yang, Jiayang Xu, Kaixuan Luan, Xinyu Zhan, Hongshun Qiu, Shijun Shi, Hao Li, Shuai Yang, Li Zhang, Checheng Yu, Cewu Lu, Lixin Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.02312
• Pdf URL: https://arxiv.org/pdf/2504.02312
• Copy Paste: [[2504.02312]] OmniCam: Unified Multimodal Video Generation via Camera Control(https://arxiv.org/abs/2504.02312)
• Keywords: generation
• Abstract: Camera control, which achieves diverse visual effects by changing camera position and pose, has attracted widespread attention. However, existing methods face challenges such as complex interaction and limited control capabilities. To address these issues, we present OmniCam, a unified multimodal camera control framework. Leveraging large language models and video diffusion models, OmniCam generates spatio-temporally consistent videos. It supports various combinations of input modalities: the user can provide text or video with expected trajectory as camera path guidance, and image or video as content reference, enabling precise control over camera motion. To facilitate the training of OmniCam, we introduce the OmniTr dataset, which contains a large collection of high-quality long-sequence trajectories, videos, and corresponding descriptions. Experimental results demonstrate that our model achieves state-of-the-art performance in high-quality camera-controlled video generation across various metrics.
• 摘要：相机控制通过改变相机位置和姿势来实现各种视觉效果，引起了广泛的关注。但是，现有方法面临诸如复杂互动和有限的控制功能之类的挑战。为了解决这些问题，我们提出了统一的多模式相机控制框架Omnicam。 Omnicam利用大型语言模型和视频扩散模型生成时空一致的视频。它支持输入方式的各种组合：用户可以提供带有预期轨迹作为相机路径指导的文本或视频，以及图像或视频作为内容参考，从而可以精确控制相机运动。为了促进OMNICAM的培训，我们介绍了Omnitr数据集，其中包含大量高质量的长期轨迹，视频和相应描述。实验结果表明，我们的模型在各种指标的高质量摄像机控制视频生成中实现了最先进的性能。

Title: ConsDreamer: Advancing Multi-View Consistency for Zero-Shot Text-to-3D Generation

• Authors: Yuan Zhou, Shilong Jin, Litao Hua, Wanjun Lv, Haoran Duan, Jungong Han
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.02316
• Pdf URL: https://arxiv.org/pdf/2504.02316
• Copy Paste: [[2504.02316]] ConsDreamer: Advancing Multi-View Consistency for Zero-Shot Text-to-3D Generation(https://arxiv.org/abs/2504.02316)
• Keywords: generation
• Abstract: Recent advances in zero-shot text-to-3D generation have revolutionized 3D content creation by enabling direct synthesis from textual descriptions. While state-of-the-art methods leverage 3D Gaussian Splatting with score distillation to enhance multi-view rendering through pre-trained text-to-image (T2I) models, they suffer from inherent view biases in T2I priors. These biases lead to inconsistent 3D generation, particularly manifesting as the multi-face Janus problem, where objects exhibit conflicting features across views. To address this fundamental challenge, we propose ConsDreamer, a novel framework that mitigates view bias by refining both the conditional and unconditional terms in the score distillation process: (1) a View Disentanglement Module (VDM) that eliminates viewpoint biases in conditional prompts by decoupling irrelevant view components and injecting precise camera parameters; and (2) a similarity-based partial order loss that enforces geometric consistency in the unconditional term by aligning cosine similarities with azimuth relationships. Extensive experiments demonstrate that ConsDreamer effectively mitigates the multi-face Janus problem in text-to-3D generation, outperforming existing methods in both visual quality and consistency.
• 摘要：零击文本到3D代的最新进展通过从文本描述中启用直接综合，彻底改变了3D内容的创建。尽管最新的方法利用了3D高斯脱落，并通过得分蒸馏来通过预先训练的文本对图像（T2I）模型来增强多视图渲染，但它们在T2i先生中具有固有的视图偏见。这些偏见导致3D代表不一致，特别是表现为多面Janus问题，在该问题中，对象跨视图表现出矛盾的特征。为了应对这一基本挑战，我们提出了一个新颖的框架，这是一个新颖的框架，通过在得分蒸馏过程中同时完善条件和无条件的术语来减轻视图偏见：（1）视图解开词（VDM），消除了视图点在有条件的提示中消除观点偏见，从而通过排除Irrepplant Inferevant View Components和Indemplant Indecome Indionement Compares sameress参数来偏见。 （2）基于相似性的部分阶损失，通过将余弦相似性与方位关系对准相似性，从而在无条件的项中实现几何一致性。广泛的实验表明，Consdreamer有效地减轻了文本到3D生成中的多面Janus问题，在视觉质量和一致性方面都优于现有方法。

Title: Towards Assessing Deep Learning Test Input Generators

• Authors: Seif Mzoughi, Ahmed Hajyahmed, Mohamed Elshafei, Foutse Khomh anb Diego Elias Costa
• Subjects: cs.LG, cs.CV, cs.SE
• Abstract URL: https://arxiv.org/abs/2504.02329
• Pdf URL: https://arxiv.org/pdf/2504.02329
• Copy Paste: [[2504.02329]] Towards Assessing Deep Learning Test Input Generators(https://arxiv.org/abs/2504.02329)
• Keywords: generation
• Abstract: Deep Learning (DL) systems are increasingly deployed in safety-critical applications, yet they remain vulnerable to robustness issues that can lead to significant failures. While numerous Test Input Generators (TIGs) have been developed to evaluate DL robustness, a comprehensive assessment of their effectiveness across different dimensions is still lacking. This paper presents a comprehensive assessment of four state-of-the-art TIGs--DeepHunter, DeepFault, AdvGAN, and SinVAD--across multiple critical aspects: fault-revealing capability, naturalness, diversity, and efficiency. Our empirical study leverages three pre-trained models (LeNet-5, VGG16, and EfficientNetB3) on datasets of varying complexity (MNIST, CIFAR-10, and ImageNet-1K) to evaluate TIG performance. Our findings reveal important trade-offs in robustness revealing capability, variation in test case generation, and computational efficiency across TIGs. The results also show that TIG performance varies significantly with dataset complexity, as tools that perform well on simpler datasets may struggle with more complex ones. In contrast, others maintain steadier performance or better scalability. This paper offers practical guidance for selecting appropriate TIGs aligned with specific objectives and dataset characteristics. Nonetheless, more work is needed to address TIG limitations and advance TIGs for real-world, safety-critical systems.
• 摘要：深度学习（DL）系统越来越多地在安全至关重要的应用中部署，但它们仍然容易受到可造成重大失败的鲁棒性问题的影响。尽管已经开发了许多测试输入发电机（TIG）来评估DL鲁棒性，但仍缺乏对它们在不同维度上的有效性的全面评估。本文对四个最先进的tigs（dephunter，deepfault，advgan和sinvad）进行了全面评估 - 概括了多个关键方面：避免故障的能力，自然性，多样性和效率。我们的实证研究利用了不同复杂性（MNIST，CIFAR-10和Imagenet-1K）的数据集上的三个预训练模型（LENET-5，VGG16和有效网络）来评估TIG性能。我们的发现揭示了鲁棒性的重要权衡，揭示了能力，测试案例生成的变化以及TIG的计算效率。结果还表明，TIG性能随数据集复杂性而有很大差异，因为在更简单的数据集上表现良好的工具可能会在更复杂的数据集中遇到困难。相比之下，其他人保持稳定的性能或更好的可扩展性。本文提供了实用的指导，以选择与特定目标和数据集特征一致的合适的TIG。尽管如此，还需要做更多的工作来解决TIG限制并推动对现实世界中的安全 - 关键系统的提升。

Title: LPA3D: 3D Room-Level Scene Generation from In-the-Wild Images

• Authors: Ming-Jia Yang, Yu-Xiao Guo, Yang Liu, Bin Zhou, Xin Tong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.02337
• Pdf URL: https://arxiv.org/pdf/2504.02337
• Copy Paste: [[2504.02337]] LPA3D: 3D Room-Level Scene Generation from In-the-Wild Images(https://arxiv.org/abs/2504.02337)
• Keywords: generation, generative
• Abstract: Generating realistic, room-level indoor scenes with semantically plausible and detailed appearances from in-the-wild images is crucial for various applications in VR, AR, and robotics. The success of NeRF-based generative methods indicates a promising direction to address this challenge. However, unlike their success at the object level, existing scene-level generative methods require additional information, such as multiple views, depth images, or semantic guidance, rather than relying solely on RGB images. This is because NeRF-based methods necessitate prior knowledge of camera poses, which is challenging to approximate for indoor scenes due to the complexity of defining alignment and the difficulty of globally estimating poses from a single image, given the unseen parts behind the camera. To address this challenge, we redefine global poses within the framework of Local-Pose-Alignment (LPA) -- an anchor-based multi-local-coordinate system that uses a selected number of anchors as the roots of these coordinates. Building on this foundation, we introduce LPA-GAN, a novel NeRF-based generative approach that incorporates specific modifications to estimate the priors of camera poses under LPA. It also co-optimizes the pose predictor and scene generation processes. Our ablation study and comparisons with straightforward extensions of NeRF-based object generative methods demonstrate the effectiveness of our approach. Furthermore, visual comparisons with other techniques reveal that our method achieves superior view-to-view consistency and semantic normality.
• 摘要：从野外图像中生成具有语义上合理和详细外观的现实，室内室内场景对于VR，AR和机器人技术中的各种应用至关重要。基于NERF的生成方法的成功表明了应对这一挑战的有希望的方向。但是，与对象级别的成功不同，现有的场景级生成方法需要其他信息，例如多个视图，深度图像或语义指导，而不是仅依靠RGB图像。这是因为基于NERF的方法需要对相机姿势的先验知识，这对于室内场景而言是挑战，这是因为定义对齐的复杂性以及从单个图像的全球估算姿势的难度，鉴于相机后面的看不见部分。为了应对这一挑战，我们重新定义了本地置序对准（LPA）的框架 - 基于锚的多区域坐标系统，该系统使用选定数量的锚固量作为这些坐标的根源。在此基础的基础上，我们引入了LPA-GAN，这是一种基于NERF的新型生成方法，它结合了特定的修改，以估算LPA下的相机姿势先验。它还可以优化姿势预测因子和场景生成过程。我们的消融研究和与基于NERF的对象生成方法直接扩展的比较证明了我们方法的有效性。此外，与其他技术的视觉比较表明，我们的方法达到了卓越的视图对视图一致性和语义正态性。

Title: Reinforcement Learning for Solving the Pricing Problem in Column Generation: Applications to Vehicle Routing

• Authors: Abdo Abouelrous, Laurens Bliek, Adriana F. Gabor, Yaoxin Wu, Yingqian Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.02383
• Pdf URL: https://arxiv.org/pdf/2504.02383
• Copy Paste: [[2504.02383]] Reinforcement Learning for Solving the Pricing Problem in Column Generation: Applications to Vehicle Routing(https://arxiv.org/abs/2504.02383)
• Keywords: generation
• Abstract: In this paper, we address the problem of Column Generation (CG) using Reinforcement Learning (RL). Specifically, we use a RL model based on the attention-mechanism architecture to find the columns with most negative reduced cost in the Pricing Problem (PP). Unlike previous Machine Learning (ML) applications for CG, our model deploys an end-to-end mechanism as it independently solves the pricing problem without the help of any heuristic. We consider a variant of Vehicle Routing Problem (VRP) as a case study for our method. Through a set of experiments where our method is compared against a Dynamic Programming (DP)-based heuristic for solving the PP, we show that our method solves the linear relaxation up to a reasonable objective gap within 9% in significantly shorter running times, up to over 300 times faster for instances with 100 customers.
• 摘要：在本文中，我们使用增强学习（RL）解决了列生成（CG）的问题。具体而言，我们使用基于注意力机制架构的RL模型来找到定价问题（PP）中最负成本的列。与以前的机器学习（ML）应用程序有关CG的应用程序，我们的模型部署了一种端到端机制，因为它在没有任何启发式的帮助下独立解决了定价问题。我们将车辆路由问题（VRP）视为我们方法的案例研究。通过一组实验，将我们的方法与用于解决PP的基于动态编程（DP）的启发式方法进行了比较，我们表明我们的方法可以在较短的运行时间内求解9％以内9％以内的线性放松，而有100个客户的情况更快300倍。

Title: OmniTalker: Real-Time Text-Driven Talking Head Generation with In-Context Audio-Visual Style Replication

• Authors: Zhongjian Wang, Peng Zhang, Jinwei Qi, Guangyuan Wang Sheng Xu, Bang Zhang, Liefeng Bo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.02433
• Pdf URL: https://arxiv.org/pdf/2504.02433
• Copy Paste: [[2504.02433]] OmniTalker: Real-Time Text-Driven Talking Head Generation with In-Context Audio-Visual Style Replication(https://arxiv.org/abs/2504.02433)
• Keywords: generation
• Abstract: Recent years have witnessed remarkable advances in talking head generation, owing to its potential to revolutionize the human-AI interaction from text interfaces into realistic video chats. However, research on text-driven talking heads remains underexplored, with existing methods predominantly adopting a cascaded pipeline that combines TTS systems with audio-driven talking head models. This conventional pipeline not only introduces system complexity and latency overhead but also fundamentally suffers from asynchronous audiovisual output and stylistic discrepancies between generated speech and visual expressions. To address these limitations, we introduce OmniTalker, an end-to-end unified framework that simultaneously generates synchronized speech and talking head videos from text and reference video in real-time zero-shot scenarios, while preserving both speech style and facial styles. The framework employs a dual-branch diffusion transformer architecture: the audio branch synthesizes mel-spectrograms from text, while the visual branch predicts fine-grained head poses and facial dynamics. To bridge modalities, we introduce a novel audio-visual fusion module that integrates cross-modal information to ensure temporal synchronization and stylistic coherence between audio and visual outputs. Furthermore, our in-context reference learning module effectively captures both speech and facial style characteristics from a single reference video without introducing an extra style extracting module. To the best of our knowledge, OmniTalker presents the first unified framework that jointly models speech style and facial style in a zero-shot setting, achieving real-time inference speed of 25 FPS. Extensive experiments demonstrate that our method surpasses existing approaches in generation quality, particularly excelling in style preservation and audio-video synchronization.
• 摘要：近年来，由于其潜力从文本接口到现实的视频聊天，它有可能改变人类的互动，从而在谈论脑电图中取得了显着进步。但是，对文本驱动的说话头的研究仍然没有被忽视，现有方法主要采用级联管道，将TTS系统与音频驱动的说话头模型相结合。这条传统的管道不仅引入了系统的复杂性和潜伏期开销，而且从根本上讲，它们具有异步的视听输出和产生的语音和视觉表达式之间的风格差异。为了解决这些限制，我们介绍了Omnitalker，这是一个端到端的统一框架，同时在实时零拍摄的情况下从文本和参考视频中产生同步的语音和谈话主视频，同时保留语音样式和面部风格。该框架采用双分支扩散变压器体系结构：音频分支从文本中综合了MEL-SPECTROGRAM，而Visual分支可以预测细粒的头部姿势和面部动力学。为了桥接模式，我们引入了一种新颖的视听融合模块，该模块集成了交叉模式信息，以确保音频和视觉输出之间的时间同步和风格上的连贯性。此外，我们的文章参考学习模块有效地从单个参考视频中捕获了语音和面部风格特征，而无需引入额外的样式提取模块。据我们所知，Omnitalker提出了第一个在零拍设置中共同建模语音样式和面部风格的统一框架，以实现25 fps的实时推理速度。广泛的实验表明，我们的方法超过了发电质量的现有方法，尤其是样式保存和音频视频同步。

Title: SkyReels-A2: Compose Anything in Video Diffusion Transformers

• Authors: Zhengcong Fei, Debang Li, Di Qiu, Jiahua Wang, Yikun Dou, Rui Wang, Jingtao Xu, Mingyuan Fan, Guibin Chen, Yang Li, Yahui Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.02436
• Pdf URL: https://arxiv.org/pdf/2504.02436
• Copy Paste: [[2504.02436]] SkyReels-A2: Compose Anything in Video Diffusion Transformers(https://arxiv.org/abs/2504.02436)
• Keywords: generation, generative
• Abstract: This paper presents SkyReels-A2, a controllable video generation framework capable of assembling arbitrary visual elements (e.g., characters, objects, backgrounds) into synthesized videos based on textual prompts while maintaining strict consistency with reference images for each element. We term this task elements-to-video (E2V), whose primary challenges lie in preserving the fidelity of each reference element, ensuring coherent composition of the scene, and achieving natural outputs. To address these, we first design a comprehensive data pipeline to construct prompt-reference-video triplets for model training. Next, we propose a novel image-text joint embedding model to inject multi-element representations into the generative process, balancing element-specific consistency with global coherence and text alignment. We also optimize the inference pipeline for both speed and output stability. Moreover, we introduce a carefully curated benchmark for systematic evaluation, i.e, A2 Bench. Experiments demonstrate that our framework can generate diverse, high-quality videos with precise element control. SkyReels-A2 is the first open-source commercial grade model for the generation of E2V, performing favorably against advanced closed-source commercial models. We anticipate SkyReels-A2 will advance creative applications such as drama and virtual e-commerce, pushing the boundaries of controllable video generation.
• 摘要：本文介绍了Skyreels-A2，这是一个可控制的视频生成框架，该框架能够根据文本提示将任意的视觉元素（例如字符，对象，背景）组装成综合视频，同时与每个元素的参考图像保持严格的一致性。我们认为此任务要素到视频（E2V），其主要挑战在于保留每个参考元素的保真度，确保场景的连贯构成并实现自然产出。为了解决这些问题，我们首先设计了一条综合的数据管道，以构建及时的参考 - 视频三重态以进行模型培训。接下来，我们提出了一个新型的图像文本嵌入模型，将多元素表示形式注入生成过程，并平衡元素特异性的一致性与全局相干性和文本对齐。我们还针对速度和输出稳定性优化推理管道。此外，我们引入了一个精心策划的基准，以进行系统评估，即A2基准。实验表明，我们的框架可以生成具有精确元素控制的多样化，高质量的视频。 Skyreels-A2是第一个用于生成E2V的开源商业级模型，对先进的封闭式商业模型表现出色。我们预计Skyreels-A2将推进诸如戏剧和虚拟电子商务之类的创意应用程序，从而突破可控视频的界限。

Title: ConMo: Controllable Motion Disentanglement and Recomposition for Zero-Shot Motion Transfer

• Authors: Jiayi Gao, Zijin Yin, Changcheng Hua, Yuxin Peng, Kongming Liang, Zhanyu Ma, Jun Guo, Yang Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.02451
• Pdf URL: https://arxiv.org/pdf/2504.02451
• Copy Paste: [[2504.02451]] ConMo: Controllable Motion Disentanglement and Recomposition for Zero-Shot Motion Transfer(https://arxiv.org/abs/2504.02451)
• Keywords: generation
• Abstract: The development of Text-to-Video (T2V) generation has made motion transfer possible, enabling the control of video motion based on existing footage. However, current methods have two limitations: 1) struggle to handle multi-subjects videos, failing to transfer specific subject motion; 2) struggle to preserve the diversity and accuracy of motion as transferring to subjects with varying shapes. To overcome these, we introduce \textbf{ConMo}, a zero-shot framework that disentangle and recompose the motions of subjects and camera movements. ConMo isolates individual subject and background motion cues from complex trajectories in source videos using only subject masks, and reassembles them for target video generation. This approach enables more accurate motion control across diverse subjects and improves performance in multi-subject scenarios. Additionally, we propose soft guidance in the recomposition stage which controls the retention of original motion to adjust shape constraints, aiding subject shape adaptation and semantic transformation. Unlike previous methods, ConMo unlocks a wide range of applications, including subject size and position editing, subject removal, semantic modifications, and camera motion simulation. Extensive experiments demonstrate that ConMo significantly outperforms state-of-the-art methods in motion fidelity and semantic consistency. The code is available at this https URL.
• 摘要：文本到视频（T2V）生成的开发使运动转移成为可能，从而可以基于现有镜头来控制视频运动。但是，当前方法有两个局限性：1）难以处理多主题视频，无法转移特定的主题运动； 2）努力将运动的多样性和准确性保留到具有不同形状的受试者中。为了克服这些问题，我们介绍了\ textbf {conmo}，这是一个零拍的框架，可将主体和相机运动的动作删除并重新组合。 Conmo仅使用主题掩码将单个主题和背景运动提示与源视频中的复杂轨迹隔离，并将其重新组装以进行目标视频。这种方法可实现跨不同主题的更准确的运动控制，并改善多受试者方案的性能。此外，我们在重组阶段提出了软引导，该指导控制原始运动的保留，以调整形状约束，有助于主体形状适应和语义转换。与以前的方法不同，Conmo解锁了广泛的应用，包括主题大小和位置编辑，主题删除，语义修改和摄像机运动模拟。广泛的实验表明，CONMO在运动保真度和语义一致性方面的表现显着优于最先进的方法。该代码可在此HTTPS URL上找到。

Title: A Physics-Informed Meta-Learning Framework for the Continuous Solution of Parametric PDEs on Arbitrary Geometries

• Authors: Reza Najian Asl, Yusuke Yamazaki, Kianoosh Taghikhani, Mayu Muramatsu, Markus Apel, Shahed Rezaei
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2504.02459
• Pdf URL: https://arxiv.org/pdf/2504.02459
• Copy Paste: [[2504.02459]] A Physics-Informed Meta-Learning Framework for the Continuous Solution of Parametric PDEs on Arbitrary Geometries(https://arxiv.org/abs/2504.02459)
• Keywords: super-resolution
• Abstract: In this work, we introduce implicit Finite Operator Learning (iFOL) for the continuous and parametric solution of partial differential equations (PDEs) on arbitrary geometries. We propose a physics-informed encoder-decoder network to establish the mapping between continuous parameter and solution spaces. The decoder constructs the parametric solution field by leveraging an implicit neural field network conditioned on a latent or feature code. Instance-specific codes are derived through a PDE encoding process based on the second-order meta-learning technique. In training and inference, a physics-informed loss function is minimized during the PDE encoding and decoding. iFOL expresses the loss function in an energy or weighted residual form and evaluates it using discrete residuals derived from standard numerical PDE methods. This approach results in the backpropagation of discrete residuals during both training and inference. iFOL features several key properties: (1) its unique loss formulation eliminates the need for the conventional encode-process-decode pipeline previously used in operator learning with conditional neural fields for PDEs; (2) it not only provides accurate parametric and continuous fields but also delivers solution-to-parameter gradients without requiring additional loss terms or sensitivity analysis; (3) it can effectively capture sharp discontinuities in the solution; and (4) it removes constraints on the geometry and mesh, making it applicable to arbitrary geometries and spatial sampling (zero-shot super-resolution capability). We critically assess these features and analyze the network's ability to generalize to unseen samples across both stationary and transient PDEs. The overall performance of the proposed method is promising, demonstrating its applicability to a range of challenging problems in computational mechanics.
• 摘要：在这项工作中，我们为在任意几何形状上的部分微分方程（PDE）的连续和参数解决方案引入了隐式有限运算符学习（IFOL）。我们提出了一个物理信息编码器网络，以在连续参数和解决方案空间之间建立映射。解码器通过利用以潜在或特征代码为条件的隐式神经场网络来构建参数解决方案字段。实例特定的代码是通过基于二阶元学习技术的PDE编码过程得出的。在训练和推理中，PDE编码和解码过程中，物理信息损失函数被最小化。 IFOL以能量或加权残余形式表示损失函数，并使用源自标准数值PDE方法的离散残留物进行评估。这种方法导致在训练和推理期间均导致离散残差的反向传播。 IFOL具有多种关键属性：（1）其唯一的损失公式消除了先前在运算符学习中使用的传统编码过程管道的需求； （2）它不仅提供了准确的参数和连续场，而且还提供了解决方案到参数梯度，而无需其他损失项或灵敏度分析； （3）它可以有效地捕获溶液中的急剧中断； （4）它消除了对几何和网格的约束，使其适用于任意几何和空间采样（零射击的超分辨率能力）。我们批判性地评估了这些功能，并分析网络在固定和瞬态PDE中概括地看不见样品的能力。所提出的方法的总体表现很有希望，证明了其适用于计算力学中一系列具有挑战性的问题。

Title: MG-MotionLLM: A Unified Framework for Motion Comprehension and Generation across Multiple Granularities

• Authors: Bizhu Wu, Jinheng Xie, Keming Shen, Zhe Kong, Jianfeng Ren, Ruibin Bai, Rong Qu, Linlin Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.02478
• Pdf URL: https://arxiv.org/pdf/2504.02478
• Copy Paste: [[2504.02478]] MG-MotionLLM: A Unified Framework for Motion Comprehension and Generation across Multiple Granularities(https://arxiv.org/abs/2504.02478)
• Keywords: generation
• Abstract: Recent motion-aware large language models have demonstrated promising potential in unifying motion comprehension and generation. However, existing approaches primarily focus on coarse-grained motion-text modeling, where text describes the overall semantics of an entire motion sequence in just a few words. This limits their ability to handle fine-grained motion-relevant tasks, such as understanding and controlling the movements of specific body parts. To overcome this limitation, we pioneer MG-MotionLLM, a unified motion-language model for multi-granular motion comprehension and generation. We further introduce a comprehensive multi-granularity training scheme by incorporating a set of novel auxiliary tasks, such as localizing temporal boundaries of motion segments via detailed text as well as motion detailed captioning, to facilitate mutual reinforcement for motion-text modeling across various levels of granularity. Extensive experiments show that our MG-MotionLLM achieves superior performance on classical text-to-motion and motion-to-text tasks, and exhibits potential in novel fine-grained motion comprehension and editing tasks. Project page: CVI-SZU/MG-MotionLLM
• 摘要：最近的运动感知大型语言模型表明，在统一运动理解和产生方面具有有希望的潜力。但是，现有方法主要集中于粗粒粒度的运动文本建模，其中文本仅用几个词描述了整个运动序列的整体语义。这限制了他们处理相关运动任务的能力，例如理解和控制特定身体部位的运动。为了克服这一限制，我们开拓了MG-Motionllm，这是一种用于多粒子运动理解和产生的统一运动语言模型。我们通过结合一组新颖的辅助任务，例如通过详细文本和运动详细的字幕介绍运动段的时间边界，从而促进相互加固，以促进各种颗粒状的运动模型。广泛的实验表明，我们的MG-Motionllm在经典的文本到动作和文本任务上取得了卓越的表现，并且在新颖的细粒运动理解和编辑任务中表现出潜力。项目页面：CVI-SZU/mg-Motionllm

Title: Group-based Distinctive Image Captioning with Memory Difference Encoding and Attention

• Authors: Jiuniu Wang, Wenjia Xu, Qingzhong Wang, Antoni B. Chan
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2504.02496
• Pdf URL: https://arxiv.org/pdf/2504.02496
• Copy Paste: [[2504.02496]] Group-based Distinctive Image Captioning with Memory Difference Encoding and Attention(https://arxiv.org/abs/2504.02496)
• Keywords: generation
• Abstract: Recent advances in image captioning have focused on enhancing accuracy by substantially increasing the dataset and model size. While conventional captioning models exhibit high performance on established metrics such as BLEU, CIDEr, and SPICE, the capability of captions to distinguish the target image from other similar images is under-explored. To generate distinctive captions, a few pioneers employed contrastive learning or re-weighted the ground-truth captions. However, these approaches often overlook the relationships among objects in a similar image group (e.g., items or properties within the same album or fine-grained events). In this paper, we introduce a novel approach to enhance the distinctiveness of image captions, namely Group-based Differential Distinctive Captioning Method, which visually compares each image with other images in one similar group and highlights the uniqueness of each image. In particular, we introduce a Group-based Differential Memory Attention (GDMA) module, designed to identify and emphasize object features in an image that are uniquely distinguishable within its image group, i.e., those exhibiting low similarity with objects in other images. This mechanism ensures that such unique object features are prioritized during caption generation for the image, thereby enhancing the distinctiveness of the resulting captions. To further refine this process, we select distinctive words from the ground-truth captions to guide both the language decoder and the GDMA module. Additionally, we propose a new evaluation metric, the Distinctive Word Rate (DisWordRate), to quantitatively assess caption distinctiveness. Quantitative results indicate that the proposed method significantly improves the distinctiveness of several baseline models, and achieves state-of-the-art performance on distinctiveness while not excessively sacrificing accuracy...
• 摘要：图像字幕的最新进展集中在通过大大增加数据集和模型大小来提高准确性。尽管常规字幕模型在诸如BLEU，苹果酒和香料之类的已建立指标上表现出很高的性能，但标题将目标图像与其他相似图像区分开的能力尚未探索。为了产生独特的标题，一些先驱使用对比度学习或重新加权地面真相标题。但是，这些方法通常会忽略相似图像组（例如，同一专辑或细粒事件中的项目或属性）中对象之间的关系。在本文中，我们介绍了一种新的方法来增强图像字幕的独特性，即基于组的差异独特字幕方法，该方法将每个图像与一个相似组中的其他图像进行视觉比较，并突出显示每个图像的独特性。特别是，我们介绍了一个基于组的差分记忆注意（GDMA）模块，旨在识别和强调图像中的对象特征，在其图像组中，在其图像组中可以唯一区分的对象特征，即那些与其他图像中对象相似性的较低相似性。该机制可确保在图像的字幕生成期间优先考虑这种独特的对象特征，从而增强结果字幕的独特性。为了进一步完善此过程，我们从基本真相字幕中选择独特的单词来指导语言解码器和GDMA模块。此外，我们提出了一个新的评估指标，即独特的单词率（diswordrate），以定量评估标题独特性。定量结果表明，所提出的方法显着提高了几种基线模型的独特性，并在独特性上实现了最先进的性能，同时又不过分牺牲准确性...

Title: ZClip: Adaptive Spike Mitigation for LLM Pre-Training

• Authors: Abhay Kumar, Louis Owen, Nilabhra Roy Chowdhury, Fabian Güra
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2504.02507
• Pdf URL: https://arxiv.org/pdf/2504.02507
• Copy Paste: [[2504.02507]] ZClip: Adaptive Spike Mitigation for LLM Pre-Training(https://arxiv.org/abs/2504.02507)
• Keywords: restoration
• Abstract: Training large language models (LLMs) presents numerous challenges, including gradient instability and loss spikes. These phenomena can lead to catastrophic divergence, requiring costly checkpoint restoration and data batch skipping. Traditional gradient clipping techniques, such as constant or norm-based methods, fail to address these issues effectively due to their reliance on fixed thresholds or heuristics, leading to inefficient learning and requiring frequent manual intervention. In this work, we propose ZClip, an adaptive gradient clipping algorithm that dynamically adjusts the clipping threshold based on statistical properties of gradient norms over time. Unlike prior reactive strategies, ZClip proactively adapts to training dynamics without making any prior assumptions on the scale and the temporal evolution of gradient norms. At its core, it leverages z-score-based anomaly detection to identify and mitigate large gradient spikes, preventing malignant loss spikes while not interfering with convergence otherwise. Our code is available at: this https URL.
• 摘要：培训大语言模型（LLM）提出了许多挑战，包括梯度不稳定和损失峰值。这些现象可能导致灾难性的差异，需要昂贵的检查点修复和数据批次跳过。传统的梯度裁剪技术，例如恒定或基于规范的方法，由于依赖固定阈值或启发式方法，无法有效解决这些问题，从而导致学习效率低下并需要经常进行手动干预。在这项工作中，我们提出了ZClip，这是一种自适应梯度剪辑算法，该算法会根据梯度规范的统计特性，动态调整剪辑阈值。与先前的反应性策略不同，ZCLIP主动适应了训练动力学，而无需对规模和梯度规范的时间演变做出任何先前的假设。从本质上讲，它利用基于Z分数的异常检测来识别和减轻大梯度尖峰，以防止恶性损失尖峰，而否则不会干扰收敛。我们的代码可用：此HTTPS URL。

Title: Exploration-Driven Generative Interactive Environments

• Authors: Nedko Savov, Naser Kazemi, Mohammad Mahdi, Danda Pani Paudel, Xi Wang, Luc Van Gool
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.02515
• Pdf URL: https://arxiv.org/pdf/2504.02515
• Copy Paste: [[2504.02515]] Exploration-Driven Generative Interactive Environments(https://arxiv.org/abs/2504.02515)
• Keywords: generative
• Abstract: Modern world models require costly and time-consuming collection of large video datasets with action demonstrations by people or by environment-specific agents. To simplify training, we focus on using many virtual environments for inexpensive, automatically collected interaction data. Genie, a recent multi-environment world model, demonstrates simulation abilities of many environments with shared behavior. Unfortunately, training their model requires expensive demonstrations. Therefore, we propose a training framework merely using a random agent in virtual environments. While the model trained in this manner exhibits good controls, it is limited by the random exploration possibilities. To address this limitation, we propose AutoExplore Agent - an exploration agent that entirely relies on the uncertainty of the world model, delivering diverse data from which it can learn the best. Our agent is fully independent of environment-specific rewards and thus adapts easily to new environments. With this approach, the pretrained multi-environment model can quickly adapt to new environments achieving video fidelity and controllability improvement. In order to obtain automatically large-scale interaction datasets for pretraining, we group environments with similar behavior and controls. To this end, we annotate the behavior and controls of 974 virtual environments - a dataset that we name RetroAct. For building our model, we first create an open implementation of Genie - GenieRedux and apply enhancements and adaptations in our version GenieRedux-G. Our code and data are available at this https URL.
• 摘要：现代世界模型需要昂贵且耗时的大型视频数据集，其中包括人们或特定环境特定代理的行动演示。为了简化培训，我们专注于使用许多虚拟环境来廉价，自动收集的交互数据。 Genie是最近的多种环境世界模型，它展示了许多具有共同行为的环境的模拟能力。不幸的是，培训他们的模型需要昂贵的示范。因此，我们仅在虚拟环境中使用随机代理提出一个培训框架。尽管以这种方式训练的模型表现出良好的控制，但它受到随机探索可能性的限制。为了解决这一限制，我们提出了自动塑料代理 - 一种完全依赖世界模型的不确定性的勘探代理，提供了可以从中学习最好的数据。我们的代理完全独立于特定环境的奖励，因此很容易适应新的环境。通过这种方法，预处理的多环境模型可以迅速适应新的环境，从而实现视频保真度和可控性的改善。为了获得自动大规模交互数据集以进行预处理，我们将具有相似行为和对照的环境分组。为此，我们注释了974个虚拟环境的行为和控制，这是我们命名retroact的数据集。为了构建我们的模型，我们首先创建了Genie -Genieredux的开放实现，并在Genieredux -G版本中应用增强和适应。我们的代码和数据可在此HTTPS URL上找到。

Title: Charm: The Missing Piece in ViT fine-tuning for Image Aesthetic Assessment

• Authors: Fatemeh Behrad, Tinne Tuytelaars, Johan Wagemans
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.02522
• Pdf URL: https://arxiv.org/pdf/2504.02522
• Copy Paste: [[2504.02522]] Charm: The Missing Piece in ViT fine-tuning for Image Aesthetic Assessment(https://arxiv.org/abs/2504.02522)
• Keywords: quality assessment
• Abstract: The capacity of Vision transformers (ViTs) to handle variable-sized inputs is often constrained by computational complexity and batch processing limitations. Consequently, ViTs are typically trained on small, fixed-size images obtained through downscaling or cropping. While reducing computational burden, these methods result in significant information loss, negatively affecting tasks like image aesthetic assessment. We introduce Charm, a novel tokenization approach that preserves Composition, High-resolution, Aspect Ratio, and Multi-scale information simultaneously. Charm prioritizes high-resolution details in specific regions while downscaling others, enabling shorter fixed-size input sequences for ViTs while incorporating essential information. Charm is designed to be compatible with pre-trained ViTs and their learned positional embeddings. By providing multiscale input and introducing variety to input tokens, Charm improves ViT performance and generalizability for image aesthetic assessment. We avoid cropping or changing the aspect ratio to further preserve information. Extensive experiments demonstrate significant performance improvements on various image aesthetic and quality assessment datasets (up to 8.1 %) using a lightweight ViT backbone. Code and pre-trained models are available at this https URL.
• 摘要：视觉变压器（VIT）处理可变大小输入的能力通常受到计算复杂性和批处理处理限制的限制。因此，VIT通常是通过通过缩小或裁剪获得的小型固定尺寸图像进行训练的。在减轻计算负担的同时，这些方法会导致大量信息丢失，对图像美学评估等任务产生负面影响。我们介绍了一种新型的令牌化方法，可以同时保留组成，高分辨率，宽高比和多尺度信息。魅力将特定区域的高分辨率细节优先考虑，同时降低其他区域，从而使VIT的固定尺寸输入序列更短，同时合并基本信息。魅力旨在与预训练的VIT及其学习的位置嵌入兼容。通过提供多尺度输入并将多样性引入输入令牌，魅力可以提高VIT性能和图像美学评估的普遍性。我们避免裁剪或改变纵横比以进一步保留信息。广泛的实验表明，使用轻质VIT主链，各种图像美学和质量评估数据集（高达8.1％）的性能改善。代码和预训练的模型可在此HTTPS URL上找到。

Title: Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generation

• Authors: Fa-Ting Hong, Zunnan Xu, Zixiang Zhou, Jun Zhou, Xiu Li, Qin Lin, Qinglin Lu, Dan Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.02542
• Pdf URL: https://arxiv.org/pdf/2504.02542
• Copy Paste: [[2504.02542]] Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generation(https://arxiv.org/abs/2504.02542)
• Keywords: generation
• Abstract: Talking head synthesis is vital for virtual avatars and human-computer interaction. However, most existing methods are typically limited to accepting control from a single primary modality, restricting their practical utility. To this end, we introduce \textbf{ACTalker}, an end-to-end video diffusion framework that supports both multi-signals control and single-signal control for talking head video generation. For multiple control, we design a parallel mamba structure with multiple branches, each utilizing a separate driving signal to control specific facial regions. A gate mechanism is applied across all branches, providing flexible control over video generation. To ensure natural coordination of the controlled video both temporally and spatially, we employ the mamba structure, which enables driving signals to manipulate feature tokens across both dimensions in each branch. Additionally, we introduce a mask-drop strategy that allows each driving signal to independently control its corresponding facial region within the mamba structure, preventing control conflicts. Experimental results demonstrate that our method produces natural-looking facial videos driven by diverse signals and that the mamba layer seamlessly integrates multiple driving modalities without conflict.
• 摘要：说话的头部合成对于虚拟化身和人类计算机相互作用至关重要。但是，大多数现有方法通常仅限于接受单个主要模式的控制，从而限制其实际实用程序。为此，我们介绍了\ textbf {actalker}，这是一个端到端的视频扩散框架，支持多信号控制和单信号控制，以供谈话头视频。对于多个控制，我们设计了带有多个分支的平行曼巴结构，每个结构都使用单独的驱动信号来控制特定的面部区域。在所有分支机构上都应用了栅极机制，从而为视频生成提供了灵活的控制。为了确保受控视频的自然协调在时间和空间上，我们采用了MAMBA结构，该结构使驾驶信号能够操纵每个分支中两个维度的特征令牌。此外，我们引入了一种面具滴定策略，该策略允许每个驾驶信号在曼巴结构内独立控制其相应的面部区域，从而防止控制冲突。实验结果表明，我们的方法会产生由不同信号驱动的自然面部视频，并且Mamba层无缝地集成了多种驾驶方式而不会发生冲突。

Title: Fine-Tuning Visual Autoregressive Models for Subject-Driven Generation

• Authors: Jiwoo Chung, Sangeek Hyun, Hyunjun Kim, Eunseo Koh, MinKyu Lee, Jae-Pil Heo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.02612
• Pdf URL: https://arxiv.org/pdf/2504.02612
• Copy Paste: [[2504.02612]] Fine-Tuning Visual Autoregressive Models for Subject-Driven Generation(https://arxiv.org/abs/2504.02612)
• Keywords: generation, generative
• Abstract: Recent advances in text-to-image generative models have enabled numerous practical applications, including subject-driven generation, which fine-tunes pretrained models to capture subject semantics from only a few examples. While diffusion-based models produce high-quality images, their extensive denoising steps result in significant computational overhead, limiting real-world applicability. Visual autoregressive~(VAR) models, which predict next-scale tokens rather than spatially adjacent ones, offer significantly faster inference suitable for practical deployment. In this paper, we propose the first VAR-based approach for subject-driven generation. However, na\"ıve fine-tuning VAR leads to computational overhead, language drift, and reduced diversity. To address these challenges, we introduce selective layer tuning to reduce complexity and prior distillation to mitigate language drift. Additionally, we found that the early stages have a greater influence on the generation of subject than the latter stages, which merely synthesize local details. Based on this finding, we propose scale-wise weighted tuning, which prioritizes coarser resolutions for promoting the model to focus on the subject-relevant information instead of local details. Extensive experiments validate that our method significantly outperforms diffusion-based baselines across various metrics and demonstrates its practical usage.
• 摘要：文本到图像生成模型的最新进展已实现了许多实用应用，包括主题驱动的生成，该应用微调概括的模型仅从几个示例中捕获主题语义。尽管基于扩散的模型产生了高质量的图像，但它们的广泛降解步骤导致了大量的计算开销，从而限制了现实世界的适用性。视觉自回旋〜（var）模型可预测隔壁代币而不是空间相邻的代币，它提供了适合实际部署的推断的速度。在本文中，我们提出了第一种基于主题驱动的生成的基于VAR的方法。然而，na \“微调VAR会导致计算间接费用，语言漂移和减少的多样性。为了应对这些挑战，我们介绍了选择性的层调整以减少复杂性和以前的蒸馏来减轻语言漂移。此外，我们的早期阶段对早期阶段的影响更大，而不是后来的范围，这些阶段比本地详细概述了，该阶段是在此范围内提出的。促进模型的较高决议，以关注与主题相关的信息，而不是本地细节。

Title: Variational Online Mirror Descent for Robust Learning in Schrödinger Bridge

• Authors: Dong-Sig Han, Jaein Kim, Hee Bin Yoo, Byoung-Tak Zhang
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2504.02618
• Pdf URL: https://arxiv.org/pdf/2504.02618
• Copy Paste: [[2504.02618]] Variational Online Mirror Descent for Robust Learning in Schrödinger Bridge(https://arxiv.org/abs/2504.02618)
• Keywords: generative
• Abstract: Schödinger bridge (SB) has evolved into a universal class of probabilistic generative models. In practice, however, estimated learning signals are often uncertain, and the reliability promised by existing methods is often based on speculative optimal-case scenarios. Recent studies regarding the Sinkhorn algorithm through mirror descent (MD) have gained attention, revealing geometric insights into solution acquisition of the SB problems. In this paper, we propose a variational online MD (OMD) framework for the SB problems, which provides further stability to SB solvers. We formally prove convergence and a regret bound for the novel OMD formulation of SB acquisition. As a result, we propose a simulation-free SB algorithm called Variational Mirrored Schrödinger Bridge (VMSB) by utilizing the Wasserstein-Fisher-Rao geometry of the Gaussian mixture parameterization for Schrödinger potentials. Based on the Wasserstein gradient flow theory, the algorithm offers tractable learning dynamics that precisely approximate each OMD step. In experiments, we validate the performance of the proposed VMSB algorithm across an extensive suite of benchmarks. VMSB consistently outperforms contemporary SB solvers on a range of SB problems, demonstrating the robustness predicted by our theory.
• 摘要：Schödinger桥（SB）已演变为一类普遍的概率生成模型。但是，实际上，估计的学习信号通常是不确定的，现有方法承诺的可靠性通常基于投机性最佳案例场景。关于通过镜下下降（MD）的凹痕算法的最新研究引起了人们的注意，揭示了对SB问题的溶液获取的几何见解。在本文中，我们为SB问题提出了一个差异在线MD（OMD）框架，该框架为SB求解器提供了进一步的稳定性。我们正式证明了融合和遗憾，即SB采集的新型OMD表述。结果，我们提出了一种无模拟的SB算法，称为变异镜像Schrödinger桥（VMSB），利用Gaussian混合物参数化的schrödinger电位的Wasserstein-fisher-fisher-rao几何形状。基于Wasserstein梯度流理论，该算法提供了可拖动的学习动力学，可以准确地近似每个OMD步骤。在实验中，我们验证了在广泛的基准套件中验证了所提出的VMSB算法的性能。 VMSB在一系列SB问题上始终优于当代SB求解器，这证明了我们的理论所预测的鲁棒性。

Title: MD-ProjTex: Texturing 3D Shapes with Multi-Diffusion Projection

• Authors: Ahmet Burak Yildirim, Mustafa Utku Aydogdu, Duygu Ceylan, Aysegul Dundar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.02762
• Pdf URL: https://arxiv.org/pdf/2504.02762
• Copy Paste: [[2504.02762]] MD-ProjTex: Texturing 3D Shapes with Multi-Diffusion Projection(https://arxiv.org/abs/2504.02762)
• Keywords: generation
• Abstract: We introduce MD-ProjTex, a method for fast and consistent text-guided texture generation for 3D shapes using pretrained text-to-image diffusion models. At the core of our approach is a multi-view consistency mechanism in UV space, which ensures coherent textures across different viewpoints. Specifically, MD-ProjTex fuses noise predictions from multiple views at each diffusion step and jointly updates the per-view denoising directions to maintain 3D consistency. In contrast to existing state-of-the-art methods that rely on optimization or sequential view synthesis, MD-ProjTex is computationally more efficient and achieves better quantitative and qualitative results.
• 摘要：我们介绍了MD-ProjTex，这是一种使用验证的文本对图像扩散模型，用于3D形状的快速，一致的文本制定纹理生成的方法。我们方法的核心是紫外线空间中的多视图一致性机制，该机制可确保跨不同观点的连贯纹理。具体而言，MD-ProjTex在每个扩散步骤中从多个视图中融合了噪声预测，并共同更新了视图降级方向以保持3D一致性。与依靠优化或顺序视图合成的现有最新方法相反，MD-ProjTex在计算上更有效，并且可以获得更好的定量和定性结果。

Title: Scene Splatter: Momentum 3D Scene Generation from Single Image with Video Diffusion Model

• Authors: Shengjun Zhang, Jinzhao Li, Xin Fei, Hao Liu, Yueqi Duan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2504.02764
• Pdf URL: https://arxiv.org/pdf/2504.02764
• Copy Paste: [[2504.02764]] Scene Splatter: Momentum 3D Scene Generation from Single Image with Video Diffusion Model(https://arxiv.org/abs/2504.02764)
• Keywords: generation, generative
• Abstract: In this paper, we propose Scene Splatter, a momentum-based paradigm for video diffusion to generate generic scenes from single image. Existing methods, which employ video generation models to synthesize novel views, suffer from limited video length and scene inconsistency, leading to artifacts and distortions during further reconstruction. To address this issue, we construct noisy samples from original features as momentum to enhance video details and maintain scene consistency. However, for latent features with the perception field that spans both known and unknown regions, such latent-level momentum restricts the generative ability of video diffusion in unknown regions. Therefore, we further introduce the aforementioned consistent video as a pixel-level momentum to a directly generated video without momentum for better recovery of unseen regions. Our cascaded momentum enables video diffusion models to generate both high-fidelity and consistent novel views. We further finetune the global Gaussian representations with enhanced frames and render new frames for momentum update in the next step. In this manner, we can iteratively recover a 3D scene, avoiding the limitation of video length. Extensive experiments demonstrate the generalization capability and superior performance of our method in high-fidelity and consistent scene generation.
• 摘要：在本文中，我们提出了场景飞溅，这是一种基于动量的范式，用于视频扩散，以从单个图像中生成通用场景。现有的方法采用视频生成模型来综合新观点，遭受视频长度和场景不一致的有限，导致在进一步重建过程中导致伪影和扭曲。为了解决这个问题，我们构建了来自原始功能的嘈杂样本作为增强视频细节并保持场景一致性的动量。但是，对于具有跨越已知区域和未知区域的感知场的潜在特征，这种潜在动量限制了视频扩散在未知区域中的生成能力。因此，我们进一步将上述一致的视频作为像素级动量引入了直接生成的视频，而无需动力以更好地恢复看不见的区域。我们的级联动量使视频扩散模型能够产生高保真和一致的新颖观点。我们通过增强的框架进一步对全球高斯表示，并在下一步中进行动量更新。通过这种方式，我们可以迭代恢复3D场景，避免视频长度的限制。广泛的实验证明了我们方法在高保真性和一致的场景产生中的概括能力和出色的性能。

Title: GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generation

• Authors: Zhiyuan Yan, Junyan Ye, Weijia Li, Zilong Huang, Shenghai Yuan, Xiangyang He, Kaiqing Lin, Jun He, Conghui He, Li Yuan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.02782
• Pdf URL: https://arxiv.org/pdf/2504.02782
• Copy Paste: [[2504.02782]] GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generation(https://arxiv.org/abs/2504.02782)
• Keywords: generation
• Abstract: The recent breakthroughs in OpenAI's GPT4o model have demonstrated surprisingly good capabilities in image generation and editing, resulting in significant excitement in the community. This technical report presents the first-look evaluation benchmark (named GPT-ImgEval), quantitatively and qualitatively diagnosing GPT-4o's performance across three critical dimensions: (1) generation quality, (2) editing proficiency, and (3) world knowledge-informed semantic synthesis. Across all three tasks, GPT-4o demonstrates strong performance, significantly surpassing existing methods in both image generation control and output quality, while also showcasing exceptional knowledge reasoning capabilities. Furthermore, based on the GPT-4o's generated data, we propose a classification-model-based approach to investigate the underlying architecture of GPT-4o, where our empirical results suggest the model consists of an auto-regressive (AR) combined with a diffusion-based head for image decoding, rather than the VAR-like architectures. We also provide a complete speculation on GPT-4o's overall architecture. In addition, we conduct a series of analyses to identify and visualize GPT-4o's specific limitations and the synthetic artifacts commonly observed in its image generation. We also present a comparative study of multi-round image editing between GPT-4o and Gemini 2.0 Flash, and discuss the safety implications of GPT-4o's outputs, particularly their detectability by existing image forensic models. We hope that our work can offer valuable insight and provide a reliable benchmark to guide future research, foster reproducibility, and accelerate innovation in the field of image generation and beyond. The codes and datasets used for evaluating GPT-4o can be found at this https URL.
• 摘要：OpenAI的GPT4O模型的最新突破表现出了令人惊讶的良好能力在图像生成和编辑中，从而引起了社区的极大兴奋。该技术报告介绍了第一观评估基准（命名为GPT-Imgeval），在三个关键维度上进行了定量和质量地诊断GPT-4O的性能：（1）生成质量，（2）编辑能力和（3）世界知识知识的语义合成。在这三个任务中，GPT-4O表现出强劲的性能，在图像生成控制和输出质量中都显着超过了现有方法，同时还展示了出色的知识推理能力。此外，根据GPT-4O生成的数据，我们提出了一种基于分类模型的方法来研究GPT-4O的潜在架构，在该方法中，我们的经验结果表明，该模型由自动回归（AR）与基于扩散的头部组成，以用于图像解码，而不是变化型体系模型。我们还对GPT-4O的整体体系结构提供了一个完整的猜测。此外，我们进行了一系列分析，以识别和可视化GPT-4O的特定局限性以及其图像产生中通常观察到的合成伪像。我们还介绍了GPT-4O和Gemini 2.0 Flash之间多轮图像编辑的比较研究，并讨论了GPT-4O输出的安全性，尤其是通过现有图像法医模型的可检测性。我们希望我们的工作能够提供宝贵的见解，并提供可靠的基准，以指导未来的研究，培养可重复性以及在图像生成及其他领域加速创新。可以在此HTTPS URL上找到用于评估GPT-4O的代码和数据集。

Title: F-ViTA: Foundation Model Guided Visible to Thermal Translation

• Authors: Jay N. Paranjape, Celso de Melo, Vishal M. Patel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.02801
• Pdf URL: https://arxiv.org/pdf/2504.02801
• Copy Paste: [[2504.02801]] F-ViTA: Foundation Model Guided Visible to Thermal Translation(https://arxiv.org/abs/2504.02801)
• Keywords: generative
• Abstract: Thermal imaging is crucial for scene understanding, particularly in low-light and nighttime conditions. However, collecting large thermal datasets is costly and labor-intensive due to the specialized equipment required for infrared image capture. To address this challenge, researchers have explored visible-to-thermal image translation. Most existing methods rely on Generative Adversarial Networks (GANs) or Diffusion Models (DMs), treating the task as a style transfer problem. As a result, these approaches attempt to learn both the modality distribution shift and underlying physical principles from limited training data. In this paper, we propose F-ViTA, a novel approach that leverages the general world knowledge embedded in foundation models to guide the diffusion process for improved translation. Specifically, we condition an InstructPix2Pix Diffusion Model with zero-shot masks and labels from foundation models such as SAM and Grounded DINO. This allows the model to learn meaningful correlations between scene objects and their thermal signatures in infrared imagery. Extensive experiments on five public datasets demonstrate that F-ViTA outperforms state-of-the-art (SOTA) methods. Furthermore, our model generalizes well to out-of-distribution (OOD) scenarios and can generate Long-Wave Infrared (LWIR), Mid-Wave Infrared (MWIR), and Near-Infrared (NIR) translations from the same visible image. Code: this https URL.
• 摘要：热成像对于场景理解至关重要，尤其是在弱光和夜间条件下。但是，由于红外图像捕获所需的专门设备，收集大型热数据集是昂贵且劳动力密集的。为了应对这一挑战，研究人员探索了可见的对热图像翻译。大多数现有方法依赖于生成的对抗网络（GAN）或扩散模型（DMS），将任务视为样式转移问题。结果，这些方法试图从有限的培训数据中学习模态分布转移和基本的物理原理。在本文中，我们提出了F-Vita，这是一种新型的方法，它利用基础模型中嵌入的一般世界知识来指导扩散过程以改善翻译。具体而言，我们可以通过零摄像面罩和诸如SAM和接地Dino等基础模型的标签进行指示PPIX2PIX扩散模型。这使模型可以学习场景对象与红外图像中的热签名之间的有意义的相关性。在五个公共数据集上进行的广泛实验表明，F-VITA的表现优于最先进的方法（SOTA）方法。此外，我们的模型可以很好地概括到分布（OOD）方案，并可以产生长波红外（LWIR），中波红外（MWIR）和近红外（NIR）转换，并从相同的可见图像中产生近红外（NIR）翻译。代码：此HTTPS URL。

Title: Efficient Autoregressive Shape Generation via Octree-Based Adaptive Tokenization

• Authors: Kangle Deng, Hsueh-Ti Derek Liu, Yiheng Zhu, Xiaoxia Sun, Chong Shang, Kiran Bhat, Deva Ramanan, Jun-Yan Zhu, Maneesh Agrawala, Tinghui Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.02817
• Pdf URL: https://arxiv.org/pdf/2504.02817
• Copy Paste: [[2504.02817]] Efficient Autoregressive Shape Generation via Octree-Based Adaptive Tokenization(https://arxiv.org/abs/2504.02817)
• Keywords: generation, generative
• Abstract: Many 3D generative models rely on variational autoencoders (VAEs) to learn compact shape representations. However, existing methods encode all shapes into a fixed-size token, disregarding the inherent variations in scale and complexity across 3D data. This leads to inefficient latent representations that can compromise downstream generation. We address this challenge by introducing Octree-based Adaptive Tokenization, a novel framework that adjusts the dimension of latent representations according to shape complexity. Our approach constructs an adaptive octree structure guided by a quadric-error-based subdivision criterion and allocates a shape latent vector to each octree cell using a query-based transformer. Building upon this tokenization, we develop an octree-based autoregressive generative model that effectively leverages these variable-sized representations in shape generation. Extensive experiments demonstrate that our approach reduces token counts by 50% compared to fixed-size methods while maintaining comparable visual quality. When using a similar token length, our method produces significantly higher-quality shapes. When incorporated with our downstream generative model, our method creates more detailed and diverse 3D content than existing approaches.
• 摘要：许多3D生成模型都依靠变异自动编码器（VAE）来学习紧凑的形状表示。但是，现有方法将所有形状编码为固定大小的令牌，无视3D数据中固有的尺度和复杂性变化。这会导致潜在的潜在表示，可能会损害下游生成。我们通过引入基于OCTREE的自适应令牌化来应对这一挑战，这是一个新颖的框架，可根据形状复杂性调整潜在表示的维度。我们的方法构建了一种自适应的OCTREE结构，该结构由基于二次 - 纠正的细分标准引导，并使用基于查询的变压器将形状潜在向量分配给每个OctRee单元。在这种令牌化的基础上，我们开发了一种基于OCTREE的自回归生成模型，该模型有效地利用了这些可变大小的表示形式。广泛的实验表明，与固定尺寸的方法相比，我们的方法可将令牌计数降低50％，同时保持可比的视觉质量。当使用类似的令牌长度时，我们的方法会产生明显的更高质量的形状。与我们的下游生成模型合并时，我们的方法比现有方法创造了更详细和多样的3D内容。

Title: Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing

• Authors: Xiangyu Zhao, Peiyuan Zhang, Kexian Tang, Hao Li, Zicheng Zhang, Guangtao Zhai, Junchi Yan, Hua Yang, Xue Yang, Haodong Duan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2504.02826
• Pdf URL: https://arxiv.org/pdf/2504.02826
• Copy Paste: [[2504.02826]] Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editing(https://arxiv.org/abs/2504.02826)
• Keywords: generation
• Abstract: Large Multi-modality Models (LMMs) have made significant progress in visual understanding and generation, but they still face challenges in General Visual Editing, particularly in following complex instructions, preserving appearance consistency, and supporting flexible input formats. To address this gap, we introduce RISEBench, the first benchmark for evaluating Reasoning-Informed viSual Editing (RISE). RISEBench focuses on four key reasoning types: Temporal, Causal, Spatial, and Logical Reasoning. We curate high-quality test cases for each category and propose an evaluation framework that assesses Instruction Reasoning, Appearance Consistency, and Visual Plausibility with both human judges and an LMM-as-a-judge approach. Our experiments reveal that while GPT-4o-Native significantly outperforms other open-source and proprietary models, even this state-of-the-art system struggles with logical reasoning tasks, highlighting an area that remains underexplored. As an initial effort, RISEBench aims to provide foundational insights into reasoning-aware visual editing and to catalyze future research. Though still in its early stages, we are committed to continuously expanding and refining the benchmark to support more comprehensive, reliable, and scalable evaluations of next-generation multimodal systems. Our code and data will be released at this https URL.
• 摘要：大型多模式模型（LMM）在视觉理解和产生方面取得了重大进展，但它们在一般视觉编辑中仍然面临挑战，尤其是在遵循复杂的说明，保持外观一致性以及支持灵活的输入格式时。为了解决这一差距，我们介绍了RiseBench，这是评估推理信息的视觉编辑（RISE）的第一个基准。 RiseBench专注于四种关键推理类型：时间，因果，空间和逻辑推理。我们为每个类别策划了高质量的测试用例，并提出了一个评估框架，该框架评估了与人类法官和LMM-AS-AS-A-A-Gudge方法评估教学推理，外观一致性和视觉合理性。我们的实验表明，尽管GPT-4O-NONITAING显着优于其他开源和专有模型，但即使是这种最新的系统在逻辑推理任务上也努力努力，突出了一个尚未被忽略的领域。作为最初的努力，RiseBench旨在提供有关推理意识到的视觉编辑并催化未来研究的基本见解。尽管仍处于早期阶段，但我们致力于不断扩大和完善基准，以支持对下一代多模式系统的更全面，可靠和可扩展的评估。我们的代码和数据将在此HTTPS URL上发布。