2025-05-15

Title: Multi-modal Synthetic Data Training and Model Collapse: Insights from VLMs and Diffusion Models

• Authors: Zizhao Hu, Mohammad Rostami, Jesse Thomason
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.08803
• Pdf URL: https://arxiv.org/pdf/2505.08803
• Copy Paste: [[2505.08803]] Multi-modal Synthetic Data Training and Model Collapse: Insights from VLMs and Diffusion Models(https://arxiv.org/abs/2505.08803)
• Keywords: generative
• Abstract: Recent research has highlighted the risk of generative model collapse, where performance progressively degrades when continually trained on self-generated data. However, existing exploration on model collapse is limited to single, unimodal models, limiting our understanding in more realistic scenarios, such as diverse multi-modal AI agents interacting autonomously through synthetic data and continually evolving. We expand the synthetic data training and model collapse study to multi-modal vision-language generative systems, such as vision-language models (VLMs) and text-to-image diffusion models, as well as recursive generate-train loops with multiple models. We find that model collapse, previously observed in single-modality generative models, exhibits distinct characteristics in the multi-modal context, such as improved vision-language alignment and increased variance in VLM image-captioning task. Additionally, we find that general approaches such as increased decoding budgets, greater model diversity, and relabeling with frozen models can effectively mitigate model collapse. Our findings provide initial insights and practical guidelines for reducing the risk of model collapse in self-improving multi-agent AI systems and curating robust multi-modal synthetic datasets.
• 摘要：最近的研究强调了生成模型崩溃的风险，在不断接受自生数据的培训时，性能会逐渐降低。但是，现有对模型崩溃的探索仅限于单峰模型，从而限制了我们在更现实的场景中的理解，例如通过合成数据自主相互作用并不断发展的多种多模式AI代理。我们将合成数据训练和模型崩溃研究扩展到多模式视觉生成系统，例如视觉语言模型（VLM）和文本对图像扩散模型，以及具有多个模型的递归生成训练环。我们发现，以前在单模式生成模型中观察到的模型崩溃在多模式上下文中表现出不同的特征，例如改善的视觉和语言比对和VLM图像捕获任务的差异增加。此外，我们发现一般方法，例如增加的解码预算，更大的模型多样性以及与冷冻模型进行重新标记可以有效地减轻模型崩溃。我们的发现提供了最初的见解和实用指南，以降低自我改善的多代理AI系统中模型崩溃的风险并策划强大的多模式合成数据集。

Title: Towards SFW sampling for diffusion models via external conditioning

• Authors: Camilo Carvajal Reyes, Joaquín Fontbona, Felipe Tobar
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.08817
• Pdf URL: https://arxiv.org/pdf/2505.08817
• Copy Paste: [[2505.08817]] Towards SFW sampling for diffusion models via external conditioning(https://arxiv.org/abs/2505.08817)
• Keywords: generation, generative
• Abstract: Score-based generative models (SBM), also known as diffusion models, are the de facto state of the art for image synthesis. Despite their unparalleled performance, SBMs have recently been in the spotlight for being tricked into creating not-safe-for-work (NSFW) content, such as violent images and non-consensual nudity. Current approaches that prevent unsafe generation are based on the models' own knowledge, and the majority of them require fine-tuning. This article explores the use of external sources for ensuring safe outputs in SBMs. Our safe-for-work (SFW) sampler implements a Conditional Trajectory Correction step that guides the samples away from undesired regions in the ambient space using multimodal models as the source of conditioning. Furthermore, using Contrastive Language Image Pre-training (CLIP), our method admits user-defined NSFW classes, which can vary in different settings. Our experiments on the text-to-image SBM Stable Diffusion validate that the proposed SFW sampler effectively reduces the generation of explicit content while being competitive with other fine-tuning-based approaches, as assessed via independent NSFW detectors. Moreover, we evaluate the impact of the SFW sampler on image quality and show that the proposed correction scheme comes at a minor cost with negligible effect on samples not needing correction. Our study confirms the suitability of the SFW sampler towards aligned SBM models and the potential of using model-agnostic conditioning for the prevention of unwanted images.
• 摘要：基于得分的生成模型（SBM），也称为扩散模型，是图像合成的事实​​状态。尽管SBM的表现无与伦比，但最近还是被诱惑创建非安全工作（NSFW）内容（例如暴力图像和非自愿裸体）的焦点。当前预防不安全产生的方法是基于模型自己的知识，其中大多数都需要微调。本文探讨了使用外部资源来确保SBMS中安全输出的使用。我们的安全工作（SFW）采样器实现了有条件的轨迹校正步骤，该步骤将样品从环境空间中使用多模型模型作为条件来源。此外，使用对比语言图像预训练（剪辑），我们的方法允许用户定义的NSFW类，这些类可能在不同的设置中变化。我们对文本到图像SBM稳定扩散的实验验证了所提出的SFW采样器有效地减少了显式内容的产生，同时与其他基于微调的方法具有竞争力，如通过独立的NSFW检测器评估。此外，我们评估了SFW采样器对图像质量的影响，并表明拟议的校正方案的成本较小，对不需要校正的样品的影响微不足道。我们的研究证实了SFW采样器对对齐的SBM模型的适用性，以及使用模型敏锐的条件来预防不需要的图像的潜力。

Title: Self Rewarding Self Improving

• Authors: Toby Simonds, Kevin Lopez, Akira Yoshiyama, Dominique Garmier
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.08827
• Pdf URL: https://arxiv.org/pdf/2505.08827
• Copy Paste: [[2505.08827]] Self Rewarding Self Improving(https://arxiv.org/abs/2505.08827)
• Keywords: generation
• Abstract: We demonstrate that large language models can effectively self-improve through self-judging without requiring reference solutions, leveraging the inherent asymmetry between generating and verifying solutions. Our experiments on Countdown puzzles and MIT Integration Bee problems show that models can provide reliable reward signals without ground truth answers, enabling reinforcement learning in domains previously not possible. By implementing self-judging, we achieve significant performance gains maintaining alignment with formal verification. When combined with synthetic question generation, we establish a complete self-improvement loop where models generate practice problems, solve them, and evaluate their own performance-achieving an 8% improvement with Qwen 2.5 7B over baseline and surpassing GPT-4o performance on integration tasks. Our findings demonstrate that LLM judges can provide effective reward signals for training models, unlocking many reinforcement learning environments previously limited by the difficulty of creating programmatic rewards. This suggests a potential paradigm shift toward AI systems that continuously improve through self-directed learning rather than human-guided training, potentially accelerating progress in domains with scarce training data or complex evaluation requirements.
• 摘要：我们证明，大型语言模型可以通过自我判断而无需参考解决方案有效地自我爆发，从而利用生成和验证解决方案之间的固有不对称性。我们在倒计时难题和麻省理工学院集成蜜蜂问题上的实验表明，模型可以提供可靠的奖励信号，而无需地面真相答案，从而在以前无法实现的域中进行加强学习。通过实施自我判断，我们实现了与正式验证保持一致性的显着绩效提高。当与综合问题生成结合使用时，我们建立了一个完整的自我完善循环，模型会产生实践问题，解决它们并评估自己的性能方面的8％，而QWEN 2.5 7B比基线比基线超过了GPT-4O的绩效，并且在集成任务上超过了GPT-4O的性能。我们的发现表明，LLM法官可以为培训模型提供有效的奖励信号，从而解除了以前受到创造程序奖励的困难限制的许多强化学习环境。这表明，潜在的范式向AI系统转变，这些系统通过自我指导的学习而不是人类指导的培训不断改进，并有可能加速具有稀缺培训数据或复杂评估要求的域中的进度。

Title: Generative AI for Urban Planning: Synthesizing Satellite Imagery via Diffusion Models

• Authors: Qingyi Wang, Yuebing Liang, Yunhan Zheng, Kaiyuan Xu, Jinhua Zhao, Shenhao Wang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.08833
• Pdf URL: https://arxiv.org/pdf/2505.08833
• Copy Paste: [[2505.08833]] Generative AI for Urban Planning: Synthesizing Satellite Imagery via Diffusion Models(https://arxiv.org/abs/2505.08833)
• Keywords: generation, generative
• Abstract: Generative AI offers new opportunities for automating urban planning by creating site-specific urban layouts and enabling flexible design exploration. However, existing approaches often struggle to produce realistic and practical designs at scale. Therefore, we adapt a state-of-the-art Stable Diffusion model, extended with ControlNet, to generate high-fidelity satellite imagery conditioned on land use descriptions, infrastructure, and natural environments. To overcome data availability limitations, we spatially link satellite imagery with structured land use and constraint information from OpenStreetMap. Using data from three major U.S. cities, we demonstrate that the proposed diffusion model generates realistic and diverse urban landscapes by varying land-use configurations, road networks, and water bodies, facilitating cross-city learning and design diversity. We also systematically evaluate the impacts of varying language prompts and control imagery on the quality of satellite imagery generation. Our model achieves high FID and KID scores and demonstrates robustness across diverse urban contexts. Qualitative assessments from urban planners and the general public show that generated images align closely with design descriptions and constraints, and are often preferred over real images. This work establishes a benchmark for controlled urban imagery generation and highlights the potential of generative AI as a tool for enhancing planning workflows and public engagement.
• 摘要：Generative AI通过创建特定地点的城市布局并实现灵活的设计探索，为自动化城市规划提供了新的机会。但是，现有的方法通常很难按大规模生产现实和实用的设计。因此，我们适应了使用ControlNet扩展的最先进的稳定扩散模型，以生成以土地使用描述，基础设施和自然环境为条件的高保真卫星图像。为了克服数据可用性限制，我们将卫星图像与openstreetMap的结构化土地使用和约束信息联系起来。使用来自美国三个主要城市的数据，我们证明了拟议的扩散模型通过改变土地利用配置，道路网络和水域来产生现实和多样化的城市景观，从而促进跨城学习和设计多样性。我们还系统地评估了不同语言提示和控制图像对卫星图像产生质量的影响。我们的模型达到了高FID和KID的得分，并展示了各种城市环境中的鲁棒性。城市规划师和公众的定性评估表明，生成的图像与设计描述和约束紧密相符，并且通常比真实图像更喜欢。这项工作为受控的城市图像产生建立了基准，并突出了生成AI作为增强计划工作流程和公众参与的工具的潜力。

Title: Generative AI for Autonomous Driving: Frontiers and Opportunities

• Authors: Yuping Wang, Shuo Xing, Cui Can, Renjie Li, Hongyuan Hua, Kexin Tian, Zhaobin Mo, Xiangbo Gao, Keshu Wu, Sulong Zhou, Hengxu You, Juntong Peng, Junge Zhang, Zehao Wang, Rui Song, Mingxuan Yan, Walter Zimmer, Xingcheng Zhou, Peiran Li, Zhaohan Lu, Chia-Ju Chen, Yue Huang, Ryan A. Rossi, Lichao Sun, Hongkai Yu, Zhiwen Fan, Frank Hao Yang, Yuhao Kang, Ross Greer, Chenxi Liu, Eun Hak Lee, Xuan Di, Xinyue Ye, Liu Ren, Alois Knoll, Xiaopeng Li, Shuiwang Ji, Masayoshi Tomizuka, Marco Pavone, Tianbao Yang, Jing Du, Ming-Hsuan Yang, Hua Wei, Ziran Wang, Yang Zhou, Jiachen Li, Zhengzhong Tu
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2505.08854
• Pdf URL: https://arxiv.org/pdf/2505.08854
• Copy Paste: [[2505.08854]] Generative AI for Autonomous Driving: Frontiers and Opportunities(https://arxiv.org/abs/2505.08854)
• Keywords: generation, generative
• Abstract: Generative Artificial Intelligence (GenAI) constitutes a transformative technological wave that reconfigures industries through its unparalleled capabilities for content creation, reasoning, planning, and multimodal understanding. This revolutionary force offers the most promising path yet toward solving one of engineering's grandest challenges: achieving reliable, fully autonomous driving, particularly the pursuit of Level 5 autonomy. This survey delivers a comprehensive and critical synthesis of the emerging role of GenAI across the autonomous driving stack. We begin by distilling the principles and trade-offs of modern generative modeling, encompassing VAEs, GANs, Diffusion Models, and Large Language Models (LLMs). We then map their frontier applications in image, LiDAR, trajectory, occupancy, video generation as well as LLM-guided reasoning and decision making. We categorize practical applications, such as synthetic data workflows, end-to-end driving strategies, high-fidelity digital twin systems, smart transportation networks, and cross-domain transfer to embodied AI. We identify key obstacles and possibilities such as comprehensive generalization across rare cases, evaluation and safety checks, budget-limited implementation, regulatory compliance, ethical concerns, and environmental effects, while proposing research plans across theoretical assurances, trust metrics, transport integration, and socio-technical influence. By unifying these threads, the survey provides a forward-looking reference for researchers, engineers, and policymakers navigating the convergence of generative AI and advanced autonomous mobility. An actively maintained repository of cited works is available at this https URL.
• 摘要：生成人工智能（Genai）构成了一种变革性的技术浪潮，该浪潮通过其无与伦比的内容来创建内容，推理，计划和多模式理解来重新配置行业。这种革命力量为解决工程最大的挑战之一提供了最有前途的途径：实现可靠，完全自主的驾驶，尤其是追求5级自治的驾驶。这项调查提供了Genai在自主驾驶堆栈中新兴作用的全面和关键综合。我们首先要提炼现代生成建模的原理和权衡，包括VAE，gan，扩散模型和大型语言模型（LLMS）。然后，我们将其前沿应用程序映射到图像，激光雷达，轨迹，占用率，视频产生以及LLM引导的推理和决策中。我们对实用应用程序进行了分类，例如合成数据工作流程，端到端驾驶策略，高保真数字双胞胎系统，智能运输网络以及跨域转移到体现的AI。我们确定了关键的障碍和可能性，例如在极少数情况下进行全面概括，评估和安全检查，预算有限的实施，法规合规性，道德问题和环境影响，同时在理论保证，信托指标，运输整合和社会技术影响方面提出了研究计划。通过统一这些线程，该调查为研究人员，工程师和政策制定者提供了前瞻性参考，从而导致了生成AI和先进的自动启动性的收敛性。该HTTPS URL可用一个积极维护的引用作品存储库。

Title: Learning Cocoercive Conservative Denoisers via Helmholtz Decomposition for Poisson Inverse Problems

• Authors: Deliang Wei, Peng Chen, Haobo Xu, Jiale Yao, Fang Li, Tieyong Zeng
• Subjects: cs.CV, cs.LG, math.FA, math.OC
• Abstract URL: https://arxiv.org/abs/2505.08909
• Pdf URL: https://arxiv.org/pdf/2505.08909
• Copy Paste: [[2505.08909]] Learning Cocoercive Conservative Denoisers via Helmholtz Decomposition for Poisson Inverse Problems(https://arxiv.org/abs/2505.08909)
• Keywords: restoration
• Abstract: Plug-and-play (PnP) methods with deep denoisers have shown impressive results in imaging problems. They typically require strong convexity or smoothness of the fidelity term and a (residual) non-expansive denoiser for convergence. These assumptions, however, are violated in Poisson inverse problems, and non-expansiveness can hinder denoising performance. To address these challenges, we propose a cocoercive conservative (CoCo) denoiser, which may be (residual) expansive, leading to improved denoising. By leveraging the generalized Helmholtz decomposition, we introduce a novel training strategy that combines Hamiltonian regularization to promote conservativeness and spectral regularization to ensure cocoerciveness. We prove that CoCo denoiser is a proximal operator of a weakly convex function, enabling a restoration model with an implicit weakly convex prior. The global convergence of PnP methods to a stationary point of this restoration model is established. Extensive experimental results demonstrate that our approach outperforms closely related methods in both visual quality and quantitative metrics.
• 摘要：带有深层DeNoiser的插头播放方法（PNP）在成像问题中显示出令人印象深刻的结果。他们通常需要富有凸度的凸度或平滑度，而（残留）非企业DeNoiser进行收敛。但是，这些假设在泊松逆问题中受到侵犯，而非表达性会阻碍性能。为了应对这些挑战，我们提出了一个可可保守的（可可）Denoiser，这可能是（残留）膨胀，从而改善了DeNosing。通过利用广义的Helmholtz分解，我们引入了一种新颖的训练策略，该策略结合了哈密顿式的正则化，以促进保守性和频谱正则化，以确保可熟练。我们证明可可denoiser是弱凸功能的近端运算符，从而实现了具有隐式弱凸面的恢复模型。建立了PNP方法与该恢复模型的固定点的全球收敛。广泛的实验结果表明，我们的方法在视觉质量和定量指标上的表现都超过了密切相关的方法。

Title: Generating time-consistent dynamics with discriminator-guided image diffusion models

• Authors: Philipp Hess, Maximilian Gelbrecht, Christof Schötz, Michael Aich, Yu Huang, Shangshang Yang, Niklas Boers
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.09089
• Pdf URL: https://arxiv.org/pdf/2505.09089
• Copy Paste: [[2505.09089]] Generating time-consistent dynamics with discriminator-guided image diffusion models(https://arxiv.org/abs/2505.09089)
• Keywords: generation
• Abstract: Realistic temporal dynamics are crucial for many video generation, processing and modelling applications, e.g. in computational fluid dynamics, weather prediction, or long-term climate simulations. Video diffusion models (VDMs) are the current state-of-the-art method for generating highly realistic dynamics. However, training VDMs from scratch can be challenging and requires large computational resources, limiting their wider application. Here, we propose a time-consistency discriminator that enables pretrained image diffusion models to generate realistic spatiotemporal dynamics. The discriminator guides the sampling inference process and does not require extensions or finetuning of the image diffusion model. We compare our approach against a VDM trained from scratch on an idealized turbulence simulation and a real-world global precipitation dataset. Our approach performs equally well in terms of temporal consistency, shows improved uncertainty calibration and lower biases compared to the VDM, and achieves stable centennial-scale climate simulations at daily time steps.
• 摘要：现实的时间动态对于许多视频生成，处理和建模应用至关重要，例如在计算流体动力学，天气预测或长期气候模拟中。视频扩散模型（VDMS）是生成高度逼真动态的当前最新方法。但是，从头开始培训VDM可能具有挑战性，需要大量的计算资源，从而限制其更广泛的应用。在这里，我们提出了一个时间矛盾歧视器，该歧视器可以实现验证的图像扩散模型，以生成逼真的时空动力学。鉴别器指导抽样推理过程，不需要扩展或填充图像扩散模型。我们将我们的方法与在理想化的湍流模拟和现实世界全球降水数据集中从头开始训练的VDM进行了比较。与VDM相比，我们的方法在时间一致性方面的表现同样好，不确定性校准和偏差较低，并且在每日时间步骤中实现稳定的百年纪念气候模拟。

Title: TopoDiT-3D: Topology-Aware Diffusion Transformer with Bottleneck Structure for 3D Point Cloud Generation

• Authors: Zechao Guan, Feng Yan, Shuai Du, Lin Ma, Qingshan Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.09140
• Pdf URL: https://arxiv.org/pdf/2505.09140
• Copy Paste: [[2505.09140]] TopoDiT-3D: Topology-Aware Diffusion Transformer with Bottleneck Structure for 3D Point Cloud Generation(https://arxiv.org/abs/2505.09140)
• Keywords: generation
• Abstract: Recent advancements in Diffusion Transformer (DiT) models have significantly improved 3D point cloud generation. However, existing methods primarily focus on local feature extraction while overlooking global topological information, such as voids, which are crucial for maintaining shape consistency and capturing complex geometries. To address this limitation, we propose TopoDiT-3D, a Topology-Aware Diffusion Transformer with a bottleneck structure for 3D point cloud generation. Specifically, we design the bottleneck structure utilizing Perceiver Resampler, which not only offers a mode to integrate topological information extracted through persistent homology into feature learning, but also adaptively filters out redundant local features to improve training efficiency. Experimental results demonstrate that TopoDiT-3D outperforms state-of-the-art models in visual quality, diversity, and training efficiency. Furthermore, TopoDiT-3D demonstrates the importance of rich topological information for 3D point cloud generation and its synergy with conventional local feature learning. Videos and code are available at this https URL.
• 摘要：扩散变压器（DIT）模型的最新进展显着改善了3D点云的产生。但是，现有方法主要集中于局部特征提取，同时忽略了全球拓扑信息，例如空隙，这对于保持形状一致性和捕获复杂的几何形状至关重要。为了解决此限制，我们提出了TopoDit-3D，这是一种具有瓶颈结构的拓扑感知扩散变压器，用于3D点云生成。具体而言，我们设计了使用感知器重采样器的瓶颈结构，该结构不仅提供了一种将通过持续同源性学习提取的拓扑信息集成到功能学习中的模式，而且还可以自适应地过滤冗余的本地功能，以提高训练效率。实验结果表明，TopoDit-3D在视觉质量，多样性和训练效率方面的表现优于最先进的模型。此外，TopoDit-3D展示了丰富的拓扑信息对于3D点云的生成及其与常规局部功能学习的协同作用的重要性。视频和代码可在此HTTPS URL上找到。

Title: Quotient Complex Transformer (QCformer) for Perovskite Data Analysis

• Authors: Xinyu You, Xiang Liu, Chuan-Shen Hu, Kelin Xia, Tze Chien Sum
• Subjects: cs.LG, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2505.09174
• Pdf URL: https://arxiv.org/pdf/2505.09174
• Copy Paste: [[2505.09174]] Quotient Complex Transformer (QCformer) for Perovskite Data Analysis(https://arxiv.org/abs/2505.09174)
• Keywords: generation
• Abstract: The discovery of novel functional materials is crucial in addressing the challenges of sustainable energy generation and climate change. Hybrid organic-inorganic perovskites (HOIPs) have gained attention for their exceptional optoelectronic properties in photovoltaics. Recently, geometric deep learning, particularly graph neural networks (GNNs), has shown strong potential in predicting material properties and guiding material design. However, traditional GNNs often struggle to capture the periodic structures and higher-order interactions prevalent in such systems. To address these limitations, we propose a novel representation based on quotient complexes (QCs) and introduce the Quotient Complex Transformer (QCformer) for material property prediction. A material structure is modeled as a quotient complex, which encodes both pairwise and many-body interactions via simplices of varying dimensions and captures material periodicity through a quotient operation. Our model leverages higher-order features defined on simplices and processes them using a simplex-based Transformer module. We pretrain QCformer on benchmark datasets such as the Materials Project and JARVIS, and fine-tune it on HOIP datasets. The results show that QCformer outperforms state-of-the-art models in HOIP property prediction, demonstrating its effectiveness. The quotient complex representation and QCformer model together contribute a powerful new tool for predictive modeling of perovskite materials.
• 摘要：新型功能材料的发现对于应对可持续能源产生和气候变化的挑战至关重要。杂交有机无机钙钛矿（HOIP）因其在光伏中的异常光电特性而引起了人们的关注。最近，几何深度学习，尤其是图形神经网络（GNN），在预测材料特性和指导材料设计方面表现出了强大的潜力。但是，传统的GNN通常很难捕获这种系统中普遍存在的周期性结构和高阶相互作用。为了解决这些局限性，我们提出了一种基于商复合物（QC）的新颖表示，并介绍了材料属性预测的商复合变压器（QCFormer）。材料结构被建模为商配合物，该复合物可以通过变化尺寸的简单来编码成对和多体相互作用，并通过商操作捕获物质周期性。我们的模型利用在简单上定义的高阶功能，并使用基于单纯形的变压器模块对其进行处理。我们在基准数据集（例如材料项目和jarvis）上预处理QCFormer，然后在HOIP数据集上对其进行微调。结果表明，QCFormer在HOIP属性预测中的表现优于最先进的模型，证明了其有效性。商的复杂表示和QCFormer模型共同为钙钛矿材料的预测建模提供了强大的新工具。

Title: Zero-shot Quantization: A Comprehensive Survey

• Authors: Minjun Kim, Jaehyeon Choi, Jongkeun Lee, Wonjin Cho, U Kang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.09188
• Pdf URL: https://arxiv.org/pdf/2505.09188
• Copy Paste: [[2505.09188]] Zero-shot Quantization: A Comprehensive Survey(https://arxiv.org/abs/2505.09188)
• Keywords: generation
• Abstract: Network quantization has proven to be a powerful approach to reduce the memory and computational demands of deep learning models for deployment on resource-constrained devices. However, traditional quantization methods often rely on access to training data, which is impractical in many real-world scenarios due to privacy, security, or regulatory constraints. Zero-shot Quantization (ZSQ) emerges as a promising solution, achieving quantization without requiring any real data. In this paper, we provide a comprehensive overview of ZSQ methods and their recent advancements. First, we provide a formal definition of the ZSQ problem and highlight the key challenges. Then, we categorize the existing ZSQ methods into classes based on data generation strategies, and analyze their motivations, core ideas, and key takeaways. Lastly, we suggest future research directions to address the remaining limitations and advance the field of ZSQ. To the best of our knowledge, this paper is the first in-depth survey on ZSQ.
• 摘要：事实证明，网络量化是一种有力的方法，可以减少深度学习模型的记忆和计算需求，以在资源受限的设备上部署。但是，传统的量化方法通常依赖于对培训数据的访问，这在许多实际情况下是由于隐私，安全性或法规限制而不切实际的。零射击量化（ZSQ）是一种有希望的解决方案，可以实现量化，而无需任何实际数据。在本文中，我们提供了ZSQ方法及其最近进步的全面概述。首先，我们提供了ZSQ问题的正式定义，并突出了关键挑战。然后，我们将现有的ZSQ方法分为基于数据生成策略的类，并分析其动机，核心思想和关键要点。最后，我们建议未来的研究方向解决剩余局限性并推进ZSQ领域。据我们所知，本文是对ZSQ的首次深入调查。

Title: PDE: Gene Effect Inspired Parameter Dynamic Evolution for Low-light Image Enhancement

• Authors: Tong Li, Lizhi Wang, Hansen Feng, Lin Zhu, Hua Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.09196
• Pdf URL: https://arxiv.org/pdf/2505.09196
• Copy Paste: [[2505.09196]] PDE: Gene Effect Inspired Parameter Dynamic Evolution for Low-light Image Enhancement(https://arxiv.org/abs/2505.09196)
• Keywords: generation
• Abstract: Low-light image enhancement (LLIE) is a fundamental task in computational photography, aiming to improve illumination, reduce noise, and enhance image quality. While recent advancements focus on designing increasingly complex neural network models, we observe a peculiar phenomenon: resetting certain parameters to random values unexpectedly improves enhancement performance for some images. Drawing inspiration from biological genes, we term this phenomenon the gene effect. The gene effect limits enhancement performance, as even random parameters can sometimes outperform learned ones, preventing models from fully utilizing their capacity. In this paper, we investigate the reason and propose a solution. Based on our observations, we attribute the gene effect to static parameters, analogous to how fixed genetic configurations become maladaptive when environments change. Inspired by biological evolution, where adaptation to new environments relies on gene mutation and recombination, we propose parameter dynamic evolution (PDE) to adapt to different images and mitigate the gene effect. PDE employs a parameter orthogonal generation technique and the corresponding generated parameters to simulate gene recombination and gene mutation, separately. Experiments validate the effectiveness of our techniques. The code will be released to the public.
• 摘要：低光图像增强（LLIE）是计算摄影的基本任务，旨在改善照明，降低噪声并增强图像质量。尽管最近的进步着重于设计日益复杂的神经网络模型，但我们观察到一种特殊的现象：将某些参数重置为随机值，会意外地改善某些图像的增强性能。从生物学基因中汲取灵感，我们将这种现象称为基因效应。基因效应限制了增强性能，因为即使随机参数有时也可以胜过学习的参数，从而阻止模型充分利用其容量。在本文中，我们研究了原因并提出了解决方案。根据我们的观察结果，我们将基因效应归因于静态参数，类似于固定遗传构型在环境发生变化时如何使适应不良。受生物进化的启发，在新环境中适应基因突变和重组，我们建议参数动态进化（PDE）适应不同的图像并减轻基因效应。 PDE采用参数正交生成技术和相应的生成参数来分别模拟基因重组和基因突变。实验验证了我们技术的有效性。该代码将发布给公众。

Title: Few-Shot Anomaly-Driven Generation for Anomaly Classification and Segmentation

• Authors: Guan Gui, Bin-Bin Gao, Jun Liu, Chengjie Wang, Yunsheng Wu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.09263
• Pdf URL: https://arxiv.org/pdf/2505.09263
• Copy Paste: [[2505.09263]] Few-Shot Anomaly-Driven Generation for Anomaly Classification and Segmentation(https://arxiv.org/abs/2505.09263)
• Keywords: generation
• Abstract: Anomaly detection is a practical and challenging task due to the scarcity of anomaly samples in industrial inspection. Some existing anomaly detection methods address this issue by synthesizing anomalies with noise or external data. However, there is always a large semantic gap between synthetic and real-world anomalies, resulting in weak performance in anomaly detection. To solve the problem, we propose a few-shot Anomaly-driven Generation (AnoGen) method, which guides the diffusion model to generate realistic and diverse anomalies with only a few real anomalies, thereby benefiting training anomaly detection models. Specifically, our work is divided into three stages. In the first stage, we learn the anomaly distribution based on a few given real anomalies and inject the learned knowledge into an embedding. In the second stage, we use the embedding and given bounding boxes to guide the diffusion model to generate realistic and diverse anomalies on specific objects (or textures). In the final stage, we propose a weakly-supervised anomaly detection method to train a more powerful model with generated anomalies. Our method builds upon DRAEM and DesTSeg as the foundation model and conducts experiments on the commonly used industrial anomaly detection dataset, MVTec. The experiments demonstrate that our generated anomalies effectively improve the model performance of both anomaly classification and segmentation tasks simultaneously, \eg, DRAEM and DseTSeg achieved a 5.8\% and 1.5\% improvement in AU-PR metric on segmentation task, respectively. The code and generated anomalous data are available at this https URL.
• 摘要：由于工业检查中的异常样本缺乏，异常检测是一项实用且具有挑战性的任务。一些现有的异常检测方法通过与噪声或外部数据合成异常来解决此问题。但是，合成和现实世界异常之间总是存在较大的语义差距，从而导致异常检测性能较弱。为了解决该问题，我们提出了一些以异常驱动的生成（Anogen）方法，该方法指导扩散模型以仅使用少数实际异常产生现实和多样的异常，从而使训练异常检测模型受益。具体来说，我们的工作分为三个阶段。在第一阶段，我们根据一些给定的实际异常学习了异常分布，并将学习知识注入嵌入。在第二阶段，我们使用嵌入式和给定的边界框来指导扩散模型，以在特定对象（或纹理）上生成现实和多样的异常。在最后阶段，我们提出了一种弱监督的异常检测方法，以训练具有生成异常的更强大模型。我们的方法基于Draem和Destseg作为基础模型，并在常用的工业异常检测数据集MVTEC上进行实验。实验表明，我们生成的异常有效地提高了异常分类和分割任务的模型性能，分别在分段任务上，AU-pr量表的提高了5.8 \％\％和1.5 \％。该HTTPS URL可用代码和生成的异常数据。

Title: Recent Advances in Medical Imaging Segmentation: A Survey

• Authors: Fares Bougourzi, Abdenour Hadid
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.09274
• Pdf URL: https://arxiv.org/pdf/2505.09274
• Copy Paste: [[2505.09274]] Recent Advances in Medical Imaging Segmentation: A Survey(https://arxiv.org/abs/2505.09274)
• Keywords: generative
• Abstract: Medical imaging is a cornerstone of modern healthcare, driving advancements in diagnosis, treatment planning, and patient care. Among its various tasks, segmentation remains one of the most challenging problem due to factors such as data accessibility, annotation complexity, structural variability, variation in medical imaging modalities, and privacy constraints. Despite recent progress, achieving robust generalization and domain adaptation remains a significant hurdle, particularly given the resource-intensive nature of some proposed models and their reliance on domain expertise. This survey explores cutting-edge advancements in medical image segmentation, focusing on methodologies such as Generative AI, Few-Shot Learning, Foundation Models, and Universal Models. These approaches offer promising solutions to longstanding challenges. We provide a comprehensive overview of the theoretical foundations, state-of-the-art techniques, and recent applications of these methods. Finally, we discuss inherent limitations, unresolved issues, and future research directions aimed at enhancing the practicality and accessibility of segmentation models in medical imaging. We are maintaining a \href{this https URL}{GitHub Repository} to continue tracking and updating innovations in this field.
• 摘要：医学成像是现代医疗保健，诊断，治疗计划和患者护理方面的进步的基石。在其各种任务中，由于数据可访问性，注释复杂性，结构可变性，医学成像方式的变化和隐私限制等因素，分割仍然是最具挑战性的问题之一。尽管最近取得了进展，但实现强大的概括和域的适应性仍然是一个重大障碍，尤其是考虑到某些拟议模型的资源密集性及其对领域专业知识的依赖。这项调查探讨了医学图像细分的最先进进步，重点是生成AI，少数学习，基础模型和通用模型等方法。这些方法为长期挑战提供了有希望的解决方案。我们提供了有关这些方法的理论基础，最新技术以及最新应用的全面概述。最后，我们讨论旨在增强医学成像中细分模型的实用性和可访问性的固有局限性，未解决的问题以及未来的研究方向。我们正在维护\ href {此https url} {github存储库}，以继续跟踪和更新该领域的创新。

Title: Generating Full-field Evolution of Physical Dynamics from Irregular Sparse Observations

• Authors: Panqi Chen, Yifan Sun, Lei Cheng, Yang Yang, Weichang Li, Yang Liu, Weiqing Liu, Jiang Bian, Shikai Fang
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2505.09284
• Pdf URL: https://arxiv.org/pdf/2505.09284
• Copy Paste: [[2505.09284]] Generating Full-field Evolution of Physical Dynamics from Irregular Sparse Observations(https://arxiv.org/abs/2505.09284)
• Keywords: generation, generative
• Abstract: Modeling and reconstructing multidimensional physical dynamics from sparse and off-grid observations presents a fundamental challenge in scientific research. Recently, diffusion-based generative modeling shows promising potential for physical simulation. However, current approaches typically operate on on-grid data with preset spatiotemporal resolution, but struggle with the sparsely observed and continuous nature of real-world physical dynamics. To fill the gaps, we present SDIFT, Sequential DIffusion in Functional Tucker space, a novel framework that generates full-field evolution of physical dynamics from irregular sparse observations. SDIFT leverages the functional Tucker model as the latent space representer with proven universal approximation property, and represents observations as latent functions and Tucker core sequences. We then construct a sequential diffusion model with temporally augmented UNet in the functional Tucker space, denoising noise drawn from a Gaussian process to generate the sequence of core tensors. At the posterior sampling stage, we propose a Message-Passing Posterior Sampling mechanism, enabling conditional generation of the entire sequence guided by observations at limited time steps. We validate SDIFT on three physical systems spanning astronomical (supernova explosions, light-year scale), environmental (ocean sound speed fields, kilometer scale), and molecular (organic liquid, millimeter scale) domains, demonstrating significant improvements in both reconstruction accuracy and computational efficiency compared to state-of-the-art approaches.
• 摘要：从稀疏和离网观测中对多维物理动态进行建模和重建，这在科学研究中提出了基本挑战。最近，基于扩散的生成建模显示了物理模拟的有希望的潜力。但是，当前的方法通常在具有预设时空分辨率的网格数据上运行，但在现实世界中物理动力学的稀疏和连续性上挣扎。为了填补空白，我们提出了SDIFT，在功能性Tucker空间中的顺序扩散，这是一个新型框架，从不规则的稀疏观测中产生了物理动力学的全场演变。 SDIFT利用功能性Tucker模型为具有证明的通用近似属性的潜在空间代表，并将观察值表示为潜在函数和Tucker Core序列。然后，我们在功能性塔克空间中使用暂时增强的UNET构建了一个顺序扩散模型，从高斯过程中得出噪声，以生成核心张量的序列。在后采样阶段，我们提出了一个消息后的后取样机制，从而使能够在有限时间步长下观察到的有条件生成整个序列。我们验证了跨越天文学（超新星爆炸，光年量表），环境（海洋声速，公里尺度）和分子（有机液体，毫米级）域的三个物理系统的SDIFT，这表明了与正常方法相比，重建精度和计算效率的显着提高。

Title: Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysis

• Authors: Bingxin Ke, Kevin Qu, Tianfu Wang, Nando Metzger, Shengyu Huang, Bo Li, Anton Obukhov, Konrad Schindler
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2505.09358
• Pdf URL: https://arxiv.org/pdf/2505.09358
• Copy Paste: [[2505.09358]] Marigold: Affordable Adaptation of Diffusion-Based Image Generators for Image Analysis(https://arxiv.org/abs/2505.09358)
• Keywords: generative
• Abstract: The success of deep learning in computer vision over the past decade has hinged on large labeled datasets and strong pretrained models. In data-scarce settings, the quality of these pretrained models becomes crucial for effective transfer learning. Image classification and self-supervised learning have traditionally been the primary methods for pretraining CNNs and transformer-based architectures. Recently, the rise of text-to-image generative models, particularly those using denoising diffusion in a latent space, has introduced a new class of foundational models trained on massive, captioned image datasets. These models' ability to generate realistic images of unseen content suggests they possess a deep understanding of the visual world. In this work, we present Marigold, a family of conditional generative models and a fine-tuning protocol that extracts the knowledge from pretrained latent diffusion models like Stable Diffusion and adapts them for dense image analysis tasks, including monocular depth estimation, surface normals prediction, and intrinsic decomposition. Marigold requires minimal modification of the pre-trained latent diffusion model's architecture, trains with small synthetic datasets on a single GPU over a few days, and demonstrates state-of-the-art zero-shot generalization. Project page: this https URL
• 摘要：在过去十年中，深度学习在计算机视觉中的成功取决于大型标记的数据集和强大的验证模型。在数据筛分设置中，这些预审预告片的模型的质量对于有效的转移学习至关重要。传统上，图像分类和自我监督的学习是预处理CNN和基于变压器架构的主要方法。最近，文本到图像生成模型的兴起，尤其是那些在潜在空间中使用DeNoising扩散的模型，它引入了一类新的基础模型，该模型在大规模的字幕图像数据集中训练。这些模型能够产生看不见的内容的现实图像的能力表明它们对视觉世界有深刻的了解。在这项工作中，我们提出了Marigold，这是一个有条件的生成模型家族和微调协议，该方案从预处理的潜在扩散模型中提取知识，例如稳定的扩散，并适应了它们的密集图像分析任务，包括单眼深度估计，表面正态的预测和内在的分解。万寿菊要求在几天内在单个GPU上使用小型合成数据集的训练预训练的潜扩散模型的架构最少修改，并展示了最先进的零拍概括。项目页面：此HTTPS URL

Title: Text-driven Motion Generation: Overview, Challenges and Directions

• Authors: Ali Rida Sahili, Najett Neji, Hedi Tabia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.09379
• Pdf URL: https://arxiv.org/pdf/2505.09379
• Copy Paste: [[2505.09379]] Text-driven Motion Generation: Overview, Challenges and Directions(https://arxiv.org/abs/2505.09379)
• Keywords: generation
• Abstract: Text-driven motion generation offers a powerful and intuitive way to create human movements directly from natural language. By removing the need for predefined motion inputs, it provides a flexible and accessible approach to controlling animated characters. This makes it especially useful in areas like virtual reality, gaming, human-computer interaction, and robotics. In this review, we first revisit the traditional perspective on motion synthesis, where models focused on predicting future poses from observed initial sequences, often conditioned on action labels. We then provide a comprehensive and structured survey of modern text-to-motion generation approaches, categorizing them from two complementary perspectives: (i) architectural, dividing methods into VAE-based, diffusion-based, and hybrid models; and (ii) motion representation, distinguishing between discrete and continuous motion generation strategies. In addition, we explore the most widely used datasets, evaluation methods, and recent benchmarks that have shaped progress in this area. With this survey, we aim to capture where the field currently stands, bring attention to its key challenges and limitations, and highlight promising directions for future exploration. We hope this work offers a valuable starting point for researchers and practitioners working to push the boundaries of language-driven human motion synthesis.
• 摘要：文本驱动的运动生成提供了一种有力而直观的方式，可以直接从自然语言中创造人类运动。通过消除对预定义运动输入的需求，它为控制动画字符提供了一种灵活且易于访问的方法。这使其在虚拟现实，游戏，人类计算机互动和机器人技术等领域中特别有用。在这篇综述中，我们首先重新审视运动合成的传统观点，其中模型着重于预测未来的原始序列，通常以动作标签为条件。然后，我们对现代文本到动作生成方法进行了全面的结构化调查，从两个互补的角度将其分类：（i）建筑，分隔为基于VAE的基于VAE，基于扩散和混合模型的方法； （ii）运动表示，区分离散和连续运动策略。此外，我们探索了最广泛使用的数据集，评估方法和最近在该领域取得进展的基准。通过这项调查，我们旨在捕捉该领域目前所在的位置，引起人们对其主要挑战和局限性的关注，并突出显示未来探索的有希望的方向。我们希望这项工作为研究人员和从业者提供了一个宝贵的起点，以突破语言驱动的人类运动综合的界限。

Title: CXMArena: Unified Dataset to benchmark performance in realistic CXM Scenarios

• Authors: Raghav Garg, Kapil Sharma, Karan Gupta
• Subjects: cs.LG, cs.AI, cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2505.09436
• Pdf URL: https://arxiv.org/pdf/2505.09436
• Copy Paste: [[2505.09436]] CXMArena: Unified Dataset to benchmark performance in realistic CXM Scenarios(https://arxiv.org/abs/2505.09436)
• Keywords: generation
• Abstract: Large Language Models (LLMs) hold immense potential for revolutionizing Customer Experience Management (CXM), particularly in contact center operations. However, evaluating their practical utility in complex operational environments is hindered by data scarcity (due to privacy concerns) and the limitations of current benchmarks. Existing benchmarks often lack realism, failing to incorporate deep knowledge base (KB) integration, real-world noise, or critical operational tasks beyond conversational fluency. To bridge this gap, we introduce CXMArena, a novel, large-scale synthetic benchmark dataset specifically designed for evaluating AI in operational CXM contexts. Given the diversity in possible contact center features, we have developed a scalable LLM-powered pipeline that simulates the brand's CXM entities that form the foundation of our datasets-such as knowledge articles including product specifications, issue taxonomies, and contact center conversations. The entities closely represent real-world distribution because of controlled noise injection (informed by domain experts) and rigorous automated validation. Building on this, we release CXMArena, which provides dedicated benchmarks targeting five important operational tasks: Knowledge Base Refinement, Intent Prediction, Agent Quality Adherence, Article Search, and Multi-turn RAG with Integrated Tools. Our baseline experiments underscore the benchmark's difficulty: even state of the art embedding and generation models achieve only 68% accuracy on article search, while standard embedding methods yield a low F1 score of 0.3 for knowledge base refinement, highlighting significant challenges for current models necessitating complex pipelines and solutions over conventional techniques.
• 摘要：大型语言模型（LLMS）具有彻底改变客户体验管理（CXM）的巨大潜力，尤其是在联络中心运营中。但是，数据稀缺性（由于隐私问题）和当前基准的局限性，评估其在复杂的操作环境中的实际实用性受到了阻碍。现有的基准通常缺乏现实主义，无法纳入深层知识库（KB）集成，现实世界噪声或超出对话流利的关键操作任务。为了弥合这一差距，我们介绍了CXMARENA，这是一种新型的大规模合成基准数据集，专门设计用于在操作CXM环境中评估AI。鉴于可能的接触中心功能的多样性，我们开发了可扩展的LLM驱动管道，该管道模拟了品牌的CXM实体，这些实体构成了我们数据集的基础，例如知识文章，包括产品规格，问题分类法和联系中心对话。由于受控的噪声注入（由域专家告知）和严格的自动化验证，该实体非常代表现实世界的分布。在此基础上，我们发布了CXMARENA，该CXMARENA提供了针对五个重要操作任务的专用基准：知识基础改进，意图预测，代理质量依从性，文章搜索和使用集成工具的多转移。我们的基线实验强调了基准的难度：即使是最新的嵌入和生成模型在文章搜索方面仅达到68％的精度，而标准嵌入方法的较低的F1得分为0.3，用于知识库的细化，强调了当前模型的重大挑战，即在传统技术上不需要复杂的管道和解决方案。

Title: Variational Rank Reduction Autoencoder

• Authors: Jad Mounayer, Alicia Tierz, Jerome Tomezyk, Chady Ghnatios, Francisco Chinesta
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2505.09458
• Pdf URL: https://arxiv.org/pdf/2505.09458
• Copy Paste: [[2505.09458]] Variational Rank Reduction Autoencoder(https://arxiv.org/abs/2505.09458)
• Keywords: generation, generative
• Abstract: Deterministic Rank Reduction Autoencoders (RRAEs) enforce by construction a regularization on the latent space by applying a truncated SVD. While this regularization makes Autoencoders more powerful, using them for generative purposes is counter-intuitive due to their deterministic nature. On the other hand, Variational Autoencoders (VAEs) are well known for their generative abilities by learning a probabilistic latent space. In this paper, we present Variational Rank Reduction Autoencoders (VRRAEs), a model that leverages the advantages of both RRAEs and VAEs. Our claims and results show that when carefully sampling the latent space of RRAEs and further regularizing with the Kullback-Leibler (KL) divergence (similarly to VAEs), VRRAEs outperform RRAEs and VAEs. Additionally, we show that the regularization induced by the SVD not only makes VRRAEs better generators than VAEs, but also reduces the possibility of posterior collapse. Our results include a synthetic dataset of a small size that showcases the robustness of VRRAEs against collapse, and three real-world datasets; the MNIST, CelebA, and CIFAR-10, over which VRRAEs are shown to outperform both VAEs and RRAEs on many random generation and interpolation tasks based on the FID score.
• 摘要：确定性等级降低自动编码器（RRAES）通过施加截短的SVD来实施潜在空间上的正规化。尽管这种正则化使自动编码器更强大，但由于其确定性性质，将其用于生成目的是违反直觉的。另一方面，各种自动编码器（VAE）通过学习概率潜在空间而以其生成能力而闻名。在本文中，我们介绍了各种排名降低自动编码器（VRRAES），该模型利用了RRAES和VAE的优势。我们的主张和结果表明，当仔细采样RRAE的潜在空间并与Kullback-Leibler（kl）差异（类似于VAE）进一步正规化时，VRRAES的表现要优于RRAES和VAE。此外，我们表明，由SVD引起的正则化不仅使VRRAES比VAE更好，而且还减少了后塌陷的可能性。我们的结果包括一个小尺寸的合成数据集，该数据集展示了VRRAES防止崩溃的鲁棒性和三个现实世界数据集； MNIST，CELEBA和CIFAR-10，根据FID得分，VRRAES在许多随机生成和插值任务上的表现都超过了VAE和RRAES。

Title: BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset

• Authors: Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, Le Xue, Caiming Xiong, Ran Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2505.09568
• Pdf URL: https://arxiv.org/pdf/2505.09568
• Copy Paste: [[2505.09568]] BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset(https://arxiv.org/abs/2505.09568)
• Keywords: generation, generative
• Abstract: Unifying image understanding and generation has gained growing attention in recent research on multimodal models. Although design choices for image understanding have been extensively studied, the optimal model architecture and training recipe for a unified framework with image generation remain underexplored. Motivated by the strong potential of autoregressive and diffusion models for high-quality generation and scalability, we conduct a comprehensive study of their use in unified multimodal settings, with emphasis on image representations, modeling objectives, and training strategies. Grounded in these investigations, we introduce a novel approach that employs a diffusion transformer to generate semantically rich CLIP image features, in contrast to conventional VAE-based representations. This design yields both higher training efficiency and improved generative quality. Furthermore, we demonstrate that a sequential pretraining strategy for unified models-first training on image understanding and subsequently on image generation-offers practical advantages by preserving image understanding capability while developing strong image generation ability. Finally, we carefully curate a high-quality instruction-tuning dataset BLIP3o-60k for image generation by prompting GPT-4o with a diverse set of captions covering various scenes, objects, human gestures, and more. Building on our innovative model design, training recipe, and datasets, we develop BLIP3-o, a suite of state-of-the-art unified multimodal models. BLIP3-o achieves superior performance across most of the popular benchmarks spanning both image understanding and generation tasks. To facilitate future research, we fully open-source our models, including code, model weights, training scripts, and pretraining and instruction tuning datasets.
• 摘要：在最近对多模型模型的研究中，统一的图像理解和产生引起了人们的关注。尽管已经对图像理解的设计选择进行了广泛的研究，但具有图像生成的统一框架的最佳模型架构和培训配方仍未得到充实。由自回旋和扩散模型具有高质量生成和可伸缩性的强大潜力，我们对它们在统一的多模式环境中的使用进行了全面研究，重点是图像表示，建模目标和培训策略。基于这些研究，我们引入了一种新颖的方法，该方法采用扩散变压器来生成语义上富含的剪辑图像特征，与传统的基于VAE的表示相反。该设计既可以提高训练效率，又提高了生成质量。此外，我们证明了统一模型的顺序预处理策略，即对图像理解的第一培训，以及随后在图像生成器上的实用优势，通过在开发强大的图像生成能力的同时保留图像理解能力。最后，我们通过提示带有各种字幕的GPT-4O来介绍各种场景，对象，人类手势等，仔细策划了一个高质量的指令调查数据集Blip3O-60K，以生成图像生成图像。在我们创新的模型设计，培训配方和数据集的基础上，我们开发了Blip3-O，这是一套最先进的统一多模型。 Blip3-O在大多数流行的基准测试中都达到了跨越图像理解和发电任务的卓越性能。为了促进未来的研究，我们将模型完全开放，包括代码，模型权重，培训脚本以及训练和指导调谐数据集。

Title: Don't Forget your Inverse DDIM for Image Editing

• Authors: Guillermo Gomez-Trenado, Pablo Mesejo, Oscar Cordón, Stéphane Lathuilière
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2505.09571
• Pdf URL: https://arxiv.org/pdf/2505.09571
• Copy Paste: [[2505.09571]] Don't Forget your Inverse DDIM for Image Editing(https://arxiv.org/abs/2505.09571)
• Keywords: generation
• Abstract: The field of text-to-image generation has undergone significant advancements with the introduction of diffusion models. Nevertheless, the challenge of editing real images persists, as most methods are either computationally intensive or produce poor reconstructions. This paper introduces SAGE (Self-Attention Guidance for image Editing) - a novel technique leveraging pre-trained diffusion models for image editing. SAGE builds upon the DDIM algorithm and incorporates a novel guidance mechanism utilizing the self-attention layers of the diffusion U-Net. This mechanism computes a reconstruction objective based on attention maps generated during the inverse DDIM process, enabling efficient reconstruction of unedited regions without the need to precisely reconstruct the entire input image. Thus, SAGE directly addresses the key challenges in image editing. The superiority of SAGE over other methods is demonstrated through quantitative and qualitative evaluations and confirmed by a statistically validated comprehensive user study, in which all 47 surveyed users preferred SAGE over competing methods. Additionally, SAGE ranks as the top-performing method in seven out of 10 quantitative analyses and secures second and third places in the remaining three.
• 摘要：随着扩散模型的引入，文本对图像生成的领域已经取得了重大进步。然而，编辑真实图像的挑战仍然存在，因为大多数方法在计算密集程度上要么产生不良的重建。本文介绍了SAGE（图像编辑的自我发挥指导） - 一种利用预训练的扩散模型的新技术。 Sage建立在DDIM算法上，并使用扩散U-NET的自发层进行了新颖的指导机制。该机制基于反向DDIM过程中生成的注意图计算重建目标，从而无需精确重建整个输入图像，从而有效地重建未编辑的区域。因此，Sage直接解决了图像编辑中的关键挑战。通过定量和定性评估证明了SAGE比其他方法的优势，并通过统计验证的综合用户研究证实，在该评估中，所有47位接受调查的用户都偏爱Sage而不是竞争方法。此外，SAGE在10个定量分析中的7个中的7个中排名最佳的方法，并在其余三个中获得第二和第三名。

Title: Adversarial Suffix Filtering: a Defense Pipeline for LLMs

• Authors: David Khachaturov, Robert Mullins
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2505.09602
• Pdf URL: https://arxiv.org/pdf/2505.09602
• Copy Paste: [[2505.09602]] Adversarial Suffix Filtering: a Defense Pipeline for LLMs(https://arxiv.org/abs/2505.09602)
• Keywords: generation
• Abstract: Large Language Models (LLMs) are increasingly embedded in autonomous systems and public-facing environments, yet they remain susceptible to jailbreak vulnerabilities that may undermine their security and trustworthiness. Adversarial suffixes are considered to be the current state-of-the-art jailbreak, consistently outperforming simpler methods and frequently succeeding even in black-box settings. Existing defenses rely on access to the internal architecture of models limiting diverse deployment, increase memory and computation footprints dramatically, or can be bypassed with simple prompt engineering methods. We introduce $\textbf{Adversarial Suffix Filtering}$ (ASF), a lightweight novel model-agnostic defensive pipeline designed to protect LLMs against adversarial suffix attacks. ASF functions as an input preprocessor and sanitizer that detects and filters adversarially crafted suffixes in prompts, effectively neutralizing malicious injections. We demonstrate that ASF provides comprehensive defense capabilities across both black-box and white-box attack settings, reducing the attack efficacy of state-of-the-art adversarial suffix generation methods to below 4%, while only minimally affecting the target model's capabilities in non-adversarial scenarios.
• 摘要：大型语言模型（LLM）越来越多地嵌入自主系统和面向公共的环境中，但它们仍然容易受到可能破坏其安全性和可信赖性的越狱漏洞的影响。对抗后缀被认为是当前的最新越狱，始终超过更简单的方法，即使在黑盒设置中也经常成功。现有的防御能力依赖于对模型内部体系结构的访问，从而限制了各种部署，增加内存和计算足迹，或者可以使用简单的及时工程方法绕过。我们介绍了$ \ textbf {对抗后缀过滤} $（ASF），这是一种轻巧的新型模型 - 敏捷的防御管道，旨在保护LLMS免受对抗后缀攻击。 ASF充当输入预处理和消毒剂，可在提示中检测和过滤对抗的后缀，从而有效地中和恶意注射。我们证明，ASF在Black-Box和White-Box攻击设置中均提供全面的防御能力，从而降低了最先进的对抗性后缀生成方法的攻击功效至4％以下，而仅对非对抗性场景中目标模型的能力影响最小。