2025-07-11

Title: Image Can Bring Your Memory Back: A Novel Multi-Modal Guided Attack against Image Generation Model Unlearning

• Authors: Renyang Liu, Guanlin Li, Tianwei Zhang, See-Kiong Ng
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2507.07139
• Pdf URL: https://arxiv.org/pdf/2507.07139
• Copy Paste: [[2507.07139]] Image Can Bring Your Memory Back: A Novel Multi-Modal Guided Attack against Image Generation Model Unlearning(https://arxiv.org/abs/2507.07139)
• Keywords: generation, generative
• Abstract: Recent advances in image generation models (IGMs), particularly diffusion-based architectures such as Stable Diffusion (SD), have markedly enhanced the quality and diversity of AI-generated visual content. However, their generative capability has also raised significant ethical, legal, and societal concerns, including the potential to produce harmful, misleading, or copyright-infringing content. To mitigate these concerns, machine unlearning (MU) emerges as a promising solution by selectively removing undesirable concepts from pretrained models. Nevertheless, the robustness and effectiveness of existing unlearning techniques remain largely unexplored, particularly in the presence of multi-modal adversarial inputs. To bridge this gap, we propose Recall, a novel adversarial framework explicitly designed to compromise the robustness of unlearned IGMs. Unlike existing approaches that predominantly rely on adversarial text prompts, Recall exploits the intrinsic multi-modal conditioning capabilities of diffusion models by efficiently optimizing adversarial image prompts with guidance from a single semantically relevant reference image. Extensive experiments across ten state-of-the-art unlearning methods and diverse tasks show that Recall consistently outperforms existing baselines in terms of adversarial effectiveness, computational efficiency, and semantic fidelity with the original textual prompt. These findings reveal critical vulnerabilities in current unlearning mechanisms and underscore the need for more robust solutions to ensure the safety and reliability of generative models. Code and data are publicly available at \textcolor{blue}{this https URL}.
• 摘要：图像生成模型（IGM）的最新进展，尤其是基于扩散的架构，例如稳定扩散（SD），已显着提高了AI生成的视觉内容的质量和多样性。但是，它们的生成能力也引起了重大的道德，法律和社会问题，包括产生有害，误导或版权侵入内容的​​潜力。为了减轻这些关注，通过选择性地删除预验证模型中的不良概念，将机器学习（MU）作为一种有希望的解决方案出现。然而，现有的未学习技术的鲁棒性和有效性在很大程度上尚未探索，尤其是在存在多模式的对手输入的情况下。为了弥合这一差距，我们提出了回忆，这是一个新颖的对抗框架，该框架明确设计，旨在损害未经学习的IGM的稳健性。与现有的主要依赖对抗文本提示的方法不同，召回利用了扩散模型的内在多模式调节能力，通过有效地优化对抗性图像提示，并在单个具有语义相关的参考图像的指导下进行指导。在十种最先进的学习方法和各种任务上进行的广泛实验表明，回忆在对抗性有效性，计算效率和语义忠诚方面始终优于现有基准，并具有原始文本提示。这些发现揭示了当前未学习机制中的关键脆弱性，并强调了需要更强大的解决方案以确保生成模型的安全性和可靠性。代码和数据可在\ textColor {blue} {this HTTPS url}上公开获得。

Title: Interpretable EEG-to-Image Generation with Semantic Prompts

• Authors: Arshak Rezvani, Ali Akbari, Kosar Sanjar Arani, Maryam Mirian, Emad Arasteh, Martin J. McKeown
• Subjects: cs.CV, cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2507.07157
• Pdf URL: https://arxiv.org/pdf/2507.07157
• Copy Paste: [[2507.07157]] Interpretable EEG-to-Image Generation with Semantic Prompts(https://arxiv.org/abs/2507.07157)
• Keywords: generation
• Abstract: Decoding visual experience from brain signals offers exciting possibilities for neuroscience and interpretable AI. While EEG is accessible and temporally precise, its limitations in spatial detail hinder image reconstruction. Our model bypasses direct EEG-to-image generation by aligning EEG signals with multilevel semantic captions -- ranging from object-level to abstract themes -- generated by a large language model. A transformer-based EEG encoder maps brain activity to these captions through contrastive learning. During inference, caption embeddings retrieved via projection heads condition a pretrained latent diffusion model for image generation. This text-mediated framework yields state-of-the-art visual decoding on the EEGCVPR dataset, with interpretable alignment to known neurocognitive pathways. Dominant EEG-caption associations reflected the importance of different semantic levels extracted from perceived images. Saliency maps and t-SNE projections reveal semantic topography across the scalp. Our model demonstrates how structured semantic mediation enables cognitively aligned visual decoding from EEG.
• 摘要：从大脑信号中解码视觉体验为神经科学和可解释的AI提供了令人兴奋的可能性。虽然脑电图可访问且在时间上精确，但其空间细节的局限性阻碍了图像重建。我们的模型通过与大型语言模型生成的多级语义字幕（从对象级到抽象主题）来对齐脑电图信号（从对象级到抽象主题）绕开直接的脑电图生成。基于变压器的EEG编码器通过对比度学习将大脑活动映射到这些字幕上。在推断期间，通过投影头部检索的字幕嵌入条件是图像生成的预验证的潜扩散模型。该文本介导的框架在EEGCVPR数据集上产生了最新的视觉解码，并与已知的神经认知途径有可解释的一致性。主要的EEG抗管相关反映了从感知图像中提取的不同语义水平的重要性。显着图和T-SNE投影揭示了整个头皮上的语义地形。我们的模型展示了结构化语义介导如何使脑电图的认知对齐视觉解码。

Title: Bridging the Last Mile of Prediction: Enhancing Time Series Forecasting with Conditional Guided Flow Matching

• Authors: Huibo Xu, Runlong Yu, Likang Wu, Xianquan Wang, Qi Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.07192
• Pdf URL: https://arxiv.org/pdf/2507.07192
• Copy Paste: [[2507.07192]] Bridging the Last Mile of Prediction: Enhancing Time Series Forecasting with Conditional Guided Flow Matching(https://arxiv.org/abs/2507.07192)
• Keywords: generation, generative
• Abstract: Diffusion models, a type of generative model, have shown promise in time series forecasting. But they face limitations like rigid source distributions and limited sampling paths, which hinder their performance. Flow matching offers faster generation, higher-quality outputs, and greater flexibility, while also possessing the ability to utilize valuable information from the prediction errors of prior models, which were previously inaccessible yet critically important. To address these challenges and fully unlock the untapped potential of flow matching, we propose Conditional Guided Flow Matching (CGFM). CGFM extends flow matching by incorporating the outputs of an auxiliary model, enabling a previously unattainable capability in the field: learning from the errors of the auxiliary model. For time series forecasting tasks, it integrates historical data as conditions and guidance, constructs two-sided conditional probability paths, and uses a general affine path to expand the space of probability paths, ultimately leading to improved predictions. Extensive experiments show that CGFM consistently enhances and outperforms state-of-the-art models, highlighting its effectiveness in advancing forecasting methods.
• 摘要：扩散模型是一种生成模型，在时间序列预测中显示了有望。但是他们面临诸如刚性源分布和有限的采样路径之类的限制，这阻碍了他们的性能。流匹配提供更快的生成，更高质量的输出和更大的灵活性，同时还具有从先前模型的预测错误中利用有价值的信息的能力，这些模型以前无法访问但至关重要。为了应对这些挑战并完全解锁流量匹配的尚未开发的潜力，我们提出了条件引导流匹配（CGFM）。 CGFM通过合并辅助模型的输出来扩展流量匹配，从而在现场实现了以前无法实现的能力：从辅助模型的误差中学习。对于时间序列预测任务，它将历史数据作为条件和指导集成，构建双面条件概率路径，并使用一般仿射路径来扩展概率路径的空间，最终导致预测的改进。广泛的实验表明，CGFM始终增强和胜过最先进的模型，突出了其在推进预测方法方面的有效性。

Title: A Survey on Long-Video Storytelling Generation: Architectures, Consistency, and Cinematic Quality

• Authors: Mohamed Elmoghany, Ryan Rossi, Seunghyun Yoon, Subhojyoti Mukherjee, Eslam Bakr, Puneet Mathur, Gang Wu, Viet Dac Lai, Nedim Lipka, Ruiyi Zhang, Varun Manjunatha, Chien Nguyen, Daksh Dangi, Abel Salinas, Mohammad Taesiri, Hongjie Chen, Xiaolei Huang, Joe Barrow, Nesreen Ahmed, Hoda Eldardiry, Namyong Park, Yu Wang, Jaemin Cho, Anh Totti Nguyen, Zhengzhong Tu, Thien Nguyen, Dinesh Manocha, Mohamed Elhoseiny, Franck Dernoncourt
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2507.07202
• Pdf URL: https://arxiv.org/pdf/2507.07202
• Copy Paste: [[2507.07202]] A Survey on Long-Video Storytelling Generation: Architectures, Consistency, and Cinematic Quality(https://arxiv.org/abs/2507.07202)
• Keywords: generation, generative
• Abstract: Despite the significant progress that has been made in video generative models, existing state-of-the-art methods can only produce videos lasting 5-16 seconds, often labeled "long-form videos". Furthermore, videos exceeding 16 seconds struggle to maintain consistent character appearances and scene layouts throughout the narrative. In particular, multi-subject long videos still fail to preserve character consistency and motion coherence. While some methods can generate videos up to 150 seconds long, they often suffer from frame redundancy and low temporal diversity. Recent work has attempted to produce long-form videos featuring multiple characters, narrative coherence, and high-fidelity detail. We comprehensively studied 32 papers on video generation to identify key architectural components and training strategies that consistently yield these qualities. We also construct a comprehensive novel taxonomy of existing methods and present comparative tables that categorize papers by their architectural designs and performance characteristics.
• 摘要：尽管视频生成模型取得了重大进展，但现有的最新方法只能制作持续5-16秒的视频，通常标记为“长格式视频”。此外，超过16秒的视频难以在整个叙述中保持一致的角色外观和场景布局。特别是，多主题长的视频仍然无法保持角色的一致性和运动连贯性。尽管某些方法可以生成长达150秒的视频，但它们通常会遭受框架的冗余和低时间多样性的困扰。最近的工作试图制作具有多个字符，叙事连贯性和高保真细节的长期视频。我们全面研究了32篇有关视频生成的论文，以确定始终产生这些品质的关键建筑组件和培训策略。我们还构建了现有方法的全面新颖分类学，并提出了比较表，该表可以根据其建筑设计和性能特征对论文进行分类。

Title: Scale leads to compositional generalization

• Authors: Florian Redhardt, Yassir Akram, Simon Schug
• Subjects: cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2507.07207
• Pdf URL: https://arxiv.org/pdf/2507.07207
• Copy Paste: [[2507.07207]] Scale leads to compositional generalization(https://arxiv.org/abs/2507.07207)
• Keywords: generation
• Abstract: Can neural networks systematically capture discrete, compositional task structure despite their continuous, distributed nature? The impressive capabilities of large-scale neural networks suggest that the answer to this question is yes. However, even for the most capable models, there are still frequent failure cases that raise doubts about their compositionality. Here, we seek to understand what it takes for a standard neural network to generalize over tasks that share compositional structure. We find that simply scaling data and model size leads to compositional generalization. We show that this holds across different task encodings as long as the training distribution sufficiently covers the task space. In line with this finding, we prove that standard multilayer perceptrons can approximate a general class of compositional task families to arbitrary precision using only a linear number of neurons with respect to the number of task modules. Finally, we uncover that if networks successfully compositionally generalize, the constituents of a task can be linearly decoded from their hidden activations. We show that this metric correlates with failures of text-to-image generation models to compose known concepts.
• 摘要：神经网络尽管具有连续的，分布性的性质，是否可以系统地捕获离散的，组成的任务结构？大规模神经网络的令人印象深刻的功能表明，这个问题的答案是肯定的。但是，即使对于最有能力的模型，仍然存在频繁的故障案例，引起了对其组成性的怀疑。在这里，我们试图了解标准神经网络对共享组成结构的任务的概括所需的需要。我们发现，简单地缩放数据和模型大小会导致组成概括。我们表明，只要训练分布足够涵盖任务空间，这将跨不同的任务编码。与这一发现相一致，我们证明标准的多层感知可以将一般的组成任务家族近似于任意精度，仅使用线性数量的神经元就任务模块的数量使用。最后，我们发现，如果网络成功地概括了网络，则可以从其隐藏的激活中线性解码任务的组成部分。我们表明，该度量与文本到图像生成模型的失败相关，以构成已知概念。

Title: Automated Video Segmentation Machine Learning Pipeline

• Authors: Johannes Merz, Lucien Fostier
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2507.07242
• Pdf URL: https://arxiv.org/pdf/2507.07242
• Copy Paste: [[2507.07242]] Automated Video Segmentation Machine Learning Pipeline(https://arxiv.org/abs/2507.07242)
• Keywords: generation
• Abstract: Visual effects (VFX) production often struggles with slow, resource-intensive mask generation. This paper presents an automated video segmentation pipeline that creates temporally consistent instance masks. It employs machine learning for: (1) flexible object detection via text prompts, (2) refined per-frame image segmentation and (3) robust video tracking to ensure temporal stability. Deployed using containerization and leveraging a structured output format, the pipeline was quickly adopted by our artists. It significantly reduces manual effort, speeds up the creation of preliminary composites, and provides comprehensive segmentation data, thereby enhancing overall VFX production efficiency.
• 摘要：视觉效果（VFX）的生产通常会在缓慢，资源密集型面具的产生中挣扎。本文提出了一个自动化的视频分割管道，该管道在时间上创建了时间一致的实例掩码。它使用：（1）通过文本提示，（2）精制图像分割和（3）强大的视频跟踪以确保时间稳定性的灵活对象检测。使用容器化并利用结构化的输出格式部署，该管道很快被我们的艺术家采用。它大大减少了手动努力，加快了初步复合材料的创建，并提供了全面的分割数据，从而提高了整体VFX生产效率。

Title: Frontier LLMs Still Struggle with Simple Reasoning Tasks

• Authors: Alan Malek, Jiawei Ge, Jiawei Ge, Chi Jin, András György, Csaba Szepesvári
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2507.07313
• Pdf URL: https://arxiv.org/pdf/2507.07313
• Copy Paste: [[2507.07313]] Frontier LLMs Still Struggle with Simple Reasoning Tasks(https://arxiv.org/abs/2507.07313)
• Keywords: generation
• Abstract: While state-of-the-art large language models (LLMs) demonstrate advanced reasoning capabilities-achieving remarkable performance on challenging competitive math and coding benchmarks-they also frequently fail on tasks that are easy for humans. This work studies the performance of frontier LLMs on a broad set of such "easy" reasoning problems. By extending previous work in the literature, we create a suite of procedurally generated simple reasoning tasks, including counting, first-order logic, proof trees, and travel planning, with changeable parameters (such as document length. or the number of variables in a math problem) that can arbitrarily increase the amount of computation required to produce the answer while preserving the fundamental difficulty. While previous work showed that traditional, non-thinking models can be made to fail on such problems, we demonstrate that even state-of-the-art thinking models consistently fail on such problems and for similar reasons (e.g. statistical shortcuts, errors in intermediate steps, and difficulties in processing long contexts). To further understand the behavior of the models, we introduce the unpuzzles dataset, a different "easy" benchmark consisting of trivialized versions of well-known math and logic puzzles. Interestingly, while modern LLMs excel at solving the original puzzles, they tend to fail on the trivialized versions, exhibiting several systematic failure patterns related to memorizing the originals. We show that this happens even if the models are otherwise able to solve problems with different descriptions but requiring the same logic. Our results highlight that out-of-distribution generalization is still problematic for frontier language models and the new generation of thinking models, even for simple reasoning tasks, and making tasks easier does not necessarily imply improved performance.
• 摘要：尽管最先进的大语言模型（LLMS）展示了在挑战性竞争性数学和编码基准方面表现出色的能力，但他们经常在对人类容易的任务上失败。这项工作研究了Frontier LLMS在一系列此类“简单”推理问题上的表现。通过在文献中进行以前的大量工作，我们创建了一套程序生成的简单推理任务，包括计数，一阶逻辑，证明树和旅行计划，具有可变的参数（例如文档长度。或数学问题中的变量数量），这些参数可以任意增加在保留基本难度时产生答案所需的计算量。虽然先前的工作表明，传统的，无思想的模型可以在此类问题上失败，但我们证明，即使是最先进的思维模型也始终出于类似的原因（例如统计快捷方式，中间步骤中的错误以及处理长上下文的困难）始终如一地失败。为了进一步了解模型的行为，我们介绍了Untuzzles数据集，该数据集是由著名的数学和逻辑难题的琐碎版本组成的不同的“简单”基准。有趣的是，尽管现代LLM擅长解决原始难题，但它们倾向于在琐碎的版本上失败，表现出与记忆原件有关的几种系统性故障模式。我们表明，即使模型可以通过不同的描述但需要相同的逻辑来解决问题，也会发生这种情况。我们的结果强调，对于边境语言模型和新一代思维模型，即使对于简单的推理任务以及使任务更容易的新一代思维模型仍然存在问题，这仍然是有问题的。

Title: Learning Collective Variables from Time-lagged Generation

• Authors: Seonghyun Park, Kiyoung Seong, Soojung Yang, Rafael Gómez-Bombarelli, Sungsoo Ahn
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2507.07390
• Pdf URL: https://arxiv.org/pdf/2507.07390
• Copy Paste: [[2507.07390]] Learning Collective Variables from Time-lagged Generation(https://arxiv.org/abs/2507.07390)
• Keywords: generation, generative
• Abstract: Rare events such as state transitions are difficult to observe directly with molecular dynamics simulations due to long timescales. Enhanced sampling techniques overcome this by introducing biases along carefully chosen low-dimensional features, known as collective variables (CVs), which capture the slow degrees of freedom. Machine learning approaches (MLCVs) have automated CV discovery, but existing methods typically focus on discriminating meta-stable states without fully encoding the detailed dynamics essential for accurate sampling. We propose TLC, a framework that learns CVs directly from time-lagged conditions of a generative model. Instead of modeling the static Boltzmann distribution, TLC models a time-lagged conditional distribution yielding CVs to capture the slow dynamic behavior. We validate TLC on the Alanine Dipeptide system using two CV-based enhanced sampling tasks: (i) steered molecular dynamics (SMD) and (ii) on-the-fly probability enhanced sampling (OPES), demonstrating equal or superior performance compared to existing MLCV methods in both transition path sampling and state discrimination.
• 摘要：由于长时间尺度引起的分子动力学模拟，很难直接观察到罕见的事件，例如状态转变。增强的采样技术通过沿精心选择的低维特征（称为集体变量（CVS））引入偏见来克服这一问题，该特征捕获了缓慢的自由度。机器学习方法（MLCV）具有自动化的CV发现，但是现有的方法通常着重于区分元稳定状态，而无需完全编码准确采样必不可少的详细动态。我们提出了TLC，该框架直接从生成模型的时间滞后条件中学习CVS。 TLC不是对静态玻尔兹曼分布进行建模，而是建模一个时置的条件分布，从而产生CVS以捕获缓慢的动态行为。我们使用两个基于CV的增强采样任务在丙氨酸二肽系统上验证TLC：（i）转导的分子动力学（SMD）和（ii）与现有的MLCV方法相比，在过渡路径采样和状态歧视中，与现有的MLCV方法相比，概率增强了采样（OPES）。

Title: Behave Your Motion: Habit-preserved Cross-category Animal Motion Transfer

• Authors: Zhimin Zhang, Bi'an Du, Caoyuan Ma, Zheng Wang, Wei Hu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.07394
• Pdf URL: https://arxiv.org/pdf/2507.07394
• Copy Paste: [[2507.07394]] Behave Your Motion: Habit-preserved Cross-category Animal Motion Transfer(https://arxiv.org/abs/2507.07394)
• Keywords: generative
• Abstract: Animal motion embodies species-specific behavioral habits, making the transfer of motion across categories a critical yet complex task for applications in animation and virtual reality. Existing motion transfer methods, primarily focused on human motion, emphasize skeletal alignment (motion retargeting) or stylistic consistency (motion style transfer), often neglecting the preservation of distinct habitual behaviors in animals. To bridge this gap, we propose a novel habit-preserved motion transfer framework for cross-category animal motion. Built upon a generative framework, our model introduces a habit-preservation module with category-specific habit encoder, allowing it to learn motion priors that capture distinctive habitual characteristics. Furthermore, we integrate a large language model (LLM) to facilitate the motion transfer to previously unobserved species. To evaluate the effectiveness of our approach, we introduce the DeformingThings4D-skl dataset, a quadruped dataset with skeletal bindings, and conduct extensive experiments and quantitative analyses, which validate the superiority of our proposed model.
• 摘要：动物运动体现了特定物种的行为习惯，使运动转移成为动画和虚拟现实中应用的关键但复杂的任务。现有的运动转移方法主要集中于人类运动，强调骨骼对齐（运动重新定位）或风格一致性（运动方式转移），通常忽略了动物中明显的习惯行为的保存。为了弥合这一差距，我们提出了一个新型的习惯保存的运动转移框架，用于跨类别动物运动。我们的模型基于生成框架，引入了一个具有特定于类别习惯编码器的习惯保护模块，从而可以学习捕获独特习惯特征的运动先验。此外，我们集成了大型语言模型（LLM），以促进运动转移到以前未观察到的物种。为了评估方法的有效性，我们介绍了变形的4D-SKL数据集，这是具有骨骼结合的四倍数据集，并进行了广泛的实验和定量分析，从而验证了我们所提出的模型的优势。

Title: Neural networks leverage nominally quantum and post-quantum representations

• Authors: Paul M. Riechers, Thomas J. Elliott, Adam S. Shai
• Subjects: cs.LG, quant-ph
• Abstract URL: https://arxiv.org/abs/2507.07432
• Pdf URL: https://arxiv.org/pdf/2507.07432
• Copy Paste: [[2507.07432]] Neural networks leverage nominally quantum and post-quantum representations(https://arxiv.org/abs/2507.07432)
• Keywords: generative
• Abstract: We show that deep neural networks, including transformers and RNNs, pretrained as usual on next-token prediction, intrinsically discover and represent beliefs over 'quantum' and 'post-quantum' low-dimensional generative models of their training data -- as if performing iterative Bayesian updates over the latent state of this world model during inference as they observe more context. Notably, neural nets easily find these representation whereas there is no finite classical circuit that would do the job. The corresponding geometric relationships among neural activations induced by different input sequences are found to be largely independent of neural-network architecture. Each point in this geometry corresponds to a history-induced probability density over all possible futures, and the relative displacement of these points reflects the difference in mechanism and magnitude for how these distinct pasts affect the future.
• 摘要：我们表明，包括变形金刚和RNN在内的深层神经网络在下一步的预测上像往常一样仔细预测，本质上发现并表示对训练数据的“量子”和“ Quantum”的低维生成模型的信念 - 好像在迭代的贝叶斯更新中，在本世界的潜在模型中，他们观察到了更高的上下文，就可以进行迭代的贝叶斯更新。值得注意的是，神经网很容易找到这些表示形式，而没有有限的古典电路可以完成这项工作。发现不同输入序列引起的神经激活之间的相应几何关系在很大程度上与神经网络结构无关。该几何形状中的每个点都对应于所有可能的未来的历史诱导的概率密度，这些点的相对位移反映了这些独特的过去如何影响未来的机制和幅度的差异。

Title: Degradation-Agnostic Statistical Facial Feature Transformation for Blind Face Restoration in Adverse Weather Conditions

• Authors: Chang-Hwan Son
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2507.07464
• Pdf URL: https://arxiv.org/pdf/2507.07464
• Copy Paste: [[2507.07464]] Degradation-Agnostic Statistical Facial Feature Transformation for Blind Face Restoration in Adverse Weather Conditions(https://arxiv.org/abs/2507.07464)
• Keywords: restoration, generative
• Abstract: With the increasing deployment of intelligent CCTV systems in outdoor environments, there is a growing demand for face recognition systems optimized for challenging weather conditions. Adverse weather significantly degrades image quality, which in turn reduces recognition accuracy. Although recent face image restoration (FIR) models based on generative adversarial networks (GANs) and diffusion models have shown progress, their performance remains limited due to the lack of dedicated modules that explicitly address weather-induced degradations. This leads to distorted facial textures and structures. To address these limitations, we propose a novel GAN-based blind FIR framework that integrates two key components: local Statistical Facial Feature Transformation (SFFT) and Degradation-Agnostic Feature Embedding (DAFE). The local SFFT module enhances facial structure and color fidelity by aligning the local statistical distributions of low-quality (LQ) facial regions with those of high-quality (HQ) counterparts. Complementarily, the DAFE module enables robust statistical facial feature extraction under adverse weather conditions by aligning LQ and HQ encoder representations, thereby making the restoration process adaptive to severe weather-induced degradations. Experimental results demonstrate that the proposed degradation-agnostic SFFT model outperforms existing state-of-the-art FIR methods based on GAN and diffusion models, particularly in suppressing texture distortions and accurately reconstructing facial structures. Furthermore, both the SFFT and DAFE modules are empirically validated in enhancing structural fidelity and perceptual quality in face restoration under challenging weather scenarios.
• 摘要：随着智能CCTV系统在室外环境中的越来越多的部署，人们对针对挑战性天气条件进行了优化的面部识别系统的需求不断增长。不利的天气大大降低了图像质量，从而降低了识别精度。尽管最近基于生成对抗网络（GAN）和扩散模型的面部图像恢复（FIR）模型已经显示出进展，但由于缺乏明确解决天气引起的降级的专用模块，它们的性能仍然有限。这导致面部纹理和结构扭曲。为了解决这些局限性，我们提出了一个基于GAN的新型盲型FIR框架，该框架集成了两个关键组成部分：局部统计面部特征转换（SFFT）和降解 - 无链特征嵌入（DAFE）。局部SFFT模块通过将低质量（LQ）面部区域的局部统计分布与高质量（HQ）对应物的局部统计分布对齐来增强面部结构和颜色保真度。补充，DAFE模块可以通过对齐LQ和HQ编码器表示，从而在不利的天气条件下实现了强大的统计面部特征提取，从而使恢复过程适应了严重天气引起的降解。实验结果表明，所提出的降解无形SFFT模型优于基于GAN和扩散模型的现有最新FIR方法，尤其是在抑制纹理扭曲和准确地重建面部结构方面。此外，在挑战性的天气情况下，SFFT和DAFE模块在提高结构保真度和感知质量方面都得到了经验验证。

Title: Temporal Unlearnable Examples: Preventing Personal Video Data from Unauthorized Exploitation by Object Tracking

• Authors: Qiangqiang Wu, Yi Yu, Chenqi Kong, Ziquan Liu, Jia Wan, Haoliang Li, Alex C. Kot, Antoni B. Chan
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2507.07483
• Pdf URL: https://arxiv.org/pdf/2507.07483
• Copy Paste: [[2507.07483]] Temporal Unlearnable Examples: Preventing Personal Video Data from Unauthorized Exploitation by Object Tracking(https://arxiv.org/abs/2507.07483)
• Keywords: generative
• Abstract: With the rise of social media, vast amounts of user-uploaded videos (e.g., YouTube) are utilized as training data for Visual Object Tracking (VOT). However, the VOT community has largely overlooked video data-privacy issues, as many private videos have been collected and used for training commercial models without authorization. To alleviate these issues, this paper presents the first investigation on preventing personal video data from unauthorized exploitation by deep trackers. Existing methods for preventing unauthorized data use primarily focus on image-based tasks (e.g., image classification), directly applying them to videos reveals several limitations, including inefficiency, limited effectiveness, and poor generalizability. To address these issues, we propose a novel generative framework for generating Temporal Unlearnable Examples (TUEs), and whose efficient computation makes it scalable for usage on large-scale video datasets. The trackers trained w/ TUEs heavily rely on unlearnable noises for temporal matching, ignoring the original data structure and thus ensuring training video data-privacy. To enhance the effectiveness of TUEs, we introduce a temporal contrastive loss, which further corrupts the learning of existing trackers when using our TUEs for training. Extensive experiments demonstrate that our approach achieves state-of-the-art performance in video data-privacy protection, with strong transferability across VOT models, datasets, and temporal matching tasks.
• 摘要：随着社交媒体的兴起，将大量的用户更易视频（例如，YouTube）用作视觉对象跟踪（fot）的培训数据。但是，由于许多私人视频已被收集并用于培训商业模型，因此在很大程度上忽略了视频数据私人关系问题。为了减轻这些问题，本文介绍了有关防止深层跟踪器未经授权剥削的个人视频数据的首次调查。预防未经授权数据使用的现有方法主要集中于基于图像的任务（例如，图像分类），将它们直接应用于视频中，揭示了几个局限性，包括效率低下，有限的有效性和较差的普遍性。为了解决这些问题，我们提出了一个新颖的生成框架，用于生成时间无关的示例（TUE），并且其有效的计算使其可在大规模视频数据集中使用。经过训练的带有周二的跟踪器在很大程度上依赖于暂时匹配的无效噪音，忽略了原始数据结构，从而确保了培训视频数据私人关系。为了提高周二的有效性，我们引入了时间对比度损失，这进一步破坏了使用我们的星期二进行培训时的现有跟踪器的学习。广泛的实验表明，我们的方法在视频数据私人保护保护中实现了最新的性能，并且在投票模型，数据集和时间匹配任务之间具有强大的可传递性。

Title: Divergence Minimization Preference Optimization for Diffusion Model Alignment

• Authors: Binxu Li, Minkai Xu, Meihua Dang, Stefano Ermon
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2507.07510
• Pdf URL: https://arxiv.org/pdf/2507.07510
• Copy Paste: [[2507.07510]] Divergence Minimization Preference Optimization for Diffusion Model Alignment(https://arxiv.org/abs/2507.07510)
• Keywords: generative
• Abstract: Diffusion models have achieved remarkable success in generating realistic and versatile images from text prompts. Inspired by the recent advancements of language models, there is an increasing interest in further improving the models by aligning with human preferences. However, we investigate alignment from a divergence minimization perspective and reveal that existing preference optimization methods are typically trapped in suboptimal mean-seeking optimization. In this paper, we introduce Divergence Minimization Preference Optimization (DMPO), a novel and principled method for aligning diffusion models by minimizing reverse KL divergence, which asymptotically enjoys the same optimization direction as original RL. We provide rigorous analysis to justify the effectiveness of DMPO and conduct comprehensive experiments to validate its empirical strength across both human evaluations and automatic metrics. Our extensive results show that diffusion models fine-tuned with DMPO can consistently outperform or match existing techniques, specifically outperforming all existing diffusion alignment baselines by at least 64.6% in PickScore across all evaluation datasets, demonstrating the method's superiority in aligning generative behavior with desired outputs. Overall, DMPO unlocks a robust and elegant pathway for preference alignment, bridging principled theory with practical performance in diffusion models.
• 摘要：扩散模型在从文本提示中生成现实和多功能的图像方面取得了巨大的成功。受语言模型的最新进步的启发，人们对通过与人类偏好保持一致的进一步改善模型产生了越来越多的兴趣。但是，我们从差异最小化的角度研究对齐，并揭示现有的偏好优化方法通常被困在次优的寻求优化中。在本文中，我们引入了Divergence最小化偏好优化（DMPO），这是一种通过最大程度地减少反向KL差异来对齐扩散模型的新颖和原则性方法，该方法渐近地享有与原始RL相同的优化方向。我们提供严格的分析，以证明DMPO的有效性并进行全面的实验，以验证其在人类评估和自动指标中的经验强度。我们的广泛结果表明，用DMPO微调的扩散模型可以始终跑赢或匹配现有技术，特别是在所有评估数据集中，PickScore中至少超过了所有现有的扩散对齐基线，在所有评估数据集中至少超过64.6％，这证明了该方法在使生成性行为与所需的输出相吻合方面的优势。总体而言，DMPO解锁了偏好对齐的强大而优雅的途径，在扩散模型中具有实践性的桥接原则性理论。

Title: Stable-Hair v2: Real-World Hair Transfer via Multiple-View Diffusion Model

• Authors: Kuiyuan Sun, Yuxuan Zhang, Jichao Zhang, Jiaming Liu, Wei Wang, Niculae Sebe, Yao Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2507.07591
• Pdf URL: https://arxiv.org/pdf/2507.07591
• Copy Paste: [[2507.07591]] Stable-Hair v2: Real-World Hair Transfer via Multiple-View Diffusion Model(https://arxiv.org/abs/2507.07591)
• Keywords: generation
• Abstract: While diffusion-based methods have shown impressive capabilities in capturing diverse and complex hairstyles, their ability to generate consistent and high-quality multi-view outputs -- crucial for real-world applications such as digital humans and virtual avatars -- remains underexplored. In this paper, we propose Stable-Hair v2, a novel diffusion-based multi-view hair transfer framework. To the best of our knowledge, this is the first work to leverage multi-view diffusion models for robust, high-fidelity, and view-consistent hair transfer across multiple perspectives. We introduce a comprehensive multi-view training data generation pipeline comprising a diffusion-based Bald Converter, a data-augment inpainting model, and a face-finetuned multi-view diffusion model to generate high-quality triplet data, including bald images, reference hairstyles, and view-aligned source-bald pairs. Our multi-view hair transfer model integrates polar-azimuth embeddings for pose conditioning and temporal attention layers to ensure smooth transitions between views. To optimize this model, we design a novel multi-stage training strategy consisting of pose-controllable latent IdentityNet training, hair extractor training, and temporal attention training. Extensive experiments demonstrate that our method accurately transfers detailed and realistic hairstyles to source subjects while achieving seamless and consistent results across views, significantly outperforming existing methods and establishing a new benchmark in multi-view hair transfer. Code is publicly available at this https URL.
• 摘要：尽管基于扩散的方法在捕获多样化且复杂的发型方面表现出了令人印象深刻的能力，但它们产生一致和高质量的多视图输出的能力 - 对于像数字人类和虚拟头像等现实世界应用至关重要 - 仍然没有受到驱动。在本文中，我们提出了稳定的头发V2，这是一种新型的基于扩散的多视图发型框架。据我们所知，这是第一项利用多视图扩散模型的工作，以跨多个角度跨多个观点转移稳健，高保真和一致的头发转移。我们介绍了一个全面的多观培训数据生成管道，该管道包括基于扩散的秃头转换器，一个数据启动介绍模型以及面部调节的多视图扩散模型，以生成高质量的三重态数据，包括秃头图像，参考发型，参考发型和观看型与view的源源。我们的多视视头发转移模型集成了姿势调节和时间注意层的极性嵌入，以确保视图之间的平滑过渡。为了优化该模型，我们设计了一种新型的多阶段训练策略，该策略包括可控制的潜在标识网训练，提取器培训和时间注意力训练。广泛的实验表明，我们的方法准确地将详细和逼真的发型转移到来源的主题中，同时在视图之间取得无缝且一致的结果，显着优于现有方法，并在多视频发型中建立了新的基准测试。代码在此HTTPS URL上公开可用。

Title: Synthetic MC via Biological Transmitters: Therapeutic Modulation of the Gut-Brain Axis

• Authors: Sebastian Lotter, Elisabeth Mohr, Andrina Rutsch, Lukas Brand, Francesca Ronchi, Laura Díaz-Marugán
• Subjects: cs.LG, q-bio.QM, q-bio.TO
• Abstract URL: https://arxiv.org/abs/2507.07604
• Pdf URL: https://arxiv.org/pdf/2507.07604
• Copy Paste: [[2507.07604]] Synthetic MC via Biological Transmitters: Therapeutic Modulation of the Gut-Brain Axis(https://arxiv.org/abs/2507.07604)
• Keywords: generation
• Abstract: Synthetic molecular communication (SMC) is a key enabler for future healthcare systems in which Internet of Bio-Nano-Things (IoBNT) devices facilitate the continuous monitoring of a patient's biochemical signals. To close the loop between sensing and actuation, both the detection and the generation of in-body molecular communication (MC) signals is key. However, generating signals inside the human body, e.g., via synthetic nanodevices, poses a challenge in SMC, due to technological obstacles as well as legal, safety, and ethical issues. Hence, this paper considers an SMC system in which signals are generated indirectly via the modulation of a natural in-body MC system, namely the gut-brain axis (GBA). Therapeutic GBA modulation is already established as treatment for neurological diseases, e.g., drug refractory epilepsy (DRE), and performed via the administration of nutritional supplements or specific diets. However, the molecular signaling pathways that mediate the effect of such treatments are mostly unknown. Consequently, existing treatments are standardized or designed heuristically and able to help only some patients while failing to help others. In this paper, we propose to leverage personal health data, e.g., gathered by in-body IoBNT devices, to design more versatile and robust GBA modulation-based treatments as compared to the existing ones. To show the feasibility of our approach, we define a catalog of theoretical requirements for therapeutic GBA modulation. Then, we propose a machine learning model to verify these requirements for practical scenarios when only limited data on the GBA modulation exists. By evaluating the proposed model on several datasets, we confirm its excellent accuracy in identifying different modulators of the GBA. Finally, we utilize the proposed model to identify specific modulatory pathways that play an important role for therapeutic GBA modulation.
• 摘要：合成分子通信（SMC）是未来医疗保健系统的关键推动力，其中生物纳米群（IOBNT）设备有助于对患者的生化信号进行持续监测。为了关闭感应和驱动之间的循环，检测和体内分子通信（MC）信号的产生都是关键。但是，由于技术障碍以及法律，安全性和道德问题，例如通过合成纳米版在人体内部产生信号，在SMC中构成了挑战。因此，本文考虑了一个SMC系统，该系统通过自然体内MC系统（即肠脑轴（GBA））间接生成信号。治疗性GBA调节已被确定为神经疾病的治疗方法，例如药物难治性癫痫（DRE），并通过给药营养补充剂或特定饮食进行。但是，介导此类治疗作用的分子信号通路大多未知。因此，现有的治疗方法是标准化或启发式设计的，并且能够仅帮助某些患者，同时无法帮助其他患者。在本文中，我们建议利用个人健康数据，例如由内部IOBNT设备收集，以设计与现有数据相比，基于GBA调制的更广泛，强大的基于GBA调制的治疗方法。为了显示我们方法的可行性，我们定义了治疗性GBA调制的理论要求目录。然后，我们提出了一个机器学习模型，以验证仅存在GBA调制的数据时，对实际情况的这些要求。通过在几个数据集上评估所提出的模型，我们确认了它在识别GBA不同调节剂方面的出色准确性。最后，我们利用所提出的模型来确定特定的调节途径，这些途径对于治疗性GBA调节起着重要作用。

Title: Sparse Causal Discovery with Generative Intervention for Unsupervised Graph Domain Adaptation

• Authors: Junyu Luo, Yuhao Tang, Yiwei Fu, Xiao Luo, Zhizhuo Kou, Zhiping Xiao, Wei Ju, Wentao Zhang, Ming Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2507.07621
• Pdf URL: https://arxiv.org/pdf/2507.07621
• Copy Paste: [[2507.07621]] Sparse Causal Discovery with Generative Intervention for Unsupervised Graph Domain Adaptation(https://arxiv.org/abs/2507.07621)
• Keywords: generative
• Abstract: Unsupervised Graph Domain Adaptation (UGDA) leverages labeled source domain graphs to achieve effective performance in unlabeled target domains despite distribution shifts. However, existing methods often yield suboptimal results due to the entanglement of causal-spurious features and the failure of global alignment strategies. We propose SLOGAN (Sparse Causal Discovery with Generative Intervention), a novel approach that achieves stable graph representation transfer through sparse causal modeling and dynamic intervention mechanisms. Specifically, SLOGAN first constructs a sparse causal graph structure, leveraging mutual information bottleneck constraints to disentangle sparse, stable causal features while compressing domain-dependent spurious correlations through variational inference. To address residual spurious correlations, we innovatively design a generative intervention mechanism that breaks local spurious couplings through cross-domain feature recombination while maintaining causal feature semantic consistency via covariance constraints. Furthermore, to mitigate error accumulation in target domain pseudo-labels, we introduce a category-adaptive dynamic calibration strategy, ensuring stable discriminative learning. Extensive experiments on multiple real-world datasets demonstrate that SLOGAN significantly outperforms existing baselines.
• 摘要：无监督的图形域适应（UGDA）利用标有源域图的标记为源域图，尽管分布偏移，但在未标记的目标域中实现了有效的性能。但是，由于因果关系特征的纠缠以及全球一致性策略的失败，现有方法通常会产生次优的结果。我们提出口号（稀疏的因果关系，并具有生成干预），这是一种新的方法，通过稀疏因果建模和动态干预机制实现稳定的图表表示转移。具体而言，口号首先构建了稀疏的因果图结构，利用相互信息的瓶颈约束到稀疏，稳定的因果特征，同时通过变异推断压缩域依赖性的虚假相关性。为了解决残留的虚假相关性，我们创新设计了一种生成干预机制，该机制通过跨域特征重组打破了当地的伪造耦合，同时通过协方差约束保持因果特征语义一致性。此外，为了减轻目标域伪标签中的错误积累，我们引入了一种类别自适应的动态校准策略，以确保稳定的判别性学习。在多个现实世界数据集上进行的广泛实验表明，口号明显优于现有基线。

Title: T-GVC: Trajectory-Guided Generative Video Coding at Ultra-Low Bitrates

• Authors: Zhitao Wang, Hengyu Man, Wenrui Li, Xingtao Wang, Xiaopeng Fan, Debin Zhao
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2507.07633
• Pdf URL: https://arxiv.org/pdf/2507.07633
• Copy Paste: [[2507.07633]] T-GVC: Trajectory-Guided Generative Video Coding at Ultra-Low Bitrates(https://arxiv.org/abs/2507.07633)
• Keywords: generation, generative
• Abstract: Recent advances in video generation techniques have given rise to an emerging paradigm of generative video coding, aiming to achieve semantically accurate reconstructions in Ultra-Low Bitrate (ULB) scenarios by leveraging strong generative priors. However, most existing methods are limited by domain specificity (e.g., facial or human videos) or an excessive dependence on high-level text guidance, which often fails to capture motion details and results in unrealistic reconstructions. To address these challenges, we propose a Trajectory-Guided Generative Video Coding framework (dubbed T-GVC). T-GVC employs a semantic-aware sparse motion sampling pipeline to effectively bridge low-level motion tracking with high-level semantic understanding by extracting pixel-wise motion as sparse trajectory points based on their semantic importance, not only significantly reducing the bitrate but also preserving critical temporal semantic information. In addition, by incorporating trajectory-aligned loss constraints into diffusion processes, we introduce a training-free latent space guidance mechanism to ensure physically plausible motion patterns without sacrificing the inherent capabilities of generative models. Experimental results demonstrate that our framework outperforms both traditional codecs and state-of-the-art end-to-end video compression methods under ULB conditions. Furthermore, additional experiments confirm that our approach achieves more precise motion control than existing text-guided methods, paving the way for a novel direction of generative video coding guided by geometric motion modeling.
• 摘要：视频生成技术的最新进展引起了生成视频编码的新兴范式，旨在通过利用强大的生成先验来实现超低比特率（ULB）方案的语义准确重建。但是，大多数现有方法受域特异性（例如面部或人类视频）或对高级文本指导的过度依赖的限制，该指导通常无法捕获运动细节并导致不切实际的重建。为了应对这些挑战，我们提出了一个轨迹引导的生成视频编码框架（称为T-GVC）。 T-GVC采用语义意识的稀疏运动采样管道来有效地桥接低级运动跟踪，并通过将像素化运动作为稀疏轨迹点提取基于其语义重要性，而不仅会显着降低比特率，而且还可以显着降低比特率，还可以保留关键的临时语义信息，从而有效地桥接了低级语义理解。此外，通过将轨迹一致的损失约束纳入扩散过程中，我们引入了一种无训练的潜在空间指导机制，以确保物理上合理的运动模式，而无需牺牲生成模型的固有功能。实验结果表明，在ULB条件下，我们的框架表现优于传统的编解码器和最新的端到端视频压缩方法。此外，其他实验证实，与现有的文本指导方法相比，我们的方法可以实现更精确的运动控制，这为通过几何运动建模指导的新生成视频编码方向铺平了道路。

Title: Motion-Aware Adaptive Pixel Pruning for Efficient Local Motion Deblurring

• Authors: Wei Shang, Dongwei Ren, Wanying Zhang, Pengfei Zhu, Qinghua Hu, Wangmeng Zuo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2507.07708
• Pdf URL: https://arxiv.org/pdf/2507.07708
• Copy Paste: [[2507.07708]] Motion-Aware Adaptive Pixel Pruning for Efficient Local Motion Deblurring(https://arxiv.org/abs/2507.07708)
• Keywords: restoration
• Abstract: Local motion blur in digital images originates from the relative motion between dynamic objects and static imaging systems during exposure. Existing deblurring methods face significant challenges in addressing this problem due to their inefficient allocation of computational resources and inadequate handling of spatially varying blur patterns. To overcome these limitations, we first propose a trainable mask predictor that identifies blurred regions in the image. During training, we employ blur masks to exclude sharp regions. For inference optimization, we implement structural reparameterization by converting $3\times 3$ convolutions to computationally efficient $1\times 1$ convolutions, enabling pixel-level pruning of sharp areas to reduce computation. Second, we develop an intra-frame motion analyzer that translates relative pixel displacements into motion trajectories, establishing adaptive guidance for region-specific blur restoration. Our method is trained end-to-end using a combination of reconstruction loss, reblur loss, and mask loss guided by annotated blur masks. Extensive experiments demonstrate superior performance over state-of-the-art methods on both local and global blur datasets while reducing FLOPs by 49\% compared to SOTA models (e.g., LMD-ViT). The source code is available at this https URL.
• 摘要：数字图像中的局部运动模糊源自曝光过程中动态对象和静态成像系统之间的相对运动。由于其计算资源的分配效率低下，并且处理空间变化的模糊模式不足，因此现有的Deblurring方法在解决此问题方面面临重大挑战。为了克服这些局限性，我们首先提出了一个可训练的遮罩预测指标，该预测指标可以识别图像中的模糊区域。在培训期间，我们采用模糊面具来排除锋利的区域。为了进行推理优化，我们通过将$ 3 \ times 3 $卷积转换为计算高效的$ 1 \ times 1 $卷积来实现结构重新聚体化，从而使像素级修剪敏锐的区域可以减少计算。其次，我们开发了一个框内运动分析仪，该运动分析仪将相对像素位移转化为运动轨迹，从而为区域特异性模糊恢复建立自适应指导。我们的方法是通过重建损失，reblur损失和由带注释的模糊面膜引导的掩盖损失和掩盖丢失的组合进行训练的。广泛的实验表明，与SOTA模型（例如LMD-VIT）相比，在本地和全局模糊数据集上的最先进方法中的性能优于最先进的方法。源代码可在此HTTPS URL上找到。

Title: Breast Ultrasound Tumor Generation via Mask Generator and Text-Guided Network:A Clinically Controllable Framework with Downstream Evaluation

• Authors: Haoyu Pan, Hongxin Lin, Zetian Feng, Chuxuan Lin, Junyang Mo, Chu Zhang, Zijian Wu, Yi Wang, Qingqing Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2507.07721
• Pdf URL: https://arxiv.org/pdf/2507.07721
• Copy Paste: [[2507.07721]] Breast Ultrasound Tumor Generation via Mask Generator and Text-Guided Network:A Clinically Controllable Framework with Downstream Evaluation(https://arxiv.org/abs/2507.07721)
• Keywords: generation, generative
• Abstract: The development of robust deep learning models for breast ultrasound (BUS) image analysis is significantly constrained by the scarcity of expert-annotated data. To address this limitation, we propose a clinically controllable generative framework for synthesizing BUS images. This framework integrates clinical descriptions with structural masks to generate tumors, enabling fine-grained control over tumor characteristics such as morphology, echogencity, and shape. Furthermore, we design a semantic-curvature mask generator, which synthesizes structurally diverse tumor masks guided by clinical priors. During inference, synthetic tumor masks serve as input to the generative framework, producing highly personalized synthetic BUS images with tumors that reflect real-world morphological diversity. Quantitative evaluations on six public BUS datasets demonstrate the significant clinical utility of our synthetic images, showing their effectiveness in enhancing downstream breast cancer diagnosis tasks. Furthermore, visual Turing tests conducted by experienced sonographers confirm the realism of the generated images, indicating the framework's potential to support broader clinical applications.
• 摘要：乳房超声（BUS）图像分析的强大深度学习模型的开发受到专家通知数据的稀缺性的限制。为了解决这一限制，我们提出了一个可控制的生成框架，用于合成总线图像。该框架将临床描述与结构面具相结合，以产生肿瘤，从而对肿瘤特征（例如形态，回声性和形状）进行细粒度的控制。此外，我们设计了一个语义曲面掩膜发生器，该发电机综合了由临床先验引导的结构多样的肿瘤口罩。在推断期间，合成肿瘤面膜是生成框架的输入，产生具有反映现实世界形态多样性的肿瘤的高度个性化的合成总线图像。六个公共巴士数据集的定量评估证明了我们的合成图像的重要临床实用性，显示了它们在增强下游乳腺癌诊断任务方面的有效性。此外，经验丰富的超声师进行的视觉图灵测试证实了生成的图像的现实主义，这表明该框架支持更广泛的临床应用的潜力。

Title: GuardVal: Dynamic Large Language Model Jailbreak Evaluation for Comprehensive Safety Testing

• Authors: Peiyan Zhang, Haibo Jin, Liying Kang, Haohan Wang
• Subjects: cs.LG, cs.CL, cs.CR
• Abstract URL: https://arxiv.org/abs/2507.07735
• Pdf URL: https://arxiv.org/pdf/2507.07735
• Copy Paste: [[2507.07735]] GuardVal: Dynamic Large Language Model Jailbreak Evaluation for Comprehensive Safety Testing(https://arxiv.org/abs/2507.07735)
• Keywords: generation
• Abstract: Jailbreak attacks reveal critical vulnerabilities in Large Language Models (LLMs) by causing them to generate harmful or unethical content. Evaluating these threats is particularly challenging due to the evolving nature of LLMs and the sophistication required in effectively probing their vulnerabilities. Current benchmarks and evaluation methods struggle to fully address these challenges, leaving gaps in the assessment of LLM vulnerabilities. In this paper, we review existing jailbreak evaluation practices and identify three assumed desiderata for an effective jailbreak evaluation protocol. To address these challenges, we introduce GuardVal, a new evaluation protocol that dynamically generates and refines jailbreak prompts based on the defender LLM's state, providing a more accurate assessment of defender LLMs' capacity to handle safety-critical situations. Moreover, we propose a new optimization method that prevents stagnation during prompt refinement, ensuring the generation of increasingly effective jailbreak prompts that expose deeper weaknesses in the defender LLMs. We apply this protocol to a diverse set of models, from Mistral-7b to GPT-4, across 10 safety domains. Our findings highlight distinct behavioral patterns among the models, offering a comprehensive view of their robustness. Furthermore, our evaluation process deepens the understanding of LLM behavior, leading to insights that can inform future research and drive the development of more secure models.
• 摘要：越狱攻击揭示了大语言模型（LLM）的关键漏洞，从而导致它们产生有害或不道德的内容。由于LLM的不断发展的性质以及有效探索其脆弱性所需的复杂性，评估这些威胁尤其具有挑战性。当前的基准和评估方法难以充分解决这些挑战，在评估LLM漏洞的评估中留下了差距。在本文中，我们审查了现有的越狱评估实践，并确定了三个假定的Desiderata，以制定有效的越狱评估协议。为了应对这些挑战，我们引入了Guardval，这是一种新的评估协议，该协议根据Defender LLM的状态动态生成和完善越狱提示，从而更准确地评估了Defender LLMS处理安全至关重要情况的能力。此外，我们提出了一种新的优化方法，该方法可以防止在迅速完善过程中停滞，从而确保产生越来越有效的越狱提示，这些越狱促使Defender LLMS中更深的弱点。我们将此协议应用于从10个安全域中从Mistral-7B到GPT-4的各种模型。我们的发现突出了模型中不同的行为模式，为它们的稳健性提供了全面的看法。此外，我们的评估过程加深了对LLM行为的理解，从而提供了可以为未来的研究提供信息并推动更安全模型的开发的见解。

Title: Rethinking Query-based Transformer for Continual Image Segmentation

• Authors: Yuchen Zhu, Cheng Shi, Dingyou Wang, Jiajin Tang, Zhengxuan Wei, Yu Wu, Guanbin Li, Sibei Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2507.07831
• Pdf URL: https://arxiv.org/pdf/2507.07831
• Copy Paste: [[2507.07831]] Rethinking Query-based Transformer for Continual Image Segmentation(https://arxiv.org/abs/2507.07831)
• Keywords: generation
• Abstract: Class-incremental/Continual image segmentation (CIS) aims to train an image segmenter in stages, where the set of available categories differs at each stage. To leverage the built-in objectness of query-based transformers, which mitigates catastrophic forgetting of mask proposals, current methods often decouple mask generation from the continual learning process. This study, however, identifies two key issues with decoupled frameworks: loss of plasticity and heavy reliance on input data order. To address these, we conduct an in-depth investigation of the built-in objectness and find that highly aggregated image features provide a shortcut for queries to generate masks through simple feature alignment. Based on this, we propose SimCIS, a simple yet powerful baseline for CIS. Its core idea is to directly select image features for query assignment, ensuring "perfect alignment" to preserve objectness, while simultaneously allowing queries to select new classes to promote plasticity. To further combat catastrophic forgetting of categories, we introduce cross-stage consistency in selection and an innovative "visual query"-based replay mechanism. Experiments demonstrate that SimCIS consistently outperforms state-of-the-art methods across various segmentation tasks, settings, splits, and input data orders. All models and codes will be made publicly available at this https URL.
• 摘要：班级信息/持续图像分割（CIS）旨在在各个阶段训练图像分割器，每个阶段的一组可用类别都不同。为了利用基于查询的变压器的内置对象，从而减轻了灾难性的遗忘面具建议，当前的方法通常会使蒙版生成与持续的学习过程相结合。但是，这项研究确定了解耦框架的两个关键问题：可塑性的丧失和对输入数据顺序的严重依赖。为了解决这些问题，我们对内置对象进行了深入的研究，并发现高度聚合的图像功能为查询提供了一个快捷方式，可以通过简单的特征对准来生成口罩。基于此，我们提出了Simcis，这是CIS的简单而强大的基线。它的核心想法是直接选择查询分配的图像功能，确保“完美的对齐”来保留对象，同时允许查询可以选择新的类以促进可塑性。为了进一步打击灾难性的遗忘类别，我们在选择方面引入了跨阶段的一致性和创新的“视觉查询”重播机制。实验表明，在各种分割任务，设置，拆分和输入数据订单上，Simcis始终胜过最先进的方法。所有模型和代码将在此HTTPS URL上公开可用。

Title: Predicting and generating antibiotics against future pathogens with ApexOracle

• Authors: Tianang Leng, Fangping Wan, Marcelo Der Torossian Torres, Cesar de la Fuente-Nunez
• Subjects: cs.LG, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2507.07862
• Pdf URL: https://arxiv.org/pdf/2507.07862
• Copy Paste: [[2507.07862]] Predicting and generating antibiotics against future pathogens with ApexOracle(https://arxiv.org/abs/2507.07862)
• Keywords: generation
• Abstract: Antimicrobial resistance (AMR) is escalating and outpacing current antibiotic development. Thus, discovering antibiotics effective against emerging pathogens is becoming increasingly critical. However, existing approaches cannot rapidly identify effective molecules against novel pathogens or emerging drug-resistant strains. Here, we introduce ApexOracle, an artificial intelligence (AI) model that both predicts the antibacterial potency of existing compounds and designs de novo molecules active against strains it has never encountered. Departing from models that rely solely on molecular features, ApexOracle incorporates pathogen-specific context through the integration of molecular features captured via a foundational discrete diffusion language model and a dual-embedding framework that combines genomic- and literature-derived strain representations. Across diverse bacterial species and chemical modalities, ApexOracle consistently outperformed state-of-the-art approaches in activity prediction and demonstrated reliable transferability to novel pathogens with little or no antimicrobial data. Its unified representation-generation architecture further enables the in silico creation of "new-to-nature" molecules with high predicted efficacy against priority threats. By pairing rapid activity prediction with targeted molecular generation, ApexOracle offers a scalable strategy for countering AMR and preparing for future infectious-disease outbreaks.
• 摘要：抗菌耐药性（AMR）正在升级和超过当前的抗生素发育。因此，发现有效抗新兴病原体的抗生素变得越来越重要。但是，现有方法无法快速鉴定针对新型病原体或新兴药物菌株的有效分子。在这里，我们介绍了Apexoracle，这是一种人工智能（AI）模型，两者都可以预测现有化合物和设计的抗菌效力，从头开始，从未遇到过的菌株。 Apexoracle偏离了仅依赖分子特征的模型，通过整合通过基础离散扩散语言模型捕获的分子特征和结合了基因组和文献衍生的菌株表征的双重插入框架，结合了病原体特异性的上下文。在各种细菌物种和化学模式中，基座在活性预测中始终超过最先进的方法，并且显示出对新型病原体的可靠转移性，几乎没有抗菌数据。其统一的代表生成结构进一步使得在硅基创建“新到天然”的分子中具有很高的预测疗效，以应对优先级威胁。通过将快速活动预测与靶向分子产生配对，Apexoracle提供了一种可扩展的策略，以抵抗AMR并为将来的感染性疾病暴发做准备。

Title: Single-Step Latent Diffusion for Underwater Image Restoration

• Authors: Jiayi Wu, Tianfu Wang, Md Abu Bakr Siddique, Md Jahidul Islam, Cornelia Fermuller, Yiannis Aloimonos, Christopher A. Metzler
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2507.07878
• Pdf URL: https://arxiv.org/pdf/2507.07878
• Copy Paste: [[2507.07878]] Single-Step Latent Diffusion for Underwater Image Restoration(https://arxiv.org/abs/2507.07878)
• Keywords: restoration, generation
• Abstract: Underwater image restoration algorithms seek to restore the color, contrast, and appearance of a scene that is imaged underwater. They are a critical tool in applications ranging from marine ecology and aquaculture to underwater construction and archaeology. While existing pixel-domain diffusion-based image restoration approaches are effective at restoring simple scenes with limited depth variation, they are computationally intensive and often generate unrealistic artifacts when applied to scenes with complex geometry and significant depth variation. In this work we overcome these limitations by combining a novel network architecture (SLURPP) with an accurate synthetic data generation pipeline. SLURPP combines pretrained latent diffusion models -- which encode strong priors on the geometry and depth of scenes -- with an explicit scene decomposition -- which allows one to model and account for the effects of light attenuation and backscattering. To train SLURPP we design a physics-based underwater image synthesis pipeline that applies varied and realistic underwater degradation effects to existing terrestrial image datasets. This approach enables the generation of diverse training data with dense medium/degradation annotations. We evaluate our method extensively on both synthetic and real-world benchmarks and demonstrate state-of-the-art performance. Notably, SLURPP is over 200X faster than existing diffusion-based methods while offering ~ 3 dB improvement in PSNR on synthetic benchmarks. It also offers compelling qualitative improvements on real-world data. Project website this https URL.
• 摘要：水下图像恢复算法试图恢复在水下成像的场景的颜色，对比度和外观。它们是从海洋生态和水产养殖到水下建设和考古学等应用的关键工具。尽管现有的像素域扩散恢复方法可有效恢复深度变化有限的简单场景，但它们在计算上是密集的，并且当应用于具有复杂几何形状和显着深度变化的场景时，通常会产生不切实际的伪影。在这项工作中，我们通过将新型网络体系结构（SLURP）与准确的合成数据生成管道相结合来克服这些局限性。 Slurpp结合了预处理的潜扩散模型 - 对场景的几何形状和深度编码强的先验 - 与明确的场景分解 - 允许人们建模并说明光衰减和反向散射的效果。为了训练slurpp，我们设计了一个基于物理的水下图像合成管道，该管道将各种且现实的水下退化效应应用于现有的陆地图像数据集。这种方法可以通过密集的培养基/降解注释产生各种培训数据。我们对合成基准和现实基准测试进行了广泛的评估，并展示了最先进的性能。值得注意的是，SLURPP的速度比现有基于扩散的方法快200倍，同时在合成基准上提供了〜3 dB的改进。它还对现实世界数据提供了引人入胜的定性改进。项目网站此HTTPS URL。

Title: MIRA: A Novel Framework for Fusing Modalities in Medical RAG

• Authors: Jinhong Wang, Tajamul Ashraf, Zongyan Han, Jorma Laaksonen, Rao Mohammad Anwer
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2507.07902
• Pdf URL: https://arxiv.org/pdf/2507.07902
• Copy Paste: [[2507.07902]] MIRA: A Novel Framework for Fusing Modalities in Medical RAG(https://arxiv.org/abs/2507.07902)
• Keywords: generation
• Abstract: Multimodal Large Language Models (MLLMs) have significantly advanced AI-assisted medical diagnosis, but they often generate factually inconsistent responses that deviate from established medical knowledge. Retrieval-Augmented Generation (RAG) enhances factual accuracy by integrating external sources, but it presents two key challenges. First, insufficient retrieval can miss critical information, whereas excessive retrieval can introduce irrelevant or misleading content, disrupting model output. Second, even when the model initially provides correct answers, over-reliance on retrieved data can lead to factual errors. To address these issues, we introduce the Multimodal Intelligent Retrieval and Augmentation (MIRA) framework, designed to optimize factual accuracy in MLLM. MIRA consists of two key components: (1) a calibrated Rethinking and Rearrangement module that dynamically adjusts the number of retrieved contexts to manage factual risk, and (2) A medical RAG framework integrating image embeddings and a medical knowledge base with a query-rewrite module for efficient multimodal reasoning. This enables the model to effectively integrate both its inherent knowledge and external references. Our evaluation of publicly available medical VQA and report generation benchmarks demonstrates that MIRA substantially enhances factual accuracy and overall performance, achieving new state-of-the-art results. Code is released at this https URL.
• 摘要：多模式大语模型（MLLM）具有明显的AI辅助医学诊断，但它们通常会产生实际上不一致的反应，这些反应偏离了既定的医学知识。检索演示的一代（RAG）通过整合外部来源提高了事实准确性，但它提出了两个关键挑战。首先，检索不足可能会错过关键信息，而过多的检索可以引入无关紧要或误导性的内容，从而破坏模型输出。其次，即使模型最初提供正确的答案，对检索到的数据的过度依赖也会导致事实错误。为了解决这些问题，我们介绍了多模式的智能检索和增强（MIRA）框架，旨在优化MLLM的事实准确性。 MIRA由两个关键组成部分组成：（1）校准的重新思考和重新排列模块，该模块动态调整了管理事实风险的检索上下文的数量，以及（2）一个将图像嵌入图像嵌入的医学抹布框架和与查询 - 鲁棒模块集成的医学知识基础，以有效的多模型推理。这使该模型能够有效整合其固有的知识和外部参考。我们对公开可用的医疗VQA和报告生成基准的评估表明，MIRA显着提高了事实准确性和整体绩效，从而实现了新的最新结果。代码在此HTTPS URL上发布。

Title: Low Resource Reconstruction Attacks Through Benign Prompts

• Authors: Sol Yarkoni, Roi Livni
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.07947
• Pdf URL: https://arxiv.org/pdf/2507.07947
• Copy Paste: [[2507.07947]] Low Resource Reconstruction Attacks Through Benign Prompts(https://arxiv.org/abs/2507.07947)
• Keywords: generative
• Abstract: The recent advances in generative models such as diffusion models have raised several risks and concerns related to privacy, copyright infringements and data stewardship. To better understand and control the risks, various researchers have created techniques, experiments and attacks that reconstruct images, or part of images, from the training set. While these techniques already establish that data from the training set can be reconstructed, they often rely on high-resources, excess to the training set as well as well-engineered and designed prompts. In this work, we devise a new attack that requires low resources, assumes little to no access to the actual training set, and identifies, seemingly, benign prompts that lead to potentially-risky image reconstruction. This highlights the risk that images might even be reconstructed by an uninformed user and unintentionally. For example, we identified that, with regard to one existing model, the prompt ``blue Unisex T-Shirt'' can generate the face of a real-life human model. Our method builds on an intuition from previous works which leverages domain knowledge and identifies a fundamental vulnerability that stems from the use of scraped data from e-commerce platforms, where templated layouts and images are tied to pattern-like prompts.
• 摘要：生成模型（例如扩散模型）的最新进展提高了与隐私，侵犯版权和数据管理有关的几种风险和问题。为了更好地理解和控制风险，各种研究人员创建了从培训集中重建图像或图像的一部分的技术，实验和攻击。尽管这些技术已经确定可以重建培训集中的数据，但它们通常依靠高资源，过多地训练集以及精心设计和设计的提示。在这项工作中，我们设计了一项新的攻击，该攻击需要低资源，几乎没有访问实际的训练集，并且可以识别出可能导致潜在风险的图像重建的良性提示。这强调了图像甚至可能由不知情的用户重建和无意间重建的风险。例如，我们确定，关于一种现有模型，提示``蓝色男女unisex t恤''可以产生现实生活中的人类模型的面孔。我们的方法建立在以前的作品的直觉上，该工作利用域知识并确定了一个基本脆弱性，该漏洞源于从电子商务平台中使用刮擦数据的基本脆弱性，在该平台上使用了模板的布局和图像与类似模式的提示绑定。

Title: Scaling RL to Long Videos

• Authors: Yukang Chen, Wei Huang, Baifeng Shi, Qinghao Hu, Hanrong Ye, Ligeng Zhu, Zhijian Liu, Pavlo Molchanov, Jan Kautz, Xiaojuan Qi, Sifei Liu, Hongxu Yin, Yao Lu, Song Han
• Subjects: cs.CV, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2507.07966
• Pdf URL: https://arxiv.org/pdf/2507.07966
• Copy Paste: [[2507.07966]] Scaling RL to Long Videos(https://arxiv.org/abs/2507.07966)
• Keywords: generation
• Abstract: We introduce a full-stack framework that scales up reasoning in vision-language models (VLMs) to long videos, leveraging reinforcement learning. We address the unique challenges of long video reasoning by integrating three critical components: (1) a large-scale dataset, LongVideo-Reason, comprising 52K long video QA pairs with high-quality reasoning annotations across diverse domains such as sports, games, and vlogs; (2) a two-stage training pipeline that extends VLMs with chain-of-thought supervised fine-tuning (CoT-SFT) and reinforcement learning (RL); and (3) a training infrastructure for long video RL, named Multi-modal Reinforcement Sequence Parallelism (MR-SP), which incorporates sequence parallelism and a vLLM-based engine tailored for long video, using cached video embeddings for efficient rollout and prefilling. In experiments, LongVILA-R1-7B achieves strong performance on long video QA benchmarks such as VideoMME. It also outperforms Video-R1-7B and even matches Gemini-1.5-Pro across temporal reasoning, goal and purpose reasoning, spatial reasoning, and plot reasoning on our LongVideo-Reason-eval benchmark. Notably, our MR-SP system achieves up to 2.1x speedup on long video RL training. LongVILA-R1 demonstrates consistent performance gains as the number of input video frames scales. LongVILA-R1 marks a firm step towards long video reasoning in VLMs. In addition, we release our training system for public availability that supports RL training on various modalities (video, text, and audio), various models (VILA and Qwen series), and even image and video generation models. On a single A100 node (8 GPUs), it supports RL training on hour-long videos (e.g., 3,600 frames / around 256k tokens).
• 摘要：我们介绍了一个全栈框架，该框架将视觉模型（VLM）中的推理扩展到了长时间的视频，从而利用了强化学习。我们通过整合三个关键组件来应对长期视频推理的独特挑战：（1）一个大规模的数据集，Longvideo-Reseason，包括52K长的视频QA对，具有高质量的推理注释，包括体育，游戏和Vlogs等各种领域； （2）一条两阶段的培训管道，通过经过经过经过经过监管的链条的微调（COT-SFT）和增强学习（RL）扩展VLM； （3）长期视频RL的训练基础架构，称为多模式增强序列并行性（MR-SP），该序列并行了序列并行性和针对长视频的VLLM基于VLLM的发动机，并使用加速视频嵌入式嵌入式嵌入式嵌入式嵌入式嵌入式嵌入，以有效地推出和预填充。在实验中，Longvila-R1-7B在长时间视频基准（例如Videmomme）上取得了强劲的性能。它还胜过视频R1-7B，甚至匹配跨时间的推理，目标和目的推理，空间推理以及在我们的Longvideo-Reason-Reason-Reason-eval基准上的情节推理的匹配。值得注意的是，我们的MR-SP系统在长期视频RL培训中达到了2.1倍的速度。 Longvila-R1作为输入视频帧量表的数量表现出一致的性能增长。 Longvila-R1标志着在VLM中迈出了长期视频推理的坚定一步。此外，我们发布了公共可用性的培训系统，该系统支持各种模式（视频，文本和音频），各种模型（Vila和Qwen系列）的RL培训，甚至图像和视频生成模型。在单个A100节点（8 GPU）上，它支持长达一个小时的视频（例如3,600帧 /约256K令牌）的RL培训。

Title: Martian World Models: Controllable Video Synthesis with Physically Accurate 3D Reconstructions

• Authors: Longfei Li, Zhiwen Fan, Wenyan Cong, Xinhang Liu, Yuyang Yin, Matt Foutter, Panwang Pan, Chenyu You, Yue Wang, Zhangyang Wang, Yao Zhao, Marco Pavone, Yunchao Wei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2507.07978
• Pdf URL: https://arxiv.org/pdf/2507.07978
• Copy Paste: [[2507.07978]] Martian World Models: Controllable Video Synthesis with Physically Accurate 3D Reconstructions(https://arxiv.org/abs/2507.07978)
• Keywords: generation
• Abstract: Synthesizing realistic Martian landscape videos is crucial for mission rehearsal and robotic simulation. However, this task poses unique challenges due to the scarcity of high-quality Martian data and the significant domain gap between Martian and terrestrial imagery. To address these challenges, we propose a holistic solution composed of two key components: 1) A data curation pipeline Multimodal Mars Synthesis (M3arsSynth), which reconstructs 3D Martian environments from real stereo navigation images, sourced from NASA's Planetary Data System (PDS), and renders high-fidelity multiview 3D video sequences. 2) A Martian terrain video generator, MarsGen, which synthesizes novel videos visually realistic and geometrically consistent with the 3D structure encoded in the data. Our M3arsSynth engine spans a wide range of Martian terrains and acquisition dates, enabling the generation of physically accurate 3D surface models at metric-scale resolution. MarsGen, fine-tuned on M3arsSynth data, synthesizes videos conditioned on an initial image frame and, optionally, camera trajectories or textual prompts, allowing for video generation in novel environments. Experimental results show that our approach outperforms video synthesis models trained on terrestrial datasets, achieving superior visual fidelity and 3D structural consistency.
• 摘要：综合现实的火星景观视频对于任务排练和机器人模拟至关重要。但是，由于高质量的火星数据和火星和陆地图像之间的重要领域差距，这项任务构成了独特的挑战。为了应对这些挑战，我们提出了一个由两个关键组成部分组成的整体解决方案：1）数据策划管道多模式MARS合成（M3ARSSYNTH），该策略从NASA的Planetary Data System（PDS）以及High-FideLity MultivieLity Multiviveive 3DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD DD-3D DD DD DD DD介绍。 2）火星地形视频生成器Marsgen，它综合了新颖的视频在视觉上逼真的，并且在几何上与数据中编码的3D结构一致。我们的M3ARSSYNTH发动机跨越了各种各样的火星地形和采集日期，从而在度量尺度分辨率下可以生成物理准确的3D表面模型。 Marsgen在M3ARSSYNTH数据上进行了微调，合成了以初始图像框架为条件的视频，并且可以选择地，摄像机轨迹或文本提示，允许在新颖的环境中进行视频生成。实验结果表明，我们的方法的表现优于在陆地数据集中训练的视频合成模型，实现了优越的视觉保真度和3D结构一致性。

Title: Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modeling

• Authors: Haoyu Wu, Diankun Wu, Tianyu He, Junliang Guo, Yang Ye, Yueqi Duan, Jiang Bian
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2507.07982
• Pdf URL: https://arxiv.org/pdf/2507.07982
• Copy Paste: [[2507.07982]] Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modeling(https://arxiv.org/abs/2507.07982)
• Keywords: generation
• Abstract: Videos inherently represent 2D projections of a dynamic 3D world. However, our analysis suggests that video diffusion models trained solely on raw video data often fail to capture meaningful geometric-aware structure in their learned representations. To bridge this gap between video diffusion models and the underlying 3D nature of the physical world, we propose Geometry Forcing, a simple yet effective method that encourages video diffusion models to internalize latent 3D representations. Our key insight is to guide the model's intermediate representations toward geometry-aware structure by aligning them with features from a pretrained geometric foundation model. To this end, we introduce two complementary alignment objectives: Angular Alignment, which enforces directional consistency via cosine similarity, and Scale Alignment, which preserves scale-related information by regressing unnormalized geometric features from normalized diffusion representation. We evaluate Geometry Forcing on both camera view-conditioned and action-conditioned video generation tasks. Experimental results demonstrate that our method substantially improves visual quality and 3D consistency over the baseline methods. Project page: this https URL.
• 摘要：视频固有地代表了动态3D世界的2D投影。但是，我们的分析表明，仅在原始视频数据上训练的视频扩散模型通常无法捕获其学习的表示形式中有意义的几何感知结构。为了弥合视频扩散模型与物理世界的基本3D性质之间的差距，我们提出了几何强迫，这是一种简单而有效的方法，可以鼓励视频扩散模型内部化潜在的3D表示。我们的关键见解是通过将模型的中间表示与几何形状 - 感知结构进行对齐，使它们与预审前的几何基础模型的特征对齐。为此，我们介绍了两个互补的对准目标：角对准，通过余弦相似性和比例对齐来实现定向一致性，从而通过从归一化扩散表示中回归未归一化的几何特征来保留与规模相关的信息。我们评估了相机视图和动作条件的视频生成任务上的几何强迫。实验结果表明，我们的方法基本上提高了基线方法的视觉质量和3D一致性。项目页面：此HTTPS URL。