2025-03-13

Title: Training Plug-n-Play Knowledge Modules with Deep Context Distillation

• Authors: Lucas Caccia, Alan Ansell, Edoardo Ponti, Ivan Vulić, Alessandro Sordoni
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.08727
• Pdf URL: https://arxiv.org/pdf/2503.08727
• Copy Paste: [[2503.08727]] Training Plug-n-Play Knowledge Modules with Deep Context Distillation(https://arxiv.org/abs/2503.08727)
• Keywords: generation
• Abstract: Dynamically integrating new or rapidly evolving information after (Large) Language Model pre-training remains challenging, particularly in low-data scenarios or when dealing with private and specialized documents. In-context learning and retrieval-augmented generation (RAG) face limitations, including their high inference costs and their inability to capture global document information. In this paper, we propose a way of modularizing knowledge by training document-level Knowledge Modules (KMs). KMs are lightweight components implemented as parameter-efficient LoRA modules, which are trained to store information about new documents and can be easily plugged into models on demand. We show that next-token prediction performs poorly as the training objective for KMs. We instead propose Deep Context Distillation: we learn KMs parameters such as to simulate hidden states and logits of a teacher that takes the document in context. Our method outperforms standard next-token prediction and pre-instruction training techniques, across two datasets. Finally, we highlight synergies between KMs and retrieval-augmented generation.
• 摘要：（大型）语言模型预训练之后，动态整合新的或快速发展的信息仍然具有挑战性，尤其是在低数据表情况下或处理私人和专业文档时。在上下文中的学习和检索增强的生成（RAG）面临限制，包括其高推理成本以及无法捕获全球文档信息的限制。在本文中，我们提出了一种通过培训文档级知识模块（KMS）进行模块化知识的方式。 KMS是实现的轻量级组件，该组件以参数有效的Lora模块实现，经过培训可以存储有关新文档的信息，并且可以轻松地按需插入型号。我们表明，作为KMS的培训目标，下一个预测的表现较差。相反，我们提出了深层的上下文蒸馏：我们学习KMS参数，例如模拟将文档在上下文中获取文档的隐藏状态和逻辑。我们的方法优于两个数据集的标准下一步预测和前指导培训技术。最后，我们重点介绍了KMS与检索型一代之间的协同作用。

Title: Representing 3D Shapes With 64 Latent Vectors for 3D Diffusion Models

• Authors: In Cho, Youngbeom Yoo, Subin Jeon, Seon Joo Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.08737
• Pdf URL: https://arxiv.org/pdf/2503.08737
• Copy Paste: [[2503.08737]] Representing 3D Shapes With 64 Latent Vectors for 3D Diffusion Models(https://arxiv.org/abs/2503.08737)
• Keywords: generation
• Abstract: Constructing a compressed latent space through a variational autoencoder (VAE) is the key for efficient 3D diffusion models. This paper introduces COD-VAE, a VAE that encodes 3D shapes into a COmpact set of 1D latent vectors without sacrificing quality. COD-VAE introduces a two-stage autoencoder scheme to improve compression and decoding efficiency. First, our encoder block progressively compresses point clouds into compact latent vectors via intermediate point patches. Second, our triplane-based decoder reconstructs dense triplanes from latent vectors instead of directly decoding neural fields, significantly reducing computational overhead of neural fields decoding. Finally, we propose uncertainty-guided token pruning, which allocates resources adaptively by skipping computations in simpler regions and improves the decoder efficiency. Experimental results demonstrate that COD-VAE achieves 16x compression compared to the baseline while maintaining quality. This enables 20.8x speedup in generation, highlighting that a large number of latent vectors is not a prerequisite for high-quality reconstruction and generation.
• 摘要：通过变量自动编码器（VAE）构建压缩潜在空间是有效的3D扩散模型的关键。本文介绍了COD-VAE，该VAE是一种编码3D形状的VAE，不牺牲质量，将3D形状编码为紧凑的1D潜在媒介。 COD-VAE引入了两阶段的自动编码器方案，以提高压缩和解码效率。首先，我们的编码器块通过中间点贴片逐渐压缩为紧凑的潜在向量。其次，我们基于三烷基的解码器重建了潜在载体的密集三分球，而不是直接解码神经场，从而大大降低了神经场解码的计算开销。最后，我们提出了不确定性引导的代币修剪，该修剪通过跳过更简单区域的计算并提高解码器效率来自适应地分配资源。实验结果表明，与基线相比，COD-VAE在保持质量的同时，达到了16倍的压缩。这使20.8倍的加速能力发表，强调了大量潜在向量不是高质量重建和生成的先决条件。

Title: Towards Efficient Parametric State Estimation in Circulating Fuel Reactors with Shallow Recurrent Decoder Networks

• Authors: Stefano Riva, Carolina Introini, J. Nathan Kutz, Antonio Cammi
• Subjects: cs.LG, cs.CE, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2503.08904
• Pdf URL: https://arxiv.org/pdf/2503.08904
• Copy Paste: [[2503.08904]] Towards Efficient Parametric State Estimation in Circulating Fuel Reactors with Shallow Recurrent Decoder Networks(https://arxiv.org/abs/2503.08904)
• Keywords: generation
• Abstract: The recent developments in data-driven methods have paved the way to new methodologies to provide accurate state reconstruction of engineering systems; nuclear reactors represent particularly challenging applications for this task due to the complexity of the strongly coupled physics involved and the extremely harsh and hostile environments, especially for new technologies such as Generation-IV reactors. Data-driven techniques can combine different sources of information, including computational proxy models and local noisy measurements on the system, to robustly estimate the state. This work leverages the novel Shallow Recurrent Decoder architecture to infer the entire state vector (including neutron fluxes, precursors concentrations, temperature, pressure and velocity) of a reactor from three out-of-core time-series neutron flux measurements alone. In particular, this work extends the standard architecture to treat parametric time-series data, ensuring the possibility of investigating different accidental scenarios and showing the capabilities of this approach to provide an accurate state estimation in various operating conditions. This paper considers as a test case the Molten Salt Fast Reactor (MSFR), a Generation-IV reactor concept, characterised by strong coupling between the neutronics and the thermal hydraulics due to the liquid nature of the fuel. The promising results of this work are further strengthened by the possibility of quantifying the uncertainty associated with the state estimation, due to the considerably low training cost. The accurate reconstruction of every characteristic field in real-time makes this approach suitable for monitoring and control purposes in the framework of a reactor digital twin.
• 摘要：数据驱动方法的最新发展已铺平了通往新方法的方式，以提供工程系统的准确状态重建。由于涉及强大耦合物理学以及极为严峻和敌对的环境，核反应堆对这项任务的应用特别具有挑战性，尤其是对于新技术（例如IV代反应堆）。数据驱动的技术可以结合不同信息来源，包括计算代理模型和系统上的局部噪声测量，以稳健地估计状态。这项工作利用了新型的复发解码器结构来推断反应堆从三个单独的核心时间序列中子磁通测量值中的反应堆的整个状态矢量（包括中子通量，前体浓度，温度，压力和速度）。特别是，这项工作扩展了标准体系结构来处理参数时间序列数据，确保可能研究不同的意外情况并显示这种方法在各种操作条件下提供准确的状态估计的能力。本文认为是测试案例，熔融盐快速反应堆（MSFR）是IV代反应堆概念，其特征是由于燃料的液体性质，中子学和热液压之间的强耦合。由于培训成本相当低，可以量化与国家估计相关的不确定性的可能性进一步加强了这项工作的有希望的结果。实时重建每个特征领域的准确重建使得这种方法适合在反应堆数字双胞胎框架中监视和控制目的。

Title: From Models To Experiments: Shallow Recurrent Decoder Networks on the DYNASTY Experimental Facility

• Authors: Carolina Introini, Stefano Riva, J. Nathan Kutz, Antonio Cammi
• Subjects: cs.LG, physics.comp-ph, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2503.08907
• Pdf URL: https://arxiv.org/pdf/2503.08907
• Copy Paste: [[2503.08907]] From Models To Experiments: Shallow Recurrent Decoder Networks on the DYNASTY Experimental Facility(https://arxiv.org/abs/2503.08907)
• Keywords: generation
• Abstract: The Shallow Recurrent Decoder networks are a novel paradigm recently introduced for state estimation, combining sparse observations with high-dimensional model data. This architecture features important advantages compared to standard data-driven methods including: the ability to use only three sensors (even randomly selected) for reconstructing the entire dynamics of a physical system; the ability to train on compressed data spanned by a reduced basis; the ability to measure a single field variable (easy to measure) and reconstruct coupled spatio-temporal fields that are not observable and minimal hyper-parameter tuning. This approach has been verified on different test cases within different fields including nuclear reactors, even though an application to a real experimental facility, adopting the employment of in-situ observed quantities, is missing. This work aims to fill this gap by applying the Shallow Recurrent Decoder architecture to the DYNASTY facility, built at Politecnico di Milano, which studies the natural circulation established by internally heated fluids for Generation IV applications, especially in the case of Circulating Fuel reactors. The RELAP5 code is used to generate the high-fidelity data, and temperature measurements extracted by the facility are used as input for the state estimation. The results of this work will provide a validation of the Shallow Recurrent Decoder architecture to engineering systems, showing the capabilities of this approach to provide and accurate state estimation.
• 摘要：浅层复发的解码器网络是最近引入的新型范式进行状态估计，将稀疏观测结果与高维模型数据相结合。与标准数据驱动的方法相比，该体系结构具有重要的优势，包括：仅使用三个传感器（甚至随机选择）重建物理系统的整个动力学；通过减少的压缩数据进行训练的能力；测量单场变量（易于测量）和重建耦合时空场的能力，这些时空场是无法观察到的，并且高参数调谐。尽管缺少对实际实验设施的应用，采用了使用现场观察到的数量，但在不同领域的不同测试案例中已经验证了这种方法。这项工作旨在通过将浅层复发解码器建筑应用于Politecnico di Milano建造的浅层解码器建筑，该建筑研究由内部加热的液体建立的自然循环，以用于IV代应用，尤其是在循环燃料反应堆的情况下。 Relap5代码用于生成高保真数据，而设施提取的温度测量值则用作状态估计的输入。这项工作的结果将对工程系统的浅反复解码器架构进行验证，以显示这种方法提供和准确的状态估计的能力。

Title: Multilevel Generative Samplers for Investigating Critical Phenomena

• Authors: Ankur Singha, Elia Cellini, Kim A. Nicoli, Karl Jansen, Stefan Kühn, Shinichi Nakajima
• Subjects: cs.LG, hep-lat, stat.ML
• Abstract URL: https://arxiv.org/abs/2503.08918
• Pdf URL: https://arxiv.org/pdf/2503.08918
• Copy Paste: [[2503.08918]] Multilevel Generative Samplers for Investigating Critical Phenomena(https://arxiv.org/abs/2503.08918)
• Keywords: generative
• Abstract: Investigating critical phenomena or phase transitions is of high interest in physics and chemistry, for which Monte Carlo (MC) simulations, a crucial tool for numerically analyzing macroscopic properties of given systems, are often hindered by an emerging divergence of correlation length -- known as scale invariance at criticality (SIC) in the renormalization group theory. SIC causes the system to behave the same at any length scale, from which many existing sampling methods suffer: long-range correlations cause critical slowing down in Markov chain Monte Carlo (MCMC), and require intractably large receptive fields for generative samplers. In this paper, we propose a Renormalization-informed Generative Critical Sampler (RiGCS) -- a novel sampler specialized for near-critical systems, where SIC is leveraged as an advantage rather than a nuisance. Specifically, RiGCS builds on MultiLevel Monte Carlo (MLMC) with Heat Bath (HB) algorithms, which perform ancestral sampling from low-resolution to high-resolution lattice configurations with site-wise-independent conditional HB sampling. Although MLMC-HB is highly efficient under exact SIC, it suffers from a low acceptance rate under slight SIC violation. Notably, SIC violation always occurs in finite-size systems, and may induce long-range and higher-order interactions in the renormalized distributions, which are not considered by independent HB samplers. RiGCS enhances MLMC-HB by replacing a part of the conditional HB sampler with generative models that capture those residual interactions and improve the sampling efficiency. Our experiments show that the effective sample size of RiGCS is a few orders of magnitude higher than state-of-the-art generative model baselines in sampling configurations for 128x128 two-dimensional Ising systems.
• 摘要：研究批判现象或相转换在物理和化学方面具有很高的兴趣，蒙特卡洛（MC）模拟是一种用于数值分析给定系统的宏观特性的关键工具，通常受到培养长度的相关长度的差异（SIC）在关键的范围（SIC）中的差异（SIC）。 SIC使系统在任何长度尺度上的表现相同，许多现有的采样方法受到了：远程相关性导致马尔可夫链蒙特卡洛（MCMC）的关键减速，并且需要对生成采样器的巨大接收场。在本文中，我们提出了一种重新归一化的生成临界抽样器（RIGCS），这是一种专门用于近临界系统的新型采样器，其中SIC被用作优势而不是滋扰。具体而言，RIGCS用热浴（HB）算法建立在多级蒙特卡洛（MLMC）上，该算法从低分辨率到高分辨率的晶格构型进行祖先采样，并具有与站点无关的条件HB采样。尽管MLMC-HB在精确的SIC下效率很高，但在轻微的SIC违规情况下，其接受率较低。值得注意的是，SIC违规始终发生在有限尺寸的系统中，并且可能在重新归一化的分布中引起长期和高阶相互作用，而独立的HB采样器未考虑。 RIGCS通过使用生成模型替换有条件的HB采样器的一部分来增强MLMC-HB，从而捕获这些残留相互作用并提高采样效率。我们的实验表明，在128x128二维ISING Systems的采样配置中，RIGC的有效样本大小比最先进的生成模型基准高几个数量级。

Title: Enhancing Large Language Models for Hardware Verification: A Novel SystemVerilog Assertion Dataset

• Authors: Anand Menon, Samit S Miftah, Shamik Kundu, Souvik Kundu, Amisha Srivastava, Arnab Raha, Gabriel Theodor Sonnenschein, Suvadeep Banerjee, Deepak Mathaikutty, Kanad Basu
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2503.08923
• Pdf URL: https://arxiv.org/pdf/2503.08923
• Copy Paste: [[2503.08923]] Enhancing Large Language Models for Hardware Verification: A Novel SystemVerilog Assertion Dataset(https://arxiv.org/abs/2503.08923)
• Keywords: generation
• Abstract: Hardware verification is crucial in modern SoC design, consuming around 70% of development time. SystemVerilog assertions ensure correct functionality. However, existing industrial practices rely on manual efforts for assertion generation, which becomes increasingly untenable as hardware systems become complex. Recent research shows that Large Language Models (LLMs) can automate this process. However, proprietary SOTA models like GPT-4o often generate inaccurate assertions and require expensive licenses, while smaller open-source LLMs need fine-tuning to manage HDL code complexities. To address these issues, we introduce **VERT**, an open-source dataset designed to enhance SystemVerilog assertion generation using LLMs. VERT enables researchers in academia and industry to fine-tune open-source models, outperforming larger proprietary ones in both accuracy and efficiency while ensuring data privacy through local fine-tuning and eliminating costly licenses. The dataset is curated by systematically augmenting variables from open-source HDL repositories to generate synthetic code snippets paired with corresponding assertions. Experimental results demonstrate that fine-tuned models like Deepseek Coder 6.7B and Llama 3.1 8B outperform GPT-4o, achieving up to 96.88% improvement over base models and 24.14% over GPT-4o on platforms including OpenTitan, CVA6, OpenPiton and Pulpissimo. VERT is available at this https URL.
• 摘要：硬件验证对于现代SOC设计至关重要，耗时约为开发时间的70％。 SystemVerilog断言可确保正确的功能。但是，现有的工业实践依赖于为生成的手动努力，随着硬件系统变得复杂，这变得越来越站不住脚。最近的研究表明，大型语言模型（LLM）可以自动化此过程。但是，诸如GPT-4O之类的专有SOTA模型通常会产生不准确的断言，并且需要昂贵的许可证，而较小的开源LLMS则需要微调来管理HDL代码复杂性。为了解决这些问题，我们介绍了** vert **，这是一个开源数据集，旨在增强使用LLMS的SystemVerilog断言。 Vert使学术界和行业的研究人员能够微调开源模型，在准确性和效率上都优于更大的专有模型，同时通过当地的微调和消除昂​​贵的许可来确保数据隐私。通过系统地增强从开源HDL存储库来策划数据集，以生成与相应断言配对的合成代码片段。实验结果表明，诸如DeepSeek Coder 6.7b和Llama 3.1 8B的微调模型均优于GPT-4O，比基本型号提高了96.88％的提高，而在包括Opentitan，CVA6，CVA6，OpenPiton和Pulpissimo等平台上的GPT-4O相比24.14％。 Vert可在此HTTPS URL上找到。

Title: Near-Optimal Sample Complexity for Iterated CVaR Reinforcement Learning with a Generative Model

• Authors: Zilong Deng, Simon Khan, Shaofeng Zou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2503.08934
• Pdf URL: https://arxiv.org/pdf/2503.08934
• Copy Paste: [[2503.08934]] Near-Optimal Sample Complexity for Iterated CVaR Reinforcement Learning with a Generative Model(https://arxiv.org/abs/2503.08934)
• Keywords: generative
• Abstract: In this work, we study the sample complexity problem of risk-sensitive Reinforcement Learning (RL) with a generative model, where we aim to maximize the Conditional Value at Risk (CVaR) with risk tolerance level $\tau$ at each step, named Iterated CVaR. %We consider the sample complexity of obtaining an $\epsilon$-optimal policy in an infinite horizon discounted MDP, given access to a generative model. % We first build a connection between Iterated CVaR RL with $(s, a)$-rectangular distributional robust RL with the specific uncertainty set for CVaR. We develop nearly matching upper and lower bounds on the sample complexity for this problem. Specifically, we first prove that a value iteration-based algorithm, ICVaR-VI, achieves an $\epsilon$-optimal policy with at most $\tilde{O}\left(\frac{SA}{(1-\gamma)^4\tau^2\epsilon^2}\right)$ samples, where $\gamma$ is the discount factor, and $S, A$ are the sizes of the state and action spaces. Furthermore, if $\tau \geq \gamma$, then the sample complexity can be further improved to $\tilde{O}\left( \frac{SA}{(1-\gamma)^3\epsilon^2} \right)$. We further show a minimax lower bound of ${\tilde{O}}\left(\frac{(1-\gamma \tau)SA}{(1-\gamma)^4\tau\epsilon^2}\right)$. For a constant risk level $0<\tau\leq 1$, our upper and lower bounds match with each other, demonstrating the tightness and optimality of our analyses. We also investigate a limiting case with a small risk level $\tau$, called Worst-Path RL, where the objective is to maximize the minimum possible cumulative reward. We develop matching upper and lower bounds of $\tilde{O}\left(\frac{SA}{p_{\min}}\right)$, where $p_{\min}$ denotes the minimum non-zero reaching probability of the transition kernel.
• 摘要：在这项工作中，我们使用生成模型研究了对风险敏感的增强学习（RL）的样本复杂性问题，我们的目标是在每个步骤中以风险容忍度水平$ \ tau $在风险中最大化有条件的价值（CVAR），名为Iterated Cvar。 ％我们考虑在无限的地平线折扣MDP中获得$ \ epsilon $最佳政策的样本复杂性，允许访问生成模型。 ％我们首先与$（s，a）$ - 矩形分布鲁棒RL之间的迭代CVAR RL之间建立连接，并使用CVAR的特定不确定性集。对于此问题，我们在样品复杂性上开发了几乎匹配的上限和下限。具体而言，我们首先证明了一种基于价值的算法ICVAR-VI，达到了$ \ epsilon $ - 最佳政策，最多最多可以使用$ \ tilde {o} \ left（\ frac {sa}} {（sa} {（1- \ gamma）^4 \ tau^4 \ tau^4 \ tau^2 \ eps $ \ eps $ \ eps $ plase $ wer折扣因子和$ s，$是国家和行动空间的尺寸。此外，如果$ \ tau \ geq \ gamma $，则可以将样品复杂性进一步提高到$ \ tilde {o} \ left（\ frac {sa}} {（1- \ gamma）^3 \ 3 \ epsilon^2}} \ right）$。我们进一步显示了$ {\ tilde {o}} \ left（\ frac {（1- \ gamma \ tau）sa} {（1- \ gamma）^4 \ tau \ epsilon^2} \ right）$ {\ tilde {o}} \ left的最小下限。对于恒定的风险级别$ 0 <\ tau \ leq 1 $，我们的上限和下限彼此匹配，证明了我们的分析的紧密度和最佳性。我们还调查了一个限制案例，其中一个较小的风险水平$ \ tau $，称为最差路径RL，其目的是最大程度地提高可能的累积奖励。我们开发了$ \ tilde {o} \ left（\ frac {sa} {p _ {\ min}} \ right）$的匹配上限和下限，其中$ p _ {\ min} $表示最小非零触发过渡核的非零可能性。

Title: I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data?

• Authors: Yuhang Liu, Dong Gong, Erdun Gao, Zhen Zhang, Biwei Huang, Mingming Gong, Anton van den Hengel, Javen Qinfeng Shi
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2503.08980
• Pdf URL: https://arxiv.org/pdf/2503.08980
• Copy Paste: [[2503.08980]] I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data?(https://arxiv.org/abs/2503.08980)
• Keywords: generative
• Abstract: The remarkable achievements of large language models (LLMs) have led many to conclude that they exhibit a form of intelligence. This is as opposed to explanations of their capabilities based on their ability to perform relatively simple manipulations of vast volumes of data. To illuminate the distinction between these explanations, we introduce a novel generative model that generates tokens on the basis of human interpretable concepts represented as latent discrete variables. Under mild conditions, even when the mapping from the latent space to the observed space is non-invertible, we establish an identifiability result: the representations learned by LLMs through next-token prediction can be approximately modeled as the logarithm of the posterior probabilities of these latent discrete concepts, up to an invertible linear transformation. This theoretical finding not only provides evidence that LLMs capture underlying generative factors, but also strongly reinforces the linear representation hypothesis, which posits that LLMs learn linear representations of human-interpretable concepts. Empirically, we validate our theoretical results through evaluations on both simulation data and the Pythia, Llama, and DeepSeek model families.
• 摘要：大型语言模型（LLM）的显着成就使许多人得出结论，他们表现出一种智能形式。这是基于他们对大量数据进行相对简单操纵的能力的解释而不是对其能力的解释。为了阐明这些解释之间的区别，我们介绍了一种新颖的生成模型，该模型基于人类可解释的概念产生代币，该概念表示为潜在的离散变量。在温和的条件下，即使从潜在空间到观察到的空间的映射是不可简化的，我们也建立了一个可识别性结果：LLM通过下一步预测所学的表示形式近似建模为这些潜在离散概念的后验概率的对数，直至可逆的线性转换。这一理论发现不仅提供了LLM捕获潜在的生成因素的证据，而且还强烈加强了线性表示假设，这表明LLMS学习了人类可介入的概念的线性表示。从经验上讲，我们通过对仿真数据和毕曲霉，骆驼和DeepSeek模型家族的评估来验证我们的理论结果。

Title: Dual-Domain Homogeneous Fusion with Cross-Modal Mamba and Progressive Decoder for 3D Object Detection

• Authors: Xuzhong Hu, Zaipeng Duan, Pei An, Jun zhang, Jie Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.08992
• Pdf URL: https://arxiv.org/pdf/2503.08992
• Copy Paste: [[2503.08992]] Dual-Domain Homogeneous Fusion with Cross-Modal Mamba and Progressive Decoder for 3D Object Detection(https://arxiv.org/abs/2503.08992)
• Keywords: generation
• Abstract: Fusing LiDAR point cloud features and image features in a homogeneous BEV space has been widely adopted for 3D object detection in autonomous driving. However, such methods are limited by the excessive compression of multi-modal features. While some works explore feature fusion in dense voxel spaces, they suffer from high computational costs and inefficiencies in query generation. To address these limitations, we propose a Dual-Domain Homogeneous Fusion network (DDHFusion), which leverages the complementary advantages of both BEV and voxel domains while mitigating their respective drawbacks. Specifically, we first transform image features into BEV and sparse voxel spaces using LSS and our proposed semantic-aware feature sampling module which can significantly reduces computational overhead by filtering unimportant voxels. For feature encoding, we design two networks for BEV and voxel feature fusion, incorporating novel cross-modal voxel and BEV Mamba blocks to resolve feature misalignment and enable efficient yet comprehensive scene perception. The output voxel features are injected into the BEV space to compensate for the loss of 3D details caused by height compression. For feature decoding, a progressive query generation module is implemented in the BEV domain to alleviate false negatives during query selection caused by feature compression and small object sizes. Finally, a progressive decoder can sequentially aggregate not only context-rich BEV features but also geometry-aware voxel features, ensuring more precise confidence prediction and bounding box regression. On the NuScenes dataset, DDHfusion achieves state-of-the-art performance, and further experiments demonstrate its superiority over other homogeneous fusion methods.
• 摘要：在均匀的BEV空间中，在自动驾驶中的3D对象检测中，融合LiDAR点云特征和图像特征已被广泛采用。但是，这种方法受多模式特征的过度压缩受到限制。虽然某些作品探索了密集的体素空间中的特征融合，但它们的计算成本很高，并且在查询产生中效率低下。为了解决这些限制，我们提出了一个双域均匀融合网络（DDHFusion），该网络利用BEV和Voxel域的互补优势，同时减轻了它们各自的缺点。具体而言，我们首先使用LSS和我们提出的语义感知功能采样模块将图像特征转换为BEV和稀疏体素空间，该模块可以通过过滤不重要的体素来大大减少计算开销。对于特征编码，我们为BEV和Voxel特征融合设计了两个网络，并结合了新型的跨模式体素和Bev Mamba块，以解决特征错位，并实现有效但全面的场景感知。输出体素特征被注入BEV空间，以补偿由高度压缩引起的3D细节的损失。对于功能解码，在BEV域中实现了一个渐进的查询生成模块，以减轻特征压缩和小物体大小引起的查询选择期间的错误负面因素。最后，渐进解码器不仅可以依次汇总上下文富含BEV的特征，还可以几何感知体素特征，从而确保更精确的置信度预测和边界框回归。在Nuscenes数据集上，DDHFusion实现了最新的性能，进一步的实验证明了其优于其他同质融合方法。

Title: Prompt to Restore, Restore to Prompt: Cyclic Prompting for Universal Adverse Weather Removal

• Authors: Rongxin Liao, Feng Li, Yanyan Wei, Zenglin Shi, Le Zhang, Huihui Bai, Meng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09013
• Pdf URL: https://arxiv.org/pdf/2503.09013
• Copy Paste: [[2503.09013]] Prompt to Restore, Restore to Prompt: Cyclic Prompting for Universal Adverse Weather Removal(https://arxiv.org/abs/2503.09013)
• Keywords: restoration
• Abstract: Universal adverse weather removal (UAWR) seeks to address various weather degradations within a unified framework. Recent methods are inspired by prompt learning using pre-trained vision-language models (e.g., CLIP), leveraging degradation-aware prompts to facilitate weather-free image restoration, yielding significant improvements. In this work, we propose CyclicPrompt, an innovative cyclic prompt approach designed to enhance the effectiveness, adaptability, and generalizability of UAWR. CyclicPrompt Comprises two key components: 1) a composite context prompt that integrates weather-related information and context-aware representations into the network to guide restoration. This prompt differs from previous methods by marrying learnable input-conditional vectors with weather-specific knowledge, thereby improving adaptability across various degradations. 2) The erase-and-paste mechanism, after the initial guided restoration, substitutes weather-specific knowledge with constrained restoration priors, inducing high-quality weather-free concepts into the composite prompt to further fine-tune the restoration process. Therefore, we can form a cyclic "Prompt-Restore-Prompt" pipeline that adeptly harnesses weather-specific knowledge, textual contexts, and reliable textures. Extensive experiments on synthetic and real-world datasets validate the superior performance of CyclicPrompt. The code is available at: this https URL.
• 摘要：普遍不利天气去除（UAWR）试图解决统一框架内的各种天气退化。最近的方法是由使用预训练的视觉语言模型（例如剪辑）迅速学习的灵感来源，利用降解感知的提示以促进无天气的图像恢复，从而产生显着改善。在这项工作中，我们提出了循环规定，这是一种创新的环状及时方法，旨在提高UAWR的有效性，适应性和概括性。环保措施包括两个关键组成部分：1）一个复合上下文提示，将与天气相关的信息和上下文感知的表示形式集成到网络中以指导恢复。该提示与以前的方法不同，通过将可学习的输入条件向量与特定于天气的知识结合在一起，从而改善了各种降解的适应性。 2）擦除和帕斯特机制在初始引导的修复后，用约束的恢复先验代替了特定于天气的知识，将高质量的无天气概念诱导到复合材料中，以进一步调整恢复过程。因此，我们可以形成一个循环“及时的填充物”管道，该管道擅长利用特定于天气的知识，文本上下文和可靠的纹理。关于合成和现实世界数据集的广泛实验验证了环状策略的出色性能。该代码可用：此HTTPS URL。

Title: Measure Twice, Cut Once: Grasping Video Structures and Event Semantics with LLMs for Video Temporal Localization

• Authors: Zongshang Pang, Mayu Otani, Yuta Nakashima
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09027
• Pdf URL: https://arxiv.org/pdf/2503.09027
• Copy Paste: [[2503.09027]] Measure Twice, Cut Once: Grasping Video Structures and Event Semantics with LLMs for Video Temporal Localization(https://arxiv.org/abs/2503.09027)
• Keywords: generation
• Abstract: Localizing user-queried events through natural language is crucial for video understanding models. Recent methods predominantly adapt Video LLMs to generate event boundary timestamps to handle temporal localization tasks, which struggle to leverage LLMs' powerful semantic understanding. In this work, we introduce MeCo, a novel timestamp-free framework that enables video LLMs to fully harness their intrinsic semantic capabilities for temporal localization tasks. Rather than outputting boundary timestamps, MeCo partitions videos into holistic event and transition segments based on the proposed structural token generation and grounding pipeline, derived from video LLMs' temporal structure understanding capability. We further propose a query-focused captioning task that compels the LLM to extract fine-grained, event-specific details, bridging the gap between localization and higher-level semantics and enhancing localization performance. Extensive experiments on diverse temporal localization tasks show that MeCo consistently outperforms boundary-centric methods, underscoring the benefits of a semantic-driven approach for temporal localization with video LLMs.
• 摘要：通过自然语言本地化用户引起的事件对于视频理解模型至关重要。最近的方法主要适应视频LLM，以生成事件边界时间戳来处理时间定位任务，这些任务很难利用LLMS强大的语义理解。在这项工作中，我们介绍了MECO，这是一个新颖的无时间戳框架，使视频LLM能够完全利用其内在的语义功能来实现时间定位任务。 MeCo没有输出边界时间戳，而是根据提出的结构令牌生成和接地管道将视频分配为整体事件和过渡段，这些视频源自Video LLMS的时间结构理解能力。我们进一步提出了一项以查询为重点的字幕任务，该任务迫使LLM提取精细的事件特定细节，从而弥合本地化和高级语义之间的差距并增强本地化性能。关于各种时间定位任务的广泛实验表明，MECO始终超过边界的方法，强调了使用视频LLMS进行时间驱动的时间定位的语义驱动方法的好处。

Title: Theoretical Guarantees for High Order Trajectory Refinement in Generative Flows

• Authors: Chengyue Gong, Xiaoyu Li, Yingyu Liang, Jiangxuan Long, Zhenmei Shi, Zhao Song, Yu Tian
• Subjects: cs.LG, cs.AI, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2503.09069
• Pdf URL: https://arxiv.org/pdf/2503.09069
• Copy Paste: [[2503.09069]] Theoretical Guarantees for High Order Trajectory Refinement in Generative Flows(https://arxiv.org/abs/2503.09069)
• Keywords: generative
• Abstract: Flow matching has emerged as a powerful framework for generative modeling, offering computational advantages over diffusion models by leveraging deterministic Ordinary Differential Equations (ODEs) instead of stochastic dynamics. While prior work established the worst case optimality of standard flow matching under Wasserstein distances, the theoretical guarantees for higher-order flow matching - which incorporates acceleration terms to refine sample trajectories - remain unexplored. In this paper, we bridge this gap by proving that higher-order flow matching preserves worst case optimality as a distribution estimator. We derive upper bounds on the estimation error for second-order flow matching, demonstrating that the convergence rates depend polynomially on the smoothness of the target distribution (quantified via Besov spaces) and key parameters of the ODE dynamics. Our analysis employs neural network approximations with carefully controlled depth, width, and sparsity to bound acceleration errors across both small and large time intervals, ultimately unifying these results into a general worst case optimal bound for all time steps.
• 摘要：流量匹配已成为生成建模的强大框架，通过利用确定性的普通微分方程（ODE）而不是随机动力学，可以通过利用确定性的普通微分方程（ODE）提供计算优势。虽然先前的工作确立了在瓦斯斯特因距离下标准流量匹配的最坏情况的最佳性，但高阶流量匹配的理论保证（将加速度术语结合到完善样品轨迹 - 仍未探索。在本文中，我们通过证明高阶流量匹配可以保留最坏情况最佳性作为分配估计器来弥合这一差距。我们在二阶流量匹配的估计误差上得出了上限，这表明收敛速率在多项式取决于目标分布的平滑度（通过BESOV空间进行量化）和ODE动力学的关键参数。我们的分析采用仔细控制深度，宽度和稀疏性的神经网络近似，以在小时和大时间间隔内绑定加速度误差，最终将这些结果统一为一般的最坏情况，以实现所有时间步骤的最佳结合。

Title: Everything Can Be Described in Words: A Simple Unified Multi-Modal Framework with Semantic and Temporal Alignment

• Authors: Xiaowei Bi, Zheyuan Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.09081
• Pdf URL: https://arxiv.org/pdf/2503.09081
• Copy Paste: [[2503.09081]] Everything Can Be Described in Words: A Simple Unified Multi-Modal Framework with Semantic and Temporal Alignment(https://arxiv.org/abs/2503.09081)
• Keywords: generation
• Abstract: Long Video Question Answering (LVQA) is challenging due to the need for temporal reasoning and large-scale multimodal data processing. Existing methods struggle with retrieving cross-modal information from long videos, especially when relevant details are sparsely distributed. We introduce UMaT (Unified Multi-modal as Text), a retrieval-augmented generation (RAG) framework that efficiently processes extremely long videos while maintaining cross-modal coherence. UMaT converts visual and auditory data into a unified textual representation, ensuring semantic and temporal alignment. Short video clips are analyzed using a vision-language model, while automatic speech recognition (ASR) transcribes dialogue. These text-based representations are structured into temporally aligned segments, with adaptive filtering to remove redundancy and retain salient details. The processed data is embedded into a vector database, enabling precise retrieval of dispersed yet relevant content. Experiments on a benchmark LVQA dataset show that UMaT outperforms existing methods in multimodal integration, long-form video understanding, and sparse information retrieval. Its scalability and interpretability allow it to process videos over an hour long while maintaining semantic and temporal coherence. These findings underscore the importance of structured retrieval and multimodal synchronization for advancing LVQA and long-form AI systems.
• 摘要：由于需要时间推理和大规模的多模式数据处理，长期视频问答（LVQA）是具有挑战性的。现有的方法努力从长视频中检索跨模式信息，尤其是当相关细节稀少时。我们介绍了UMAT（统一的多模式为文本），这是一个检索型发电（RAG）框架，可有效地处理非常长的视频，同时保持跨模式连贯性。 UMAT将视觉和听觉数据转换为统一的文本表示形式，以确保语义和时间对齐。使用视觉语言模型对简短的视频剪辑进行分析，而自动语音识别（ASR）则转录对话。这些基于文本的表示形式被构造成时间对齐的段，并具有自适应过滤以消除冗余并保留显着细节。处理后的数据嵌入到矢量数据库中，从而精确地检索了分散但相关的内容。基准LVQA数据集的实验表明，UMAT在多模式集成，长期视频理解和稀疏信息检索中的表现优于现有方法。它的可伸缩性和可解释性使其可以在一个小时的时间内处理视频，同时保持语义和时间连贯性。这些发现强调了结构化检索和多模式同步对推进LVQA和长形AI系统的重要性。

Title: Sometimes Painful but Certainly Promising: Feasibility and Trade-offs of Language Model Inference at the Edge

• Authors: Maximilian Abstreiter, Sasu Tarkoma, Roberto Morabito
• Subjects: cs.LG, cs.AI, cs.DC, cs.PF
• Abstract URL: https://arxiv.org/abs/2503.09114
• Pdf URL: https://arxiv.org/pdf/2503.09114
• Copy Paste: [[2503.09114]] Sometimes Painful but Certainly Promising: Feasibility and Trade-offs of Language Model Inference at the Edge(https://arxiv.org/abs/2503.09114)
• Keywords: generation, generative
• Abstract: The rapid rise of Language Models (LMs) has expanded the capabilities of natural language processing, powering applications from text generation to complex decision-making. While state-of-the-art LMs often boast hundreds of billions of parameters and are primarily deployed in data centers, recent trends show a growing focus on compact models-typically under 10 billion parameters-enabled by techniques such as quantization and other model compression techniques. This shift paves the way for LMs on edge devices, offering potential benefits such as enhanced privacy, reduced latency, and improved data sovereignty. However, the inherent complexity of even these smaller models, combined with the limited computing resources of edge hardware, raises critical questions about the practical trade-offs in executing LM inference outside the cloud. To address these challenges, we present a comprehensive evaluation of generative LM inference on representative CPU-based and GPU-accelerated edge devices. Our study measures key performance indicators-including memory usage, inference speed, and energy consumption-across various device configurations. Additionally, we examine throughput-energy trade-offs, cost considerations, and usability, alongside an assessment of qualitative model performance. While quantization helps mitigate memory overhead, it does not fully eliminate resource bottlenecks, especially for larger models. Our findings quantify the memory and energy constraints that must be considered for practical real-world deployments, offering concrete insights into the trade-offs between model size, inference performance, and efficiency. The exploration of LMs at the edge is still in its early stages. We hope this study provides a foundation for future research, guiding the refinement of models, the enhancement of inference efficiency, and the advancement of edge-centric AI systems.
• 摘要：语言模型（LMS）的快速崛起扩大了自然语言处理的功能，从文本生成到复杂的决策为应用程序提供了动力。虽然最先进的LMS通常拥有数百万个参数，并且主要部署在数据中心中，但最近的趋势表明，人们对紧凑型模型的关注越来越多，在100亿以下参数（诸如量化和其他模型压缩技术）中所启用的100亿次参数。这种转变为在边缘设备上的LMS铺平了道路，提供了潜在的好处，例如增强的隐私，延迟减少和改进的数据主权。但是，即使是这些较小模型的固有复杂性，再加上边缘硬件的有限计算资源，对在云外执行LM推理时的实际权衡提出了关键问题。为了应对这些挑战，我们对基于CPU和GPU加速的边缘设备进行了对生成LM推断的全面评估。我们的研究衡量了关键性能指标，包括记忆使用，推理速度和能源消耗速度各种设备配置。此外，我们还研究了吞吐量的权衡，成本注意事项和可用性，以及对定性模型性能的评估。虽然量化有助于减轻内存开销，但并不能完全消除资源瓶颈，尤其是对于大型型号。我们的发现量化了实际现实部署必须考虑的记忆和能量限制，从而为模型大小，推理性能和效率之间的权衡提供了具体的见解。 LMS在边缘的探索仍处于早期阶段。我们希望这项研究为未来的研究提供了基础，指导模型的完善，推理效率的提高以及以边缘为中心的AI系统的发展。

Title: Training Data Provenance Verification: Did Your Model Use Synthetic Data from My Generative Model for Training?

• Authors: Yuechen Xie, Jie Song, Huiqiong Wang, Mingli Song
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09122
• Pdf URL: https://arxiv.org/pdf/2503.09122
• Copy Paste: [[2503.09122]] Training Data Provenance Verification: Did Your Model Use Synthetic Data from My Generative Model for Training?(https://arxiv.org/abs/2503.09122)
• Keywords: generative
• Abstract: High-quality open-source text-to-image models have lowered the threshold for obtaining photorealistic images significantly, but also face potential risks of misuse. Specifically, suspects may use synthetic data generated by these generative models to train models for specific tasks without permission, when lacking real data resources especially. Protecting these generative models is crucial for the well-being of their owners. In this work, we propose the first method to this important yet unresolved issue, called Training data Provenance Verification (TrainProVe). The rationale behind TrainProVe is grounded in the principle of generalization error bound, which suggests that, for two models with the same task, if the distance between their training data distributions is smaller, their generalization ability will be closer. We validate the efficacy of TrainProVe across four text-to-image models (Stable Diffusion v1.4, latent consistency model, PixArt-$\alpha$, and Stable Cascade). The results show that TrainProVe achieves a verification accuracy of over 99\% in determining the provenance of suspicious model training data, surpassing all previous methods. Code is available at this https URL.
• 摘要：高质量的开源文本对图像模型已降低了明显获得影像图像的阈值，但也面临滥用的潜在风险。具体而言，嫌疑人可能会使用这些生成模型生成的合成数据在未经许可的情况下训练模型，尤其是在不允许的情况下，尤其是缺乏真实的数据资源。保护这些生成模型对于其所有者的福祉至关重要。在这项工作中，我们提出了这一重要但尚未解决的问题的第一种方法，称为培训数据出处验证（TrainProve）。 Trainprove背后的基本原理是基于概括误差的原理，这表明，对于具有相同任务的两个模型，如果他们的训练数据分布之间的距离较小，则其概括能力将更加接近。我们验证了Trainprove在四个文本到图像模型中的功效（稳定扩散V1.4，潜在一致性模型，Pixart-$ \ alpha $和稳定的Cascade）。结果表明，Trainprove在确定可疑模型训练数据的出处时达到了超过99 \％的验证精度，超过了所有以前的方法。代码可在此HTTPS URL上找到。

Title: AdvAD: Exploring Non-Parametric Diffusion for Imperceptible Adversarial Attacks

• Authors: Jin Li, Ziqiang He, Anwei Luo, Jian-Fang Hu, Z. Jane Wang, Xiangui Kang
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09124
• Pdf URL: https://arxiv.org/pdf/2503.09124
• Copy Paste: [[2503.09124]] AdvAD: Exploring Non-Parametric Diffusion for Imperceptible Adversarial Attacks(https://arxiv.org/abs/2503.09124)
• Keywords: generation, generative
• Abstract: Imperceptible adversarial attacks aim to fool DNNs by adding imperceptible perturbation to the input data. Previous methods typically improve the imperceptibility of attacks by integrating common attack paradigms with specifically designed perception-based losses or the capabilities of generative models. In this paper, we propose Adversarial Attacks in Diffusion (AdvAD), a novel modeling framework distinct from existing attack paradigms. AdvAD innovatively conceptualizes attacking as a non-parametric diffusion process by theoretically exploring basic modeling approach rather than using the denoising or generation abilities of regular diffusion models requiring neural networks. At each step, much subtler yet effective adversarial guidance is crafted using only the attacked model without any additional network, which gradually leads the end of diffusion process from the original image to a desired imperceptible adversarial example. Grounded in a solid theoretical foundation of the proposed non-parametric diffusion process, AdvAD achieves high attack efficacy and imperceptibility with intrinsically lower overall perturbation strength. Additionally, an enhanced version AdvAD-X is proposed to evaluate the extreme of our novel framework under an ideal scenario. Extensive experiments demonstrate the effectiveness of the proposed AdvAD and AdvAD-X. Compared with state-of-the-art imperceptible attacks, AdvAD achieves an average of 99.9$\%$ (+17.3$\%$) ASR with 1.34 (-0.97) $l_2$ distance, 49.74 (+4.76) PSNR and 0.9971 (+0.0043) SSIM against four prevalent DNNs with three different architectures on the ImageNet-compatible dataset. Code is available at this https URL.
• 摘要：不可察觉的对抗性攻击旨在通过向输入数据添加不可察觉的扰动来欺骗DNN。以前的方法通常通过将共同的攻击范式与专门设计的基于感知的损失或生成模型的功能整合在一起，从而提高了攻击的不可思议。在本文中，我们提出了扩散中的对抗攻击（Advad），这是一个与现有攻击范式不同的新型建模框架。通过从理论上探索基本的建模方法，而不是使用需要神经网络的常规扩散模型的转换或发电能力，Advad创新地概念化了攻击作为非参数扩散过程。在每个步骤中，仅使用攻击模型而没有任何其他网络来制定许多微妙而有效的对抗指导，这逐渐将从原始图像的扩散过程结束到了所需的不可感知的对抗性示例。 Advad以拟议中的非参数扩散过程的扎实理论基础为基础，实现了高发作的功效和不可信性，而本质上较低的总体扰动强度。此外，提出了增强版本的Advad-X，以评估我们在理想情况下的新型框架的极端。广泛的实验证明了拟议的Advad和Advad-X的有效性。 Compared with state-of-the-art imperceptible attacks, AdvAD achieves an average of 99.9$\%$ (+17.3$\%$) ASR with 1.34 (-0.97) $l_2$ distance, 49.74 (+4.76) PSNR and 0.9971 (+0.0043) SSIM against four prevalent DNNs with three different architectures on the ImageNet-compatible数据集。代码可在此HTTPS URL上找到。

Title: MP-HSIR: A Multi-Prompt Framework for Universal Hyperspectral Image Restoration

• Authors: Zhehui Wu, Yong Chen, Naoto Yokoya, Wei He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09131
• Pdf URL: https://arxiv.org/pdf/2503.09131
• Copy Paste: [[2503.09131]] MP-HSIR: A Multi-Prompt Framework for Universal Hyperspectral Image Restoration(https://arxiv.org/abs/2503.09131)
• Keywords: restoration
• Abstract: Hyperspectral images (HSIs) often suffer from diverse and unknown degradations during imaging, leading to severe spectral and spatial distortions. Existing HSI restoration methods typically rely on specific degradation assumptions, limiting their effectiveness in complex scenarios. In this paper, we propose MP-HSIR, a novel multi-prompt framework that effectively integrates spectral, textual, and visual prompts to achieve universal HSI restoration across diverse degradation types and intensities. Specifically, we develop a prompt-guided spatial-spectral transformer, which incorporates spatial self-attention and a prompt-guided dual-branch spectral self-attention. Since degradations affect spectral features differently, we introduce spectral prompts in the local spectral branch to provide universal low-rank spectral patterns as prior knowledge for enhancing spectral reconstruction. Furthermore, the text-visual synergistic prompt fuses high-level semantic representations with fine-grained visual features to encode degradation information, thereby guiding the restoration process. Extensive experiments on 9 HSI restoration tasks, including all-in-one scenarios, generalization tests, and real-world cases, demonstrate that MP-HSIR not only consistently outperforms existing all-in-one methods but also surpasses state-of-the-art task-specific approaches across multiple tasks. The code and models will be released at this https URL.
• 摘要：高光谱图像（HSIS）在成像过程中通常会遭受多种和未知的降解，导致严重的光谱和空间扭曲。现有的HSI恢复方法通常依赖于特定的降解假设，从而在复杂的情况下限制了它们的有效性。在本文中，我们提出了MP-HSIR，这是一种新型的多项目框架，可有效整合光谱，文本和视觉提示，以实现跨不同降解类型和强度的通用HSI恢复。具体而言，我们开发了一个迅速引入的空间传热变压器，该变压器结合了空间自我注意力和迅速引入的双支分支光谱自我注意。由于降解对光谱特征的影响有所不同，因此我们在局部光谱分支中引入光谱提示，以提供通用的低级光谱模式，作为增强光谱重建的先验知识。此外，文本视觉协同提示将高级语义表示与细粒度的视觉特征融合在一起，以编码降解信息，从而指导修复过程。对9个HSI恢复任务的广泛实验，包括多合一场景，泛化测试和现实世界中的情况，表明MP-HSIR不仅始终超过现有的多合一方法，而且超过了跨多个任务的最新任务特定方法。代码和模型将在此HTTPS URL上发布。

Title: Generative Frame Sampler for Long Video Understanding

• Authors: Linli Yao, Haoning Wu, Kun Ouyang, Yuanxing Zhang, Caiming Xiong, Bei Chen, Xu Sun, Junnan Li
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2503.09146
• Pdf URL: https://arxiv.org/pdf/2503.09146
• Copy Paste: [[2503.09146]] Generative Frame Sampler for Long Video Understanding(https://arxiv.org/abs/2503.09146)
• Keywords: generative
• Abstract: Despite recent advances in Video Large Language Models (VideoLLMs), effectively understanding long-form videos remains a significant challenge. Perceiving lengthy videos containing thousands of frames poses substantial computational burden. To mitigate this issue, this paper introduces Generative Frame Sampler (GenS), a plug-and-play module integrated with VideoLLMs to facilitate efficient lengthy video perception. Built upon a lightweight VideoLLM, GenS leverages its inherent vision-language capabilities to identify question-relevant frames. To facilitate effective retrieval, we construct GenS-Video-150K, a large-scale video instruction dataset with dense frame relevance annotations. Extensive experiments demonstrate that GenS consistently boosts the performance of various VideoLLMs, including open-source models (Qwen2-VL-7B, Aria-25B, VILA-40B, LLaVA-Video-7B/72B) and proprietary assistants (GPT-4o, Gemini). When equipped with GenS, open-source VideoLLMs achieve impressive state-of-the-art results on long-form video benchmarks: LLaVA-Video-72B reaches 66.8 (+4.3) on LongVideoBench and 77.0 (+2.7) on MLVU, while Aria obtains 39.2 on HourVideo surpassing the Gemini-1.5-pro by 1.9 points. We will release all datasets and models at this https URL.
• 摘要：尽管视频大型语言模型（Videolms）最近取得了进步，但有效理解长期视频仍然是一个重大挑战。感知到包含数千帧的冗长视频会造成重大的计算负担。为了减轻此问题，本文引入了生成框架采样器（Gens），这是一个与视频集成的插件模块，以促进有效的冗长视频感知。 Gens建立在轻巧的视频基础上，利用其固有的视觉语言功能来识别与问题相关的框架。为了促进有效的检索，我们构建了Gens-Video-150k，这是一个具有密集框架相关性注释的大型视频指令数据集。广泛的实验表明，GENS始终提高各种视频的性能，包括开源模型（QWEN2-VL-7B，ARIA-25B，VILA-40B，LLAVA-VIDEO-7B/72B）和所有权助理（GPT-4O，GEMINI）。在配备长期的视频基准上，开源视频卷在长期视频基准上获得了令人印象深刻的最新结果：llava-video-72b在LongvideObench上达到66.8（+4.3），Mlvu上的77.0（+2.7）在MLV上达到77.0（+2.7），而Aria在Hourvideo Survide survide survide survide survide survide survide survide survice survide survice survice survide surving sup the geminiii survise surving surving surving surving the Gemsinii-1.99中。我们将在此HTTPS URL上发布所有数据集和模型。

Title: Memory-enhanced Retrieval Augmentation for Long Video Understanding

• Authors: Huaying Yuan, Zheng Liu, Minhao Qin, Hongjin Qian, Y Shu, Zhicheng Dou, Ji-Rong Wen
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2503.09149
• Pdf URL: https://arxiv.org/pdf/2503.09149
• Copy Paste: [[2503.09149]] Memory-enhanced Retrieval Augmentation for Long Video Understanding(https://arxiv.org/abs/2503.09149)
• Keywords: generation
• Abstract: Retrieval-augmented generation (RAG) shows strong potential in addressing long-video understanding (LVU) tasks. However, traditional RAG methods remain fundamentally limited due to their dependence on explicit search queries, which are unavailable in many situations. To overcome this challenge, we introduce a novel RAG-based LVU approach inspired by the cognitive memory of human beings, which is called MemVid. Our approach operates with four basics steps: memorizing holistic video information, reasoning about the task's information needs based on the memory, retrieving critical moments based on the information needs, and focusing on the retrieved moments to produce the final answer. To enhance the system's memory-grounded reasoning capabilities and achieve optimal end-to-end performance, we propose a curriculum learning strategy. This approach begins with supervised learning on well-annotated reasoning results, then progressively explores and reinforces more plausible reasoning outcomes through reinforcement learning. We perform extensive evaluations on popular LVU benchmarks, including MLVU, VideoMME and LVBench. In our experiment, MemVid significantly outperforms existing RAG-based methods and popular LVU models, which demonstrate the effectiveness of our approach. Our model and source code will be made publicly available upon acceptance.
• 摘要：检索增强的一代（RAG）在解决长期video理解（LVU）任务方面具有强大的潜力。但是，传统的抹布方法由于其依赖于明确的搜索查询而在根本上仍然有限，而这些查询在许多情况下都无法使用。为了克服这一挑战，我们引入了一种新型基于抹布的LVU方法，其灵感来自人类的认知记忆，这称为Memvid。我们的方法采用四个基本步骤运行：记住整体视频信息，根据内存来理解任务信息需求，根据信息需求检索关键时刻，并专注于产生最终答案的检索时刻。为了增强系统的内存基础推理能力并实现最佳的端到端性能，我们提出了一种课程学习策略。这种方法始于对良好的推理结果的监督学习，然后逐步探索并通过加强学习来增强更多合理的推理结果。我们对包括MLVU，VideMomme和LVBench在内的流行LVU基准进行了广泛的评估。在我们的实验中，Memvid显着胜过现有的基于抹布的方法和流行的LVU模型，这些方法证明了我们方法的有效性。我们的模型和源代码将在接受后公开提供。

Title: Reangle-A-Video: 4D Video Generation as Video-to-Video Translation

• Authors: Hyeonho Jeong, Suhyeon Lee, Jong Chul Ye
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.09151
• Pdf URL: https://arxiv.org/pdf/2503.09151
• Copy Paste: [[2503.09151]] Reangle-A-Video: 4D Video Generation as Video-to-Video Translation(https://arxiv.org/abs/2503.09151)
• Keywords: generation
• Abstract: We introduce Reangle-A-Video, a unified framework for generating synchronized multi-view videos from a single input video. Unlike mainstream approaches that train multi-view video diffusion models on large-scale 4D datasets, our method reframes the multi-view video generation task as video-to-videos translation, leveraging publicly available image and video diffusion priors. In essence, Reangle-A-Video operates in two stages. (1) Multi-View Motion Learning: An image-to-video diffusion transformer is synchronously fine-tuned in a self-supervised manner to distill view-invariant motion from a set of warped videos. (2) Multi-View Consistent Image-to-Images Translation: The first frame of the input video is warped and inpainted into various camera perspectives under an inference-time cross-view consistency guidance using DUSt3R, generating multi-view consistent starting images. Extensive experiments on static view transport and dynamic camera control show that Reangle-A-Video surpasses existing methods, establishing a new solution for multi-view video generation. We will publicly release our code and data. Project page: this https URL
• 摘要：我们介绍了Reangle-A-Video，这是一个统一的框架，用于从单个输入视频中生成同步的多视频视频。与大规模4D数据集上训练多视频视频扩散模型的主流方法不同，我们的方法将多视频视频生成任务重新塑造为视频到视频翻译，利用公开可用的图像和视频扩散先验。从本质上讲，Reangle-A-Video分为两个阶段。 （1）多视图运动学习：图像到视频扩散变压器以一种自我监督的方式同步微调，以将视图不变的运动从一组扭曲的视频中提炼出来。 （2）多视图一致的图像到图像翻译：输入视频的第一帧被扭曲，并在推理时间跨视图的一致性指导下使用DUST3R旋转，并将其划分为各种相机视角，从而生成多视图一致的启动图像。关于静态视图传输和动态相机控制的广泛实验表明，Reangle-A-Video超过了现有方法，为多视频视频生成建立了新的解决方案。我们将公开发布我们的代码和数据。项目页面：此HTTPS URL

Title: WonderVerse: Extendable 3D Scene Generation with Video Generative Models

• Authors: Hao Feng, Zhi Zuo, Jia-hui Pan, Ka-hei Hui, Yi-hua Shao, Qi Dou, Wei Xie, Zheng-zhe Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09160
• Pdf URL: https://arxiv.org/pdf/2503.09160
• Copy Paste: [[2503.09160]] WonderVerse: Extendable 3D Scene Generation with Video Generative Models(https://arxiv.org/abs/2503.09160)
• Keywords: generation, generative
• Abstract: We introduce \textit{WonderVerse}, a simple but effective framework for generating extendable 3D scenes. Unlike existing methods that rely on iterative depth estimation and image inpainting, often leading to geometric distortions and inconsistencies, WonderVerse leverages the powerful world-level priors embedded within video generative foundation models to create highly immersive and geometrically coherent 3D environments. Furthermore, we propose a new technique for controllable 3D scene extension to substantially increase the scale of the generated environments. Besides, we introduce a novel abnormal sequence detection module that utilizes camera trajectory to address geometric inconsistency in the generated videos. Finally, WonderVerse is compatible with various 3D reconstruction methods, allowing both efficient and high-quality generation. Extensive experiments on 3D scene generation demonstrate that our WonderVerse, with an elegant and simple pipeline, delivers extendable and highly-realistic 3D scenes, markedly outperforming existing works that rely on more complex architectures.
• 摘要：我们介绍\ textit {Wonderverse}，这是一个简单但有效的框架，用于生成可扩展的3D场景。与依赖迭代深度估计和图像插入的现有方法不同，通常会导致几何扭曲和不一致之处，Wonderverse利用了嵌入了视频生成基础模型中的强大的世界级别的先验，以创建高度沉浸式和几何形式和几何相一致的3D环境。此外，我们提出了一种用于可控3D场景扩展的新技术，以大大增加生成环境的规模。此外，我们引入了一个新型的异常序列检测模块，该模块利用摄像头轨迹来解决生成视频中的几何不一致。最后，Wonderverse与各种3D重建方法兼容，允许有效和高质量的生成。 3D场景一代的广泛实验表明，我们的Wonderverse凭借优雅而简单的管道提供了可扩展且高度逼真的3D场景，明显超过了依赖更复杂体系结构的现有作品。

Title: Incomplete Multi-view Clustering via Diffusion Contrastive Generation

• Authors: Yuanyang Zhang, Yijie Lin, Weiqing Yan, Li Yao, Xinhang Wan, Guangyuan Li, Chao Zhang, Guanzhou Ke, Jie Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09185
• Pdf URL: https://arxiv.org/pdf/2503.09185
• Copy Paste: [[2503.09185]] Incomplete Multi-view Clustering via Diffusion Contrastive Generation(https://arxiv.org/abs/2503.09185)
• Keywords: generation
• Abstract: Incomplete multi-view clustering (IMVC) has garnered increasing attention in recent years due to the common issue of missing data in multi-view datasets. The primary approach to address this challenge involves recovering the missing views before applying conventional multi-view clustering methods. Although imputation-based IMVC methods have achieved significant improvements, they still encounter notable limitations: 1) heavy reliance on paired data for training the data recovery module, which is impractical in real scenarios with high missing data rates; 2) the generated data often lacks diversity and discriminability, resulting in suboptimal clustering results. To address these shortcomings, we propose a novel IMVC method called Diffusion Contrastive Generation (DCG). Motivated by the consistency between the diffusion and clustering processes, DCG learns the distribution characteristics to enhance clustering by applying forward diffusion and reverse denoising processes to intra-view data. By performing contrastive learning on a limited set of paired multi-view samples, DCG can align the generated views with the real views, facilitating accurate recovery of views across arbitrary missing view scenarios. Additionally, DCG integrates instance-level and category-level interactive learning to exploit the consistent and complementary information available in multi-view data, achieving robust and end-to-end clustering. Extensive experiments demonstrate that our method outperforms state-of-the-art approaches.
• 摘要：由于多视图数据集中缺少数据的常见问题，近年来，不完整的多视图群集（IMVC）近年来引起了人们的关注。应对此挑战的主要方法涉及在应用常规的多视图聚类方法之前恢复丢失的视图。尽管基于插补的IMVC方法已经取得了重大改进，但它们仍然遇到明显的局限性：1）严重依赖对数据恢复模块的配对数据，这在实际情况下是不切实际的，数据速率较高； 2）生成的数据通常缺乏多样性和可区分性，从而导致次优聚类结果。为了解决这些缺点，我们提出了一种新型的IMVC方法，称为扩散对比生成（DCG）。 DCG受到扩散过程和聚类过程之间的一致性的动机，通过将正向扩散和反向去核过程应用于视图数据的数据来学习分布特性，以增强聚类。通过对有限的配对多视图样本进行对比度学习，DCG可以将生成的视图与真实视图对齐，从而促进跨任意缺失的视图方案的准确恢复视图。此外，DCG集成了实例级别和类别级交互式学习，以利用多视图数据中可用的一致和互补信息，从而实现了强大的和端到端的群集。广泛的实验表明，我们的方法优于最先进的方法。

Title: Teaching LMMs for Image Quality Scoring and Interpreting

• Authors: Zicheng Zhang, Haoning Wu, Ziheng Jia, Weisi Lin, Guangtao Zhai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09197
• Pdf URL: https://arxiv.org/pdf/2503.09197
• Copy Paste: [[2503.09197]] Teaching LMMs for Image Quality Scoring and Interpreting(https://arxiv.org/abs/2503.09197)
• Keywords: quality assessment
• Abstract: Image quality scoring and interpreting are two fundamental components of Image Quality Assessment (IQA). The former quantifies image quality, while the latter enables descriptive question answering about image quality. Traditionally, these two tasks have been addressed independently. However, from the perspective of the Human Visual System (HVS) and the Perception-Decision Integration Model, they are inherently interconnected: interpreting serves as the foundation for scoring, while scoring provides an abstract summary of interpreting. Thus, unifying these capabilities within a single model is both intuitive and logically coherent. In this paper, we propose Q-SiT (Quality Scoring and Interpreting joint Teaching), a unified framework that enables large multimodal models (LMMs) to learn both image quality scoring and interpreting simultaneously. We achieve this by transforming conventional IQA datasets into learnable question-answering datasets and incorporating human-annotated quality interpreting data for training. Furthermore, we introduce an efficient scoring & interpreting balance strategy, which first determines the optimal data mix ratio on lightweight LMMs and then maps this ratio to primary LMMs for fine-tuning adjustment. This strategy not only mitigates task interference and enhances cross-task knowledge transfer but also significantly reduces computational costs compared to direct optimization on full-scale LMMs. With this joint learning framework and corresponding training strategy, we develop Q-SiT, the first model capable of simultaneously performing image quality scoring and interpreting tasks, along with its lightweight variant, Q-SiT-mini. Experimental results demonstrate that Q-SiT achieves strong performance in both tasks with superior generalization IQA this http URL page at this https URL.
• 摘要：图像质量评分和解释是图像质量评估（IQA）的两个基本组成部分。前者量化了图像质量，而后者可以对图像质量回答的描述性问题。传统上，这两个任务已被独立解决。但是，从人类视觉系统（HVS）和感知决策集成模型的角度来看，它们固有地相互联系：解释是评分的基础，而评分则提供了解释的抽象摘要。因此，在单个模型中统一这些功能既直观又是逻辑上的连贯性。在本文中，我们提出了Q-SIT（质量评分和解释联合教学），这是一个统一的框架，使大型多模型模型（LMMS）同时学习图像质量评分和解释。我们通过将常规的IQA数据集转换为可学习的提问数据集并结合人类通知的质量解释数据来实现这一目标。此外，我们引入了有效的评分和解释平衡策略，该策略首先确定轻量级LMM的最佳数据混合率，然后将此比率映射到主要LMMS以进行微调调整。该策略不仅减轻了任务干扰并增强了交叉任务知识转移，而且与直接优化全尺度LMM相比，计算成本大大降低了。借助这个联合学习框架和相应的培训策略，我们开发了Q-SIT，这是第一个能够同时执行图像质量评分和解释任务的模型，以及其轻巧的Q-SIT Mini。实验结果表明，Q-SIT在此HTTPS URL上的HTTP URL页面上具有出色概括IQA的两项任务中的强劲性能。

Title: Other Vehicle Trajectories Are Also Needed: A Driving World Model Unifies Ego-Other Vehicle Trajectories in Video Latant Space

• Authors: Jian Zhu, Zhengyu Jia, Tian Gao, Jiaxin Deng, Shidi Li, Fu Liu, Peng Jia, Xianpeng Lang, Xiaolong Sun
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.09215
• Pdf URL: https://arxiv.org/pdf/2503.09215
• Copy Paste: [[2503.09215]] Other Vehicle Trajectories Are Also Needed: A Driving World Model Unifies Ego-Other Vehicle Trajectories in Video Latant Space(https://arxiv.org/abs/2503.09215)
• Keywords: generation
• Abstract: Advanced end-to-end autonomous driving systems predict other vehicles' motions and plan ego vehicle's trajectory. The world model that can foresee the outcome of the trajectory has been used to evaluate the end-to-end autonomous driving system. However, existing world models predominantly emphasize the trajectory of the ego vehicle and leave other vehicles uncontrollable. This limitation hinders their ability to realistically simulate the interaction between the ego vehicle and the driving scenario. In addition, it remains a challenge to match multiple trajectories with each vehicle in the video to control the video generation. To address above issues, a driving \textbf{W}orld \textbf{M}odel named EOT-WM is proposed in this paper, unifying \textbf{E}go-\textbf{O}ther vehicle \textbf{T}rajectories in videos. Specifically, we first project ego and other vehicle trajectories in the BEV space into the image coordinate to match each trajectory with its corresponding vehicle in the video. Then, trajectory videos are encoded by the Spatial-Temporal Variational Auto Encoder to align with driving video latents spatially and temporally in the unified visual space. A trajectory-injected diffusion Transformer is further designed to denoise the noisy video latents for video generation with the guidance of ego-other vehicle trajectories. In addition, we propose a metric based on control latent similarity to evaluate the controllability of trajectories. Extensive experiments are conducted on the nuScenes dataset, and the proposed model outperforms the state-of-the-art method by 30\% in FID and 55\% in FVD. The model can also predict unseen driving scenes with self-produced trajectories.
• 摘要：先进的端到端自动驾驶系统可以预测其他车辆的动作和计划自我车辆的轨迹。可以预见轨迹结果的世界模型已用于评估端到端自动驾驶系统。但是，现有的世界模型主要强调自我车辆的轨迹，并使其他车辆无法控制。这种限制阻碍了他们实际模拟自我车辆与驾驶场景之间的相互作用的能力。此外，将多个轨迹与视频中的每辆车匹配以控制视频生成仍然是一个挑战。要解决上述问题，本文提出了一个名为eot-wm的驱动器\ textbf {w} orld \ textbf {m} odel odel odel unify \ textbf {e} go- \ textbf {o} ther ther ther ther ther textbf \ textbf {t textbf {t} textbf {t} rajectores in Videos中的rajectories。具体来说，我们首先将BEV空间中的自我和其他车辆轨迹投射到图像坐标中，以使每个轨迹与视频中的相应车辆匹配。然后，轨迹视频由空间变化自动编码器编码，以与统一的视觉空间中的空间和时间上的驾驶视频潜伏期保持一致。进一步设计了注入轨迹的扩散变压器，旨在通过自我轨迹的指导来代替视频发电的嘈杂的视频潜在。此外，我们提出了一个基于控制潜在相似性的度量，以评估轨迹的可控性。在Nuscenes数据集上进行了广泛的实验，而所提出的模型在FID中优于最先进的方法，而FVD中的55 \％则优于30 \％。该模型还可以通过自生产轨迹预测看不见的驾驶场景。

Title: Active Learning Inspired ControlNet Guidance for Augmenting Semantic Segmentation Datasets

• Authors: Hannah Kniesel, Pedro Hermosilla, Timo Ropinski
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09221
• Pdf URL: https://arxiv.org/pdf/2503.09221
• Copy Paste: [[2503.09221]] Active Learning Inspired ControlNet Guidance for Augmenting Semantic Segmentation Datasets(https://arxiv.org/abs/2503.09221)
• Keywords: generation
• Abstract: Recent advances in conditional image generation from diffusion models have shown great potential in achieving impressive image quality while preserving the constraints introduced by the user. In particular, ControlNet enables precise alignment between ground truth segmentation masks and the generated image content, allowing the enhancement of training datasets in segmentation tasks. This raises a key question: Can ControlNet additionally be guided to generate the most informative synthetic samples for a specific task? Inspired by active learning, where the most informative real-world samples are selected based on sample difficulty or model uncertainty, we propose the first approach to integrate active learning-based selection metrics into the backward diffusion process for sample generation. Specifically, we explore uncertainty, query by committee, and expected model change, which are commonly used in active learning, and demonstrate their application for guiding the sample generation process through gradient approximation. Our method is training-free, modifying only the backward diffusion process, allowing it to be used on any pretrained ControlNet. Using this process, we show that segmentation models trained with guided synthetic data outperform those trained on non-guided synthetic data. Our work underscores the need for advanced control mechanisms for diffusion-based models, which are not only aligned with image content but additionally downstream task performance, highlighting the true potential of synthetic data generation.
• 摘要：从扩散模型中有条件图像产生的最新进展显示出在达到令人印象深刻的图像质量的同时，保留用户引入的约束。特别是，ControlNet可以在地面真理分割掩码和生成的图像内容之间进行精确的对齐，从而可以增强分段任务中的培训数据集。这提出了一个关键问题：是否还可以指导ControlNet来为特定任务生成最有用的合成样本吗？受主动学习的启发，其中最有用的现实世界样本是根据样本难度或模型不确定性选择的，我们提出了将基于主动学习的选择指标集成到样本生成的向后扩散过程中的第一种方法。具体来说，我们探索不确定性，委员会查询以及预期的模型变化，这些变化通常用于主动学习中，并证明了他们通过梯度近似指导样本生成过程的应用。我们的方法是无训练的，仅修改向后扩散过程，从而可以在任何预验证的控制网络上使用。使用此过程，我们表明，经过指导合成数据训练的分割模型优于接受非引导合成数据训练的分割模型。我们的工作强调了基于扩散的模型的高级控制机制的需求，这些模型不仅与图像内容保持一致，而且还与下游任务性能保持一致，从而突出了合成数据生成的真正潜力。

Title: NAMI: Efficient Image Generation via Progressive Rectified Flow Transformers

• Authors: Yuhang Ma, Bo Cheng, Shanyuan Liu, Ao Ma, Xiaoyu Wu, Liebucha Wu, Dawei Leng, Yuhui Yin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09242
• Pdf URL: https://arxiv.org/pdf/2503.09242
• Copy Paste: [[2503.09242]] NAMI: Efficient Image Generation via Progressive Rectified Flow Transformers(https://arxiv.org/abs/2503.09242)
• Keywords: generation
• Abstract: Flow-based transformer models for image generation have achieved state-of-the-art performance with larger model parameters, but their inference deployment cost remains high. To enhance inference performance while maintaining generation quality, we propose progressive rectified flow transformers. We divide the rectified flow into different stages according to resolution, using fewer transformer layers at the low-resolution stages to generate image layouts and concept contours, and progressively adding more layers as the resolution increases. Experiments demonstrate that our approach achieves fast convergence and reduces inference time while ensuring generation quality. The main contributions of this paper are summarized as follows: (1) We introduce progressive rectified flow transformers that enable multi-resolution training, accelerating model convergence; (2) NAMI leverages piecewise flow and spatial cascading of Diffusion Transformer (DiT) to rapidly generate images, reducing inference time by 40% to generate a 1024 resolution image; (3) We propose NAMI-1K benchmark to evaluate human preference performance, aiming to mitigate distributional bias and prevent data leakage from open-source benchmarks. The results show that our model is competitive with state-of-the-art models.
• 摘要：图像生成的基于流量的变压器模型已通过较大的模型参数实现了最先进的性能，但是它们的推理部署成本仍然很高。为了增强推理性能的同时保持发电质量，我们提出了进行性纠正的流动变压器。我们根据分辨率将整流的流量分为不同的阶段，在低分辨率阶段使用更少的变压器层来生成图像布局和概念轮廓，并随着分辨率增加而逐渐添加更多的层。实验表明，我们的方法可实现快速的收敛性并减少推理时间，同时确保发电质量。本文的主要贡献总结如下：（1）我们介绍了进行性整流的流动变压器，以实现多分辨率培训，加速模型收敛； （2）NAMI利用扩散变压器（DIT）的分段流量和空间级联来快速生成图像，将推理时间减少40％以生成1024分辨率图像； （3）我们建议NAMI-1K基准测试以评估人类偏好性能，旨在减轻分布偏见并防止开源基准中的数据泄漏。结果表明，我们的模型与最先进的模型具有竞争力。

Title: UniCombine: Unified Multi-Conditional Combination with Diffusion Transformer

• Authors: Haoxuan Wang, Jinlong Peng, Qingdong He, Hao Yang, Ying Jin, Jiafu Wu, Xiaobin Hu, Yanjie Pan, Zhenye Gan, Mingmin Chi, Bo Peng, Yabiao Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.09277
• Pdf URL: https://arxiv.org/pdf/2503.09277
• Copy Paste: [[2503.09277]] UniCombine: Unified Multi-Conditional Combination with Diffusion Transformer(https://arxiv.org/abs/2503.09277)
• Keywords: generation, generative
• Abstract: With the rapid development of diffusion models in image generation, the demand for more powerful and flexible controllable frameworks is increasing. Although existing methods can guide generation beyond text prompts, the challenge of effectively combining multiple conditional inputs while maintaining consistency with all of them remains unsolved. To address this, we introduce UniCombine, a DiT-based multi-conditional controllable generative framework capable of handling any combination of conditions, including but not limited to text prompts, spatial maps, and subject images. Specifically, we introduce a novel Conditional MMDiT Attention mechanism and incorporate a trainable LoRA module to build both the training-free and training-based versions. Additionally, we propose a new pipeline to construct SubjectSpatial200K, the first dataset designed for multi-conditional generative tasks covering both the subject-driven and spatially-aligned conditions. Extensive experimental results on multi-conditional generation demonstrate the outstanding universality and powerful capability of our approach with state-of-the-art performance.
• 摘要：随着图像生成中扩散模型的快速发展，对更强大和灵活的可控框架的需求正在增加。尽管现有方法可以指导生成文本提示，但在保持多种条件输入的同时保持一致性的挑战仍未解决。为了解决这个问题，我们引入了Unicombine，这是一种基于DIT的多条件可控生成框架，能够处理任何条件组合，包括但不限于文本提示，空间图和主题图像。具体而言，我们引入了一种新型的条件MMDIT注意机制，并结合了可训练的Lora模块，以构建无训练和基于培训的版本。此外，我们提出了一条新的管道来构建主题Spatial200K，这是第一个用于涵盖主题驱动和空间对准条件的多条件生成任务的数据集。多条件生成的广泛实验结果证明了我们方法的出色普遍性和强大的能力，并表明了最先进的表现。

Title: IQPFR: An Image Quality Prior for Blind Face Restoration and Beyond

• Authors: Peng Hu, Chunming He, Lei Xu, Jingduo Tian, Sina Farsiu, Yulun Zhang, Pei Liu, Xiu Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09294
• Pdf URL: https://arxiv.org/pdf/2503.09294
• Copy Paste: [[2503.09294]] IQPFR: An Image Quality Prior for Blind Face Restoration and Beyond(https://arxiv.org/abs/2503.09294)
• Keywords: restoration, quality assessment
• Abstract: Blind Face Restoration (BFR) addresses the challenge of reconstructing degraded low-quality (LQ) facial images into high-quality (HQ) outputs. Conventional approaches predominantly rely on learning feature representations from ground-truth (GT) data; however, inherent imperfections in GT datasets constrain restoration performance to the mean quality level of the training data, rather than attaining maximally attainable visual quality. To overcome this limitation, we propose a novel framework that incorporates an Image Quality Prior (IQP) derived from No-Reference Image Quality Assessment (NR-IQA) models to guide the restoration process toward optimal HQ reconstructions. Our methodology synergizes this IQP with a learned codebook prior through two critical innovations: (1) During codebook learning, we devise a dual-branch codebook architecture that disentangles feature extraction into universal structural components and HQ-specific attributes, ensuring comprehensive representation of both common and high-quality facial characteristics. (2) In the codebook lookup stage, we implement a quality-conditioned Transformer-based framework. NR-IQA-derived quality scores act as dynamic conditioning signals to steer restoration toward the highest feasible quality standard. This score-conditioned paradigm enables plug-and-play enhancement of existing BFR architectures without modifying the original structure. We also formulate a discrete representation-based quality optimization strategy that circumvents over-optimization artifacts prevalent in continuous latent space approaches. Extensive experiments demonstrate that our method outperforms state-of-the-art techniques across multiple benchmarks. Besides, our quality-conditioned framework demonstrates consistent performance improvements when integrated with prior BFR models. The code will be released.
• 摘要：盲人面部修复（BFR）解决了将降级低质量（LQ）面部图像重建为高质量（HQ）输出的挑战。常规方法主要依赖于地面图（GT）数据的学习特征表示。但是，GT数据集中的固有缺陷将恢复性能限制为训练数据的平均质量水平，而不是获得最大可达到的视觉质量。为了克服这一限制，我们提出了一个新型框架，该框架结合了源自无参考图像质量评估（NR-IQA）模型的图像质量（IQP），以指导恢复过程达到最佳HQ重建。我们的方法论通过两项关键创新提前与博学的代码书协同结合：（1）在代码书学习期间，我们设计了一种双分支代码簿架构，该架构将解散的特征分解为通用的结构组件和HQ特异性属性，从而确保对常见和高素质面部特征的全面表示。 （2）在Codebook查找阶段，我们实现了基于质量条件的基于质量的变压器框架。 NR-IQA衍生的质量得分充当动态调节信号，以将恢复到最高可行的质量标准。该分数条件的范式可实现现有BFR体系结构的插件，而无需修改原始结构。我们还制定了一种基于离散表示的质量优化策略，该策略规避在连续的潜在空间方法中普遍存在的过度优化伪像。广泛的实验表明，我们的方法优于多个基准测试的最先进技术。此外，与先前的BFR模型集成在一起时，我们的质量条件框架表现出一致的性能改进。代码将发布。

Title: Revealing Unintentional Information Leakage in Low-Dimensional Facial Portrait Representations

• Authors: Kathleen Anderson, Thomas Martinetz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09306
• Pdf URL: https://arxiv.org/pdf/2503.09306
• Copy Paste: [[2503.09306]] Revealing Unintentional Information Leakage in Low-Dimensional Facial Portrait Representations(https://arxiv.org/abs/2503.09306)
• Keywords: generation
• Abstract: We evaluate the information that can unintentionally leak into the low dimensional output of a neural network, by reconstructing an input image from a 40- or 32-element feature vector that intends to only describe abstract attributes of a facial portrait. The reconstruction uses blackbox-access to the image encoder which generates the feature vector. Other than previous work, we leverage recent knowledge about image generation and facial similarity, implementing a method that outperforms the current state-of-the-art. Our strategy uses a pretrained StyleGAN and a new loss function that compares the perceptual similarity of portraits by mapping them into the latent space of a FaceNet embedding. Additionally, we present a new technique that fuses the output of an ensemble, to deliberately generate specific aspects of the recreated image.
• 摘要：我们通过从40或32个元素特征向量向外重建输入图像来评估可能无意间泄漏到神经网络的低维输出中的信息，该图像只打算仅描述面部肖像的抽象属性。重建使用BlackBox-access到生成特征向量的图像编码器。除了以前的工作外，我们还利用有关图像产生和面部相似性的最新知识，实施了一种胜过当前最新技术的方法。我们的策略使用了验证的样式和新的损失功能，该功能通过将肖像映射到面部嵌入的潜在空间来比较了肖像的感知相似性。此外，我们提出了一种融合合奏输出的新技术，以故意生成重新创建图像的特定方面。

Title: Revealing the Implicit Noise-based Imprint of Generative Models

• Authors: Xinghan Li, Jingjing Chen, Yue Yu, Xue Song, Haijun Shan, Yu-Gang Jiang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09314
• Pdf URL: https://arxiv.org/pdf/2503.09314
• Copy Paste: [[2503.09314]] Revealing the Implicit Noise-based Imprint of Generative Models(https://arxiv.org/abs/2503.09314)
• Keywords: generation, generative
• Abstract: With the rapid advancement of vision generation models, the potential security risks stemming from synthetic visual content have garnered increasing attention, posing significant challenges for AI-generated image detection. Existing methods suffer from inadequate generalization capabilities, resulting in unsatisfactory performance on emerging generative models. To address this issue, this paper presents a novel framework that leverages noise-based model-specific imprint for the detection task. Specifically, we propose a novel noise-based imprint simulator to capture intrinsic patterns imprinted in images generated by different models. By aggregating imprints from various generative models, imprints of future models can be extrapolated to expand training data, thereby enhancing generalization and robustness. Furthermore, we design a new pipeline that pioneers the use of noise patterns, derived from a noise-based imprint extractor, alongside other visual features for AI-generated image detection, resulting in a significant improvement in performance. Our approach achieves state-of-the-art performance across three public benchmarks including GenImage, Synthbuster and Chameleon.
• 摘要：随着视觉产生模型的快速发展，合成视觉内容引起的潜在安全风险越来越引起人们的注意，对AI生成的图像检测提出了重大挑战。现有方法的概括能力不足，导致新兴生成模型的性能不令人满意。为了解决这个问题，本文提出了一个新颖的框架，该框架利用基于噪声的模型特定烙印来进行检测任务。具体而言，我们提出了一种新型的基于噪声的烙印模拟器，以捕获不同模型产生的图像中的固有模式。通过汇总来自各种生成模型的烙印，可以将未来模型的烙印推送出来扩大训练数据，从而增强概括和鲁棒性。此外，我们设计了一条新的管道，该管道开创了源自基于噪声的烙印器的噪声模式的使用，以及其他视觉特征，用于AI生成的图像检测，从而显着改善了性能。我们的方法在包括Genimage，Synthbuster和Chameleon在内的三个公共基准中实现了最先进的表现。

Title: SDD-4DGS: Static-Dynamic Aware Decoupling in Gaussian Splatting for 4D Scene Reconstruction

• Authors: Dai Sun, Huhao Guan, Kun Zhang, Xike Xie, S. Kevin Zhou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.09332
• Pdf URL: https://arxiv.org/pdf/2503.09332
• Copy Paste: [[2503.09332]] SDD-4DGS: Static-Dynamic Aware Decoupling in Gaussian Splatting for 4D Scene Reconstruction(https://arxiv.org/abs/2503.09332)
• Keywords: restoration
• Abstract: Dynamic and static components in scenes often exhibit distinct properties, yet most 4D reconstruction methods treat them indiscriminately, leading to suboptimal performance in both cases. This work introduces SDD-4DGS, the first framework for static-dynamic decoupled 4D scene reconstruction based on Gaussian Splatting. Our approach is built upon a novel probabilistic dynamic perception coefficient that is naturally integrated into the Gaussian reconstruction pipeline, enabling adaptive separation of static and dynamic components. With carefully designed implementation strategies to realize this theoretical framework, our method effectively facilitates explicit learning of motion patterns for dynamic elements while maintaining geometric stability for static structures. Extensive experiments on five benchmark datasets demonstrate that SDD-4DGS consistently outperforms state-of-the-art methods in reconstruction fidelity, with enhanced detail restoration for static structures and precise modeling of dynamic motions. The code will be released.
• 摘要：场景中的动态和静态组件通常具有不同的特性，但是大多数4D重建方法都不可差治疗它们，从而在两种情况下都造成了次优性能。这项工作介绍了SDD-4DGS，这是基于高斯裂缝的静态动态解耦4D场景重建的第一个框架。我们的方法建立在一种新型的概率动态感知系数的基础上，该动态感知系数自然地整合到高斯重建管道中，从而实现了静态和动态组件的适应性分离。通过精心设计的实施策略来实现这一理论框架，我们的方法有效地促进了动态元素的运动模式的明确学习，同时保持静态结构的几何稳定性。在五个基准数据集上进行的广泛实验表明，SDD-4DG在重建忠诚度中始终优于最先进的方法，并增强了对静态结构的详细恢复和动态运动的精确建模。代码将发布。

Title: Unified Dense Prediction of Video Diffusion

• Authors: Lehan Yang, Lu Qi, Xiangtai Li, Sheng Li, Varun Jampani, Ming-Hsuan Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09344
• Pdf URL: https://arxiv.org/pdf/2503.09344
• Copy Paste: [[2503.09344]] Unified Dense Prediction of Video Diffusion(https://arxiv.org/abs/2503.09344)
• Keywords: generation
• Abstract: We present a unified network for simultaneously generating videos and their corresponding entity segmentation and depth maps from text prompts. We utilize colormap to represent entity masks and depth maps, tightly integrating dense prediction with RGB video generation. Introducing dense prediction information improves video generation's consistency and motion smoothness without increasing computational costs. Incorporating learnable task embeddings brings multiple dense prediction tasks into a single model, enhancing flexibility and further boosting performance. We further propose a large-scale dense prediction video dataset~\datasetname, addressing the issue that existing datasets do not concurrently contain captions, videos, segmentation, or depth maps. Comprehensive experiments demonstrate the high efficiency of our method, surpassing the state-of-the-art in terms of video quality, consistency, and motion smoothness.
• 摘要：我们提出了一个统一的网络，用于同时生成视频及其相应的实体细分和从文本提示中生成的深度图。我们利用colormap代表实体面具和深度图，将密集的预测与RGB视频生成紧密整合在一起。引入密集的预测信息可改善视频发电的一致性和运动平滑度，而无需增加计算成本。结合可学习的任务嵌入将使多个密集的预测任务分为单个模型，增强灵活性并进一步提高性能。我们进一步提出了一个大规模密集的预测视频数据集〜\ datasetname，解决了现有数据集不同时包含字幕，视频，细分或深度图的问题。全面的实验证明了我们方法的效率很高，在视频质量，一致性和运动平滑度方面超过了最新的效率。

Title: PerCoV2: Improved Ultra-Low Bit-Rate Perceptual Image Compression with Implicit Hierarchical Masked Image Modeling

• Authors: Nikolai Körber, Eduard Kromer, Andreas Siebert, Sascha Hauke, Daniel Mueller-Gritschneder, Björn Schuller
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2503.09368
• Pdf URL: https://arxiv.org/pdf/2503.09368
• Copy Paste: [[2503.09368]] PerCoV2: Improved Ultra-Low Bit-Rate Perceptual Image Compression with Implicit Hierarchical Masked Image Modeling(https://arxiv.org/abs/2503.09368)
• Keywords: generation
• Abstract: We introduce PerCoV2, a novel and open ultra-low bit-rate perceptual image compression system designed for bandwidth- and storage-constrained applications. Building upon prior work by Careil et al., PerCoV2 extends the original formulation to the Stable Diffusion 3 ecosystem and enhances entropy coding efficiency by explicitly modeling the discrete hyper-latent image distribution. To this end, we conduct a comprehensive comparison of recent autoregressive methods (VAR and MaskGIT) for entropy modeling and evaluate our approach on the large-scale MSCOCO-30k benchmark. Compared to previous work, PerCoV2 (i) achieves higher image fidelity at even lower bit-rates while maintaining competitive perceptual quality, (ii) features a hybrid generation mode for further bit-rate savings, and (iii) is built solely on public components. Code and trained models will be released at this https URL.
• 摘要：我们介绍了Percov2，这是一种新颖而开放的超低位比率感知图像压缩系统，专为带宽和存储约束的应用而设计。在Careil等人的先前工作的基础上，Percov2将原始配方扩展到稳定的扩散3生态系统，并通过明确对离散的超级图像分布进行显式建模，从而提高熵编码效率。为此，我们对熵建模的最新自回旋方法（VAR和MaskGit）进行了全面比较，并评估了我们在大型MSCOCO-30K基准测试中的方法。与以前的工作相比，Percov2（i）在保持竞争性感知质量的同时，在较低的比特率下实现了更高的图像保真度，（ii）具有用于进一步储蓄的混合生成模式，并且（iii）仅基于公共组件。代码和训练有素的模型将在此HTTPS URL上发布。

Title: Bidirectional Prototype-Reward co-Evolution for Test-Time Adaptation of Vision-Language Models

• Authors: Xiaozhen Qiao, Peng Huang, Jiakang Yuan, Xianda Guo, Bowen Ye, Zhe Sun, Xuelong Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09394
• Pdf URL: https://arxiv.org/pdf/2503.09394
• Copy Paste: [[2503.09394]] Bidirectional Prototype-Reward co-Evolution for Test-Time Adaptation of Vision-Language Models(https://arxiv.org/abs/2503.09394)
• Keywords: quality assessment
• Abstract: Test-time adaptation (TTA) is crucial in maintaining Vision-Language Models (VLMs) performance when facing real-world distribution shifts, particularly when the source data or target labels are inaccessible. Existing TTA methods rely on CLIP's output probability distribution for feature evaluation, which can introduce biases under domain shifts. This misalignment may cause features to be misclassified due to text priors or incorrect textual associations. To address these limitations, we propose Bidirectional Prototype-Reward co-Evolution (BPRE), a novel TTA framework for VLMs that integrates feature quality assessment with prototype evolution through a synergistic feedback loop. BPRE first employs a Multi-Dimensional Quality-Aware Reward Module to evaluate feature quality and guide prototype refinement precisely. The continuous refinement of prototype quality through Prototype-Reward Interactive Evolution will subsequently enhance the computation of more robust Multi-Dimensional Quality-Aware Reward Scores. Through the bidirectional interaction, the precision of rewards and the evolution of prototypes mutually reinforce each other, forming a self-evolving cycle. Extensive experiments are conducted across 15 diverse recognition datasets encompassing natural distribution shifts and cross-dataset generalization scenarios. Results demonstrate that BPRE consistently achieves superior average performance compared to state-of-the-art methods across different model architectures, such as ResNet-50 and ViT-B/16. By emphasizing comprehensive feature evaluation and bidirectional knowledge refinement, BPRE advances VLM generalization capabilities, offering a new perspective on TTA.
• 摘要：测试时间适应（TTA）在面对现实世界分布变化时保持视觉模型（VLMS）的性能至关重要，尤其是当源数据或目标标签无法访问时。现有的TTA方法依赖于夹子的输出概率分布来进行功能评估，这可以在域移动下引入偏见。由于文本先验或文本关联不正确，这种未对准可能导致特征被误分类。为了解决这些局限性，我们提出了双向原型 - 奖励共同进化（BPRE），这是VLM的新型TTA框架，通过协同反馈循环将特征质量评估与原型演化相结合。 BPRE首先采用多维质量感知的奖励模块来评估功能质量，并精确地指导原型细化。通过原型奖励交互式演变对原型质量的连续完善将随后增强更强大的多维质量意识奖励分数的计算。通过双向相互作用，奖励的精度和原型的演变相互加强，形成一个自我发展的循环。在15个不同的识别数据集中进行了广泛的实验，其中包括自然分布变化和交叉概括场景。结果表明，与不同模型体系结构（例如Resnet-50和VIT-B/16）的最新方法相比，BPRE始终达到的平均性能始终取得了较高的平均性能。通过强调全面的功能评估和双向知识的完善，BPRE提高了VLM泛化功能，从而提供了有关TTA的新观点。

Title: Close-up-GS: Enhancing Close-Up View Synthesis in 3D Gaussian Splatting with Progressive Self-Training

• Authors: Jiatong Xia, Lingqiao Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.09396
• Pdf URL: https://arxiv.org/pdf/2503.09396
• Copy Paste: [[2503.09396]] Close-up-GS: Enhancing Close-Up View Synthesis in 3D Gaussian Splatting with Progressive Self-Training(https://arxiv.org/abs/2503.09396)
• Keywords: generation, generative
• Abstract: 3D Gaussian Splatting (3DGS) has demonstrated impressive performance in synthesizing novel views after training on a given set of viewpoints. However, its rendering quality deteriorates when the synthesized view deviates significantly from the training views. This decline occurs due to (1) the model's difficulty in generalizing to out-of-distribution scenarios and (2) challenges in interpolating fine details caused by substantial resolution changes and occlusions. A notable case of this limitation is close-up view generation--producing views that are significantly closer to the object than those in the training set. To tackle this issue, we propose a novel approach for close-up view generation based by progressively training the 3DGS model with self-generated data. Our solution is based on three key ideas. First, we leverage the See3D model, a recently introduced 3D-aware generative model, to enhance the details of rendered views. Second, we propose a strategy to progressively expand the ``trust regions'' of the 3DGS model and update a set of reference views for See3D. Finally, we introduce a fine-tuning strategy to carefully update the 3DGS model with training data generated from the above schemes. We further define metrics for close-up views evaluation to facilitate better research on this problem. By conducting evaluations on specifically selected scenarios for close-up views, our proposed approach demonstrates a clear advantage over competitive solutions.
• 摘要：在给定的一组观点训练后，3D高斯裂（3DG）在综合新观点方面表现出了令人印象深刻的表现。但是，当合成视图显着偏离培训观点时，其渲染质量会恶化。这种下降是由于（1）模型在分布外情景中的难度以及（2）在插值的挑战中介绍了由实质性分辨率变化和遮挡引起的细节。该限制的一个值得注意的情况是特写视图的生成 - 产生的观点比训练集中的观点更接近对象。为了解决这个问题，我们通过逐步培训使用自生数据的3DGS模型来提出一种新颖的方法，用于近距离视图生成。我们的解决方案是基于三个关键想法。首先，我们利用最近引入的3D感知生成模型See3D模型来增强渲染视图的细节。其次，我们提出了一种策略，以逐步扩展3DGS模型的``信任区域''并更新See3D的一组参考视图。最后，我们介绍了一种微调策略，以通过上述方案生成的培训数据仔细更新3DGS模型。我们进一步定义了指标以进行特写视图评估，以促进对此问题的更好研究。通过对特定选择的特写视图方案进行评估，我们提出的方法证明了比竞争解决方案具有明显的优势。

Title: VLog: Video-Language Models by Generative Retrieval of Narration Vocabulary

• Authors: Kevin Qinghong Lin, Mike Zheng Shou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09402
• Pdf URL: https://arxiv.org/pdf/2503.09402
• Copy Paste: [[2503.09402]] VLog: Video-Language Models by Generative Retrieval of Narration Vocabulary(https://arxiv.org/abs/2503.09402)
• Keywords: generative
• Abstract: Human daily activities can be concisely narrated as sequences of routine events (e.g., turning off an alarm) in video streams, forming an event vocabulary. Motivated by this, we introduce VLog, a novel video understanding framework that define video narrations as vocabulary, going beyond the typical subword vocabularies in existing generative video-language models. Built on the lightweight language model GPT-2, VLog feature three key innovations: (i) A generative retrieval model, marrying language model's complex reasoning capabilities with contrastive retrieval's efficient similarity search. (ii) A hierarchical vocabulary derived from large-scale video narrations using our narration pair encoding algorithm, enabling efficient indexing of specific events (e.g., cutting a tomato) by identifying broader scenarios (e.g., kitchen) with expressive postfixes (e.g., by the left hand). (iii) A vocabulary update strategy leveraging generative models to extend the vocabulary for novel events encountered during inference. To validate our approach, we introduce VidCap-Eval, a development set requiring concise narrations with reasoning relationships (e.g., before and after). Experiments on EgoSchema, COIN, and HiREST further demonstrate the effectiveness of VLog, highlighting its ability to generate concise, contextually accurate, and efficient narrations, offering a novel perspective on video understanding. Codes are released at this https URL.
• 摘要：人类的日常活动可以简洁地叙述为视频流中的常规事件序列（例如，关闭警报），形成事件词汇。在此激励的情况下，我们介绍了Vlog，这是一个新颖的视频理解框架，将视频叙述定义为词汇，超越了现有的生成视频语言模型中典型的子字词汇。 VLOG建立在轻巧的语言模型GPT-2上，具有三个关键创新：（i）生成检索模型，将语言模型的复杂推理功能与对比检索的有效相似性搜索结合在一起。 （ii）使用我们的叙述对编码算法的大规模视频叙述衍生出的层次词汇，通过识别更广泛的场景（例如，厨房），用表达的后置换（例如，由左手）来识别更广泛的场景（例如，厨房），从而有效地对特定事件（例如切番茄）进行有效的索引。 （iii）一种利用生成模型的词汇更新策略来扩展推理过程中遇到的新事件的词汇。为了验证我们的方法，我们介绍了Vidcap-eval，这是一个开发集，需要具有推理关系的简洁叙述（例如，之前和之后）。关于Egoschema，Coin和Hirest的实验进一步展示了视频博客的有效性，突出了其产生简洁，上下文准确且高效的叙述的能力，为视频理解提供了新的观点。代码在此HTTPS URL上发布。

Title: Multi-Agent Image Restoration

• Authors: Xu Jiang, Gehui Li, Bin Chen, Jian Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.09403
• Pdf URL: https://arxiv.org/pdf/2503.09403
• Copy Paste: [[2503.09403]] Multi-Agent Image Restoration(https://arxiv.org/abs/2503.09403)
• Keywords: restoration
• Abstract: Image restoration (IR) is challenging due to the complexity of real-world degradations. While many specialized and all-in-one IR models have been developed, they fail to effectively handle complex, mixed degradations. Recent agentic methods RestoreAgent and AgenticIR leverage intelligent, autonomous workflows to alleviate this issue, yet they suffer from suboptimal results and inefficiency due to their resource-intensive finetunings, and ineffective searches and tool execution trials for satisfactory outputs. In this paper, we propose MAIR, a novel Multi-Agent approach for complex IR problems. We introduce a real-world degradation prior, categorizing degradations into three types: (1) scene, (2) imaging, and (3) compression, which are observed to occur sequentially in real world, and reverse them in the opposite order. Built upon this three-stage restoration framework, MAIR emulates a team of collaborative human specialists, including a "scheduler" for overall planning and multiple "experts" dedicated to specific degradations. This design minimizes search space and trial efforts, improving image quality while reducing inference costs. In addition, a registry mechanism is introduced to enable easy integration of new tools. Experiments on both synthetic and real-world datasets show that proposed MAIR achieves competitive performance and improved efficiency over the previous agentic IR system. Code and models will be made available.
• 摘要：由于现实世界降解的复杂性，图像恢复（IR）具有挑战性。尽管已经开发了许多专业和多合一的IR模型，但它们无法有效处理复杂的混合降解。最近的代理方法还原和代理商利用了智能，自动工作流程来减轻此问题，但由于其资源密集型的填充，他们的效果和效率低下，却遭受了较低的效果，搜索和工具执行试验无效。在本文中，我们提出了Mair，这是一种用于复杂IR问题的新型多代理方法。我们引入了现实世界中的降级，将降解分为三种类型：（1）场景，（2）成像和（3）压缩，观察到这些压缩是在现实世界中依次发生的，并以相反的顺序反转它们。 Mair建立在这个三阶段的恢复框架的基础上，模仿了一个协作人类专家团队，其中包括用于整体计划的“调度程序”和多个致力于特定退化的“专家”。这种设计可最大程度地减少搜索空间和试验，从而提高图像质量，同时降低推理成本。此外，还引入了注册机制，以简化新工具的整合。合成和现实世界数据集的实验表明，提出的MAIR可以在先前的代理IR系统上实现竞争性能并提高效率。代码和型号将提供。

Title: Diff-CL: A Novel Cross Pseudo-Supervision Method for Semi-supervised Medical Image Segmentation

• Authors: Xiuzhen Guo, Lianyuan Yu, Ji Shi, Na Lei, Hongxiao Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09408
• Pdf URL: https://arxiv.org/pdf/2503.09408
• Copy Paste: [[2503.09408]] Diff-CL: A Novel Cross Pseudo-Supervision Method for Semi-supervised Medical Image Segmentation(https://arxiv.org/abs/2503.09408)
• Keywords: generative
• Abstract: Semi-supervised learning utilizes insights from unlabeled data to improve model generalization, thereby reducing reliance on large labeled datasets. Most existing studies focus on limited samples and fail to capture the overall data distribution. We contend that combining distributional information with detailed information is crucial for achieving more robust and accurate segmentation results. On the one hand, with its robust generative capabilities, diffusion models (DM) learn data distribution effectively. However, it struggles with fine detail capture, leading to generated images with misleading details. Combining DM with convolutional neural networks (CNNs) enables the former to learn data distribution while the latter corrects fine details. While capturing complete high-frequency details by CNNs requires substantial computational resources and is susceptible to local noise. On the other hand, given that both labeled and unlabeled data come from the same distribution, we believe that regions in unlabeled data similar to overall class semantics to labeled data are likely to belong to the same class, while regions with minimal similarity are less likely to. This work introduces a semi-supervised medical image segmentation framework from the distribution perspective (Diff-CL). Firstly, we propose a cross-pseudo-supervision learning mechanism between diffusion and convolution segmentation networks. Secondly, we design a high-frequency mamba module to capture boundary and detail information globally. Finally, we apply contrastive learning for label propagation from labeled to unlabeled data. Our method achieves state-of-the-art (SOTA) performance across three datasets, including left atrium, brain tumor, and NIH pancreas datasets.
• 摘要：半监督学习利用来自未标记数据的见解来改善模型概括，从而减少了对大型标记数据集的依赖。大多数现有的研究集中在有限的样本上，并且无法捕获总体数据分布。我们认为，将分布信息与详细信息结合在一起对于获得更健壮和准确的分割结果至关重要。一方面，具有强大的生成能力，扩散模型（DM）有效地学习了数据分布。但是，它在细节捕获方面挣扎，导致产生的图像具有误导性的细节。将DM与卷积神经网络（CNN）相结合，使前者可以学习数据分布，而后者则可以纠正细节。在捕获CNN的完整高频细节的同时，需要大量的计算资源，并且容易受到当地噪声的影响。另一方面，鉴于标记和未标记的数据都来自相同的分布，我们认为，与标记数据相似的无标记数据中的区域可能属于同一类，而相似性最小的区域的可能性较小。这项工作从分布角度（DIFF-CL）引入了半监督的医学图像分割框架。首先，我们在扩散和卷积分割网络之间提出了一种跨越伪造的学习机制。其次，我们设计了一个高频MAMBA模块，以在全球范围内捕获边界和详细信息。最后，我们将对比度学习用于标签传播，从标记到未标记的数据。我们的方法在包括左心房，脑肿瘤和NIH胰腺数据集在内的三个数据集中实现了最新的（SOTA）性能。

Title: Alias-Free Latent Diffusion Models:Improving Fractional Shift Equivariance of Diffusion Latent Space

• Authors: Yifan Zhou, Zeqi Xiao, Shuai Yang, Xingang Pan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09419
• Pdf URL: https://arxiv.org/pdf/2503.09419
• Copy Paste: [[2503.09419]] Alias-Free Latent Diffusion Models:Improving Fractional Shift Equivariance of Diffusion Latent Space(https://arxiv.org/abs/2503.09419)
• Keywords: generation
• Abstract: Latent Diffusion Models (LDMs) are known to have an unstable generation process, where even small perturbations or shifts in the input noise can lead to significantly different outputs. This hinders their applicability in applications requiring consistent results. In this work, we redesign LDMs to enhance consistency by making them shift-equivariant. While introducing anti-aliasing operations can partially improve shift-equivariance, significant aliasing and inconsistency persist due to the unique challenges in LDMs, including 1) aliasing amplification during VAE training and multiple U-Net inferences, and 2) self-attention modules that inherently lack shift-equivariance. To address these issues, we redesign the attention modules to be shift-equivariant and propose an equivariance loss that effectively suppresses the frequency bandwidth of the features in the continuous domain. The resulting alias-free LDM (AF-LDM) achieves strong shift-equivariance and is also robust to irregular warping. Extensive experiments demonstrate that AF-LDM produces significantly more consistent results than vanilla LDM across various applications, including video editing and image-to-image translation. Code is available at: this https URL
• 摘要：已知潜在扩散模型（LDMS）具有不稳定的生成过程，即使输入噪声的少量扰动或变化也会导致显着不同的输出。这阻碍了他们在需要一致结果的应用中的适用性。在这项工作中，我们通过使其变速均衡性来重新设计LDM，以提高一致性。在引入反叠缩操作的同时，由于LDMS中的独特挑战，包括1）在VAE培训和多个U-NET推断期间的异化放大，以及2）自我关注模块固有地缺乏固有的转移稳定性。为了解决这些问题，我们将注意力模块重新设计为移位等值级，并提出了均衡损失，从而有效地抑制了连续域中特征的频率带宽。由此产生的无别名LDM（AF-LDM）实现了强大的移位均衡性，并且对不规则的翘曲也很强。广泛的实验表明，在各种应用程序中，AF-LDM比Vanilla LDM产生的结果明显高得多，包括视频编辑和图像到图像翻译。代码可用：此HTTPS URL

Title: Multimodal Language Modeling for High-Accuracy Single Cell Transcriptomics Analysis and Generation

• Authors: Yaorui Shi, Jiaqi Yang, Sihang Li, Junfeng Fang, Xiang Wang, Zhiyuan Liu, Yang Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.09427
• Pdf URL: https://arxiv.org/pdf/2503.09427
• Copy Paste: [[2503.09427]] Multimodal Language Modeling for High-Accuracy Single Cell Transcriptomics Analysis and Generation(https://arxiv.org/abs/2503.09427)
• Keywords: generation, generative
• Abstract: Pre-trained language models (PLMs) have revolutionized scientific research, yet their application to single-cell analysis remains limited. Text PLMs cannot process single-cell RNA sequencing data, while cell PLMs lack the ability to handle free text, restricting their use in multimodal tasks. Existing efforts to bridge these modalities often suffer from information loss or inadequate single-modal pre-training, leading to suboptimal performances. To address these challenges, we propose Single-Cell MultiModal Generative Pre-trained Transformer (scMMGPT), a unified PLM for joint cell and text modeling. scMMGPT effectively integrates the state-of-the-art cell and text PLMs, facilitating cross-modal knowledge sharing for improved performance. To bridge the text-cell modality gap, scMMGPT leverages dedicated cross-modal projectors, and undergoes extensive pre-training on 27 million cells -- the largest dataset for multimodal cell-text PLMs to date. This large-scale pre-training enables scMMGPT to excel in joint cell-text tasks, achieving an 84\% relative improvement of textual discrepancy for cell description generation, 20.5\% higher accuracy for cell type annotation, and 4\% improvement in $k$-NN accuracy for text-conditioned pseudo-cell generation, outperforming baselines.
• 摘要：预训练的语言模型（PLM）彻底改变了科学研究，但它们在单细胞分析中的应用仍然有限。文本PLM无法处理单细胞RNA测序数据，而细胞PLM缺乏处理自由文本的能力，从而限制了它们在多模式任务中的使用。现有的桥接这些模式的努力通常会遭受信息丢失或单模式预训练不足，从而导致次优的性能。为了应对这些挑战，我们提出了单电池多模式生成预训练的变压器（SCMMGPT），这是一种用于关节电池和文本建模的统一PLM。 SCMMGPT有效地集成了最先进的单元格和文本PLM，从而促进了跨模式知识共享，以提高性能。为了弥合文本模式差距，SCMMGPT利用专用的跨模式投影仪，并在2700万个单元格上进行了广泛的预训练 - 迄今为止，最大的多模式细胞文本PLM的数据集。 This large-scale pre-training enables scMMGPT to excel in joint cell-text tasks, achieving an 84\% relative improvement of textual discrepancy for cell description generation, 20.5\% higher accuracy for cell type annotation, and 4\% improvement in $k$-NN accuracy for text-conditioned pseudo-cell generation, outperforming baselines.

Title: SuperCarver: Texture-Consistent 3D Geometry Super-Resolution for High-Fidelity Surface Detail Generation

• Authors: Qijian Zhang, Xiaozheng Jian, Xuan Zhang, Wenping Wang, Junhui Hou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09439
• Pdf URL: https://arxiv.org/pdf/2503.09439
• Copy Paste: [[2503.09439]] SuperCarver: Texture-Consistent 3D Geometry Super-Resolution for High-Fidelity Surface Detail Generation(https://arxiv.org/abs/2503.09439)
• Keywords: super-resolution, generation
• Abstract: Traditional production workflow of high-precision 3D mesh assets necessitates a cumbersome and laborious process of manual sculpting by specialized modelers. The recent years have witnessed remarkable advances in AI-empowered 3D content creation. However, although the latest state-of-the-arts are already capable of generating plausible structures and intricate appearances from images or text prompts, the actual mesh surfaces are typically over-smoothing and lack geometric details. This paper introduces SuperCarver, a 3D geometry super-resolution framework particularly tailored for adding texture-consistent surface details to given coarse meshes. Technically, we start by rendering the original textured mesh into the image domain from multiple viewpoints. To achieve geometric detail generation, we develop a deterministic prior-guided normal diffusion model fine-tuned on a carefully curated dataset of paired low-poly and high-poly normal renderings. To optimize mesh structures from potentially imperfect normal map predictions, we design a simple yet effective noise-resistant inverse rendering scheme based on distance field deformation. Extensive experiments show that SuperCarver generates realistic and expressive surface details as depicted by specific texture appearances, making it a powerful tool for automatically upgrading massive outdated low-quality assets and shortening the iteration cycle of high-quality mesh production in practical applications.
• 摘要：高精度3D网格资产的传统生产工作流程需要专业建模者的手动雕刻过程繁琐而又费力的过程。近年来，在AI授权的3D内容创建中取得了显着进步。但是，尽管最新的最新技术已经能够从图像或文本提示中生成合理的结构和复杂的外观，但实际的网格表面通常是过度光滑的，并且缺乏几何细节。本文介绍了SuperCarver，SuperCarver是一个3D几何超分辨率的框架，该框架是为添加纹理一致的表面细节而量身定制的。从技术上讲，我们首先从多个观点将原始纹理网格渲染到图像域。为了实现几何细节的生成，我们开发了一个确定性的先前引导的正常扩散模型，该模型在经过精心策划的配对的低点和高聚正常渲染的数据集中进行了微调。为了通过潜在的不完美的正常地图预测优化网格结构，我们根据距离场变形设计了一种简单而有效的抗噪声呈现方案。广泛的实验表明，SuperCarver产生了由特定纹理外观所描绘的真实和表现力的表面细节，使其成为自动升级大量过时的低质量低质量资产并缩短实际应用中高质量网格生产的迭代周期的强大工具。

Title: Sparse Autoencoder as a Zero-Shot Classifier for Concept Erasing in Text-to-Image Diffusion Models

• Authors: Zhihua Tian, Sirun Nan, Ming Xu, Shengfang Zhai, Wenjie Qu, Jian Liu, Kui Ren, Ruoxi Jia, Jiaheng Zhang
• Subjects: cs.CV, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2503.09446
• Pdf URL: https://arxiv.org/pdf/2503.09446
• Copy Paste: [[2503.09446]] Sparse Autoencoder as a Zero-Shot Classifier for Concept Erasing in Text-to-Image Diffusion Models(https://arxiv.org/abs/2503.09446)
• Keywords: generation
• Abstract: Text-to-image (T2I) diffusion models have achieved remarkable progress in generating high-quality images but also raise people's concerns about generating harmful or misleading content. While extensive approaches have been proposed to erase unwanted concepts without requiring retraining from scratch, they inadvertently degrade performance on normal generation tasks. In this work, we propose Interpret then Deactivate (ItD), a novel framework to enable precise concept removal in T2I diffusion models while preserving overall performance. ItD first employs a sparse autoencoder (SAE) to interpret each concept as a combination of multiple features. By permanently deactivating the specific features associated with target concepts, we repurpose SAE as a zero-shot classifier that identifies whether the input prompt includes target concepts, allowing selective concept erasure in diffusion models. Moreover, we demonstrate that ItD can be easily extended to erase multiple concepts without requiring further training. Comprehensive experiments across celebrity identities, artistic styles, and explicit content demonstrate ItD's effectiveness in eliminating targeted concepts without interfering with normal concept generation. Additionally, ItD is also robust against adversarial prompts designed to circumvent content filters. Code is available at: this https URL.
• 摘要：文本对图像（T2I）扩散模型在产生高质量的图像方面取得了显着的进步，同时也引起了人们对产生有害或误导性内容的担忧。尽管已经提出了广泛的方法来消除不需要的概念而不需要从头开始重新培训，但它们会无意中降低了正常生成任务的绩效。在这项工作中，我们提出了解释然后停用（ITD），这是一个新颖的框架，可以在T2I扩散模型中进行精确的概念去除，同时保留整体性能。 ITD首先采用稀疏的自动编码器（SAE）将每个概念解释为多个功能的组合。通过永久停用与目标概念相关的特定特征，我们将SAE重新为零摄像分类器，该分类器识别输入提示是否包括目标概念，从而允许在扩散模型中选择性概念擦除。此外，我们证明可以轻松地扩展ITD以消除多个概念而无需进行进一步的培训。跨名人身份，艺术风格和明确内容的全面实验证明了ITD在消除有针对性概念的情况下的有效性，而不会干扰正常的概念生成。此外，ITD还针对旨在绕过内容过滤器的对抗提示。代码可用：此HTTPS URL。

Title: How Well Does Your Tabular Generator Learn the Structure of Tabular Data?

• Authors: Xiangjian Jiang, Nikola Simidjievski, Mateja Jamnik
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2503.09453
• Pdf URL: https://arxiv.org/pdf/2503.09453
• Copy Paste: [[2503.09453]] How Well Does Your Tabular Generator Learn the Structure of Tabular Data?(https://arxiv.org/abs/2503.09453)
• Keywords: generative
• Abstract: Heterogeneous tabular data poses unique challenges in generative modelling due to its fundamentally different underlying data structure compared to homogeneous modalities, such as images and text. Although previous research has sought to adapt the successes of generative modelling in homogeneous modalities to the tabular domain, defining an effective generator for tabular data remains an open problem. One major reason is that the evaluation criteria inherited from other modalities often fail to adequately assess whether tabular generative models effectively capture or utilise the unique structural information encoded in tabular data. In this paper, we carefully examine the limitations of the prevailing evaluation framework and introduce $\textbf{TabStruct}$, a novel evaluation benchmark that positions structural fidelity as a core evaluation dimension. Specifically, TabStruct evaluates the alignment of causal structures in real and synthetic data, providing a direct measure of how effectively tabular generative models learn the structure of tabular data. Through extensive experiments using generators from eight categories on seven datasets with expert-validated causal graphical structures, we show that structural fidelity offers a task-independent, domain-agnostic evaluation dimension. Our findings highlight the importance of tabular data structure and offer practical guidance for developing more effective and robust tabular generative models. Code is available at this https URL.
• 摘要：异质表格数据与均匀方式（例如图像和文本）相比，由于其根本不同的潜在数据结构，因此在生成建模中构成了独特的挑战。尽管先前的研究试图使均匀模式的生成建模的成功适应表格域，但为表格数据定义了有效的发电机仍然是一个空旷的问题。一个主要原因是，从其他方式继承的评估标准通常无法充分评估表格生成模型是否有效捕获或利用表格数据中编码的唯一结构信息。在本文中，我们仔细检查了普遍评估框架的局限性，并引入了$ \ textbf {tabStruct} $，这是一种新颖的评估基准，将结构保真度定位为核心评估维度。具体而言，TABSTRUCT评估实际和合成数据中因果结构的对准，从而直接衡量了如何有效表达表格生成模型学习表格数据的结构。通过使用具有专家验证的因果图形结构的七个数据集中的八个类别的发电机进行广泛的实验，我们表明结构保真度提供了独立于任务的，域名的评估维度。我们的发现突出了表格数据结构的重要性，并为开发更有效和强大的表格生成模型提供了实用的指导。代码可在此HTTPS URL上找到。

Title: SurgicalVLM-Agent: Towards an Interactive AI Co-Pilot for Pituitary Surgery

• Authors: Jiayuan Huang, Runlong He, Danyal Z. Khan, Evangelos Mazomenos, Danail Stoyanov, Hani J. Marcus, Matthew J. Clarkson, Mobarakol Islam
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09474
• Pdf URL: https://arxiv.org/pdf/2503.09474
• Copy Paste: [[2503.09474]] SurgicalVLM-Agent: Towards an Interactive AI Co-Pilot for Pituitary Surgery(https://arxiv.org/abs/2503.09474)
• Keywords: generation
• Abstract: Image-guided surgery demands adaptive, real-time decision support, yet static AI models struggle with structured task planning and providing interactive guidance. Large vision-language models (VLMs) offer a promising solution by enabling dynamic task planning and predictive decision support. We introduce SurgicalVLM-Agent, an AI co-pilot for image-guided pituitary surgery, capable of conversation, planning, and task execution. The agent dynamically processes surgeon queries and plans the tasks such as MRI tumor segmentation, endoscope anatomy segmentation, overlaying preoperative imaging with intraoperative views, instrument tracking, and surgical visual question answering (VQA). To enable structured task planning, we develop the PitAgent dataset, a surgical context-aware dataset covering segmentation, overlaying, instrument localization, tool tracking, tool-tissue interactions, phase identification, and surgical activity recognition. Additionally, we propose FFT-GaLore, a fast Fourier transform (FFT)-based gradient projection technique for efficient low-rank adaptation, optimizing fine-tuning for LLaMA 3.2 in surgical environments. We validate SurgicalVLM-Agent by assessing task planning and prompt generation on our PitAgent dataset and evaluating zero-shot VQA using a public pituitary dataset. Results demonstrate state-of-the-art performance in task planning and query interpretation, with highly semantically meaningful VQA responses, advancing AI-driven surgical assistance.
• 摘要：图像指导的手术需要适应性，实时决策支持，但静态AI模型在结构化任务计划中挣扎并提供互动指导。大型视觉模型（VLMS）通过实现动态任务计划和预测性决策支持提供了有希望的解决方案。我们介绍了SurgicalVLM-Engent，这是一种用于图像引导的垂体手术的AI副驾驶，能够对话，计划和执行任务。该代理动态处理外科医生的查询并计划诸如MRI肿瘤分割，内窥镜解剖学分割，具有术中视图，仪器跟踪和手术视觉问题答录（VQA）的术前成像（VQA）。为了启用结构化的任务计划，我们开发了Pitagent数据集，这是一个手术上下文感知的数据集，涵盖细分，覆盖，仪器定位，工具跟踪，工具组织交互，相位识别和手术活动识别。此外，我们提出了FFT-Galore，这是一种快速的傅立叶变换（FFT）基于基于有效的低级适应性的梯度投影技术，在手术环境中优化了Llama 3.2的微调。我们通过评估任务计划并迅速在我们的Pitagent数据集上生成并使用公共垂体数据集评估零射击VQA来验证手术vlm-agent。结果表明，在任务计划和查询解释中具有最先进的表现，并具有具有高度有意义的VQA响应，从而提高了AI驱动的手术援助。

Title: DAMM-Diffusion: Learning Divergence-Aware Multi-Modal Diffusion Model for Nanoparticles Distribution Prediction

• Authors: Junjie Zhou, Shouju Wang, Yuxia Tang, Qi Zhu, Daoqiang Zhang, Wei Shao
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2503.09491
• Pdf URL: https://arxiv.org/pdf/2503.09491
• Copy Paste: [[2503.09491]] DAMM-Diffusion: Learning Divergence-Aware Multi-Modal Diffusion Model for Nanoparticles Distribution Prediction(https://arxiv.org/abs/2503.09491)
• Keywords: generative
• Abstract: The prediction of nanoparticles (NPs) distribution is crucial for the diagnosis and treatment of tumors. Recent studies indicate that the heterogeneity of tumor microenvironment (TME) highly affects the distribution of NPs across tumors. Hence, it has become a research hotspot to generate the NPs distribution by the aid of multi-modal TME components. However, the distribution divergence among multi-modal TME components may cause side effects i.e., the best uni-modal model may outperform the joint generative model. To address the above issues, we propose a \textbf{D}ivergence-\textbf{A}ware \textbf{M}ulti-\textbf{M}odal \textbf{Diffusion} model (i.e., \textbf{DAMM-Diffusion}) to adaptively generate the prediction results from uni-modal and multi-modal branches in a unified network. In detail, the uni-modal branch is composed of the U-Net architecture while the multi-modal branch extends it by introducing two novel fusion modules i.e., Multi-Modal Fusion Module (MMFM) and Uncertainty-Aware Fusion Module (UAFM). Specifically, the MMFM is proposed to fuse features from multiple modalities, while the UAFM module is introduced to learn the uncertainty map for cross-attention computation. Following the individual prediction results from each branch, the Divergence-Aware Multi-Modal Predictor (DAMMP) module is proposed to assess the consistency of multi-modal data with the uncertainty map, which determines whether the final prediction results come from multi-modal or uni-modal predictions. We predict the NPs distribution given the TME components of tumor vessels and cell nuclei, and the experimental results show that DAMM-Diffusion can generate the distribution of NPs with higher accuracy than the comparing methods. Additional results on the multi-modal brain image synthesis task further validate the effectiveness of the proposed method.
• 摘要：纳米颗粒（NP）分布的预测对于诊断和治疗肿瘤至关重要。最近的研究表明，肿瘤微环境（TME）的异质性高度影响NP跨肿瘤的分布。因此，它已成为一种研究热点，可以借助多模式TME组件生成NPS分布。但是，多模式TME组件之间的分布差异可能会导致副作用，即，最佳的单模模型可能胜过关节生成模型。要解决上述问题，我们提出了a \ textbf {d} ivergence- \ textbf {a} ware \ textbf {m} ulti- \ textbf {m} odal \ textbf \ textbf {diffusion}在统一网络中。详细说明，单模式分支由U-NET架构组成，而多模式分支通过引入两个新型融合模块，即多模式融合模块（MMFM）和不确定性吸引融合模块（UAFM）来扩展它。具体而言，提出了MMFM以融合来自多种模式的功能，而引入了UAFM模块以学习跨科计算的不确定性图。遵循每个分支的单个预测结果，提出了差异感知的多模式预测变量（DAMMP）模块，以评估多模式数据与不确定性图的一致性，该数据确定最终预测结果是来自多模式还是单模式预测。我们预测鉴于肿瘤血管和细胞核的TME成分的NPS分布，实验结果表明，与比较方法相比，DAMM扩散可以产生具有更高准确性的NP的分布。多模式大脑图像合成任务的其他结果进一步验证了该方法的有效性。

Title: Robust Multimodal Survival Prediction with the Latent Differentiation Conditional Variational AutoEncoder

• Authors: Junjie Zhou, Jiao Tang, Yingli Zuo, Peng Wan, Daoqiang Zhang, Wei Shao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09496
• Pdf URL: https://arxiv.org/pdf/2503.09496
• Copy Paste: [[2503.09496]] Robust Multimodal Survival Prediction with the Latent Differentiation Conditional Variational AutoEncoder(https://arxiv.org/abs/2503.09496)
• Keywords: generative
• Abstract: The integrative analysis of histopathological images and genomic data has received increasing attention for survival prediction of human cancers. However, the existing studies always hold the assumption that full modalities are available. As a matter of fact, the cost for collecting genomic data is high, which sometimes makes genomic data unavailable in testing samples. A common way of tackling such incompleteness is to generate the genomic representations from the pathology images. Nevertheless, such strategy still faces the following two challenges: (1) The gigapixel whole slide images (WSIs) are huge and thus hard for representation. (2) It is difficult to generate the genomic embeddings with diverse function categories in a unified generative framework. To address the above challenges, we propose a Conditional Latent Differentiation Variational AutoEncoder (LD-CVAE) for robust multimodal survival prediction, even with missing genomic data. Specifically, a Variational Information Bottleneck Transformer (VIB-Trans) module is proposed to learn compressed pathological representations from the gigapixel WSIs. To generate different functional genomic features, we develop a novel Latent Differentiation Variational AutoEncoder (LD-VAE) to learn the common and specific posteriors for the genomic embeddings with diverse functions. Finally, we use the product-of-experts technique to integrate the genomic common posterior and image posterior for the joint latent distribution estimation in LD-CVAE. We test the effectiveness of our method on five different cancer datasets, and the experimental results demonstrate its superiority in both complete and missing modality scenarios.
• 摘要：组织病理学图像和基因组数据的综合分析已受到越来越多的关注人类癌症的生存预测。但是，现有的研究始终认为可以使用完整的方式。事实上，收集基因组数据的成本很高，这有时使基因组数据在测试样品中不可用。解决这种不完整的一种常见方法是从病理图像中产生基因组表征。然而，这种策略仍然面临以下两个挑战：（1）Gigapixel全幻灯片图像（WSIS）很大，因此很难代表。 （2）在统一生成框架中，很难生成具有不同功能类别的基因组嵌入。为了应对上述挑战，我们提出了一个有条件的潜在分化变异自动编码器（LD-CVAE），即使缺少基因组数据，也有稳健的多模式存活预测。具体而言，提出了变分信息瓶颈变压器（VIB-TRANS）模块以从Gigapixel WSIS学习压缩病理表示。为了产生不同的功能基因组特征，我们开发了一种新型的潜在分化变异自动编码器（LD-VAE），以学习具有不同功能的基因组嵌入的常见和特定后代。最后，我们使用专家专家技术来整合LD-CVAE联合潜在分布估计的基因组公共后部和图像后部。我们测试了方法对五个不同癌症数据集的有效性，并且实验结果证明了它在完整和缺失模态方面的优势。

Title: CM-Diff: A Single Generative Network for Bidirectional Cross-Modality Translation Diffusion Model Between Infrared and Visible Images

• Authors: Bin Hu, Chenqiang Gao, Shurui Liu, Junjie Guo, Fang Chen, Fangcen Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09514
• Pdf URL: https://arxiv.org/pdf/2503.09514
• Copy Paste: [[2503.09514]] CM-Diff: A Single Generative Network for Bidirectional Cross-Modality Translation Diffusion Model Between Infrared and Visible Images(https://arxiv.org/abs/2503.09514)
• Keywords: generative
• Abstract: The image translation method represents a crucial approach for mitigating information deficiencies in the infrared and visible modalities, while also facilitating the enhancement of modality-specific datasets. However, existing methods for infrared and visible image translation either achieve unidirectional modality translation or rely on cycle consistency for bidirectional modality translation, which may result in suboptimal performance. In this work, we present the cross-modality translation diffusion model (CM-Diff) for simultaneously modeling data distributions in both the infrared and visible modalities. We address this challenge by combining translation direction labels for guidance during training with cross-modality feature control. Specifically, we view the establishment of the mapping relationship between the two modalities as the process of learning data distributions and understanding modality differences, achieved through a novel Bidirectional Diffusion Training (BDT) strategy. Additionally, we propose a Statistical Constraint Inference (SCI) strategy to ensure the generated image closely adheres to the data distribution of the target modality. Experimental results demonstrate the superiority of our CM-Diff over state-of-the-art methods, highlighting its potential for generating dual-modality datasets.
• 摘要：图像翻译方法代表了一种至关重要的方法，用于减轻红外和可见方式中的信息缺陷，同时还促进了特定于模态特异性数据集的增强。但是，现有的红外和可见图像翻译方法要么实现单向模态翻译，要么依赖于双向模态翻译的循环一致性，这可能会导致次优性能。在这项工作中，我们介绍了用于同时对红外和可见模式中的数据分布进行建模的跨模式翻译扩散模型（CM-DIFF）。我们通过结合翻译方向标签来应对这一挑战，以便在训练期间与跨模式功能控制进行指导。具体而言，我们将两种方式之间的映射关系建立是学习数据分布和理解模态差异的过程，这是通过新型的双向扩散训练（BDT）策略实现的。此外，我们提出了一个统计约束推断（SCI）策略，以确保生成的图像紧密遵守目标模式的数据分布。实验结果证明了我们的CM-DIFF优于最先进的方法，突出了其生成双模式数据集的潜力。

Title: GenHPE: Generative Counterfactuals for 3D Human Pose Estimation with Radio Frequency Signals

• Authors: Shuokang Huang, Julie A. McCann
• Subjects: cs.CV, cs.AI, cs.MM, eess.SP
• Abstract URL: https://arxiv.org/abs/2503.09537
• Pdf URL: https://arxiv.org/pdf/2503.09537
• Copy Paste: [[2503.09537]] GenHPE: Generative Counterfactuals for 3D Human Pose Estimation with Radio Frequency Signals(https://arxiv.org/abs/2503.09537)
• Keywords: generative
• Abstract: Human pose estimation (HPE) detects the positions of human body joints for various applications. Compared to using cameras, HPE using radio frequency (RF) signals is non-intrusive and more robust to adverse conditions, exploiting the signal variations caused by human interference. However, existing studies focus on single-domain HPE confined by domain-specific confounders, which cannot generalize to new domains and result in diminished HPE performance. Specifically, the signal variations caused by different human body parts are entangled, containing subject-specific confounders. RF signals are also intertwined with environmental noise, involving environment-specific confounders. In this paper, we propose GenHPE, a 3D HPE approach that generates counterfactual RF signals to eliminate domain-specific confounders. GenHPE trains generative models conditioned on human skeleton labels, learning how human body parts and confounders interfere with RF signals. We manipulate skeleton labels (i.e., removing body parts) as counterfactual conditions for generative models to synthesize counterfactual RF signals. The differences between counterfactual signals approximately eliminate domain-specific confounders and regularize an encoder-decoder model to learn domain-independent representations. Such representations help GenHPE generalize to new subjects/environments for cross-domain 3D HPE. We evaluate GenHPE on three public datasets from WiFi, ultra-wideband, and millimeter wave. Experimental results show that GenHPE outperforms state-of-the-art methods and reduces estimation errors by up to 52.2mm for cross-subject HPE and 10.6mm for cross-environment HPE.
• 摘要：人姿势估计（HPE）检测到各种应用中人体关节的位置。与使用摄像头相比，使用射频（RF）信号的HPE是非侵入性的，并且对不利条件更强大，从而利用了人类干扰引起的信号变化。但是，现有研究的重点是限制于域特异性混杂因素的单域HPE，该HPE无法推广到新的领域并导致HPE性能降低。具体而言，由不同的人体部位引起的信号变化是纠缠的，包含特定于主体的混杂因素。 RF信号还与环境噪声交织在一起，涉及环境特定的混杂因子。在本文中，我们提出了GenHPE，这是一种3D HPE方法，该方法生成反事实RF信号以消除特定于域的混杂因子。 Genhpe训练以人骨骼标签为条件的生成模型，学习人体零件和混淆者如何干扰RF信号。我们操纵骨骼标签（即去除身体部位）作为生成模型的反事实条件，以合成反事实RF信号。反事实信号之间的差异大致消除了域特异性的混杂因素，并将编码器模型正规化以学习无独立的表示。这种表示有助于GenHPE推广到跨域3D HPE的新主题/环境。我们在WiFi，Ultra Wideband和毫米波的三个公共数据集上评估GenHPE。实验结果表明，GENHPE胜过最先进的方法，而跨主题HPE的估计误差最高为52.2mm，跨环境HPE的估计误差为10.6mm。

Title: Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models

• Authors: Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, Volodymyr Kuleshov
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.09573
• Pdf URL: https://arxiv.org/pdf/2503.09573
• Copy Paste: [[2503.09573]] Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models(https://arxiv.org/abs/2503.09573)
• Keywords: generation
• Abstract: Diffusion language models offer unique benefits over autoregressive models due to their potential for parallelized generation and controllability, yet they lag in likelihood modeling and are limited to fixed-length generation. In this work, we introduce a class of block diffusion language models that interpolate between discrete denoising diffusion and autoregressive models. Block diffusion overcomes key limitations of both approaches by supporting flexible-length generation and improving inference efficiency with KV caching and parallel token sampling. We propose a recipe for building effective block diffusion models that includes an efficient training algorithm, estimators of gradient variance, and data-driven noise schedules to minimize the variance. Block diffusion sets a new state-of-the-art performance among diffusion models on language modeling benchmarks and enables generation of arbitrary-length sequences. We provide the code, along with the model weights and blog post on the project page: this https URL
• 摘要：扩散语言模型由于其具有并行的生成和可控性的潜力而为自回归模型提供了独特的好处，但它们的可能性模型却滞后，并且仅限于固定长度的生成。在这项工作中，我们介绍了一类块扩散语言模型，这些模型在离散的denoising扩散和自回旋模型之间插值。通过支持柔性长度的产生并通过KV缓存和平行令牌采样来提高推理效率，克服了两种方法的关键局限性。我们提出了一种构建有效块扩散模型的配方，其中包括有效的训练算法，梯度差异的估计器和数据驱动的噪声时间表以最大程度地减少方差。块扩散在语言建模基准上的扩散模型之间设置了新的最新性能，并可以生成任意长度的序列。我们提供代码，以及项目页面上的模型权重和博客文章：此HTTPS URL

Title: Minimax Optimality of the Probability Flow ODE for Diffusion Models

• Authors: Changxiao Cai, Gen Li
• Subjects: cs.LG, cs.IT, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2503.09583
• Pdf URL: https://arxiv.org/pdf/2503.09583
• Copy Paste: [[2503.09583]] Minimax Optimality of the Probability Flow ODE for Diffusion Models(https://arxiv.org/abs/2503.09583)
• Keywords: generative
• Abstract: Score-based diffusion models have become a foundational paradigm for modern generative modeling, demonstrating exceptional capability in generating samples from complex high-dimensional distributions. Despite the dominant adoption of probability flow ODE-based samplers in practice due to their superior sampling efficiency and precision, rigorous statistical guarantees for these methods have remained elusive in the literature. This work develops the first end-to-end theoretical framework for deterministic ODE-based samplers that establishes near-minimax optimal guarantees under mild assumptions on target data distributions. Specifically, focusing on subgaussian distributions with $\beta$-Hölder smooth densities for $\beta\leq 2$, we propose a smooth regularized score estimator that simultaneously controls both the $L^2$ score error and the associated mean Jacobian error. Leveraging this estimator within a refined convergence analysis of the ODE-based sampling process, we demonstrate that the resulting sampler achieves the minimax rate in total variation distance, modulo logarithmic factors. Notably, our theory comprehensively accounts for all sources of error in the sampling process and does not require strong structural conditions such as density lower bounds or Lipschitz/smooth scores on target distributions, thereby covering a broad range of practical data distributions.
• 摘要：基于得分的扩散模型已成为现代生成建模的基础范式，证明了从复杂的高维分布生成样品方面的非凡能力。尽管由于其出色的采样效率和精度，在实践中主要采用了概率流动的采样器，但这些方法的严格统计保证在文献中仍然难以捉摸。这项工作开发了基于确定性ODE的采样器的第一个端到端的理论框架，该框架在目标数据分布的轻度假设下建立了接近米诺克斯的最佳保证。具体而言，专注于$ \ beta $-Hölder光滑密度，$ \ beta \ leq 2 $，我们提出了一个平稳的正规分数估计器，同时控制$ l^2 $得分错误和相关的平均雅各布式错误。利用该估计器在基于ODE的采样过程的精制收敛分析中，我们证明了所得的采样器在总变化距离（Modulo Googarithmic因子）中实现了最小值。值得注意的是，我们的理论全面说明了采样过程中的所有错误源，并且不需要强大的结构条件，例如密度下限或Lipschitz/Lipschitz/平滑分数在目标分布上，从而涵盖了广泛的实用数据分布。

Title: Parsing the Language of Expression: Enhancing Symbolic Regression with Domain-Aware Symbolic Priors

• Authors: Sikai Huang, Yixin Berry Wen, Tara Adusumilli, Kusum Choudhary, Haizhao Yang
• Subjects: cs.LG, cs.SC
• Abstract URL: https://arxiv.org/abs/2503.09592
• Pdf URL: https://arxiv.org/pdf/2503.09592
• Copy Paste: [[2503.09592]] Parsing the Language of Expression: Enhancing Symbolic Regression with Domain-Aware Symbolic Priors(https://arxiv.org/abs/2503.09592)
• Keywords: generation
• Abstract: Symbolic regression is essential for deriving interpretable expressions that elucidate complex phenomena by exposing the underlying mathematical and physical relationships in data. In this paper, we present an advanced symbolic regression method that integrates symbol priors from diverse scientific domains - including physics, biology, chemistry, and engineering - into the regression process. By systematically analyzing domain-specific expressions, we derive probability distributions of symbols to guide expression generation. We propose novel tree-structured recurrent neural networks (RNNs) that leverage these symbol priors, enabling domain knowledge to steer the learning process. Additionally, we introduce a hierarchical tree structure for representing expressions, where unary and binary operators are organized to facilitate more efficient learning. To further accelerate training, we compile characteristic expression blocks from each domain and include them in the operator dictionary, providing relevant building blocks. Experimental results demonstrate that leveraging symbol priors significantly enhances the performance of symbolic regression, resulting in faster convergence and higher accuracy.
• 摘要：符号回归对于通过暴露数据中的基本数学和物理关系来阐明复杂现象的可解释表达至关重要。在本文中，我们提出了一种先进的符号回归方法，该方法将来自各种科学领域的符号先验（包括物理，生物学，化学和工程学）整合到回归过程中。通过系统地分析域特异性表达式，我们得出符号的概率分布以指导表达生成。我们提出了利用这些符号先验的新型树结构的复发性神经网络（RNN），从而使领域知识可以引导学习过程。此外，我们引入了一种层次树结构，用于表示表达式，其中组织了一级和二进制操作员以促进更有效的学习。为了进一步加速培训，我们将每个域的特征表达式块编译到操作员词典中，提供相关的构件。实验结果表明，利用符号先验会显着提高符号回归的性能，从而更快地收敛和更高的精度。

Title: PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Drop

• Authors: Chenyu Li, Oscar Michel, Xichen Pan, Sainan Liu, Mike Roberts, Saining Xie
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09595
• Pdf URL: https://arxiv.org/pdf/2503.09595
• Copy Paste: [[2503.09595]] PISA Experiments: Exploring Physics Post-Training for Video Diffusion Models by Watching Stuff Drop(https://arxiv.org/abs/2503.09595)
• Keywords: generation, generative
• Abstract: Large-scale pre-trained video generation models excel in content creation but are not reliable as physically accurate world simulators out of the box. This work studies the process of post-training these models for accurate world modeling through the lens of the simple, yet fundamental, physics task of modeling object freefall. We show state-of-the-art video generation models struggle with this basic task, despite their visually impressive outputs. To remedy this problem, we find that fine-tuning on a relatively small amount of simulated videos is effective in inducing the dropping behavior in the model, and we can further improve results through a novel reward modeling procedure we introduce. Our study also reveals key limitations of post-training in generalization and distribution modeling. Additionally, we release a benchmark for this task that may serve as a useful diagnostic tool for tracking physical accuracy in large-scale video generative model development.
• 摘要：大规模的预训练的视频生成模型在内容创建中表现出色，但由于物理上精确的世界模拟器并不可靠。这项工作研究了训练后的这些模型的过程，从而通过简单但基本的物理学任务进行建模对象自由落体的镜头进行准确的世界建模。我们展示了最先进的视频生成模型在这项基本任务上遇到了令人印象深刻的输出。为了解决这个问题，我们发现对相对少量的模拟视频进行微调可以有效诱导模型中的脱落行为，并且我们可以通过引入的新型奖励建模程序进一步改善结果。我们的研究还揭示了在概括和分布建模中训练后的关键局限性。此外，我们为该任务发布了一个基准，该基准可能是在大规模视频生成模型开发中跟踪身体准确性的有用诊断工具。

Title: RewardSDS: Aligning Score Distillation via Reward-Weighted Sampling

• Authors: Itay Chachy, Guy Yariv, Sagie Benaim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.09601
• Pdf URL: https://arxiv.org/pdf/2503.09601
• Copy Paste: [[2503.09601]] RewardSDS: Aligning Score Distillation via Reward-Weighted Sampling(https://arxiv.org/abs/2503.09601)
• Keywords: generation
• Abstract: Score Distillation Sampling (SDS) has emerged as an effective technique for leveraging 2D diffusion priors for tasks such as text-to-3D generation. While powerful, SDS struggles with achieving fine-grained alignment to user intent. To overcome this, we introduce RewardSDS, a novel approach that weights noise samples based on alignment scores from a reward model, producing a weighted SDS loss. This loss prioritizes gradients from noise samples that yield aligned high-reward output. Our approach is broadly applicable and can extend SDS-based methods. In particular, we demonstrate its applicability to Variational Score Distillation (VSD) by introducing RewardVSD. We evaluate RewardSDS and RewardVSD on text-to-image, 2D editing, and text-to-3D generation tasks, showing significant improvements over SDS and VSD on a diverse set of metrics measuring generation quality and alignment to desired reward models, enabling state-of-the-art performance. Project page is available at https://itaychachy. this http URL.
• 摘要：得分蒸馏采样（SDS）已成为利用2D扩散先验的有效技术，例如文本到3D生成。尽管有力，SD却努力与用户意图实现细粒度的一致性。为了克服这一点，我们引入了奖励，这是一种新型方法，可以根据奖励模型的对齐分数加权噪声样本，从而产生加权SDS损失。这种损失优先于产生高回报输出的噪声样本中的梯度。我们的方法广泛适用，可以扩展基于SDS的方法。特别是，我们通过引入RewardVSD来证明其适用于变异得分蒸馏（VSD）。我们在文本到图像，2D编辑以及文本到3D代任务上评估奖励和奖励VSD，显示了对SDS和VSD的显着改进，以衡量所需的奖励模型，以衡量生成质量和对齐方式，从而实现先进的绩效。项目页面可从https：// itaychachy获得。此HTTP URL。