2025-03-07

Title: Positive-Unlabeled Diffusion Models for Preventing Sensitive Data Generation

• Authors: Hiroshi Takahashi, Tomoharu Iwata, Atsutoshi Kumagai, Yuuki Yamanaka, Tomoya Yamashita
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2503.03789
• Pdf URL: https://arxiv.org/pdf/2503.03789
• Copy Paste: [[2503.03789]] Positive-Unlabeled Diffusion Models for Preventing Sensitive Data Generation(https://arxiv.org/abs/2503.03789)
• Keywords: generation, generative
• Abstract: Diffusion models are powerful generative models but often generate sensitive data that are unwanted by users, mainly because the unlabeled training data frequently contain such sensitive data. Since labeling all sensitive data in the large-scale unlabeled training data is impractical, we address this problem by using a small amount of labeled sensitive data. In this paper, we propose positive-unlabeled diffusion models, which prevent the generation of sensitive data using unlabeled and sensitive data. Our approach can approximate the evidence lower bound (ELBO) for normal (negative) data using only unlabeled and sensitive (positive) data. Therefore, even without labeled normal data, we can maximize the ELBO for normal data and minimize it for labeled sensitive data, ensuring the generation of only normal data. Through experiments across various datasets and settings, we demonstrated that our approach can prevent the generation of sensitive images without compromising image quality.
• 摘要：扩散模型是强大的生成模型，但通常会生成用户不需要的敏感数据，这主要是因为未标记的训练数据经常包含此类敏感数据。由于在大规模未标记的培训数据中标记所有敏感数据是不切实际的，因此我们通过使用少量标记的敏感数据来解决此问题。在本文中，我们提出了阳性未标记的扩散模型，该模型可以防止使用未标记和敏感数据产生敏感数据。我们的方法仅使用未标记和敏感（正）数据近似正常数据的证据（ELBO）。因此，即使没有标记的正常数据，我们也可以最大化ELBO以获得正常数据，并将其最小化以获得标记的敏感数据，从而确保仅生成正常数据。通过在各种数据集和设置上进行的实验，我们证明了我们的方法可以防止敏感图像的产生而不会损害图像质量。

Title: Rebalanced Multimodal Learning with Data-aware Unimodal Sampling

• Authors: Qingyuan Jiang, Zhouyang Chi, Xiao Ma, Qirong Mao, Yang Yang, Jinhui Tang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.03792
• Pdf URL: https://arxiv.org/pdf/2503.03792
• Copy Paste: [[2503.03792]] Rebalanced Multimodal Learning with Data-aware Unimodal Sampling(https://arxiv.org/abs/2503.03792)
• Keywords: generation
• Abstract: To address the modality learning degeneration caused by modality imbalance, existing multimodal learning~(MML) approaches primarily attempt to balance the optimization process of each modality from the perspective of model learning. However, almost all existing methods ignore the modality imbalance caused by unimodal data sampling, i.e., equal unimodal data sampling often results in discrepancies in informational content, leading to modality imbalance. Therefore, in this paper, we propose a novel MML approach called \underline{D}ata-aware \underline{U}nimodal \underline{S}ampling~(\method), which aims to dynamically alleviate the modality imbalance caused by sampling. Specifically, we first propose a novel cumulative modality discrepancy to monitor the multimodal learning process. Based on the learning status, we propose a heuristic and a reinforcement learning~(RL)-based data-aware unimodal sampling approaches to adaptively determine the quantity of sampled data at each iteration, thus alleviating the modality imbalance from the perspective of sampling. Meanwhile, our method can be seamlessly incorporated into almost all existing multimodal learning approaches as a plugin. Experiments demonstrate that \method~can achieve the best performance by comparing with diverse state-of-the-art~(SOTA) baselines.
• 摘要：为了解决模式失衡引起的模态学习变性，现有的多模式学习〜（MML）方法主要试图从模型学习的角度平衡每个方式的优化过程。但是，几乎所有现有的方法都忽略了由单峰数据采样引起的模式失衡，即相等的单峰数据采样通常会导致信息内容差异，从而导致模态不平衡。因此，在本文中，我们提出了一种新型的MML方法，称为\下列{d} ata-ata-ata-ata-aware \ usewandline {u} nimodal \ nimodal \ usewissline {s}放大〜（\ method），旨在动态减轻由抽样引起的方式失衡。具体而言，我们首先提出了一种新颖的累积方式差异来监测多模式学习过程。基于学习状况，我们提出了一种启发式和增强学习〜（RL）基于数据感知的单峰采样方法，以适应每次迭代时的采样数据的数量，从而减轻了采样的角度的模态失衡。同时，我们的方法可以无缝地纳入几乎所有现有的多模式学习方法中。实验表明，\方法可以通过与各种最先进的（SOTA）基准进行比较来实现最佳性能。

Title: Neural Descriptors: Self-Supervised Learning of Robust Local Surface Descriptors Using Polynomial Patches

• Authors: Gal Yona, Roy Velich, Ron Kimmel, Ehud Rivlin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.03907
• Pdf URL: https://arxiv.org/pdf/2503.03907
• Copy Paste: [[2503.03907]] Neural Descriptors: Self-Supervised Learning of Robust Local Surface Descriptors Using Polynomial Patches(https://arxiv.org/abs/2503.03907)
• Keywords: generation
• Abstract: Classical shape descriptors such as Heat Kernel Signature (HKS), Wave Kernel Signature (WKS), and Signature of Histograms of OrienTations (SHOT), while widely used in shape analysis, exhibit sensitivity to mesh connectivity, sampling patterns, and topological noise. While differential geometry offers a promising alternative through its theory of differential invariants, which are theoretically guaranteed to be robust shape descriptors, the computation of these invariants on discrete meshes often leads to unstable numerical approximations, limiting their practical utility. We present a self-supervised learning approach for extracting geometric features from 3D surfaces. Our method combines synthetic data generation with a neural architecture designed to learn sampling-invariant features. By integrating our features into existing shape correspondence frameworks, we demonstrate improved performance on standard benchmarks including FAUST, SCAPE, TOPKIDS, and SHREC'16, showing particular robustness to topological noise and partial shapes.
• 摘要：经典形状的描述符，例如热内核签名（HKS），波核签名（WKS）和方向直方图（SHOT）的签名，尽管广泛用于形状分析，但对网格连接，采样模式和拓扑噪声表现出敏感性。尽管差异几何形状通过其差异不变性理论提供了一种有希望的替代方法，从理论上讲，这些理论可以保证是强大的形状描述符，但这些不变的分离网格上的计算通常会导致不稳定的数值近似值，从而限制了其实际效用。我们提出了一种自我监督的学习方法，用于从3D表面提取几何特征。我们的方法将合成数据的生成与旨在学习采样不变特征的神经结构相结合。通过将我们的功能集成到现有的形状通信框架中，我们在包括浮士德，Scape，Topkids和Sherec'16在内的标准基准上的性能提高了，对拓扑噪声和部分形状表现出了特殊的鲁棒性。

Title: Personalized Federated Fine-tuning for Heterogeneous Data: An Automatic Rank Learning Approach via Two-Level LoRA

• Authors: Jie Hao, Yuman Wu, Ali Payani, Myungjin Lee, Mingrui Liu
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2503.03920
• Pdf URL: https://arxiv.org/pdf/2503.03920
• Copy Paste: [[2503.03920]] Personalized Federated Fine-tuning for Heterogeneous Data: An Automatic Rank Learning Approach via Two-Level LoRA(https://arxiv.org/abs/2503.03920)
• Keywords: generation
• Abstract: We study the task of personalized federated fine-tuning with heterogeneous data in the context of language models, where clients collaboratively fine-tune a language model (e.g., BERT, GPT) without sharing their local data, achieving personalization simultaneously. While recent efforts have applied parameter-efficient fine-tuning techniques like low-rank adaptation (LoRA) in federated settings, they typically use single or multiple independent low-rank adapters with predefined maximal and minimal ranks, which may not be optimal for diverse data sources over clients. To address this issue, we propose PF2LoRA, a new personalized federated fine-tuning algorithm built on a novel \emph{automatic rank learning approach via two-level LoRA}. Given the pretrained language model whose weight is frozen, our algorithm aims to learn two levels of adaptation simultaneously: the first level aims to learn a common adapter for all clients, while the second level fosters individual client personalization. A key advantage of PF2LoRA is its ability to adaptively determine a suitable rank based on an individual client's data, rather than relying on a predefined rank that is agnostic to data heterogeneity. We present a synthetic example that highlights how PF2LoRA automatically learns the ground-truth rank for each client, tailoring the adaptation to match the properties of their individual data. Notably, this approach introduces minimal additional memory overhead, as the second-level adaptation comprises a small number of parameters compared to the first level. Our experiments on natural language understanding and generation tasks demonstrate that PF2LoRA significantly outperforms existing federated fine-tuning methods.
• 摘要：我们在语言模型的背景下研究了通过异质数据进行个性化联合微调的任务，在这种情况下，客户在不共享其本地数据并同时实现个性化的情况下，协同调整语言模型（例如Bert，GPT）。尽管最近的努力应用了参数效率高的微调技术，例如在联邦设置中的低排名适应（LORA），但他们通常使用具有预定义最大和最小排名的单个或多个独立的低级适配器，这可能不是客户多种数据源的最佳选择。为了解决这个问题，我们提出了PF2LORA，这是一种新的个性化联合微调算法，建立在小说\ emph {自动等级学习方法中，这是通过两级Lora}。鉴于重量被冻结的语言模型，我们的算法旨在同时学习两个级别的适应性：第一级旨在为所有客户学习一个共同的适配器，而第二级级别则促进了个人客户个性化。 PF2LORA的关键优势在于它的能力基于单个客户的数据自适应确定合适的等级，而不是依赖于不可知的数据异质性的预定级等级。我们提出了一个合成的示例，该示例突出显示了PF2LORA如何自动学习每个客户的地面等级，从而调整适应以匹配其单个数据的属性。值得注意的是，这种方法引入了最小的附加内存开销，因为第二级适应性与第一级相比包含少量参数。我们对自然语言理解和发电任务的实验表明，PF2LORA显着胜过现有的联合微调方法。

Title: GuardDoor: Safeguarding Against Malicious Diffusion Editing via Protective Backdoors

• Authors: Yaopei Zeng, Yuanpu Cao, Lu Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.03944
• Pdf URL: https://arxiv.org/pdf/2503.03944
• Copy Paste: [[2503.03944]] GuardDoor: Safeguarding Against Malicious Diffusion Editing via Protective Backdoors(https://arxiv.org/abs/2503.03944)
• Keywords: generative
• Abstract: The growing accessibility of diffusion models has revolutionized image editing but also raised significant concerns about unauthorized modifications, such as misinformation and plagiarism. Existing countermeasures largely rely on adversarial perturbations designed to disrupt diffusion model outputs. However, these approaches are found to be easily neutralized by simple image preprocessing techniques, such as compression and noise addition. To address this limitation, we propose GuardDoor, a novel and robust protection mechanism that fosters collaboration between image owners and model providers. Specifically, the model provider participating in the mechanism fine-tunes the image encoder to embed a protective backdoor, allowing image owners to request the attachment of imperceptible triggers to their images. When unauthorized users attempt to edit these protected images with this diffusion model, the model produces meaningless outputs, reducing the risk of malicious image editing. Our method demonstrates enhanced robustness against image preprocessing operations and is scalable for large-scale deployment. This work underscores the potential of cooperative frameworks between model providers and image owners to safeguard digital content in the era of generative AI.
• 摘要：扩散模型的不断增长彻底改变了图像编辑，但也引起了人们对未经授权修改的重大关注，例如误导和窃。现有的对策在很大程度上依赖于旨在破坏扩散模型输出的对抗扰动。但是，发现这些方法很容易通过简单的图像预处理技术（例如压缩和噪声）进行中和。为了解决这一限制，我们提出了Guarddoor，这是一种新颖而强大的保护机制，促进了图像所有者和模型提供者之间的协作。具体而言，参与机制的模型提供商微调图像编码器嵌入了保护性后门，从而使图像所有者可以要求将不可察觉的触发器附加到其图像上。当未经授权的用户尝试通过此扩散模型编辑这些受保护的图像时，该模型会产生毫无意义的输出，从而降低了恶意图像编辑的风险。我们的方法证明了针对图像预处理操作的鲁棒性增强，并且对于大规模部署而言是可扩展的。这项工作强调了模型提供商和图像所有者之间合作框架的潜力，以保护生成AI时代的数字内容。

Title: Generative Learning of Densities on Manifolds

• Authors: Dimitris G. Giovanis, Ellis Crabtree, Roger G. Ghanem, Ioannis G. kevrekidis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2503.03963
• Pdf URL: https://arxiv.org/pdf/2503.03963
• Copy Paste: [[2503.03963]] Generative Learning of Densities on Manifolds(https://arxiv.org/abs/2503.03963)
• Keywords: generative
• Abstract: A generative modeling framework is proposed that combines diffusion models and manifold learning to efficiently sample data densities on manifolds. The approach utilizes Diffusion Maps to uncover possible low-dimensional underlying (latent) spaces in the high-dimensional data (ambient) space. Two approaches for sampling from the latent data density are described. The first is a score-based diffusion model, which is trained to map a standard normal distribution to the latent data distribution using a neural network. The second one involves solving an Itô stochastic differential equation in the latent space. Additional realizations of the data are generated by lifting the samples back to the ambient space using Double Diffusion Maps, a recently introduced technique typically employed in studying dynamical system reduction; here the focus lies in sampling densities rather than system dynamics. The proposed approaches enable sampling high dimensional data densities restricted to low-dimensional, a priori unknown manifolds. The efficacy of the proposed framework is demonstrated through a benchmark problem and a material with multiscale structure.
• 摘要：提出了一个生成建模框架，该框架结合了扩散模型和流形学习，以有效地在多种流形上采样数据密度。该方法利用扩散图在高维数据（环境）空间中发现可能的低维基（潜在）空间。描述了从潜在数据密度采样的两种方法。第一个是基于分数的扩散模型，该模型经过训练，可以使用神经网络将标准正态分布映射到潜在数据分布。第二个涉及在潜在空间中求解ITô随机微分方程。通过使用双扩散图将样品提升回到环境空间来生成数据的其他实现，这是一种通常用于研究动态系统减少的技术；这里的重点在于采样密度而不是系统动力学。所提出的方法使得可以采样高维数据密度，仅限于低维（先验未知的歧管）。通过基准问题和具有多尺度结构的材料来证明所提出框架的功效。

Title: All-atom Diffusion Transformers: Unified generative modelling of molecules and materials

• Authors: Chaitanya K. Joshi, Xiang Fu, Yi-Lun Liao, Vahe Gharakhanyan, Benjamin Kurt Miller, Anuroop Sriram, Zachary W. Ulissi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.03965
• Pdf URL: https://arxiv.org/pdf/2503.03965
• Copy Paste: [[2503.03965]] All-atom Diffusion Transformers: Unified generative modelling of molecules and materials(https://arxiv.org/abs/2503.03965)
• Keywords: generative
• Abstract: Diffusion models are the standard toolkit for generative modelling of 3D atomic systems. However, for different types of atomic systems - such as molecules and materials - the generative processes are usually highly specific to the target system despite the underlying physics being the same. We introduce the All-atom Diffusion Transformer (ADiT), a unified latent diffusion framework for jointly generating both periodic materials and non-periodic molecular systems using the same model: (1) An autoencoder maps a unified, all-atom representations of molecules and materials to a shared latent embedding space; and (2) A diffusion model is trained to generate new latent embeddings that the autoencoder can decode to sample new molecules or materials. Experiments on QM9 and MP20 datasets demonstrate that jointly trained ADiT generates realistic and valid molecules as well as materials, exceeding state-of-the-art results from molecule and crystal-specific models. ADiT uses standard Transformers for both the autoencoder and diffusion model, resulting in significant speedups during training and inference compared to equivariant diffusion models. Scaling ADiT up to half a billion parameters predictably improves performance, representing a step towards broadly generalizable foundation models for generative chemistry. Open source code: this https URL
• 摘要：扩散模型是用于3D原子系统生成建模的标准工具包。但是，对于不同类型的原子系统（例如分子和材料），尽管潜在的物理学相同，但生成过程通常对目标系统高度特异。我们使用相同的模型介绍了全原子扩散变压器（ADIT），该统一的潜在扩散框架，用于共同生成周期性材料和非周期性分子系统：（1）自动编码器将分子和材料的统一，全部原子化的分子和材料映射到共享潜在的潜在胚胎嵌入空间； （2）训练了一个扩散模型，以生成新的潜在嵌入，自动编码器可以解码以采样新分子或材料。 QM9和MP20数据集的实验表明，经过训练的ADIT会产生逼真的和有效的分子以及材料，超过了分子和晶体特异性模型的最先进的结果。 ADIT对自动编码器和扩散模型都使用标准变压器，与等效性扩散模型相比，在训练和推理过程中导致了显着的加速。可以预见的是，扩展最高可达的参数可提高性能，这代表了迈向广泛概括的生成化学基础模型的一步。开源代码：此HTTPS URL

Title: TextDoctor: Unified Document Image Inpainting via Patch Pyramid Diffusion Models

• Authors: Wanglong Lu, Lingming Su, Jingjing Zheng, Vinícius Veloso de Melo, Farzaneh Shoeleh, John Hawkin, Terrence Tricco, Hanli Zhao, Xianta Jiang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.04021
• Pdf URL: https://arxiv.org/pdf/2503.04021
• Copy Paste: [[2503.04021]] TextDoctor: Unified Document Image Inpainting via Patch Pyramid Diffusion Models(https://arxiv.org/abs/2503.04021)
• Keywords: restoration
• Abstract: Digital versions of real-world text documents often suffer from issues like environmental corrosion of the original document, low-quality scanning, or human interference. Existing document restoration and inpainting methods typically struggle with generalizing to unseen document styles and handling high-resolution images. To address these challenges, we introduce TextDoctor, a novel unified document image inpainting method. Inspired by human reading behavior, TextDoctor restores fundamental text elements from patches and then applies diffusion models to entire document images instead of training models on specific document types. To handle varying text sizes and avoid out-of-memory issues, common in high-resolution documents, we propose using structure pyramid prediction and patch pyramid diffusion models. These techniques leverage multiscale inputs and pyramid patches to enhance the quality of inpainting both globally and locally. Extensive qualitative and quantitative experiments on seven public datasets validated that TextDoctor outperforms state-of-the-art methods in restoring various types of high-resolution document images.
• 摘要：现实世界中文本文档的数字版本通常会遇到原始文档的环境腐蚀，低质量扫描或人类干扰等问题。现有的文档修复和介绍方法通常在概括地看不见的文档样式和处理高分辨率图像方面努力。为了应对这些挑战，我们介绍了TextDoctor，这是一种新颖的统一文档图像介入方法。受到人类阅读行为的启发，TextDoctor恢复了补丁的基本文本元素，然后将扩散模型应用于整个文档图像，而不是对特定文档类型的培训模型。为了处理不同的文本大小并避免在高分辨率文档中常见的有内存的问题，我们建议使用结构金字塔预测和贴片金字塔扩散模型。这些技术利用多尺度输入和金字塔贴片来提高全球和本地覆盖的质量。在七个公共数据集上进行的广泛的定性和定量实验验证了TextDoctor在还原各种类型的高分辨率文档图像方面优于最先进的方法。

Title: Self-Supervised Large Scale Point Cloud Completion for Archaeological Site Restoration

• Authors: Aocheng Li, James R. Zimmer-Dauphinee, Rajesh Kalyanam, Ian Lindsay, Parker VanValkenburgh, Steven Wernke, Daniel Aliaga
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.04030
• Pdf URL: https://arxiv.org/pdf/2503.04030
• Copy Paste: [[2503.04030]] Self-Supervised Large Scale Point Cloud Completion for Archaeological Site Restoration(https://arxiv.org/abs/2503.04030)
• Keywords: restoration
• Abstract: Point cloud completion helps restore partial incomplete point clouds suffering occlusions. Current self-supervised methods fail to give high fidelity completion for large objects with missing surfaces and unbalanced distribution of available points. In this paper, we present a novel method for restoring large-scale point clouds with limited and imbalanced ground-truth. Using rough boundary annotations for a region of interest, we project the original point clouds into a multiple-center-of-projection (MCOP) image, where fragments are projected to images of 5 channels (RGB, depth, and rotation). Completion of the original point cloud is reduced to inpainting the missing pixels in the MCOP images. Due to lack of complete structures and an unbalanced distribution of existing parts, we develop a self-supervised scheme which learns to infill the MCOP image with points resembling existing "complete" patches. Special losses are applied to further enhance the regularity and consistency of completed MCOP images, which is mapped back to 3D to form final restoration. Extensive experiments demonstrate the superiority of our method in completing 600+ incomplete and unbalanced archaeological structures in Peru.
• 摘要：点云完成有助于恢复部分不完整的点云遭受遮挡。当前的自我监督方法无法为缺少表面和可用点不平衡分布的大对象提供高保真度完成。在本文中，我们提出了一种新颖的方法，用于恢复具有有限和不平衡地面真相的大规模点云。使用粗糙的边界注释对感兴趣的区域，我们将原始点云投影到多中心的预测（MCOP）图像中，其中片段被投影到5个通道的图像（RGB，深度和旋转）。原始点云的完成将减少为介绍MCOP图像中缺少的像素。由于缺乏完整的结构和现有零件的不平衡分布，我们开发了一种自我保护的方案，该方案学会用类似于现有的“完整”补丁的点填充MCOP图像。采用特殊损失来进一步增强完成的MCOP图像的规律性和一致性，该图像映射回3D以形成最终的修复。广泛的实验证明了我们方法在秘鲁完成600多个不完整和不平衡的考古结构方面的优越性。

Title: GaussianGraph: 3D Gaussian-based Scene Graph Generation for Open-world Scene Understanding

• Authors: Xihan Wang, Dianyi Yang, Yu Gao, Yufeng Yue, Yi Yang, Mengyin Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.04034
• Pdf URL: https://arxiv.org/pdf/2503.04034
• Copy Paste: [[2503.04034]] GaussianGraph: 3D Gaussian-based Scene Graph Generation for Open-world Scene Understanding(https://arxiv.org/abs/2503.04034)
• Keywords: generation
• Abstract: Recent advancements in 3D Gaussian Splatting(3DGS) have significantly improved semantic scene understanding, enabling natural language queries to localize objects within a scene. However, existing methods primarily focus on embedding compressed CLIP features to 3D Gaussians, suffering from low object segmentation accuracy and lack spatial reasoning capabilities. To address these limitations, we propose GaussianGraph, a novel framework that enhances 3DGS-based scene understanding by integrating adaptive semantic clustering and scene graph generation. We introduce a "Control-Follow" clustering strategy, which dynamically adapts to scene scale and feature distribution, avoiding feature compression and significantly improving segmentation accuracy. Additionally, we enrich scene representation by integrating object attributes and spatial relations extracted from 2D foundation models. To address inaccuracies in spatial relationships, we propose 3D correction modules that filter implausible relations through spatial consistency verification, ensuring reliable scene graph construction. Extensive experiments on three datasets demonstrate that GaussianGraph outperforms state-of-the-art methods in both semantic segmentation and object grounding tasks, providing a robust solution for complex scene understanding and interaction.
• 摘要：3D高斯分裂（3DG）的最新进展已显着改善了语义场景的理解，从而使自然语言查询能够在场景中定位对象。但是，现有的方法主要集中于将压缩夹的特征嵌入到3D高斯，患有低对象分割精度和缺乏空间推理能力。为了解决这些局限性，我们提出了高斯格拉普（Gaussiangraph），这是一个新颖的框架，通过整合自适应语义聚类和场景图生成来增强基于3DGS的场景理解。我们引入了“控制遵循”聚类策略，该策略会动态适应场景尺度和特征分布，避免了特征压缩并显着提高了分割精度。此外，我们通过整合从2D基础模型中提取的对象属性和空间关系来丰富场景表示。为了解决空间关系中的不准确性，我们提出了3D校正模块，该模块通过空间一致性验证过滤不可用的关系，从而确保了可靠的场景图构造。在三个数据集上进行的广泛实验表明，高斯图在语义分割和对象接地任务中的最先进方法优于最先进的方法，为复杂的场景理解和交互提供了强大的解决方案。

Title: Neural Network Surrogate Model for Junction Temperature and Hotspot Position in $3$D Multi-Layer High Bandwidth Memory (HBM) Chiplets under Varying Thermal Conditions

• Authors: Chengxin Zhang, Yujie Liu, Quan Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2503.04049
• Pdf URL: https://arxiv.org/pdf/2503.04049
• Copy Paste: [[2503.04049]] Neural Network Surrogate Model for Junction Temperature and Hotspot Position in $3$D Multi-Layer High Bandwidth Memory (HBM) Chiplets under Varying Thermal Conditions(https://arxiv.org/abs/2503.04049)
• Keywords: generation
• Abstract: As the demand for computational power increases, high-bandwidth memory (HBM) has become a critical technology for next-generation computing systems. However, the widespread adoption of HBM presents significant thermal management challenges, particularly in multilayer through-silicon-via (TSV) stacked structures under varying thermal conditions, where accurate prediction of junction temperature and hotspot position is essential during the early design. This work develops a data-driven neural network model for the fast prediction of junction temperature and hotspot position in 3D HBM chiplets. The model, trained with a data set of $13,494$ different combinations of thermal condition parameters, sampled from a vast parameter space characterized by high-dimensional combination (up to $3^{27}$), can accurately and quickly infer the junction temperature and hotspot position for any thermal conditions in the parameter space. Moreover, it shows good generalizability for other thermal conditions not considered in the parameter space. The data set is constructed using accurate finite element solvers. This method not only minimizes the reliance on costly experimental tests and extensive computational resources for finite element analysis but also accelerates the design and optimization of complex HBM systems, making it a valuable tool for improving thermal management and performance in high-performance computing applications.
• 摘要：随着计算能力需求的增加，高带宽内存（HBM）已成为下一代计算系统的关键技术。但是，HBM的广泛采用提出了重大的热管理挑战，尤其是在不同的热条件下的多层通过 - 硅via（TSV）堆叠结构中，在早期设计期间，准确预测连接温度和热点位置是必不可少的。这项工作开发了一个数据驱动的神经网络模型，用于在3D HBM芯片中快速预测连接温度和热点位置。该模型以$ 13,494 $不同的热条件参数组合的数据集进行了训练，这些组合是根据以高维组合为特征的巨大参数空间（最高$ 3^{27} $）的，可以准确，快速地推断出在参数空间中任何热条件的连接温度和热点位置。此外，它显示出对参数空间中未考虑的其他热条件的良好概括性。数据集是使用准确的有限元求解器构建的。这种方法不仅可以最大程度地减少对有限元分析的昂贵实验测试和广泛的计算资源的依赖，还可以加速复杂的HBM系统的设计和优化，使其成为改善高性能计算应用中的热管理和性能的宝贵工具。

Title: Underlying Semantic Diffusion for Effective and Efficient In-Context Learning

• Authors: Zhong Ji, Weilong Cao, Yan Zhang, Yanwei Pang, Jungong Han, Xuelong Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.04050
• Pdf URL: https://arxiv.org/pdf/2503.04050
• Copy Paste: [[2503.04050]] Underlying Semantic Diffusion for Effective and Efficient In-Context Learning(https://arxiv.org/abs/2503.04050)
• Keywords: generation
• Abstract: Diffusion models has emerged as a powerful framework for tasks like image controllable generation and dense prediction. However, existing models often struggle to capture underlying semantics (e.g., edges, textures, shapes) and effectively utilize in-context learning, limiting their contextual understanding and image generation quality. Additionally, high computational costs and slow inference speeds hinder their real-time applicability. To address these challenges, we propose Underlying Semantic Diffusion (US-Diffusion), an enhanced diffusion model that boosts underlying semantics learning, computational efficiency, and in-context learning capabilities on multi-task scenarios. We introduce Separate & Gather Adapter (SGA), which decouples input conditions for different tasks while sharing the architecture, enabling better in-context learning and generalization across diverse visual domains. We also present a Feedback-Aided Learning (FAL) framework, which leverages feedback signals to guide the model in capturing semantic details and dynamically adapting to task-specific contextual cues. Furthermore, we propose a plug-and-play Efficient Sampling Strategy (ESS) for dense sampling at time steps with high-noise levels, which aims at optimizing training and inference efficiency while maintaining strong in-context learning performance. Experimental results demonstrate that US-Diffusion outperforms the state-of-the-art method, achieving an average reduction of 7.47 in FID on Map2Image tasks and an average reduction of 0.026 in RMSE on Image2Map tasks, while achieving approximately 9.45 times faster inference speed. Our method also demonstrates superior training efficiency and in-context learning capabilities, excelling in new datasets and tasks, highlighting its robustness and adaptability across diverse visual domains.
• 摘要：扩散模型已成为图像可控生成和密集预测等任务的强大框架。但是，现有模型通常很难捕获基本的语义（例如边缘，纹理，形状），并有效地利用内在的学习，从而限制了它们的上下文理解和图像产生质量。此外，高计算成本和缓慢的推理速度阻碍了其实时适用性。为了应对这些挑战，我们提出了潜在的语义扩散（US-扩展），这是一个增强的扩散模型，可提高基本语义学习，计算效率和对多任务场景的内在学习能力。我们介绍了单独的收集适配器（SGA），该适配器（SGA）在共享体系结构的同时，将其分解了不同任务的输入条件，从而可以在不同的视觉域中更好地在上下文学习和概括。我们还提出了一个反馈辅助学习（FAL）框架，该框架利用反馈信号指导模型捕获语义细节并动态适应特定于任务的上下文提示。此外，我们提出了一种有效的有效抽样策略（ESS），以在具有高噪声水平的时间步骤进行密集的采样，旨在优化培训和推理效率，同时保持强大的内在学习绩效。实验结果表明，US-Fusion的表现优于最先进的方法，在MAP2IMAGE任务上平均减少了7.47的FID，而在Image2map任务上平均降低了0.026，同时达到了大约9.45倍的推进速度。我们的方法还表明，在新的数据集和任务中表现出色，在新的数据集和任务中表现出色，突出了其跨不同视觉域的鲁棒性和适应性。

Title: DuCos: Duality Constrained Depth Super-Resolution via Foundation Model

• Authors: Zhiqiang Yan, Zhengxue Wang, Haoye Dong, Jun Li, Jian Yang, Gim Hee Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.04171
• Pdf URL: https://arxiv.org/pdf/2503.04171
• Copy Paste: [[2503.04171]] DuCos: Duality Constrained Depth Super-Resolution via Foundation Model(https://arxiv.org/abs/2503.04171)
• Keywords: super-resolution
• Abstract: We introduce DuCos, a novel depth super-resolution framework grounded in Lagrangian duality theory, offering a flexible integration of multiple constraints and reconstruction objectives to enhance accuracy and robustness. Our DuCos is the first to significantly improve generalization across diverse scenarios with foundation models as prompts. The prompt design consists of two key components: Correlative Fusion (CF) and Gradient Regulation (GR). CF facilitates precise geometric alignment and effective fusion between prompt and depth features, while GR refines depth predictions by enforcing consistency with sharp-edged depth maps derived from foundation models. Crucially, these prompts are seamlessly embedded into the Lagrangian constraint term, forming a synergistic and principled framework. Extensive experiments demonstrate that DuCos outperforms existing state-of-the-art methods, achieving superior accuracy, robustness, and generalization. The source codes and pre-trained models will be publicly available.
• 摘要：我们介绍了Ducos，这是一个以拉格朗日二元性理论为基础的新型深度超分辨率框架，提供了多个约束和重建目标的灵活整合，以提高准确性和鲁棒性。我们的DUCOS是第一个以基础模型作为提示来显着改善各种情况的概括的人。及时设计由两个关键组成部分组成：相关融合（CF）和梯度调节（GR）。 CF促进了迅速和深度特征之间精确的几何对齐和有效融合，而GR通过与基础模型得出的锋利的深度图实施一致性来完善深度预测。至关重要的是，这些提示将无缝嵌入到拉格朗日的约束术语中，形成了协同和原则性的框架。广泛的实验表明，DUCOS的表现优于现有的最新方法，实现了卓越的准确性，鲁棒性和概括。源代码和预培训模型将公开可用。

Title: Energy-Guided Optimization for Personalized Image Editing with Pretrained Text-to-Image Diffusion Models

• Authors: Rui Jiang, Xinghe Fu, Guangcong Zheng, Teng Li, Taiping Yao, Xi Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.04215
• Pdf URL: https://arxiv.org/pdf/2503.04215
• Copy Paste: [[2503.04215]] Energy-Guided Optimization for Personalized Image Editing with Pretrained Text-to-Image Diffusion Models(https://arxiv.org/abs/2503.04215)
• Keywords: generation
• Abstract: The rapid advancement of pretrained text-driven diffusion models has significantly enriched applications in image generation and editing. However, as the demand for personalized content editing increases, new challenges emerge especially when dealing with arbitrary objects and complex scenes. Existing methods usually mistakes mask as the object shape prior, which struggle to achieve a seamless integration result. The mostly used inversion noise initialization also hinders the identity consistency towards the target object. To address these challenges, we propose a novel training-free framework that formulates personalized content editing as the optimization of edited images in the latent space, using diffusion models as the energy function guidance conditioned by reference text-image pairs. A coarse-to-fine strategy is proposed that employs text energy guidance at the early stage to achieve a natural transition toward the target class and uses point-to-point feature-level image energy guidance to perform fine-grained appearance alignment with the target object. Additionally, we introduce the latent space content composition to enhance overall identity consistency with the target. Extensive experiments demonstrate that our method excels in object replacement even with a large domain gap, highlighting its potential for high-quality, personalized image editing.
• 摘要：经过验证的文本驱动扩散模型的快速发展在图像生成和编辑中具有显着丰富的应用。但是，随着对个性化内容编辑的需求增加，新挑战尤其是在处理任意对象和复杂场景时。现有的方法通常将掩模误认为是对象形状的先验，而这些方法很难获得无缝的集成结果。大多数使用的反转噪声初始化还阻碍了目标对象的身份一致性。为了应对这些挑战，我们提出了一个新颖的无培训框架，该框架将个性化内容编辑作为在潜在空间中的优化，使用扩散模型作为由参考文本图像对调节的能量函数指南。提出了一种在早期阶段采用文本能量指导的粗到精细策略，以实现向目标类别的自然过渡，并使用点对点特征级图像能量指导与目标对象进行细粒度的外观对齐。此外，我们介绍了潜在的空间内容组成，以增强与目标的总体身份一致性。广泛的实验表明，即使有较大的域间隙，我们的方法也可以在对象更换中脱颖而出，从而突出了其高质量，个性化图像编辑的潜力。

Title: Spiking Meets Attention: Efficient Remote Sensing Image Super-Resolution with Attention Spiking Neural Networks

• Authors: Yi Xiao, Qiangqiang Yuan, Kui Jiang, Qiang Zhang, Tingting Zheng, Chia-Wen Lin, Liangpei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.04223
• Pdf URL: https://arxiv.org/pdf/2503.04223
• Copy Paste: [[2503.04223]] Spiking Meets Attention: Efficient Remote Sensing Image Super-Resolution with Attention Spiking Neural Networks(https://arxiv.org/abs/2503.04223)
• Keywords: super-resolution
• Abstract: Spiking neural networks (SNNs) are emerging as a promising alternative to traditional artificial neural networks (ANNs), offering biological plausibility and energy efficiency. Despite these merits, SNNs are frequently hampered by limited capacity and insufficient representation power, yet remain underexplored in remote sensing super-resolution (SR) tasks. In this paper, we first observe that spiking signals exhibit drastic intensity variations across diverse textures, highlighting an active learning state of the neurons. This observation motivates us to apply SNNs for efficient SR of RSIs. Inspired by the success of attention mechanisms in representing salient information, we devise the spiking attention block (SAB), a concise yet effective component that optimizes membrane potentials through inferred attention weights, which, in turn, regulates spiking activity for superior feature representation. Our key contributions include: 1) we bridge the independent modulation between temporal and channel dimensions, facilitating joint feature correlation learning, and 2) we access the global self-similar patterns in large-scale remote sensing imagery to infer spatial attention weights, incorporating effective priors for realistic and faithful reconstruction. Building upon SAB, we proposed SpikeSR, which achieves state-of-the-art performance across various remote sensing benchmarks such as AID, DOTA, and DIOR, while maintaining high computational efficiency. The code of SpikeSR will be available upon paper acceptance.
• 摘要：尖峰神经网络（SNN）正在成为传统人工神经网络（ANN）的有希望的替代方案，提供了生物学上的合理性和能量效率。尽管有这些优点，但SNN经常受到有限的容量和不足的表示功率的阻碍，但在遥感超分辨率（SR）任务中仍然没有被忽视。在本文中，我们首先观察到尖峰信号在各种纹理之间表现出急剧的强度变化，突出了神经元的主动学习状态。这种观察促使我们将SNN应用于有效的RSIS。受到注意机制在表示显着信息的成功启发，我们设计了尖峰注意块（SAB），这是一种简洁而有效的组件，通过推断注意力重量来优化膜电位，进而调节尖峰活动以获得出色的特征表示。我们的主要贡献包括：1）我们在时间和通道维度之间桥接独立的调制，促进联合特征相关学习，以及2）我们访问大型遥感图像中的全球自相似模式，以推断空间注意力的重量，并结合现实和忠实重建的有效先验。在SAB的基础上，我们提出了Spikesr，该Spikesr在辅助，DOTA和DIOR等各种遥感基准中实现了最先进的性能，同时保持了较高的计算效率。 Spikesr守则将在纸上接受后获得。

Title: How to Mitigate Overfitting in Weak-to-strong Generalization?

• Authors: Junhao Shi, Qinyuan Cheng, Zhaoye Fei, Yining Zheng, Qipeng Guo, Xipeng Qiu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.04249
• Pdf URL: https://arxiv.org/pdf/2503.04249
• Copy Paste: [[2503.04249]] How to Mitigate Overfitting in Weak-to-strong Generalization?(https://arxiv.org/abs/2503.04249)
• Keywords: generation
• Abstract: Aligning powerful AI models on tasks that surpass human evaluation capabilities is the central problem of \textbf{superalignment}. To address this problem, weak-to-strong generalization aims to elicit the capabilities of strong models through weak supervisors and ensure that the behavior of strong models aligns with the intentions of weak supervisors without unsafe behaviors such as deception. Although weak-to-strong generalization exhibiting certain generalization capabilities, strong models exhibit significant overfitting in weak-to-strong generalization: Due to the strong fit ability of strong models, erroneous labels from weak supervisors may lead to overfitting in strong models. In addition, simply filtering out incorrect labels may lead to a degeneration in question quality, resulting in a weak generalization ability of strong models on hard questions. To mitigate overfitting in weak-to-strong generalization, we propose a two-stage framework that simultaneously improves the quality of supervision signals and the quality of input questions. Experimental results in three series of large language models and two mathematical benchmarks demonstrate that our framework significantly improves PGR compared to naive weak-to-strong generalization, even achieving up to 100\% PGR on some models.
• 摘要：在超过人类评估功能的任务上对齐强大的AI模型是\ textbf {Supergailment}的核心问题。为了解决这一问题，弱到很概括旨在通过薄弱的主管来引起强模型的能力，并确保强大模型的行为与没有欺骗等不安全行为的弱主管的意图一致。尽管弱到较长的概括具有某些概括能力，但强模型在弱到较大的概括方面表现出明显的过度拟合：由于强大的模型的拟合能力很强，因此来自弱主管的错误标签可能会导致在强模型中过度拟合。此外，简单地滤除错误的标签可能会导致质量质量的变性，从而导致强大模型在硬性问题上的概括能力较弱。为了减轻过度拟合弱到弱的概括，我们提出了一个两阶段的框架，同时提高了监督信号的质量和输入问题的质量。与天真的弱到较强的概括相比，三个系列大语模型和两个数学基准的实验结果表明，我们的框架显着改善了PGR，甚至在某些模型上甚至达到了100 \％的PGR。

Title: An Egocentric Vision-Language Model based Portable Real-time Smart Assistant

• Authors: Yifei Huang, Jilan Xu, Baoqi Pei, Yuping He, Guo Chen, Mingfang Zhang, Lijin Yang, Zheng Nie, Jinyao Liu, Guoshun Fan, Dechen Lin, Fang Fang, Kunpeng Li, Chang Yuan, Xinyuan Chen, Yaohui Wang, Yali Wang, Yu Qiao, Limin Wang
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2503.04250
• Pdf URL: https://arxiv.org/pdf/2503.04250
• Copy Paste: [[2503.04250]] An Egocentric Vision-Language Model based Portable Real-time Smart Assistant(https://arxiv.org/abs/2503.04250)
• Keywords: generation
• Abstract: We present Vinci, a vision-language system designed to provide real-time, comprehensive AI assistance on portable devices. At its core, Vinci leverages EgoVideo-VL, a novel model that integrates an egocentric vision foundation model with a large language model (LLM), enabling advanced functionalities such as scene understanding, temporal grounding, video summarization, and future planning. To enhance its utility, Vinci incorporates a memory module for processing long video streams in real time while retaining contextual history, a generation module for producing visual action demonstrations, and a retrieval module that bridges egocentric and third-person perspectives to provide relevant how-to videos for skill acquisition. Unlike existing systems that often depend on specialized hardware, Vinci is hardware-agnostic, supporting deployment across a wide range of devices, including smartphones and wearable cameras. In our experiments, we first demonstrate the superior performance of EgoVideo-VL on multiple public benchmarks, showcasing its vision-language reasoning and contextual understanding capabilities. We then conduct a series of user studies to evaluate the real-world effectiveness of Vinci, highlighting its adaptability and usability in diverse scenarios. We hope Vinci can establish a new framework for portable, real-time egocentric AI systems, empowering users with contextual and actionable insights. Including the frontend, backend, and models, all codes of Vinci are available at this https URL.
• 摘要：我们提出了Vinci，这是一种视觉语言系统，旨在在便携式设备上提供实时，全面的AI帮助。 Vinci以核心利用Egovideo-VL，这是一个新颖的模型，将以Egentric Vision Foundit Foundation模型与大语言模型（LLM）整合在一起，从而实现了高级功能，例如场景理解，时间基础，视频摘要，视频摘要和未来计划。为了增强其效用，VINCI合并了一个内存模块，用于实时处理长视频流，同时保留上下文历史记录，一个用于产生视觉动作演示的生成模块以及一个桥梁的检索模块，该模块桥接了以自我为中心的和第三人称的观点，以提供相关的技能视频以供技能获取。与经常依赖专用硬件的现有系统不同，Vinci是硬件敏捷的，可支持在包括智能手机和可穿戴摄像头在内的各种设备上的部署。在我们的实验中，我们首先证明了Egovideo-vl在多个公共基准上的出色表现，展示了其视野推理和上下文理解能力。然后，我们进行了一系列用户研究，以评估VINCI的现实有效性，从而强调了其在各种情况下的适应性和可用性。我们希望Vinci能够为便携式实时Egentric AI系统建立一个新的框架，从而通过上下文和可行的见解为用户增强了能力。包括前端，后端和型号在内，Vinci的所有代码都可以在此HTTPS URL上找到。

Title: Knowledge Retention for Continual Model-Based Reinforcement Learning

• Authors: Yixiang Sun, Haotian Fu, Michael Littman, George Konidaris
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.04256
• Pdf URL: https://arxiv.org/pdf/2503.04256
• Copy Paste: [[2503.04256]] Knowledge Retention for Continual Model-Based Reinforcement Learning(https://arxiv.org/abs/2503.04256)
• Keywords: generative
• Abstract: We propose DRAGO, a novel approach for continual model-based reinforcement learning aimed at improving the incremental development of world models across a sequence of tasks that differ in their reward functions but not the state space or dynamics. DRAGO comprises two key components: Synthetic Experience Rehearsal, which leverages generative models to create synthetic experiences from past tasks, allowing the agent to reinforce previously learned dynamics without storing data, and Regaining Memories Through Exploration, which introduces an intrinsic reward mechanism to guide the agent toward revisiting relevant states from prior tasks. Together, these components enable the agent to maintain a comprehensive and continually developing world model, facilitating more effective learning and adaptation across diverse environments. Empirical evaluations demonstrate that DRAGO is able to preserve knowledge across tasks, achieving superior performance in various continual learning scenarios.
• 摘要：我们提出了Drago，这是一种用于连续基于模型的强化学习的新颖方法，旨在改善一系列任务的增量发展，这些任务在其奖励功能上有所不同，但没有状态空间或动态。 Drago包括两个关键组成部分：综合体验彩排，它利用生成模型从过去的任务中创建合成体验，允许代理商在不存储数据的情况下加强先前学到的动态，并通过探索重新获得记忆，从而引入了固有的奖励机制，以指导代理机制从先前的任务中恢复相关状态。这些组件共同使代理商能够保持全面，不断发展的世界模型，从而促进各种环境中更有效的学习和适应。经验评估表明，Drago能够在各种持续学习方案中保持跨任务的知识，从而实现卓越的表现。

Title: S2Gaussian: Sparse-View Super-Resolution 3D Gaussian Splatting

• Authors: Yecong Wan, Mingwen Shao, Yuanshuo Cheng, Wangmeng Zuo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.04314
• Pdf URL: https://arxiv.org/pdf/2503.04314
• Copy Paste: [[2503.04314]] S2Gaussian: Sparse-View Super-Resolution 3D Gaussian Splatting(https://arxiv.org/abs/2503.04314)
• Keywords: super-resolution
• Abstract: In this paper, we aim ambitiously for a realistic yet challenging problem, namely, how to reconstruct high-quality 3D scenes from sparse low-resolution views that simultaneously suffer from deficient perspectives and clarity. Whereas existing methods only deal with either sparse views or low-resolution observations, they fail to handle such hybrid and complicated scenarios. To this end, we propose a novel Sparse-view Super-resolution 3D Gaussian Splatting framework, dubbed S2Gaussian, that can reconstruct structure-accurate and detail-faithful 3D scenes with only sparse and low-resolution views. The S2Gaussian operates in a two-stage fashion. In the first stage, we initially optimize a low-resolution Gaussian representation with depth regularization and densify it to initialize the high-resolution Gaussians through a tailored Gaussian Shuffle Split operation. In the second stage, we refine the high-resolution Gaussians with the super-resolved images generated from both original sparse views and pseudo-views rendered by the low-resolution Gaussians. In which a customized blur-free inconsistency modeling scheme and a 3D robust optimization strategy are elaborately designed to mitigate multi-view inconsistency and eliminate erroneous updates caused by imperfect supervision. Extensive experiments demonstrate superior results and in particular establishing new state-of-the-art performances with more consistent geometry and finer details.
• 摘要：在本文中，我们雄心勃勃地针对一个现实而又具有挑战性的问题，即，如何从稀疏的低分辨率观点中重建高质量的3D场景，这些观点同时遭受了不足的观点和清晰度的影响。尽管现有方法仅处理稀疏视图或低分辨率观察结果，但它们无法处理这种混合和复杂的方案。为此，我们提出了一个新颖的稀疏视图超分辨率3D高斯脱落框架，称为S2Gaussian，它可以重建结构 - 精确和细节幻想的3D场景，只有稀疏和低分辨率的视图。 S2Gaussian以两阶段的方式运作。在第一阶段，我们最初通过深度正则化优化了低分辨率的高斯表示，并通过量身定制的高斯洗牌拆分操作来使其致密以初始化高分辨率高斯。在第二阶段，我们通过低分辨率高斯人呈现的原始稀疏视图和伪视图产生的超级分辨图像来完善高分辨率高斯人。在其中精心设计了一种自定义的无模糊不一致的建模方案和3D强大的优化策略，以减轻多视图的不一致，并消除由不完善的监督引起的错误更新。广泛的实验表明了卓越的结果，尤其是建立了新的最先进的表演，并具有更一致的几何形状和更精细的细节。

Title: scDD: Latent Codes Based scRNA-seq Dataset Distillation with Foundation Model Knowledge

• Authors: Zhen Yu, Jianan Han, Yang Liu, Qingchao Chen
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.04357
• Pdf URL: https://arxiv.org/pdf/2503.04357
• Copy Paste: [[2503.04357]] scDD: Latent Codes Based scRNA-seq Dataset Distillation with Foundation Model Knowledge(https://arxiv.org/abs/2503.04357)
• Keywords: generation
• Abstract: Single-cell RNA sequencing (scRNA-seq) technology has profiled hundreds of millions of human cells across organs, diseases, development and perturbations to date. However, the high-dimensional sparsity, batch effect noise, category imbalance, and ever-increasing data scale of the original sequencing data pose significant challenges for multi-center knowledge transfer, data fusion, and cross-validation between scRNA-seq datasets. To address these barriers, (1) we first propose a latent codes-based scRNA-seq dataset distillation framework named scDD, which transfers and distills foundation model knowledge and original dataset information into a compact latent space and generates synthetic scRNA-seq dataset by a generator to replace the original dataset. Then, (2) we propose a single-step conditional diffusion generator named SCDG, which perform single-step gradient back-propagation to help scDD optimize distillation quality and avoid gradient decay caused by multi-step back-propagation. Meanwhile, SCDG ensures the scRNA-seq data characteristics and inter-class discriminability of the synthetic dataset through flexible conditional control and generation quality assurance. Finally, we propose a comprehensive benchmark to evaluate the performance of scRNA-seq dataset distillation in different data analysis tasks. It is validated that our proposed method can achieve 7.61% absolute and 15.70% relative improvement over previous state-of-the-art methods on average task.
• 摘要：迄今为止，单细胞RNA测序（SCRNA-SEQ）技术已经介绍了数亿人类细胞，疾病，发育和扰动。但是，原始测序数据的高维稀疏性，批处理效应噪声，类别失衡以及不断增长的数据量表对多中心知识传输，数据融合和SCRNA-SEQ数据集之间的交叉验证构成了重大挑战。为了解决这些障碍，（1）我们首先提出了一个名为SCDD的基于潜在代码的SCRNA-SCRNA-SEQ数据集蒸馏框架，该框架将基础模型知识和原始数据集传输到紧凑的潜在空间中，并生成合成的SCRNA-SCRNA-SEQ数据集，由发电机替换原始数据集。然后，（2）我们提出了一个名为SCDG的单步条件扩散发生器，该发电机执行单步梯度后传播，以帮助SCDD优化蒸馏质量并避免由多步后场引起的梯度衰变。同时，SCDG确保通过柔性条件控制和发电质量保证，合成数据集的SCRNA-SEQ数据特征和类间可区分性。最后，我们提出了一个综合基准，以评估不同数据分析任务中SCRNA-Seq数据集蒸馏的性能。曾经验证的是，我们提出的方法可以平均实现7.61％的绝对方法和15.70％的相对改善。

Title: Scale-Invariant Adversarial Attack against Arbitrary-scale Super-resolution

• Authors: Yihao Huang, Xin Luo, Qing Guo, Felix Juefei-Xu, Xiaojun Jia, Weikai Miao, Geguang Pu, Yang Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.04385
• Pdf URL: https://arxiv.org/pdf/2503.04385
• Copy Paste: [[2503.04385]] Scale-Invariant Adversarial Attack against Arbitrary-scale Super-resolution(https://arxiv.org/abs/2503.04385)
• Keywords: super-resolution
• Abstract: The advent of local continuous image function (LIIF) has garnered significant attention for arbitrary-scale super-resolution (SR) techniques. However, while the vulnerabilities of fixed-scale SR have been assessed, the robustness of continuous representation-based arbitrary-scale SR against adversarial attacks remains an area warranting further exploration. The elaborately designed adversarial attacks for fixed-scale SR are scale-dependent, which will cause time-consuming and memory-consuming problems when applied to arbitrary-scale SR. To address this concern, we propose a simple yet effective ``scale-invariant'' SR adversarial attack method with good transferability, termed SIAGT. Specifically, we propose to construct resource-saving attacks by exploiting finite discrete points of continuous representation. In addition, we formulate a coordinate-dependent loss to enhance the cross-model transferability of the attack. The attack can significantly deteriorate the SR images while introducing imperceptible distortion to the targeted low-resolution (LR) images. Experiments carried out on three popular LIIF-based SR approaches and four classical SR datasets show remarkable attack performance and transferability of SIAGT.
• 摘要：局部连续图像函数（LIIF）的出现引起了对任意规模超分辨率（SR）技术的极大关注。但是，尽管已经评估了固定规模的SR的脆弱性，但基于连续代表的任意尺度SR的鲁棒性针对对抗性攻击仍然是一个需要进一步探索的领域。针对固定规模的SR的精心设计的对抗性攻击依赖比例依赖性，当应用于任意规模的SR时，这将导致耗时和记忆耗费的问题。为了解决这一问题，我们提出了一种简单而有效的``规模不变''SR对抗攻击方法，具有良好的可传递性，称为SIAGT。具体而言，我们建议通过利用连续表示的有限离散点来构建节省资源的攻击。此外，我们制定了依赖坐标的损失，以增强攻击的跨模型可传递性。该攻击可能会显着恶化SR图像，同时将无法察觉的失真引入目标低分辨率（LR）图像。在三种流行的基于LIIF的SR方法和四个经典SR数据集上进行的实验表现出显着的攻击性能和SIAGT的可传递性。

Title: TPC: Cross-Temporal Prediction Connection for Vision-Language Model Hallucination Reduction

• Authors: Chao Wang, Weiwei Fu, Yang Zhou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2503.04457
• Pdf URL: https://arxiv.org/pdf/2503.04457
• Copy Paste: [[2503.04457]] TPC: Cross-Temporal Prediction Connection for Vision-Language Model Hallucination Reduction(https://arxiv.org/abs/2503.04457)
• Keywords: generation
• Abstract: Vision-language models (VLMs) have achieved remarkable advancements, capitalizing on the impressive capabilities of large language models (LLMs) across diverse tasks. Despite this, a critical challenge known as hallucination occurs when models overconfidently describe objects or attributes absent from the image, a problem exacerbated by the tendency of VLMs to rely on linguistic priors. This limitation reduces model reliability in high-stakes applications. In this work, we have observed the characteristic of logits' continuity consistency enhancement and introduced a straightforward and efficient method, Cross-Temporal Prediction Connection (TPC), designed to enhance the semantic consistency of logits by connecting them temporally across timesteps. TPC amplifies information flow and improves coherence, effectively reducing hallucination. Extensive experiments show that TPC surpasses existing representatives, delivering superior performance in both accuracy and efficiency while maintaining robustness in open-ended text generation tasks.
• 摘要：视觉模型（VLM）取得了显着进步，利用了各种任务跨越大型语言模型（LLM）的令人印象深刻的能力。尽管如此，当模型过于保密地描述图像中缺少的对象或属性时，就会发生一个关键的挑战，这是VLM依靠语言先验的趋势加剧的问题。此限制可降低高风险应用中的模型可靠性。在这项工作中，我们观察到了逻辑连续性一致性增强的特征，并引入了一种直接有效的方法，即跨时空预测连接（TPC），旨在通过跨时间浏览暂时连接逻辑来增强逻辑的语义一致性。 TPC扩大信息流并提高连贯性，有效地减少了幻觉。广泛的实验表明，TPC超过了现有代表，在准确性和效率方面表现出色，同时保持开放式文本生成任务的鲁棒性。

Title: Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judges

• Authors: Francisco Eiras, Eliott Zemour, Eric Lin, Vaikkunth Mugunthan
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2503.04474
• Pdf URL: https://arxiv.org/pdf/2503.04474
• Copy Paste: [[2503.04474]] Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judges(https://arxiv.org/abs/2503.04474)
• Keywords: generation
• Abstract: Large Language Model (LLM) based judges form the underpinnings of key safety evaluation processes such as offline benchmarking, automated red-teaming, and online guardrailing. This widespread requirement raises the crucial question: can we trust the evaluations of these evaluators? In this paper, we highlight two critical challenges that are typically overlooked: (i) evaluations in the wild where factors like prompt sensitivity and distribution shifts can affect performance and (ii) adversarial attacks that target the judge. We highlight the importance of these through a study of commonly used safety judges, showing that small changes such as the style of the model output can lead to jumps of up to 0.24 in the false negative rate on the same dataset, whereas adversarial attacks on the model generation can fool some judges into misclassifying 100% of harmful generations as safe ones. These findings reveal gaps in commonly used meta-evaluation benchmarks and weaknesses in the robustness of current LLM judges, indicating that low attack success under certain judges could create a false sense of security.
• 摘要：大型语言模型（LLM）的法官构成了关键安全评估过程的基础，例如离线基准测试，自动红色团队和在线护栏。这一普遍要求提出了一个关键问题：我们可以相信这些评估者的评估吗？在本文中，我们重点介绍了通常被忽略的两个关键挑战：（i）在野外进行评估，其中迅速灵敏度和分配变化等因素会影响绩效，以及（ii）针对法官的对抗性攻击。我们通过对常用安全法官的研究来强调这些的重要性，这表明诸如模型输出的样式诸如模型输出的样式可以导致同一数据集中的假负率高达0.24，而对模型产生的对抗性攻击可能会使一些法官欺骗一些法官将有害世代的100％分类为安全的人。这些发现揭示了当前LLM法官鲁棒性的常用元评估基准和弱点的差距，表明某些法官在某些法官下的攻击成功率低可能会产生错误的安全感。

Title: A Novel Solution for Drone Photogrammetry with Low-overlap Aerial Images using Monocular Depth Estimation

• Authors: Jiageng Zhong, Qi Zhou, Ming Li, Armin Gruen, Xuan Liao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.04513
• Pdf URL: https://arxiv.org/pdf/2503.04513
• Copy Paste: [[2503.04513]] A Novel Solution for Drone Photogrammetry with Low-overlap Aerial Images using Monocular Depth Estimation(https://arxiv.org/abs/2503.04513)
• Keywords: generation
• Abstract: Low-overlap aerial imagery poses significant challenges to traditional photogrammetric methods, which rely heavily on high image overlap to produce accurate and complete mapping products. In this study, we propose a novel workflow based on monocular depth estimation to address the limitations of conventional techniques. Our method leverages tie points obtained from aerial triangulation to establish a relationship between monocular depth and metric depth, thus transforming the original depth map into a metric depth map, enabling the generation of dense depth information and the comprehensive reconstruction of the scene. For the experiments, a high-overlap drone dataset containing 296 images is processed using Metashape to generate depth maps and DSMs as ground truth. Subsequently, we create a low-overlap dataset by selecting 20 images for experimental evaluation. Results demonstrate that while the recovered depth maps and resulting DSMs achieve meter-level accuracy, they provide significantly better completeness compared to traditional methods, particularly in regions covered by single images. This study showcases the potential of monocular depth estimation in low-overlap aerial photogrammetry.
• 摘要：低空飞行的航空影像对传统的摄影法对高图像重叠构成了重大挑战，以产生准确而完整的映射产品。在这项研究中，我们提出了一个基于单眼深度估计的新型工作流程，以解决传统技术的局限性。我们的方法利用了从空中三角剖分获得的扎点，以建立单眼深度和度量深度之间的关系，从而将原始深度图转换为度量深度图，从而能够产生密集的深度信息以及场景的全面重建。对于实验，使用Metashape处理包含296张图像的高空无人机数据集，以生成深度图和DSM作为地面真实。随后，我们通过选择20张图像进行实验评估来创建一个低重叠的数据集。结果表明，尽管恢复的深度图和结果DSM达到了仪表级的准确性，但与传统方法相比，它们提供了明显更好的完整性，尤其是在单个图像所涵盖的区域中。这项研究展示了低重叠空中摄影测量法中单眼深度估计的潜力。

Title: The Best of Both Worlds: Integrating Language Models and Diffusion Models for Video Generation

• Authors: Aoxiong Yin, Kai Shen, Yichong Leng, Xu Tan, Xinyu Zhou, Juncheng Li, Siliang Tang
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.04606
• Pdf URL: https://arxiv.org/pdf/2503.04606
• Copy Paste: [[2503.04606]] The Best of Both Worlds: Integrating Language Models and Diffusion Models for Video Generation(https://arxiv.org/abs/2503.04606)
• Keywords: generation
• Abstract: Recent advancements in text-to-video (T2V) generation have been driven by two competing paradigms: autoregressive language models and diffusion models. However, each paradigm has intrinsic limitations: language models struggle with visual quality and error accumulation, while diffusion models lack semantic understanding and causal modeling. In this work, we propose LanDiff, a hybrid framework that synergizes the strengths of both paradigms through coarse-to-fine generation. Our architecture introduces three key innovations: (1) a semantic tokenizer that compresses 3D visual features into compact 1D discrete representations through efficient semantic compression, achieving a $\sim$14,000$\times$ compression ratio; (2) a language model that generates semantic tokens with high-level semantic relationships; (3) a streaming diffusion model that refines coarse semantics into high-fidelity videos. Experiments show that LanDiff, a 5B model, achieves a score of 85.43 on the VBench T2V benchmark, surpassing the state-of-the-art open-source models Hunyuan Video (13B) and other commercial models such as Sora, Keling, and Hailuo. Furthermore, our model also achieves state-of-the-art performance in long video generation, surpassing other open-source models in this field. Our demo can be viewed at this https URL.
• 摘要：文本到视频（T2V）一代的最新进展是由两个竞争范式驱动的：自回归语言模型和扩散模型。但是，每个范式都有内在的局限性：语言模型在视觉质量和错误积累中挣扎，而扩散模型缺乏语义理解和因果建模。在这项工作中，我们提出了Landiff，这是一个混合框架，通过粗到精细的一代协同范围。我们的体系结构介绍了三个关键创新：（1）语义令牌通过有效的语义压缩将3D视觉功能压缩为紧凑的1D离散表示形式，达到$ \ sim $ 14,000 $ \ times $ compression $压缩比； （2）一种具有高级语义关系的语义令牌的语言模型； （3）将粗略语义改进到高保真视频中的流式扩散模型。实验表明，Landiff是一种5B型号，在VBENCH T2V基准上获得了85.43的得分，超过了最先进的开源模型Hunyuan视频（13B）和其他商业模型，例如Sora，Keling和Hailuo。此外，我们的模型还可以在长期视频生成中实现最先进的性能，超过了该领域的其他开源模型。可以通过此HTTPS URL查看我们的演示。

Title: PathoPainter: Augmenting Histopathology Segmentation via Tumor-aware Inpainting

• Authors: Hong Liu, Haosen Yang, Evi M.C. Huijben, Mark Schuiveling, Ruisheng Su, Josien P.W. Pluim, Mitko Veta
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.04634
• Pdf URL: https://arxiv.org/pdf/2503.04634
• Copy Paste: [[2503.04634]] PathoPainter: Augmenting Histopathology Segmentation via Tumor-aware Inpainting(https://arxiv.org/abs/2503.04634)
• Keywords: generation
• Abstract: Tumor segmentation plays a critical role in histopathology, but it requires costly, fine-grained image-mask pairs annotated by pathologists. Thus, synthesizing histopathology data to expand the dataset is highly desirable. Previous works suffer from inaccuracies and limited diversity in image-mask pairs, both of which affect training segmentation, particularly in small-scale datasets and the inherently complex nature of histopathology images. To address this challenge, we propose PathoPainter, which reformulates image-mask pair generation as a tumor inpainting task. Specifically, our approach preserves the background while inpainting the tumor region, ensuring precise alignment between the generated image and its corresponding mask. To enhance dataset diversity while maintaining biological plausibility, we incorporate a sampling mechanism that conditions tumor inpainting on regional embeddings from a different image. Additionally, we introduce a filtering strategy to exclude uncertain synthetic regions, further improving the quality of the generated data. Our comprehensive evaluation spans multiple datasets featuring diverse tumor types and various training data scales. As a result, segmentation improved significantly with our synthetic data, surpassing existing segmentation data synthesis approaches, e.g., 75.69% -> 77.69% on CAMELYON16. The code is available at this https URL.
• 摘要：肿瘤分割在组织病理学中起着至关重要的作用，但需要由病理学家注释的昂贵，细粒度的图像面膜对。因此，高度希望合成的组织病理学数据扩展数据集。以前的作品遭受了图像面罩对的不准确性和有限的多样性，这两者都会影响训练分割，尤其是在小规模数据集和组织病理学图像的固有复杂性质中。为了应对这一挑战，我们提出了Pathopainter，该蛋白酶将图像面膜对生成重新定义为肿瘤介绍任务。具体而言，我们的方法保留了背景，同时介绍了肿瘤区域，以确保生成的图像与其相应的掩模之间的精确比对。为了增强数据集多样性，同时维持生物学上的合理性，我们结合了一种采样机制，该机制可以调节来自不同图像的区域嵌入肿瘤。此外，我们引入了一种过滤策略，以排除不确定的合成区域，从而进一步提高生成的数据的质量。我们的全面评估涵盖了多个具有不同肿瘤类型和各种训练数据量表的数据集。结果，通过我们的合成数据，分割显着改善，超过了现有的分割数据综合方法，例如，CamelyOn16上的75.69％ - > 77.69％。该代码可在此HTTPS URL上找到。

Title: Simulating the Real World: A Unified Survey of Multimodal Generative Models

• Authors: Yuqi Hu, Longguang Wang, Xian Liu, Ling-Hao Chen, Yuwei Guo, Yukai Shi, Ce Liu, Anyi Rao, Zeyu Wang, Hui Xiong
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2503.04641
• Pdf URL: https://arxiv.org/pdf/2503.04641
• Copy Paste: [[2503.04641]] Simulating the Real World: A Unified Survey of Multimodal Generative Models(https://arxiv.org/abs/2503.04641)
• Keywords: generation, generative
• Abstract: Understanding and replicating the real world is a critical challenge in Artificial General Intelligence (AGI) research. To achieve this, many existing approaches, such as world models, aim to capture the fundamental principles governing the physical world, enabling more accurate simulations and meaningful interactions. However, current methods often treat different modalities, including 2D (images), videos, 3D, and 4D representations, as independent domains, overlooking their interdependencies. Additionally, these methods typically focus on isolated dimensions of reality without systematically integrating their connections. In this survey, we present a unified survey for multimodal generative models that investigate the progression of data dimensionality in real-world simulation. Specifically, this survey starts from 2D generation (appearance), then moves to video (appearance+dynamics) and 3D generation (appearance+geometry), and finally culminates in 4D generation that integrate all dimensions. To the best of our knowledge, this is the first attempt to systematically unify the study of 2D, video, 3D and 4D generation within a single framework. To guide future research, we provide a comprehensive review of datasets, evaluation metrics and future directions, and fostering insights for newcomers. This survey serves as a bridge to advance the study of multimodal generative models and real-world simulation within a unified framework.
• 摘要：理解和复制现实世界是人工通用智能（AGI）研究的关键挑战。为了实现这一目标，许多现有的方法，例如世界模型，旨在捕获有关物理世界的基本原则，从而实现更准确的模拟和有意义的互动。但是，当前方法通常将不同的方式视为不同的模态，包括2D（图像），视频，3D和4D表示，作为独立域，忽略了它们的相互依赖性。此外，这些方法通常集中在现实的孤立维度上，而不会系统地整合其连接。在这项调查中，我们提出了一项针对多模式生成模型的统一调查，该模型研究了现实世界模拟中数据维度的进展。具体而言，这项调查从2D代（外观）开始，然后移至视频（外观+动力学）和3D（外观+几何），最后最终以4D代的结束，从而整合了所有维度。据我们所知，这是系统地将2D，视频，3D和4D代表的研究统一的第一次尝试。为了指导未来的研究，我们将对数据集，评估指标和未来方向进行全面审查，并为新移民提供见解。这项调查是推进统一框架内多模式生成模型和现实世界模拟的研究的桥梁。

Title: Implicit Neural Representation for Video and Image Super-Resolution

• Authors: Mary Aiyetigbo, Wanqi Yuan, Feng Luo, Nianyi Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.04665
• Pdf URL: https://arxiv.org/pdf/2503.04665
• Copy Paste: [[2503.04665]] Implicit Neural Representation for Video and Image Super-Resolution(https://arxiv.org/abs/2503.04665)
• Keywords: super-resolution
• Abstract: We present a novel approach for super-resolution that utilizes implicit neural representation (INR) to effectively reconstruct and enhance low-resolution videos and images. By leveraging the capacity of neural networks to implicitly encode spatial and temporal features, our method facilitates high-resolution reconstruction using only low-resolution inputs and a 3D high-resolution grid. This results in an efficient solution for both image and video super-resolution. Our proposed method, SR-INR, maintains consistent details across frames and images, achieving impressive temporal stability without relying on the computationally intensive optical flow or motion estimation typically used in other video super-resolution techniques. The simplicity of our approach contrasts with the complexity of many existing methods, making it both effective and efficient. Experimental evaluations show that SR-INR delivers results on par with or superior to state-of-the-art super-resolution methods, while maintaining a more straightforward structure and reduced computational demands. These findings highlight the potential of implicit neural representations as a powerful tool for reconstructing high-quality, temporally consistent video and image signals from low-resolution data.
• 摘要：我们提出了一种新型的超分辨率方法，该方法利用隐式神经表示（INR）有效地重建和增强了低分辨率的视频和图像。通过利用神经网络隐式编码空间和时间特征的能力，我们的方法仅使用低分辨率输入和3D高分辨率网格来促进高分辨率重建。这为图像和视频超分辨率提供了有效的解决方案。我们提出的方法SR-INR在框架和图像之间保持一致的细节，实现了令人印象深刻的时间稳定性，而无需依赖于其他视频超级分辨率技术中通常使用的计算密集的光流或运动估计。我们方法的简单性与许多现有方法的复杂性形成鲜明对比，使其既有效又有效。实验评估表明，SR-INR可以与最先进的超分辨率方法相提并论，同时保持更直接的结构和减少的计算需求。这些发现突出了隐式神经表示作为重建低分辨率数据中高质量，时间一致的视频和图像信号的强大工具的潜力。

Title: What Are You Doing? A Closer Look at Controllable Human Video Generation

• Authors: Emanuele Bugliarello, Anurag Arnab, Roni Paiss, Pieter-Jan Kindermans, Cordelia Schmid
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.04666
• Pdf URL: https://arxiv.org/pdf/2503.04666
• Copy Paste: [[2503.04666]] What Are You Doing? A Closer Look at Controllable Human Video Generation(https://arxiv.org/abs/2503.04666)
• Keywords: generation
• Abstract: High-quality benchmarks are crucial for driving progress in machine learning research. However, despite the growing interest in video generation, there is no comprehensive dataset to evaluate human generation. Humans can perform a wide variety of actions and interactions, but existing datasets, like TikTok and TED-Talks, lack the diversity and complexity to fully capture the capabilities of video generation models. We close this gap by introducing `What Are You Doing?' (WYD): a new benchmark for fine-grained evaluation of controllable image-to-video generation of humans. WYD consists of 1{,}544 captioned videos that have been meticulously collected and annotated with 56 fine-grained categories. These allow us to systematically measure performance across 9 aspects of human generation, including actions, interactions and motion. We also propose and validate automatic metrics that leverage our annotations and better capture human evaluations. Equipped with our dataset and metrics, we perform in-depth analyses of seven state-of-the-art models in controllable image-to-video generation, showing how WYD provides novel insights about the capabilities of these models. We release our data and code to drive forward progress in human video generation modeling at this https URL.
• 摘要：高质量的基准测试对于推动机器学习研究的进度至关重要。但是，尽管对视频生成的兴趣越来越浓厚，但仍没有评估人类发电的全面数据集。人类可以执行各种各样的动作和互动，但是现有的数据集（如Tiktok和Ted-Talks）缺乏多样性和复杂性，无法完全捕获视频生成模型的功能。我们通过介绍“您在做什么？”来缩小这一差距。 （WYD）：一种新的基准测试，用于对人类的可控图像与视频产生进行细粒度评估。 WYD由1 {，} 544个字幕视频组成，这些视频已经过精心收集并用56个细粒类别注释。这些使我们能够系统地测量人类一代9个方面的性能，包括行动，互动和运动。我们还建议并验证自动指标，以利用我们的注释并更好地捕获人类评估。配备了我们的数据集和指标，我们在可控的图像到视频生成中对七个最先进的模型进行了深入的分析，展示了WYD如何提供有关这些模型功能的新颖见解。我们发布我们的数据和代码，以推动此HTTPS URL的人类视频生成建模的前进进度。

Title: FluidNexus: 3D Fluid Reconstruction and Prediction from a Single Video

• Authors: Yue Gao, Hong-Xing Yu, Bo Zhu, Jiajun Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2503.04720
• Pdf URL: https://arxiv.org/pdf/2503.04720
• Copy Paste: [[2503.04720]] FluidNexus: 3D Fluid Reconstruction and Prediction from a Single Video(https://arxiv.org/abs/2503.04720)
• Keywords: generation
• Abstract: We study reconstructing and predicting 3D fluid appearance and velocity from a single video. Current methods require multi-view videos for fluid reconstruction. We present FluidNexus, a novel framework that bridges video generation and physics simulation to tackle this task. Our key insight is to synthesize multiple novel-view videos as references for reconstruction. FluidNexus consists of two key components: (1) a novel-view video synthesizer that combines frame-wise view synthesis with video diffusion refinement for generating realistic videos, and (2) a physics-integrated particle representation coupling differentiable simulation and rendering to simultaneously facilitate 3D fluid reconstruction and prediction. To evaluate our approach, we collect two new real-world fluid datasets featuring textured backgrounds and object interactions. Our method enables dynamic novel view synthesis, future prediction, and interaction simulation from a single fluid video. Project website: this https URL.
• 摘要：我们研究了单个视频的重建和预测3D流体外观和速度。当前方法需要多视频视频以进行流体重建。我们提出了FluidNexus，这是一个新颖的框架，它桥接了视频生成和物理模拟以解决此任务。我们的主要见解是将多个小说视频综合为重建的参考。 FluidNexus由两个关键组成部分组成：（1）一种新颖的视频合成器，将框架视图合成与视频扩散的细化结合在一起，用于生成逼真的视频，以及（2）物理综合的粒子表示耦合耦合，将不同的可不同的合成和呈现与3D流体恢复和预测同时实现。为了评估我们的方法，我们收集了两个新的现实流体数据集，这些数据集具有纹理背景和对象交互。我们的方法使动态的新型视图合成，未来预测和来自单个流体视频的相互作用模拟。项目网站：此HTTPS URL。