2025-02-05

Title: Leveraging Stable Diffusion for Monocular Depth Estimation via Image Semantic Encoding

• Authors: Jingming Xia, Guanqun Cao, Guang Ma, Yiben Luo, Qinzhao Li, John Oyekan
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.01666
• Pdf URL: https://arxiv.org/pdf/2502.01666
• Copy Paste: [[2502.01666]] Leveraging Stable Diffusion for Monocular Depth Estimation via Image Semantic Encoding(https://arxiv.org/abs/2502.01666)
• Keywords: generative
• Abstract: Monocular depth estimation involves predicting depth from a single RGB image and plays a crucial role in applications such as autonomous driving, robotic navigation, 3D reconstruction, etc. Recent advancements in learning-based methods have significantly improved depth estimation performance. Generative models, particularly Stable Diffusion, have shown remarkable potential in recovering fine details and reconstructing missing regions through large-scale training on diverse datasets. However, models like CLIP, which rely on textual embeddings, face limitations in complex outdoor environments where rich context information is needed. These limitations reduce their effectiveness in such challenging scenarios. Here, we propose a novel image-based semantic embedding that extracts contextual information directly from visual features, significantly improving depth prediction in complex environments. Evaluated on the KITTI and Waymo datasets, our method achieves performance comparable to state-of-the-art models while addressing the shortcomings of CLIP embeddings in handling outdoor scenes. By leveraging visual semantics directly, our method demonstrates enhanced robustness and adaptability in depth estimation tasks, showcasing its potential for application to other visual perception tasks.
• 摘要：单目深度估计涉及从单个 RGB 图像预测深度，在自动驾驶、机器人导航、3D 重建等应用中起着至关重要的作用。基于学习的方法的最新进展显著提高了深度估计性能。生成模型，尤其是稳定扩散，通过对不同数据集进行大规模训练，在恢复精细细节和重建缺失区域方面表现出了巨大的潜力。然而，像 CLIP 这样依赖于文本嵌入的模型在需要丰富上下文信息的复杂户外环境中面临限制。这些限制降低了它们在这种具有挑战性的场景中的有效性。在这里，我们提出了一种新颖的基于图像的语义嵌入，它直接从视觉特征中提取上下文信息，显著改善了复杂环境中的深度预测。在 KITTI 和 Waymo 数据集上进行评估后，我们的方法实现了与最先进模型相当的性能，同时解决了 CLIP 嵌入在处理户外场景方面的缺点。通过直接利用视觉语义，我们的方法在深度估计任务中表现出增强的鲁棒性和适应性，展示了其应用于其他视觉感知任务的潜力。

Title: Refining Alignment Framework for Diffusion Models with Intermediate-Step Preference Ranking

• Authors: Jie Ren, Yuhang Zhang, Dongrui Liu, Xiaopeng Zhang, Qi Tian
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.01667
• Pdf URL: https://arxiv.org/pdf/2502.01667
• Copy Paste: [[2502.01667]] Refining Alignment Framework for Diffusion Models with Intermediate-Step Preference Ranking(https://arxiv.org/abs/2502.01667)
• Keywords: generation
• Abstract: Direct preference optimization (DPO) has shown success in aligning diffusion models with human preference. Previous approaches typically assume a consistent preference label between final generations and noisy samples at intermediate steps, and directly apply DPO to these noisy samples for fine-tuning. However, we theoretically identify inherent issues in this assumption and its impacts on the effectiveness of preference alignment. We first demonstrate the inherent issues from two perspectives: gradient direction and preference order, and then propose a Tailored Preference Optimization (TailorPO) framework for aligning diffusion models with human preference, underpinned by some theoretical insights. Our approach directly ranks intermediate noisy samples based on their step-wise reward, and effectively resolves the gradient direction issues through a simple yet efficient design. Additionally, we incorporate the gradient guidance of diffusion models into preference alignment to further enhance the optimization effectiveness. Experimental results demonstrate that our method significantly improves the model's ability to generate aesthetically pleasing and human-preferred images.
• 摘要：直接偏好优化 (DPO) 已成功将扩散模型与人类偏好对齐。以前的方法通常假设最终生成和中间步骤的噪声样本之间存在一致的偏好标签，并直接将 DPO 应用于这些噪声样本进行微调。然而，我们从理论上确定了这种假设的固有问题及其对偏好对齐有效性的影响。我们首先从梯度方向和偏好顺序两个角度展示了固有问题，然后提出了一个定制偏好优化 (TailorPO) 框架来将扩散模型与人类偏好对齐，并以一些理论见解为基础。我们的方法直接根据中间噪声样本的逐步奖励对其进行排序，并通过简单而有效的设计有效地解决了梯度方向问题。此外，我们将扩散模型的梯度指导纳入偏好对齐，以进一步增强优化效果。实验结果表明，我们的方法显着提高了模型生成美观且人类喜欢的图像的能力。

Title: Semantic Communication based on Generative AI: A New Approach to Image Compression and Edge Optimization

• Authors: Francesco Pezone
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.01675
• Pdf URL: https://arxiv.org/pdf/2502.01675
• Copy Paste: [[2502.01675]] Semantic Communication based on Generative AI: A New Approach to Image Compression and Edge Optimization(https://arxiv.org/abs/2502.01675)
• Keywords: generative
• Abstract: As digital technologies advance, communication networks face challenges in handling the vast data generated by intelligent devices. Autonomous vehicles, smart sensors, and IoT systems necessitate new paradigms. This thesis addresses these challenges by integrating semantic communication and generative models for optimized image compression and edge network resource allocation. Unlike bit-centric systems, semantic communication prioritizes transmitting meaningful data specifically selected to convey the meaning rather than obtain a faithful representation of the original data. The communication infrastructure can benefit to significant improvements in bandwidth efficiency and latency reduction. Central to this work is the design of semantic-preserving image compression using Generative Adversarial Networks and Denoising Diffusion Probabilistic Models. These models compress images by encoding only semantically relevant features, allowing for high-quality reconstruction with minimal transmission. Additionally, a Goal-Oriented edge network optimization framework is introduced, leveraging the Information Bottleneck principle and stochastic optimization to dynamically allocate resources and enhance efficiency. By integrating semantic communication into edge networks, this approach balances computational efficiency and communication effectiveness, making it suitable for real-time applications. The thesis compares semantic-aware models with conventional image compression techniques using classical and semantic evaluation metrics. Results demonstrate the potential of combining generative AI and semantic communication to create more efficient semantic-goal-oriented communication networks that meet the demands of modern data-driven applications.
• 摘要：随着数字技术的进步，通信网络在处理智能设备产生的大量数据方面面临挑战。自动驾驶汽车、智能传感器和物联网系统需要新的范式。本论文通过整合语义通信和生成模型来优化图像压缩和边缘网络资源分配，以应对这些挑战。与以比特为中心的系统不同，语义通信优先传输专门选择用来传达含义的有意义的数据，而不是获取原始数据的忠实表示。通信基础设施可以受益于带宽效率的显着提高和延迟的减少。这项工作的核心是使用生成对抗网络和去噪扩散概率模型设计保留语义的图像压缩。这些模型通过仅对语义相关特征进行编码来压缩图像，从而以最少的传输实现高质量的重建。此外，还引入了一个面向目标的边缘网络优化框架，利用信息瓶颈原理和随机优化来动态分配资源并提高效率。通过将语义通信集成到边缘网络中，这种方法平衡了计算效率和通信效率，使其适用于实时应用。这篇论文使用经典和语义评估指标将语义感知模型与传统图像压缩技术进行了比较。结果表明，将生成式人工智能与语义通信相结合可以创建更高效​​的面向语义目标的通信网络，以满足现代数据驱动型应用的需求。

Title: HuViDPO:Enhancing Video Generation through Direct Preference Optimization for Human-Centric Alignment

• Authors: Lifan Jiang, Boxi Wu, Jiahui Zhang, Xiaotong Guan, Shuang Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.01690
• Pdf URL: https://arxiv.org/pdf/2502.01690
• Copy Paste: [[2502.01690]] HuViDPO:Enhancing Video Generation through Direct Preference Optimization for Human-Centric Alignment(https://arxiv.org/abs/2502.01690)
• Keywords: generation
• Abstract: With the rapid development of AIGC technology, significant progress has been made in diffusion model-based technologies for text-to-image (T2I) and text-to-video (T2V). In recent years, a few studies have introduced the strategy of Direct Preference Optimization (DPO) into T2I tasks, significantly enhancing human preferences in generated images. However, existing T2V generation methods lack a well-formed pipeline with exact loss function to guide the alignment of generated videos with human preferences using DPO strategies. Additionally, challenges such as the scarcity of paired video preference data hinder effective model training. At the same time, the lack of training datasets poses a risk of insufficient flexibility and poor video generation quality in the generated videos. Based on those problems, our work proposes three targeted solutions in sequence. 1) Our work is the first to introduce the DPO strategy into the T2V tasks. By deriving a carefully structured loss function, we utilize human feedback to align video generation with human preferences. We refer to this new method as HuViDPO. 2) Our work constructs small-scale human preference datasets for each action category and fine-tune this model, improving the aesthetic quality of the generated videos while reducing training costs. 3) We adopt a First-Frame-Conditioned strategy, leveraging the rich in formation from the first frame to guide the generation of subsequent frames, enhancing flexibility in video generation. At the same time, we employ a SparseCausal Attention mechanism to enhance the quality of the generated this http URL details and examples can be accessed on our website: this https URL. this http URL.
• 摘要：随着AIGC技术的快速发展，基于扩散模型的文本转图像（T2I）和文本转视频（T2V）技术取得了重大进展。近年来，一些研究将直接偏好优化（DPO）策略引入到T2I任务中，显著增强了生成图像中的人类偏好。然而，现有的T2V生成方法缺乏一个结构良好且具有精确损失函数的流水线来指导使用DPO策略将生成的视频与人类偏好对齐。此外，配对视频偏好数据稀缺等挑战阻碍了有效的模型训练。同时，训练数据集的缺乏也带来了生成的视频灵活性不足和视频生成质量差的风险。基于这些问题，我们的工作依次提出了三个有针对性的解决方案。1）我们的工作首次将DPO策略引入到T2V任务中。通过推导一个精心构造的损失函数，我们利用人类反馈将视频生成与人类偏好对齐。我们将这种新方法称为HuViDPO。 2）我们的工作为每个动作类别构建小规模的人类偏好数据集并微调该模型，在降低训练成本的同时提高生成视频的美学质量。 3）我们采用首帧条件策略，利用第一帧的丰富信息来指导后续帧的生成，增强视频生成的灵活性。同时，我们采用 SparseCausal Attention 机制来提高生成的视频的质量。此 http URL 详细信息和示例可在我们的网站上访问：此 https URL。此 http URL。

Title: Fast Direct: Query-Efficient Online Black-box Guidance for Diffusion-model Target Generation

• Authors: Kim Yong Tan, Yueming Lyu, Ivor Tsang, Yew-Soon Ong
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.01692
• Pdf URL: https://arxiv.org/pdf/2502.01692
• Copy Paste: [[2502.01692]] Fast Direct: Query-Efficient Online Black-box Guidance for Diffusion-model Target Generation(https://arxiv.org/abs/2502.01692)
• Keywords: generation
• Abstract: Guided diffusion-model generation is a promising direction for customizing the generation process of a pre-trained diffusion-model to address the specific downstream tasks. Existing guided diffusion models either rely on training of the guidance model with pre-collected datasets or require the objective functions to be differentiable. However, for most real-world tasks, the offline datasets are often unavailable, and their objective functions are often not differentiable, such as image generation with human preferences, molecular generation for drug discovery, and material design. Thus, we need an \textbf{online} algorithm capable of collecting data during runtime and supporting a \textbf{black-box} objective function. Moreover, the \textbf{query efficiency} of the algorithm is also critical because the objective evaluation of the query is often expensive in the real-world scenarios. In this work, we propose a novel and simple algorithm, \textbf{Fast Direct}, for query-efficient online black-box target generation. Our Fast Direct builds a pseudo-target on the data manifold to update the noise sequence of the diffusion model with a universal direction, which is promising to perform query-efficient guided generation. Extensive experiments on twelve high-resolution ($\small {1024 \times 1024}$) image target generation tasks and six 3D-molecule target generation tasks show $\textbf{6}\times$ up to $\textbf{10}\times$ query efficiency improvement and $\textbf{11}\times$ up to $\textbf{44}\times$ query efficiency improvement, respectively. Our implementation is publicly available at: this https URL
• 摘要：引导扩散模型生成是一个有前途的方向，用于定制预训练扩散模型的生成过程以解决特定的下游任务。现有的引导扩散模型要么依赖于使用预先收集的数据集对引导模型进行训练，要么要求目标函数可微分。然而，对于大多数现实世界的任务，离线数据集通常不可用，并且它们的目标函数通常不可微分，例如具有人类偏好的图像生成、用于药物发现的分子生成和材料设计。因此，我们需要一种能够在运行时收集数据并支持 \textbf{黑盒} 目标函数的 \textbf{在线} 算法。此外，算法的 \textbf{查询效率} 也至关重要，因为在现实世界场景中查询的客观评估通常很昂贵。在这项工作中，我们提出了一种新颖而简单的算法 \textbf{Fast Direct}，用于查询高效的在线黑盒目标生成。我们的 Fast Direct 在数据流形上构建了一个伪目标，以通用方向更新扩散模型的噪声序列，有望执行查询高效的引导生成。在 12 个高分辨率 ($\small {1024 \times 1024}$) 图像目标生成任务和 6 个 3D 分子目标生成任务上进行的大量实验分别表明查询效率提高了 $\textbf{6}\times$ 到 $\textbf{10}\times$，查询效率提高了 $\textbf{11}\times$ 到 $\textbf{44}\times$。我们的实现公开可用：此 https URL

Title: EdgeMark: An Automation and Benchmarking System for Embedded Artificial Intelligence Tools

• Authors: Mohammad Amin Hasanpour, Mikkel Kirkegaard, Xenofon Fafoutis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.01700
• Pdf URL: https://arxiv.org/pdf/2502.01700
• Copy Paste: [[2502.01700]] EdgeMark: An Automation and Benchmarking System for Embedded Artificial Intelligence Tools(https://arxiv.org/abs/2502.01700)
• Keywords: generation
• Abstract: The integration of artificial intelligence (AI) into embedded devices, a paradigm known as embedded artificial intelligence (eAI) or tiny machine learning (TinyML), is transforming industries by enabling intelligent data processing at the edge. However, the many tools available in this domain leave researchers and developers wondering which one is best suited to their needs. This paper provides a review of existing eAI tools, highlighting their features, trade-offs, and limitations. Additionally, we introduce EdgeMark, an open-source automation system designed to streamline the workflow for deploying and benchmarking machine learning (ML) models on embedded platforms. EdgeMark simplifies model generation, optimization, conversion, and deployment while promoting modularity, reproducibility, and scalability. Experimental benchmarking results showcase the performance of widely used eAI tools, including TensorFlow Lite Micro (TFLM), Edge Impulse, Ekkono, and Renesas eAI Translator, across a wide range of models, revealing insights into their relative strengths and weaknesses. The findings provide guidance for researchers and developers in selecting the most suitable tools for specific application requirements, while EdgeMark lowers the barriers to adoption of eAI technologies.
• 摘要：将人工智能 (AI) 集成到嵌入式设备中，这种模式称为嵌入式人工智能 (eAI) 或微型机器学习 (TinyML)，它通过在边缘实现智能数据处理来改变行业。然而，该领域可用的工具众多，让研究人员和开发人员不知道哪一种最适合他们的需求。本文回顾了现有的 eAI 工具，重点介绍了它们的功能、权衡和局限性。此外，我们还介绍了 EdgeMark，这是一个开源自动化系统，旨在简化在嵌入式平台上部署和基准测试机器学习 (ML) 模型的工作流程。EdgeMark 简化了模型生成、优化、转换和部署，同时提高了模块化、可重复性和可扩展性。实验基准测试结果展示了广泛使用的 eAI 工具（包括 TensorFlow Lite Micro (TFLM)、Edge Impulse、Ekkono 和 Renesas eAI Translator）在各种模型中的性能，揭示了它们的相对优势和劣势。研究结果为研究人员和开发人员选择最适合特定应用需求的工具提供了指导，而 EdgeMark 降低了采用 eAI 技术的障碍。

Title: Al-Khwarizmi: Discovering Physical Laws with Foundation Models

• Authors: Christopher E. Mower, Haitham Bou-Ammar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.01702
• Pdf URL: https://arxiv.org/pdf/2502.01702
• Copy Paste: [[2502.01702]] Al-Khwarizmi: Discovering Physical Laws with Foundation Models(https://arxiv.org/abs/2502.01702)
• Keywords: generation
• Abstract: Inferring physical laws from data is a central challenge in science and engineering, including but not limited to healthcare, physical sciences, biosciences, social sciences, sustainability, climate, and robotics. Deep networks offer high-accuracy results but lack interpretability, prompting interest in models built from simple components. The Sparse Identification of Nonlinear Dynamics (SINDy) method has become the go-to approach for building such modular and interpretable models. SINDy leverages sparse regression with L1 regularization to identify key terms from a library of candidate functions. However, SINDy's choice of candidate library and optimization method requires significant technical expertise, limiting its widespread applicability. This work introduces Al-Khwarizmi, a novel agentic framework for physical law discovery from data, which integrates foundational models with SINDy. Leveraging LLMs, VLMs, and Retrieval-Augmented Generation (RAG), our approach automates physical law discovery, incorporating prior knowledge and iteratively refining candidate solutions via reflection. Al-Khwarizmi operates in two steps: it summarizes system observations-comprising textual descriptions, raw data, and plots-followed by a secondary step that generates candidate feature libraries and optimizer configurations to identify hidden physics laws correctly. Evaluating our algorithm on over 198 models, we demonstrate state-of-the-art performance compared to alternatives, reaching a 20 percent increase against the best-performing alternative.
• 摘要：从数据中推断物理定律是科学和工程领域面临的一个核心挑战，包括但不限于医疗保健、物理科学、生物科学、社会科学、可持续性、气候和机器人技术。深度网络提供高精度结果但缺乏可解释性，这引发了人们对由简单组件构建的模型的兴趣。稀疏非线性动力学识别 (SINDy) 方法已成为构建此类模块化和可解释模型的首选方法。SINDy 利用稀疏回归和 L1 正则化从候选函数库中识别关键术语。然而，SINDy 对候选库和优化方法的选择需要大量的技术专业知识，限制了其广泛的适用性。这项工作引入了 Al-Khwarizmi，这是一种用于从数据中发现物理定律的新型代理框架，它将基础模型与 SINDy 集成在一起。利用 LLM、VLM 和检索增强生成 (RAG)，我们的方法可以自动发现物理定律，结合先验知识并通过反射迭代地优化候选解决方案。 Al-Khwarizmi 分为两个步骤：首先总结系统观察结果（包括文本描述、原始数据和图表），然后是生成候选特征库和优化器配置以正确识别隐藏的物理定律的第二步。在超过 198 个模型上评估我们的算法后，我们展示了与其他替代方案相比最先进的性能，与表现最佳的替代方案相比，性能提高了 20%。

Title: CLIP-DQA: Blindly Evaluating Dehazed Images from Global and Local Perspectives Using CLIP

• Authors: Yirui Zeng, Jun Fu, Hadi Amirpour, Huasheng Wang, Guanghui Yue, Hantao Liu, Ying Chen, Wei Zhou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.01707
• Pdf URL: https://arxiv.org/pdf/2502.01707
• Copy Paste: [[2502.01707]] CLIP-DQA: Blindly Evaluating Dehazed Images from Global and Local Perspectives Using CLIP(https://arxiv.org/abs/2502.01707)
• Keywords: quality assessment
• Abstract: Blind dehazed image quality assessment (BDQA), which aims to accurately predict the visual quality of dehazed images without any reference information, is essential for the evaluation, comparison, and optimization of image dehazing algorithms. Existing learning-based BDQA methods have achieved remarkable success, while the small scale of DQA datasets limits their performance. To address this issue, in this paper, we propose to adapt Contrastive Language-Image Pre-Training (CLIP), pre-trained on large-scale image-text pairs, to the BDQA task. Specifically, inspired by the fact that the human visual system understands images based on hierarchical features, we take global and local information of the dehazed image as the input of CLIP. To accurately map the input hierarchical information of dehazed images into the quality score, we tune both the vision branch and language branch of CLIP with prompt learning. Experimental results on two authentic DQA datasets demonstrate that our proposed approach, named CLIP-DQA, achieves more accurate quality predictions over existing BDQA methods. The code is available at this https URL.
• 摘要：盲去雾图像质量评估 (BDQA) 旨在在没有任何参考信息的情况下准确预测去雾图像的视觉质量，对于图像去雾算法的评估、比较和优化至关重要。现有的基于学习的 BDQA 方法已经取得了显著的成功，而 DQA 数据集的规模限制了它们的性能。为了解决这个问题，在本文中，我们建议将对大规模图像-文本对进行预训练的对比语言-图像预训练 (CLIP) 应用于 BDQA 任务。具体而言，受人类视觉系统基于分层特征理解图像这一事实的启发，我们将去雾图像的全局和局部信息作为 CLIP 的输入。为了将去雾图像的输入分层信息准确地映射到质量得分中，我们通过快速学习调整了 CLIP 的视觉分支和语言分支。在两个真实的 DQA 数据集上进行的实验结果表明，我们提出的方法（CLIP-DQA）比现有的 BDQA 方法实现了更准确的质量预测。代码可从此 https URL 获取。

Title: MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generation

• Authors: Haibo Tong, Zhaoyang Wang, Zhaorun Chen, Haonian Ji, Shi Qiu, Siwei Han, Kexin Geng, Zhongkai Xue, Yiyang Zhou, Peng Xia, Mingyu Ding, Rafael Rafailov, Chelsea Finn, Huaxiu Yao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.01719
• Pdf URL: https://arxiv.org/pdf/2502.01719
• Copy Paste: [[2502.01719]] MJ-VIDEO: Fine-Grained Benchmarking and Rewarding Video Preferences in Video Generation(https://arxiv.org/abs/2502.01719)
• Keywords: generation
• Abstract: Recent advancements in video generation have significantly improved the ability to synthesize videos from text instructions. However, existing models still struggle with key challenges such as instruction misalignment, content hallucination, safety concerns, and bias. Addressing these limitations, we introduce MJ-BENCH-VIDEO, a large-scale video preference benchmark designed to evaluate video generation across five critical aspects: Alignment, Safety, Fineness, Coherence & Consistency, and Bias & Fairness. This benchmark incorporates 28 fine-grained criteria to provide a comprehensive evaluation of video preference. Building upon this dataset, we propose MJ-VIDEO, a Mixture-of-Experts (MoE)-based video reward model designed to deliver fine-grained reward. MJ-VIDEO can dynamically select relevant experts to accurately judge the preference based on the input text-video pair. This architecture enables more precise and adaptable preference judgments. Through extensive benchmarking on MJ-BENCH-VIDEO, we analyze the limitations of existing video reward models and demonstrate the superior performance of MJ-VIDEO in video preference assessment, achieving 17.58% and 15.87% improvements in overall and fine-grained preference judgments, respectively. Additionally, introducing MJ-VIDEO for preference tuning in video generation enhances the alignment performance.
• 摘要：视频生成领域的最新进展显著提高了从文本指令合成视频的能力。然而，现有模型仍然面临着指令错位、内容幻觉、安全问题和偏见等关键挑战。为了解决这些限制，我们推出了 MJ-BENCH-VIDEO，这是一个大规模视频偏好基准，旨在从五个关键方面评估视频生成：对齐、安全性、精细度、连贯性和一致性以及偏见和公平性。该基准结合了 28 个细粒度标准，以提供对视频偏好的全面评估。基于这个数据集，我们提出了 MJ-VIDEO，这是一个基于混合专家 (MoE) 的视频奖励模型，旨在提供细粒度的奖励。MJ-VIDEO 可以根据输入的文本-视频对动态选择相关专家来准确判断偏好。这种架构可以实现更精确、适应性更强的偏好判断。通过对 MJ-BENCH-VIDEO 进行广泛的基准测试，我们分析了现有视频奖励模型的局限性，并展示了 MJ-VIDEO 在视频偏好评估方面的卓越性能，在整体和细粒度偏好判断中分别实现了 17.58% 和 15.87% 的提升。此外，在视频生成中引入 MJ-VIDEO 进行偏好调整可增强对齐性能。

Title: Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity

• Authors: Haocheng Xi, Shuo Yang, Yilong Zhao, Chenfeng Xu, Muyang Li, Xiuyu Li, Yujun Lin, Han Cai, Jintao Zhang, Dacheng Li, Jianfei Chen, Ion Stoica, Kurt Keutzer, Song Han
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.01776
• Pdf URL: https://arxiv.org/pdf/2502.01776
• Copy Paste: [[2502.01776]] Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsity(https://arxiv.org/abs/2502.01776)
• Keywords: generation
• Abstract: Diffusion Transformers (DiTs) dominate video generation but their high computational cost severely limits real-world applicability, usually requiring tens of minutes to generate a few seconds of video even on high-performance GPUs. This inefficiency primarily arises from the quadratic computational complexity of 3D Full Attention with respect to the context length. In this paper, we propose a training-free framework termed Sparse VideoGen (SVG) that leverages the inherent sparsity in 3D Full Attention to boost inference efficiency. We reveal that the attention heads can be dynamically classified into two groups depending on distinct sparse patterns: (1) Spatial Head, where only spatially-related tokens within each frame dominate the attention output, and (2) Temporal Head, where only temporally-related tokens across different frames dominate. Based on this insight, SVG proposes an online profiling strategy to capture the dynamic sparse patterns and predicts the type of attention head. Combined with a novel hardware-efficient tensor layout transformation and customized kernel implementations, SVG achieves up to 2.28x and 2.33x end-to-end speedup on CogVideoX-v1.5 and HunyuanVideo, respectively, while preserving generation quality.
• 摘要：扩散变换器 (DiT) 在视频生成中占主导地位，但其高计算成本严重限制了现实世界的适用性，即使在高性能 GPU 上也通常需要数十分钟才能生成几秒钟的视频。这种低效率主要源于 3D 全注意力相对于上下文长度的二次计算复杂度。在本文中，我们提出了一个称为稀疏视频生成 (SVG) 的免训练框架，该框架利用 3D 全注意力中固有的稀疏性来提高推理效率。我们发现，根据不同的稀疏模式，注意力头可以动态地分为两类：(1) 空间头，其中只有每帧内的空间相关标记主导注意力输出，以及 (2) 时间头，其中只有不同帧之间的时间相关标记占主导地位。基于这一见解，SVG 提出了一种在线分析策略来捕获动态稀疏模式并预测注意力头的类型。结合新颖的硬件高效张量布局转换和定制的内核实现，SVG 在 CogVideoX-v1.5 和 HunyuanVideo 上分别实现了高达 2.28 倍和 2.33 倍的端到端加速，同时保持了生成质量。

Title: Self-supervised Subgraph Neural Network With Deep Reinforcement Walk Exploration

• Authors: Jianming Huang, Hiroyuki Kasai
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.01809
• Pdf URL: https://arxiv.org/pdf/2502.01809
• Copy Paste: [[2502.01809]] Self-supervised Subgraph Neural Network With Deep Reinforcement Walk Exploration(https://arxiv.org/abs/2502.01809)
• Keywords: generation
• Abstract: Graph data, with its structurally variable nature, represents complex real-world phenomena like chemical compounds, protein structures, and social networks. Traditional Graph Neural Networks (GNNs) primarily utilize the message-passing mechanism, but their expressive power is limited and their prediction lacks explainability. To address these limitations, researchers have focused on graph substructures. Subgraph neural networks (SGNNs) and GNN explainers have emerged as potential solutions, but each has its limitations. SGNNs computes graph representations based on the bags of subgraphs to enhance the expressive power. However, they often rely on predefined algorithm-based sampling strategies, which is inefficient. GNN explainers adopt data-driven approaches to generate important subgraphs to provide explanation. Nevertheless, their explanation is difficult to be translated into practical improvements on GNNs. To overcome these issues, we propose a novel self-supervised framework that integrates SGNNs with the generation approach of GNN explainers, named the Reinforcement Walk Exploration SGNN (RWE-SGNN). Our approach features a sampling model trained in an explainer fashion, optimizing subgraphs to enhance model performance. To achieve a data-driven sampling approach, unlike traditional subgraph generation approaches, we propose a novel walk exploration process, which efficiently extracts important substructures, simplifying the embedding process and avoiding isomorphism problems. Moreover, we prove that our proposed walk exploration process has equivalent generation capability to the traditional subgraph generation process. Experimental results on various graph datasets validate the effectiveness of our proposed method, demonstrating significant improvements in performance and precision.
• 摘要：图数据具有结构多变的性质，代表了复杂的现实世界现象，如化合物、蛋白质结构和社交网络。传统的图神经网络 (GNN) 主要利用消息传递机制，但其表达能力有限，预测缺乏可解释性。为了解决这些限制，研究人员将重点放在图子结构上。子图神经网络 (SGNN) 和 GNN 解释器已成为潜在的解决方案，但每种方法都有其局限性。SGNN 根据子图包计算图表示以增强表达能力。然而，它们通常依赖于预定义的基于算法的采样策略，这是低效的。GNN 解释器采用数据驱动的方法来生成重要的子图以提供解释。然而，它们的解释很难转化为 GNN 的实际改进。为了克服这些问题，我们提出了一种新颖的自监督框架，将 SGNN 与 GNN 解释器的生成方法相结合，称为强化步行探索 SGNN (RWE-SGNN)。我们的方法采用以解释器方式训练的采样模型，优化子图以提高模型性能。为了实现数据驱动的采样方法，与传统的子图生成方法不同，我们提出了一种新颖的步行探索过程，它可以有效地提取重要的子结构，简化嵌入过程并避免同构问题。此外，我们证明了我们提出的步行探索过程具有与传统子图生成过程相当的生成能力。在各种图数据集上的实验结果验证了我们提出的方法的有效性，表明性能和精度有显著的提高。

Title: PolyhedronNet: Representation Learning for Polyhedra with Surface-attributed Graph

• Authors: Dazhou Yu, Genpei Zhang, Liang Zhao
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.01814
• Pdf URL: https://arxiv.org/pdf/2502.01814
• Copy Paste: [[2502.01814]] PolyhedronNet: Representation Learning for Polyhedra with Surface-attributed Graph(https://arxiv.org/abs/2502.01814)
• Keywords: generation
• Abstract: Ubiquitous geometric objects can be precisely and efficiently represented as polyhedra. The transformation of a polyhedron into a vector, known as polyhedra representation learning, is crucial for manipulating these shapes with mathematical and statistical tools for tasks like classification, clustering, and generation. Recent years have witnessed significant strides in this domain, yet most efforts focus on the vertex sequence of a polyhedron, neglecting the complex surface modeling crucial in real-world polyhedral objects. This study proposes \textbf{PolyhedronNet}, a general framework tailored for learning representations of 3D polyhedral objects. We propose the concept of the surface-attributed graph to seamlessly model the vertices, edges, faces, and their geometric interrelationships within a polyhedron. To effectively learn the representation of the entire surface-attributed graph, we first propose to break it down into local rigid representations to effectively learn each local region's relative positions against the remaining regions without geometric information loss. Subsequently, we propose PolyhedronGNN to hierarchically aggregate the local rigid representation via intra-face and inter-face geometric message passing modules, to obtain a global representation that minimizes information loss while maintaining rotation and translation invariance. Our experimental evaluations on four distinct datasets, encompassing both classification and retrieval tasks, substantiate PolyhedronNet's efficacy in capturing comprehensive and informative representations of 3D polyhedral objects. Code and data are available at {this https URL}.
• 摘要：无处不在的几何对象可以精确而有效地表示为多面体。将多面体转换为矢量，称为多面体表示学习，对于使用数学和统计工具操纵这些形状以完成分类、聚类和生成等任务至关重要。近年来，这一领域取得了重大进展，但大多数努力都集中在多面体的顶点序列上，而忽略了现实世界多面体对象中至关重要的复杂表面建模。本研究提出了 \textbf{PolyhedronNet}，这是一个专为学习 3D 多面体对象表示而量身定制的通用框架。我们提出了表面属性图的概念，以无缝地建模多面体内的顶点、边、面及其几何相互关系。为了有效地学习整个表面属性图的表示，我们首先建议将其分解为局部刚性表示，以有效地学习每个局部区域相对于其余区域的相对位置，而不会丢失几何信息。随后，我们提出 PolyhedronGNN 通过界面内和界面间几何消息传递模块分层聚合局部刚性表示，以获得在保持旋转和平移不变性的同时最大程度减少信息损失的全局表示。我们对四个不同数据集（包括分类和检索任务）进行的实验评估证实了 PolyhedronNet 在捕获 3D 多面体对象的全面和信息丰富的表示方面的有效性。代码和数据可在 {此 https URL} 上找到。

Title: Low Resource Video Super-resolution using Memory and Residual Deformable Convolutions

• Authors: Kavitha Viswanathan, Shashwat Pathak, Piyush Bharambe, Harsh Choudhary, Amit Sethi
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.01816
• Pdf URL: https://arxiv.org/pdf/2502.01816
• Copy Paste: [[2502.01816]] Low Resource Video Super-resolution using Memory and Residual Deformable Convolutions(https://arxiv.org/abs/2502.01816)
• Keywords: super-resolution
• Abstract: Transformer-based video super-resolution (VSR) models have set new benchmarks in recent years, but their substantial computational demands make most of them unsuitable for deployment on resource-constrained devices. Achieving a balance between model complexity and output quality remains a formidable challenge in VSR. Although lightweight models have been introduced to address this issue, they often struggle to deliver state-of-the-art performance. We propose a novel lightweight, parameter-efficient deep residual deformable convolution network for VSR. Unlike prior methods, our model enhances feature utilization through residual connections and employs deformable convolution for precise frame alignment, addressing motion dynamics effectively. Furthermore, we introduce a single memory tensor to capture information accrued from the past frames and improve motion estimation across frames. This design enables an efficient balance between computational cost and reconstruction quality. With just 2.3 million parameters, our model achieves state-of-the-art SSIM of 0.9175 on the REDS4 dataset, surpassing existing lightweight and many heavy models in both accuracy and resource efficiency. Architectural insights from our model pave the way for real-time VSR on streaming data.
• 摘要：近年来，基于 Transformer 的视频超分辨率 (VSR) 模型树立了新的标杆，但它们巨大的计算需求使大多数模型不适合部署在资源受限的设备上。在 VSR 中，实现模型复杂性和输出质量之间的平衡仍然是一项艰巨的挑战。尽管已经引入了轻量级模型来解决这个问题，但它们往往难以提供最先进的性能。我们为 VSR 提出了一种新颖的轻量级、参数高效的深度残差可变形卷积网络。与之前的方法不同，我们的模型通过残差连接增强了特征利用率，并采用可变形卷积进行精确的帧对齐，从而有效地解决了运动动态问题。此外，我们引入了一个单一的记忆张量来捕获从过去帧中积累的信息并改进跨帧的运动估计。这种设计可以在计算成本和重建质量之间实现有效的平衡。仅使用 230 万个参数，我们的模型在 REDS4 数据集上实现了 0.9175 的最先进的 SSIM，在准确性和资源效率方面都超越了现有的轻量级和许多重型模型。我们模型的架构见解为流数据的实时 VSR 铺平了道路。

Title: Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning

• Authors: Hanyang Zhao, Haoxian Chen, Ji Zhang, David D. Yao, Wenpin Tang
• Subjects: cs.LG, cs.AI, math.OC
• Abstract URL: https://arxiv.org/abs/2502.01819
• Pdf URL: https://arxiv.org/pdf/2502.01819
• Copy Paste: [[2502.01819]] Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning(https://arxiv.org/abs/2502.01819)
• Keywords: generative
• Abstract: Reinforcement learning from human feedback (RLHF), which aligns a diffusion model with input prompt, has become a crucial step in building reliable generative AI models. Most works in this area use a discrete-time formulation, which is prone to induced errors, and often not applicable to models with higher-order/black-box solvers. The objective of this study is to develop a disciplined approach to fine-tune diffusion models using continuous-time RL, formulated as a stochastic control problem with a reward function that aligns the end result (terminal state) with input prompt. The key idea is to treat score matching as controls or actions, and thereby making connections to policy optimization and regularization in continuous-time RL. To carry out this idea, we lay out a new policy optimization framework for continuous-time RL, and illustrate its potential in enhancing the value networks design space via leveraging the structural property of diffusion models. We validate the advantages of our method by experiments in downstream tasks of fine-tuning large-scale Text2Image models of Stable Diffusion v1.5.
• 摘要：人类反馈强化学习 (RLHF) 将扩散模型与输入提示对齐，已成为构建可靠的生成式 AI 模型的关键步骤。该领域的大多数工作都使用离散时间公式，这容易导致错误，并且通常不适用于具有高阶/黑盒求解器的模型。本研究的目的是开发一种使用连续时间 RL 微调扩散模型的规范方法，该方法被表述为一个随机控制问题，具有将最终结果（终端状态）与输入提示对齐的奖励函数。关键思想是将分数匹配视为控制或动作，从而与连续时间 RL 中的策略优化和正则化建立联系。为了实现这个想法，我们为连续时间 RL 制定了一个新的策略优化框架，并说明了它通过利用扩散模型的结构特性来增强价值网络设计空间的潜力。我们通过对 Stable Diffusion v1.5 的大规模 Text2Image 模型进行微调的下游任务中的实验来验证我们方法的优势。

Title: Texture Image Synthesis Using Spatial GAN Based on Vision Transformers

• Authors: Elahe Salari, Zohreh Azimifar
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.01842
• Pdf URL: https://arxiv.org/pdf/2502.01842
• Copy Paste: [[2502.01842]] Texture Image Synthesis Using Spatial GAN Based on Vision Transformers(https://arxiv.org/abs/2502.01842)
• Keywords: generative
• Abstract: Texture synthesis is a fundamental task in computer vision, whose goal is to generate visually realistic and structurally coherent textures for a wide range of applications, from graphics to scientific simulations. While traditional methods like tiling and patch-based techniques often struggle with complex textures, recent advancements in deep learning have transformed this field. In this paper, we propose ViT-SGAN, a new hybrid model that fuses Vision Transformers (ViTs) with a Spatial Generative Adversarial Network (SGAN) to address the limitations of previous methods. By incorporating specialized texture descriptors such as mean-variance (mu, sigma) and textons into the self-attention mechanism of ViTs, our model achieves superior texture synthesis. This approach enhances the model's capacity to capture complex spatial dependencies, leading to improved texture quality that is superior to state-of-the-art models, especially for regular and irregular textures. Comparison experiments with metrics such as FID, IS, SSIM, and LPIPS demonstrate the substantial improvement of ViT-SGAN, which underlines its efficiency in generating diverse realistic textures.
• 摘要：纹理合成是计算机视觉中的一项基本任务，其目标是为从图形到科学模拟等广泛应用生成视觉逼真、结构一致的纹理。虽然传统方法（如平铺和基于块的技术）通常难以处理复杂纹理，但深度学习的最新进展已经改变了这一领域。在本文中，我们提出了 ViT-SGAN，这是一种新的混合模型，它将视觉变换器 (ViT) 与空间生成对抗网络 (SGAN) 融合在一起，以解决以前方法的局限性。通过将均值-方差（mu、sigma）和纹理元等专门的纹理描述符合并到 ViT 的自注意力机制中，我们的模型实现了卓越的纹理合成。这种方法增强了模型捕获复杂空间依赖性的能力，从而提高了纹理质量，优于最先进的模型，尤其是对于规则和不规则纹理。与 FID、IS、SSIM 和 LPIPS 等指标的比较实验表明 ViT-SGAN 有了显著的改进，凸显了其在生成多样化真实纹理方面的效率。

Title: UVGS: Reimagining Unstructured 3D Gaussian Splatting using UV Mapping

• Authors: Aashish Rai, Dilin Wang, Mihir Jain, Nikolaos Sarafianos, Arthur Chen, Srinath Sridhar, Aayush Prakash
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.01846
• Pdf URL: https://arxiv.org/pdf/2502.01846
• Copy Paste: [[2502.01846]] UVGS: Reimagining Unstructured 3D Gaussian Splatting using UV Mapping(https://arxiv.org/abs/2502.01846)
• Keywords: generation
• Abstract: 3D Gaussian Splatting (3DGS) has demonstrated superior quality in modeling 3D objects and scenes. However, generating 3DGS remains challenging due to their discrete, unstructured, and permutation-invariant nature. In this work, we present a simple yet effective method to overcome these challenges. We utilize spherical mapping to transform 3DGS into a structured 2D representation, termed UVGS. UVGS can be viewed as multi-channel images, with feature dimensions as a concatenation of Gaussian attributes such as position, scale, color, opacity, and rotation. We further find that these heterogeneous features can be compressed into a lower-dimensional (e.g., 3-channel) shared feature space using a carefully designed multi-branch network. The compressed UVGS can be treated as typical RGB images. Remarkably, we discover that typical VAEs trained with latent diffusion models can directly generalize to this new representation without additional training. Our novel representation makes it effortless to leverage foundational 2D models, such as diffusion models, to directly model 3DGS. Additionally, one can simply increase the 2D UV resolution to accommodate more Gaussians, making UVGS a scalable solution compared to typical 3D backbones. This approach immediately unlocks various novel generation applications of 3DGS by inherently utilizing the already developed superior 2D generation capabilities. In our experiments, we demonstrate various unconditional, conditional generation, and inpainting applications of 3DGS based on diffusion models, which were previously non-trivial.
• 摘要：3D 高斯分层 (3DGS) 在建模 3D 对象和场景方面表现出色。然而，由于 3DGS 具有离散、非结构化和置换不变性，因此生成 3DGS 仍然具有挑战性。在这项工作中，我们提出了一种简单而有效的方法来克服这些挑战。我们利用球面映射将 3DGS 转换为结构化的 2D 表示，称为 UVGS。UVGS 可以被视为多通道图像，其特征维度是高斯属性（例如位置、比例、颜色、不透明度和旋转）的串联。我们进一步发现，可以使用精心设计的多分支网络将这些异构特征压缩到低维（例如 3 通道）共享特征空间中。压缩后的 UVGS 可以视为典型的 RGB 图像。值得注意的是，我们发现使用潜在扩散模型训练的典型 VAE 可以直接推广到这种新表示，而无需额外的训练。我们新颖的表示方法使得利用基础 2D 模型（例如扩散模型）直接建模 3DGS 变得毫不费力。此外，只需增加 2D UV 分辨率即可容纳更多高斯分布，这使得 UVGS 成为与典型 3D 主干相比可扩展的解决方案。这种方法通过固有利用已经开发的卓越 2D 生成功能，立即解锁了 3DGS 的各种新型生成应用。在我们的实验中，我们展示了基于扩散模型的 3DGS 的各种无条件、条件生成和修复应用，这些应用以前并不简单。

Title: SimBEV: A Synthetic Multi-Task Multi-Sensor Driving Data Generation Tool and Dataset

• Authors: Goodarz Mehr, Azim Eskandarian
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2502.01894
• Pdf URL: https://arxiv.org/pdf/2502.01894
• Copy Paste: [[2502.01894]] SimBEV: A Synthetic Multi-Task Multi-Sensor Driving Data Generation Tool and Dataset(https://arxiv.org/abs/2502.01894)
• Keywords: generation
• Abstract: Bird's-eye view (BEV) perception for autonomous driving has garnered significant attention in recent years, in part because BEV representation facilitates the fusion of multi-sensor data. This enables a variety of perception tasks including BEV segmentation, a concise view of the environment that can be used to plan a vehicle's trajectory. However, this representation is not fully supported by existing datasets, and creation of new datasets can be a time-consuming endeavor. To address this problem, in this paper we introduce SimBEV, an extensively configurable and scalable randomized synthetic data generation tool that incorporates information from multiple sources to capture accurate BEV ground truth data, supports a comprehensive array of sensors, and enables a variety of perception tasks including BEV segmentation and 3D object detection. We use SimBEV to create the SimBEV dataset, a large collection of annotated perception data from diverse driving scenarios.
• 摘要：近年来，自动驾驶的鸟瞰图 (BEV) 感知引起了广泛关注，部分原因是 BEV 表示有助于多传感器数据的融合。这使得各种感知任务成为可能，包括 BEV 分割、可用于规划车辆轨迹的环境简明视图。但是，现有数据集并不完全支持这种表示，而创建新数据集可能是一项耗时的工作。为了解决这个问题，我们在本文中介绍了 SimBEV，这是一种可广泛配置和可扩展的随机合成数据生成工具，它结合了来自多个来源的信息以捕获准确的 BEV 地面实况数据，支持全面的传感器阵列，并支持各种感知任务，包括 BEV 分割和 3D 物体检测。我们使用 SimBEV 创建 SimBEV 数据集，这是来自不同驾驶场景的大量带注释的感知数据集合。

Title: Anomaly Detection via Autoencoder Composite Features and NCE

• Authors: Yalin Liao, Austin J. Brockmeier
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.01920
• Pdf URL: https://arxiv.org/pdf/2502.01920
• Copy Paste: [[2502.01920]] Anomaly Detection via Autoencoder Composite Features and NCE(https://arxiv.org/abs/2502.01920)
• Keywords: generative
• Abstract: Unsupervised anomaly detection is a challenging task. Autoencoders (AEs) or generative models are often employed to model the data distribution of normal inputs and subsequently identify anomalous, out-of-distribution inputs by high reconstruction error or low likelihood, respectively. However, AEs may generalize and achieve small reconstruction errors on abnormal inputs. We propose a decoupled training approach for anomaly detection that both an AE and a likelihood model trained with noise contrastive estimation (NCE). After training the AE, NCE estimates a probability density function, to serve as the anomaly score, on the joint space of the AE's latent representation combined with features of the reconstruction quality. To further reduce the false negative rate in NCE we systematically varying the reconstruction features to augment the training and optimize the contrastive Gaussian noise distribution. Experimental assessments on multiple benchmark datasets demonstrate that the proposed approach matches the performance of prevalent state-of-the-art anomaly detection algorithms.
• 摘要：无监督异常检测是一项具有挑战性的任务。自动编码器 (AE) 或生成模型通常用于对正常输入的数据分布进行建模，然后分别通过高重构误差或低似然度识别异常、分布外的输入。然而，AE 可以推广并在异常输入上实现较小的重构误差。我们提出了一种用于异常检测的解耦训练方法，该方法既使用噪声对比估计 (NCE) 训练 AE 和似然模型。在训练 AE 之后，NCE 在 AE 的潜在表示与重构质量特征相结合的联合空间上估计一个概率密度函数，作为异常分数。为了进一步降低 NCE 中的假阴性率，我们系统地改变重构特征以增强训练并优化对比高斯噪声分布​​。在多个基准数据集上的实验评估表明，所提出的方法与流行的最先进异常检测算法的性能相匹配。

Title: Toward a Low-Cost Perception System in Autonomous Vehicles: A Spectrum Learning Approach

• Authors: Mohammed Alsakabi, Aidan Erickson, John M. Dolan, Ozan K. Tonguz
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2502.01940
• Pdf URL: https://arxiv.org/pdf/2502.01940
• Copy Paste: [[2502.01940]] Toward a Low-Cost Perception System in Autonomous Vehicles: A Spectrum Learning Approach(https://arxiv.org/abs/2502.01940)
• Keywords: generative
• Abstract: We present a cost-effective new approach for generating denser depth maps for Autonomous Driving (AD) and Autonomous Vehicles (AVs) by integrating the images obtained from deep neural network (DNN) 4D radar detectors with conventional camera RGB images. Our approach introduces a novel pixel positional encoding algorithm inspired by Bartlett's spatial spectrum estimation technique. This algorithm transforms both radar depth maps and RGB images into a unified pixel image subspace called the Spatial Spectrum, facilitating effective learning based on their similarities and differences. Our method effectively leverages high-resolution camera images to train radar depth map generative models, addressing the limitations of conventional radar detectors in complex vehicular environments, thus sharpening the radar output. We develop spectrum estimation algorithms tailored for radar depth maps and RGB images, a comprehensive training framework for data-driven generative models, and a camera-radar deployment scheme for AV operation. Our results demonstrate that our approach also outperforms the state-of-the-art (SOTA) by 27.95% in terms of Unidirectional Chamfer Distance (UCD).
• 摘要：我们提出了一种经济高效的新方法，通过将深度神经网络 (DNN) 4D 雷达探测器获得的图像与传统摄像机 RGB 图像相结合，为自动驾驶 (AD) 和自动驾驶汽车 (AV) 生成更密集的深度图。我们的方法引入了一种受 Bartlett 空间谱估计技术启发的新型像素位置编码算法。该算法将雷达深度图和 RGB 图像转换为称为空间谱的统一像素图像子空间，从而促进基于它们的相似性和差异性的有效学习。我们的方法有效地利用高分辨率摄像机图像来训练雷达深度图生成模型，解决了传统雷达探测器在复杂车辆环境中的局限性，从而提高了雷达输出。我们开发了针对雷达深度图和 RGB 图像的频谱估计算法、数据驱动生成模型的综合训练框架以及用于 AV 操作的摄像机雷达部署方案。我们的结果表明，我们的方法在单向倒角距离 (UCD) 方面也比最先进方法 (SOTA) 高出 27.95%。

Title: LAYOUTDREAMER: Physics-guided Layout for Text-to-3D Compositional Scene Generation

• Authors: Yang Zhou, Zongjin He, Qixuan Li, Chao Wang
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2502.01949
• Pdf URL: https://arxiv.org/pdf/2502.01949
• Copy Paste: [[2502.01949]] LAYOUTDREAMER: Physics-guided Layout for Text-to-3D Compositional Scene Generation(https://arxiv.org/abs/2502.01949)
• Keywords: generation
• Abstract: Recently, the field of text-guided 3D scene generation has garnered significant attention. High-quality generation that aligns with physical realism and high controllability is crucial for practical 3D scene applications. However, existing methods face fundamental limitations: (i) difficulty capturing complex relationships between multiple objects described in the text, (ii) inability to generate physically plausible scene layouts, and (iii) lack of controllability and extensibility in compositional scenes. In this paper, we introduce LayoutDreamer, a framework that leverages 3D Gaussian Splatting (3DGS) to facilitate high-quality, physically consistent compositional scene generation guided by text. Specifically, given a text prompt, we convert it into a directed scene graph and adaptively adjust the density and layout of the initial compositional 3D Gaussians. Subsequently, dynamic camera adjustments are made based on the training focal point to ensure entity-level generation quality. Finally, by extracting directed dependencies from the scene graph, we tailor physical and layout energy to ensure both realism and flexibility. Comprehensive experiments demonstrate that LayoutDreamer outperforms other compositional scene generation quality and semantic alignment methods. Specifically, it achieves state-of-the-art (SOTA) performance in the multiple objects generation metric of T3Bench.
• 摘要：最近，文本引导的 3D 场景生成领域引起了广泛关注。符合物理真实感和高可控性的高质量生成对于实际的 3D 场景应用至关重要。然而，现有的方法面临着根本性的限制：(i) 难以捕捉文本中描述的多个对象之间的复杂关系，(ii) 无法生成物理上合理的场景布局，以及 (iii) 缺乏合成场景的可控性和可扩展性。在本文中，我们介绍了 LayoutDreamer，这是一个利用 3D Gaussian Splatting (3DGS) 来促进高质量、物理上一致的合成场景生成的框架，由文本引导。具体来说，给定一个文本提示，我们将其转换为有向场景图，并自适应地调整初始合成 3D 高斯的密度和布局。随后，根据训练焦点进行动态相机调整，以确保实体级生成质量。最后，通过从场景图中提取有向依赖关系，我们定制物理和布局能量以确保真实感和灵活性。综合实验表明，LayoutDreamer 的表现优于其他构图场景生成质量和语义对齐方法。具体而言，它在 T3Bench 的多对象生成指标中实现了最佳 (SOTA) 性能。

Title: MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Serving

• Authors: Shiju Zhao, Junhao Hu, Rongxiao Huang, Jiaqi Zheng, Guihai Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.01960
• Pdf URL: https://arxiv.org/pdf/2502.01960
• Copy Paste: [[2502.01960]] MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Serving(https://arxiv.org/abs/2502.01960)
• Keywords: generation
• Abstract: The context caching technique is employed to accelerate the Multimodal Large Language Model (MLLM) inference by prevailing serving platforms currently. However, this approach merely reuses the Key-Value (KV) cache of the initial sequence of prompt, resulting in full KV cache recomputation even if the prefix differs slightly. This becomes particularly inefficient in the context of interleaved text and images, as well as multimodal retrieval-augmented generation. This paper proposes position-independent caching as a more effective approach for multimodal information management. We have designed and implemented a caching system, named MPIC, to address both system-level and algorithm-level challenges. MPIC stores the KV cache on local or remote disks when receiving multimodal data, and calculates and loads the KV cache in parallel during inference. To mitigate accuracy degradation, we have incorporated integrated reuse and recompute mechanisms within the system. The experimental results demonstrate that MPIC can achieve up to 54% reduction in response time compared to existing context caching systems, while maintaining negligible or no accuracy loss.
• 摘要：目前主流的服务平台均采用上下文缓存技术来加速多模态大型语言模型 (MLLM) 推理。然而，这种方法仅仅重用了初始提示序列的键值 (KV) 缓存，即使前缀稍有不同，也会导致整个 KV 缓存重新计算。这在交错文本和图像以及多模态检索增强生成的背景下尤其低效。本文提出了位置无关缓存作为一种更有效的多模态信息管理方法。我们设计并实现了一个名为 MPIC 的缓存系统，以解决系统级和算法级的挑战。MPIC 在接收多模态数据时将 KV 缓存存储在本地或远程磁盘上，并在推理过程中并行计算和加载 KV 缓存。为了减轻准确度下降的影响，我们在系统中加入了集成的重用和重新计算机制。实验结果表明，与现有的上下文缓存系统相比，MPIC 可以将响应时间缩短高达 54%，同时保持准确性损失可忽略不计或没有损失。

Title: AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs

• Authors: Hongxin Li, Jingfan Chen, Jingran Su, Yuntao Chen, Qing Li, Zhaoxiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.01977
• Pdf URL: https://arxiv.org/pdf/2502.01977
• Copy Paste: [[2502.01977]] AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs(https://arxiv.org/abs/2502.01977)
• Keywords: generation
• Abstract: User interface understanding with vision-language models has received much attention due to its potential for enabling next-generation software automation. However, existing UI datasets either only provide large-scale context-free element annotations or contextualized functional descriptions for elements at a much smaller scale. In this work, we propose the \methodname{} pipeline for automatically annotating UI elements with detailed functionality descriptions at scale. Specifically, we leverage large language models (LLMs) to infer element functionality by comparing the UI content changes before and after simulated interactions with specific UI elements. To improve annotation quality, we propose LLM-aided rejection and verification, eliminating invalid and incorrect annotations without human labor. We construct an \methodname{}-704k dataset using the proposed pipeline, featuring multi-resolution, multi-device screenshots, diverse data domains, and detailed functionality annotations that have never been provided by previous datasets. Human evaluation shows that the AutoGUI pipeline achieves annotation correctness comparable to trained human annotators. Extensive experimental results show that our \methodname{}-704k dataset remarkably enhances VLM's UI grounding capabilities, exhibits significant scaling effects, and outperforms existing web pre-training data types. We envision AutoGUI as a scalable pipeline for generating massive data to build GUI-oriented VLMs. AutoGUI dataset can be viewed at this anonymous URL: this https URL.
• 摘要：由于具有实现下一代软件自动化的潜力，使用视觉语言模型进行用户界面理解已受到广泛关注。然而，现有的 UI 数据集要么只提供大规模的上下文无关元素注释，要么只提供小规模元素的上下文化功能描述。在这项工作中，我们提出了 \methodname{} 流程，用于自动大规模注释 UI 元素并提供详细的功能描述。具体来说，我们利用大型语言模型 (LLM) 通过比较与特定 UI 元素模拟交互前后的 UI 内容变化来推断元素功能。为了提高注释质量，我们提出了 LLM 辅助拒绝和验证，从而无需人工即可消除无效和不正确的注释。我们使用所提出的流程构建了一个 \methodname{}-704k 数据集，该数据集具有多分辨率、多设备屏幕截图、多样化数据域和以前的数据集从未提供过的详细功能注释。人工评估表明，AutoGUI 流程实现了与训练有素的人工注释者相当的注释正确性。大量实验结果表明，我们的 \methodname{}-704k 数据集显著增强了 VLM 的 UI 基础能力，表现出显著的扩展效果，并且优于现有的 Web 预训练数据类型。我们将 AutoGUI 设想为一个可扩展的管道，用于生成大量数据以构建面向 GUI 的 VLM。AutoGUI 数据集可在此匿名 URL 上查看：此 https URL。

Title: Generative Data Mining with Longtail-Guided Diffusion

• Authors: David S. Hayden, Mao Ye, Timur Garipov, Gregory P. Meyer, Carl Vondrick, Zhao Chen, Yuning Chai, Eric Wolff, Siddhartha S. Srinivasa
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.01980
• Pdf URL: https://arxiv.org/pdf/2502.01980
• Copy Paste: [[2502.01980]] Generative Data Mining with Longtail-Guided Diffusion(https://arxiv.org/abs/2502.01980)
• Keywords: generative
• Abstract: It is difficult to anticipate the myriad challenges that a predictive model will encounter once deployed. Common practice entails a reactive, cyclical approach: model deployment, data mining, and retraining. We instead develop a proactive longtail discovery process by imagining additional data during training. In particular, we develop general model-based longtail signals, including a differentiable, single forward pass formulation of epistemic uncertainty that does not impact model parameters or predictive performance but can flag rare or hard inputs. We leverage these signals as guidance to generate additional training data from a latent diffusion model in a process we call Longtail Guidance (LTG). Crucially, we can perform LTG without retraining the diffusion model or the predictive model, and we do not need to expose the predictive model to intermediate diffusion states. Data generated by LTG exhibit semantically meaningful variation, yield significant generalization improvements on image classification benchmarks, and can be analyzed to proactively discover, explain, and address conceptual gaps in a predictive model.
• 摘要：很难预测部署预测模型后会遇到的无数挑战。常见的做法是采用被动的、周期性的方法：模型部署、数据挖掘和再训练。相反，我们通过在训练期间想象额外的数据来开发主动的长尾发现过程。具体来说，我们开发了基于模型的通用长尾信号，包括可微分的、单次前向传递的认知不确定性公式，它不会影响模型参数或预测性能，但可以标记罕见或困难的输入。我们利用这些信号作为指导，从潜在扩散模型生成额外的训练数据，这个过程我们称之为长尾指导 (LTG)。至关重要的是，我们可以在不重新训练扩散模型或预测模型的情况下执行 LTG，并且我们不需要将预测模型暴露于中间扩散状态。LTG 生成的数据表现出语义上有意义的变化，在图像分类基准上产生显着的泛化改进，并且可以进行分析以主动发现、解释和解决预测模型中的概念差距。

Title: Rethinking Timesteps Samplers and Prediction Types

• Authors: Bin Xie, Gady Agam
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.01990
• Pdf URL: https://arxiv.org/pdf/2502.01990
• Copy Paste: [[2502.01990]] Rethinking Timesteps Samplers and Prediction Types(https://arxiv.org/abs/2502.01990)
• Keywords: generative
• Abstract: Diffusion models suffer from the huge consumption of time and resources to train. For example, diffusion models need hundreds of GPUs to train for several weeks for a high-resolution generative task to meet the requirements of an extremely large number of iterations and a large batch size. Training diffusion models become a millionaire's game. With limited resources that only fit a small batch size, training a diffusion model always fails. In this paper, we investigate the key reasons behind the difficulties of training diffusion models with limited resources. Through numerous experiments and demonstrations, we identified a major factor: the significant variation in the training losses across different timesteps, which can easily disrupt the progress made in previous iterations. Moreover, different prediction types of $x_0$ exhibit varying effectiveness depending on the task and timestep. We hypothesize that using a mixed-prediction approach to identify the most accurate $x_0$ prediction type could potentially serve as a breakthrough in addressing this issue. In this paper, we outline several challenges and insights, with the hope of inspiring further research aimed at tackling the limitations of training diffusion models with constrained resources, particularly for high-resolution tasks.
• 摘要：扩散模型的训练耗费大量时间和资源。例如，为了完成高分辨率生成任务，扩散模型需要数百个 GPU 进行数周的训练，才能满足极大量迭代和大批量的要求。训练扩散模型成为百万富翁的游戏。由于资源有限，只能适应小批量，训练扩散模型总是会失败。在本文中，我们探讨了在有限资源下训练扩散模型困难的主要原因。通过大量的实验和演示，我们发现了一个主要因素：不同时间步的训练损失存在显著差异，这很容易破坏前几次迭代的进展。此外，不同类型的 $x_0$ 预测效果会因任务和时间步的不同而不同。我们假设使用混合预测方法来识别最准确的 $x_0$ 预测类型可能会成为解决这一问题的突破。在本文中，我们概述了一些挑战和见解，希望能够激发进一步的研究，旨在解决使用受限资源训练扩散模型的局限性，特别是对于高分辨率任务。

Title: One Diffusion Step to Real-World Super-Resolution via Flow Trajectory Distillation

• Authors: Jianze Li, Jiezhang Cao, Yong Guo, Wenbo Li, Yulun Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.01993
• Pdf URL: https://arxiv.org/pdf/2502.01993
• Copy Paste: [[2502.01993]] One Diffusion Step to Real-World Super-Resolution via Flow Trajectory Distillation(https://arxiv.org/abs/2502.01993)
• Keywords: super-resolution
• Abstract: Diffusion models (DMs) have significantly advanced the development of real-world image super-resolution (Real-ISR), but the computational cost of multi-step diffusion models limits their application. One-step diffusion models generate high-quality images in a one sampling step, greatly reducing computational overhead and inference latency. However, most existing one-step diffusion methods are constrained by the performance of the teacher model, where poor teacher performance results in image artifacts. To address this limitation, we propose FluxSR, a novel one-step diffusion Real-ISR technique based on flow matching models. We use the state-of-the-art diffusion model FLUX.1-dev as both the teacher model and the base model. First, we introduce Flow Trajectory Distillation (FTD) to distill a multi-step flow matching model into a one-step Real-ISR. Second, to improve image realism and address high-frequency artifact issues in generated images, we propose TV-LPIPS as a perceptual loss and introduce Attention Diversification Loss (ADL) as a regularization term to reduce token similarity in transformer, thereby eliminating high-frequency artifacts. Comprehensive experiments demonstrate that our method outperforms existing one-step diffusion-based Real-ISR methods. The code and model will be released at this https URL.
• 摘要：扩散模型 (DM) 极大地推动了真实世界图像超分辨率 (Real-ISR) 的发展，但多步扩散模型的计算成本限制了它们的应用。一步扩散模型在一个采样步骤中生成高质量图像，大大减少了计算开销和推理延迟。然而，大多数现有的一步扩散方法都受到教师模型性能的限制，教师模型性能不佳会导致图像伪影。为了解决这一限制，我们提出了 FluxSR，一种基于流匹配模型的新型一步扩散 Real-ISR 技术。我们使用最先进的扩散模型 FLUX.1-dev 作为教师模型和基础模型。首先，我们引入流轨迹蒸馏 (FTD) 将多步流匹配模型蒸馏为一步 Real-ISR。其次，为了提高图像真实感并解决生成图像中的高频伪影问题，我们提出了 TV-LPIPS 作为感知损失，并引入注意力多样化损失 (ADL) 作为正则化项，以降低 Transformer 中的 token 相似性，从而消除高频伪影。综合实验表明，我们的方法优于现有的基于一步扩散的 Real-ISR 方法。代码和模型将在此 https URL 上发布。

Title: Analytical Lyapunov Function Discovery: An RL-based Generative Approach

• Authors: Haohan Zou, Jie Feng, Hao Zhao, Yuanyuan Shi
• Subjects: cs.LG, cs.AI, cs.SC, eess.SY
• Abstract URL: https://arxiv.org/abs/2502.02014
• Pdf URL: https://arxiv.org/pdf/2502.02014
• Copy Paste: [[2502.02014]] Analytical Lyapunov Function Discovery: An RL-based Generative Approach(https://arxiv.org/abs/2502.02014)
• Keywords: generative
• Abstract: Despite advances in learning-based methods, finding valid Lyapunov functions for nonlinear dynamical systems remains challenging. Current neural network approaches face two main issues: challenges in scalable verification and limited interpretability. To address these, we propose an end-to-end framework using transformers to construct analytical Lyapunov functions (local), which simplifies formal verification, enhances interpretability, and provides valuable insights for control engineers. Our framework consists of a transformer-based trainer that generates candidate Lyapunov functions and a falsifier that verifies candidate expressions and refines the model via risk-seeking policy gradient. Unlike Alfarano et al. (2024), which utilizes pre-training and seeks global Lyapunov functions for low-dimensional systems, our model is trained from scratch via reinforcement learning (RL) and succeeds in finding local Lyapunov functions for high-dimensional and non-polynomial systems. Given the analytical nature of the candidates, we employ efficient optimization methods for falsification during training and formal verification tools for the final verification. We demonstrate the efficiency of our approach on a range of nonlinear dynamical systems with up to ten dimensions and show that it can discover Lyapunov functions not previously identified in the control literature.
• 摘要：尽管基于学习的方法取得了进展，但为非线性动态系统找到有效的 Lyapunov 函数仍然具有挑战性。当前的神经网络方法面临两个主要问题：可扩展验证的挑战和有限的可解释性。为了解决这些问题，我们提出了一个端到端框架，使用转换器来构建分析 Lyapunov 函数（局部），这简化了形式验证，增强了可解释性，并为控制工程师提供了宝贵的见解。我们的框架由一个基于转换器的训练器组成，该训练器生成候选 Lyapunov 函数，以及一个伪造器，该伪造器验证候选表达式并通过风险寻求策略梯度改进模型。与 Alfarano 等人 (2024) 利用预训练并寻找低维系统的全局 Lyapunov 函数不同，我们的模型通过强化学习 (RL) 从头开始​​训练，并成功找到高维和非多项式系统的局部 Lyapunov 函数。鉴于候选函数的分析性质，我们在训练期间采用有效的优化方法进行伪造，并使用形式验证工具进行最终验证。我们证明了我们的方法在多达十维的一系列非线性动力系统中的有效性，并表明它可以发现控制文献中以前未发现的李雅普诺夫函数。

Title: A Periodic Bayesian Flow for Material Generation

• Authors: Hanlin Wu, Yuxuan Song, Jingjing Gong, Ziyao Cao, Yawen Ouyang, Jianbing Zhang, Hao Zhou, Wei-Ying Ma, Jingjing Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02016
• Pdf URL: https://arxiv.org/pdf/2502.02016
• Copy Paste: [[2502.02016]] A Periodic Bayesian Flow for Material Generation(https://arxiv.org/abs/2502.02016)
• Keywords: generation, generative
• Abstract: Generative modeling of crystal data distribution is an important yet challenging task due to the unique periodic physical symmetry of crystals. Diffusion-based methods have shown early promise in modeling crystal distribution. More recently, Bayesian Flow Networks were introduced to aggregate noisy latent variables, resulting in a variance-reduced parameter space that has been shown to be advantageous for modeling Euclidean data distributions with structural constraints (Song et al., 2023). Inspired by this, we seek to unlock its potential for modeling variables located in non-Euclidean manifolds e.g. those within crystal structures, by overcoming challenging theoretical issues. We introduce CrysBFN, a novel crystal generation method by proposing a periodic Bayesian flow, which essentially differs from the original Gaussian-based BFN by exhibiting non-monotonic entropy dynamics. To successfully realize the concept of periodic Bayesian flow, CrysBFN integrates a new entropy conditioning mechanism and empirically demonstrates its significance compared to time-conditioning. Extensive experiments over both crystal ab initio generation and crystal structure prediction tasks demonstrate the superiority of CrysBFN, which consistently achieves new state-of-the-art on all benchmarks. Surprisingly, we found that CrysBFN enjoys a significant improvement in sampling efficiency, e.g., ~100x speedup 10 v.s. 2000 steps network forwards) compared with previous diffusion-based methods on MP-20 dataset. Code is available at this https URL.
• 摘要：由于晶体具有独特的周期性物理对称性，晶体数据分布的生成建模是一项重要而又具有挑战性的任务。基于扩散的方法在晶体分布建模方面已显示出早期的前景。最近，引入了贝叶斯流网络来聚合嘈杂的潜在变量，从而产生了一个方差减少的参数空间，这已被证明有利于对具有结构约束的欧几里得数据分布进行建模（Song 等人，2023 年）。受此启发，我们寻求通过克服具有挑战性的理论问题来释放其对位于非欧几里得流形中的变量（例如晶体结构内的变量）进行建模的潜力。我们通过提出周期性贝叶斯流引入了一种新颖的晶体生成方法 CrysBFN，它与原始的基于高斯的 BFN 本质上不同，因为它表现出非单调的熵动力学。为了成功实现周期性贝叶斯流的概念，CrysBFN 集成了一种新的熵调节机制，并通过经验证明了其与时间调节相比的重要性。对晶体从头算生成和晶体结构预测任务的大量实验证明了 CrysBFN 的优越性，它在所有基准上始终达到新的最高水平。令人惊讶的是，我们发现 CrysBFN 在采样效率方面有显著提高，例如，与 MP-20 数据集上基于扩散的先前方法相比，速度提高了约 100 倍（10 vs. 2000 步网络转发）。代码可在此 https URL 上找到。

Title: ContinuouSP: Generative Model for Crystal Structure Prediction with Invariance and Continuity

• Authors: Yuji Tone, Masatoshi Hanai, Mitsuaki Kawamura, Kenjiro Taura, Toyotaro Suzumura
• Subjects: cs.LG, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2502.02026
• Pdf URL: https://arxiv.org/pdf/2502.02026
• Copy Paste: [[2502.02026]] ContinuouSP: Generative Model for Crystal Structure Prediction with Invariance and Continuity(https://arxiv.org/abs/2502.02026)
• Keywords: generative
• Abstract: The discovery of new materials using crystal structure prediction (CSP) based on generative machine learning models has become a significant research topic in recent years. In this paper, we study invariance and continuity in the generative machine learning for CSP. We propose a new model, called ContinuouSP, which effectively handles symmetry and periodicity in crystals. We clearly formulate the invariance and the continuity, and construct a model based on the energy-based model. Our preliminary evaluation demonstrates the effectiveness of this model with the CSP task.
• 摘要：近年来，基于生成式机器学习模型的晶体结构预测 (CSP) 发现新材料已成为一个重要的研究课题。在本文中，我们研究了 CSP 生成式机器学习中的不变性和连续性。我们提出了一种名为 ContinuouSP 的新模型，它可以有效地处理晶体中的对称性和周期性。我们清楚地表述了不变性和连续性，并基于基于能量的模型构建了一个模型。我们的初步评估证明了该模型在 CSP 任务中的有效性。

Title: CASIM: Composite Aware Semantic Injection for Text to Motion Generation

• Authors: Che-Jui Chang, Qingze Tony Liu, Honglu Zhou, Vladimir Pavlovic, Mubbasir Kapadia
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2502.02063
• Pdf URL: https://arxiv.org/pdf/2502.02063
• Copy Paste: [[2502.02063]] CASIM: Composite Aware Semantic Injection for Text to Motion Generation(https://arxiv.org/abs/2502.02063)
• Keywords: generation, generative
• Abstract: Recent advances in generative modeling and tokenization have driven significant progress in text-to-motion generation, leading to enhanced quality and realism in generated motions. However, effectively leveraging textual information for conditional motion generation remains an open challenge. We observe that current approaches, primarily relying on fixed-length text embeddings (e.g., CLIP) for global semantic injection, struggle to capture the composite nature of human motion, resulting in suboptimal motion quality and controllability. To address this limitation, we propose the Composite Aware Semantic Injection Mechanism (CASIM), comprising a composite-aware semantic encoder and a text-motion aligner that learns the dynamic correspondence between text and motion tokens. Notably, CASIM is model and representation-agnostic, readily integrating with both autoregressive and diffusion-based methods. Experiments on HumanML3D and KIT benchmarks demonstrate that CASIM consistently improves motion quality, text-motion alignment, and retrieval scores across state-of-the-art methods. Qualitative analyses further highlight the superiority of our composite-aware approach over fixed-length semantic injection, enabling precise motion control from text prompts and stronger generalization to unseen text inputs.
• 摘要：生成建模和标记化方面的最新进展推动了文本到运动生成的重大进展，从而提高了生成运动的质量和真实感。然而，有效地利用文本信息进行条件运动生成仍然是一个悬而未决的挑战。我们观察到，当前的方法主要依靠固定长度的文本嵌入（例如 CLIP）进行全局语义注入，难以捕捉人体运动的复合性质，导致运动质量和可控性不理想。为了解决这一限制，我们提出了复合感知语义注入机制 (CASIM)，它包括一个复合感知语义编码器和一个文本运动对齐器，用于学习文本和运动标记之间的动态对应关系。值得注意的是，CASIM 与模型和表示无关，可轻松与自回归和基于扩散的方法集成。在 HumanML3D 和 KIT 基准上的实验表明，CASIM 可在最先进的方法中持续提高运动质量、文本运动对齐和检索分数。定性分析进一步强调了我们的复合感知方法相对于固定长度语义注入的优越性，能够根据文本提示进行精确的运动控制，并能够更强大地概括为看不见的文本输入。

Title: Position Paper: Building Trust in Synthetic Data for Clinical AI

• Authors: Krishan Agyakari Raja Babu, Supriti Mulay, Om Prabhu, Mohanasankar Sivaprakasam
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.02076
• Pdf URL: https://arxiv.org/pdf/2502.02076
• Copy Paste: [[2502.02076]] Position Paper: Building Trust in Synthetic Data for Clinical AI(https://arxiv.org/abs/2502.02076)
• Keywords: generative
• Abstract: Deep generative models and synthetic medical data have shown significant promise in addressing key challenges in healthcare, such as privacy concerns, data bias, and the scarcity of realistic datasets. While research in this area has grown rapidly and demonstrated substantial theoretical potential, its practical adoption in clinical settings remains limited. Despite the benefits synthetic data offers, questions surrounding its reliability and credibility persist, leading to a lack of trust among clinicians. This position paper argues that fostering trust in synthetic medical data is crucial for its clinical adoption. It aims to spark a discussion on the viability of synthetic medical data in clinical practice, particularly in the context of current advancements in AI. We present empirical evidence from brain tumor segmentation to demonstrate that the quality, diversity, and proportion of synthetic data directly impact trust in clinical AI models. Our findings provide insights to improve the deployment and acceptance of synthetic data-driven AI systems in real-world clinical workflows.
• 摘要：深度生成模型和合成医疗数据在解决医疗保健领域的关键挑战（例如隐私问题、数据偏见和现实数据集的稀缺性）方面显示出巨大的潜力。虽然该领域的研究发展迅速并显示出巨大的理论潜力，但它在临床环境中的实际应用仍然有限。尽管合成数据提供了好处，但围绕其可靠性和可信度的问题仍然存在，导致临床医生缺乏信任。本立场文件认为，培养对合成医疗数据的信任对于其临床应用至关重要。它旨在引发关于合成医疗数据在临床实践中的可行性的讨论，特别是在当前人工智能进步的背景下。我们从脑肿瘤分割中提供了经验证据，以证明合成数据的质量、多样性和比例直接影响对临床人工智能模型的信任。我们的研究结果为改善合成数据驱动的人工智能系统在现实世界临床工作流程中的部署和接受度提供了见解。

Title: IPO: Iterative Preference Optimization for Text-to-Video Generation

• Authors: Xiaomeng Yang, Zhiyu Tan, Xuecheng Nie, Hao Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02088
• Pdf URL: https://arxiv.org/pdf/2502.02088
• Copy Paste: [[2502.02088]] IPO: Iterative Preference Optimization for Text-to-Video Generation(https://arxiv.org/abs/2502.02088)
• Keywords: generation
• Abstract: Video foundation models have achieved significant advancement with the help of network upgrade as well as model scale-up. However, they are still hard to meet requirements of applications due to unsatisfied generation quality. To solve this problem, we propose to align video foundation models with human preferences from the perspective of post-training in this paper. Consequently, we introduce an Iterative Preference Optimization strategy to enhance generated video quality by incorporating human feedback. Specifically, IPO exploits a critic model to justify video generations for pairwise ranking as in Direct Preference Optimization or point-wise scoring as in Kahneman-Tversky Optimization. Given this, IPO optimizes video foundation models with guidance of signals from preference feedback, which helps improve generated video quality in subject consistency, motion smoothness and aesthetic quality, etc. In addition, IPO incorporates the critic model with the multi-modality large language model, which enables it to automatically assign preference labels without need of retraining or relabeling. In this way, IPO can efficiently perform multi-round preference optimization in an iterative manner, without the need of tediously manual labeling. Comprehensive experiments demonstrate that the proposed IPO can effectively improve the video generation quality of a pretrained model and help a model with only 2B parameters surpass the one with 5B parameters. Besides, IPO achieves new state-of-the-art performance on VBench benchmark. We will release our source codes, models as well as dataset to advance future research and applications.
• 摘要：随着网络升级和模型扩展，视频基础模型取得了长足进步，但生成质量仍难以满足应用要求。针对这一问题，本文提出从后训练的角度将视频基础模型与人类偏好对齐，并引入迭代偏好优化策略，通过结合人类反馈来提升生成视频质量。具体而言，IPO 利用评价模型来证明视频生成适合成对排序（如直接偏好优化）或逐点评分（如卡尼曼-特沃斯基优化）。在此基础上，IPO 利用偏好反馈信号指导视频基础模型优化，从而提高生成视频在主题一致性、运动流畅度和美学质量等方面的质量。此外，IPO 将评价模型与多模态大语言模型相结合，使其能够自动分配偏好标签，而无需重新训练或重新标记。这样，IPO 可以高效地以迭代方式进行多轮偏好优化，而无需繁琐的手动标记。综合实验表明，所提出的 IPO 可以有效提高预训练模型的视频生成质量，并帮助仅具有 2B 参数的模型超越具有 5B 参数的模型。此外，IPO 在 VBench 基准上取得了新的最佳性能。我们将发布我们的源代码、模型以及数据集，以推动未来的研究和应用。

Title: Dual-Flow: Transferable Multi-Target, Instance-Agnostic Attacks via In-the-wild Cascading Flow Optimization

• Authors: Yixiao Chen, Shikun Sun, Jianshu Li, Ruoyu Li, Zhe Li, Junliang Xing
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.02096
• Pdf URL: https://arxiv.org/pdf/2502.02096
• Copy Paste: [[2502.02096]] Dual-Flow: Transferable Multi-Target, Instance-Agnostic Attacks via In-the-wild Cascading Flow Optimization(https://arxiv.org/abs/2502.02096)
• Keywords: generative
• Abstract: Adversarial attacks are widely used to evaluate model robustness, and in black-box scenarios, the transferability of these attacks becomes crucial. Existing generator-based attacks have excellent generalization and transferability due to their instance-agnostic nature. However, when training generators for multi-target tasks, the success rate of transfer attacks is relatively low due to the limitations of the model's capacity. To address these challenges, we propose a novel Dual-Flow framework for multi-target instance-agnostic adversarial attacks, utilizing Cascading Distribution Shift Training to develop an adversarial velocity function. Extensive experiments demonstrate that Dual-Flow significantly improves transferability over previous multi-target generative attacks. For example, it increases the success rate from Inception-v3 to ResNet-152 by 34.58%. Furthermore, our attack method, such as adversarially trained models, shows substantially stronger robustness against defense mechanisms.
• 摘要：对抗攻击被广泛用于评估模型的鲁棒性，而在黑盒场景中，这些攻击的可迁移性变得至关重要。现有的基于生成器的攻击由于其实例不可知性而具有出色的泛化和可迁移性。然而，在为多目标任务训练生成器时，由于模型容量的限制，迁移攻击的成功率相对较低。为了应对这些挑战，我们提出了一种用于多目标实例不可知对抗攻击的新型 Dual-Flow 框架，利用级联分布移位训练来开发对抗速度函数。大量实验表明，与之前的多目标生成攻击相比，Dual-Flow 显著提高了可迁移性。例如，它将从 Inception-v3 到 ResNet-152 的成功率提高了 34.58%。此外，我们的攻击方法（例如对抗训练的模型）表现出对防御机制的更强的鲁棒性。

Title: DOC-Depth: A novel approach for dense depth ground truth generation

• Authors: Simon de Moreau, Mathias Corsia, Hassan Bouchiba, Yasser Almehio, Andrei Bursuc, Hafid El-Idrissi, Fabien Moutarde
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2502.02144
• Pdf URL: https://arxiv.org/pdf/2502.02144
• Copy Paste: [[2502.02144]] DOC-Depth: A novel approach for dense depth ground truth generation(https://arxiv.org/abs/2502.02144)
• Keywords: generation
• Abstract: Accurate depth information is essential for many computer vision applications. Yet, no available dataset recording method allows for fully dense accurate depth estimation in a large scale dynamic environment. In this paper, we introduce DOC-Depth, a novel, efficient and easy-to-deploy approach for dense depth generation from any LiDAR sensor. After reconstructing consistent dense 3D environment using LiDAR odometry, we address dynamic objects occlusions automatically thanks to DOC, our state-of-the art dynamic object classification method. Additionally, DOC-Depth is fast and scalable, allowing for the creation of unbounded datasets in terms of size and time. We demonstrate the effectiveness of our approach on the KITTI dataset, improving its density from 16.1% to 71.2% and release this new fully dense depth annotation, to facilitate future research in the domain. We also showcase results using various LiDAR sensors and in multiple environments. All software components are publicly available for the research community.
• 摘要：准确的深度信息对于许多计算机视觉应用至关重要。然而，没有可用的数据集记录方法可以在大规模动态环境中实现完全密集的精确深度估计。在本文中，我们介绍了 DOC-Depth，这是一种新颖、高效且​​易于部署的方法，可用于从任何 LiDAR 传感器生成密集深度。在使用 LiDAR 里程计重建一致的密集 3D 环境后，我们借助我们最先进的动态物体分类方法 DOC 自动解决动态物体遮挡问题。此外，DOC-Depth 快速且可扩展，允许创建大小和时间不受限制的数据集。我们在 KITTI 数据集上展示了我们的方法的有效性，将其密度从 16.1% 提高到 71.2%，并发布了这种新的完全密集深度注释，以促进该领域的未来研究。我们还展示了使用各种 LiDAR 传感器和在多种环境中的结果。所有软件组件均向研究界公开。

Title: On the Guidance of Flow Matching

• Authors: Ruiqi Feng, Tailin Wu, Chenglei Yu, Wenhao Deng, Peiyan Hu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02150
• Pdf URL: https://arxiv.org/pdf/2502.02150
• Copy Paste: [[2502.02150]] On the Guidance of Flow Matching(https://arxiv.org/abs/2502.02150)
• Keywords: generation, generative
• Abstract: Flow matching has shown state-of-the-art performance in various generative tasks, ranging from image generation to decision-making, where guided generation is pivotal. However, the guidance of flow matching is more general than and thus substantially different from that of its predecessor, diffusion models. Therefore, the challenge in guidance for general flow matching remains largely underexplored. In this paper, we propose the first framework of general guidance for flow matching. From this framework, we derive a family of guidance techniques that can be applied to general flow matching. These include a new training-free asymptotically exact guidance, novel training losses for training-based guidance, and two classes of approximate guidance that cover classical gradient guidance methods as special cases. We theoretically investigate these different methods to give a practical guideline for choosing suitable methods in different scenarios. Experiments on synthetic datasets, image inverse problems, and offline reinforcement learning demonstrate the effectiveness of our proposed guidance methods and verify the correctness of our flow matching guidance framework. Code to reproduce the experiments can be found at this https URL.
• 摘要：流匹配在各种生成任务中都表现出了最先进的性能，从图像生成到决策，其中引导生成是关键。然而，流匹配的引导比其前身扩散模型更通用，因此有很大不同。因此，一般流匹配的引导挑战仍然在很大程度上未被充分探索。在本文中，我们提出了第一个流匹配通用指导框架。从这个框架中，我们推导出一系列可应用于一般流匹配的指导技术。这些包括一种新的无训练渐近精确指导、基于训练的指导的新型训练损失，以及两类近似指导，它们涵盖了经典梯度指导方法作为特殊情况。我们从理论上研究了这些不同的方法，以提供在不同场景中选择合适方法的实用指南。在合成数据集、图像逆问题和离线强化学习上的实验证明了我们提出的指导方法的有效性，并验证了我们的流匹配指导框架的正确性。可以在此 https URL 中找到重现实验的代码。

Title: Generative Kernel Spectral Clustering

• Authors: David Winant, Sonny Achten, Johan A. K. Suykens
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02185
• Pdf URL: https://arxiv.org/pdf/2502.02185
• Copy Paste: [[2502.02185]] Generative Kernel Spectral Clustering(https://arxiv.org/abs/2502.02185)
• Keywords: generative
• Abstract: Modern clustering approaches often trade interpretability for performance, particularly in deep learning-based methods. We present Generative Kernel Spectral Clustering (GenKSC), a novel model combining kernel spectral clustering with generative modeling to produce both well-defined clusters and interpretable representations. By augmenting weighted variance maximization with reconstruction and clustering losses, our model creates an explorable latent space where cluster characteristics can be visualized through traversals along cluster directions. Results on MNIST and FashionMNIST datasets demonstrate the model's ability to learn meaningful cluster representations.
• 摘要：现代聚类方法通常会牺牲可解释性来换取性能，尤其是在基于深度学习的方法中。我们提出了生成核谱聚类 (GenKSC)，这是一种将核谱聚类与生成模型相结合的新型模型，可生成定义明确的聚类和可解释的表示。通过使用重构和聚类损失来增强加权方差最大化，我们的模型创建了一个可探索的潜在空间，其中可以通过沿聚类方向的遍历来可视化聚类特征。MNIST 和 FashionMNIST 数据集上的结果证明了该模型学习有意义的聚类表示的能力。

Title: ShapeShifter: 3D Variations Using Multiscale and Sparse Point-Voxel Diffusion

• Authors: Nissim Maruani, Wang Yifan, Matthew Fisher, Pierre Alliez, Mathieu Desbrun
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02187
• Pdf URL: https://arxiv.org/pdf/2502.02187
• Copy Paste: [[2502.02187]] ShapeShifter: 3D Variations Using Multiscale and Sparse Point-Voxel Diffusion(https://arxiv.org/abs/2502.02187)
• Keywords: generation, generative
• Abstract: This paper proposes ShapeShifter, a new 3D generative model that learns to synthesize shape variations based on a single reference model. While generative methods for 3D objects have recently attracted much attention, current techniques often lack geometric details and/or require long training times and large resources. Our approach remedies these issues by combining sparse voxel grids and point, normal, and color sampling within a multiscale neural architecture that can be trained efficiently and in parallel. We show that our resulting variations better capture the fine details of their original input and can handle more general types of surfaces than previous SDF-based methods. Moreover, we offer interactive generation of 3D shape variants, allowing more human control in the design loop if needed.
• 摘要：本文提出了 ShapeShifter，这是一种新的 3D 生成模型，它基于单一参考模型学习合成形状变化。虽然 3D 对象的生成方法最近引起了广泛关注，但当前的技术通常缺乏几何细节和/或需要较长的训练时间和大量资源。我们的方法通过将稀疏体素网格和点、法线和颜色采样结合在可以高效并行训练的多尺度神经架构中来解决这些问题。我们表明，与以前基于 SDF 的方法相比，我们得到的变体可以更好地捕捉其原始输入的精细细节，并且可以处理更通用的表面类型。此外，我们提供 3D 形状变体的交互式生成，如果需要，可以在设计循环中提供更多的人为控制。

Title: From Uncertain to Safe: Conformal Fine-Tuning of Diffusion Models for Safe PDE Control

• Authors: Peiyan Hu, Xiaowei Qian, Wenhao Deng, Rui Wang, Haodong Feng, Ruiqi Feng, Tao Zhang, Long Wei, Yue Wang, Zhi-Ming Ma, Tailin Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02205
• Pdf URL: https://arxiv.org/pdf/2502.02205
• Copy Paste: [[2502.02205]] From Uncertain to Safe: Conformal Fine-Tuning of Diffusion Models for Safe PDE Control(https://arxiv.org/abs/2502.02205)
• Keywords: generation
• Abstract: The application of deep learning for partial differential equation (PDE)-constrained control is gaining increasing attention. However, existing methods rarely consider safety requirements crucial in real-world applications. To address this limitation, we propose Safe Diffusion Models for PDE Control (SafeDiffCon), which introduce the uncertainty quantile as model uncertainty quantification to achieve optimal control under safety constraints through both post-training and inference phases. Firstly, our approach post-trains a pre-trained diffusion model to generate control sequences that better satisfy safety constraints while achieving improved control objectives via a reweighted diffusion loss, which incorporates the uncertainty quantile estimated using conformal prediction. Secondly, during inference, the diffusion model dynamically adjusts both its generation process and parameters through iterative guidance and fine-tuning, conditioned on control targets while simultaneously integrating the estimated uncertainty quantile. We evaluate SafeDiffCon on three control tasks: 1D Burgers' equation, 2D incompressible fluid, and controlled nuclear fusion problem. Results demonstrate that SafeDiffCon is the only method that satisfies all safety constraints, whereas other classical and deep learning baselines fail. Furthermore, while adhering to safety constraints, SafeDiffCon achieves the best control performance.
• 摘要：深度学习在偏微分方程 (PDE) 约束控制中的应用越来越受到关注。然而，现有的方法很少考虑实际应用中至关重要的安全要求。为了解决这一限制，我们提出了用于 PDE 控制的安全扩散模型 (SafeDiffCon)，该模型引入不确定性分位数作为模型不确定性量化，以通过后训练和推理阶段实现安全约束下的最佳控制。首先，我们的方法对预训练的扩散模型进行后训练，以生成更好地满足安全约束的控制序列，同时通过重新加权的扩散损失实现改进的控制目标，该损失结合了使用共形预测估计的不确定性分位数。其次，在推理过程中，扩散模型通过迭代引导和微调动态调整其生成过程和参数，以控制目标为条件，同时积分估计的不确定性分位数。我们在三个控制任务上评估 SafeDiffCon：一维 Burgers 方程、二维不可压缩流体和受控核聚变问题。结果表明，SafeDiffCon 是唯一满足所有安全约束的方法，而其他经典和深度学习基线则失败了。此外，在遵守安全约束的同时，SafeDiffCon 实现了最佳控制性能。

Title: InterLCM: Low-Quality Images as Intermediate States of Latent Consistency Models for Effective Blind Face Restoration

• Authors: Senmao Li, Kai Wang, Joost van de Weijer, Fahad Shahbaz Khan, Chun-Le Guo, Shiqi Yang, Yaxing Wang, Jian Yang, Ming-Ming Cheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.02215
• Pdf URL: https://arxiv.org/pdf/2502.02215
• Copy Paste: [[2502.02215]] InterLCM: Low-Quality Images as Intermediate States of Latent Consistency Models for Effective Blind Face Restoration(https://arxiv.org/abs/2502.02215)
• Keywords: restoration
• Abstract: Diffusion priors have been used for blind face restoration (BFR) by fine-tuning diffusion models (DMs) on restoration datasets to recover low-quality images. However, the naive application of DMs presents several key limitations. (i) The diffusion prior has inferior semantic consistency (e.g., ID, structure and color.), increasing the difficulty of optimizing the BFR model; (ii) reliance on hundreds of denoising iterations, preventing the effective cooperation with perceptual losses, which is crucial for faithful restoration. Observing that the latent consistency model (LCM) learns consistency noise-to-data mappings on the ODE-trajectory and therefore shows more semantic consistency in the subject identity, structural information and color preservation, we propose InterLCM to leverage the LCM for its superior semantic consistency and efficiency to counter the above issues. Treating low-quality images as the intermediate state of LCM, InterLCM achieves a balance between fidelity and quality by starting from earlier LCM steps. LCM also allows the integration of perceptual loss during training, leading to improved restoration quality, particularly in real-world scenarios. To mitigate structural and semantic uncertainties, InterLCM incorporates a Visual Module to extract visual features and a Spatial Encoder to capture spatial details, enhancing the fidelity of restored images. Extensive experiments demonstrate that InterLCM outperforms existing approaches in both synthetic and real-world datasets while also achieving faster inference speed.
• 摘要：扩散先验已用于盲人脸部恢复 (BFR)，通过在恢复数据集上微调扩散模型 (DM) 来恢复低质量图像。然而，DM 的简单应用存在几个关键限制。(i) 扩散先验的语义一致性较差（例如，ID、结构和颜色），增加了优化 BFR 模型的难度；(ii) 依赖于数百次去噪迭代，阻碍了与感知损失的有效配合，而感知损失对于忠实恢复至关重要。观察到潜在一致性模型 (LCM) 在 ODE 轨迹上学习一致性噪声到数据映射，因此在主体身份、结构信息和颜色保留方面表现出更多的语义一致性，我们提出 InterLCM 以利用 LCM 的卓越语义一致性和效率来解决上述问题。将低质量图像视为 LCM 的中间状态，InterLCM 通过从早期的 LCM 步骤开始实现保真度和质量之间的平衡。 LCM 还允许在训练过程中整合感知损失，从而提高恢复质量，尤其是在现实世界场景中。为了减轻结构和语义不确定性，InterLCM 结合了视觉模块来提取视觉特征和空间编码器来捕捉空间细节，从而提高了恢复图像的保真度。大量实验表明，InterLCM 在合成和现实世界数据集中的表现都优于现有方法，同时还实现了更快的推理速度。

Title: Flatten Graphs as Sequences: Transformers are Scalable Graph Generators

• Authors: Dexiong Chen, Markus Krimmel, Karsten Borgwardt
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.02216
• Pdf URL: https://arxiv.org/pdf/2502.02216
• Copy Paste: [[2502.02216]] Flatten Graphs as Sequences: Transformers are Scalable Graph Generators(https://arxiv.org/abs/2502.02216)
• Keywords: generation
• Abstract: We introduce AutoGraph, a novel autoregressive framework for generating large attributed graphs using decoder-only transformers. At the core of our approach is a reversible "flattening" process that transforms graphs into random sequences. By sampling and learning from these sequences, AutoGraph enables transformers to model and generate complex graph structures in a manner akin to natural language. In contrast to diffusion models that rely on computationally intensive node features, our approach operates exclusively on these sequences. The sampling complexity and sequence length scale linearly with the number of edges, making AutoGraph highly scalable for generating large sparse graphs. Empirically, AutoGraph achieves state-of-the-art performance across diverse synthetic and molecular graph generation benchmarks, while delivering a 100-fold generation and a 3-fold training speedup compared to leading diffusion models. Additionally, it demonstrates promising transfer capabilities and supports substructure-conditioned generation without additional fine-tuning. By extending language modeling techniques to graph generation, this work paves the way for developing graph foundation models.
• 摘要：我们引入了 AutoGraph，这是一种新颖的自回归框架，用于使用仅解码器的转换器生成大型属性图。我们方法的核心是一个可逆的“扁平化”过程，将图转换为随机序列。通过对这些序列进行采样和学习，AutoGraph 使转换器能够以类似于自然语言的方式建模和生成复杂的图结构。与依赖计算密集型节点特征的扩散模型相比，我们的方法仅对这些序列进行操作。采样复杂度和序列长度与边数成线性比例，使 AutoGraph 具有高度可扩展性，可用于生成大型稀疏图。从经验上讲，AutoGraph 在各种合成和分子图生成基准中实现了最先进的性能，同时与领先的扩散模型相比，生成速度提高了 100 倍，训练速度提高了 3 倍。此外，它还展示了有希望的传输能力，并支持子结构条件生成而无需额外的微调。通过将语言建模技术扩展到图生成，这项工作为开发图基础模型铺平了道路。

Title: Exploring the latent space of diffusion models directly through singular value decomposition

• Authors: Li Wang, Boyan Gao, Yanran Li, Zhao Wang, Xiaosong Yang, David A. Clifton, Jun Xiao
• Subjects: cs.CV, cs.AI, cs.MM
• Abstract URL: https://arxiv.org/abs/2502.02225
• Pdf URL: https://arxiv.org/pdf/2502.02225
• Copy Paste: [[2502.02225]] Exploring the latent space of diffusion models directly through singular value decomposition(https://arxiv.org/abs/2502.02225)
• Keywords: generation
• Abstract: Despite the groundbreaking success of diffusion models in generating high-fidelity images, their latent space remains relatively under-explored, even though it holds significant promise for enabling versatile and interpretable image editing capabilities. The complicated denoising trajectory and high dimensionality of the latent space make it extremely challenging to interpret. Existing methods mainly explore the feature space of U-Net in Diffusion Models (DMs) instead of the latent space itself. In contrast, we directly investigate the latent space via Singular Value Decomposition (SVD) and discover three useful properties that can be used to control generation results without the requirements of data collection and maintain identity fidelity generated images. Based on these properties, we propose a novel image editing framework that is capable of learning arbitrary attributes from one pair of latent codes destined by text prompts in Stable Diffusion Models. To validate our approach, extensive experiments are conducted to demonstrate its effectiveness and flexibility in image editing. We will release our codes soon to foster further research and applications in this area.
• 摘要：尽管扩散模型在生成高保真图像方面取得了突破性的成功，但它们的潜在空间仍然相对未被充分探索，尽管它在实现多功能和可解释的图像编辑功能方面具有重大前景。复杂的去噪轨迹和潜在空间的高维性使其极难解释。现有方法主要探索扩散模型 (DM) 中 U-Net 的特征空间，而不是潜在空间本身。相比之下，我们通过奇异值分解 (SVD) 直接研究潜在空间，并发现三个有用的属性，可用于控制生成结果而无需数据收集并保持生成图像的身份保真度。基于这些属性，我们提出了一种新颖的图像编辑框架，该框架能够从稳定扩散模型中由文本提示指定的一对潜在代码中学习任意属性。为了验证我们的方法，我们进行了大量实验来证明其在图像编辑中的有效性和灵活性。我们将很快发布我们的代码，以促进该领域的进一步研究和应用。

Title: A User Guide to Sampling Strategies for Sliced Optimal Transport

• Authors: Keanu Sisouk, Julie Delon, Julien Tierny
• Subjects: cs.LG, math.PR
• Abstract URL: https://arxiv.org/abs/2502.02275
• Pdf URL: https://arxiv.org/pdf/2502.02275
• Copy Paste: [[2502.02275]] A User Guide to Sampling Strategies for Sliced Optimal Transport(https://arxiv.org/abs/2502.02275)
• Keywords: generation
• Abstract: This paper serves as a user guide to sampling strategies for sliced optimal transport. We provide reminders and additional regularity results on the Sliced Wasserstein distance. We detail the construction methods, generation time complexity, theoretical guarantees, and conditions for each strategy. Additionally, we provide insights into their suitability for sliced optimal transport in theory. Extensive experiments on both simulated and real-world data offer a representative comparison of the strategies, culminating in practical recommendations for their best usage.
• 摘要：本文是切片最优传输采样策略的用户指南。我们提供了切片 Wasserstein 距离的提醒和额外的规律性结果。我们详细介绍了每种策略的构造方法、生成时间复杂度、理论保证和条件。此外，我们还从理论上深入了解了它们是否适用于切片最优传输。对模拟数据和真实数据进行的大量实验提供了这些策略的代表性比较，最终为最佳使用提供了实用建议。

Title: GP-GS: Gaussian Processes for Enhanced Gaussian Splatting

• Authors: Zhihao Guo, Jingxuan Su, Shenglin Wang, Jinlong Fan, Jing Zhang, Liangxiu Han, Peng Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02283
• Pdf URL: https://arxiv.org/pdf/2502.02283
• Copy Paste: [[2502.02283]] GP-GS: Gaussian Processes for Enhanced Gaussian Splatting(https://arxiv.org/abs/2502.02283)
• Keywords: generation
• Abstract: 3D Gaussian Splatting has emerged as an efficient photorealistic novel view synthesis method. However, its reliance on sparse Structure-from-Motion (SfM) point clouds consistently compromises the scene reconstruction quality. To address these limitations, this paper proposes a novel 3D reconstruction framework Gaussian Processes Gaussian Splatting (GP-GS), where a multi-output Gaussian Process model is developed to achieve adaptive and uncertainty-guided densification of sparse SfM point clouds. Specifically, we propose a dynamic sampling and filtering pipeline that adaptively expands the SfM point clouds by leveraging GP-based predictions to infer new candidate points from the input 2D pixels and depth maps. The pipeline utilizes uncertainty estimates to guide the pruning of high-variance predictions, ensuring geometric consistency and enabling the generation of dense point clouds. The densified point clouds provide high-quality initial 3D Gaussians to enhance reconstruction performance. Extensive experiments conducted on synthetic and real-world datasets across various scales validate the effectiveness and practicality of the proposed framework.
• 摘要：3D 高斯分层已成为一种高效、逼真的新型视图合成方法。然而，它对稀疏的运动结构 (SfM) 点云的依赖会持续损害场景重建质量。为了解决这些限制，本文提出了一种新型 3D 重建框架高斯过程高斯分层 (GP-GS)，其中开发了一个多输出高斯过程模型来实现稀疏 SfM 点云的自适应和不确定性引导的致密化。具体来说，我们提出了一种动态采样和过滤管道，该管道通过利用基于 GP 的预测从输入的 2D 像素和深度图中推断新的候选点，自适应地扩展 SfM 点云。该管道利用不确定性估计来指导高方差预测的修剪，确保几何一致性并实现密集点云的生成。致密点云提供高质量的初始 3D 高斯函数以增强重建性能。在不同规模的合成和真实数据集上进行的大量实验验证了所提出框架的有效性和实用性。

Title: Density Ratio Estimation with Conditional Probability Paths

• Authors: Hanlin Yu, Arto Klami, Aapo Hyvärinen, Anna Korba, Omar Chehab
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02300
• Pdf URL: https://arxiv.org/pdf/2502.02300
• Copy Paste: [[2502.02300]] Density Ratio Estimation with Conditional Probability Paths(https://arxiv.org/abs/2502.02300)
• Keywords: generative
• Abstract: Density ratio estimation in high dimensions can be reframed as integrating a certain quantity, the time score, over probability paths which interpolate between the two densities. In practice, the time score has to be estimated based on samples from the two densities. However, existing methods for this problem remain computationally expensive and can yield inaccurate estimates. Inspired by recent advances in generative modeling, we introduce a novel framework for time score estimation, based on a conditioning variable. Choosing the conditioning variable judiciously enables a closed-form objective function. We demonstrate that, compared to previous approaches, our approach results in faster learning of the time score and competitive or better estimation accuracies of the density ratio on challenging tasks. Furthermore, we establish theoretical guarantees on the error of the estimated density ratio.
• 摘要：高维密度比估计可以重新定义为将某个量（时间分数）与插值在两个密度之间的概率路径进行积分。实际上，时间分数必须根据来自两个密度的样本进行估计。但是，现有的解决此问题的方法仍然计算成本高昂，并且可能产生不准确的估计。受到生成建模最新进展的启发，我们引入了一种基于条件变量的时间分数估计新框架。明智地选择条件变量可以实现闭式目标函数。我们证明，与以前的方法相比，我们的方法可以更快地学习时间分数，并且在具有挑战性的任务上对密度比的估计准确度具有竞争力或更好。此外，我们还为估计密度比的误差建立了理论保证。

Title: MotionLab: Unified Human Motion Generation and Editing via the Motion-Condition-Motion Paradigm

• Authors: Ziyan Guo, Zeyu Hu, Na Zhao, De Wen Soh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.02358
• Pdf URL: https://arxiv.org/pdf/2502.02358
• Copy Paste: [[2502.02358]] MotionLab: Unified Human Motion Generation and Editing via the Motion-Condition-Motion Paradigm(https://arxiv.org/abs/2502.02358)
• Keywords: generation
• Abstract: Human motion generation and editing are key components of computer graphics and vision. However, current approaches in this field tend to offer isolated solutions tailored to specific tasks, which can be inefficient and impractical for real-world applications. While some efforts have aimed to unify motion-related tasks, these methods simply use different modalities as conditions to guide motion generation. Consequently, they lack editing capabilities, fine-grained control, and fail to facilitate knowledge sharing across tasks. To address these limitations and provide a versatile, unified framework capable of handling both human motion generation and editing, we introduce a novel paradigm: Motion-Condition-Motion, which enables the unified formulation of diverse tasks with three concepts: source motion, condition, and target this http URL on this paradigm, we propose a unified framework, MotionLab, which incorporates rectified flows to learn the mapping from source motion to target motion, guided by the specified this http URL MotionLab, we introduce the 1) MotionFlow Transformer to enhance conditional generation and editing without task-specific modules; 2) Aligned Rotational Position Encoding} to guarantee the time synchronization between source motion and target motion; 3) Task Specified Instruction Modulation; and 4) Motion Curriculum Learning for effective multi-task learning and knowledge sharing across tasks. Notably, our MotionLab demonstrates promising generalization capabilities and inference efficiency across multiple benchmarks for human motion. Our code and additional video results are available at: this https URL.
• 摘要：人体运动生成和编辑是计算机图形学和视觉的关键组成部分。然而，目前该领域的方法倾向于提供针对特定任务的孤立解决方案，这对于实际应用来说可能效率低下且不切实际。虽然一些努力旨在统一与运动相关的任务，但这些方法只是使用不同的模态作为指导运动生成的条件。因此，它们缺乏编辑功能和细粒度控制，并且无法促进跨任务的知识共享。为了解决这些限制并提供一个能够处理人体运动生成和编辑的多功能统一框架，我们引入了一个新范式：运动-条件-运动，它能够用三个概念统一制定不同的任务：源运动、条件和目标此http URL在此范式上，我们提出了一个统一的框架MotionLab，它结合了整流来学习从源运动到目标运动的映射，在指定的此http URL MotionLab的指导下，我们引入了1）MotionFlow Transformer来增强条件生成和编辑，而无需特定于任务的模块； 2) 对齐旋转位置编码} 以确保源运动和目标运动之间的时间同步；3) 任务指定指令调制；4) 运动课程学习，实现有效的多任务学习和跨任务知识共享。值得注意的是，我们的 MotionLab 在多个人体运动基准测试中展示了良好的泛化能力和推理效率。我们的代码和其他视频结果可在以下网址获得：此 https URL。

Title: Field Matching: an Electrostatic Paradigm to Generate and Transfer Data

• Authors: Alexander Kolesov, Manukhov Stepan, Vladimir V. Palyulin, Alexander Korotin
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.02367
• Pdf URL: https://arxiv.org/pdf/2502.02367
• Copy Paste: [[2502.02367]] Field Matching: an Electrostatic Paradigm to Generate and Transfer Data(https://arxiv.org/abs/2502.02367)
• Keywords: generative
• Abstract: We propose Electrostatic Field Matching (EFM), a novel method that is suitable for both generative modeling and distribution transfer tasks. Our approach is inspired by the physics of an electrical capacitor. We place source and target distributions on the capacitor plates and assign them positive and negative charges, respectively. We then learn the electrostatic field of the capacitor using a neural network approximator. To map the distributions to each other, we start at one plate of the capacitor and move the samples along the learned electrostatic field lines until they reach the other plate. We theoretically justify that this approach provably yields the distribution transfer. In practice, we demonstrate the performance of our EFM in toy and image data experiments.
• 摘要：我们提出了静电场匹配 (EFM)，这是一种适用于生成建模和分布转移任务的新方法。我们的方法受到电容器物理原理的启发。我们将源分布和目标分布放置在电容器板上，并分别为其分配正电荷和负电荷。然后，我们使用神经网络近似器学习电容器的静电场。为了将分布相互映射，我们从电容器的一个板开始，并沿着学习到的静电场线移动样本，直到它们到达另一个板。我们从理论上证明了这种方法可以实现分布转移。在实践中，我们在玩具和图像数据实验中展示了 EFM 的性能。

Title: MaintaAvatar: A Maintainable Avatar Based on Neural Radiance Fields by Continual Learning

• Authors: Shengbo Gu, Yu-Kun Qiu, Yu-Ming Tang, Ancong Wu, Wei-Shi Zheng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02372
• Pdf URL: https://arxiv.org/pdf/2502.02372
• Copy Paste: [[2502.02372]] MaintaAvatar: A Maintainable Avatar Based on Neural Radiance Fields by Continual Learning(https://arxiv.org/abs/2502.02372)
• Keywords: generation
• Abstract: The generation of a virtual digital avatar is a crucial research topic in the field of computer vision. Many existing works utilize Neural Radiance Fields (NeRF) to address this issue and have achieved impressive results. However, previous works assume the images of the training person are available and fixed while the appearances and poses of a subject could constantly change and increase in real-world scenarios. How to update the human avatar but also maintain the ability to render the old appearance of the person is a practical challenge. One trivial solution is to combine the existing virtual avatar models based on NeRF with continual learning methods. However, there are some critical issues in this approach: learning new appearances and poses can cause the model to forget past information, which in turn leads to a degradation in the rendering quality of past appearances, especially color bleeding issues, and incorrect human body poses. In this work, we propose a maintainable avatar (MaintaAvatar) based on neural radiance fields by continual learning, which resolves the issues by utilizing a Global-Local Joint Storage Module and a Pose Distillation Module. Overall, our model requires only limited data collection to quickly fine-tune the model while avoiding catastrophic forgetting, thus achieving a maintainable virtual avatar. The experimental results validate the effectiveness of our MaintaAvatar model.
• 摘要：虚拟数字化身的生成是计算机视觉领域的一个重要研究课题。许多现有的研究利用神经辐射场 (NeRF) 来解决这一问题，并取得了令人印象深刻的成果。然而，以前的研究假设训练人员的图像是可用且固定的，而主体的外观和姿势在现实场景中可能会不断变化和增加。如何更新人类化身，同时又能保持渲染人物旧外观的能力是一个实际挑战。一个简单的解决方案是将现有的基于 NeRF 的虚拟化身模型与持续学习方法相结合。然而，这种方法存在一些关键问题：学习新的外观和姿势会导致模型忘记过去的信息，进而导致过去外观的渲染质量下降，尤其是颜色渗色问题和不正确的人体姿势。在这项工作中，我们提出了一种基于神经辐射场的可维护化身 (MaintaAvatar)，通过持续学习，利用全局-局部联合存储模块和姿势蒸馏模块解决了这些问题。总体而言，我们的模型只需要有限的数据收集就可以快速微调模型，同时避免灾难性遗忘，从而实现可维护的虚拟化身。实验结果验证了我们的MaintaAvatar模型的有效性。

Title: Extending SEEDS to a Supervoxel Algorithm for Medical Image Analysis

• Authors: Chenhui Zhao, Yan Jiang, Todd C. Hollon
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.02409
• Pdf URL: https://arxiv.org/pdf/2502.02409
• Copy Paste: [[2502.02409]] Extending SEEDS to a Supervoxel Algorithm for Medical Image Analysis(https://arxiv.org/abs/2502.02409)
• Keywords: generation
• Abstract: In this work, we extend the SEEDS superpixel algorithm from 2D images to 3D volumes, resulting in 3D SEEDS, a faster, better, and open-source supervoxel algorithm for medical image analysis. We compare 3D SEEDS with the widely used supervoxel algorithm SLIC on 13 segmentation tasks across 10 organs. 3D SEEDS accelerates supervoxel generation by a factor of 10, improves the achievable Dice score by +6.5%, and reduces the under-segmentation error by -0.16%. The code is available at this https URL
• 摘要：在这项工作中，我们将 SEEDS 超像素算法从 2D 图像扩展到 3D 体积，从而产生了 3D SEEDS，这是一种更快、更好且开源的医学图像分析超体素算法。我们在 10 个器官的 13 个分割任务上对 3D SEEDS 与广泛使用的超体素算法 SLIC 进行了比较。3D SEEDS 将超体素生成速度提高了 10 倍，将可实现的 Dice 分数提高了 +6.5%，并将欠分割误差降低了 -0.16%。代码可从此 https URL 获取

Title: Towards Fast Graph Generation via Autoregressive Noisy Filtration Modeling

• Authors: Markus Krimmel, Jenna Wiens, Karsten Borgwardt, Dexiong Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02415
• Pdf URL: https://arxiv.org/pdf/2502.02415
• Copy Paste: [[2502.02415]] Towards Fast Graph Generation via Autoregressive Noisy Filtration Modeling(https://arxiv.org/abs/2502.02415)
• Keywords: generation, generative
• Abstract: Graph generative models often face a critical trade-off between learning complex distributions and achieving fast generation speed. We introduce Autoregressive Noisy Filtration Modeling (ANFM), a novel approach that addresses both challenges. ANFM leverages filtration, a concept from topological data analysis, to transform graphs into short sequences of monotonically increasing subgraphs. This formulation extends the sequence families used in previous autoregressive models. To learn from these sequences, we propose a novel autoregressive graph mixer model. Our experiments suggest that exposure bias might represent a substantial hurdle in autoregressive graph generation and we introduce two mitigation strategies to address it: noise augmentation and a reinforcement learning approach. Incorporating these techniques leads to substantial performance gains, making ANFM competitive with state-of-the-art diffusion models across diverse synthetic and real-world datasets. Notably, ANFM produces remarkably short sequences, achieving a 100-fold speedup in generation time compared to diffusion models. This work marks a significant step toward high-throughput graph generation.
• 摘要：图生成模型通常面临学习复杂分布和实现快速生成速度之间的关键权衡。我们引入了自回归噪声过滤模型 (ANFM)，这是一种解决这两个挑战的新方法。ANFM 利用拓扑数据分析中的概念过滤将图转换为单调递增子图的短序列。此公式扩展了以前自回归模型中使用的序列系列。为了从这些序列中学习，我们提出了一种新颖的自回归图混合器模型。我们的实验表明，曝光偏差可能是自回归图生成中的一个重大障碍，我们引入了两种缓解策略来解决它：噪声增强和强化学习方法。结合这些技术可以显着提高性能，使 ANFM 在各种合成和现实世界数据集中与最先进的扩散模型相媲美。值得注意的是，ANFM 产生的序列非常短，与扩散模型相比，生成时间加快了 100 倍。这项工作标志着朝着高吞吐量图生成迈出了重要一步。

Title: TransformDAS: Mapping {\Phi}-OTDR Signals to Riemannian Manifold for Robust Classification

• Authors: Jiaju Kang, Puyu Han, Yang Chun, Xu Wang, Luqi Gong
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02428
• Pdf URL: https://arxiv.org/pdf/2502.02428
• Copy Paste: [[2502.02428]] TransformDAS: Mapping {\Phi}-OTDR Signals to Riemannian Manifold for Robust Classification(https://arxiv.org/abs/2502.02428)
• Keywords: generation, generative
• Abstract: Phase-sensitive optical time-domain reflectometry ({\Phi}-OTDR) is a widely used distributed fiber optic sensing system in engineering. Machine learning algorithms for {\Phi}-OTDR event classification require high volumes and quality of datasets; however, high-quality datasets are currently extremely scarce in the field, leading to a lack of robustness in models, which is manifested by higher false alarm rates in real-world scenarios. One promising approach to address this issue is to augment existing data using generative models combined with a small amount of real-world data. We explored mapping both {\Phi}-OTDR features in a GAN-based generative pipeline and signal features in a Transformer classifier to hyperbolic space to seek more effective model generalization. The results indicate that state-of-the-art models exhibit stronger generalization performance and lower false alarm rates in real-world scenarios when trained on augmented datasets. TransformDAS, in particular, demonstrates the best classification performance, highlighting the benefits of Riemannian manifold mapping in {\Phi}-OTDR data generation and model classification.
• 摘要：相位敏感光时域反射仪 ({\Phi}-OTDR) 是一种在工程中广泛使用的分布式光纤传感系统。用于 {\Phi}-OTDR 事件分类的机器学习算法需要大量和高质量的数据集；然而，目前该领域的高质量数据集极为稀缺，导致模型缺乏稳健性，这表现为现实场景中的更高误报率。解决此问题的一种有前途的方法是使用生成模型结合少量现实数据来增强现有数据。我们探索了将基于 GAN 的生成管道中的 {\Phi}-OTDR 特征和 Transformer 分类器中的信号特征映射到双曲空间，以寻求更有效的模型泛化。结果表明，当在增强数据集上训练时，最先进的模型在现实场景中表现出更强的泛化性能和更低的误报率。尤其是TransformDAS，它表现出了最佳的分类性能，凸显了黎曼流形映射在{\Phi}-OTDR数据生成和模型分类中的优势。

Title: Sparse Data Generation Using Diffusion Models

• Authors: Phil Ostheimer, Mayank Nagda, Marius Kloft, Sophie Fellenz
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02448
• Pdf URL: https://arxiv.org/pdf/2502.02448
• Copy Paste: [[2502.02448]] Sparse Data Generation Using Diffusion Models(https://arxiv.org/abs/2502.02448)
• Keywords: generation
• Abstract: Sparse data is ubiquitous, appearing in numerous domains, from economics and recommender systems to astronomy and biomedical sciences. However, efficiently and realistically generating sparse data remains a significant challenge. We introduce Sparse Data Diffusion (SDD), a novel method for generating sparse data. SDD extends continuous state-space diffusion models by explicitly modeling sparsity through the introduction of Sparsity Bits. Empirical validation on image data from various domains-including two scientific applications, physics and biology-demonstrates that SDD achieves high fidelity in representing data sparsity while preserving the quality of the generated data.
• 摘要：稀疏数据无处不在，出现在从经济学和推荐系统到天文学和生物医学等众多领域。然而，高效而真实地生成稀疏数据仍然是一项重大挑战。我们引入了稀疏数据扩散 (SDD)，这是一种生成稀疏数据的新方法。SDD 通过引入稀疏位明确地对稀疏性进行建模，从而扩展了连续状态空间扩散模型。对来自各个领域（包括物理学和生物学两个科学应用）的图像数据进行的经验验证表明，SDD 在表示数据稀疏性方面实现了高保真度，同时保持了生成数据的质量。

Title: Personalization Toolkit: Training Free Personalization of Large Vision Language Models

• Authors: Soroush Seifi, Vaggelis Dorovatas, Daniel Olmeda Reino, Rahaf Aljundi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.02452
• Pdf URL: https://arxiv.org/pdf/2502.02452
• Copy Paste: [[2502.02452]] Personalization Toolkit: Training Free Personalization of Large Vision Language Models(https://arxiv.org/abs/2502.02452)
• Keywords: generation
• Abstract: Large Vision Language Models (LVLMs) have significant potential to deliver personalized assistance by adapting to individual users' unique needs and preferences. Personalization of LVLMs is an emerging area that involves customizing models to recognize specific object instances and provide tailored responses. However, existing approaches rely on time-consuming test-time training for each user and object, rendering them impractical. This paper proposes a novel, training-free approach to LVLM personalization by leveraging pre-trained vision foundation models to extract distinct features, retrieval-augmented generation (RAG) techniques to recognize instances in the visual input, and visual prompting methods. Our model-agnostic vision toolkit enables flexible and efficient personalization without extensive retraining. We demonstrate state-of-the-art results, outperforming conventional training-based approaches and establish a new standard for LVLM personalization.
• 摘要：大型视觉语言模型 (LVLM) 具有通过适应个人用户的独特需求和偏好来提供个性化帮助的巨大潜力。LVLM 的个性化是一个新兴领域，涉及定制模型以识别特定对象实例并提供量身定制的响应。然而，现有的方法依赖于对每个用户和对象进行耗时的测试时间训练，这使得它们不切实际。本文提出了一种新颖的、无需训练的 LVLM 个性化方法，利用预先训练的视觉基础模型来提取不同的特征，利用检索增强生成 (RAG) 技术来识别视觉输入中的实例，以及视觉提示方法。我们的模型无关的视觉工具包可实现灵活高效的个性化，而无需大量的再训练。我们展示了最先进的结果，优于传统的基于训练的方法，并为 LVLM 个性化建立了新的标准。

Title: High-Fidelity Human Avatars from Laptop Webcams using Edge Compute

• Authors: Akash Haridas Imran N. Junejo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.02468
• Pdf URL: https://arxiv.org/pdf/2502.02468
• Copy Paste: [[2502.02468]] High-Fidelity Human Avatars from Laptop Webcams using Edge Compute(https://arxiv.org/abs/2502.02468)
• Keywords: generation
• Abstract: Applications of generating photo-realistic human avatars are many, however, high-fidelity avatar generation traditionally required expensive professional camera rigs and artistic labor, but recent research has enabled constructing them automatically from smartphones with RGB and IR sensors. However, these new methods still rely on the presence of high-resolution cameras on modern smartphones and often require offloading the processing to powerful servers with GPUs. Modern applications such as video conferencing call for the ability to generate these avatars from consumer-grade laptop webcams using limited compute available on-device. In this work, we develop a novel method based on 3D morphable models, landmark detection, photo-realistic texture GANs, and differentiable rendering to tackle the problem of low webcam image quality and edge computation. We build an automatic system to generate high-fidelity animatable avatars under these limitations, leveraging the neural compute capabilities of mobile chips.
• 摘要：生成照片般逼真的人类头像的应用有很多，然而，高保真头像生成传统上需要昂贵的专业摄像装置和艺术劳动，但最近的研究已经能够使用带有 RGB 和 IR 传感器的智能手机自动构建它们。但是，这些新方法仍然依赖于现代智能手机上高分辨率摄像头的存在，并且通常需要将处理任务转移到具有 GPU 的强大服务器上。视频会议等现代应用要求能够使用设备上有限的计算能力从消费级笔记本电脑网络摄像头生成这些头像。在这项工作中，我们开发了一种基于 3D 可变形模型、地标检测、照片般逼真的纹理 GAN 和可微分渲染的新方法来解决网络摄像头图像质量低和边缘计算的问题。我们构建了一个自动系统，在这些限制下生成高保真可动画头像，利用移动芯片的神经计算能力。

Title: Distributional Diffusion Models with Scoring Rules

• Authors: Valentin De Bortoli, Alexandre Galashov, J. Swaroop Guntupalli, Guangyao Zhou, Kevin Murphy, Arthur Gretton, Arnaud Doucet
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.02483
• Pdf URL: https://arxiv.org/pdf/2502.02483
• Copy Paste: [[2502.02483]] Distributional Diffusion Models with Scoring Rules(https://arxiv.org/abs/2502.02483)
• Keywords: generation
• Abstract: Diffusion models generate high-quality synthetic data. They operate by defining a continuous-time forward process which gradually adds Gaussian noise to data until fully corrupted. The corresponding reverse process progressively "denoises" a Gaussian sample into a sample from the data distribution. However, generating high-quality outputs requires many discretization steps to obtain a faithful approximation of the reverse process. This is expensive and has motivated the development of many acceleration methods. We propose to accomplish sample generation by learning the posterior {\em distribution} of clean data samples given their noisy versions, instead of only the mean of this distribution. This allows us to sample from the probability transitions of the reverse process on a coarse time scale, significantly accelerating inference with minimal degradation of the quality of the output. This is accomplished by replacing the standard regression loss used to estimate conditional means with a scoring rule. We validate our method on image and robot trajectory generation, where we consistently outperform standard diffusion models at few discretization steps.
• 摘要：扩散模型可生成高质量的合成数据。它们通过定义连续时间正向过程来运行，该过程逐渐将高斯噪声添加到数据中，直到完全损坏。相应的反向过程逐渐将高斯样本“去噪”为数据分布中的样本。但是，生成高质量的输出需要许多离散化步骤才能获得反向过程的忠实近似值。这很昂贵，并促使了许多加速方法的发展。我们建议通过学习干净数据样本的后验分布（给定它们的噪声版本）来完成样本生成，而不仅仅是该分布的平均值。这使我们能够在粗略的时间尺度上从反向过程的概率转换中进行采样，从而显着加速推理，同时将输出质量的下降降至最低。这是通过用评分规则替换用于估计条件均值的标准回归损失来实现的。我们在图像和机器人轨迹生成上验证了我们的方法，我们在几个离散化步骤中始终优于标准扩散模型。

Title: Do Graph Diffusion Models Accurately Capture and Generate Substructure Distributions?

• Authors: Xiyuan Wang, Yewei Liu, Lexi Pang, Siwei Chen, Muhan Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02488
• Pdf URL: https://arxiv.org/pdf/2502.02488
• Copy Paste: [[2502.02488]] Do Graph Diffusion Models Accurately Capture and Generate Substructure Distributions?(https://arxiv.org/abs/2502.02488)
• Keywords: generation
• Abstract: Diffusion models have gained popularity in graph generation tasks; however, the extent of their expressivity concerning the graph distributions they can learn is not fully understood. Unlike models in other domains, popular backbones for graph diffusion models, such as Graph Transformers, do not possess universal expressivity to accurately model the distribution scores of complex graph data. Our work addresses this limitation by focusing on the frequency of specific substructures as a key characteristic of target graph distributions. When evaluating existing models using this metric, we find that they fail to maintain the distribution of substructure counts observed in the training set when generating new graphs. To address this issue, we establish a theoretical connection between the expressivity of Graph Neural Networks (GNNs) and the overall performance of graph diffusion models, demonstrating that more expressive GNN backbones can better capture complex distribution patterns. By integrating advanced GNNs into the backbone architecture, we achieve significant improvements in substructure generation.
• 摘要：扩散模型在图生成任务中越来越受欢迎；然而，它们对可以学习的图分布的表达能力程度尚不完全清楚。与其他领域的模型不同，图扩散模型的流行主干（例如 Graph Transformers）不具备通用的表达能力，无法准确模拟复杂图数据的分布分数。我们的工作通过关注特定子结构的频率作为目标图分布的一个关键特征来解决这一限制。当使用此指标评估现有模型时，我们发现它们在生成新图时无法保持训练集中观察到的子结构计数的分布。为了解决这个问题，我们在图神经网络 (GNN) 的表达能力和图扩散模型的整体性能之间建立了理论联系，表明更具表达力的 GNN 主干可以更好地捕捉复杂的分布模式。通过将先进的 GNN 集成到主干架构中，我们在子结构生成方面取得了显著的改进。

Title: VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models

• Authors: Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, Shelly Sheynin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.02492
• Pdf URL: https://arxiv.org/pdf/2502.02492
• Copy Paste: [[2502.02492]] VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models(https://arxiv.org/abs/2502.02492)
• Keywords: generation, generative
• Abstract: Despite tremendous recent progress, generative video models still struggle to capture real-world motion, dynamics, and physics. We show that this limitation arises from the conventional pixel reconstruction objective, which biases models toward appearance fidelity at the expense of motion coherence. To address this, we introduce VideoJAM, a novel framework that instills an effective motion prior to video generators, by encouraging the model to learn a joint appearance-motion representation. VideoJAM is composed of two complementary units. During training, we extend the objective to predict both the generated pixels and their corresponding motion from a single learned representation. During inference, we introduce Inner-Guidance, a mechanism that steers the generation toward coherent motion by leveraging the model's own evolving motion prediction as a dynamic guidance signal. Notably, our framework can be applied to any video model with minimal adaptations, requiring no modifications to the training data or scaling of the model. VideoJAM achieves state-of-the-art performance in motion coherence, surpassing highly competitive proprietary models while also enhancing the perceived visual quality of the generations. These findings emphasize that appearance and motion can be complementary and, when effectively integrated, enhance both the visual quality and the coherence of video generation. Project website: this https URL
• 摘要：尽管最近取得了巨大进展，但生成视频模型仍然难以捕捉现实世界的运动、动态和物理特性。我们表明，这种限制源于传统的像素重建目标，该目标以牺牲运动连贯性为代价，使模型偏向于外观保真度。为了解决这个问题，我们引入了 VideoJAM，这是一个新颖的框架，它通过鼓励模型学习联合外观-运动表示，在视频生成器之前灌输有效的运动。VideoJAM 由两个互补单元组成。在训练期间，我们扩展目标以从单个学习到的表示中预测生成的像素及其相应的运动。在推理过程中，我们引入了内部指导，这是一种通过利用模型自身不断发展的运动预测作为动态指导信号来引导生成走向连贯运动的机制。值得注意的是，我们的框架可以应用于任何视频模型，只需进行最少的调整，不需要修改训练数据或缩放模型。VideoJAM 在运动连贯性方面实现了最先进的性能，超越了极具竞争力的专有模型，同时还提高了生成的视觉质量。这些发现强调了外观和动作可以互补，如果能有效整合，可以提高视频生成的视觉质量和连贯性。项目网站：这个 https URL

Title: Learning to generate physical ocean states: Towards hybrid climate modeling

• Authors: Etienne Meunier, David Kamm, Guillaume Gachon, Redouane Lguensat, Julie Deshayes
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02499
• Pdf URL: https://arxiv.org/pdf/2502.02499
• Copy Paste: [[2502.02499]] Learning to generate physical ocean states: Towards hybrid climate modeling(https://arxiv.org/abs/2502.02499)
• Keywords: generation, generative
• Abstract: Ocean General Circulation Models require extensive computational resources to reach equilibrium states, while deep learning emulators, despite offering fast predictions, lack the physical interpretability and long-term stability necessary for climate scientists to understand climate sensitivity (to greenhouse gas emissions) and mechanisms of abrupt % variability such as tipping points. We propose to take the best from both worlds by leveraging deep generative models to produce physically consistent oceanic states that can serve as initial conditions for climate projections. We assess the viability of this hybrid approach through both physical metrics and numerical experiments, and highlight the benefits of enforcing physical constraints during generation. Although we train here on ocean variables from idealized numerical simulations, we claim that this hybrid approach, combining the computational efficiency of deep learning with the physical accuracy of numerical models, can effectively reduce the computational burden of running climate models to equilibrium, and reduce uncertainties in climate projections by minimizing drifts in baseline simulations.
• 摘要：海洋大气环流模型需要大量计算资源才能达到平衡状态，而深度学习模拟器虽然可以提供快速预测，但缺乏气候科学家了解气候敏感性（对温室气体排放）和突然百分比变化机制（如临界点）所必需的物理可解释性和长期稳定性。我们建议利用深度生成模型来产生物理上一致的海洋状态，作为气候预测的初始条件，从而兼顾两方面的优势。我们通过物理指标和数值实验来评估这种混合方法的可行性，并强调在生成过程中实施物理约束的好处。虽然我们在这里使用理想化数值模拟中的海洋变量进行训练，但我们认为这种混合方法结合了深度学习的计算效率和数值模型的物理精度，可以有效减少运行气候模型达到平衡的计算负担，并通过最大限度地减少基线模拟中的漂移来减少气候预测中的不确定性。

Title: Generative Modeling on Lie Groups via Euclidean Generalized Score Matching

• Authors: Marco Bertolini, Tuan Le, Djork-Arné Clevert
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02513
• Pdf URL: https://arxiv.org/pdf/2502.02513
• Copy Paste: [[2502.02513]] Generative Modeling on Lie Groups via Euclidean Generalized Score Matching(https://arxiv.org/abs/2502.02513)
• Keywords: generation, generative
• Abstract: We extend Euclidean score-based diffusion processes to generative modeling on Lie groups. Through the formalism of Generalized Score Matching, our approach yields a Langevin dynamics which decomposes as a direct sum of Lie algebra representations, enabling generative processes on Lie groups while operating in Euclidean space. Unlike equivariant models, which restrict the space of learnable functions by quotienting out group orbits, our method can model any target distribution on any (non-Abelian) Lie group. Standard score matching emerges as a special case of our framework when the Lie group is the translation group. We prove that our generalized generative processes arise as solutions to a new class of paired stochastic differential equations (SDEs), introduced here for the first time. We validate our approach through experiments on diverse data types, demonstrating its effectiveness in real-world applications such as SO(3)-guided molecular conformer generation and modeling ligand-specific global SE(3) transformations for molecular docking, showing improvement in comparison to Riemannian diffusion on the group itself. We show that an appropriate choice of Lie group enhances learning efficiency by reducing the effective dimensionality of the trajectory space and enables the modeling of transitions between complex data distributions. Additionally, we demonstrate the universality of our approach by deriving how it extends to flow matching.
• 摘要：我们将基于欧几里得分数的扩散过程扩展到李群上的生成建模。通过广义分数匹配的形式化，我们的方法产生了朗之万动力学，该动力学分解为李代数表示的直接和，从而能够在欧几里得空间中操作李群上的生成过程。与通过商出群轨道来限制可学习函数空间的等变模型不同，我们的方法可以在任何（非阿贝尔）李群上建模任何目标分布。当李群是平移群时，标准分数匹配成为我们框架的一个特例。我们证明我们的广义生成过程是作为一类新的成对随机微分方程 (SDE) 的解出现的，这是这里首次引入的。我们通过对各种数据类型的实验验证了我们的方法，证明了它在实际应用中的有效性，例如 SO(3) 引导的分子构象生成和为分子对接建模配体特定的全局 SE(3) 变换，与群本身的黎曼扩散相比，该方法有所改进。我们表明，选择合适的李群可以通过降低轨迹空间的有效维数来提高学习效率，并能够对复杂数据分布之间的转换进行建模。此外，我们通过推导该方法如何扩展到流匹配来证明该方法的通用性。

Title: Privacy Attacks on Image AutoRegressive Models

• Authors: Antoni Kowalczuk, Jan Dubiński, Franziska Boenisch, Adam Dziedzic
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02514
• Pdf URL: https://arxiv.org/pdf/2502.02514
• Copy Paste: [[2502.02514]] Privacy Attacks on Image AutoRegressive Models(https://arxiv.org/abs/2502.02514)
• Keywords: generation
• Abstract: Image autoregressive (IAR) models have surpassed diffusion models (DMs) in both image quality (FID: 1.48 vs. 1.58) and generation speed. However, their privacy risks remain largely unexplored. To address this, we conduct a comprehensive privacy analysis comparing IARs to DMs. We develop a novel membership inference attack (MIA) that achieves a significantly higher success rate in detecting training images (TPR@FPR=1%: 86.38% for IARs vs. 4.91% for DMs). Using this MIA, we perform dataset inference (DI) and find that IARs require as few as six samples to detect dataset membership, compared to 200 for DMs, indicating higher information leakage. Additionally, we extract hundreds of training images from an IAR (e.g., 698 from VAR-d30). Our findings highlight a fundamental privacy-utility trade-off: while IARs excel in generation quality and speed, they are significantly more vulnerable to privacy attacks. This suggests that incorporating techniques from DMs, such as per-token probability modeling using diffusion, could help mitigate IARs' privacy risks. Our code is available at this https URL.
• 摘要：图像自回归 (IAR) 模型在图像质量 (FID：1.48 vs. 1.58) 和生成速度方面都超越了扩散模型 (DM)。然而，它们的隐私风险在很大程度上仍未被探索。为了解决这个问题，我们对 IAR 和 DM 进行了全面的隐私分析。我们开发了一种新颖的成员资格推理攻击 (MIA)，在检测训练图像方面取得了显著更高的成功率 (TPR@FPR=1%：IAR 为 86.38%，DM 为 4.91%)。使用此 MIA，我们执行数据集推理 (DI)，发现 IAR 只需六个样本即可检测数据集成员资格，而 DM 则需要 200 个样本，这表明信息泄露更高。此外，我们从 IAR 中提取了数百张训练图像（例如，来自 VAR-d30 的 698 张）。我们的研究结果强调了一个基本的隐私效用权衡：虽然 IAR 在生成质量和速度方面表现出色，但它们更容易受到隐私攻击。这表明，结合 DM 的技术（例如使用扩散的每令牌概率建模）可以帮助减轻 IAR 的隐私风险。我们的代码可在此 https URL 上找到。

Title: Brief analysis of DeepSeek R1 and it's implications for Generative AI

• Authors: Sarah Mercer, Samuel Spillard, Daniel P. Martin
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.02523
• Pdf URL: https://arxiv.org/pdf/2502.02523
• Copy Paste: [[2502.02523]] Brief analysis of DeepSeek R1 and it's implications for Generative AI(https://arxiv.org/abs/2502.02523)
• Keywords: generative
• Abstract: In late January 2025, DeepSeek released their new reasoning model (DeepSeek R1); which was developed at a fraction of the cost yet remains competitive with OpenAI's models, despite the US's GPU export ban. This report discusses the model, and what its release means for the field of Generative AI more widely. We briefly discuss other models released from China in recent weeks, their similarities; innovative use of Mixture of Experts (MoE), Reinforcement Learning (RL) and clever engineering appear to be key factors in the capabilities of these models. This think piece has been written to a tight time-scale, providing broad coverage of the topic, and serves as introductory material for those looking to understand the model's technical advancements, as well as it's place in the ecosystem. Several further areas of research are identified.
• 摘要：2025 年 1 月下旬，DeepSeek 发布了其新的推理模型 (DeepSeek R1)；尽管美国实施了 GPU 出口禁令，但该模型的开发成本仅为 OpenAI 模型的一小部分，且仍具有竞争力。本报告讨论了该模型，以及它的发布对生成式人工智能领域的更广泛意义。我们简要讨论了最近几周中国发布的其他模型及其相似之处；创新使用混合专家 (MoE)、强化学习 (RL) 和巧妙的工程设计似乎是这些模型能力的关键因素。这篇思考文章是在紧迫的时间范围内撰写的，广泛涵盖了该主题，并可作为那些希望了解该模型的技术进步及其在生态系统中地位的人的入门材料。确定了几个进一步的研究领域。

Title: Diff9D: Diffusion-Based Domain-Generalized Category-Level 9-DoF Object Pose Estimation

• Authors: Jian Liu, Wei Sun, Hui Yang, Pengchao Deng, Chongpei Liu, Nicu Sebe, Hossein Rahmani, Ajmal Mian
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2502.02525
• Pdf URL: https://arxiv.org/pdf/2502.02525
• Copy Paste: [[2502.02525]] Diff9D: Diffusion-Based Domain-Generalized Category-Level 9-DoF Object Pose Estimation(https://arxiv.org/abs/2502.02525)
• Keywords: generative
• Abstract: Nine-degrees-of-freedom (9-DoF) object pose and size estimation is crucial for enabling augmented reality and robotic manipulation. Category-level methods have received extensive research attention due to their potential for generalization to intra-class unknown objects. However, these methods require manual collection and labeling of large-scale real-world training data. To address this problem, we introduce a diffusion-based paradigm for domain-generalized category-level 9-DoF object pose estimation. Our motivation is to leverage the latent generalization ability of the diffusion model to address the domain generalization challenge in object pose estimation. This entails training the model exclusively on rendered synthetic data to achieve generalization to real-world scenes. We propose an effective diffusion model to redefine 9-DoF object pose estimation from a generative perspective. Our model does not require any 3D shape priors during training or inference. By employing the Denoising Diffusion Implicit Model, we demonstrate that the reverse diffusion process can be executed in as few as 3 steps, achieving near real-time performance. Finally, we design a robotic grasping system comprising both hardware and software components. Through comprehensive experiments on two benchmark datasets and the real-world robotic system, we show that our method achieves state-of-the-art domain generalization performance. Our code will be made public at this https URL.
• 摘要：九自由度 (9-DoF) 物体姿态和大小估计对于实现增强现实和机器人操控至关重要。类别级方法因其具有推广到类内未知物体的潜力而受到广泛的研究关注。然而，这些方法需要手动收集和标记大规模真实世界训练数据。为了解决这个问题，我们引入了一个基于扩散的范式，用于领域广义的类别级 9-DoF 物体姿态估计。我们的动机是利用扩散模型的潜在泛化能力来解决物体姿态估计中的领域泛化挑战。这需要专门在渲染的合成数据上训练模型，以实现对真实世界场景的泛化。我们提出了一个有效的扩散模型，从生成的角度重新定义 9-DoF 物体姿态估计。我们的模型在训练或推理过程中不需要任何 3D 形状先验。通过采用去噪扩散隐式模型，我们证明反向扩散过程只需 3 步即可执行，实现近乎实时的性能。最后，我们设计了一个由硬件和软件组件组成的机器人抓取系统。通过对两个基准数据集和现实世界机器人系统进行全面实验，我们表明我们的方法实现了最先进的领域泛化性能。我们的代码将在此 https URL 上公开。

Title: Flow Q-Learning

• Authors: Seohong Park, Qiyang Li, Sergey Levine
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02538
• Pdf URL: https://arxiv.org/pdf/2502.02538
• Copy Paste: [[2502.02538]] Flow Q-Learning(https://arxiv.org/abs/2502.02538)
• Keywords: generation
• Abstract: We present flow Q-learning (FQL), a simple and performant offline reinforcement learning (RL) method that leverages an expressive flow-matching policy to model arbitrarily complex action distributions in data. Training a flow policy with RL is a tricky problem, due to the iterative nature of the action generation process. We address this challenge by training an expressive one-step policy with RL, rather than directly guiding an iterative flow policy to maximize values. This way, we can completely avoid unstable recursive backpropagation, eliminate costly iterative action generation at test time, yet still mostly maintain expressivity. We experimentally show that FQL leads to strong performance across 73 challenging state- and pixel-based OGBench and D4RL tasks in offline RL and offline-to-online RL. Project page: this https URL
• 摘要：我们提出了流 Q 学习 (FQL)，这是一种简单且高效的离线强化学习 (RL) 方法，它利用富有表现力的流匹配策略来对数据中任意复杂的动作分布进行建模。由于动作生成过程的迭代性质，使用 RL 训练流策略是一个棘手的问题。我们通过使用 RL 训练富有表现力的一步策略来解决这一挑战，而不是直接引导迭代流策略来最大化值。这样，我们可以完全避免不稳定的递归反向传播，在测试时消除昂贵的迭代动作生成，但仍能保持大部分表现力。我们通过实验表明，FQL 在离线 RL 和离线到在线 RL 中的 73 个具有挑战性的基于状态和像素的 OGBench 和 D4RL 任务中表现出色。项目页面：此 https URL

Title: Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation

• Authors: Junha Lee, Chunghyun Park, Jaesung Choe, Yu-Chiang Frank Wang, Jan Kautz, Minsu Cho, Chris Choy
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.02548
• Pdf URL: https://arxiv.org/pdf/2502.02548
• Copy Paste: [[2502.02548]] Mosaic3D: Foundation Dataset and Model for Open-Vocabulary 3D Segmentation(https://arxiv.org/abs/2502.02548)
• Keywords: generation
• Abstract: We tackle open-vocabulary 3D scene understanding by introducing a novel data generation pipeline and training framework. Our method addresses three critical requirements for effective training: precise 3D region segmentation, comprehensive textual descriptions, and sufficient dataset scale. By leveraging state-of-the-art open-vocabulary image segmentation models and region-aware Vision-Language Models, we develop an automatic pipeline that generates high-quality 3D mask-text pairs. Applying this pipeline to multiple 3D scene datasets, we create Mosaic3D-5.6M, a dataset of over 30K annotated scenes with 5.6M mask-text pairs, significantly larger than existing datasets. Building upon this data, we propose Mosaic3D, a foundation model combining a 3D encoder trained with contrastive learning and a lightweight mask decoder for open-vocabulary 3D semantic and instance segmentation. Our approach achieves state-of-the-art results on open-vocabulary 3D semantic and instance segmentation tasks including ScanNet200, Matterport3D, and ScanNet++, with ablation studies validating the effectiveness of our large-scale training data.
• 摘要：我们通过引入一种新颖的数据生成管道和训练框架来解决开放词汇 3D 场景理解问题。我们的方法解决了有效训练的三个关键要求：精确的 3D 区域分割、全面的文本描述和足够的数据集规模。通过利用最先进的开放词汇图像分割模型和区域感知视觉语言模型，我们开发了一种自动管道，可生成高质量的 3D 掩码-文本对。将此管道应用于多个 3D 场景数据集，我们创建了 Mosaic3D-5.6M，这是一个包含超过 30K 个带注释场景和 5.6M 个掩码-文本对的数据集，比现有数据集大得多。基于这些数据，我们提出了 Mosaic3D，这是一个基础模型，结合了使用对比学习训练的 3D 编码器和轻量级掩码解码器，用于开放词汇 3D 语义和实例分割。我们的方法在开放词汇 3D 语义和实例分割任务（包括 ScanNet200、Matterport3D 和 ScanNet++）上取得了最先进的结果，并且消融研究验证了我们的大规模训练数据的有效性。

Title: Open Materials Generation with Stochastic Interpolants

• Authors: Philipp Hoellmer, Thomas Egg, Maya M. Martirossyan, Eric Fuemmeler, Amit Gupta, Zeren Shui, Pawan Prakash, Adrian Roitberg, Mingjie Liu, George Karypis, Mark Transtrum, Richard G. Hennig, Ellad B. Tadmor, Stefano Martiniani
• Subjects: cs.LG, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2502.02582
• Pdf URL: https://arxiv.org/pdf/2502.02582
• Copy Paste: [[2502.02582]] Open Materials Generation with Stochastic Interpolants(https://arxiv.org/abs/2502.02582)
• Keywords: generation, generative
• Abstract: The discovery of new materials is essential for enabling technological advancements. Computational approaches for predicting novel materials must effectively learn the manifold of stable crystal structures within an infinite design space. We introduce Open Materials Generation (OMG), a unifying framework for the generative design and discovery of inorganic crystalline materials. OMG employs stochastic interpolants (SI) to bridge an arbitrary base distribution to the target distribution of inorganic crystals via a broad class of tunable stochastic processes, encompassing both diffusion models and flow matching as special cases. In this work, we adapt the SI framework by integrating an equivariant graph representation of crystal structures and extending it to account for periodic boundary conditions in unit cell representations. Additionally, we couple the SI flow over spatial coordinates and lattice vectors with discrete flow matching for atomic species. We benchmark OMG's performance on two tasks: Crystal Structure Prediction (CSP) for specified compositions, and 'de novo' generation (DNG) aimed at discovering stable, novel, and unique structures. In our ground-up implementation of OMG, we refine and extend both CSP and DNG metrics compared to previous works. OMG establishes a new state-of-the-art in generative modeling for materials discovery, outperforming purely flow-based and diffusion-based implementations. These results underscore the importance of designing flexible deep learning frameworks to accelerate progress in materials science.
• 摘要：新材料的发现对于技术进步至关重要。预测新材料的计算方法必须有效地学习无限设计空间内稳定晶体结构的多样性。我们引入了开放材料生成 (OMG)，这是一个用于生成设计和发现无机晶体材料的统一框架。OMG 采用随机插值 (SI) 通过广泛的可调随机过程将任意基础分布与无机晶体的目标分布联系起来，包括扩散模型和流匹配作为特殊情况。在这项工作中，我们通过集成晶体结构的等变图表示并将其扩展为考虑晶胞表示中的周期性边界条件来调整 SI 框架。此外，我们将空间坐标和晶格矢量上的 SI 流与原子种类的离散流匹配耦合。我们在两个任务上对 OMG 的性能进行了基准测试：针对指定成分的晶体结构预测 (CSP) 和旨在发现稳定、新颖和独特结构的“从头”生成 (DNG)。在我们从头开始实施 OMG 的过程中，我们与之前的研究相比改进并扩展了 CSP 和 DNG 指标。OMG 在材料发现的生成模型中建立了新的领先地位，其表现优于纯基于流和基于扩散的实现。这些结果强调了设计灵活的深度学习框架以加速材料科学进步的重要性。

Title: QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Search

• Authors: Zongyu Lin, Yao Tang, Xingcheng Yao, Da Yin, Ziniu Hu, Yizhou Sun, Kai-Wei Chang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.02584
• Pdf URL: https://arxiv.org/pdf/2502.02584
• Copy Paste: [[2502.02584]] QLASS: Boosting Language Agent Inference via Q-Guided Stepwise Search(https://arxiv.org/abs/2502.02584)
• Keywords: generation
• Abstract: Language agents have become a promising solution to complex interactive tasks. One of the key ingredients to the success of language agents is the reward model on the trajectory of the agentic workflow, which provides valuable guidance during training or inference. However, due to the lack of annotations of intermediate interactions, most existing works use an outcome reward model to optimize policies across entire trajectories. This may lead to sub-optimal policies and hinder the overall performance. To address this, we propose QLASS (Q-guided Language Agent Stepwise Search), to automatically generate annotations by estimating Q-values in a stepwise manner for open language agents. By introducing a reasoning tree and performing process reward modeling, QLASS provides effective intermediate guidance for each step. With the stepwise guidance, we propose a Q-guided generation strategy to enable language agents to better adapt to long-term value, resulting in significant performance improvement during model inference on complex interactive agent tasks. Notably, even with almost half the annotated data, QLASS retains strong performance, demonstrating its efficiency in handling limited supervision. We also empirically demonstrate that QLASS can lead to more effective decision making through qualitative analysis. We will release our code and data.
• 摘要：语言代理已成为复杂交互任务的有前途的解决方案。语言代理成功的关键因素之一是代理工作流轨迹上的奖励模型，它在训练或推理期间提供有价值的指导。然而，由于缺乏中间交互的注释，大多数现有工作使用结果奖励模型来优化整个轨迹上的策略。这可能会导致次优策略并阻碍整体性能。为了解决这个问题，我们提出了 QLASS（Q 引导语言代理分步搜索），通过逐步估计开放语言代理的 Q 值来自动生成注释。通过引入推理树并执行过程奖励建模，QLASS 为每个步骤提供有效的中间指导。通过分步指导，我们提出了一种 Q 引导生成策略，使语言代理能够更好地适应长期价值，从而显著提高复杂交互式代理任务的模型推理性能。值得注意的是，即使使用几乎一半的注释数据，QLASS 仍保持强劲的性能，证明了其在处理有限监督方面的效率。我们还通过实证研究证明，QLASS 可以通过定性分析实现更有效的决策。我们将发布我们的代码和数据。

Title: COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for Fine-Grained Understanding and Generation

• Authors: Xueqing Deng, Qihang Yu, Ali Athar, Chenglin Yang, Linjie Yang, Xiaojie Jin, Xiaohui Shen, Liang-Chieh Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.02589
• Pdf URL: https://arxiv.org/pdf/2502.02589
• Copy Paste: [[2502.02589]] COCONut-PanCap: Joint Panoptic Segmentation and Grounded Captions for Fine-Grained Understanding and Generation(https://arxiv.org/abs/2502.02589)
• Keywords: generation, generative
• Abstract: This paper introduces the COCONut-PanCap dataset, created to enhance panoptic segmentation and grounded image captioning. Building upon the COCO dataset with advanced COCONut panoptic masks, this dataset aims to overcome limitations in existing image-text datasets that often lack detailed, scene-comprehensive descriptions. The COCONut-PanCap dataset incorporates fine-grained, region-level captions grounded in panoptic segmentation masks, ensuring consistency and improving the detail of generated captions. Through human-edited, densely annotated descriptions, COCONut-PanCap supports improved training of vision-language models (VLMs) for image understanding and generative models for text-to-image tasks. Experimental results demonstrate that COCONut-PanCap significantly boosts performance across understanding and generation tasks, offering complementary benefits to large-scale datasets. This dataset sets a new benchmark for evaluating models on joint panoptic segmentation and grounded captioning tasks, addressing the need for high-quality, detailed image-text annotations in multi-modal learning.
• 摘要：本文介绍了 COCONut-PanCap 数据集，该数据集旨在增强全景分割和基于图像的字幕。该数据集以 COCO 数据集为基础，采用先进的 COCONut 全景蒙版，旨在克服现有图像文本数据集的局限性，这些数据集通常缺乏详细的场景综合描述。COCONut-PanCap 数据集结合了基于全景分割蒙版的细粒度区域级字幕，确保了一致性并提高了生成的字幕的细节。通过人工编辑的密集注释描述，COCONut-PanCap 支持改进用于图像理解的视觉语言模型 (VLM) 和用于文本到图像任务的生成模型的训练。实验结果表明，COCONut-PanCap 显著提高了理解和生成任务的性能，为大规模数据集提供了互补优势。该数据集为评估联合全景分割和基于字幕任务的模型设定了新的基准，满足了多模态学习中对高质量、详细的图像文本注释的需求。