2025-01-10

Title: Video Summarisation with Incident and Context Information using Generative AI

• Authors: Ulindu De Silva, Leon Fernando, Kalinga Bandara, Rashmika Nawaratne
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04764
• Pdf URL: https://arxiv.org/pdf/2501.04764
• Copy Paste: [[2501.04764]] Video Summarisation with Incident and Context Information using Generative AI(https://arxiv.org/abs/2501.04764)
• Keywords: generative
• Abstract: The proliferation of video content production has led to vast amounts of data, posing substantial challenges in terms of analysis efficiency and resource utilization. Addressing this issue calls for the development of robust video analysis tools. This paper proposes a novel approach leveraging Generative Artificial Intelligence (GenAI) to facilitate streamlined video analysis. Our tool aims to deliver tailored textual summaries of user-defined queries, offering a focused insight amidst extensive video datasets. Unlike conventional frameworks that offer generic summaries or limited action recognition, our method harnesses the power of GenAI to distil relevant information, enhancing analysis precision and efficiency. Employing YOLO-V8 for object detection and Gemini for comprehensive video and text analysis, our solution achieves heightened contextual accuracy. By combining YOLO with Gemini, our approach furnishes textual summaries extracted from extensive CCTV footage, enabling users to swiftly navigate and verify pertinent events without the need for exhaustive manual review. The quantitative evaluation revealed a similarity of 72.8%, while the qualitative assessment rated an accuracy of 85%, demonstrating the capability of the proposed method.
• 摘要：视频内容制作的激增导致了大量数据，对分析效率和资源利用率提出了巨大挑战。解决这一问题需要开发强大的视频分析工具。本文提出了一种利用生成人工智能 (GenAI) 来促进简化视频分析的新方法。我们的工具旨在提供用户定义查询的定制文本摘要，在大量视频数据集中提供有针对性的见解。与提供通用摘要或有限动作识别的传统框架不同，我们的方法利用 GenAI 的强大功能来提取相关信息，从而提高分析精度和效率。我们的解决方案采用 YOLO-V8 进行对象检测，采用 Gemini 进行全面的视频和文本分析，实现了更高的上下文准确性。通过将 YOLO 与 Gemini 相结合，我们的方法提供了从大量闭路电视录像中提取的文本摘要，使用户能够快速浏览和验证相关事件，而无需进行详尽的手动审查。定量评估显示相似度为 72.8%，定性评估准确度为 85%，证明了所提出方法的能力。

Title: TREAD: Token Routing for Efficient Architecture-agnostic Diffusion Training

• Authors: Felix Krause, Timy Phan, Vincent Tao Hu, Björn Ommer
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04765
• Pdf URL: https://arxiv.org/pdf/2501.04765
• Copy Paste: [[2501.04765]] TREAD: Token Routing for Efficient Architecture-agnostic Diffusion Training(https://arxiv.org/abs/2501.04765)
• Keywords: generation
• Abstract: Diffusion models have emerged as the mainstream approach for visual generation. However, these models usually suffer from sample inefficiency and high training costs. This issue is particularly pronounced in the standard diffusion transformer architecture due to its quadratic complexity relative to input length. Recent works have addressed this by reducing the number of tokens processed in the model, often through masking. In contrast, this work aims to improve the training efficiency of the diffusion backbone by using predefined routes that store this information until it is reintroduced to deeper layers of the model, rather than discarding these tokens entirely. Further, we combine multiple routes and introduce an adapted auxiliary loss that accounts for all applied routes. Our method is not limited to the common transformer-based model - it can also be applied to state-space models. Unlike most current approaches, TREAD achieves this without architectural modifications. Finally, we show that our method reduces the computational cost and simultaneously boosts model performance on the standard benchmark ImageNet-1K 256 x 256 in class-conditional synthesis. Both of these benefits multiply to a convergence speedup of 9.55x at 400K training iterations compared to DiT and 25.39x compared to the best benchmark performance of DiT at 7M training iterations.
• 摘要：扩散模型已成为视觉生成的主流方法。然而，这些模型通常存在样本效率低和训练成本高的问题。这个问题在标准扩散变压器架构中尤其明显，因为它相对于输入长度具有二次复杂度。最近的研究通过减少模型中处理的标记数量（通常是通过掩码）解决了这个问题。相反，这项工作旨在通过使用预定义路由来提高扩散主干的训练效率，这些路由存储这些信息，直到将其重新引入模型的更深层，而不是完全丢弃这些标记。此外，我们结合了多条路线，并引入了一种适应性辅助损失，该损失考虑了所有应用的路线。我们的方法不仅限于常见的基于变压器的模型 - 它也可以应用于状态空间模型。与大多数当前方法不同，TREAD 无需修改架构即可实现这一点。最后，我们展示了我们的方法降低了计算成本，同时提高了类条件合成中标准基准 ImageNet-1K 256 x 256 上的模型性能。与 DiT 相比，这两项优势在 400K 次训练迭代中收敛速度提高了 9.55 倍，与 DiT 在 7M 次训练迭代中的最佳基准性能相比，收敛速度提高了 25.39 倍。

Title: Back Home: A Machine Learning Approach to Seashell Classification and Ecosystem Restoration

• Authors: Alexander Valverde, Luis Solano
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04873
• Pdf URL: https://arxiv.org/pdf/2501.04873
• Copy Paste: [[2501.04873]] Back Home: A Machine Learning Approach to Seashell Classification and Ecosystem Restoration(https://arxiv.org/abs/2501.04873)
• Keywords: restoration
• Abstract: In Costa Rica, an average of 5 tons of seashells are extracted from ecosystems annually. Confiscated seashells, cannot be returned to their ecosystems due to the lack of origin recognition. To address this issue, we developed a convolutional neural network (CNN) specifically for seashell identification. We built a dataset from scratch, consisting of approximately 19000 images from the Pacific and Caribbean coasts. Using this dataset, the model achieved a classification accuracy exceeding 85%. The model has been integrated into a user-friendly application, which has classified over 36,000 seashells to date, delivering real-time results within 3 seconds per image. To further enhance the system's accuracy, an anomaly detection mechanism was incorporated to filter out irrelevant or anomalous inputs, ensuring only valid seashell images are processed.
• 摘要：在哥斯达黎加，每年平均有 5 吨贝壳从生态系统中被采掘出来。由于无法识别来源，被没收的贝壳无法归还到生态系统中。为了解决这个问题，我们开发了一个专门用于贝壳识别的卷积神经网络 (CNN)。我们从头开始构建了一个数据集，其中包含来自太平洋和加勒比海岸的大约 19000 张图像。使用该数据集，该模型实现了超过 85% 的分类准确率。该模型已集成到一个用户友好的应用程序中，该应用程序迄今为止已对超过 36,000 个贝壳进行了分类，每张图像在 3 秒内提供实时结果。为了进一步提高系统的准确性，我们加入了异常检测机制来过滤掉不相关或异常的输入，确保只处理有效的贝壳图像。

Title: Online Continual Learning: A Systematic Literature Review of Approaches, Challenges, and Benchmarks

• Authors: Seyed Amir Bidaki, Amir Mohammadkhah, Kiyan Rezaee, Faeze Hassani, Sadegh Eskandari, Maziar Salahi, Mohammad M. Ghassemi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04897
• Pdf URL: https://arxiv.org/pdf/2501.04897
• Copy Paste: [[2501.04897]] Online Continual Learning: A Systematic Literature Review of Approaches, Challenges, and Benchmarks(https://arxiv.org/abs/2501.04897)
• Keywords: generative
• Abstract: Online Continual Learning (OCL) is a critical area in machine learning, focusing on enabling models to adapt to evolving data streams in real-time while addressing challenges such as catastrophic forgetting and the stability-plasticity trade-off. This study conducts the first comprehensive Systematic Literature Review (SLR) on OCL, analyzing 81 approaches, extracting over 1,000 features (specific tasks addressed by these approaches), and identifying more than 500 components (sub-models within approaches, including algorithms and tools). We also review 83 datasets spanning applications like image classification, object detection, and multimodal vision-language tasks. Our findings highlight key challenges, including reducing computational overhead, developing domain-agnostic solutions, and improving scalability in resource-constrained environments. Furthermore, we identify promising directions for future research, such as leveraging self-supervised learning for multimodal and sequential data, designing adaptive memory mechanisms that integrate sparse retrieval and generative replay, and creating efficient frameworks for real-world applications with noisy or evolving task boundaries. By providing a rigorous and structured synthesis of the current state of OCL, this review offers a valuable resource for advancing this field and addressing its critical challenges and opportunities. The complete SLR methodology steps and extracted data are publicly available through the provided link: this https URL Systematic-Literature-Review-on-Online-Continual-Learning
• 摘要：在线持续学习 (OCL) 是机器学习的一个关键领域，专注于使模型能够实时适应不断变化的数据流，同时解决灾难性遗忘和稳定性-可塑性权衡等挑战。本研究对 OCL 进行了首次全面的系统文献综述 (SLR)，分析了 81 种方法，提取了 1,000 多个特征（这些方法解决的特定任务），并识别了 500 多个组件（方法中的子模型，包括算法和工具）。我们还审查了 83 个数据集，涵盖图像分类、对象检测和多模态视觉语言任务等应用。我们的研究结果强调了关键挑战，包括减少计算开销、开发与领域无关的解决方案以及提高资源受限环境中的可扩展性。此外，我们确定了未来研究的有希望的方向，例如利用自监督学习处理多模态和顺序数据，设计集成稀疏检索和生成重放的自适应记忆机制，以及为具有嘈杂或不断变化的任务边界的实际应用创建有效的框架。通过对 OCL 的现状进行严格而结构化的综合，本综述为推进该领域并应对其关键挑战和机遇提供了宝贵的资源。完整的 SLR 方法步骤和提取的数据可通过提供的链接公开获取：此 https URL Systematic-Literature-Review-on-Online-Continual-Learning

Title: From Mesh Completion to AI Designed Crown

• Authors: Golriz Hosseinimanesh, Farnoosh Ghadiri, Francois Guibault, Farida Cheriet, Julia Keren
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04914
• Pdf URL: https://arxiv.org/pdf/2501.04914
• Copy Paste: [[2501.04914]] From Mesh Completion to AI Designed Crown(https://arxiv.org/abs/2501.04914)
• Keywords: generation
• Abstract: Designing a dental crown is a time-consuming and labor intensive process. Our goal is to simplify crown design and minimize the tediousness of making manual adjustments while still ensuring the highest level of accuracy and consistency. To this end, we present a new end- to-end deep learning approach, coined Dental Mesh Completion (DMC), to generate a crown mesh conditioned on a point cloud context. The dental context includes the tooth prepared to receive a crown and its surroundings, namely the two adjacent teeth and the three closest teeth in the opposing jaw. We formulate crown generation in terms of completing this point cloud context. A feature extractor first converts the input point cloud into a set of feature vectors that represent local regions in the point cloud. The set of feature vectors is then fed into a transformer to predict a new set of feature vectors for the missing region (crown). Subsequently, a point reconstruction head, followed by a multi-layer perceptron, is used to predict a dense set of points with normals. Finally, a differentiable point-to-mesh layer serves to reconstruct the crown surface mesh. We compare our DMC method to a graph-based convolutional neural network which learns to deform a crown mesh from a generic crown shape to the target geometry. Extensive experiments on our dataset demonstrate the effectiveness of our method, which attains an average of 0.062 Chamfer this http URL code is available at:this https URL
• 摘要：设计牙冠是一个耗时且劳动密集的过程。我们的目标是简化牙冠设计，尽量减少手动调整的繁琐，同时仍确保最高的准确性和一致性。为此，我们提出了一种新的端到端深度学习方法，称为牙科网格完成 (DMC)，以生成基于点云上下文的牙冠网格。牙科背景包括准备接受牙冠的牙齿及其周围环境，即对颌中的两颗相邻牙齿和三颗最近的牙齿。我们根据完成此点云背景来制定牙冠生成。特征提取器首先将输入点云转换为一组表示点云中局部区域的特征向量。然后将该组特征向量输入到转换器中，以预测缺失区域（牙冠）的一组新特征向量。随后，使用点重建头，然后使用多层感知器，预测具有法线的密集点集。最后，可微分的点到网格层用于重建牙冠表面网格。我们将 DMC 方法与基于图的卷积神经网络进行了比较，后者学习将皇冠网格从通用皇冠形状变形为目标几何形状。在我们的数据集上进行的大量实验证明了我们方法的有效性，平均精度达到 0.062 Chamfer 此 http URL 代码可在以下网址获取：此 https URL

Title: SpecTf: Transformers Enable Data-Driven Imaging Spectroscopy Cloud Detection

• Authors: Jake H. Lee, Michael Kiper, David R. Thompson, Philip G. Brodrick
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04916
• Pdf URL: https://arxiv.org/pdf/2501.04916
• Copy Paste: [[2501.04916]] SpecTf: Transformers Enable Data-Driven Imaging Spectroscopy Cloud Detection(https://arxiv.org/abs/2501.04916)
• Keywords: generation
• Abstract: Current and upcoming generations of visible-shortwave infrared (VSWIR) imaging spectrometers promise unprecedented capacity to quantify Earth System processes across the globe. However, reliable cloud screening remains a fundamental challenge for these instruments, where traditional spatial and temporal approaches are limited by cloud variability and limited temporal coverage. The Spectroscopic Transformer (SpecTf) addresses these challenges with a spectroscopy-specific deep learning architecture that performs cloud detection using only spectral information (no spatial or temporal data are required). By treating spectral measurements as sequences rather than image channels, SpecTf learns fundamental physical relationships without relying on spatial context. Our experiments demonstrate that SpecTf significantly outperforms the current baseline approach implemented for the EMIT instrument, and performs comparably with other machine learning methods with orders of magnitude fewer learned parameters. Critically, we demonstrate SpecTf's inherent interpretability through its attention mechanism, revealing physically meaningful spectral features the model has learned. Finally, we present SpecTf's potential for cross-instrument generalization by applying it to a different instrument on a different platform without modifications, opening the door to instrument agnostic data driven algorithms for future imaging spectroscopy tasks.
• 摘要：当前和即将推出的可见短波红外 (VSWIR) 成像光谱仪有望以前所未有的能力量化全球地球系统过程。然而，可靠的云筛选仍然是这些仪器面临的根本挑战，传统的空间和时间方法受到云变化和时间覆盖范围有限的限制。光谱变压器 (SpecTf) 使用光谱专用的深度学习架构解决了这些挑战，该架构仅使用光谱信息（不需要空间或时间数据）执行云检测。通过将光谱测量视为序列而不是图像通道，SpecTf 可以学习基本的物理关系而不依赖于空间背景。我们的实验表明，SpecTf 的表现明显优于当前为 EMIT 仪器实施的基线方法，并且与其他机器学习方法的表现相当，但学习的参数要少几个数量级。至关重要的是，我们通过其注意力机制展示了 SpecTf 固有的可解释性，揭示了模型学习到的具有物理意义的光谱特征。最后，我们通过将 SpecTf 不经修改地应用于不同平台上的不同仪器，展示了其跨仪器泛化的潜力，为未来成像光谱任务的仪器无关数据驱动算法打开了大门。

Title: MambaHSI: Spatial-Spectral Mamba for Hyperspectral Image Classification

• Authors: Yapeng Li, Yong Luo, Lefei Zhang, Zengmao Wang, Bo Du
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04944
• Pdf URL: https://arxiv.org/pdf/2501.04944
• Copy Paste: [[2501.04944]] MambaHSI: Spatial-Spectral Mamba for Hyperspectral Image Classification(https://arxiv.org/abs/2501.04944)
• Keywords: generation
• Abstract: Transformer has been extensively explored for hyperspectral image (HSI) classification. However, transformer poses challenges in terms of speed and memory usage because of its quadratic computational complexity. Recently, the Mamba model has emerged as a promising approach, which has strong long-distance modeling capabilities while maintaining a linear computational complexity. However, representing the HSI is challenging for the Mamba due to the requirement for an integrated spatial and spectral understanding. To remedy these drawbacks, we propose a novel HSI classification model based on a Mamba model, named MambaHSI, which can simultaneously model long-range interaction of the whole image and integrate spatial and spectral information in an adaptive manner. Specifically, we design a spatial Mamba block (SpaMB) to model the long-range interaction of the whole image at the pixel-level. Then, we propose a spectral Mamba block (SpeMB) to split the spectral vector into multiple groups, mine the relations across different spectral groups, and extract spectral features. Finally, we propose a spatial-spectral fusion module (SSFM) to adaptively integrate spatial and spectral features of a HSI. To our best knowledge, this is the first image-level HSI classification model based on the Mamba. We conduct extensive experiments on four diverse HSI datasets. The results demonstrate the effectiveness and superiority of the proposed model for HSI classification. This reveals the great potential of Mamba to be the next-generation backbone for HSI models. Codes are available at this https URL .
• 摘要：Transformer 已被广泛用于高光谱图像 (HSI) 分类。然而，由于其二次计算复杂度，Transformer 在速度和内存使用方面带来了挑战。最近，Mamba 模型已成为一种有前途的方法，它具有强大的长距离建模能力，同时保持了线性计算复杂度。然而，由于需要综合的空间和光谱理解，表示 HSI 对 Mamba 来说具有挑战性。为了弥补这些缺点，我们提出了一种基于 Mamba 模型的新型 HSI 分类模型，名为 MambaHSI，它可以同时对整个图像的长距离交互进行建模，并以自适应方式整合空间和光谱信息。具体来说，我们设计了一个空间 Mamba 块 (SpaMB) 来在像素级对整个图像的长距离交互进行建模。然后，我们提出了一个光谱 Mamba 块 (SpeMB) 将光谱向量分成多个组，挖掘不同光谱组之间的关系，并提取光谱特征。最后，我们提出了一个空间光谱融合模块 (SSFM) 来自适应地整合 HSI 的空间和光谱特征。据我们所知，这是第一个基于 Mamba 的图像级 HSI 分类模型。我们对四个不同的 HSI 数据集进行了广泛的实验。结果证明了所提出的模型对 HSI 分类的有效性和优越性。这揭示了 Mamba 成为下一代 HSI 模型骨干的巨大潜力。代码可在此 https URL 上找到。

Title: ResPanDiff: Diffusion Model with Disentangled Modulations for Image Fusion

• Authors: Shiqi Cao, Liangjian Deng, Shangqi Deng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05091
• Pdf URL: https://arxiv.org/pdf/2501.05091
• Copy Paste: [[2501.05091]] ResPanDiff: Diffusion Model with Disentangled Modulations for Image Fusion(https://arxiv.org/abs/2501.05091)
• Keywords: generation
• Abstract: The implementation of diffusion-based pansharpening task is predominantly constrained by its slow inference speed, which results from numerous sampling steps. Despite the existing techniques aiming to accelerate sampling, they often compromise performance when fusing multi-source images. To ease this limitation, we introduce a novel and efficient diffusion model named Diffusion Model for Pansharpening by Inferring Residual Inference (ResPanDiff), which significantly reduces the number of diffusion steps without sacrificing the performance to tackle pansharpening task. In ResPanDiff, we innovatively propose a Markov chain that transits from noisy residuals to the residuals between the LRMS and HRMS images, thereby reducing the number of sampling steps and enhancing performance. Additionally, we design the latent space to help model extract more features at the encoding stage, Shallow Cond-Injection~(SC-I) to help model fetch cond-injected hidden features with higher dimensions, and loss functions to give a better guidance for the residual generation task. enabling the model to achieve superior performance in residual generation. Furthermore, experimental evaluations on pansharpening datasets demonstrate that the proposed method achieves superior outcomes compared to recent state-of-the-art~(SOTA) techniques, requiring only 15 sampling steps, which reduces over $90\%$ step compared with the benchmark diffusion models. Our experiments also include thorough discussions and ablation studies to underscore the effectiveness of our approach.
• 摘要：基于扩散的全色锐化任务的实现主要受限于其推理速度慢，这是由于采样步骤繁多造成的。尽管现有技术旨在加速采样，但它们在融合多源图像时往往会损害性能。为了缓解这一限制，我们引入了一种新颖而高效的扩散模型，称为通过推断残差进行全色锐化的扩散模型 (ResPanDiff)，该模型在不牺牲性能的情况下显著减少了扩散步骤的数量以解决全色锐化任务。在 ResPanDiff 中，我们创新地提出了一种马尔可夫链，该链从噪声残差过渡到 LRMS 和 HRMS 图像之间的残差，从而减少了采样步骤的数量并提高了性能。此外，我们设计了潜在空间以帮助模型在编码阶段提取更多特征，浅条件注入~(SC-I) 以帮助模型获取具有更高维度的条件注入隐藏特征，并设计了损失函数以更好地指导残差生成任务。使模型在残差生成方面取得优异的性能。此外，在全色锐化数据集上的实验评估表明，与最近的最先进~(SOTA) 技术相比，所提出的方法取得了优异的结果，只需要 15 个采样步骤，与基准扩散模型相比，减少了超过 $90\%$ 的步骤。我们的实验还包括彻底的讨论和消融研究，以强调我们方法的有效性。

Title: Hierarchical Decomposed Dual-domain Deep Learning for Sparse-View CT Reconstruction

• Authors: Yoseob Han
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2501.05093
• Pdf URL: https://arxiv.org/pdf/2501.05093
• Copy Paste: [[2501.05093]] Hierarchical Decomposed Dual-domain Deep Learning for Sparse-View CT Reconstruction(https://arxiv.org/abs/2501.05093)
• Keywords: restoration
• Abstract: Objective: X-ray computed tomography employing sparse projection views has emerged as a contemporary technique to mitigate radiation dose. However, due to the inadequate number of projection views, an analytic reconstruction method utilizing filtered backprojection results in severe streaking artifacts. Recently, deep learning strategies employing image-domain networks have demonstrated remarkable performance in eliminating the streaking artifact caused by analytic reconstruction methods with sparse projection views. Nevertheless, it is difficult to clarify the theoretical justification for applying deep learning to sparse view CT reconstruction, and it has been understood as restoration by removing image artifacts, not reconstruction. Approach: By leveraging the theory of deep convolutional framelets and the hierarchical decomposition of measurement, this research reveals the constraints of conventional image- and projection-domain deep learning methodologies, subsequently, the research proposes a novel dual-domain deep learning framework utilizing hierarchical decomposed measurements. Specifically, the research elucidates how the performance of the projection-domain network can be enhanced through a low-rank property of deep convolutional framelets and a bowtie support of hierarchical decomposed measurement in the Fourier domain. Main Results: This study demonstrated performance improvement of the proposed framework based on the low-rank property, resulting in superior reconstruction performance compared to conventional analytic and deep learning methods. Significance: By providing a theoretically justified deep learning approach for sparse-view CT reconstruction, this study not only offers a superior alternative to existing methods but also opens new avenues for research in medical imaging.
• 摘要：目的：采用稀疏投影视图的X射线计算机断层扫描已成为一种当代减轻辐射剂量的技术。然而，由于投影视图数量不足，采用滤波反投影的解析重建方法会导致严重的条纹伪影。最近，采用图像域网络的深度学习策略在消除稀疏投影视图解析重建方法引起的条纹伪影方面表现出色。然而，很难阐明将深度学习应用于稀疏视图CT重建的理论依据，并且它被理解为通过去除图像伪影进行的恢复，而不是重建。方法：通过利用深度卷积框架理论和测量的分层分解，本研究揭示了传统图像和投影域深度学习方法的局限性，随后，该研究提出了一种利用分层分解测量的新型双域深度学习框架。具体而言，该研究阐明了如何通过深度卷积框架的低秩属性和傅里叶域中分层分解测量的领结支持来增强投影域网络的性能。主要结果：本研究证明了基于低秩属性的所提框架的性能改进，与传统的分析和深度学习方法相比，其重建性能更优异。意义：通过为稀疏视图 CT 重建提供理论上合理的深度学习方法，本研究不仅为现有方法提供了一种更好的替代方案，而且还为医学成像研究开辟了新的途径。

Title: Motion-X++: A Large-Scale Multimodal 3D Whole-body Human Motion Dataset

• Authors: Yuhong Zhang, Jing Lin, Ailing Zeng, Guanlin Wu, Shunlin Lu, Yurong Fu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, Lei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05098
• Pdf URL: https://arxiv.org/pdf/2501.05098
• Copy Paste: [[2501.05098]] Motion-X++: A Large-Scale Multimodal 3D Whole-body Human Motion Dataset(https://arxiv.org/abs/2501.05098)
• Keywords: generation
• Abstract: In this paper, we introduce Motion-X++, a large-scale multimodal 3D expressive whole-body human motion dataset. Existing motion datasets predominantly capture body-only poses, lacking facial expressions, hand gestures, and fine-grained pose descriptions, and are typically limited to lab settings with manually labeled text descriptions, thereby restricting their scalability. To address this issue, we develop a scalable annotation pipeline that can automatically capture 3D whole-body human motion and comprehensive textural labels from RGB videos and build the Motion-X dataset comprising 81.1K text-motion pairs. Furthermore, we extend Motion-X into Motion-X++ by improving the annotation pipeline, introducing more data modalities, and scaling up the data quantities. Motion-X++ provides 19.5M 3D whole-body pose annotations covering 120.5K motion sequences from massive scenes, 80.8K RGB videos, 45.3K audios, 19.5M frame-level whole-body pose descriptions, and 120.5K sequence-level semantic labels. Comprehensive experiments validate the accuracy of our annotation pipeline and highlight Motion-X++'s significant benefits for generating expressive, precise, and natural motion with paired multimodal labels supporting several downstream tasks, including text-driven whole-body motion generation,audio-driven motion generation, 3D whole-body human mesh recovery, and 2D whole-body keypoints estimation, etc.
• 摘要：在本文中，我们介绍了 Motion-X++，这是一个大规模多模态 3D 富有表现力的全身人体运动数据集。现有的运动数据集主要捕捉身体姿势，缺乏面部表情、手势和细粒度姿势描述，并且通常仅限于具有手动标记文本描述的实验室设置，从而限制了它们的可扩展性。为了解决这个问题，我们开发了一个可扩展的注释管道，可以自动从 RGB 视频中捕捉 3D 全身人体运动和全面的纹理标签，并构建包含 81.1K 文本运动对的 Motion-X 数据集。此外，我们通过改进注释管道、引入更多数据模态和扩大数据量，将 Motion-X 扩展为 Motion-X++。 Motion-X++ 提供 19.5M 3D 全身姿势注释，涵盖来自海量场景的 120.5K 运动序列、80.8K RGB 视频、45.3K 音频、19.5M 帧级全身姿势描述和 120.5K 序列级语义标签。全面的实验验证了我们的注释流程的准确性，并突出了 Motion-X++ 在使用成对多模态标签生成富有表现力、精确和自然的运动方面的显著优势，支持多个下游任务，包括文本驱动的全身运动生成、音频驱动的运动生成、3D 全身人体网格恢复和 2D 全身关键点估计等。

Title: EquiBoost: An Equivariant Boosting Approach to Molecular Conformation Generation

• Authors: Yixuan Yang, Xingyu Fang, Zhaowen Cheng, Pengju Yan, Xiaolin Li
• Subjects: cs.LG, physics.chem-ph, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2501.05109
• Pdf URL: https://arxiv.org/pdf/2501.05109
• Copy Paste: [[2501.05109]] EquiBoost: An Equivariant Boosting Approach to Molecular Conformation Generation(https://arxiv.org/abs/2501.05109)
• Keywords: generation
• Abstract: Molecular conformation generation plays key roles in computational drug design. Recently developed deep learning methods, particularly diffusion models have reached competitive performance over traditional cheminformatical approaches. However, these methods are often time-consuming or require extra support from traditional methods. We propose EquiBoost, a boosting model that stacks several equivariant graph transformers as weak learners, to iteratively refine 3D conformations of molecules. Without relying on diffusion techniques, EquiBoost balances accuracy and efficiency more effectively than diffusion-based methods. Notably, compared to the previous state-of-the-art diffusion method, EquiBoost improves generation quality and preserves diversity, achieving considerably better precision of Average Minimum RMSD (AMR) on the GEOM datasets. This work rejuvenates boosting and sheds light on its potential to be a robust alternative to diffusion models in certain scenarios.
• 摘要：分子构象生成在计算药物设计中起着关键作用。最近开发的深度学习方法，特别是扩散模型，已经达到了与传统化学信息学方法相当的性能。然而，这些方法通常很耗时或需要传统方法的额外支持。我们提出了 EquiBoost，这是一种增强模型，它将几个等变图变换器堆叠为弱学习器，以迭代方式细化分子的 3D 构象。EquiBoost 不依赖扩散技术，因此比基于扩散的方法更有效地平衡了准确性和效率。值得注意的是，与之前最先进的扩散方法相比，EquiBoost 提高了生成质量并保留了多样性，在 GEOM 数据集上实现了平均最小 RMSD (AMR) 的明显更好的精度。这项工作使增强模型焕发了活力，并揭示了它在某些情况下成为扩散模型的强大替代方案的潜力。

Title: 3DIS-FLUX: simple and efficient multi-instance generation with DiT rendering

• Authors: Dewei Zhou, Ji Xie, Zongxin Yang, Yi Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05131
• Pdf URL: https://arxiv.org/pdf/2501.05131
• Copy Paste: [[2501.05131]] 3DIS-FLUX: simple and efficient multi-instance generation with DiT rendering(https://arxiv.org/abs/2501.05131)
• Keywords: generation
• Abstract: The growing demand for controllable outputs in text-to-image generation has driven significant advancements in multi-instance generation (MIG), enabling users to define both instance layouts and attributes. Currently, the state-of-the-art methods in MIG are primarily adapter-based. However, these methods necessitate retraining a new adapter each time a more advanced model is released, resulting in significant resource consumption. A methodology named Depth-Driven Decoupled Instance Synthesis (3DIS) has been introduced, which decouples MIG into two distinct phases: 1) depth-based scene construction and 2) detail rendering with widely pre-trained depth control models. The 3DIS method requires adapter training solely during the scene construction phase, while enabling various models to perform training-free detail rendering. Initially, 3DIS focused on rendering techniques utilizing U-Net architectures such as SD1.5, SD2, and SDXL, without exploring the potential of recent DiT-based models like FLUX. In this paper, we present 3DIS-FLUX, an extension of the 3DIS framework that integrates the FLUX model for enhanced rendering capabilities. Specifically, we employ the FLUX.1-Depth-dev model for depth map controlled image generation and introduce a detail renderer that manipulates the Attention Mask in FLUX's Joint Attention mechanism based on layout information. This approach allows for the precise rendering of fine-grained attributes of each instance. Our experimental results indicate that 3DIS-FLUX, leveraging the FLUX model, outperforms the original 3DIS method, which utilized SD2 and SDXL, and surpasses current state-of-the-art adapter-based methods in terms of both performance and image quality. Project Page: this https URL.
• 摘要：文本到图像生成中对可控输出的需求不断增长，推动了多实例生成 (MIG) 的重大进步，使用户能够定义实例布局和属性。目前，MIG 中最先进的方法主要是基于适配器的。然而，这些方法每次发布更先进的模型时都需要重新训练新的适配器，从而导致大量资源消耗。一种名为深度驱动解耦实例合成 (3DIS) 的方法已被引入，它将 MIG 解耦为两个不同的阶段：1) 基于深度的场景构建和 2) 使用广泛预训练的深度控制模型进行细节渲染。3DIS 方法只需要在场景构建阶段进行适配器训练，同时使各种模型能够执行无需训练的细节渲染。最初，3DIS 专注于利用 U-Net 架构（例如 SD1.5、SD2 和 SDXL）的渲染技术，而没有探索 FLUX 等基于 DiT 的最新模型的潜力。在本文中，我们介绍了 3DIS-FLUX，这是 3DIS 框架的扩展，它集成了 FLUX 模型以增强渲染功能。具体来说，我们采用 FLUX.1-Depth-dev 模型来生成深度图控制的图像，并引入了一个细节渲染器，该渲染器基于布局信息操纵 FLUX 联合注意力机制中的注意力掩码。这种方法可以精确渲染每个实例的细粒度属性。我们的实验结果表明，利用 FLUX 模型的 3DIS-FLUX 优于使用 SD2 和 SDXL 的原始 3DIS 方法，并且在性能和图像质量方面超越了当前最先进的基于适配器的方法。项目页面：此 https URL。

Title: FaceMe: Robust Blind Face Restoration with Personal Identification

• Authors: Siyu Liu, Zheng-Peng Duan, Jia OuYang, Jiayi Fu, Hyunhee Park, Zikun Liu, Chun-Le Guo, Chongyi Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05177
• Pdf URL: https://arxiv.org/pdf/2501.05177
• Copy Paste: [[2501.05177]] FaceMe: Robust Blind Face Restoration with Personal Identification(https://arxiv.org/abs/2501.05177)
• Keywords: restoration
• Abstract: Blind face restoration is a highly ill-posed problem due to the lack of necessary context. Although existing methods produce high-quality outputs, they often fail to faithfully preserve the individual's identity. In this paper, we propose a personalized face restoration method, FaceMe, based on a diffusion model. Given a single or a few reference images, we use an identity encoder to extract identity-related features, which serve as prompts to guide the diffusion model in restoring high-quality and identity-consistent facial images. By simply combining identity-related features, we effectively minimize the impact of identity-irrelevant features during training and support any number of reference image inputs during inference. Additionally, thanks to the robustness of the identity encoder, synthesized images can be used as reference images during training, and identity changing during inference does not require fine-tuning the model. We also propose a pipeline for constructing a reference image training pool that simulates the poses and expressions that may appear in real-world scenarios. Experimental results demonstrate that our FaceMe can restore high-quality facial images while maintaining identity consistency, achieving excellent performance and robustness.
• 摘要：由于缺乏必要的背景信息，盲人脸恢复是一个高度不适定的问题。尽管现有的方法可以产生高质量的输出，但它们往往无法忠实地保留个人的身份。在本文中，我们提出了一种基于扩散模型的个性化人脸恢复方法 FaceMe。给定一张或几张参考图像，我们使用身份编码器提取与身份相关的特征，这些特征作为提示来指导扩散模型恢复高质量且身份一致的人脸图像。通过简单地组合与身份相关的特征，我们有效地将与身份无关的特征在训练过程中的影响降至最低，并在推理过程中支持任意数量的参考图像输入。此外，由于身份编码器的鲁棒性，合成图像可以在训练过程中用作参考图像，并且在推理过程中更改身份不需要对模型进行微调。我们还提出了一个用于构建参考图像训练池的流程，该池模拟了真实场景中可能出现的姿势和表情。实验结果表明，我们的 FaceMe 可以在保持身份一致性的同时恢复高质量的人脸图像，实现出色的性能和鲁棒性。

Title: Light Transport-aware Diffusion Posterior Sampling for Single-View Reconstruction of 3D Volumes

• Authors: Ludwic Leonard, Nils Thuerey, Ruediger Westermann
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05226
• Pdf URL: https://arxiv.org/pdf/2501.05226
• Copy Paste: [[2501.05226]] Light Transport-aware Diffusion Posterior Sampling for Single-View Reconstruction of 3D Volumes(https://arxiv.org/abs/2501.05226)
• Keywords: generative
• Abstract: We introduce a single-view reconstruction technique of volumetric fields in which multiple light scattering effects are omnipresent, such as in clouds. We model the unknown distribution of volumetric fields using an unconditional diffusion model trained on a novel benchmark dataset comprising 1,000 synthetically simulated volumetric density fields. The neural diffusion model is trained on the latent codes of a novel, diffusion-friendly, monoplanar representation. The generative model is used to incorporate a tailored parametric diffusion posterior sampling technique into different reconstruction tasks. A physically-based differentiable volume renderer is employed to provide gradients with respect to light transport in the latent space. This stands in contrast to classic NeRF approaches and makes the reconstructions better aligned with observed data. Through various experiments, we demonstrate single-view reconstruction of volumetric clouds at a previously unattainable quality.
• 摘要：我们介绍了一种体积场的单视图重建技术，其中多种光散射效应无处不在，例如在云中。我们使用一个无条件扩散模型来对体积场的未知分布进行建模，该模型在一个包含 1,000 个合成模拟的体积密度场的新型基准数据集上进行训练。神经扩散模型是在一种新型、扩散友好的单平面表示的潜在代码上进行训练的。生成模型用于将定制的参数扩散后验采样技术结合到不同的重建任务中。采用基于物理的可微体积渲染器来提供潜在空间中光传输的梯度。这与经典的 NeRF 方法形成鲜明对比，使重建与观察到的数据更加一致。通过各种实验，我们展示了以前无法达到的质量的体积云的单视图重建。

Title: Deriving Coding-Specific Sub-Models from LLMs using Resource-Efficient Pruning

• Authors: Laura Puccioni, Alireza Farshin, Mariano Scazzariello, Changjie Wang, Marco Chiesa, Dejan Kostic
• Subjects: cs.LG, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2501.05248
• Pdf URL: https://arxiv.org/pdf/2501.05248
• Copy Paste: [[2501.05248]] Deriving Coding-Specific Sub-Models from LLMs using Resource-Efficient Pruning(https://arxiv.org/abs/2501.05248)
• Keywords: generation
• Abstract: Large Language Models (LLMs) have demonstrated their exceptional performance in various complex code generation tasks. However, their broader adoption is limited by significant computational demands and high resource requirements, particularly memory and processing power. To mitigate such requirements, model pruning techniques are used to create more compact models with significantly fewer parameters. However, current approaches do not focus on the efficient extraction of programming-language-specific sub-models. In this work, we explore the idea of efficiently deriving coding-specific sub-models through unstructured pruning (i.e., Wanda). We investigate the impact of different domain-specific calibration datasets on pruning outcomes across three distinct domains and extend our analysis to extracting four language-specific sub-models: Python, Java, C++, and JavaScript. We are the first to efficiently extract programming-language-specific sub-models using appropriate calibration datasets while maintaining acceptable accuracy w.r.t. full models. We are also the first to provide analytical evidence that domain-specific tasks activate distinct regions within LLMs, supporting the creation of specialized sub-models through unstructured pruning. We believe that this work has significant potential to enhance LLM accessibility for coding by reducing computational requirements to enable local execution on consumer-grade hardware, and supporting faster inference times critical for real-time development feedback.
• 摘要：大型语言模型 (LLM) 在各种复杂的代码生成任务中都表现出色。然而，它们更广泛的应用受到大量计算需求和高资源要求（尤其是内存和处理能力）的限制。为了减轻这些要求，模型修剪技术用于创建具有明显更少参数的更紧凑模型。然而，当前的方法并不注重有效提取特定于编程语言的子模型。在这项工作中，我们探索了通过非结构化修剪（即 Wanda）有效推导特定于编码的子模型的想法。我们研究了不同领域特定校准数据集对三个不同领域的修剪结果的影响，并将我们的分析扩展到提取四种特定于语言的子模型：Python、Java、C++ 和 JavaScript。我们是第一个使用适当的校准数据集有效提取特定于编程语言的子模型，同时保持相对于完整模型的可接受准确性。我们还首次提供了分析证据，证明特定领域的任务会激活 LLM 中的不同区域，从而支持通过非结构化修剪创建专门的子模型。我们认为，这项工作具有巨大的潜力，可以降低计算要求，实现消费级硬件上的本地执行，并支持对实时开发反馈至关重要的更快推理时间，从而提高 LLM 的编码可访问性。

Title: Patch-GAN Transfer Learning with Reconstructive Models for Cloud Removal

• Authors: Wanli Ma, Oktay Karakus, Paul L. Rosin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05265
• Pdf URL: https://arxiv.org/pdf/2501.05265
• Copy Paste: [[2501.05265]] Patch-GAN Transfer Learning with Reconstructive Models for Cloud Removal(https://arxiv.org/abs/2501.05265)
• Keywords: generation, generative
• Abstract: Cloud removal plays a crucial role in enhancing remote sensing image analysis, yet accurately reconstructing cloud-obscured regions remains a significant challenge. Recent advancements in generative models have made the generation of realistic images increasingly accessible, offering new opportunities for this task. Given the conceptual alignment between image generation and cloud removal tasks, generative models present a promising approach for addressing cloud removal in remote sensing. In this work, we propose a deep transfer learning approach built on a generative adversarial network (GAN) framework to explore the potential of the novel masked autoencoder (MAE) image reconstruction model in cloud removal. Due to the complexity of remote sensing imagery, we further propose using a patch-wise discriminator to determine whether each patch of the image is real or not. The proposed reconstructive transfer learning approach demonstrates significant improvements in cloud removal performance compared to other GAN-based methods. Additionally, whilst direct comparisons with some of the state-of-the-art cloud removal techniques are limited due to unclear details regarding their train/test data splits, the proposed model achieves competitive results based on available benchmarks.
• 摘要：云去除在增强遥感图像分析方面起着至关重要的作用，但准确重建云遮蔽区域仍然是一项重大挑战。生成模型的最新进展使得生成逼真的图像变得越来越容易，为这项任务提供了新的机会。鉴于图像生成和云去除任务之间的概念一致性，生成模型为解决遥感中的云去除问题提供了一种有前途的方法。在这项工作中，我们提出了一种基于生成对抗网络 (GAN) 框架的深度迁移学习方法，以探索新型掩蔽自编码器 (MAE) 图像重建模型在云去除中的潜力。由于遥感图像的复杂性，我们进一步建议使用逐块鉴别器来确定图像的每个块是否真实。与其他基于 GAN 的方法相比，所提出的重建迁移学习方法在云去除性能方面表现出显着的改进。此外，虽然由于有关其训练/测试数据分割的细节不明确，与一些最先进的云去除技术的直接比较受到限制，但所提出的模型根据可用的基准取得了有竞争力的结果。

Title: CellViT++: Energy-Efficient and Adaptive Cell Segmentation and Classification Using Foundation Models

• Authors: Fabian Hörst, Moritz Rempe, Helmut Becker, Lukas Heine, Julius Keyl, Jens Kleesiek
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05269
• Pdf URL: https://arxiv.org/pdf/2501.05269
• Copy Paste: [[2501.05269]] CellViT++: Energy-Efficient and Adaptive Cell Segmentation and Classification Using Foundation Models(https://arxiv.org/abs/2501.05269)
• Keywords: generation
• Abstract: Digital Pathology is a cornerstone in the diagnosis and treatment of diseases. A key task in this field is the identification and segmentation of cells in hematoxylin and eosin-stained images. Existing methods for cell segmentation often require extensive annotated datasets for training and are limited to a predefined cell classification scheme. To overcome these limitations, we propose $\text{CellViT}^{\scriptscriptstyle ++}$, a framework for generalized cell segmentation in digital pathology. $\text{CellViT}^{\scriptscriptstyle ++}$ utilizes Vision Transformers with foundation models as encoders to compute deep cell features and segmentation masks simultaneously. To adapt to unseen cell types, we rely on a computationally efficient approach. It requires minimal data for training and leads to a drastically reduced carbon footprint. We demonstrate excellent performance on seven different datasets, covering a broad spectrum of cell types, organs, and clinical settings. The framework achieves remarkable zero-shot segmentation and data-efficient cell-type classification. Furthermore, we show that $\text{CellViT}^{\scriptscriptstyle ++}$ can leverage immunofluorescence stainings to generate training datasets without the need for pathologist annotations. The automated dataset generation approach surpasses the performance of networks trained on manually labeled data, demonstrating its effectiveness in creating high-quality training datasets without expert annotations. To advance digital pathology, $\text{CellViT}^{\scriptscriptstyle ++}$ is available as an open-source framework featuring a user-friendly, web-based interface for visualization and annotation. The code is available under this https URL.
• 摘要：数字病理学是疾病诊断和治疗的基石。该领域的一项关键任务是识别和分割苏木精和伊红染色图像中的细胞。现有的细胞分割方法通常需要大量带注释的数据集进行训练，并且仅限于预定义的细胞分类方案。为了克服这些限制，我们提出了 $\text{CellViT}^{\scriptscriptstyle ++}$，这是一个用于数字病理学中广义细胞分割的框架。$\text{CellViT}^{\scriptscriptstyle ++}$ 利用带有基础模型的 Vision Transformers 作为编码器来同时计算深度细胞特征和分割掩模。为了适应看不见的细胞类型，我们依靠一种计算效率高的方法。它需要最少的数据进行训练，并可大幅减少碳足迹。我们在七种不同的数据集上展示了出色的性能，涵盖了广泛的细胞类型、器官和临床环境。该框架实现了卓越的零样本分割和数据高效的细胞类型分类。此外，我们表明，$\text{CellViT}^{\scriptscriptstyle ++}$ 可以利用免疫荧光染色生成训练数据集，而无需病理学家注释。自动数据集生成方法的性能超过了使用手动标记数据训练的网络，证明了其在创建高质量训练数据集方面没有专家注释的有效性。为了推进数字病理学，$\text{CellViT}^{\scriptscriptstyle ++}$ 是一个开源框架，具有用户友好的基于 Web 的可视化和注释界面。代码可在此 https URL 下找到。

Title: JAQ: Joint Efficient Architecture Design and Low-Bit Quantization with Hardware-Software Co-Exploration

• Authors: Mingzi Wang, Yuan Meng, Chen Tang, Weixiang Zhang, Yijian Qin, Yang Yao, Yingxin Li, Tongtong Feng, Xin Wang, Xun Guan, Zhi Wang, Wenwu Zhu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05339
• Pdf URL: https://arxiv.org/pdf/2501.05339
• Copy Paste: [[2501.05339]] JAQ: Joint Efficient Architecture Design and Low-Bit Quantization with Hardware-Software Co-Exploration(https://arxiv.org/abs/2501.05339)
• Keywords: generation
• Abstract: The co-design of neural network architectures, quantization precisions, and hardware accelerators offers a promising approach to achieving an optimal balance between performance and efficiency, particularly for model deployment on resource-constrained edge devices. In this work, we propose the JAQ Framework, which jointly optimizes the three critical dimensions. However, effectively automating the design process across the vast search space of those three dimensions poses significant challenges, especially when pursuing extremely low-bit quantization. Specifical, the primary challenges include: (1) Memory overhead in software-side: Low-precision quantization-aware training can lead to significant memory usage due to storing large intermediate features and latent weights for back-propagation, potentially causing memory exhaustion. (2) Search time-consuming in hardware-side: The discrete nature of hardware parameters and the complex interplay between compiler optimizations and individual operators make the accelerator search time-consuming. To address these issues, JAQ mitigates the memory overhead through a channel-wise sparse quantization (CSQ) scheme, selectively applying quantization to the most sensitive components of the model during optimization. Additionally, JAQ designs BatchTile, which employs a hardware generation network to encode all possible tiling modes, thereby speeding up the search for the optimal compiler mapping strategy. Extensive experiments demonstrate the effectiveness of JAQ, achieving approximately 7% higher Top-1 accuracy on ImageNet compared to previous methods and reducing the hardware search time per iteration to 0.15 seconds.
• 摘要：神经网络架构、量化精度和硬件加速器的共同设计提供了一种有前途的方法，以实现性能和效率之间的最佳平衡，特别是对于在资源受限的边缘设备上部署模型而言。在这项工作中，我们提出了 JAQ 框架，它联合优化了三个关键维度。然而，在三个维度的广阔搜索空间中有效地自动化设计过程带来了重大挑战，尤其是在追求极低位量化时。具体来说，主要挑战包括：（1）软件端的内存开销：低精度量化感知训练会导致大量内存使用，因为要存储大量中间特征和反向传播的潜在权重，从而可能导致内存耗尽。（2）硬件端搜索耗时：硬件参数的离散性质以及编译器优化和各个运算符之间的复杂相互作用使加速器搜索耗时。为了解决这些问题，JAQ 通过通道稀疏量化 (CSQ) 方案减轻了内存开销，在优化过程中有选择地将量化应用于模型中最敏感的组件。此外，JAQ 还设计了 BatchTile，它使用硬件生成网络对所有可能的平铺模式进行编码，从而加快了对最佳编译器映射策略的搜索。大量实验证明了 JAQ 的有效性，与之前的方法相比，它在 ImageNet 上的 Top-1 准确率提高了约 7%，并且每次迭代的硬件搜索时间缩短至 0.15 秒。

Title: CROPS: Model-Agnostic Training-Free Framework for Safe Image Synthesis with Latent Diffusion Models

• Authors: Junha Park, Ian Ryu, Jaehui Hwang, Hyungkeun Park, Jiyoon Kim, Jong-Seok Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05359
• Pdf URL: https://arxiv.org/pdf/2501.05359
• Copy Paste: [[2501.05359]] CROPS: Model-Agnostic Training-Free Framework for Safe Image Synthesis with Latent Diffusion Models(https://arxiv.org/abs/2501.05359)
• Keywords: generation
• Abstract: With advances in diffusion models, image generation has shown significant performance improvements. This raises concerns about the potential abuse of image generation, such as the creation of explicit or violent images, commonly referred to as Not Safe For Work (NSFW) content. To address this, the Stable Diffusion model includes several safety checkers to censor initial text prompts and final output images generated from the model. However, recent research has shown that these safety checkers have vulnerabilities against adversarial attacks, allowing them to generate NSFW images. In this paper, we find that these adversarial attacks are not robust to small changes in text prompts or input latents. Based on this, we propose CROPS (Circular or RandOm Prompts for Safety), a model-agnostic framework that easily defends against adversarial attacks generating NSFW images without requiring additional training. Moreover, we develop an approach that utilizes one-step diffusion models for efficient NSFW detection (CROPS-1), further reducing computational resources. We demonstrate the superiority of our method in terms of performance and applicability.
• 摘要：随着扩散模型的进步，图像生成已显示出显着的性能改进。这引发了人们对图像生成可能被滥用的担忧，例如创建露骨或暴力图像，通常称为不适合工作 (NSFW) 的内容。为了解决这个问题，稳定扩散模型包含几个安全检查器来审查初始文本提示和从模型生成的最终输出图像。然而，最近的研究表明，这些安全检查器存在对抗攻击漏洞，允许它们生成 NSFW 图像。在本文中，我们发现这些对抗攻击对文本提示或输入潜伏期的微小变化不具有鲁棒性。基于此，我们提出了 CROPS（循环或随机安全提示），这是一个与模型无关的框架，可以轻松防御生成 NSFW 图像的对抗攻击，而无需额外的训练。此外，我们开发了一种利用一步扩散模型进行有效 NSFW 检测 (CROPS-1) 的方法，进一步减少了计算资源。我们证明了我们方法在性能和适用性方面的优越性。

Title: 1-2-1: Renaissance of Single-Network Paradigm for Virtual Try-On

• Authors: Shuliang Ning, Yipeng Qin, Xiaoguang Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05369
• Pdf URL: https://arxiv.org/pdf/2501.05369
• Copy Paste: [[2501.05369]] 1-2-1: Renaissance of Single-Network Paradigm for Virtual Try-On(https://arxiv.org/abs/2501.05369)
• Keywords: generative
• Abstract: Virtual Try-On (VTON) has become a crucial tool in ecommerce, enabling the realistic simulation of garments on individuals while preserving their original appearance and pose. Early VTON methods relied on single generative networks, but challenges remain in preserving fine-grained garment details due to limitations in feature extraction and fusion. To address these issues, recent approaches have adopted a dual-network paradigm, incorporating a complementary "ReferenceNet" to enhance garment feature extraction and fusion. While effective, this dual-network approach introduces significant computational overhead, limiting its scalability for high-resolution and long-duration image/video VTON applications. In this paper, we challenge the dual-network paradigm by proposing a novel single-network VTON method that overcomes the limitations of existing techniques. Our method, namely MNVTON, introduces a Modality-specific Normalization strategy that separately processes text, image and video inputs, enabling them to share the same attention layers in a VTON network. Extensive experimental results demonstrate the effectiveness of our approach, showing that it consistently achieves higher-quality, more detailed results for both image and video VTON tasks. Our results suggest that the single-network paradigm can rival the performance of dualnetwork approaches, offering a more efficient alternative for high-quality, scalable VTON applications.
• 摘要：虚拟试穿 (VTON) 已成为电子商务中的重要工具，它能够在保留个人原始外观和姿势的同时逼真地模拟服装在个人身上的穿着。早期的 VTON 方法依赖于单个生成网络，但由于特征提取和融合的限制，在保留细粒度服装细节方面仍然存在挑战。为了解决这些问题，最近的方法采用了双网络范式，结合了互补的“ReferenceNet”来增强服装特征提取和融合。虽然这种双网络方法有效，但它会带来大量计算开销，限制其在高分辨率和长时间图像/视频 VTON 应用中的可扩展性。在本文中，我们通过提出一种新颖的单网络 VTON 方法来挑战双网络范式，该方法克服了现有技术的局限性。我们的方法，即 MNVTON，引入了一种模态特定的规范化策略，该策略分别处理文本、图像和视频输入，使它们能够在 VTON 网络中共享相同的注意层。大量实验结果证明了我们方法的有效性，表明该方法始终能够为图像和视频 VTON 任务实现更高质量、更详细的结果。我们的结果表明，单网络范式的性能可以与双网络方法相媲美，为高质量、可扩展的 VTON 应用提供了更高效的替代方案。

Title: Accelerated Diffusion Models via Speculative Sampling

• Authors: Valentin De Bortoli, Alexandre Galashov, Arthur Gretton, Arnaud Doucet
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.05370
• Pdf URL: https://arxiv.org/pdf/2501.05370
• Copy Paste: [[2501.05370]] Accelerated Diffusion Models via Speculative Sampling(https://arxiv.org/abs/2501.05370)
• Keywords: generation
• Abstract: Speculative sampling is a popular technique for accelerating inference in Large Language Models by generating candidate tokens using a fast draft model and accepting or rejecting them based on the target model's distribution. While speculative sampling was previously limited to discrete sequences, we extend it to diffusion models, which generate samples via continuous, vector-valued Markov chains. In this context, the target model is a high-quality but computationally expensive diffusion model. We propose various drafting strategies, including a simple and effective approach that does not require training a draft model and is applicable out of the box to any diffusion model. Our experiments demonstrate significant generation speedup on various diffusion models, halving the number of function evaluations, while generating exact samples from the target model.
• 摘要：推测性采样是一种流行的技术，它通过使用快速草稿模型生成候选标记并根据目标模型的分布接受或拒绝它们来加速大型语言模型中的推理。虽然推测性采样以前仅限于离散序列，但我们将其扩展到扩散模型，该模型通过连续的向量值马尔可夫链生成样本。在这种情况下，目标模型是一个高质量但计算成本高的扩散模型。我们提出了各种草稿策略，包括一种简单有效的方法，该方法不需要训练草稿模型并且可以开箱即用地应用于任何扩散模型。我们的实验表明，各种扩散模型的生成速度显著加快，将函数评估次数减半，同时从目标模型生成精确样本。

Title: Arc2Avatar: Generating Expressive 3D Avatars from a Single Image via ID Guidance

• Authors: Dimitrios Gerogiannis, Foivos Paraperas Papantoniou, Rolandos Alexandros Potamias, Alexandros Lattas, Stefanos Zafeiriou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05379
• Pdf URL: https://arxiv.org/pdf/2501.05379
• Copy Paste: [[2501.05379]] Arc2Avatar: Generating Expressive 3D Avatars from a Single Image via ID Guidance(https://arxiv.org/abs/2501.05379)
• Keywords: generation
• Abstract: Inspired by the effectiveness of 3D Gaussian Splatting (3DGS) in reconstructing detailed 3D scenes within multi-view setups and the emergence of large 2D human foundation models, we introduce Arc2Avatar, the first SDS-based method utilizing a human face foundation model as guidance with just a single image as input. To achieve that, we extend such a model for diverse-view human head generation by fine-tuning on synthetic data and modifying its conditioning. Our avatars maintain a dense correspondence with a human face mesh template, allowing blendshape-based expression generation. This is achieved through a modified 3DGS approach, connectivity regularizers, and a strategic initialization tailored for our task. Additionally, we propose an optional efficient SDS-based correction step to refine the blendshape expressions, enhancing realism and diversity. Experiments demonstrate that Arc2Avatar achieves state-of-the-art realism and identity preservation, effectively addressing color issues by allowing the use of very low guidance, enabled by our strong identity prior and initialization strategy, without compromising detail.
• 摘要：受到 3D Gaussian Splatting (3DGS) 在多视图设置中重建详细 3D 场景的有效性以及大型 2D 人体基础模型的出现的启发，我们推出了 Arc2Avatar，这是第一个基于 SDS 的方法，利用人脸基础模型作为指导，仅需一张图像作为输入。为了实现这一点，我们通过对合成数据进行微调和修改其条件，将这种模型扩展到多视图人头生成。我们的化身与人脸网格模板保持紧密对应，从而允许基于混合形状的表情生成。这是通过改进的 3DGS 方法、连接正则化器和为我们的任务量身定制的战略初始化来实现的。此外，我们提出了一个可选的基于 SDS 的高效校正步骤来改进混合形状表情，增强真实感和多样性。实验表明，Arc2Avatar 实现了最先进的真实感和身份保存，通过允许使用非常低的指导，有效地解决了颜色问题，这得益于我们强大的身份先验和初始化策略，而不会影响细节。

Title: TimeDP: Learning to Generate Multi-Domain Time Series with Domain Prompts

• Authors: Yu-Hao Huang, Chang Xu, Yueying Wu, Wu-Jun Li, Jiang Bian
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05403
• Pdf URL: https://arxiv.org/pdf/2501.05403
• Copy Paste: [[2501.05403]] TimeDP: Learning to Generate Multi-Domain Time Series with Domain Prompts(https://arxiv.org/abs/2501.05403)
• Keywords: generation
• Abstract: Time series generation models are crucial for applications like data augmentation and privacy preservation. Most existing time series generation models are typically designed to generate data from one specified domain. While leveraging data from other domain for better generalization is proved to work in other application areas, this approach remains challenging for time series modeling due to the large divergence in patterns among different real world time series categories. In this paper, we propose a multi-domain time series diffusion model with domain prompts, named TimeDP. In TimeDP, we utilize a time series semantic prototype module which defines time series prototypes to represent time series basis, each prototype vector serving as "word" representing some elementary time series feature. A prototype assignment module is applied to extract the extract domain specific prototype weights, for learning domain prompts as generation condition. During sampling, we extract "domain prompt" with few-shot samples from the target domain and use the domain prompts as condition to generate time series samples. Experiments demonstrate that our method outperforms baselines to provide the state-of-the-art in-domain generation quality and strong unseen domain generation capability.
• 摘要：时间序列生成模型对于数据增强和隐私保护等应用至关重要。大多数现有的时间序列生成模型通常被设计为从一个指定的域生成数据。虽然利用来自其他域的数据进行更好的泛化已被证明在其他应用领域是可行的，但由于不同现实世界时间序列类别之间的模式差异很大，这种方法对于时间序列建模仍然具有挑战性。在本文中，我们提出了一个具有领域提示的多域时间序列扩散模型，名为 TimeDP。在 TimeDP 中，我们使用一个时间序列语义原型模块，该模块定义时间序列原型来表示时间序列基础，每个原型向量作为表示某些基本时间序列特征的“单词”。原型分配模块用于提取特定于域的原型权重，以学习域提示作为生成条件。在采样过程中，我们从目标域中提取少量样本的“域提示”，并使用域提示作为条件来生成时间序列样本。实验表明，我们的方法优于基线，可提供最先进的域内生成质量和强大的未知域生成能力。

Title: Zero-1-to-G: Taming Pretrained 2D Diffusion Model for Direct 3D Generation

• Authors: Xuyi Meng, Chen Wang, Jiahui Lei, Kostas Daniilidis, Jiatao Gu, Lingjie Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05427
• Pdf URL: https://arxiv.org/pdf/2501.05427
• Copy Paste: [[2501.05427]] Zero-1-to-G: Taming Pretrained 2D Diffusion Model for Direct 3D Generation(https://arxiv.org/abs/2501.05427)
• Keywords: generation, generative
• Abstract: Recent advances in 2D image generation have achieved remarkable quality,largely driven by the capacity of diffusion models and the availability of large-scale datasets. However, direct 3D generation is still constrained by the scarcity and lower fidelity of 3D datasets. In this paper, we introduce Zero-1-to-G, a novel approach that addresses this problem by enabling direct single-view generation on Gaussian splats using pretrained 2D diffusion models. Our key insight is that Gaussian splats, a 3D representation, can be decomposed into multi-view images encoding different attributes. This reframes the challenging task of direct 3D generation within a 2D diffusion framework, allowing us to leverage the rich priors of pretrained 2D diffusion models. To incorporate 3D awareness, we introduce cross-view and cross-attribute attention layers, which capture complex correlations and enforce 3D consistency across generated splats. This makes Zero-1-to-G the first direct image-to-3D generative model to effectively utilize pretrained 2D diffusion priors, enabling efficient training and improved generalization to unseen objects. Extensive experiments on both synthetic and in-the-wild datasets demonstrate superior performance in 3D object generation, offering a new approach to high-quality 3D generation.
• 摘要：2D 图像生成的最新进展已取得了显著的质量，这主要得益于扩散模型的容量和大规模数据集的可用性。然而，直接 3D 生成仍然受到 3D 数据集稀缺性和低保真度的限制。在本文中，我们介绍了一种新方法 Zero-1-to-G，该方法通过使用预训练的 2D 扩散模型在高斯图上直接进行单视图生成来解决此问题。我们的主要见解是，高斯图（一种 3D 表示）可以分解为编码不同属性的多视图图像。这重新定义了在 2D 扩散框架内直接 3D 生成的艰巨任务，使我们能够利用预训练的 2D 扩散模型的丰富先验。为了融入 3D 感知，我们引入了跨视图和跨属性注意层，它们可以捕捉复杂的相关性并在生成的图之间强制 3D 一致性。这使得 Zero-1-to-G 成为第一个直接图像到 3D 生成模型，能够有效利用预训练的 2D 扩散先验，从而实现高效训练并提高对未见过物体的泛化能力。在合成数据集和自然数据集上进行的大量实验表明，该模型在 3D 物体生成方面具有卓越的性能，为高质量 3D 生成提供了一种新方法。

Title: Progressive Growing of Video Tokenizers for Highly Compressed Latent Spaces

• Authors: Aniruddha Mahapatra, Long Mai, Yitian Zhang, David Bourgin, Feng Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05442
• Pdf URL: https://arxiv.org/pdf/2501.05442
• Copy Paste: [[2501.05442]] Progressive Growing of Video Tokenizers for Highly Compressed Latent Spaces(https://arxiv.org/abs/2501.05442)
• Keywords: generation
• Abstract: Video tokenizers are essential for latent video diffusion models, converting raw video data into spatiotemporally compressed latent spaces for efficient training. However, extending state-of-the-art video tokenizers to achieve a temporal compression ratio beyond 4x without increasing channel capacity poses significant challenges. In this work, we propose an alternative approach to enhance temporal compression. We find that the reconstruction quality of temporally subsampled videos from a low-compression encoder surpasses that of high-compression encoders applied to original videos. This indicates that high-compression models can leverage representations from lower-compression models. Building on this insight, we develop a bootstrapped high-temporal-compression model that progressively trains high-compression blocks atop well-trained lower-compression models. Our method includes a cross-level feature-mixing module to retain information from the pretrained low-compression model and guide higher-compression blocks to capture the remaining details from the full video sequence. Evaluation of video benchmarks shows that our method significantly improves reconstruction quality while increasing temporal compression compared to direct extensions of existing video tokenizers. Furthermore, the resulting compact latent space effectively trains a video diffusion model for high-quality video generation with a reduced token budget.
• 摘要：视频标记器对于潜在视频扩散模型至关重要，可将原始视频数据转换为时空压缩的潜在空间，以实现高效训练。然而，在不增加通道容量的情况下扩展最先进的视频标记器以实现超过 4 倍的时间压缩比，是一项重大挑战。在这项工作中，我们提出了一种增强时间压缩的替代方法。我们发现，低压缩编码器的时间子采样视频的重建质量优于应用于原始视频的高压缩编码器。这表明高压缩模型可以利用低压缩模型的表示。基于这一见解，我们开发了一个引导式高时间压缩模型，该模型在训练有素的低压缩模型之上逐步训练高压缩块。我们的方法包括一个跨级特征混合模块，用于保留来自预训练的低压缩模型的信息并引导高压缩块从完整视频序列中捕获其余细节。视频基准评估表明，与现有视频标记器的直接扩展相比，我们的方法显著提高了重建质量，同时提高了时间压缩率。此外，由此产生的紧凑潜在空间可有效训练视频扩散模型，以减少标记预算生成高质量视频。

Title: Consistent Flow Distillation for Text-to-3D Generation

• Authors: Runjie Yan, Yinbo Chen, Xiaolong Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05445
• Pdf URL: https://arxiv.org/pdf/2501.05445
• Copy Paste: [[2501.05445]] Consistent Flow Distillation for Text-to-3D Generation(https://arxiv.org/abs/2501.05445)
• Keywords: generation, generative
• Abstract: Score Distillation Sampling (SDS) has made significant strides in distilling image-generative models for 3D generation. However, its maximum-likelihood-seeking behavior often leads to degraded visual quality and diversity, limiting its effectiveness in 3D applications. In this work, we propose Consistent Flow Distillation (CFD), which addresses these limitations. We begin by leveraging the gradient of the diffusion ODE or SDE sampling process to guide the 3D generation. From the gradient-based sampling perspective, we find that the consistency of 2D image flows across different viewpoints is important for high-quality 3D generation. To achieve this, we introduce multi-view consistent Gaussian noise on the 3D object, which can be rendered from various viewpoints to compute the flow gradient. Our experiments demonstrate that CFD, through consistent flows, significantly outperforms previous methods in text-to-3D generation.
• 摘要：分数蒸馏采样 (SDS) 在为 3D 生成提取图像生成模型方面取得了重大进展。然而，其最大似然寻求行为往往会导致视觉质量和多样性下降，限制了其在 3D 应用中的有效性。在这项工作中，我们提出了一致流蒸馏 (CFD)，它解决了这些限制。我们首先利用扩散 ODE 或 SDE 采样过程的梯度来指导 3D 生成。从基于梯度的采样角度来看，我们发现不同视点之间的 2D 图像流的一致性对于高质量 3D 生成非常重要。为了实现这一点，我们在 3D 对象上引入了多视图一致高斯噪声，可以从各个视点渲染该对象以计算流梯度。我们的实验表明，通过一致的流，CFD 在文本到 3D 生成中的表现明显优于以前的方法。