2025-01-10

Title: Video Summarisation with Incident and Context Information using Generative AI

• Authors: Ulindu De Silva, Leon Fernando, Kalinga Bandara, Rashmika Nawaratne
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.04764
• Pdf URL: https://arxiv.org/pdf/2501.04764
• Copy Paste: [[2501.04764]] Video Summarisation with Incident and Context Information using Generative AI(https://arxiv.org/abs/2501.04764)
• Keywords: generative
• Abstract: The proliferation of video content production has led to vast amounts of data, posing substantial challenges in terms of analysis efficiency and resource utilization. Addressing this issue calls for the development of robust video analysis tools. This paper proposes a novel approach leveraging Generative Artificial Intelligence (GenAI) to facilitate streamlined video analysis. Our tool aims to deliver tailored textual summaries of user-defined queries, offering a focused insight amidst extensive video datasets. Unlike conventional frameworks that offer generic summaries or limited action recognition, our method harnesses the power of GenAI to distil relevant information, enhancing analysis precision and efficiency. Employing YOLO-V8 for object detection and Gemini for comprehensive video and text analysis, our solution achieves heightened contextual accuracy. By combining YOLO with Gemini, our approach furnishes textual summaries extracted from extensive CCTV footage, enabling users to swiftly navigate and verify pertinent events without the need for exhaustive manual review. The quantitative evaluation revealed a similarity of 72.8%, while the qualitative assessment rated an accuracy of 85%, demonstrating the capability of the proposed method.

Title: TREAD: Token Routing for Efficient Architecture-agnostic Diffusion Training

• Authors: Felix Krause, Timy Phan, Vincent Tao Hu, Björn Ommer
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04765
• Pdf URL: https://arxiv.org/pdf/2501.04765
• Copy Paste: [[2501.04765]] TREAD: Token Routing for Efficient Architecture-agnostic Diffusion Training(https://arxiv.org/abs/2501.04765)
• Keywords: diffusion
• Abstract: Diffusion models have emerged as the mainstream approach for visual generation. However, these models usually suffer from sample inefficiency and high training costs. This issue is particularly pronounced in the standard diffusion transformer architecture due to its quadratic complexity relative to input length. Recent works have addressed this by reducing the number of tokens processed in the model, often through masking. In contrast, this work aims to improve the training efficiency of the diffusion backbone by using predefined routes that store this information until it is reintroduced to deeper layers of the model, rather than discarding these tokens entirely. Further, we combine multiple routes and introduce an adapted auxiliary loss that accounts for all applied routes. Our method is not limited to the common transformer-based model - it can also be applied to state-space models. Unlike most current approaches, TREAD achieves this without architectural modifications. Finally, we show that our method reduces the computational cost and simultaneously boosts model performance on the standard benchmark ImageNet-1K 256 x 256 in class-conditional synthesis. Both of these benefits multiply to a convergence speedup of 9.55x at 400K training iterations compared to DiT and 25.39x compared to the best benchmark performance of DiT at 7M training iterations.

Title: Leveraging Registers in Vision Transformers for Robust Adaptation

• Authors: Srikar Yellapragada, Kowshik Thopalli, Vivek Narayanaswamy, Wesam Sakla, Yang Liu, Yamen Mubarka, Dimitris Samaras, Jayaraman J. Thiagarajan
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04784
• Pdf URL: https://arxiv.org/pdf/2501.04784
• Copy Paste: [[2501.04784]] Leveraging Registers in Vision Transformers for Robust Adaptation(https://arxiv.org/abs/2501.04784)
• Keywords: anomaly
• Abstract: Vision Transformers (ViTs) have shown success across a variety of tasks due to their ability to capture global image representations. Recent studies have identified the existence of high-norm tokens in ViTs, which can interfere with unsupervised object discovery. To address this, the use of "registers" which are additional tokens that isolate high norm patch tokens while capturing global image-level information has been proposed. While registers have been studied extensively for object discovery, their generalization properties particularly in out-of-distribution (OOD) scenarios, remains underexplored. In this paper, we examine the utility of register token embeddings in providing additional features for improving generalization and anomaly rejection. To that end, we propose a simple method that combines the special CLS token embedding commonly employed in ViTs with the average-pooled register embeddings to create feature representations which are subsequently used for training a downstream classifier. We find that this enhances OOD generalization and anomaly rejection, while maintaining in-distribution (ID) performance. Extensive experiments across multiple ViT backbones trained with and without registers reveal consistent improvements of 2-4\% in top-1 OOD accuracy and a 2-3\% reduction in false positive rates for anomaly detection. Importantly, these gains are achieved without additional computational overhead.

Title: Back Home: A Machine Learning Approach to Seashell Classification and Ecosystem Restoration

• Authors: Alexander Valverde, Luis Solano
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.04873
• Pdf URL: https://arxiv.org/pdf/2501.04873
• Copy Paste: [[2501.04873]] Back Home: A Machine Learning Approach to Seashell Classification and Ecosystem Restoration(https://arxiv.org/abs/2501.04873)
• Keywords: anomaly
• Abstract: In Costa Rica, an average of 5 tons of seashells are extracted from ecosystems annually. Confiscated seashells, cannot be returned to their ecosystems due to the lack of origin recognition. To address this issue, we developed a convolutional neural network (CNN) specifically for seashell identification. We built a dataset from scratch, consisting of approximately 19000 images from the Pacific and Caribbean coasts. Using this dataset, the model achieved a classification accuracy exceeding 85%. The model has been integrated into a user-friendly application, which has classified over 36,000 seashells to date, delivering real-time results within 3 seconds per image. To further enhance the system's accuracy, an anomaly detection mechanism was incorporated to filter out irrelevant or anomalous inputs, ensuring only valid seashell images are processed.

Title: Online Continual Learning: A Systematic Literature Review of Approaches, Challenges, and Benchmarks

• Authors: Seyed Amir Bidaki, Amir Mohammadkhah, Kiyan Rezaee, Faeze Hassani, Sadegh Eskandari, Maziar Salahi, Mohammad M. Ghassemi
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04897
• Pdf URL: https://arxiv.org/pdf/2501.04897
• Copy Paste: [[2501.04897]] Online Continual Learning: A Systematic Literature Review of Approaches, Challenges, and Benchmarks(https://arxiv.org/abs/2501.04897)
• Keywords: self-supervised, generative
• Abstract: Online Continual Learning (OCL) is a critical area in machine learning, focusing on enabling models to adapt to evolving data streams in real-time while addressing challenges such as catastrophic forgetting and the stability-plasticity trade-off. This study conducts the first comprehensive Systematic Literature Review (SLR) on OCL, analyzing 81 approaches, extracting over 1,000 features (specific tasks addressed by these approaches), and identifying more than 500 components (sub-models within approaches, including algorithms and tools). We also review 83 datasets spanning applications like image classification, object detection, and multimodal vision-language tasks. Our findings highlight key challenges, including reducing computational overhead, developing domain-agnostic solutions, and improving scalability in resource-constrained environments. Furthermore, we identify promising directions for future research, such as leveraging self-supervised learning for multimodal and sequential data, designing adaptive memory mechanisms that integrate sparse retrieval and generative replay, and creating efficient frameworks for real-world applications with noisy or evolving task boundaries. By providing a rigorous and structured synthesis of the current state of OCL, this review offers a valuable resource for advancing this field and addressing its critical challenges and opportunities. The complete SLR methodology steps and extracted data are publicly available through the provided link: this https URL Systematic-Literature-Review-on-Online-Continual-Learning

Title: Demystifying Domain-adaptive Post-training for Financial LLMs

• Authors: Zixuan Ke, Yifei Ming, Xuan-Phi Nguyen, Caiming Xiong, Shafiq Joty
• Subjects: cs.CL, cs.AI, cs.CE, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.04961
• Pdf URL: https://arxiv.org/pdf/2501.04961
• Copy Paste: [[2501.04961]] Demystifying Domain-adaptive Post-training for Financial LLMs(https://arxiv.org/abs/2501.04961)
• Keywords: generative
• Abstract: Domain-adaptive post-training of large language models (LLMs) has emerged as a promising approach for specialized domains such as medicine and finance. However, significant challenges remain in identifying optimal adaptation criteria and training strategies across varying data and model configurations. To address these challenges, we introduce FINDAP, a systematic and fine-grained investigation into domain-adaptive post-training of LLMs for the finance domain. Our approach begins by identifying the core capabilities required for the target domain and designing a comprehensive evaluation suite aligned with these needs. We then analyze the effectiveness of key post-training stages, including continual pretraining, instruction tuning, and preference alignment. Building on these insights, we propose an effective training recipe centered on a novel preference data distillation method, which leverages process signals from a generative reward model. The resulting model, Llama-Fin, achieves state-of-the-art performance across a wide range of financial tasks. Our analysis also highlights how each post-training stage contributes to distinct capabilities, uncovering specific challenges and effective solutions, providing valuable insights for domain adaptation of LLMs. Project page: this https URL

Title: ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmark

• Authors: Ronghao Dang, Yuqian Yuan, Wenqi Zhang, Yifei Xin, Boqiang Zhang, Long Li, Liuyi Wang, Qinyang Zeng, Xin Li, Lidong Bing
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2501.05031
• Pdf URL: https://arxiv.org/pdf/2501.05031
• Copy Paste: [[2501.05031]] ECBench: Can Multi-modal Foundation Models Understand the Egocentric World? A Holistic Embodied Cognition Benchmark(https://arxiv.org/abs/2501.05031)
• Keywords: foundation model
• Abstract: The enhancement of generalization in robots by large vision-language models (LVLMs) is increasingly evident. Therefore, the embodied cognitive abilities of LVLMs based on egocentric videos are of great interest. However, current datasets for embodied video question answering lack comprehensive and systematic evaluation frameworks. Critical embodied cognitive issues, such as robotic self-cognition, dynamic scene perception, and hallucination, are rarely addressed. To tackle these challenges, we propose ECBench, a high-quality benchmark designed to systematically evaluate the embodied cognitive abilities of LVLMs. ECBench features a diverse range of scene video sources, open and varied question formats, and 30 dimensions of embodied cognition. To ensure quality, balance, and high visual dependence, ECBench uses class-independent meticulous human annotation and multi-round question screening strategies. Additionally, we introduce ECEval, a comprehensive evaluation system that ensures the fairness and rationality of the indicators. Utilizing ECBench, we conduct extensive evaluations of proprietary, open-source, and task-specific LVLMs. ECBench is pivotal in advancing the embodied cognitive capabilities of LVLMs, laying a solid foundation for developing reliable core models for embodied agents. All data and code are available at this https URL.

Title: Towards Fingerprint Mosaicking Artifact Detection: A Self-Supervised Deep Learning Approach

• Authors: Laurenz Ruzicka, Alexander Spenke, Stephan Bergmann, Gerd Nolden, Bernhard Kohn, Clemens Heitzinger
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05034
• Pdf URL: https://arxiv.org/pdf/2501.05034
• Copy Paste: [[2501.05034]] Towards Fingerprint Mosaicking Artifact Detection: A Self-Supervised Deep Learning Approach(https://arxiv.org/abs/2501.05034)
• Keywords: self-supervised
• Abstract: Fingerprint mosaicking, which is the process of combining multiple fingerprint images into a single master fingerprint, is an essential process in modern biometric systems. However, it is prone to errors that can significantly degrade fingerprint image quality. This paper proposes a novel deep learning-based approach to detect and score mosaicking artifacts in fingerprint images. Our method leverages a self-supervised learning framework to train a model on large-scale unlabeled fingerprint data, eliminating the need for manual artifact annotation. The proposed model effectively identifies mosaicking errors, achieving high accuracy on various fingerprint modalities, including contactless, rolled, and pressed fingerprints and furthermore proves to be robust to different data sources. Additionally, we introduce a novel mosaicking artifact score to quantify the severity of errors, enabling automated evaluation of fingerprint images. By addressing the challenges of mosaicking artifact detection, our work contributes to improving the accuracy and reliability of fingerprint-based biometric systems.

Title: ResPanDiff: Diffusion Model with Disentangled Modulations for Image Fusion

• Authors: Shiqi Cao, Liangjian Deng, Shangqi Deng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05091
• Pdf URL: https://arxiv.org/pdf/2501.05091
• Copy Paste: [[2501.05091]] ResPanDiff: Diffusion Model with Disentangled Modulations for Image Fusion(https://arxiv.org/abs/2501.05091)
• Keywords: diffusion
• Abstract: The implementation of diffusion-based pansharpening task is predominantly constrained by its slow inference speed, which results from numerous sampling steps. Despite the existing techniques aiming to accelerate sampling, they often compromise performance when fusing multi-source images. To ease this limitation, we introduce a novel and efficient diffusion model named Diffusion Model for Pansharpening by Inferring Residual Inference (ResPanDiff), which significantly reduces the number of diffusion steps without sacrificing the performance to tackle pansharpening task. In ResPanDiff, we innovatively propose a Markov chain that transits from noisy residuals to the residuals between the LRMS and HRMS images, thereby reducing the number of sampling steps and enhancing performance. Additionally, we design the latent space to help model extract more features at the encoding stage, Shallow Cond-Injection~(SC-I) to help model fetch cond-injected hidden features with higher dimensions, and loss functions to give a better guidance for the residual generation task. enabling the model to achieve superior performance in residual generation. Furthermore, experimental evaluations on pansharpening datasets demonstrate that the proposed method achieves superior outcomes compared to recent state-of-the-art~(SOTA) techniques, requiring only 15 sampling steps, which reduces over $90\%$ step compared with the benchmark diffusion models. Our experiments also include thorough discussions and ablation studies to underscore the effectiveness of our approach.

Title: Advancing ALS Applications with Large-Scale Pre-training: Dataset Development and Downstream Assessment

• Authors: Haoyi Xiu, Xin Liu, Taehoon Kim, Kyoung-Sook Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05095
• Pdf URL: https://arxiv.org/pdf/2501.05095
• Copy Paste: [[2501.05095]] Advancing ALS Applications with Large-Scale Pre-training: Dataset Development and Downstream Assessment(https://arxiv.org/abs/2501.05095)
• Keywords: self-supervised
• Abstract: The pre-training and fine-tuning paradigm has revolutionized satellite remote sensing applications. However, this approach remains largely underexplored for airborne laser scanning (ALS), an important technology for applications such as forest management and urban planning. In this study, we address this gap by constructing a large-scale ALS point cloud dataset and evaluating its impact on downstream applications. Our dataset comprises ALS point clouds collected across the contiguous United States, provided by the United States Geological Survey's 3D Elevation Program. To ensure efficient data collection while capturing diverse land cover and terrain types, we introduce a geospatial sampling method that selects point cloud tiles based on land cover maps and digital elevation models. As a baseline self-supervised learning model, we adopt BEV-MAE, a state-of-the-art masked autoencoder for 3D outdoor point clouds, and pre-train it on the constructed dataset. The pre-trained models are subsequently fine-tuned for downstream tasks, including tree species classification, terrain scene recognition, and point cloud semantic segmentation. Our results show that the pre-trained models significantly outperform their scratch counterparts across all downstream tasks, demonstrating the transferability of the representations learned from the proposed dataset. Furthermore, we observe that scaling the dataset using our geospatial sampling method consistently enhances performance, whereas pre-training on datasets constructed with random sampling fails to achieve similar improvements. These findings highlight the utility of the constructed dataset and the effectiveness of our sampling strategy in the pre-training and fine-tuning paradigm. The source code and pre-trained models will be made publicly available at \url{this https URL}.

Title: Optimizing Multitask Industrial Processes with Predictive Action Guidance

• Authors: Naval Kishore Mehta, Arvind, Shyam Sunder Prasad, Sumeet Saurav, Sanjay Singh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05108
• Pdf URL: https://arxiv.org/pdf/2501.05108
• Copy Paste: [[2501.05108]] Optimizing Multitask Industrial Processes with Predictive Action Guidance(https://arxiv.org/abs/2501.05108)
• Keywords: anomaly
• Abstract: Monitoring complex assembly processes is critical for maintaining productivity and ensuring compliance with assembly standards. However, variability in human actions and subjective task preferences complicate accurate task anticipation and guidance. To address these challenges, we introduce the Multi-Modal Transformer Fusion and Recurrent Units (MMTFRU) Network for egocentric activity anticipation, utilizing multimodal fusion to improve prediction accuracy. Integrated with the Operator Action Monitoring Unit (OAMU), the system provides proactive operator guidance, preventing deviations in the assembly process. OAMU employs two strategies: (1) Top-5 MMTF-RU predictions, combined with a reference graph and an action dictionary, for next-step recommendations; and (2) Top-1 MMTF-RU predictions, integrated with a reference graph, for detecting sequence deviations and predicting anomaly scores via an entropy-informed confidence mechanism. We also introduce Time-Weighted Sequence Accuracy (TWSA) to evaluate operator efficiency and ensure timely task completion. Our approach is validated on the industrial Meccano dataset and the largescale EPIC-Kitchens-55 dataset, demonstrating its effectiveness in dynamic environments.

Title: EquiBoost: An Equivariant Boosting Approach to Molecular Conformation Generation

• Authors: Yixuan Yang, Xingyu Fang, Zhaowen Cheng, Pengju Yan, Xiaolin Li
• Subjects: cs.LG, physics.chem-ph, q-bio.BM
• Abstract URL: https://arxiv.org/abs/2501.05109
• Pdf URL: https://arxiv.org/pdf/2501.05109
• Copy Paste: [[2501.05109]] EquiBoost: An Equivariant Boosting Approach to Molecular Conformation Generation(https://arxiv.org/abs/2501.05109)
• Keywords: diffusion
• Abstract: Molecular conformation generation plays key roles in computational drug design. Recently developed deep learning methods, particularly diffusion models have reached competitive performance over traditional cheminformatical approaches. However, these methods are often time-consuming or require extra support from traditional methods. We propose EquiBoost, a boosting model that stacks several equivariant graph transformers as weak learners, to iteratively refine 3D conformations of molecules. Without relying on diffusion techniques, EquiBoost balances accuracy and efficiency more effectively than diffusion-based methods. Notably, compared to the previous state-of-the-art diffusion method, EquiBoost improves generation quality and preserves diversity, achieving considerably better precision of Average Minimum RMSD (AMR) on the GEOM datasets. This work rejuvenates boosting and sheds light on its potential to be a robust alternative to diffusion models in certain scenarios.

Title: Learning In-Distribution Representations for Anomaly Detection

• Authors: William T. Lunardi, Abdulrahman Banabila, Dania Herzalla, Martin L. Andreoni
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05130
• Pdf URL: https://arxiv.org/pdf/2501.05130
• Copy Paste: [[2501.05130]] Learning In-Distribution Representations for Anomaly Detection(https://arxiv.org/abs/2501.05130)
• Keywords: self-supervised, anomaly
• Abstract: Anomaly detection involves identifying data patterns that deviate from the anticipated norm. Traditional methods struggle in high-dimensional spaces due to the curse of dimensionality. In recent years, self-supervised learning, particularly through contrastive objectives, has driven advances in anomaly detection. However, vanilla contrastive learning struggles to align with the unique demands of anomaly detection, as it lacks a pretext task tailored to the homogeneous nature of In-Distribution (ID) data and the diversity of Out-of-Distribution (OOD) anomalies. Methods that attempt to address these challenges, such as introducing hard negatives through synthetic outliers, Outlier Exposure (OE), and supervised objectives, often rely on pretext tasks that fail to balance compact clustering of ID samples with sufficient separation from OOD data. In this work, we propose Focused In-distribution Representation Modeling (FIRM), a contrastive learning objective specifically designed for anomaly detection. Unlike existing approaches, FIRM incorporates synthetic outliers into its pretext task in a way that actively shapes the representation space, promoting compact clustering of ID samples while enforcing strong separation from outliers. This formulation addresses the challenges of class collision, enhancing both the compactness of ID representations and the discriminative power of the learned feature space. We show that FIRM surpasses other contrastive methods in standard benchmarks, significantly enhancing anomaly detection compared to both traditional and supervised contrastive learning objectives. Our ablation studies confirm that FIRM consistently improves the quality of representations and shows robustness across a range of scoring methods. The code is available at: this https URL.

Title: FaceMe: Robust Blind Face Restoration with Personal Identification

• Authors: Siyu Liu, Zheng-Peng Duan, Jia OuYang, Jiayi Fu, Hyunhee Park, Zikun Liu, Chun-Le Guo, Chongyi Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05177
• Pdf URL: https://arxiv.org/pdf/2501.05177
• Copy Paste: [[2501.05177]] FaceMe: Robust Blind Face Restoration with Personal Identification(https://arxiv.org/abs/2501.05177)
• Keywords: diffusion
• Abstract: Blind face restoration is a highly ill-posed problem due to the lack of necessary context. Although existing methods produce high-quality outputs, they often fail to faithfully preserve the individual's identity. In this paper, we propose a personalized face restoration method, FaceMe, based on a diffusion model. Given a single or a few reference images, we use an identity encoder to extract identity-related features, which serve as prompts to guide the diffusion model in restoring high-quality and identity-consistent facial images. By simply combining identity-related features, we effectively minimize the impact of identity-irrelevant features during training and support any number of reference image inputs during inference. Additionally, thanks to the robustness of the identity encoder, synthesized images can be used as reference images during training, and identity changing during inference does not require fine-tuning the model. We also propose a pipeline for constructing a reference image training pool that simulates the poses and expressions that may appear in real-world scenarios. Experimental results demonstrate that our FaceMe can restore high-quality facial images while maintaining identity consistency, achieving excellent performance and robustness.

Title: EVA-S2PLoR: A Secure Element-wise Multiplication Meets Logistic Regression on Heterogeneous Database

• Authors: Tianle Tao, Shizhao Peng, Tianyu Mei, Shoumo Li, Haogang Zhu
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05223
• Pdf URL: https://arxiv.org/pdf/2501.05223
• Copy Paste: [[2501.05223]] EVA-S2PLoR: A Secure Element-wise Multiplication Meets Logistic Regression on Heterogeneous Database(https://arxiv.org/abs/2501.05223)
• Keywords: anomaly
• Abstract: Accurate nonlinear computation is a key challenge in privacy-preserving machine learning (PPML). Most existing frameworks approximate it through linear operations, resulting in significant precision loss. This paper proposes an efficient, verifiable and accurate security 2-party logistic regression framework (EVA-S2PLoR), which achieves accurate nonlinear function computation through a novel secure element-wise multiplication protocol and its derived protocols. Our framework primarily includes secure 2-party vector element-wise multiplication, addition to multiplication, reciprocal, and sigmoid function based on data disguising technology, where high efficiency and accuracy are guaranteed by the simple computation flow based on the real number domain and the few number of fixed communication rounds. We provide secure and robust anomaly detection through dimension transformation and Monte Carlo methods. EVA-S2PLoR outperforms many advanced frameworks in terms of precision (improving the performance of the sigmoid function by about 10 orders of magnitude compared to most frameworks) and delivers the best overall performance in secure logistic regression experiments.

Title: Light Transport-aware Diffusion Posterior Sampling for Single-View Reconstruction of 3D Volumes

• Authors: Ludwic Leonard, Nils Thuerey, Ruediger Westermann
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05226
• Pdf URL: https://arxiv.org/pdf/2501.05226
• Copy Paste: [[2501.05226]] Light Transport-aware Diffusion Posterior Sampling for Single-View Reconstruction of 3D Volumes(https://arxiv.org/abs/2501.05226)
• Keywords: diffusion, generative
• Abstract: We introduce a single-view reconstruction technique of volumetric fields in which multiple light scattering effects are omnipresent, such as in clouds. We model the unknown distribution of volumetric fields using an unconditional diffusion model trained on a novel benchmark dataset comprising 1,000 synthetically simulated volumetric density fields. The neural diffusion model is trained on the latent codes of a novel, diffusion-friendly, monoplanar representation. The generative model is used to incorporate a tailored parametric diffusion posterior sampling technique into different reconstruction tasks. A physically-based differentiable volume renderer is employed to provide gradients with respect to light transport in the latent space. This stands in contrast to classic NeRF approaches and makes the reconstructions better aligned with observed data. Through various experiments, we demonstrate single-view reconstruction of volumetric clouds at a previously unattainable quality.

Title: Patch-GAN Transfer Learning with Reconstructive Models for Cloud Removal

• Authors: Wanli Ma, Oktay Karakus, Paul L. Rosin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05265
• Pdf URL: https://arxiv.org/pdf/2501.05265
• Copy Paste: [[2501.05265]] Patch-GAN Transfer Learning with Reconstructive Models for Cloud Removal(https://arxiv.org/abs/2501.05265)
• Keywords: generative
• Abstract: Cloud removal plays a crucial role in enhancing remote sensing image analysis, yet accurately reconstructing cloud-obscured regions remains a significant challenge. Recent advancements in generative models have made the generation of realistic images increasingly accessible, offering new opportunities for this task. Given the conceptual alignment between image generation and cloud removal tasks, generative models present a promising approach for addressing cloud removal in remote sensing. In this work, we propose a deep transfer learning approach built on a generative adversarial network (GAN) framework to explore the potential of the novel masked autoencoder (MAE) image reconstruction model in cloud removal. Due to the complexity of remote sensing imagery, we further propose using a patch-wise discriminator to determine whether each patch of the image is real or not. The proposed reconstructive transfer learning approach demonstrates significant improvements in cloud removal performance compared to other GAN-based methods. Additionally, whilst direct comparisons with some of the state-of-the-art cloud removal techniques are limited due to unclear details regarding their train/test data splits, the proposed model achieves competitive results based on available benchmarks.

Title: CellViT++: Energy-Efficient and Adaptive Cell Segmentation and Classification Using Foundation Models

• Authors: Fabian Hörst, Moritz Rempe, Helmut Becker, Lukas Heine, Julius Keyl, Jens Kleesiek
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05269
• Pdf URL: https://arxiv.org/pdf/2501.05269
• Copy Paste: [[2501.05269]] CellViT++: Energy-Efficient and Adaptive Cell Segmentation and Classification Using Foundation Models(https://arxiv.org/abs/2501.05269)
• Keywords: foundation model
• Abstract: Digital Pathology is a cornerstone in the diagnosis and treatment of diseases. A key task in this field is the identification and segmentation of cells in hematoxylin and eosin-stained images. Existing methods for cell segmentation often require extensive annotated datasets for training and are limited to a predefined cell classification scheme. To overcome these limitations, we propose $\text{CellViT}^{\scriptscriptstyle ++}$, a framework for generalized cell segmentation in digital pathology. $\text{CellViT}^{\scriptscriptstyle ++}$ utilizes Vision Transformers with foundation models as encoders to compute deep cell features and segmentation masks simultaneously. To adapt to unseen cell types, we rely on a computationally efficient approach. It requires minimal data for training and leads to a drastically reduced carbon footprint. We demonstrate excellent performance on seven different datasets, covering a broad spectrum of cell types, organs, and clinical settings. The framework achieves remarkable zero-shot segmentation and data-efficient cell-type classification. Furthermore, we show that $\text{CellViT}^{\scriptscriptstyle ++}$ can leverage immunofluorescence stainings to generate training datasets without the need for pathologist annotations. The automated dataset generation approach surpasses the performance of networks trained on manually labeled data, demonstrating its effectiveness in creating high-quality training datasets without expert annotations. To advance digital pathology, $\text{CellViT}^{\scriptscriptstyle ++}$ is available as an open-source framework featuring a user-friendly, web-based interface for visualization and annotation. The code is available under this https URL.

Title: Comparison Study: Glacier Calving Front Delineation in Synthetic Aperture Radar Images With Deep Learning

• Authors: Nora Gourmelon, Konrad Heidler, Erik Loebel, Daniel Cheng, Julian Klink, Anda Dong, Fei Wu, Noah Maul, Moritz Koch, Marcel Dreier, Dakota Pyles, Thorsten Seehaus, Matthias Braun, Andreas Maier, Vincent Christlein
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05281
• Pdf URL: https://arxiv.org/pdf/2501.05281
• Copy Paste: [[2501.05281]] Comparison Study: Glacier Calving Front Delineation in Synthetic Aperture Radar Images With Deep Learning(https://arxiv.org/abs/2501.05281)
• Keywords: foundation model
• Abstract: Calving front position variation of marine-terminating glaciers is an indicator of ice mass loss and a crucial parameter in numerical glacier models. Deep Learning (DL) systems can automatically extract this position from Synthetic Aperture Radar (SAR) imagery, enabling continuous, weather- and illumination-independent, large-scale monitoring. This study presents the first comparison of DL systems on a common calving front benchmark dataset. A multi-annotator study with ten annotators is performed to contrast the best-performing DL system against human performance. The best DL model's outputs deviate 221 m on average, while the average deviation of the human annotators is 38 m. This significant difference shows that current DL systems do not yet match human performance and that further research is needed to enable fully automated monitoring of glacier calving fronts. The study of Vision Transformers, foundation models, and the inclusion and processing strategy of more information are identified as avenues for future research.

Title: CROPS: Model-Agnostic Training-Free Framework for Safe Image Synthesis with Latent Diffusion Models

• Authors: Junha Park, Ian Ryu, Jaehui Hwang, Hyungkeun Park, Jiyoon Kim, Jong-Seok Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05359
• Pdf URL: https://arxiv.org/pdf/2501.05359
• Copy Paste: [[2501.05359]] CROPS: Model-Agnostic Training-Free Framework for Safe Image Synthesis with Latent Diffusion Models(https://arxiv.org/abs/2501.05359)
• Keywords: diffusion
• Abstract: With advances in diffusion models, image generation has shown significant performance improvements. This raises concerns about the potential abuse of image generation, such as the creation of explicit or violent images, commonly referred to as Not Safe For Work (NSFW) content. To address this, the Stable Diffusion model includes several safety checkers to censor initial text prompts and final output images generated from the model. However, recent research has shown that these safety checkers have vulnerabilities against adversarial attacks, allowing them to generate NSFW images. In this paper, we find that these adversarial attacks are not robust to small changes in text prompts or input latents. Based on this, we propose CROPS (Circular or RandOm Prompts for Safety), a model-agnostic framework that easily defends against adversarial attacks generating NSFW images without requiring additional training. Moreover, we develop an approach that utilizes one-step diffusion models for efficient NSFW detection (CROPS-1), further reducing computational resources. We demonstrate the superiority of our method in terms of performance and applicability.

Title: 1-2-1: Renaissance of Single-Network Paradigm for Virtual Try-On

• Authors: Shuliang Ning, Yipeng Qin, Xiaoguang Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05369
• Pdf URL: https://arxiv.org/pdf/2501.05369
• Copy Paste: [[2501.05369]] 1-2-1: Renaissance of Single-Network Paradigm for Virtual Try-On(https://arxiv.org/abs/2501.05369)
• Keywords: generative
• Abstract: Virtual Try-On (VTON) has become a crucial tool in ecommerce, enabling the realistic simulation of garments on individuals while preserving their original appearance and pose. Early VTON methods relied on single generative networks, but challenges remain in preserving fine-grained garment details due to limitations in feature extraction and fusion. To address these issues, recent approaches have adopted a dual-network paradigm, incorporating a complementary "ReferenceNet" to enhance garment feature extraction and fusion. While effective, this dual-network approach introduces significant computational overhead, limiting its scalability for high-resolution and long-duration image/video VTON applications. In this paper, we challenge the dual-network paradigm by proposing a novel single-network VTON method that overcomes the limitations of existing techniques. Our method, namely MNVTON, introduces a Modality-specific Normalization strategy that separately processes text, image and video inputs, enabling them to share the same attention layers in a VTON network. Extensive experimental results demonstrate the effectiveness of our approach, showing that it consistently achieves higher-quality, more detailed results for both image and video VTON tasks. Our results suggest that the single-network paradigm can rival the performance of dualnetwork approaches, offering a more efficient alternative for high-quality, scalable VTON applications.

Title: Accelerated Diffusion Models via Speculative Sampling

• Authors: Valentin De Bortoli, Alexandre Galashov, Arthur Gretton, Arnaud Doucet
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2501.05370
• Pdf URL: https://arxiv.org/pdf/2501.05370
• Copy Paste: [[2501.05370]] Accelerated Diffusion Models via Speculative Sampling(https://arxiv.org/abs/2501.05370)
• Keywords: diffusion
• Abstract: Speculative sampling is a popular technique for accelerating inference in Large Language Models by generating candidate tokens using a fast draft model and accepting or rejecting them based on the target model's distribution. While speculative sampling was previously limited to discrete sequences, we extend it to diffusion models, which generate samples via continuous, vector-valued Markov chains. In this context, the target model is a high-quality but computationally expensive diffusion model. We propose various drafting strategies, including a simple and effective approach that does not require training a draft model and is applicable out of the box to any diffusion model. Our experiments demonstrate significant generation speedup on various diffusion models, halving the number of function evaluations, while generating exact samples from the target model.

Title: Arc2Avatar: Generating Expressive 3D Avatars from a Single Image via ID Guidance

• Authors: Dimitrios Gerogiannis, Foivos Paraperas Papantoniou, Rolandos Alexandros Potamias, Alexandros Lattas, Stefanos Zafeiriou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05379
• Pdf URL: https://arxiv.org/pdf/2501.05379
• Copy Paste: [[2501.05379]] Arc2Avatar: Generating Expressive 3D Avatars from a Single Image via ID Guidance(https://arxiv.org/abs/2501.05379)
• Keywords: foundation model
• Abstract: Inspired by the effectiveness of 3D Gaussian Splatting (3DGS) in reconstructing detailed 3D scenes within multi-view setups and the emergence of large 2D human foundation models, we introduce Arc2Avatar, the first SDS-based method utilizing a human face foundation model as guidance with just a single image as input. To achieve that, we extend such a model for diverse-view human head generation by fine-tuning on synthetic data and modifying its conditioning. Our avatars maintain a dense correspondence with a human face mesh template, allowing blendshape-based expression generation. This is achieved through a modified 3DGS approach, connectivity regularizers, and a strategic initialization tailored for our task. Additionally, we propose an optional efficient SDS-based correction step to refine the blendshape expressions, enhancing realism and diversity. Experiments demonstrate that Arc2Avatar achieves state-of-the-art realism and identity preservation, effectively addressing color issues by allowing the use of very low guidance, enabled by our strong identity prior and initialization strategy, without compromising detail.

Title: Mechanistic understanding and validation of large AI models with SemanticLens

• Authors: Maximilian Dreyer, Jim Berend, Tobias Labarta, Johanna Vielhaben, Thomas Wiegand, Sebastian Lapuschkin, Wojciech Samek
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05398
• Pdf URL: https://arxiv.org/pdf/2501.05398
• Copy Paste: [[2501.05398]] Mechanistic understanding and validation of large AI models with SemanticLens(https://arxiv.org/abs/2501.05398)
• Keywords: foundation model
• Abstract: Unlike human-engineered systems such as aeroplanes, where each component's role and dependencies are well understood, the inner workings of AI models remain largely opaque, hindering verifiability and undermining trust. This paper introduces SemanticLens, a universal explanation method for neural networks that maps hidden knowledge encoded by components (e.g., individual neurons) into the semantically structured, multimodal space of a foundation model such as CLIP. In this space, unique operations become possible, including (i) textual search to identify neurons encoding specific concepts, (ii) systematic analysis and comparison of model representations, (iii) automated labelling of neurons and explanation of their functional roles, and (iv) audits to validate decision-making against requirements. Fully scalable and operating without human input, SemanticLens is shown to be effective for debugging and validation, summarizing model knowledge, aligning reasoning with expectations (e.g., adherence to the ABCDE-rule in melanoma classification), and detecting components tied to spurious correlations and their associated training data. By enabling component-level understanding and validation, the proposed approach helps bridge the "trust gap" between AI models and traditional engineered systems. We provide code for SemanticLens on this https URL and a demo on this https URL.

Title: TimeDP: Learning to Generate Multi-Domain Time Series with Domain Prompts

• Authors: Yu-Hao Huang, Chang Xu, Yueying Wu, Wu-Jun Li, Jiang Bian
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05403
• Pdf URL: https://arxiv.org/pdf/2501.05403
• Copy Paste: [[2501.05403]] TimeDP: Learning to Generate Multi-Domain Time Series with Domain Prompts(https://arxiv.org/abs/2501.05403)
• Keywords: diffusion
• Abstract: Time series generation models are crucial for applications like data augmentation and privacy preservation. Most existing time series generation models are typically designed to generate data from one specified domain. While leveraging data from other domain for better generalization is proved to work in other application areas, this approach remains challenging for time series modeling due to the large divergence in patterns among different real world time series categories. In this paper, we propose a multi-domain time series diffusion model with domain prompts, named TimeDP. In TimeDP, we utilize a time series semantic prototype module which defines time series prototypes to represent time series basis, each prototype vector serving as "word" representing some elementary time series feature. A prototype assignment module is applied to extract the extract domain specific prototype weights, for learning domain prompts as generation condition. During sampling, we extract "domain prompt" with few-shot samples from the target domain and use the domain prompts as condition to generate time series samples. Experiments demonstrate that our method outperforms baselines to provide the state-of-the-art in-domain generation quality and strong unseen domain generation capability.

Title: A Novel Pathology Foundation Model by Mayo Clinic, Charit\'e, and Aignostics

• Authors: Maximilian Alber, Stephan Tietz, Jonas Dippel, Timo Milbich, Timothée Lesort, Panos Korfiatis, Moritz Krügener, Beatriz Perez Cancer, Neelay Shah, Alexander Möllers, Philipp Seegerer, Alexandra Carpen-Amarie, Kai Standvoss, Gabriel Dernbach, Edwin de Jong, Simon Schallenberg, Andreas Kunft, Helmut Hoffer von Ankershoffen, Gavin Schaeferle, Patrick Duffy, Matt Redlon, Philipp Jurmeister, David Horst, Lukas Ruff, Klaus-Robert Müller, Frederick Klauschen, Andrew Norgan
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05409
• Pdf URL: https://arxiv.org/pdf/2501.05409
• Copy Paste: [[2501.05409]] A Novel Pathology Foundation Model by Mayo Clinic, Charit\'e, and Aignostics(https://arxiv.org/abs/2501.05409)
• Keywords: foundation model
• Abstract: Recent advances in digital pathology have demonstrated the effectiveness of foundation models across diverse applications. In this report, we present a novel vision foundation model based on the RudolfV approach. Our model was trained on a dataset comprising 1.2 million histopathology whole slide images, collected from two medical institutions: Mayo Clinic and Charité - Universtätsmedizin Berlin. Comprehensive evaluations show that our model achieves state-of-the-art performance across twenty-one public benchmark datasets, even though it is neither the largest model by parameter count nor by training dataset size.

Title: Zero-1-to-G: Taming Pretrained 2D Diffusion Model for Direct 3D Generation

• Authors: Xuyi Meng, Chen Wang, Jiahui Lei, Kostas Daniilidis, Jiatao Gu, Lingjie Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05427
• Pdf URL: https://arxiv.org/pdf/2501.05427
• Copy Paste: [[2501.05427]] Zero-1-to-G: Taming Pretrained 2D Diffusion Model for Direct 3D Generation(https://arxiv.org/abs/2501.05427)
• Keywords: diffusion, generative
• Abstract: Recent advances in 2D image generation have achieved remarkable quality,largely driven by the capacity of diffusion models and the availability of large-scale datasets. However, direct 3D generation is still constrained by the scarcity and lower fidelity of 3D datasets. In this paper, we introduce Zero-1-to-G, a novel approach that addresses this problem by enabling direct single-view generation on Gaussian splats using pretrained 2D diffusion models. Our key insight is that Gaussian splats, a 3D representation, can be decomposed into multi-view images encoding different attributes. This reframes the challenging task of direct 3D generation within a 2D diffusion framework, allowing us to leverage the rich priors of pretrained 2D diffusion models. To incorporate 3D awareness, we introduce cross-view and cross-attribute attention layers, which capture complex correlations and enforce 3D consistency across generated splats. This makes Zero-1-to-G the first direct image-to-3D generative model to effectively utilize pretrained 2D diffusion priors, enabling efficient training and improved generalization to unseen objects. Extensive experiments on both synthetic and in-the-wild datasets demonstrate superior performance in 3D object generation, offering a new approach to high-quality 3D generation.

Title: The GAN is dead; long live the GAN! A Modern GAN Baseline

• Authors: Yiwen Huang, Aaron Gokaslan, Volodymyr Kuleshov, James Tompkin
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05441
• Pdf URL: https://arxiv.org/pdf/2501.05441
• Copy Paste: [[2501.05441]] The GAN is dead; long live the GAN! A Modern GAN Baseline(https://arxiv.org/abs/2501.05441)
• Keywords: diffusion
• Abstract: There is a widely-spread claim that GANs are difficult to train, and GAN architectures in the literature are littered with empirical tricks. We provide evidence against this claim and build a modern GAN baseline in a more principled manner. First, we derive a well-behaved regularized relativistic GAN loss that addresses issues of mode dropping and non-convergence that were previously tackled via a bag of ad-hoc tricks. We analyze our loss mathematically and prove that it admits local convergence guarantees, unlike most existing relativistic losses. Second, our new loss allows us to discard all ad-hoc tricks and replace outdated backbones used in common GANs with modern architectures. Using StyleGAN2 as an example, we present a roadmap of simplification and modernization that results in a new minimalist baseline -- R3GAN. Despite being simple, our approach surpasses StyleGAN2 on FFHQ, ImageNet, CIFAR, and Stacked MNIST datasets, and compares favorably against state-of-the-art GANs and diffusion models.

Title: Progressive Growing of Video Tokenizers for Highly Compressed Latent Spaces

• Authors: Aniruddha Mahapatra, Long Mai, Yitian Zhang, David Bourgin, Feng Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2501.05442
• Pdf URL: https://arxiv.org/pdf/2501.05442
• Copy Paste: [[2501.05442]] Progressive Growing of Video Tokenizers for Highly Compressed Latent Spaces(https://arxiv.org/abs/2501.05442)
• Keywords: diffusion
• Abstract: Video tokenizers are essential for latent video diffusion models, converting raw video data into spatiotemporally compressed latent spaces for efficient training. However, extending state-of-the-art video tokenizers to achieve a temporal compression ratio beyond 4x without increasing channel capacity poses significant challenges. In this work, we propose an alternative approach to enhance temporal compression. We find that the reconstruction quality of temporally subsampled videos from a low-compression encoder surpasses that of high-compression encoders applied to original videos. This indicates that high-compression models can leverage representations from lower-compression models. Building on this insight, we develop a bootstrapped high-temporal-compression model that progressively trains high-compression blocks atop well-trained lower-compression models. Our method includes a cross-level feature-mixing module to retain information from the pretrained low-compression model and guide higher-compression blocks to capture the remaining details from the full video sequence. Evaluation of video benchmarks shows that our method significantly improves reconstruction quality while increasing temporal compression compared to direct extensions of existing video tokenizers. Furthermore, the resulting compact latent space effectively trains a video diffusion model for high-quality video generation with a reduced token budget.

Title: Consistent Flow Distillation for Text-to-3D Generation

• Authors: Runjie Yan, Yinbo Chen, Xiaolong Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05445
• Pdf URL: https://arxiv.org/pdf/2501.05445
• Copy Paste: [[2501.05445]] Consistent Flow Distillation for Text-to-3D Generation(https://arxiv.org/abs/2501.05445)
• Keywords: diffusion, generative
• Abstract: Score Distillation Sampling (SDS) has made significant strides in distilling image-generative models for 3D generation. However, its maximum-likelihood-seeking behavior often leads to degraded visual quality and diversity, limiting its effectiveness in 3D applications. In this work, we propose Consistent Flow Distillation (CFD), which addresses these limitations. We begin by leveraging the gradient of the diffusion ODE or SDE sampling process to guide the 3D generation. From the gradient-based sampling perspective, we find that the consistency of 2D image flows across different viewpoints is important for high-quality 3D generation. To achieve this, we introduce multi-view consistent Gaussian noise on the 3D object, which can be rendered from various viewpoints to compute the flow gradient. Our experiments demonstrate that CFD, through consistent flows, significantly outperforms previous methods in text-to-3D generation.

Title: Relative Pose Estimation through Affine Corrections of Monocular Depth Priors

• Authors: Yifan Yu, Shaohui Liu, Rémi Pautrat, Marc Pollefeys, Viktor Larsson
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2501.05446
• Pdf URL: https://arxiv.org/pdf/2501.05446
• Copy Paste: [[2501.05446]] Relative Pose Estimation through Affine Corrections of Monocular Depth Priors(https://arxiv.org/abs/2501.05446)
• Keywords: foundation model
• Abstract: Monocular depth estimation (MDE) models have undergone significant advancements over recent years. Many MDE models aim to predict affine-invariant relative depth from monocular images, while recent developments in large-scale training and vision foundation models enable reasonable estimation of metric (absolute) depth. However, effectively leveraging these predictions for geometric vision tasks, in particular relative pose estimation, remains relatively under explored. While depths provide rich constraints for cross-view image alignment, the intrinsic noise and ambiguity from the monocular depth priors present practical challenges to improving upon classic keypoint-based solutions. In this paper, we develop three solvers for relative pose estimation that explicitly account for independent affine (scale and shift) ambiguities, covering both calibrated and uncalibrated conditions. We further propose a hybrid estimation pipeline that combines our proposed solvers with classic point-based solvers and epipolar constraints. We find that the affine correction modeling is beneficial to not only the relative depth priors but also, surprisingly, the ``metric" ones. Results across multiple datasets demonstrate large improvements of our approach over classic keypoint-based baselines and PnP-based solutions, under both calibrated and uncalibrated setups. We also show that our method improves consistently with different feature matchers and MDE models, and can further benefit from very recent advances on both modules. Code is available at this https URL.

Title: Decentralized Diffusion Models

• Authors: David McAllister, Matthew Tancik, Jiaming Song, Angjoo Kanazawa
• Subjects: cs.CV, cs.DC, cs.LG
• Abstract URL: https://arxiv.org/abs/2501.05450
• Pdf URL: https://arxiv.org/pdf/2501.05450
• Copy Paste: [[2501.05450]] Decentralized Diffusion Models(https://arxiv.org/abs/2501.05450)
• Keywords: diffusion
• Abstract: Large-scale AI model training divides work across thousands of GPUs, then synchronizes gradients across them at each step. This incurs a significant network burden that only centralized, monolithic clusters can support, driving up infrastructure costs and straining power systems. We propose Decentralized Diffusion Models, a scalable framework for distributing diffusion model training across independent clusters or datacenters by eliminating the dependence on a centralized, high-bandwidth networking fabric. Our method trains a set of expert diffusion models over partitions of the dataset, each in full isolation from one another. At inference time, the experts ensemble through a lightweight router. We show that the ensemble collectively optimizes the same objective as a single model trained over the whole dataset. This means we can divide the training burden among a number of "compute islands," lowering infrastructure costs and improving resilience to localized GPU failures. Decentralized diffusion models empower researchers to take advantage of smaller, more cost-effective and more readily available compute like on-demand GPU nodes rather than central integrated systems. We conduct extensive experiments on ImageNet and LAION Aesthetics, showing that decentralized diffusion models FLOP-for-FLOP outperform standard diffusion models. We finally scale our approach to 24 billion parameters, demonstrating that high-quality diffusion models can now be trained with just eight individual GPU nodes in less than a week.