2024-11-22

Title: Unveiling Redundancy in Diffusion Transformers (DiTs): A Systematic Study

• Authors: Xibo Sun, Jiarui Fang, Aoyu Li, Jinzhe Pan
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13588
• Pdf URL: https://arxiv.org/pdf/2411.13588
• Copy Paste: [[2411.13588]] Unveiling Redundancy in Diffusion Transformers (DiTs): A Systematic Study(https://arxiv.org/abs/2411.13588)
• Keywords: diffusion
• Abstract: The increased model capacity of Diffusion Transformers (DiTs) and the demand for generating higher resolutions of images and videos have led to a significant rise in inference latency, impacting real-time performance adversely. While prior research has highlighted the presence of high similarity in activation values between adjacent diffusion steps (referred to as redundancy) and proposed various caching mechanisms to mitigate computational overhead, the exploration of redundancy in existing literature remains limited, with findings often not generalizable across different DiT models. This study aims to address this gap by conducting a comprehensive investigation into redundancy across a broad spectrum of mainstream DiT models. Our experimental analysis reveals substantial variations in the distribution of redundancy across diffusion steps among different DiT models. Interestingly, within a single model, the redundancy distribution remains stable regardless of variations in input prompts, step counts, or scheduling strategies. Given the lack of a consistent pattern across diverse models, caching strategies designed for a specific group of models may not easily transfer to others. To overcome this challenge, we introduce a tool for analyzing the redundancy of individual models, enabling subsequent research to develop tailored caching strategies for specific model architectures. The project is publicly available at this https URL.

Title: Video2BEV: Transforming Drone Videos to BEVs for Video-based Geo-localization

• Authors: Hao Ju, Zhedong Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13610
• Pdf URL: https://arxiv.org/pdf/2411.13610
• Copy Paste: [[2411.13610]] Video2BEV: Transforming Drone Videos to BEVs for Video-based Geo-localization(https://arxiv.org/abs/2411.13610)
• Keywords: diffusion
• Abstract: Existing approaches to drone visual geo-localization predominantly adopt the image-based setting, where a single drone-view snapshot is matched with images from other platforms. Such task formulation, however, underutilizes the inherent video output of the drone and is sensitive to occlusions and environmental constraints. To address these limitations, we formulate a new video-based drone geo-localization task and propose the Video2BEV paradigm. This paradigm transforms the video into a Bird's Eye View (BEV), simplifying the subsequent matching process. In particular, we employ Gaussian Splatting to reconstruct a 3D scene and obtain the BEV projection. Different from the existing transform methods, \eg, polar transform, our BEVs preserve more fine-grained details without significant distortion. To further improve model scalability toward diverse BEVs and satellite figures, our Video2BEV paradigm also incorporates a diffusion-based module for generating hard negative samples, which facilitates discriminative feature learning. To validate our approach, we introduce UniV, a new video-based geo-localization dataset that extends the image-based University-1652 dataset. UniV features flight paths at $30^\circ$ and $45^\circ$ elevation angles with increased frame rates of up to 10 frames per second (FPS). Extensive experiments on the UniV dataset show that our Video2BEV paradigm achieves competitive recall rates and outperforms conventional video-based methods. Compared to other methods, our proposed approach exhibits robustness at lower elevations with more occlusions.

Title: Non-Linear Outlier Synthesis for Out-of-Distribution Detection

• Authors: Lars Doorenbos, Raphael Sznitman, Pablo Márquez-Neila
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13619
• Pdf URL: https://arxiv.org/pdf/2411.13619
• Copy Paste: [[2411.13619]] Non-Linear Outlier Synthesis for Out-of-Distribution Detection(https://arxiv.org/abs/2411.13619)
• Keywords: diffusion
• Abstract: The reliability of supervised classifiers is severely hampered by their limitations in dealing with unexpected inputs, leading to great interest in out-of-distribution (OOD) detection. Recently, OOD detectors trained on synthetic outliers, especially those generated by large diffusion models, have shown promising results in defining robust OOD decision boundaries. Building on this progress, we present NCIS, which enhances the quality of synthetic outliers by operating directly in the diffusion's model embedding space rather than combining disjoint models as in previous work and by modeling class-conditional manifolds with a conditional volume-preserving network for more expressive characterization of the training distribution. We demonstrate that these improvements yield new state-of-the-art OOD detection results on standard ImageNet100 and CIFAR100 benchmarks and provide insights into the importance of data pre-processing and other key design choices. We make our code available at \url{this https URL}.

Title: Unsupervised Foundation Model-Agnostic Slide-Level Representation Learning

• Authors: Tim Lenz, Peter Neidlinger, Marta Ligero, Georg Wölflein, Marko van Treeck, Jakob Nikolas Kather
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13623
• Pdf URL: https://arxiv.org/pdf/2411.13623
• Copy Paste: [[2411.13623]] Unsupervised Foundation Model-Agnostic Slide-Level Representation Learning(https://arxiv.org/abs/2411.13623)
• Keywords: self-supervised, foundation model
• Abstract: Representation learning of pathology whole-slide images (WSIs) has primarily relied on weak supervision with Multiple Instance Learning (MIL). This approach leads to slide representations highly tailored to a specific clinical task. Self-supervised learning (SSL) has been successfully applied to train histopathology foundation models (FMs) for patch embedding generation. However, generating patient or slide level embeddings remains challenging. Existing approaches for slide representation learning extend the principles of SSL from patch level learning to entire slides by aligning different augmentations of the slide or by utilizing multimodal data. By integrating tile embeddings from multiple FMs, we propose a new single modality SSL method in feature space that generates useful slide representations. Our contrastive pretraining strategy, called COBRA, employs multiple FMs and an architecture based on Mamba-2. COBRA exceeds performance of state-of-the-art slide encoders on four different public CPTAC cohorts on average by at least +3.8% AUC, despite only being pretrained on 3048 WSIs from TCGA. Additionally, COBRA is readily compatible at inference time with previously unseen feature extractors.

Title: Extending Video Masked Autoencoders to 128 frames

• Authors: Nitesh Bharadwaj Gundavarapu, Luke Friedman, Raghav Goyal, Chaitra Hegde, Eirikur Agustsson, Sagar M. Waghmare, Mikhail Sirotenko, Ming-Hsuan Yang, Tobias Weyand, Boqing Gong, Leonid Sigal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13683
• Pdf URL: https://arxiv.org/pdf/2411.13683
• Copy Paste: [[2411.13683]] Extending Video Masked Autoencoders to 128 frames(https://arxiv.org/abs/2411.13683)
• Keywords: self-supervised, foundation model
• Abstract: Video understanding has witnessed significant progress with recent video foundation models demonstrating strong performance owing to self-supervised pre-training objectives; Masked Autoencoders (MAE) being the design of choice. Nevertheless, the majority of prior works that leverage MAE pre-training have focused on relatively short video representations (16 / 32 frames in length) largely due to hardware memory and compute limitations that scale poorly with video length due to the dense memory-intensive self-attention decoding. One natural strategy to address these challenges is to subsample tokens to reconstruct during decoding (or decoder masking). In this work, we propose an effective strategy for prioritizing tokens which allows training on longer video sequences (128 frames) and gets better performance than, more typical, random and uniform masking strategies. The core of our approach is an adaptive decoder masking strategy that prioritizes the most important tokens and uses quantized tokens as reconstruction objectives. Our adaptive strategy leverages a powerful MAGVIT-based tokenizer that jointly learns the tokens and their priority. We validate our design choices through exhaustive ablations and observe improved performance of the resulting long-video (128 frames) encoders over short-video (32 frames) counterparts. With our long-video masked autoencoder (LVMAE) strategy, we surpass state-of-the-art on Diving48 by 3.9 points and EPIC-Kitchens-100 verb classification by 2.5 points while relying on a simple core architecture and video-only pre-training (unlike some of the prior works that require millions of labeled video-text pairs or specialized encoders).

Title: Investigating Graph Neural Networks and Classical Feature-Extraction Techniques in Activity-Cliff and Molecular Property Prediction

• Authors: Markus Dablander
• Subjects: cs.LG, q-bio.BM, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.13688
• Pdf URL: https://arxiv.org/pdf/2411.13688
• Copy Paste: [[2411.13688]] Investigating Graph Neural Networks and Classical Feature-Extraction Techniques in Activity-Cliff and Molecular Property Prediction(https://arxiv.org/abs/2411.13688)
• Keywords: self-supervised
• Abstract: Molecular featurisation refers to the transformation of molecular data into numerical feature vectors. It is one of the key research areas in molecular machine learning and computational drug discovery. Recently, message-passing graph neural networks (GNNs) have emerged as a novel method to learn differentiable features directly from molecular graphs. While such techniques hold great promise, further investigations are needed to clarify if and when they indeed manage to definitively outcompete classical molecular featurisations such as extended-connectivity fingerprints (ECFPs) and physicochemical-descriptor vectors (PDVs). We systematically explore and further develop classical and graph-based molecular featurisation methods for two important tasks: molecular property prediction, in particular, quantitative structure-activity relationship (QSAR) prediction, and the largely unexplored challenge of activity-cliff (AC) prediction. We first give a technical description and critical analysis of PDVs, ECFPs and message-passing GNNs, with a focus on graph isomorphism networks (GINs). We then conduct a rigorous computational study to compare the performance of PDVs, ECFPs and GINs for QSAR and AC-prediction. Following this, we mathematically describe and computationally evaluate a novel twin neural network model for AC-prediction. We further introduce an operation called substructure pooling for the vectorisation of structural fingerprints as a natural counterpart to graph pooling in GNN architectures. We go on to propose Sort & Slice, a simple substructure-pooling technique for ECFPs that robustly outperforms hash-based folding at molecular property prediction. Finally, we outline two ideas for future research: (i) a graph-based self-supervised learning strategy to make classical molecular featurisations trainable, and (ii) trainable substructure-pooling via differentiable self-attention.

Title: Federated Continual Learning for Edge-AI: A Comprehensive Survey

• Authors: Zi Wang, Fei Wu, Feng Yu, Yurui Zhou, Jia Hu, Geyong Min
• Subjects: cs.LG, cs.AI, cs.DC, cs.NI
• Abstract URL: https://arxiv.org/abs/2411.13740
• Pdf URL: https://arxiv.org/pdf/2411.13740
• Copy Paste: [[2411.13740]] Federated Continual Learning for Edge-AI: A Comprehensive Survey(https://arxiv.org/abs/2411.13740)
• Keywords: foundation model
• Abstract: Edge-AI, the convergence of edge computing and artificial intelligence (AI), has become a promising paradigm that enables the deployment of advanced AI models at the network edge, close to users. In Edge-AI, federated continual learning (FCL) has emerged as an imperative framework, which fuses knowledge from different clients while preserving data privacy and retaining knowledge from previous tasks as it learns new ones. By so doing, FCL aims to ensure stable and reliable performance of learning models in dynamic and distributed environments. In this survey, we thoroughly review the state-of-the-art research and present the first comprehensive survey of FCL for Edge-AI. We categorize FCL methods based on three task characteristics: federated class continual learning, federated domain continual learning, and federated task continual learning. For each category, an in-depth investigation and review of the representative methods are provided, covering background, challenges, problem formalisation, solutions, and limitations. Besides, existing real-world applications empowered by FCL are reviewed, indicating the current progress and potential of FCL in diverse application domains. Furthermore, we discuss and highlight several prospective research directions of FCL such as algorithm-hardware co-design for FCL and FCL with foundation models, which could provide insights into the future development and practical deployment of FCL in the era of Edge-AI.

Title: Segment Any Class (SAC): Multi-Class Few-Shot Semantic Segmentation via Class Region Proposals

• Authors: Hussni Mohd Zakir, Eric Tatt Wei Ho
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13774
• Pdf URL: https://arxiv.org/pdf/2411.13774
• Copy Paste: [[2411.13774]] Segment Any Class (SAC): Multi-Class Few-Shot Semantic Segmentation via Class Region Proposals(https://arxiv.org/abs/2411.13774)
• Keywords: foundation model
• Abstract: The Segment-Anything Model (SAM) is a vision foundation model for segmentation with a prompt-driven framework. SAM generates class-agnostic masks based on user-specified instance-referring prompts. However, adapting SAM for automated segmentation -- where manual input is absent -- of specific object classes often requires additional model training. We present Segment Any Class (SAC), a novel, training-free approach that task-adapts SAM for Multi-class segmentation. SAC generates Class-Region Proposals (CRP) on query images which allows us to automatically generate class-aware prompts on probable locations of class instances. CRPs are derived from elementary intra-class and inter-class feature distinctions without any additional training. Our method is versatile, accommodating any N-way K-shot configurations for the multi-class few-shot semantic segmentation (FSS) task. Unlike gradient-learning adaptation of generalist models which risk the loss of generalization and potentially suffer from catastrophic forgetting, SAC solely utilizes automated prompting and achieves superior results over state-of-the-art methods on the COCO-20i benchmark, particularly excelling in high N-way class scenarios. SAC is an interesting demonstration of a prompt-only approach to adapting foundation models for novel tasks with small, limited datasets without any modifications to the foundation model itself. This method offers interesting benefits such as intrinsic immunity to concept or feature loss and rapid, online task adaptation of foundation models.

Title: GalaxyEdit: Large-Scale Image Editing Dataset with Enhanced Diffusion Adapter

• Authors: Aniruddha Bala, Rohan Jaiswal, Loay Rashid, Siddharth Roheda
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13794
• Pdf URL: https://arxiv.org/pdf/2411.13794
• Copy Paste: [[2411.13794]] GalaxyEdit: Large-Scale Image Editing Dataset with Enhanced Diffusion Adapter(https://arxiv.org/abs/2411.13794)
• Keywords: diffusion
• Abstract: Training of large-scale text-to-image and image-to-image models requires a huge amount of annotated data. While text-to-image datasets are abundant, data available for instruction-based image-to-image tasks like object addition and removal is limited. This is because of the several challenges associated with the data generation process, such as, significant human effort, limited automation, suboptimal end-to-end models, data diversity constraints and high expenses. We propose an automated data generation pipeline aimed at alleviating such limitations, and introduce GalaxyEdit - a large-scale image editing dataset for add and remove operations. We fine-tune the SD v1.5 model on our dataset and find that our model can successfully handle a broader range of objects and complex editing instructions, outperforming state-of-the-art methods in FID scores by 11.2\% and 26.1\% for add and remove tasks respectively. Furthermore, in light of on-device usage scenarios, we expand our research to include task-specific lightweight adapters leveraging the ControlNet-xs architecture. While ControlNet-xs excels in canny and depth guided generation, we propose to improve the communication between the control network and U-Net for more intricate add and remove tasks. We achieve this by enhancing ControlNet-xs with non-linear interaction layers based on Volterra filters. Our approach outperforms ControlNet-xs in both add/remove and canny-guided image generation tasks, highlighting the effectiveness of the proposed enhancement.

Title: MagicDriveDiT: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control

• Authors: Ruiyuan Gao, Kai Chen, Bo Xiao, Lanqing Hong, Zhenguo Li, Qiang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13807
• Pdf URL: https://arxiv.org/pdf/2411.13807
• Copy Paste: [[2411.13807]] MagicDriveDiT: High-Resolution Long Video Generation for Autonomous Driving with Adaptive Control(https://arxiv.org/abs/2411.13807)
• Keywords: diffusion
• Abstract: The rapid advancement of diffusion models has greatly improved video synthesis, especially in controllable video generation, which is essential for applications like autonomous driving. However, existing methods are limited by scalability and how control conditions are integrated, failing to meet the needs for high-resolution and long videos for autonomous driving applications. In this paper, we introduce MagicDriveDiT, a novel approach based on the DiT architecture, and tackle these challenges. Our method enhances scalability through flow matching and employs a progressive training strategy to manage complex scenarios. By incorporating spatial-temporal conditional encoding, MagicDriveDiT achieves precise control over spatial-temporal latents. Comprehensive experiments show its superior performance in generating realistic street scene videos with higher resolution and more frames. MagicDriveDiT significantly improves video generation quality and spatial-temporal controls, expanding its potential applications across various tasks in autonomous driving.

Title: CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentation

• Authors: Lin Sun, Jiale Cao, Jin Xie, Xiaoheng Jiang, Yanwei Pang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13836
• Pdf URL: https://arxiv.org/pdf/2411.13836
• Copy Paste: [[2411.13836]] CLIPer: Hierarchically Improving Spatial Representation of CLIP for Open-Vocabulary Semantic Segmentation(https://arxiv.org/abs/2411.13836)
• Keywords: diffusion, foundation model
• Abstract: Contrastive Language-Image Pre-training (CLIP) exhibits strong zero-shot classification ability on various image-level tasks, leading to the research to adapt CLIP for pixel-level open-vocabulary semantic segmentation without additional training. The key is to improve spatial representation of image-level CLIP, such as replacing self-attention map at last layer with self-self attention map or vision foundation model based attention map. In this paper, we present a novel hierarchical framework, named CLIPer, that hierarchically improves spatial representation of CLIP. The proposed CLIPer includes an early-layer fusion module and a fine-grained compensation module. We observe that, the embeddings and attention maps at early layers can preserve spatial structural information. Inspired by this, we design the early-layer fusion module to generate segmentation map with better spatial coherence. Afterwards, we employ a fine-grained compensation module to compensate the local details using the self-attention maps of diffusion model. We conduct the experiments on seven segmentation datasets. Our proposed CLIPer achieves the state-of-the-art performance on these datasets. For instance, using ViT-L, CLIPer has the mIoU of 69.8% and 43.3% on VOC and COCO Object, outperforming ProxyCLIP by 9.2% and 4.1% respectively.

Title: Detecting Human Artifacts from Text-to-Image Models

• Authors: Kaihong Wang, Lingzhi Zhang, Jianming Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13842
• Pdf URL: https://arxiv.org/pdf/2411.13842
• Copy Paste: [[2411.13842]] Detecting Human Artifacts from Text-to-Image Models(https://arxiv.org/abs/2411.13842)
• Keywords: diffusion, generative
• Abstract: Despite recent advancements, text-to-image generation models often produce images containing artifacts, especially in human figures. These artifacts appear as poorly generated human bodies, including distorted, missing, or extra body parts, leading to visual inconsistencies with typical human anatomy and greatly impairing overall fidelity. In this study, we address this challenge by curating Human Artifact Dataset (HAD), the first large-scale dataset specifically designed to identify and localize human artifacts. HAD comprises over 37,000 images generated by several popular text-to-image models, annotated for human artifact localization. Using this dataset, we train the Human Artifact Detection Models (HADM), which can identify diverse artifact types across multiple generative domains and demonstrate strong generalization, even on images from unseen generators. Additionally, to further improve generators' perception of human structural coherence, we use the predictions from our HADM as feedback for diffusion model finetuning. Our experiments confirm a reduction in human artifacts in the resulting model. Furthermore, we showcase a novel application of our HADM in an iterative inpainting framework to correct human artifacts in arbitrary images directly, demonstrating its utility in improving image quality. Our dataset and detection models are available at: \url{this https URL}.

Title: Dealing with Synthetic Data Contamination in Online Continual Learning

• Authors: Maorong Wang, Nicolas Michel, Jiafeng Mao, Toshihiko Yamasaki
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.13852
• Pdf URL: https://arxiv.org/pdf/2411.13852
• Copy Paste: [[2411.13852]] Dealing with Synthetic Data Contamination in Online Continual Learning(https://arxiv.org/abs/2411.13852)
• Keywords: diffusion
• Abstract: Image generation has shown remarkable results in generating high-fidelity realistic images, in particular with the advancement of diffusion-based models. However, the prevalence of AI-generated images may have side effects for the machine learning community that are not clearly identified. Meanwhile, the success of deep learning in computer vision is driven by the massive dataset collected on the Internet. The extensive quantity of synthetic data being added to the Internet would become an obstacle for future researchers to collect "clean" datasets without AI-generated content. Prior research has shown that using datasets contaminated by synthetic images may result in performance degradation when used for training. In this paper, we investigate the potential impact of contaminated datasets on Online Continual Learning (CL) research. We experimentally show that contaminated datasets might hinder the training of existing online CL methods. Also, we propose Entropy Selection with Real-synthetic similarity Maximization (ESRM), a method to alleviate the performance deterioration caused by synthetic images when training online CL models. Experiments show that our method can significantly alleviate performance deterioration, especially when the contamination is severe. For reproducibility, the source code of our work is available at this https URL.

Title: Decoupled Sparse Priors Guided Diffusion Compression Model for Point Clouds

• Authors: Xiaoge Zhang, Zijie Wu, Mehwish Nasim, Mingtao Feng, Ajmal Mian
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.13860
• Pdf URL: https://arxiv.org/pdf/2411.13860
• Copy Paste: [[2411.13860]] Decoupled Sparse Priors Guided Diffusion Compression Model for Point Clouds(https://arxiv.org/abs/2411.13860)
• Keywords: diffusion
• Abstract: Lossy compression methods rely on an autoencoder to transform a point cloud into latent points for storage, leaving the inherent redundancy of latent representations unexplored. To reduce redundancy in latent points, we propose a sparse priors guided method that achieves high reconstruction quality, especially at high compression ratios. This is accomplished by a dual-density scheme separately processing the latent points (intended for reconstruction) and the decoupled sparse priors (intended for storage). Our approach features an efficient dual-density data flow that relaxes size constraints on latent points, and hybridizes a progressive conditional diffusion model to encapsulate essential details for reconstruction within the conditions, which are decoupled hierarchically to intra-point and inter-point priors. Specifically, our method encodes the original point cloud into latent points and decoupled sparse priors through separate encoders. Latent points serve as intermediates, while sparse priors act as adaptive conditions. We then employ a progressive attention-based conditional denoiser to generate latent points conditioned on the decoupled priors, allowing the denoiser to dynamically attend to geometric and semantic cues from the priors at each encoding and decoding layer. Additionally, we integrate the local distribution into the arithmetic encoder and decoder to enhance local context modeling of the sparse points. The original point cloud is reconstructed through a point decoder. Compared to state-of-the-art, our method obtains superior rate-distortion trade-off, evidenced by extensive evaluations on the ShapeNet dataset and standard test datasets from MPEG group including 8iVFB, and Owlii.

Title: Sli2Vol+: Segmenting 3D Medical Images Based on an Object Estimation Guided Correspondence Flow Network

• Authors: Delin An, Pengfei Gu, Milan Sonka, Chaoli Wang, Danny Z. Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13873
• Pdf URL: https://arxiv.org/pdf/2411.13873
• Copy Paste: [[2411.13873]] Sli2Vol+: Segmenting 3D Medical Images Based on an Object Estimation Guided Correspondence Flow Network(https://arxiv.org/abs/2411.13873)
• Keywords: self-supervised
• Abstract: Deep learning (DL) methods have shown remarkable successes in medical image segmentation, often using large amounts of annotated data for model training. However, acquiring a large number of diverse labeled 3D medical image datasets is highly difficult and expensive. Recently, mask propagation DL methods were developed to reduce the annotation burden on 3D medical images. For example, Sli2Vol~\cite{yeung2021sli2vol} proposed a self-supervised framework (SSF) to learn correspondences by matching neighboring slices via slice reconstruction in the training stage; the learned correspondences were then used to propagate a labeled slice to other slices in the test stage. But, these methods are still prone to error accumulation due to the inter-slice propagation of reconstruction errors. Also, they do not handle discontinuities well, which can occur between consecutive slices in 3D images, as they emphasize exploiting object continuity. To address these challenges, in this work, we propose a new SSF, called \proposed, {for segmenting any anatomical structures in 3D medical images using only a single annotated slice per training and testing volume.} Specifically, in the training stage, we first propagate an annotated 2D slice of a training volume to the other slices, generating pseudo-labels (PLs). Then, we develop a novel Object Estimation Guided Correspondence Flow Network to learn reliable correspondences between consecutive slices and corresponding PLs in a self-supervised manner. In the test stage, such correspondences are utilized to propagate a single annotated slice to the other slices of a test volume. We demonstrate the effectiveness of our method on various medical image segmentation tasks with different datasets, showing better generalizability across different organs, modalities, and modals. Code is available at \url{this https URL}

Title: Dressing the Imagination: A Dataset for AI-Powered Translation of Text into Fashion Outfits and A Novel KAN Adapter for Enhanced Feature Adaptation

• Authors: Gayatri Deshmukh, Somsubhra De, Chirag Sehgal, Jishu Sen Gupta, Sparsh Mittal
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13901
• Pdf URL: https://arxiv.org/pdf/2411.13901
• Copy Paste: [[2411.13901]] Dressing the Imagination: A Dataset for AI-Powered Translation of Text into Fashion Outfits and A Novel KAN Adapter for Enhanced Feature Adaptation(https://arxiv.org/abs/2411.13901)
• Keywords: generative
• Abstract: Specialized datasets that capture the fashion industry's rich language and styling elements can boost progress in AI-driven fashion design. We present FLORA (Fashion Language Outfit Representation for Apparel Generation), the first comprehensive dataset containing 4,330 curated pairs of fashion outfits and corresponding textual descriptions. Each description utilizes industry-specific terminology and jargon commonly used by professional fashion designers, providing precise and detailed insights into the outfits. Hence, the dataset captures the delicate features and subtle stylistic elements necessary to create high-fidelity fashion designs. We demonstrate that fine-tuning generative models on the FLORA dataset significantly enhances their capability to generate accurate and stylistically rich images from textual descriptions of fashion sketches. FLORA will catalyze the creation of advanced AI models capable of comprehending and producing subtle, stylistically rich fashion designs. It will also help fashion designers and end-users to bring their ideas to life. As a second orthogonal contribution, we introduce KAN Adapters, which leverage Kolmogorov-Arnold Networks (KAN) as adaptive modules. They serve as replacements for traditional MLP-based LoRA adapters. With learnable spline-based activations, KAN Adapters excel in modeling complex, non-linear relationships, achieving superior fidelity, faster convergence and semantic alignment. Extensive experiments and ablation studies on our proposed FLORA dataset validate the superiority of KAN Adapters over LoRA adapters. To foster further research and collaboration, we will open-source both the FLORA and our implementation code.

Title: Learning to Cooperate with Humans using Generative Agents

• Authors: Yancheng Liang, Daphne Chen, Abhishek Gupta, Simon S. Du, Natasha Jaques
• Subjects: cs.LG, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2411.13934
• Pdf URL: https://arxiv.org/pdf/2411.13934
• Copy Paste: [[2411.13934]] Learning to Cooperate with Humans using Generative Agents(https://arxiv.org/abs/2411.13934)
• Keywords: generative
• Abstract: Training agents that can coordinate zero-shot with humans is a key mission in multi-agent reinforcement learning (MARL). Current algorithms focus on training simulated human partner policies which are then used to train a Cooperator agent. The simulated human is produced either through behavior cloning over a dataset of human cooperation behavior, or by using MARL to create a population of simulated agents. However, these approaches often struggle to produce a Cooperator that can coordinate well with real humans, since the simulated humans fail to cover the diverse strategies and styles employed by people in the real world. We show \emph{learning a generative model of human partners} can effectively address this issue. Our model learns a latent variable representation of the human that can be regarded as encoding the human's unique strategy, intention, experience, or style. This generative model can be flexibly trained from any (human or neural policy) agent interaction data. By sampling from the latent space, we can use the generative model to produce different partners to train Cooperator agents. We evaluate our method -- \textbf{G}enerative \textbf{A}gent \textbf{M}odeling for \textbf{M}ulti-agent \textbf{A}daptation (GAMMA) -- on Overcooked, a challenging cooperative cooking game that has become a standard benchmark for zero-shot coordination. We conduct an evaluation with real human teammates, and the results show that GAMMA consistently improves performance, whether the generative model is trained on simulated populations or human datasets. Further, we propose a method for posterior sampling from the generative model that is biased towards the human data, enabling us to efficiently improve performance with only a small amount of expensive human interaction data.

Title: A Dataset for Evaluating Online Anomaly Detection Approaches for Discrete Multivariate Time Series

• Authors: Lucas Correia, Jan-Christoph Goos, Thomas Bäck, Anna V. Kononova
• Subjects: cs.LG, cs.AI, cs.CE, eess.SY
• Abstract URL: https://arxiv.org/abs/2411.13951
• Pdf URL: https://arxiv.org/pdf/2411.13951
• Copy Paste: [[2411.13951]] A Dataset for Evaluating Online Anomaly Detection Approaches for Discrete Multivariate Time Series(https://arxiv.org/abs/2411.13951)
• Keywords: anomaly
• Abstract: Benchmarking anomaly detection approaches for multivariate time series is challenging due to the lack of high-quality datasets. Current publicly available datasets are too small, not diverse and feature trivial anomalies, which hinders measurable progress in this research area. We propose a solution: a diverse, extensive, and non-trivial dataset generated via state-of-the-art simulation tools that reflects realistic behaviour of an automotive powertrain, including its multivariate, dynamic and variable-state properties. To cater for both unsupervised and semi-supervised anomaly detection settings, as well as time series generation and forecasting, we make different versions of the dataset available, where training and test subsets are offered in contaminated and clean versions, depending on the task. We also provide baseline results from a small selection of approaches based on deterministic and variational autoencoders, as well as a non-parametric approach. As expected, the baseline experimentation shows that the approaches trained on the semi-supervised version of the dataset outperform their unsupervised counterparts, highlighting a need for approaches more robust to contaminated training data.

Title: Zero-Shot Low-Light Image Enhancement via Joint Frequency Domain Priors Guided Diffusion

• Authors: Jinhong He, Shivakumara Palaiahnakote, Aoxiang Ning, Minglong Xue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13961
• Pdf URL: https://arxiv.org/pdf/2411.13961
• Copy Paste: [[2411.13961]] Zero-Shot Low-Light Image Enhancement via Joint Frequency Domain Priors Guided Diffusion(https://arxiv.org/abs/2411.13961)
• Keywords: diffusion
• Abstract: Due to the singularity of real-world paired datasets and the complexity of low-light environments, this leads to supervised methods lacking a degree of scene generalisation. Meanwhile, limited by poor lighting and content guidance, existing zero-shot methods cannot handle unknown severe degradation well. To address this problem, we will propose a new zero-shot low-light enhancement method to compensate for the lack of light and structural information in the diffusion sampling process by effectively combining the wavelet and Fourier frequency domains to construct rich a priori information. The key to the inspiration comes from the similarity between the wavelet and Fourier frequency domains: both light and structure information are closely related to specific frequency domain regions, respectively. Therefore, by transferring the diffusion process to the wavelet low-frequency domain and combining the wavelet and Fourier frequency domains by continuously decomposing them in the inverse process, the constructed rich illumination prior is utilised to guide the image generation enhancement process. Sufficient experiments show that the framework is robust and effective in various scenarios. The code will be available at: \href{this https URL}{this https URL}.

Title: Transforming Static Images Using Generative Models for Video Salient Object Detection

• Authors: Suhwan Cho, Minhyeok Lee, Jungho Lee, Sangyoun Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.13975
• Pdf URL: https://arxiv.org/pdf/2411.13975
• Copy Paste: [[2411.13975]] Transforming Static Images Using Generative Models for Video Salient Object Detection(https://arxiv.org/abs/2411.13975)
• Keywords: diffusion, generative
• Abstract: In many video processing tasks, leveraging large-scale image datasets is a common strategy, as image data is more abundant and facilitates comprehensive knowledge transfer. A typical approach for simulating video from static images involves applying spatial transformations, such as affine transformations and spline warping, to create sequences that mimic temporal progression. However, in tasks like video salient object detection, where both appearance and motion cues are critical, these basic image-to-video techniques fail to produce realistic optical flows that capture the independent motion properties of each object. In this study, we show that image-to-video diffusion models can generate realistic transformations of static images while understanding the contextual relationships between image components. This ability allows the model to generate plausible optical flows, preserving semantic integrity while reflecting the independent motion of scene elements. By augmenting individual images in this way, we create large-scale image-flow pairs that significantly enhance model training. Our approach achieves state-of-the-art performance across all public benchmark datasets, outperforming existing approaches.

Title: On the Fairness, Diversity and Reliability of Text-to-Image Generative Models

• Authors: Jordan Vice, Naveed Akhtar, Richard Hartley, Ajmal Mian
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13981
• Pdf URL: https://arxiv.org/pdf/2411.13981
• Copy Paste: [[2411.13981]] On the Fairness, Diversity and Reliability of Text-to-Image Generative Models(https://arxiv.org/abs/2411.13981)
• Keywords: generative
• Abstract: The widespread availability of multimodal generative models has sparked critical discussions on their fairness, reliability, and potential for misuse. While text-to-image models can produce high-fidelity, user-guided images, they also exhibit unpredictable behavior and vulnerabilities, which can be exploited to manipulate class or concept representations. To address this, we propose an evaluation framework designed to assess model reliability through their responses to globally- and locally-applied `semantic' perturbations in the embedding space, pinpointing inputs that trigger unreliable behavior. Our approach offers deeper insights into two essential aspects: (i) generative diversity, evaluating the breadth of visual representations for learned concepts, and (ii) generative fairness, examining how removing concepts from input prompts affects semantic guidance. Beyond these evaluations, our method lays the groundwork for detecting unreliable, bias-injected models and retrieval of bias provenance. We will release our code. Keywords: Fairness, Reliability, AI Ethics, Bias, Text-to-Image Models

Title: Safety Without Semantic Disruptions: Editing-free Safe Image Generation via Context-preserving Dual Latent Reconstruction

• Authors: Jordan Vice, Naveed Akhtar, Richard Hartley, Ajmal Mian
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.13982
• Pdf URL: https://arxiv.org/pdf/2411.13982
• Copy Paste: [[2411.13982]] Safety Without Semantic Disruptions: Editing-free Safe Image Generation via Context-preserving Dual Latent Reconstruction(https://arxiv.org/abs/2411.13982)
• Keywords: generative
• Abstract: Training multimodal generative models on large, uncurated datasets can result in users being exposed to harmful, unsafe and controversial or culturally-inappropriate outputs. While model editing has been proposed to remove or filter undesirable concepts in embedding and latent spaces, it can inadvertently damage learned manifolds, distorting concepts in close semantic proximity. We identify limitations in current model editing techniques, showing that even benign, proximal concepts may become misaligned. To address the need for safe content generation, we propose a modular, dynamic solution that leverages safety-context embeddings and a dual reconstruction process using tunable weighted summation in the latent space to generate safer images. Our method preserves global context without compromising the structural integrity of the learned manifolds. We achieve state-of-the-art results on safe image generation benchmarks, while offering controllable variation of model safety. We identify trade-offs between safety and censorship, which presents a necessary perspective in the development of ethical AI models. We will release our code. Keywords: Text-to-Image Models, Generative AI, Safety, Reliability, Model Editing

Title: Generative Intervention Models for Causal Perturbation Modeling

• Authors: Nora Schneider, Lars Lorch, Niki Kilbertus, Bernhard Schölkopf, Andreas Krause
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.14003
• Pdf URL: https://arxiv.org/pdf/2411.14003
• Copy Paste: [[2411.14003]] Generative Intervention Models for Causal Perturbation Modeling(https://arxiv.org/abs/2411.14003)
• Keywords: generative
• Abstract: We consider the problem of predicting perturbation effects via causal models. In many applications, it is a priori unknown which mechanisms of a system are modified by an external perturbation, even though the features of the perturbation are available. For example, in genomics, some properties of a drug may be known, but not their causal effects on the regulatory pathways of cells. We propose a generative intervention model (GIM) that learns to map these perturbation features to distributions over atomic interventions in a jointly-estimated causal model. Contrary to prior approaches, this enables us to predict the distribution shifts of unseen perturbation features while gaining insights about their mechanistic effects in the underlying data-generating process. On synthetic data and scRNA-seq drug perturbation data, GIMs achieve robust out-of-distribution predictions on par with unstructured approaches, while effectively inferring the underlying perturbation mechanisms, often better than other causal inference methods.

Title: FunctionChat-Bench: Comprehensive Evaluation of Language Models' Generative Capabilities in Korean Tool-use Dialogs

• Authors: Shinbok Lee, Gaeun Seo, Daniel Lee, Byeongil Ko, Sunghee Jung, Myeongcheol Shin
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14054
• Pdf URL: https://arxiv.org/pdf/2411.14054
• Copy Paste: [[2411.14054]] FunctionChat-Bench: Comprehensive Evaluation of Language Models' Generative Capabilities in Korean Tool-use Dialogs(https://arxiv.org/abs/2411.14054)
• Keywords: generative
• Abstract: This study investigates language models' generative capabilities in tool-use dialogs. We categorize the models' outputs in tool-use dialogs into four distinct types: Tool Call, Answer Completion, Slot Question, and Relevance Detection, which serve as aspects for evaluation. We introduce FunctionChat-Bench, comprising 700 evaluation items and automated assessment programs. Using this benchmark, we evaluate several language models that support function calling. Our findings indicate that while language models may exhibit high accuracy in single-turn Tool Call scenarios, this does not necessarily translate to superior generative performance in multi-turn environments. We argue that the capabilities required for function calling extend beyond generating tool call messages; they must also effectively generate conversational messages that engage the user.

Title: MMGenBench: Evaluating the Limits of LMMs from the Text-to-Image Generation Perspective

• Authors: Hailang Huang, Yong Wang, Zixuan Huang, Huaqiu Li, Tongwen Huang, Xiangxiang Chu, Richong Zhang
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14062
• Pdf URL: https://arxiv.org/pdf/2411.14062
• Copy Paste: [[2411.14062]] MMGenBench: Evaluating the Limits of LMMs from the Text-to-Image Generation Perspective(https://arxiv.org/abs/2411.14062)
• Keywords: generative
• Abstract: Large Multimodal Models (LMMs) have demonstrated remarkable capabilities. While existing benchmarks for evaluating LMMs mainly focus on image comprehension, few works evaluate them from the image generation perspective. To address this issue, we propose a straightforward automated evaluation pipeline. Specifically, this pipeline requires LMMs to generate an image-prompt from a given input image. Subsequently, it employs text-to-image generative models to create a new image based on these generated prompts. Finally, we evaluate the performance of LMMs by comparing the original image with the generated one. Furthermore, we introduce MMGenBench-Test, a comprehensive benchmark developed to evaluate LMMs across 13 distinct image patterns, and MMGenBench-Domain, targeting the performance evaluation of LMMs within the generative image domain. A thorough evaluation involving over 50 popular LMMs demonstrates the effectiveness and reliability in both the pipeline and benchmark. Our observations indicate that numerous LMMs excelling in existing benchmarks fail to adequately complete the basic tasks, related to image understanding and description. This finding highlights the substantial potential for performance improvement in current LMMs and suggests avenues for future model optimization. Concurrently, our pipeline facilitates the efficient assessment of LMMs performance across diverse domains by using solely image inputs.

Title: Multi LoRA Meets Vision: Merging multiple adapters to create a multi task model

• Authors: Ege Kesim, Selahattin Serdar Helli
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.14064
• Pdf URL: https://arxiv.org/pdf/2411.14064
• Copy Paste: [[2411.14064]] Multi LoRA Meets Vision: Merging multiple adapters to create a multi task model(https://arxiv.org/abs/2411.14064)
• Keywords: generative
• Abstract: Parameter efficient finetuning (PEFT) methods are widely used in LLMs and generative models in computer vision. Especially one can use multiple of these during inference to change the behavior of the base model. In this paper we investigated whether multiple LoRA adapters trained on computer vision tasks can be merged together and used during inference without loss in performance. By achieving this, multitask models can be created just by merging different LoRAs. Merging these will reduce inference time and it will not require any additional retraining. We have trained adapters on six different tasks and evaluated their performance when they are merged together. For comparison we used a model with a frozen backbone and finetuned its head. Our results show that even with simple merging techniques creating a multitask model by merging adapters is achievable by slightly loosing performance in some cases. In our experiments we merged up to three adapters together. Depending on the task and the similarity of the data adapters were trained on, merges can outperform head finetuning. We have observed that LoRAs trained with dissimilar datasets tend to perform better compared to model trained on similar datasets.

Title: RAG-Thief: Scalable Extraction of Private Data from Retrieval-Augmented Generation Applications with Agent-based Attacks

• Authors: Changyue Jiang, Xudong Pan, Geng Hong, Chenfu Bao, Min Yang
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.14110
• Pdf URL: https://arxiv.org/pdf/2411.14110
• Copy Paste: [[2411.14110]] RAG-Thief: Scalable Extraction of Private Data from Retrieval-Augmented Generation Applications with Agent-based Attacks(https://arxiv.org/abs/2411.14110)
• Keywords: generative
• Abstract: While large language models (LLMs) have achieved notable success in generative tasks, they still face limitations, such as lacking up-to-date knowledge and producing hallucinations. Retrieval-Augmented Generation (RAG) enhances LLM performance by integrating external knowledge bases, providing additional context which significantly improves accuracy and knowledge coverage. However, building these external knowledge bases often requires substantial resources and may involve sensitive information. In this paper, we propose an agent-based automated privacy attack called RAG-Thief, which can extract a scalable amount of private data from the private database used in RAG applications. We conduct a systematic study on the privacy risks associated with RAG applications, revealing that the vulnerability of LLMs makes the private knowledge bases suffer significant privacy risks. Unlike previous manual attacks which rely on traditional prompt injection techniques, RAG-Thief starts with an initial adversarial query and learns from model responses, progressively generating new queries to extract as many chunks from the knowledge base as possible. Experimental results show that our RAG-Thief can extract over 70% information from the private knowledge bases within customized RAG applications deployed on local machines and real-world platforms, including OpenAI's GPTs and ByteDance's Coze. Our findings highlight the privacy vulnerabilities in current RAG applications and underscore the pressing need for stronger safeguards.

Title: Point Cloud Resampling with Learnable Heat Diffusion

• Authors: Wenqiang Xu, Wenrui Dai, Duoduo Xue, Ziyang Zheng, Chenglin Li, Junni Zou, Hongkai Xiong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14120
• Pdf URL: https://arxiv.org/pdf/2411.14120
• Copy Paste: [[2411.14120]] Point Cloud Resampling with Learnable Heat Diffusion(https://arxiv.org/abs/2411.14120)
• Keywords: diffusion, generative
• Abstract: Generative diffusion models have shown empirical successes in point cloud resampling, generating a denser and more uniform distribution of points from sparse or noisy 3D point clouds by progressively refining noise into structure. However, existing diffusion models employ manually predefined schemes, which often fail to recover the underlying point cloud structure due to the rigid and disruptive nature of the geometric degradation. To address this issue, we propose a novel learnable heat diffusion framework for point cloud resampling, which directly parameterizes the marginal distribution for the forward process by learning the adaptive heat diffusion schedules and local filtering scales of the time-varying heat kernel, and consequently, generates an adaptive conditional prior for the reverse process. Unlike previous diffusion models with a fixed prior, the adaptive conditional prior selectively preserves geometric features of the point cloud by minimizing a refined variational lower bound, guiding the points to evolve towards the underlying surface during the reverse process. Extensive experimental results demonstrate that the proposed point cloud resampling achieves state-of-the-art performance in representative reconstruction tasks including point cloud denoising and upsampling.

Title: RestorerID: Towards Tuning-Free Face Restoration with ID Preservation

• Authors: Jiacheng Ying, Mushui Liu, Zhe Wu, Runming Zhang, Zhu Yu, Siming Fu, Si-Yuan Cao, Chao Wu, Yunlong Yu, Hui-Liang Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14125
• Pdf URL: https://arxiv.org/pdf/2411.14125
• Copy Paste: [[2411.14125]] RestorerID: Towards Tuning-Free Face Restoration with ID Preservation(https://arxiv.org/abs/2411.14125)
• Keywords: diffusion
• Abstract: Blind face restoration has made great progress in producing high-quality and lifelike images. Yet it remains challenging to preserve the ID information especially when the degradation is heavy. Current reference-guided face restoration approaches either require face alignment or personalized test-tuning, which are unfaithful or time-consuming. In this paper, we propose a tuning-free method named RestorerID that incorporates ID preservation during face restoration. RestorerID is a diffusion model-based method that restores low-quality images with varying levels of degradation by using a single reference image. To achieve this, we propose a unified framework to combine the ID injection with the base blind face restoration model. In addition, we design a novel Face ID Rebalancing Adapter (FIR-Adapter) to tackle the problems of content unconsistency and contours misalignment that are caused by information conflicts between the low-quality input and reference image. Furthermore, by employing an Adaptive ID-Scale Adjusting strategy, RestorerID can produce superior restored images across various levels of degradation. Experimental results on the Celeb-Ref dataset and real-world scenarios demonstrate that RestorerID effectively delivers high-quality face restoration with ID preservation, achieving a superior performance compared to the test-tuning approaches and other reference-guided ones. The code of RestorerID is available at \url{this https URL}.

Title: GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMs

• Authors: Advik Raj Basani, Xiao Zhang
• Subjects: cs.LG, cs.AI, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14133
• Pdf URL: https://arxiv.org/pdf/2411.14133
• Copy Paste: [[2411.14133]] GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMs(https://arxiv.org/abs/2411.14133)
• Keywords: generative
• Abstract: Large Language Models (LLMs) have shown impressive proficiency across a range of natural language processing tasks yet remain vulnerable to adversarial prompts, known as jailbreak attacks, carefully designed to elicit harmful responses from LLMs. Traditional methods rely on manual heuristics, which suffer from limited generalizability. While being automatic, optimization-based attacks often produce unnatural jailbreak prompts that are easy to detect by safety filters or require high computational overhead due to discrete token optimization. Witnessing the limitations of existing jailbreak methods, we introduce Generative Adversarial Suffix Prompter (GASP), a novel framework that combines human-readable prompt generation with Latent Bayesian Optimization (LBO) to improve adversarial suffix creation in a fully black-box setting. GASP leverages LBO to craft adversarial suffixes by efficiently exploring continuous embedding spaces, gradually optimizing the model to improve attack efficacy while balancing prompt coherence through a targeted iterative refinement procedure. Our experiments show that GASP can generate natural jailbreak prompts, significantly improving attack success rates, reducing training times, and accelerating inference speed, thus making it an efficient and scalable solution for red-teaming LLMs.

Title: Novel View Extrapolation with Video Diffusion Priors

• Authors: Kunhao Liu, Ling Shao, Shijian Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14208
• Pdf URL: https://arxiv.org/pdf/2411.14208
• Copy Paste: [[2411.14208]] Novel View Extrapolation with Video Diffusion Priors(https://arxiv.org/abs/2411.14208)
• Keywords: diffusion, generative
• Abstract: The field of novel view synthesis has made significant strides thanks to the development of radiance field methods. However, most radiance field techniques are far better at novel view interpolation than novel view extrapolation where the synthesis novel views are far beyond the observed training views. We design ViewExtrapolator, a novel view synthesis approach that leverages the generative priors of Stable Video Diffusion (SVD) for realistic novel view extrapolation. By redesigning the SVD denoising process, ViewExtrapolator refines the artifact-prone views rendered by radiance fields, greatly enhancing the clarity and realism of the synthesized novel views. ViewExtrapolator is a generic novel view extrapolator that can work with different types of 3D rendering such as views rendered from point clouds when only a single view or monocular video is available. Additionally, ViewExtrapolator requires no fine-tuning of SVD, making it both data-efficient and computation-efficient. Extensive experiments demonstrate the superiority of ViewExtrapolator in novel view extrapolation. Project page: \url{this https URL}.

Title: Generative Outpainting To Enhance the Memorability of Short-Form Videos

• Authors: Alan Byju, Aman Sudhindra Ladwa, Lorin Sweeney, Alan F. Smeaton
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14213
• Pdf URL: https://arxiv.org/pdf/2411.14213
• Copy Paste: [[2411.14213]] Generative Outpainting To Enhance the Memorability of Short-Form Videos(https://arxiv.org/abs/2411.14213)
• Keywords: generative
• Abstract: With the expanding use of the short-form video format in advertising, social media, entertainment, education and more, there is a need for such media to both captivate and be remembered. Video memorability indicates to us how likely a video is to be remembered by a viewer who has no emotional or personal connection with its content. This paper presents the results of using generative outpainting to expand the screen size of a short-form video with a view to improving its memorability. Advances in machine learning and deep learning are compared and leveraged to understand how extending the borders of video screensizes can affect their memorability to viewers. Using quantitative evaluation we determine the best-performing model for outpainting and the impact of outpainting based on image saliency on video memorability scores

Title: Efficient Aspect-Based Summarization of Climate Change Reports with Small Language Models

• Authors: Iacopo Ghinassi, Leonardo Catalano, Tommaso Colella
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.14272
• Pdf URL: https://arxiv.org/pdf/2411.14272
• Copy Paste: [[2411.14272]] Efficient Aspect-Based Summarization of Climate Change Reports with Small Language Models(https://arxiv.org/abs/2411.14272)
• Keywords: generative
• Abstract: The use of Natural Language Processing (NLP) for helping decision-makers with Climate Change action has recently been highlighted as a use case aligning with a broader drive towards NLP technologies for social good. In this context, Aspect-Based Summarization (ABS) systems that extract and summarize relevant information are particularly useful as they provide stakeholders with a convenient way of finding relevant information in expert-curated reports. In this work, we release a new dataset for ABS of Climate Change reports and we employ different Large Language Models (LLMs) and so-called Small Language Models (SLMs) to tackle this problem in an unsupervised way. Considering the problem at hand, we also show how SLMs are not significantly worse for the problem while leading to reduced carbon footprint; we do so by applying for the first time an existing framework considering both energy efficiency and task performance to the evaluation of zero-shot generative models for ABS. Overall, our results show that modern language models, both big and small, can effectively tackle ABS for Climate Change reports but more research is needed when we frame the problem as a Retrieval Augmented Generation (RAG) problem and our work and dataset will help foster efforts in this direction.

Title: Adaptive Anomaly Detection for Identifying Attacks in Cyber-Physical Systems: A Systematic Literature Review

• Authors: Pablo Moriano, Steven C. Hespeler, Mingyan Li, Maria Mahbub
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2411.14278
• Pdf URL: https://arxiv.org/pdf/2411.14278
• Copy Paste: [[2411.14278]] Adaptive Anomaly Detection for Identifying Attacks in Cyber-Physical Systems: A Systematic Literature Review(https://arxiv.org/abs/2411.14278)
• Keywords: anomaly
• Abstract: Modern cyberattacks in cyber-physical systems (CPS) rapidly evolve and cannot be deterred effectively with most current methods which focused on characterizing past threats. Adaptive anomaly detection (AAD) is among the most promising techniques to detect evolving cyberattacks focused on fast data processing and model adaptation. AAD has been researched in the literature extensively; however, to the best of our knowledge, our work is the first systematic literature review (SLR) on the current research within this field. We present a comprehensive SLR, gathering 397 relevant papers and systematically analyzing 65 of them (47 research and 18 survey papers) on AAD in CPS studies from 2013 to 2023 (November). We introduce a novel taxonomy considering attack types, CPS application, learning paradigm, data management, and algorithms. Our analysis indicates, among other findings, that reviewed works focused on a single aspect of adaptation (either data processing or model adaptation) but rarely in both at the same time. We aim to help researchers to advance the state of the art and help practitioners to become familiar with recent progress in this field. We identify the limitations of the state of the art and provide recommendations for future research directions.

Title: StereoCrafter-Zero: Zero-Shot Stereo Video Generation with Noisy Restart

• Authors: Jian Shi, Qian Wang, Zhenyu Li, Peter Wonka
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14295
• Pdf URL: https://arxiv.org/pdf/2411.14295
• Copy Paste: [[2411.14295]] StereoCrafter-Zero: Zero-Shot Stereo Video Generation with Noisy Restart(https://arxiv.org/abs/2411.14295)
• Keywords: diffusion
• Abstract: Generating high-quality stereo videos that mimic human binocular vision requires maintaining consistent depth perception and temporal coherence across frames. While diffusion models have advanced image and video synthesis, generating high-quality stereo videos remains challenging due to the difficulty of maintaining consistent temporal and spatial coherence between left and right views. We introduce \textit{StereoCrafter-Zero}, a novel framework for zero-shot stereo video generation that leverages video diffusion priors without the need for paired training data. Key innovations include a noisy restart strategy to initialize stereo-aware latents and an iterative refinement process that progressively harmonizes the latent space, addressing issues like temporal flickering and view inconsistencies. Comprehensive evaluations, including quantitative metrics and user studies, demonstrate that \textit{StereoCrafter-Zero} produces high-quality stereo videos with improved depth consistency and temporal smoothness, even when depth estimations are imperfect. Our framework is robust and adaptable across various diffusion models, setting a new benchmark for zero-shot stereo video generation and enabling more immersive visual experiences. Our code can be found in~\url{this https URL}.

Title: Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable Single-stage Image-to-3D Generation

• Authors: Yuanhao Cai, He Zhang, Kai Zhang, Yixun Liang, Mengwei Ren, Fujun Luan, Qing Liu, Soo Ye Kim, Jianming Zhang, Zhifei Zhang, Yuqian Zhou, Zhe Lin, Alan Yuille
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2411.14384
• Pdf URL: https://arxiv.org/pdf/2411.14384
• Copy Paste: [[2411.14384]] Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable Single-stage Image-to-3D Generation(https://arxiv.org/abs/2411.14384)
• Keywords: diffusion
• Abstract: Existing feed-forward image-to-3D methods mainly rely on 2D multi-view diffusion models that cannot guarantee 3D consistency. These methods easily collapse when changing the prompt view direction and mainly handle object-centric prompt images. In this paper, we propose a novel single-stage 3D diffusion model, DiffusionGS, for object and scene generation from a single view. DiffusionGS directly outputs 3D Gaussian point clouds at each timestep to enforce view consistency and allow the model to generate robustly given prompt views of any directions, beyond object-centric inputs. Plus, to improve the capability and generalization ability of DiffusionGS, we scale up 3D training data by developing a scene-object mixed training strategy. Experiments show that our method enjoys better generation quality (2.20 dB higher in PSNR and 23.25 lower in FID) and over 5x faster speed (~6s on an A100 GPU) than SOTA methods. The user study and text-to-3D applications also reveals the practical values of our method. Our Project page at this https URL shows the video and interactive generation results.

Title: From RNNs to Foundation Models: An Empirical Study on Commercial Building Energy Consumption

• Authors: Shourya Bose, Yijiang Li, Amy Van Sant, Yu Zhang, Kibaek Kim
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14421
• Pdf URL: https://arxiv.org/pdf/2411.14421
• Copy Paste: [[2411.14421]] From RNNs to Foundation Models: An Empirical Study on Commercial Building Energy Consumption(https://arxiv.org/abs/2411.14421)
• Keywords: foundation model
• Abstract: Accurate short-term energy consumption forecasting for commercial buildings is crucial for smart grid operations. While smart meters and deep learning models enable forecasting using past data from multiple buildings, data heterogeneity from diverse buildings can reduce model performance. The impact of increasing dataset heterogeneity in time series forecasting, while keeping size and model constant, is understudied. We tackle this issue using the ComStock dataset, which provides synthetic energy consumption data for U.S. commercial buildings. Two curated subsets, identical in size and region but differing in building type diversity, are used to assess the performance of various time series forecasting models, including fine-tuned open-source foundation models (FMs). The results show that dataset heterogeneity and model architecture have a greater impact on post-training forecasting performance than the parameter count. Moreover, despite the higher computational cost, fine-tuned FMs demonstrate competitive performance compared to base models trained from scratch.

Title: Unleashing the Potential of Multi-modal Foundation Models and Video Diffusion for 4D Dynamic Physical Scene Simulation

• Authors: Zhuoman Liu, Weicai Ye, Yan Luximon, Pengfei Wan, Di Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.14423
• Pdf URL: https://arxiv.org/pdf/2411.14423
• Copy Paste: [[2411.14423]] Unleashing the Potential of Multi-modal Foundation Models and Video Diffusion for 4D Dynamic Physical Scene Simulation(https://arxiv.org/abs/2411.14423)
• Keywords: diffusion, foundation model
• Abstract: Realistic simulation of dynamic scenes requires accurately capturing diverse material properties and modeling complex object interactions grounded in physical principles. However, existing methods are constrained to basic material types with limited predictable parameters, making them insufficient to represent the complexity of real-world materials. We introduce a novel approach that leverages multi-modal foundation models and video diffusion to achieve enhanced 4D dynamic scene simulation. Our method utilizes multi-modal models to identify material types and initialize material parameters through image queries, while simultaneously inferring 3D Gaussian splats for detailed scene representation. We further refine these material parameters using video diffusion with a differentiable Material Point Method (MPM) and optical flow guidance rather than render loss or Score Distillation Sampling (SDS) loss. This integrated framework enables accurate prediction and realistic simulation of dynamic interactions in real-world scenarios, advancing both accuracy and flexibility in physics-based simulations.

Title: Stable Flow: Vital Layers for Training-Free Image Editing

• Authors: Omri Avrahami, Or Patashnik, Ohad Fried, Egor Nemchinov, Kfir Aberman, Dani Lischinski, Daniel Cohen-Or
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.14430
• Pdf URL: https://arxiv.org/pdf/2411.14430
• Copy Paste: [[2411.14430]] Stable Flow: Vital Layers for Training-Free Image Editing(https://arxiv.org/abs/2411.14430)
• Keywords: diffusion
• Abstract: Diffusion models have revolutionized the field of content synthesis and editing. Recent models have replaced the traditional UNet architecture with the Diffusion Transformer (DiT), and employed flow-matching for improved training and sampling. However, they exhibit limited generation diversity. In this work, we leverage this limitation to perform consistent image edits via selective injection of attention features. The main challenge is that, unlike the UNet-based models, DiT lacks a coarse-to-fine synthesis structure, making it unclear in which layers to perform the injection. Therefore, we propose an automatic method to identify "vital layers" within DiT, crucial for image formation, and demonstrate how these layers facilitate a range of controlled stable edits, from non-rigid modifications to object addition, using the same mechanism. Next, to enable real-image editing, we introduce an improved image inversion method for flow models. Finally, we evaluate our approach through qualitative and quantitative comparisons, along with a user study, and demonstrate its effectiveness across multiple applications. The project page is available at this https URL