2024-11-15

Title: Aligning Visual Contrastive learning models via Preference Optimization

• Authors: Amirabbas Afzali, Borna Khodabandeh, Ali Rasekh, Mahyar JafariNodeh, Sepehr kazemi, Simon Gottschalk
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.08923
• Pdf URL: https://arxiv.org/pdf/2411.08923
• Copy Paste: [[2411.08923]] Aligning Visual Contrastive learning models via Preference Optimization(https://arxiv.org/abs/2411.08923)
• Keywords: generative
• Abstract: Contrastive learning models have demonstrated impressive abilities to capture semantic similarities by aligning representations in the embedding space. However, their performance can be limited by the quality of the training data and its inherent biases. While Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) have been applied to generative models to align them with human preferences, their use in contrastive learning has yet to be explored. This paper introduces a novel method for training contrastive learning models using Preference Optimization (PO) to break down complex concepts. Our method systematically aligns model behavior with desired preferences, enhancing performance on the targeted task. In particular, we focus on enhancing model robustness against typographic attacks, commonly seen in contrastive models like CLIP. We further apply our method to disentangle gender understanding and mitigate gender biases, offering a more nuanced control over these sensitive attributes. Our experiments demonstrate that models trained using PO outperform standard contrastive learning techniques while retaining their ability to handle adversarial challenges and maintain accuracy on other downstream tasks. This makes our method well-suited for tasks requiring fairness, robustness, and alignment with specific preferences. We evaluate our method on several vision-language tasks, tackling challenges such as typographic attacks. Additionally, we explore the model's ability to disentangle gender concepts and mitigate gender bias, showcasing the versatility of our approach.

Title: Retrieval of sun-induced plant fluorescence in the O$_2$-A absorption band from DESIS imagery

• Authors: Jim Buffat, Miguel Pato, Kevin Alonso, Stefan Auer, Emiliano Carmona, Stefan Maier, Rupert Müller, Patrick Rademske, Uwe Rascher, Hanno Scharr
• Subjects: cs.CV, cs.AI, physics.geo-ph
• Abstract URL: https://arxiv.org/abs/2411.08925
• Pdf URL: https://arxiv.org/pdf/2411.08925
• Copy Paste: [[2411.08925]] Retrieval of sun-induced plant fluorescence in the O$_2$-A absorption band from DESIS imagery(https://arxiv.org/abs/2411.08925)
• Keywords: self-supervised
• Abstract: We provide the first method allowing to retrieve spaceborne SIF maps at 30 m ground resolution with a strong correlation ($r^2=0.6$) to high-quality airborne estimates of sun-induced fluorescence (SIF). SIF estimates can provide explanatory information for many tasks related to agricultural management and physiological studies. While SIF products from airborne platforms are accurate and spatially well resolved, the data acquisition of such products remains science-oriented and limited to temporally constrained campaigns. Spaceborne SIF products on the other hand are available globally with often sufficient revisit times. However, the spatial resolution of spaceborne SIF products is too small for agricultural applications. In view of ESA's upcoming FLEX mission we develop a method for SIF retrieval in the O$_2$-A band of hyperspectral DESIS imagery to provide first insights for spaceborne SIF retrieval at high spatial resolution. To this end, we train a simulation-based self-supervised network with a novel perturbation based regularizer and test performance improvements under additional supervised regularization of atmospheric variable prediction. In a validation study with corresponding HyPlant derived SIF estimates at 740 nm we find that our model reaches a mean absolute difference of 0.78 mW / nm / sr / m$^2$.

Title: Structured Pattern Expansion with Diffusion Models

• Authors: Marzia Riso, Giuseppe Vecchio, Fabio Pellacini
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2411.08930
• Pdf URL: https://arxiv.org/pdf/2411.08930
• Copy Paste: [[2411.08930]] Structured Pattern Expansion with Diffusion Models(https://arxiv.org/abs/2411.08930)
• Keywords: diffusion, generative
• Abstract: Recent advances in diffusion models have significantly improved the synthesis of materials, textures, and 3D shapes. By conditioning these models via text or images, users can guide the generation, reducing the time required to create digital assets. In this paper, we address the synthesis of structured, stationary patterns, where diffusion models are generally less reliable and, more importantly, less controllable. Our approach leverages the generative capabilities of diffusion models specifically adapted for the pattern domain. It enables users to exercise direct control over the synthesis by expanding a partially hand-drawn pattern into a larger design while preserving the structure and details of the input. To enhance pattern quality, we fine-tune an image-pretrained diffusion model on structured patterns using Low-Rank Adaptation (LoRA), apply a noise rolling technique to ensure tileability, and utilize a patch-based approach to facilitate the generation of large-scale assets. We demonstrate the effectiveness of our method through a comprehensive set of experiments, showing that it outperforms existing models in generating diverse, consistent patterns that respond directly to user input.

Title: Inconsistencies In Consistency Models: Better ODE Solving Does Not Imply Better Samples

• Authors: Noël Vouitsis, Rasa Hosseinzadeh, Brendan Leigh Ross, Valentin Villecroze, Satya Krishna Gorti, Jesse C. Cresswell, Gabriel Loaiza-Ganem
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.08954
• Pdf URL: https://arxiv.org/pdf/2411.08954
• Copy Paste: [[2411.08954]] Inconsistencies In Consistency Models: Better ODE Solving Does Not Imply Better Samples(https://arxiv.org/abs/2411.08954)
• Keywords: diffusion
• Abstract: Although diffusion models can generate remarkably high-quality samples, they are intrinsically bottlenecked by their expensive iterative sampling procedure. Consistency models (CMs) have recently emerged as a promising diffusion model distillation method, reducing the cost of sampling by generating high-fidelity samples in just a few iterations. Consistency model distillation aims to solve the probability flow ordinary differential equation (ODE) defined by an existing diffusion model. CMs are not directly trained to minimize error against an ODE solver, rather they use a more computationally tractable objective. As a way to study how effectively CMs solve the probability flow ODE, and the effect that any induced error has on the quality of generated samples, we introduce Direct CMs, which \textit{directly} minimize this error. Intriguingly, we find that Direct CMs reduce the ODE solving error compared to CMs but also result in significantly worse sample quality, calling into question why exactly CMs work well in the first place. Full code is available at: this https URL.

Title: Anomaly Detection in Large-Scale Cloud Systems: An Industry Case and Dataset

• Authors: Mohammad Saiful Islam, Mohamed Sami Rakha, William Pourmajidi, Janakan Sivaloganathan, John Steinbacher, Andriy Miranskyy
• Subjects: cs.LG, cs.DC, cs.SE
• Abstract URL: https://arxiv.org/abs/2411.09047
• Pdf URL: https://arxiv.org/pdf/2411.09047
• Copy Paste: [[2411.09047]] Anomaly Detection in Large-Scale Cloud Systems: An Industry Case and Dataset(https://arxiv.org/abs/2411.09047)
• Keywords: anomaly
• Abstract: As Large-Scale Cloud Systems (LCS) become increasingly complex, effective anomaly detection is critical for ensuring system reliability and performance. However, there is a shortage of large-scale, real-world datasets available for benchmarking anomaly detection methods. To address this gap, we introduce a new high-dimensional dataset from IBM Cloud, collected over 4.5 months from the IBM Cloud Console. This dataset comprises 39,365 rows and 117,448 columns of telemetry data. Additionally, we demonstrate the application of machine learning models for anomaly detection and discuss the key challenges faced in this process. This study and the accompanying dataset provide a resource for researchers and practitioners in cloud system monitoring. It facilitates more efficient testing of anomaly detection methods in real-world data, helping to advance the development of robust solutions to maintain the health and performance of large-scale cloud infrastructures.

Title: Continuous GNN-based Anomaly Detection on Edge using Efficient Adaptive Knowledge Graph Learning

• Authors: Sanggeon Yun, Ryozo Masukawa, William Youngwoo Chung, Minhyoung Na, Nathaniel Bastian, Mohsen Imani
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09072
• Pdf URL: https://arxiv.org/pdf/2411.09072
• Copy Paste: [[2411.09072]] Continuous GNN-based Anomaly Detection on Edge using Efficient Adaptive Knowledge Graph Learning(https://arxiv.org/abs/2411.09072)
• Keywords: anomaly
• Abstract: The increasing demand for robust security solutions across various industries has made Video Anomaly Detection (VAD) a critical task in applications such as intelligent surveillance, evidence investigation, and violence detection. Traditional approaches to VAD often rely on finetuning large pre-trained models, which can be computationally expensive and impractical for real-time or resource-constrained environments. To address this, MissionGNN introduced a more efficient method by training a graph neural network (GNN) using a fixed knowledge graph (KG) derived from large language models (LLMs) like GPT-4. While this approach demonstrated significant efficiency in computational power and memory, it faces limitations in dynamic environments where frequent updates to the KG are necessary due to evolving behavior trends and shifting data patterns. These updates typically require cloud-based computation, posing challenges for edge computing applications. In this paper, we propose a novel framework that facilitates continuous KG adaptation directly on edge devices, overcoming the limitations of cloud dependency. Our method dynamically modifies the KG through a three-phase process: pruning, alternating, and creating nodes, enabling real-time adaptation to changing data trends. This continuous learning approach enhances the robustness of anomaly detection models, making them more suitable for deployment in dynamic and resource-constrained environments.

Title: Efficiently learning and sampling multimodal distributions with data-based initialization

• Authors: Frederic Koehler, Holden Lee, Thuy-Duong Vuong
• Subjects: cs.LG, cs.DS, math.PR, stat.ML
• Abstract URL: https://arxiv.org/abs/2411.09117
• Pdf URL: https://arxiv.org/pdf/2411.09117
• Copy Paste: [[2411.09117]] Efficiently learning and sampling multimodal distributions with data-based initialization(https://arxiv.org/abs/2411.09117)
• Keywords: diffusion
• Abstract: We consider the problem of sampling a multimodal distribution with a Markov chain given a small number of samples from the stationary measure. Although mixing can be arbitrarily slow, we show that if the Markov chain has a $k$th order spectral gap, initialization from a set of $\tilde O(k/\varepsilon^2)$ samples from the stationary distribution will, with high probability over the samples, efficiently generate a sample whose conditional law is $\varepsilon$-close in TV distance to the stationary measure. In particular, this applies to mixtures of $k$ distributions satisfying a Poincaré inequality, with faster convergence when they satisfy a log-Sobolev inequality. Our bounds are stable to perturbations to the Markov chain, and in particular work for Langevin diffusion over $\mathbb R^d$ with score estimation error, as well as Glauber dynamics combined with approximation error from pseudolikelihood estimation. This justifies the success of data-based initialization for score matching methods despite slow mixing for the data distribution, and improves and generalizes the results of Koehler and Vuong (2023) to have linear, rather than exponential, dependence on $k$ and apply to arbitrary semigroups. As a consequence of our results, we show for the first time that a natural class of low-complexity Ising measures can be efficiently learned from samples.

Title: Mono2Stereo: Monocular Knowledge Transfer for Enhanced Stereo Matching

• Authors: Yuran Wang, Yingping Liang, Hesong Li, Ying Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09151
• Pdf URL: https://arxiv.org/pdf/2411.09151
• Copy Paste: [[2411.09151]] Mono2Stereo: Monocular Knowledge Transfer for Enhanced Stereo Matching(https://arxiv.org/abs/2411.09151)
• Keywords: self-supervised
• Abstract: The generalization and performance of stereo matching networks are limited due to the domain gap of the existing synthetic datasets and the sparseness of GT labels in the real datasets. In contrast, monocular depth estimation has achieved significant advancements, benefiting from large-scale depth datasets and self-supervised strategies. To bridge the performance gap between monocular depth estimation and stereo matching, we propose leveraging monocular knowledge transfer to enhance stereo matching, namely Mono2Stereo. We introduce knowledge transfer with a two-stage training process, comprising synthetic data pre-training and real-world data fine-tuning. In the pre-training stage, we design a data generation pipeline that synthesizes stereo training data from monocular images. This pipeline utilizes monocular depth for warping and novel view synthesis and employs our proposed Edge-Aware (EA) inpainting module to fill in missing contents in the generated images. In the fine-tuning stage, we introduce a Sparse-to-Dense Knowledge Distillation (S2DKD) strategy encouraging the distributions of predictions to align with dense monocular depths. This strategy mitigates issues with edge blurring in sparse real-world labels and enhances overall consistency. Experimental results demonstrate that our pre-trained model exhibits strong zero-shot generalization capabilities. Furthermore, domain-specific fine-tuning using our pre-trained model and S2DKD strategy significantly increments in-domain performance. The code will be made available soon.

Title: VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation

• Authors: Youpeng Wen, Junfan Lin, Yi Zhu, Jianhua Han, Hang Xu, Shen Zhao, Xiaodan Liang
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2411.09153
• Pdf URL: https://arxiv.org/pdf/2411.09153
• Copy Paste: [[2411.09153]] VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation(https://arxiv.org/abs/2411.09153)
• Keywords: diffusion
• Abstract: Recent advancements utilizing large-scale video data for learning video generation models demonstrate significant potential in understanding complex physical dynamics. It suggests the feasibility of leveraging diverse robot trajectory data to develop a unified, dynamics-aware model to enhance robot manipulation. However, given the relatively small amount of available robot data, directly fitting data without considering the relationship between visual observations and actions could lead to suboptimal data utilization. To this end, we propose VidMan (Video Diffusion for Robot Manipulation), a novel framework that employs a two-stage training mechanism inspired by dual-process theory from neuroscience to enhance stability and improve data utilization efficiency. Specifically, in the first stage, VidMan is pre-trained on the Open X-Embodiment dataset (OXE) for predicting future visual trajectories in a video denoising diffusion manner, enabling the model to develop a long horizontal awareness of the environment's dynamics. In the second stage, a flexible yet effective layer-wise self-attention adapter is introduced to transform VidMan into an efficient inverse dynamics model that predicts action modulated by the implicit dynamics knowledge via parameter sharing. Our VidMan framework outperforms state-of-the-art baseline model GR-1 on the CALVIN benchmark, achieving a 11.7% relative improvement, and demonstrates over 9% precision gains on the OXE small-scale dataset. These results provide compelling evidence that world models can significantly enhance the precision of robot action prediction. Codes and models will be public.

Title: Unstructured Text Enhanced Open-domain Dialogue System: A Systematic Survey

• Authors: Longxuan Ma, Mingda Li, Weinan Zhang, Jiapeng Li, Ting Liu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2411.09166
• Pdf URL: https://arxiv.org/pdf/2411.09166
• Copy Paste: [[2411.09166]] Unstructured Text Enhanced Open-domain Dialogue System: A Systematic Survey(https://arxiv.org/abs/2411.09166)
• Keywords: generative
• Abstract: Incorporating external knowledge into dialogue generation has been proven to benefit the performance of an open-domain Dialogue System (DS), such as generating informative or stylized responses, controlling conversation topics. In this article, we study the open-domain DS that uses unstructured text as external knowledge sources (\textbf{U}nstructured \textbf{T}ext \textbf{E}nhanced \textbf{D}ialogue \textbf{S}ystem, \textbf{UTEDS}). The existence of unstructured text entails distinctions between UTEDS and traditional data-driven DS and we aim to analyze these differences. We first give the definition of the UTEDS related concepts, then summarize the recently released datasets and models. We categorize UTEDS into Retrieval and Generative models and introduce them from the perspective of model components. The retrieval models consist of Fusion, Matching, and Ranking modules, while the generative models comprise Dialogue and Knowledge Encoding, Knowledge Selection, and Response Generation modules. We further summarize the evaluation methods utilized in UTEDS and analyze the current models' performance. At last, we discuss the future development trends of UTEDS, hoping to inspire new research in this field.

Title: Advancing Diffusion Models: Alias-Free Resampling and Enhanced Rotational Equivariance

• Authors: Md Fahim Anjum
• Subjects: cs.CV, cs.AI, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2411.09174
• Pdf URL: https://arxiv.org/pdf/2411.09174
• Copy Paste: [[2411.09174]] Advancing Diffusion Models: Alias-Free Resampling and Enhanced Rotational Equivariance(https://arxiv.org/abs/2411.09174)
• Keywords: diffusion, generative
• Abstract: Recent advances in image generation, particularly via diffusion models, have led to impressive improvements in image synthesis quality. Despite this, diffusion models are still challenged by model-induced artifacts and limited stability in image fidelity. In this work, we hypothesize that the primary cause of this issue is the improper resampling operation that introduces aliasing in the diffusion model and a careful alias-free resampling dictated by image processing theory can improve the model's performance in image synthesis. We propose the integration of alias-free resampling layers into the UNet architecture of diffusion models without adding extra trainable parameters, thereby maintaining computational efficiency. We then assess whether these theory-driven modifications enhance image quality and rotational equivariance. Our experimental results on benchmark datasets, including CIFAR-10, MNIST, and MNIST-M, reveal consistent gains in image quality, particularly in terms of FID and KID scores. Furthermore, we propose a modified diffusion process that enables user-controlled rotation of generated images without requiring additional training. Our findings highlight the potential of theory-driven enhancements such as alias-free resampling in generative models to improve image quality while maintaining model efficiency and pioneer future research directions to incorporate them into video-generating diffusion models, enabling deeper exploration of the applications of alias-free resampling in generative modeling.

Title: JoyVASA: Portrait and Animal Image Animation with Diffusion-Based Audio-Driven Facial Dynamics and Head Motion Generation

• Authors: Xuyang Cao, Sheng Shi, Jun Zhao, Yang Yao, Jintao Fei, Minyu Gao, Guoxin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09209
• Pdf URL: https://arxiv.org/pdf/2411.09209
• Copy Paste: [[2411.09209]] JoyVASA: Portrait and Animal Image Animation with Diffusion-Based Audio-Driven Facial Dynamics and Head Motion Generation(https://arxiv.org/abs/2411.09209)
• Keywords: diffusion
• Abstract: Audio-driven portrait animation has made significant advances with diffusion-based models, improving video quality and lipsync accuracy. However, the increasing complexity of these models has led to inefficiencies in training and inference, as well as constraints on video length and inter-frame continuity. In this paper, we propose JoyVASA, a diffusion-based method for generating facial dynamics and head motion in audio-driven facial animation. Specifically, in the first stage, we introduce a decoupled facial representation framework that separates dynamic facial expressions from static 3D facial representations. This decoupling allows the system to generate longer videos by combining any static 3D facial representation with dynamic motion sequences. Then, in the second stage, a diffusion transformer is trained to generate motion sequences directly from audio cues, independent of character identity. Finally, a generator trained in the first stage uses the 3D facial representation and the generated motion sequences as inputs to render high-quality animations. With the decoupled facial representation and the identity-independent motion generation process, JoyVASA extends beyond human portraits to animate animal faces seamlessly. The model is trained on a hybrid dataset of private Chinese and public English data, enabling multilingual support. Experimental results validate the effectiveness of our approach. Future work will focus on improving real-time performance and refining expression control, further expanding the applications in portrait animation. The code will be available at: this https URL.

Title: Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentation

• Authors: Yuheng Shi, Minjing Dong, Chang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09219
• Pdf URL: https://arxiv.org/pdf/2411.09219
• Copy Paste: [[2411.09219]] Harnessing Vision Foundation Models for High-Performance, Training-Free Open Vocabulary Segmentation(https://arxiv.org/abs/2411.09219)
• Keywords: foundation model
• Abstract: While Contrastive Language-Image Pre-training (CLIP) has advanced open-vocabulary predictions, its performance on semantic segmentation remains suboptimal. This shortfall primarily stems from its spatial-invariant semantic features and constrained resolution. While previous adaptations addressed spatial invariance semantic by modifying the self-attention in CLIP's image encoder, the issue of limited resolution remains unexplored. Different from previous segment-then-splice methods that segment sub-images via a sliding window and splice the results, we introduce a splice-then-segment paradigm that incorporates Segment-Anything Model (SAM) to tackle the resolution issue since SAM excels at extracting fine-grained semantic correlations from high-resolution images. Specifically, we introduce Trident, a training-free framework that first splices features extracted by CLIP and DINO from sub-images, then leverages SAM's encoder to create a correlation matrix for global aggregation, enabling a broadened receptive field for effective segmentation. Besides, we propose a refinement strategy for CLIP's coarse segmentation outputs by transforming them into prompts for SAM, further enhancing the segmentation performance. Trident achieves a significant improvement in the mIoU across eight benchmarks compared with the current SOTA, increasing from 44.4 to this http URL is available at this https URL.

Title: Jailbreak Attacks and Defenses against Multimodal Generative Models: A Survey

• Authors: Xuannan Liu, Xing Cui, Peipei Li, Zekun Li, Huaibo Huang, Shuhan Xia, Miaoxuan Zhang, Yueying Zou, Ran He
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2411.09259
• Pdf URL: https://arxiv.org/pdf/2411.09259
• Copy Paste: [[2411.09259]] Jailbreak Attacks and Defenses against Multimodal Generative Models: A Survey(https://arxiv.org/abs/2411.09259)
• Keywords: foundation model, generative
• Abstract: The rapid evolution of multimodal foundation models has led to significant advancements in cross-modal understanding and generation across diverse modalities, including text, images, audio, and video. However, these models remain susceptible to jailbreak attacks, which can bypass built-in safety mechanisms and induce the production of potentially harmful content. Consequently, understanding the methods of jailbreak attacks and existing defense mechanisms is essential to ensure the safe deployment of multimodal generative models in real-world scenarios, particularly in security-sensitive applications. To provide comprehensive insight into this topic, this survey reviews jailbreak and defense in multimodal generative models. First, given the generalized lifecycle of multimodal jailbreak, we systematically explore attacks and corresponding defense strategies across four levels: input, encoder, generator, and output. Based on this analysis, we present a detailed taxonomy of attack methods, defense mechanisms, and evaluation frameworks specific to multimodal generative models. Additionally, we cover a wide range of input-output configurations, including modalities such as Any-to-Text, Any-to-Vision, and Any-to-Any within generative systems. Finally, we highlight current research challenges and propose potential directions for future this http URL open-source repository corresponding to this work can be found at this https URL.

Title: StreamAdapter: Efficient Test Time Adaptation from Contextual Streams

• Authors: Dilxat Muhtar, Yelong Shen, Yaming Yang, Xiaodong Liu, Yadong Lu, Jianfeng Liu, Yuefeng Zhan, Hao Sun, Weiwei Deng, Feng Sun, Xueliang Zhang, Jianfeng Gao, Weizhu Chen, Qi Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2411.09289
• Pdf URL: https://arxiv.org/pdf/2411.09289
• Copy Paste: [[2411.09289]] StreamAdapter: Efficient Test Time Adaptation from Contextual Streams(https://arxiv.org/abs/2411.09289)
• Keywords: in-context
• Abstract: In-context learning (ICL) allows large language models (LLMs) to adapt to new tasks directly from the given demonstrations without requiring gradient updates. While recent advances have expanded context windows to accommodate more demonstrations, this approach increases inference costs without necessarily improving performance. To mitigate these issues, We propose StreamAdapter, a novel approach that directly updates model parameters from context at test time, eliminating the need for explicit in-context demonstrations. StreamAdapter employs context mapping and weight absorption mechanisms to dynamically transform ICL demonstrations into parameter updates with minimal additional parameters. By reducing reliance on numerous in-context examples, StreamAdapter significantly reduce inference costs and allows for efficient inference with constant time complexity, regardless of demonstration count. Extensive experiments across diverse tasks and model architectures demonstrate that StreamAdapter achieves comparable or superior adaptation capability to ICL while requiring significantly fewer demonstrations. The superior task adaptation and context encoding capabilities of StreamAdapter on both language understanding and generation tasks provides a new perspective for adapting LLMs at test time using context, allowing for more efficient adaptation across scenarios and more cost-effective inference

Title: Exploring Zero-Shot Anomaly Detection with CLIP in Medical Imaging: Are We There Yet?

• Authors: Aldo Marzullo, Marta Bianca Maria Ranzini
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09310
• Pdf URL: https://arxiv.org/pdf/2411.09310
• Copy Paste: [[2411.09310]] Exploring Zero-Shot Anomaly Detection with CLIP in Medical Imaging: Are We There Yet?(https://arxiv.org/abs/2411.09310)
• Keywords: anomaly
• Abstract: Zero-shot anomaly detection (ZSAD) offers potential for identifying anomalies in medical imaging without task-specific training. In this paper, we evaluate CLIP-based models, originally developed for industrial tasks, on brain tumor detection using the BraTS-MET dataset. Our analysis examines their ability to detect medical-specific anomalies with no or minimal supervision, addressing the challenges posed by limited data annotation. While these models show promise in transferring general knowledge to medical tasks, their performance falls short of the precision required for clinical use. Our findings highlight the need for further adaptation before CLIP-based models can be reliably applied to medical anomaly detection.

Title: Approximate Probabilistic Inference forTime-Series Data A Robust Latent Gaussian Model With Temporal Awareness

• Authors: Anton Johansson, Arunselvan Ramaswamy
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09312
• Pdf URL: https://arxiv.org/pdf/2411.09312
• Copy Paste: [[2411.09312]] Approximate Probabilistic Inference forTime-Series Data A Robust Latent Gaussian Model With Temporal Awareness(https://arxiv.org/abs/2411.09312)
• Keywords: generative
• Abstract: The development of robust generative models for highly varied non-stationary time series data is a complex yet important problem. Traditional models for time series data prediction, such as Long Short-Term Memory (LSTM), are inefficient and generalize poorly as they cannot capture complex temporal relationships. In this paper, we present a probabilistic generative model that can be trained to capture temporal information, and that is robust to data errors. We call it Time Deep Latent Gaussian Model (tDLGM). Its novel architecture is inspired by Deep Latent Gaussian Model (DLGM). Our model is trained to minimize a loss function based on the negative log loss. One contributing factor to Time Deep Latent Gaussian Model (tDLGM) robustness is our regularizer, which accounts for data trends. Experiments conducted show that tDLGM is able to reconstruct and generate complex time series data, and that it is robust against to noise and faulty data.

Title: Time-to-Event Pretraining for 3D Medical Imaging

• Authors: Zepeng Huo, Jason Alan Fries, Alejandro Lozano, Jeya Maria Jose Valanarasu, Ethan Steinberg, Louis Blankemeier, Akshay S. Chaudhari, Curtis Langlotz, Nigam H. Shah
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09361
• Pdf URL: https://arxiv.org/pdf/2411.09361
• Copy Paste: [[2411.09361]] Time-to-Event Pretraining for 3D Medical Imaging(https://arxiv.org/abs/2411.09361)
• Keywords: self-supervised, foundation model
• Abstract: With the rise of medical foundation models and the growing availability of imaging data, scalable pretraining techniques offer a promising way to identify imaging biomarkers predictive of future disease risk. While current self-supervised methods for 3D medical imaging models capture local structural features like organ morphology, they fail to link pixel biomarkers with long-term health outcomes due to a missing context problem. Current approaches lack the temporal context necessary to identify biomarkers correlated with disease progression, as they rely on supervision derived only from images and concurrent text descriptions. To address this, we introduce time-to-event pretraining, a pretraining framework for 3D medical imaging models that leverages large-scale temporal supervision from paired, longitudinal electronic health records (EHRs). Using a dataset of 18,945 CT scans (4.2 million 2D images) and time-to-event distributions across thousands of EHR-derived tasks, our method improves outcome prediction, achieving an average AUROC increase of 23.7% and a 29.4% gain in Harrell's C-index across 8 benchmark tasks. Importantly, these gains are achieved without sacrificing diagnostic classification performance. This study lays the foundation for integrating longitudinal EHR and 3D imaging data to advance clinical risk prediction.

Title: A survey of probabilistic generative frameworks for molecular simulations

• Authors: Richard John, Lukas Herron, Pratyush Tiwary
• Subjects: cs.LG, cond-mat.dis-nn, cond-mat.soft, cond-mat.stat-mech
• Abstract URL: https://arxiv.org/abs/2411.09388
• Pdf URL: https://arxiv.org/pdf/2411.09388
• Copy Paste: [[2411.09388]] A survey of probabilistic generative frameworks for molecular simulations(https://arxiv.org/abs/2411.09388)
• Keywords: diffusion, generative
• Abstract: Generative artificial intelligence is now a widely used tool in molecular science. Despite the popularity of probabilistic generative models, numerical experiments benchmarking their performance on molecular data are lacking. In this work, we introduce and explain several classes of generative models, broadly sorted into two categories: flow-based models and diffusion models. We select three representative models: Neural Spline Flows, Conditional Flow Matching, and Denoising Diffusion Probabilistic Models, and examine their accuracy, computational cost, and generation speed across datasets with tunable dimensionality, complexity, and modal asymmetry. Our findings are varied, with no one framework being the best for all purposes. In a nutshell, (i) Neural Spline Flows do best at capturing mode asymmetry present in low-dimensional data, (ii) Conditional Flow Matching outperforms other models for high-dimensional data with low complexity, and (iii) Denoising Diffusion Probabilistic Models appears the best for low-dimensional data with high complexity. Our datasets include a Gaussian mixture model and the dihedral torsion angle distribution of the Aib\textsubscript{9} peptide, generated via a molecular dynamics simulation. We hope our taxonomy of probabilistic generative frameworks and numerical results may guide model selection for a wide range of molecular tasks.

Title: Mediffusion: Joint Diffusion for Self-Explainable Semi-Supervised Classification and Medical Image Generation

• Authors: Joanna Kaleta, Paweł Skierś, Jan Dubiński, Przemysław Korzeniowski, Kamil Deja
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09434
• Pdf URL: https://arxiv.org/pdf/2411.09434
• Copy Paste: [[2411.09434]] Mediffusion: Joint Diffusion for Self-Explainable Semi-Supervised Classification and Medical Image Generation(https://arxiv.org/abs/2411.09434)
• Keywords: diffusion, generative
• Abstract: We introduce Mediffusion -- a new method for semi-supervised learning with explainable classification based on a joint diffusion model. The medical imaging domain faces unique challenges due to scarce data labelling -- insufficient for standard training, and critical nature of the applications that require high performance, confidence, and explainability of the models. In this work, we propose to tackle those challenges with a single model that combines standard classification with a diffusion-based generative task in a single shared parametrisation. By sharing representations, our model effectively learns from both labeled and unlabeled data while at the same time providing accurate explanations through counterfactual examples. In our experiments, we show that our Mediffusion achieves results comparable to recent semi-supervised methods while providing more reliable and precise explanations.

Title: ReMP: Reusable Motion Prior for Multi-domain 3D Human Pose Estimation and Motion Inbetweening

• Authors: Hojun Jang, Young Min Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09435
• Pdf URL: https://arxiv.org/pdf/2411.09435
• Copy Paste: [[2411.09435]] ReMP: Reusable Motion Prior for Multi-domain 3D Human Pose Estimation and Motion Inbetweening(https://arxiv.org/abs/2411.09435)
• Keywords: foundation model
• Abstract: We present Reusable Motion prior (ReMP), an effective motion prior that can accurately track the temporal evolution of motion in various downstream tasks. Inspired by the success of foundation models, we argue that a robust spatio-temporal motion prior can encapsulate underlying 3D dynamics applicable to various sensor modalities. We learn the rich motion prior from a sequence of complete parametric models of posed human body shape. Our prior can easily estimate poses in missing frames or noisy measurements despite significant occlusion by employing a temporal attention mechanism. More interestingly, our prior can guide the system with incomplete and challenging input measurements to quickly extract critical information to estimate the sequence of poses, significantly improving the training efficiency for mesh sequence recovery. ReMP consistently outperforms the baseline method on diverse and practical 3D motion data, including depth point clouds, LiDAR scans, and IMU sensor data. Project page is available in this https URL.

Title: Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models

• Authors: Chutian Meng, Fan Ma, Jiaxu Miao, Chi Zhang, Yi Yang, Yueting Zhuang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09449
• Pdf URL: https://arxiv.org/pdf/2411.09449
• Copy Paste: [[2411.09449]] Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models(https://arxiv.org/abs/2411.09449)
• Keywords: diffusion, generative
• Abstract: Diffusion models have revitalized the image generation domain, playing crucial roles in both academic research and artistic expression. With the emergence of new diffusion models, assessing the performance of text-to-image models has become increasingly important. Current metrics focus on directly matching the input text with the generated image, but due to cross-modal information asymmetry, this leads to unreliable or incomplete assessment results. Motivated by this, we introduce the Image Regeneration task in this study to assess text-to-image models by tasking the T2I model with generating an image according to the reference image. We use GPT4V to bridge the gap between the reference image and the text input for the T2I model, allowing T2I models to understand image content. This evaluation process is simplified as comparisons between the generated image and the reference image are straightforward. Two regeneration datasets spanning content-diverse and style-diverse evaluation dataset are introduced to evaluate the leading diffusion models currently available. Additionally, we present ImageRepainter framework to enhance the quality of generated images by improving content comprehension via MLLM guided iterative generation and revision. Our comprehensive experiments have showcased the effectiveness of this framework in assessing the generative capabilities of models. By leveraging MLLM, we have demonstrated that a robust T2M can produce images more closely resembling the reference image.

Title: Golden Noise for Diffusion Models: A Learning Framework

• Authors: Zikai Zhou, Shitong Shao, Lichen Bai, Zhiqiang Xu, Bo Han, Zeke Xie
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09502
• Pdf URL: https://arxiv.org/pdf/2411.09502
• Copy Paste: [[2411.09502]] Golden Noise for Diffusion Models: A Learning Framework(https://arxiv.org/abs/2411.09502)
• Keywords: diffusion
• Abstract: Text-to-image diffusion model is a popular paradigm that synthesizes personalized images by providing a text prompt and a random Gaussian noise. While people observe that some noises are ``golden noises'' that can achieve better text-image alignment and higher human preference than others, we still lack a machine learning framework to obtain those golden noises. To learn golden noises for diffusion sampling, we mainly make three contributions in this paper. First, we identify a new concept termed the \textit{noise prompt}, which aims at turning a random Gaussian noise into a golden noise by adding a small desirable perturbation derived from the text prompt. Following the concept, we first formulate the \textit{noise prompt learning} framework that systematically learns ``prompted'' golden noise associated with a text prompt for diffusion models. Second, we design a noise prompt data collection pipeline and collect a large-scale \textit{noise prompt dataset}~(NPD) that contains 100k pairs of random noises and golden noises with the associated text prompts. With the prepared NPD as the training dataset, we trained a small \textit{noise prompt network}~(NPNet) that can directly learn to transform a random noise into a golden noise. The learned golden noise perturbation can be considered as a kind of prompt for noise, as it is rich in semantic information and tailored to the given text prompt. Third, our extensive experiments demonstrate the impressive effectiveness and generalization of NPNet on improving the quality of synthesized images across various diffusion models, including SDXL, DreamShaper-xl-v2-turbo, and Hunyuan-DiT. Moreover, NPNet is a small and efficient controller that acts as a plug-and-play module with very limited additional inference and computational costs, as it just provides a golden noise instead of a random noise without accessing the original pipeline.

Title: Adaptive Deviation Learning for Visual Anomaly Detection with Data Contamination

• Authors: Anindya Sundar Das, Guansong Pang, Monowar Bhuyan
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2411.09558
• Pdf URL: https://arxiv.org/pdf/2411.09558
• Copy Paste: [[2411.09558]] Adaptive Deviation Learning for Visual Anomaly Detection with Data Contamination(https://arxiv.org/abs/2411.09558)
• Keywords: anomaly
• Abstract: Visual anomaly detection targets to detect images that notably differ from normal pattern, and it has found extensive application in identifying defective parts within the manufacturing industry. These anomaly detection paradigms predominantly focus on training detection models using only clean, unlabeled normal samples, assuming an absence of contamination; a condition often unmet in real-world scenarios. The performance of these methods significantly depends on the quality of the data and usually decreases when exposed to noise. We introduce a systematic adaptive method that employs deviation learning to compute anomaly scores end-to-end while addressing data contamination by assigning relative importance to the weights of individual instances. In this approach, the anomaly scores for normal instances are designed to approximate scalar scores obtained from the known prior distribution. Meanwhile, anomaly scores for anomaly examples are adjusted to exhibit statistically significant deviations from these reference scores. Our approach incorporates a constrained optimization problem within the deviation learning framework to update instance weights, resolving this problem for each mini-batch. Comprehensive experiments on the MVTec and VisA benchmark datasets indicate that our proposed method surpasses competing techniques and exhibits both stability and robustness in the presence of data contamination.

Title: MagicQuill: An Intelligent Interactive Image Editing System

• Authors: Zichen Liu, Yue Yu, Hao Ouyang, Qiuyu Wang, Ka Leong Cheng, Wen Wang, Zhiheng Liu, Qifeng Chen, Yujun Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2411.09703
• Pdf URL: https://arxiv.org/pdf/2411.09703
• Copy Paste: [[2411.09703]] MagicQuill: An Intelligent Interactive Image Editing System(https://arxiv.org/abs/2411.09703)
• Keywords: diffusion
• Abstract: Image editing involves a variety of complex tasks and requires efficient and precise manipulation techniques. In this paper, we present MagicQuill, an integrated image editing system that enables swift actualization of creative ideas. Our system features a streamlined yet functionally robust interface, allowing for the articulation of editing operations (e.g., inserting elements, erasing objects, altering color) with minimal input. These interactions are monitored by a multimodal large language model (MLLM) to anticipate editing intentions in real time, bypassing the need for explicit prompt entry. Finally, we apply a powerful diffusion prior, enhanced by a carefully learned two-branch plug-in module, to process editing requests with precise control. Experimental results demonstrate the effectiveness of MagicQuill in achieving high-quality image edits. Please visit this https URL to try out our system.