2024-09-10

Title: Thinking Outside the BBox: Unconstrained Generative Object Compositing

• Authors: Gemma Canet Tarrés, Zhe Lin, Zhifei Zhang, Jianming Zhang, Yizhi Song, Dan Ruta, Andrew Gilbert, John Collomosse, Soo Ye Kim
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04559
• Pdf URL: https://arxiv.org/pdf/2409.04559
• Copy Paste: [[2409.04559]] Thinking Outside the BBox: Unconstrained Generative Object Compositing(https://arxiv.org/abs/2409.04559)
• Keywords: diffusion, generative
• Abstract: Compositing an object into an image involves multiple non-trivial sub-tasks such as object placement and scaling, color/lighting harmonization, viewpoint/geometry adjustment, and shadow/reflection generation. Recent generative image compositing methods leverage diffusion models to handle multiple sub-tasks at once. However, existing models face limitations due to their reliance on masking the original object during training, which constrains their generation to the input mask. Furthermore, obtaining an accurate input mask specifying the location and scale of the object in a new image can be highly challenging. To overcome such limitations, we define a novel problem of unconstrained generative object compositing, i.e., the generation is not bounded by the mask, and train a diffusion-based model on a synthesized paired dataset. Our first-of-its-kind model is able to generate object effects such as shadows and reflections that go beyond the mask, enhancing image realism. Additionally, if an empty mask is provided, our model automatically places the object in diverse natural locations and scales, accelerating the compositing workflow. Our model outperforms existing object placement and compositing models in various quality metrics and user studies.

Title: Multi-Modal Diffusion for Hand-Object Grasp Generation

• Authors: Jinkun Cao, Jingyuan Liu, Kris Kitani, Yi Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04560
• Pdf URL: https://arxiv.org/pdf/2409.04560
• Copy Paste: [[2409.04560]] Multi-Modal Diffusion for Hand-Object Grasp Generation(https://arxiv.org/abs/2409.04560)
• Keywords: diffusion
• Abstract: In this work, we focus on generating hand grasp over objects. Compared to previous works of generating hand poses with a given object, we aim to allow the generalization of both hand and object shapes by a single model. Our proposed method Multi-modal Grasp Diffusion (MGD) learns the prior and conditional posterior distribution of both modalities from heterogeneous data sources. Therefore it relieves the limitation of hand-object grasp datasets by leveraging the large-scale 3D object datasets. According to both qualitative and quantitative experiments, both conditional and unconditional generation of hand grasp achieve good visual plausibility and diversity. The proposed method also generalizes well to unseen object shapes. The code and weights will be available at \url{this https URL}.

Title: A Novel Dataset for Video-Based Autism Classification Leveraging Extra-Stimulatory Behavior

• Authors: Manuel Serna-Aguilera, Xuan Bac Nguyen, Han-Seok Seo, Khoa Luu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04598
• Pdf URL: https://arxiv.org/pdf/2409.04598
• Copy Paste: [[2409.04598]] A Novel Dataset for Video-Based Autism Classification Leveraging Extra-Stimulatory Behavior(https://arxiv.org/abs/2409.04598)
• Keywords: foundation model
• Abstract: Autism Spectrum Disorder (ASD) can affect individuals at varying degrees of intensity, from challenges in overall health, communication, and sensory processing, and this often begins at a young age. Thus, it is critical for medical professionals to be able to accurately diagnose ASD in young children, but doing so is difficult. Deep learning can be responsibly leveraged to improve productivity in addressing this task. The availability of data, however, remains a considerable obstacle. Hence, in this work, we introduce the Video ASD dataset--a dataset that contains video frame convolutional and attention map feature data--to foster further progress in the task of ASD classification. The original videos showcase children reacting to chemo-sensory stimuli, among auditory, touch, and vision This dataset contains the features of the frames spanning 2,467 videos, for a total of approximately 1.4 million frames. Additionally, head pose angles are included to account for head movement noise, as well as full-sentence text labels for the taste and smell videos that describe how the facial expression changes before, immediately after, and long after interaction with the stimuli. In addition to providing features, we also test foundation models on this data to showcase how movement noise affects performance and the need for more data and more complex labels.

Title: Self-Supervised Contrastive Learning for Videos using Differentiable Local Alignment

• Authors: Keyne Oei, Amr Gomaa, Anna Maria Feit, João Belo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04607
• Pdf URL: https://arxiv.org/pdf/2409.04607
• Copy Paste: [[2409.04607]] Self-Supervised Contrastive Learning for Videos using Differentiable Local Alignment(https://arxiv.org/abs/2409.04607)
• Keywords: self-supervised
• Abstract: Robust frame-wise embeddings are essential to perform video analysis and understanding tasks. We present a self-supervised method for representation learning based on aligning temporal video sequences. Our framework uses a transformer-based encoder to extract frame-level features and leverages them to find the optimal alignment path between video sequences. We introduce the novel Local-Alignment Contrastive (LAC) loss, which combines a differentiable local alignment loss to capture local temporal dependencies with a contrastive loss to enhance discriminative learning. Prior works on video alignment have focused on using global temporal ordering across sequence pairs, whereas our loss encourages identifying the best-scoring subsequence alignment. LAC uses the differentiable Smith-Waterman (SW) affine method, which features a flexible parameterization learned through the training phase, enabling the model to adjust the temporal gap penalty length dynamically. Evaluations show that our learned representations outperform existing state-of-the-art approaches on action recognition tasks.

Title: Multi-Conditioned Denoising Diffusion Probabilistic Model (mDDPM) for Medical Image Synthesis

• Authors: Arjun Krishna, Ge Wang, Klaus Mueller
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04670
• Pdf URL: https://arxiv.org/pdf/2409.04670
• Copy Paste: [[2409.04670]] Multi-Conditioned Denoising Diffusion Probabilistic Model (mDDPM) for Medical Image Synthesis(https://arxiv.org/abs/2409.04670)
• Keywords: diffusion, generative
• Abstract: Medical imaging applications are highly specialized in terms of human anatomy, pathology, and imaging domains. Therefore, annotated training datasets for training deep learning applications in medical imaging not only need to be highly accurate but also diverse and large enough to encompass almost all plausible examples with respect to those specifications. We argue that achieving this goal can be facilitated through a controlled generation framework for synthetic images with annotations, requiring multiple conditional specifications as input to provide control. We employ a Denoising Diffusion Probabilistic Model (DDPM) to train a large-scale generative model in the lung CT domain and expand upon a classifier-free sampling strategy to showcase one such generation framework. We show that our approach can produce annotated lung CT images that can faithfully represent anatomy, convincingly fooling experts into perceiving them as real. Our experiments demonstrate that controlled generative frameworks of this nature can surpass nearly every state-of-the-art image generative model in achieving anatomical consistency in generated medical images when trained on comparable large medical datasets.

Title: A Multi-scenario Attention-based Generative Model for Personalized Blood Pressure Time Series Forecasting

• Authors: Cheng Wan, Chenjie Xie, Longfei Liu, Dan Wu, Ye Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04704
• Pdf URL: https://arxiv.org/pdf/2409.04704
• Copy Paste: [[2409.04704]] A Multi-scenario Attention-based Generative Model for Personalized Blood Pressure Time Series Forecasting(https://arxiv.org/abs/2409.04704)
• Keywords: generative
• Abstract: Continuous blood pressure (BP) monitoring is essential for timely diagnosis and intervention in critical care settings. However, BP varies significantly across individuals, this inter-patient variability motivates the development of personalized models tailored to each patient's physiology. In this work, we propose a personalized BP forecasting model mainly using electrocardiogram (ECG) and photoplethysmogram (PPG) signals. This time-series model incorporates 2D representation learning to capture complex physiological relationships. Experiments are conducted on datasets collected from three diverse scenarios with BP measurements from 60 subjects total. Results demonstrate that the model achieves accurate and robust BP forecasts across scenarios within the Association for the Advancement of Medical Instrumentation (AAMI) standard criteria. This reliable early detection of abnormal fluctuations in BP is crucial for at-risk patients undergoing surgery or intensive care. The proposed model provides a valuable addition for continuous BP tracking to reduce mortality and improve prognosis.

Title: VidLPRO: A $\underline{Vid}$eo-$\underline{L}$anguage $\underline{P}$re-training Framework for $\underline{Ro}$botic and Laparoscopic Surgery

• Authors: Mohammadmahdi Honarmand, Muhammad Abdullah Jamal, Omid Mohareri
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.04732
• Pdf URL: https://arxiv.org/pdf/2409.04732
• Copy Paste: [[2409.04732]] VidLPRO: A $\underline{Vid}$eo-$\underline{L}$anguage $\underline{P}$re-training Framework for $\underline{Ro}$botic and Laparoscopic Surgery(https://arxiv.org/abs/2409.04732)
• Keywords: foundation model
• Abstract: We introduce VidLPRO, a novel video-language (VL) pre-training framework designed specifically for robotic and laparoscopic surgery. While existing surgical VL models primarily rely on contrastive learning, we propose a more comprehensive approach to capture the intricate temporal dynamics and align video with language. VidLPRO integrates video-text contrastive learning, video-text matching, and masked language modeling objectives to learn rich VL representations. To support this framework, we present GenSurg+, a carefully curated dataset derived from GenSurgery, comprising 17k surgical video clips paired with captions generated by GPT-4 using transcripts extracted by the Whisper model. This dataset addresses the need for large-scale, high-quality VL data in the surgical domain. Extensive experiments on benchmark datasets, including Cholec80 and AutoLaparo, demonstrate the efficacy of our approach. VidLPRO achieves state-of-the-art performance in zero-shot surgical phase recognition, significantly outperforming existing surgical VL models such as SurgVLP and HecVL. Our model demonstrates improvements of up to 21.5\% in accuracy and 15.7% in F1 score, setting a new benchmark in the field. Notably, VidLPRO exhibits robust performance even with single-frame inference, while effectively scaling with increased temporal context. Ablation studies reveal the impact of frame sampling strategies on model performance and computational efficiency. These results underscore VidLPRO's potential as a foundation model for surgical video understanding.

Title: Explicit Mutual Information Maximization for Self-Supervised Learning

• Authors: Lele Chang, Peilin Liu, Qinghai Guo, Fei Wen
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.04747
• Pdf URL: https://arxiv.org/pdf/2409.04747
• Copy Paste: [[2409.04747]] Explicit Mutual Information Maximization for Self-Supervised Learning(https://arxiv.org/abs/2409.04747)
• Keywords: self-supervised
• Abstract: Recently, self-supervised learning (SSL) has been extensively studied. Theoretically, mutual information maximization (MIM) is an optimal criterion for SSL, with a strong theoretical foundation in information theory. However, it is difficult to directly apply MIM in SSL since the data distribution is not analytically available in applications. In practice, many existing methods can be viewed as approximate implementations of the MIM criterion. This work shows that, based on the invariance property of MI, explicit MI maximization can be applied to SSL under a generic distribution assumption, i.e., a relaxed condition of the data distribution. We further illustrate this by analyzing the generalized Gaussian distribution. Based on this result, we derive a loss function based on the MIM criterion using only second-order statistics. We implement the new loss for SSL and demonstrate its effectiveness via extensive experiments.

Title: Training-Free Style Consistent Image Synthesis with Condition and Mask Guidance in E-Commerce

• Authors: Guandong Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04750
• Pdf URL: https://arxiv.org/pdf/2409.04750
• Copy Paste: [[2409.04750]] Training-Free Style Consistent Image Synthesis with Condition and Mask Guidance in E-Commerce(https://arxiv.org/abs/2409.04750)
• Keywords: diffusion
• Abstract: Generating style-consistent images is a common task in the e-commerce field, and current methods are largely based on diffusion models, which have achieved excellent results. This paper introduces the concept of the QKV (query/key/value) level, referring to modifications in the attention maps (self-attention and cross-attention) when integrating UNet with image conditions. Without disrupting the product's main composition in e-commerce images, we aim to use a train-free method guided by pre-set conditions. This involves using shared KV to enhance similarity in cross-attention and generating mask guidance from the attention map to cleverly direct the generation of style-consistent images. Our method has shown promising results in practical applications.

Title: SpotActor: Training-Free Layout-Controlled Consistent Image Generation

• Authors: Jiahao Wang, Caixia Yan, Weizhan Zhang, Haonan Lin, Mengmeng Wang, Guang Dai, Tieliang Gong, Hao Sun, Jingdong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04801
• Pdf URL: https://arxiv.org/pdf/2409.04801
• Copy Paste: [[2409.04801]] SpotActor: Training-Free Layout-Controlled Consistent Image Generation(https://arxiv.org/abs/2409.04801)
• Keywords: diffusion
• Abstract: Text-to-image diffusion models significantly enhance the efficiency of artistic creation with high-fidelity image generation. However, in typical application scenarios like comic book production, they can neither place each subject into its expected spot nor maintain the consistent appearance of each subject across images. For these issues, we pioneer a novel task, Layout-to-Consistent-Image (L2CI) generation, which produces consistent and compositional images in accordance with the given layout conditions and text prompts. To accomplish this challenging task, we present a new formalization of dual energy guidance with optimization in a dual semantic-latent space and thus propose a training-free pipeline, SpotActor, which features a layout-conditioned backward update stage and a consistent forward sampling stage. In the backward stage, we innovate a nuanced layout energy function to mimic the attention activations with a sigmoid-like objective. While in the forward stage, we design Regional Interconnection Self-Attention (RISA) and Semantic Fusion Cross-Attention (SFCA) mechanisms that allow mutual interactions across images. To evaluate the performance, we present ActorBench, a specified benchmark with hundreds of reasonable prompt-box pairs stemming from object detection datasets. Comprehensive experiments are conducted to demonstrate the effectiveness of our method. The results prove that SpotActor fulfills the expectations of this task and showcases the potential for practical applications with superior layout alignment, subject consistency, prompt conformity and background diversity.

Title: Reward-Directed Score-Based Diffusion Models via q-Learning

• Authors: Xuefeng Gao, Jiale Zha, Xun Yu Zhou
• Subjects: cs.LG, cs.AI, math.OC
• Abstract URL: https://arxiv.org/abs/2409.04832
• Pdf URL: https://arxiv.org/pdf/2409.04832
• Copy Paste: [[2409.04832]] Reward-Directed Score-Based Diffusion Models via q-Learning(https://arxiv.org/abs/2409.04832)
• Keywords: diffusion, generative
• Abstract: We propose a new reinforcement learning (RL) formulation for training continuous-time score-based diffusion models for generative AI to generate samples that maximize reward functions while keeping the generated distributions close to the unknown target data distributions. Different from most existing studies, our formulation does not involve any pretrained model for the unknown score functions of the noise-perturbed data distributions. We present an entropy-regularized continuous-time RL problem and show that the optimal stochastic policy has a Gaussian distribution with a known covariance matrix. Based on this result, we parameterize the mean of Gaussian policies and develop an actor-critic type (little) q-learning algorithm to solve the RL problem. A key ingredient in our algorithm design is to obtain noisy observations from the unknown score function via a ratio estimator. Numerically, we show the effectiveness of our approach by comparing its performance with two state-of-the-art RL methods that fine-tune pretrained models. Finally, we discuss extensions of our RL formulation to probability flow ODE implementation of diffusion models and to conditional diffusion models.

Title: Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation

• Authors: Jiaxin Cheng, Zixu Zhao, Tong He, Tianjun Xiao, Yicong Zhou, Zheng Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04847
• Pdf URL: https://arxiv.org/pdf/2409.04847
• Copy Paste: [[2409.04847]] Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation(https://arxiv.org/abs/2409.04847)
• Keywords: generative
• Abstract: Recent advancements in generative models have significantly enhanced their capacity for image generation, enabling a wide range of applications such as image editing, completion and video editing. A specialized area within generative modeling is layout-to-image (L2I) generation, where predefined layouts of objects guide the generative process. In this study, we introduce a novel regional cross-attention module tailored to enrich layout-to-image generation. This module notably improves the representation of layout regions, particularly in scenarios where existing methods struggle with highly complex and detailed textual descriptions. Moreover, while current open-vocabulary L2I methods are trained in an open-set setting, their evaluations often occur in closed-set environments. To bridge this gap, we propose two metrics to assess L2I performance in open-vocabulary scenarios. Additionally, we conduct a comprehensive user study to validate the consistency of these metrics with human preferences.

Title: Plug-and-Hide: Provable and Adjustable Diffusion Generative Steganography

• Authors: Jiahao Zhu, Zixuan Chen, Lingxiao Yang, Xiaohua Xie, Yi Zhou
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.04878
• Pdf URL: https://arxiv.org/pdf/2409.04878
• Copy Paste: [[2409.04878]] Plug-and-Hide: Provable and Adjustable Diffusion Generative Steganography(https://arxiv.org/abs/2409.04878)
• Keywords: diffusion, generative
• Abstract: Generative Steganography (GS) is a novel technique that utilizes generative models to conceal messages without relying on cover images. Contemporary GS algorithms leverage the powerful generative capabilities of Diffusion Models (DMs) to create high-fidelity stego images. However, these algorithms, while yielding relatively satisfactory generation outcomes and message extraction accuracy, significantly alter modifications to the initial Gaussian noise of DMs, thereby compromising steganographic security. In this paper, we rethink the trade-off among image quality, steganographic security, and message extraction accuracy within Diffusion Generative Steganography (DGS) settings. Our findings reveal that the normality of initial noise of DMs is crucial to these factors and can offer theoretically grounded guidance for DGS design. Based on this insight, we propose a Provable and Adjustable Message Mapping (PA-B2G) approach. It can, on one hand, theoretically guarantee reversible encoding of bit messages from arbitrary distributions into standard Gaussian noise for DMs. On the other hand, its adjustability provides a more natural and fine-grained way to trade off image quality, steganographic security, and message extraction accuracy. By integrating PA-B2G with a probability flow ordinary differential equation, we establish an invertible mapping between secret messages and stego images. PA-B2G can be seamlessly incorporated with most mainstream DMs, such as the Stable Diffusion, without necessitating additional training or fine-tuning. Comprehensive experiments corroborate our theoretical insights regarding the trade-off in DGS settings and demonstrate the effectiveness of our DGS algorithm in producing high-quality stego images while preserving desired levels of steganographic security and extraction accuracy.

Title: GS-PT: Exploiting 3D Gaussian Splatting for Comprehensive Point Cloud Understanding via Self-supervised Learning

• Authors: Keyi Liu, Yeqi Luo, Weidong Yang, Jingyi Xu, Zhijun Li, Wen-Ming Chen, Ben Fei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.04963
• Pdf URL: https://arxiv.org/pdf/2409.04963
• Copy Paste: [[2409.04963]] GS-PT: Exploiting 3D Gaussian Splatting for Comprehensive Point Cloud Understanding via Self-supervised Learning(https://arxiv.org/abs/2409.04963)
• Keywords: self-supervised
• Abstract: Self-supervised learning of point cloud aims to leverage unlabeled 3D data to learn meaningful representations without reliance on manual annotations. However, current approaches face challenges such as limited data diversity and inadequate augmentation for effective feature learning. To address these challenges, we propose GS-PT, which integrates 3D Gaussian Splatting (3DGS) into point cloud self-supervised learning for the first time. Our pipeline utilizes transformers as the backbone for self-supervised pre-training and introduces novel contrastive learning tasks through 3DGS. Specifically, the transformers aim to reconstruct the masked point cloud. 3DGS utilizes multi-view rendered images as input to generate enhanced point cloud distributions and novel view images, facilitating data augmentation and cross-modal contrastive learning. Additionally, we incorporate features from depth maps. By optimizing these tasks collectively, our method enriches the tri-modal self-supervised learning process, enabling the model to leverage the correlation across 3D point clouds and 2D images from various modalities. We freeze the encoder after pre-training and test the model's performance on multiple downstream tasks. Experimental results indicate that GS-PT outperforms the off-the-shelf self-supervised learning methods on various downstream tasks including 3D object classification, real-world classifications, and few-shot learning and segmentation.

Title: 2DSig-Detect: a semi-supervised framework for anomaly detection on image data using 2D-signatures

• Authors: Xinheng Xie, Kureha Yamaguchi, Margaux Leblanc, Simon Malzard, Varun Chhabra, Victoria Nockles, Yue Wu
• Subjects: cs.CV, math.PR, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.04982
• Pdf URL: https://arxiv.org/pdf/2409.04982
• Copy Paste: [[2409.04982]] 2DSig-Detect: a semi-supervised framework for anomaly detection on image data using 2D-signatures(https://arxiv.org/abs/2409.04982)
• Keywords: anomaly
• Abstract: The rapid advancement of machine learning technologies raises questions about the security of machine learning models, with respect to both training-time (poisoning) and test-time (evasion, impersonation, and inversion) attacks. Models performing image-related tasks, e.g. detection, and classification, are vulnerable to adversarial attacks that can degrade their performance and produce undesirable outcomes. This paper introduces a novel technique for anomaly detection in images called 2DSig-Detect, which uses a 2D-signature-embedded semi-supervised framework rooted in rough path theory. We demonstrate our method in adversarial settings for training-time and test-time attacks, and benchmark our framework against other state of the art methods. Using 2DSig-Detect for anomaly detection, we show both superior performance and a reduction in the computation time to detect the presence of adversarial perturbations in images.

Title: DreamMapping: High-Fidelity Text-to-3D Generation via Variational Distribution Mapping

• Authors: Zeyu Cai, Duotun Wang, Yixun Liang, Zhijing Shao, Ying-Cong Chen, Xiaohang Zhan, Zeyu Wang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2409.05099
• Pdf URL: https://arxiv.org/pdf/2409.05099
• Copy Paste: [[2409.05099]] DreamMapping: High-Fidelity Text-to-3D Generation via Variational Distribution Mapping(https://arxiv.org/abs/2409.05099)
• Keywords: diffusion
• Abstract: Score Distillation Sampling (SDS) has emerged as a prevalent technique for text-to-3D generation, enabling 3D content creation by distilling view-dependent information from text-to-2D guidance. However, they frequently exhibit shortcomings such as over-saturated color and excess smoothness. In this paper, we conduct a thorough analysis of SDS and refine its formulation, finding that the core design is to model the distribution of rendered images. Following this insight, we introduce a novel strategy called Variational Distribution Mapping (VDM), which expedites the distribution modeling process by regarding the rendered images as instances of degradation from diffusion-based generation. This special design enables the efficient training of variational distribution by skipping the calculations of the Jacobians in the diffusion U-Net. We also introduce timestep-dependent Distribution Coefficient Annealing (DCA) to further improve distilling precision. Leveraging VDM and DCA, we use Gaussian Splatting as the 3D representation and build a text-to-3D generation framework. Extensive experiments and evaluations demonstrate the capability of VDM and DCA to generate high-fidelity and realistic assets with optimization efficiency.

Title: WaterSeeker: Efficient Detection of Watermarked Segments in Large Documents

• Authors: Leyi Pan, Aiwei Liu, Yijian Lu, Zitian Gao, Yichen Di, Lijie Wen, Irwin King, Philip S. Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05112
• Pdf URL: https://arxiv.org/pdf/2409.05112
• Copy Paste: [[2409.05112]] WaterSeeker: Efficient Detection of Watermarked Segments in Large Documents(https://arxiv.org/abs/2409.05112)
• Keywords: anomaly
• Abstract: Watermarking algorithms for large language models (LLMs) have attained high accuracy in detecting LLM-generated text. However, existing methods primarily focus on distinguishing fully watermarked text from non-watermarked text, overlooking real-world scenarios where LLMs generate only small sections within large documents. In this scenario, balancing time complexity and detection performance poses significant challenges. This paper presents WaterSeeker, a novel approach to efficiently detect and locate watermarked segments amid extensive natural text. It first applies an efficient anomaly extraction method to preliminarily locate suspicious watermarked regions. Following this, it conducts a local traversal and performs full-text detection for more precise verification. Theoretical analysis and experimental results demonstrate that WaterSeeker achieves a superior balance between detection accuracy and computational efficiency. Moreover, WaterSeeker's localization ability supports the development of interpretable AI detection systems. This work pioneers a new direction in watermarked segment detection, facilitating more reliable AI-generated content identification.

Title: TanDepth: Leveraging Global DEMs for Metric Monocular Depth Estimation in UAVs

• Authors: Horatiu Florea, Sergiu Nedevschi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05142
• Pdf URL: https://arxiv.org/pdf/2409.05142
• Copy Paste: [[2409.05142]] TanDepth: Leveraging Global DEMs for Metric Monocular Depth Estimation in UAVs(https://arxiv.org/abs/2409.05142)
• Keywords: self-supervised
• Abstract: Aerial scene understanding systems face stringent payload restrictions and must often rely on monocular depth estimation for modelling scene geometry, which is an inherently ill-posed problem. Moreover, obtaining accurate ground truth data required by learning-based methods raises significant additional challenges in the aerial domain. Self-supervised approaches can bypass this problem, at the cost of providing only up-to-scale results. Similarly, recent supervised solutions which make good progress towards zero-shot generalization also provide only relative depth values. This work presents TanDepth, a practical, online scale recovery method for obtaining metric depth results from relative estimations at inference-time, irrespective of the type of model generating them. Tailored for Unmanned Aerial Vehicle (UAV) applications, our method leverages sparse measurements from Global Digital Elevation Models (GDEM) by projecting them to the camera view using extrinsic and intrinsic information. An adaptation to the Cloth Simulation Filter is presented, which allows selecting ground points from the estimated depth map to then correlate with the projected reference points. We evaluate and compare our method against alternate scaling methods adapted for UAVs, on a variety of real-world scenes. Considering the limited availability of data for this domain, we construct and release a comprehensive, depth-focused extension to the popular UAVid dataset to further research.

Title: OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMs

• Authors: Jintian Zhang, Cheng Peng, Mengshu Sun, Xiang Chen, Lei Liang, Zhiqiang Zhang, Jun Zhou, Huajun Chen, Ningyu Zhang
• Subjects: cs.CL, cs.AI, cs.DB, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05152
• Pdf URL: https://arxiv.org/pdf/2409.05152
• Copy Paste: [[2409.05152]] OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMs(https://arxiv.org/abs/2409.05152)
• Keywords: generative
• Abstract: Despite the recent advancements in Large Language Models (LLMs), which have significantly enhanced the generative capabilities for various NLP tasks, LLMs still face limitations in directly handling retrieval tasks. However, many practical applications demand the seamless integration of both retrieval and generation. This paper introduces a novel and efficient One-pass Generation and retrieval framework (OneGen), designed to improve LLMs' performance on tasks that require both generation and retrieval. The proposed framework bridges the traditionally separate training approaches for generation and retrieval by incorporating retrieval tokens generated autoregressively. This enables a single LLM to handle both tasks simultaneously in a unified forward pass. We conduct experiments on two distinct types of composite tasks, RAG and Entity Linking, to validate the pluggability, effectiveness, and efficiency of OneGen in training and inference. Furthermore, our results show that integrating generation and retrieval within the same context preserves the generative capabilities of LLMs while improving retrieval performance. To the best of our knowledge, OneGen is the first to enable LLMs to conduct vector retrieval during the generation.

Title: Can OOD Object Detectors Learn from Foundation Models?

• Authors: Jiahui Liu, Xin Wen, Shizhen Zhao, Yingxian Chen, Xiaojuan Qi
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05162
• Pdf URL: https://arxiv.org/pdf/2409.05162
• Copy Paste: [[2409.05162]] Can OOD Object Detectors Learn from Foundation Models?(https://arxiv.org/abs/2409.05162)
• Keywords: diffusion, foundation model, generative
• Abstract: Out-of-distribution (OOD) object detection is a challenging task due to the absence of open-set OOD data. Inspired by recent advancements in text-to-image generative models, such as Stable Diffusion, we study the potential of generative models trained on large-scale open-set data to synthesize OOD samples, thereby enhancing OOD object detection. We introduce SyncOOD, a simple data curation method that capitalizes on the capabilities of large foundation models to automatically extract meaningful OOD data from text-to-image generative models. This offers the model access to open-world knowledge encapsulated within off-the-shelf foundation models. The synthetic OOD samples are then employed to augment the training of a lightweight, plug-and-play OOD detector, thus effectively optimizing the in-distribution (ID)/OOD decision boundaries. Extensive experiments across multiple benchmarks demonstrate that SyncOOD significantly outperforms existing methods, establishing new state-of-the-art performance with minimal synthetic data usage.

Title: Lung-DETR: Deformable Detection Transformer for Sparse Lung Nodule Anomaly Detection

• Authors: Hooman Ramezani, Dionne Aleman, Daniel Létourneau
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05200
• Pdf URL: https://arxiv.org/pdf/2409.05200
• Copy Paste: [[2409.05200]] Lung-DETR: Deformable Detection Transformer for Sparse Lung Nodule Anomaly Detection(https://arxiv.org/abs/2409.05200)
• Keywords: anomaly
• Abstract: Accurate lung nodule detection for computed tomography (CT) scan imagery is challenging in real-world settings due to the sparse occurrence of nodules and similarity to other anatomical structures. In a typical positive case, nodules may appear in as few as 3% of CT slices, complicating detection. To address this, we reframe the problem as an anomaly detection task, targeting rare nodule occurrences in a predominantly normal dataset. We introduce a novel solution leveraging custom data preprocessing and Deformable Detection Transformer (Deformable- DETR). A 7.5mm Maximum Intensity Projection (MIP) is utilized to combine adjacent lung slices into single images, reducing the slice count and decreasing nodule sparsity. This enhances spatial context, allowing for better differentiation between nodules and other structures such as complex vascular structures and bronchioles. Deformable-DETR is employed to detect nodules, with a custom focal loss function to better handle the imbalanced dataset. Our model achieves state-of-the-art performance on the LUNA16 dataset with an F1 score of 94.2% (95.2% recall, 93.3% precision) on a dataset sparsely populated with lung nodules that is reflective of real-world clinical data.

Title: ICML Topological Deep Learning Challenge 2024: Beyond the Graph Domain

• Authors: Guillermo Bernárdez, Lev Telyatnikov, Marco Montagna, Federica Baccini, Mathilde Papillon, Miquel Ferriol-Galmés, Mustafa Hajij, Theodore Papamarkou, Maria Sofia Bucarelli, Olga Zaghen, Johan Mathe, Audun Myers, Scott Mahan, Hansen Lillemark, Sharvaree Vadgama, Erik Bekkers, Tim Doster, Tegan Emerson, Henry Kvinge, Katrina Agate, Nesreen K Ahmed, Pengfei Bai, Michael Banf, Claudio Battiloro, Maxim Beketov, Paul Bogdan, Martin Carrasco, Andrea Cavallo, Yun Young Choi, George Dasoulas, Matouš Elphick, Giordan Escalona, Dominik Filipiak, Halley Fritze, Thomas Gebhart, Manel Gil-Sorribes, Salvish Goomanee, Victor Guallar, Liliya Imasheva, Andrei Irimia, Hongwei Jin, Graham Johnson, Nikos Kanakaris, Boshko Koloski, Veljko Kovač, Manuel Lecha, Minho Lee, Pierrick Leroy, Theodore Long, German Magai, Alvaro Martinez, Marissa Masden, Sebastian Mežnar, Bertran Miquel-Oliver, Alexis Molina, Alexander Nikitin, Marco Nurisso, Matt Piekenbrock, Yu Qin, Patryk Rygiel, Alessandro Salatiello, Max Schattauer, Pavel Snopov, Julian Suk, Valentina Sánchez, Mauricio Tec, Francesco Vaccarino, Jonas Verhellen, Frederic Wantiez, Alexander Weers, Patrik Zajec, Blaž Škrlj, Nina Miolane
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05211
• Pdf URL: https://arxiv.org/pdf/2409.05211
• Copy Paste: [[2409.05211]] ICML Topological Deep Learning Challenge 2024: Beyond the Graph Domain(https://arxiv.org/abs/2409.05211)
• Keywords: generative
• Abstract: This paper describes the 2nd edition of the ICML Topological Deep Learning Challenge that was hosted within the ICML 2024 ELLIS Workshop on Geometry-grounded Representation Learning and Generative Modeling (GRaM). The challenge focused on the problem of representing data in different discrete topological domains in order to bridge the gap between Topological Deep Learning (TDL) and other types of structured datasets (e.g. point clouds, graphs). Specifically, participants were asked to design and implement topological liftings, i.e. mappings between different data structures and topological domains --like hypergraphs, or simplicial/cell/combinatorial complexes. The challenge received 52 submissions satisfying all the requirements. This paper introduces the main scope of the challenge, and summarizes the main results and findings.

Title: Synthetic Tabular Data Generation for Class Imbalance and Fairness: A Comparative Study

• Authors: Emmanouil Panagiotou, Arjun Roy, Eirini Ntoutsi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05215
• Pdf URL: https://arxiv.org/pdf/2409.05215
• Copy Paste: [[2409.05215]] Synthetic Tabular Data Generation for Class Imbalance and Fairness: A Comparative Study(https://arxiv.org/abs/2409.05215)
• Keywords: generative
• Abstract: Due to their data-driven nature, Machine Learning (ML) models are susceptible to bias inherited from data, especially in classification problems where class and group imbalances are prevalent. Class imbalance (in the classification target) and group imbalance (in protected attributes like sex or race) can undermine both ML utility and fairness. Although class and group imbalances commonly coincide in real-world tabular datasets, limited methods address this scenario. While most methods use oversampling techniques, like interpolation, to mitigate imbalances, recent advancements in synthetic tabular data generation offer promise but have not been adequately explored for this purpose. To this end, this paper conducts a comparative analysis to address class and group imbalances using state-of-the-art models for synthetic tabular data generation and various sampling strategies. Experimental results on four datasets, demonstrate the effectiveness of generative models for bias mitigation, creating opportunities for further exploration in this direction.

Title: NetDPSyn: Synthesizing Network Traces under Differential Privacy

• Authors: Danyu Sun, Joann Qiongna Chen, Chen Gong, Tianhao Wang, Zhou Li
• Subjects: cs.CR, cs.DB, cs.NI
• Abstract URL: https://arxiv.org/abs/2409.05249
• Pdf URL: https://arxiv.org/pdf/2409.05249
• Copy Paste: [[2409.05249]] NetDPSyn: Synthesizing Network Traces under Differential Privacy(https://arxiv.org/abs/2409.05249)
• Keywords: generative, anomaly
• Abstract: As the utilization of network traces for the network measurement research becomes increasingly prevalent, concerns regarding privacy leakage from network traces have garnered the public's attention. To safeguard network traces, researchers have proposed the trace synthesis that retains the essential properties of the raw data. However, previous works also show that synthesis traces with generative models are vulnerable under linkage attacks. This paper introduces NetDPSyn, the first system to synthesize high-fidelity network traces under privacy guarantees. NetDPSyn is built with the Differential Privacy (DP) framework as its core, which is significantly different from prior works that apply DP when training the generative model. The experiments conducted on three flow and two packet datasets indicate that NetDPSyn achieves much better data utility in downstream tasks like anomaly detection. NetDPSyn is also 2.5 times faster than the other methods on average in data synthesis.

Title: MRStyle: A Unified Framework for Color Style Transfer with Multi-Modality Reference

• Authors: Jiancheng Huang, Yu Gao, Zequn Jie, Yujie Zhong, Xintong Han, Lin Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05250
• Pdf URL: https://arxiv.org/pdf/2409.05250
• Copy Paste: [[2409.05250]] MRStyle: A Unified Framework for Color Style Transfer with Multi-Modality Reference(https://arxiv.org/abs/2409.05250)
• Keywords: diffusion
• Abstract: In this paper, we introduce MRStyle, a comprehensive framework that enables color style transfer using multi-modality reference, including image and text. To achieve a unified style feature space for both modalities, we first develop a neural network called IRStyle, which generates stylized 3D lookup tables for image reference. This is accomplished by integrating an interaction dual-mapping network with a combined supervised learning pipeline, resulting in three key benefits: elimination of visual artifacts, efficient handling of high-resolution images with low memory usage, and maintenance of style consistency even in situations with significant color style variations. For text reference, we align the text feature of stable diffusion priors with the style feature of our IRStyle to perform text-guided color style transfer (TRStyle). Our TRStyle method is highly efficient in both training and inference, producing notable open-set text-guided transfer results. Extensive experiments in both image and text settings demonstrate that our proposed method outperforms the state-of-the-art in both qualitative and quantitative evaluations.

Title: Disentangled Representations for Short-Term and Long-Term Person Re-Identification

• Authors: Chanho Eom, Wonkyung Lee, Geon Lee, Bumsub Ham
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05277
• Pdf URL: https://arxiv.org/pdf/2409.05277
• Copy Paste: [[2409.05277]] Disentangled Representations for Short-Term and Long-Term Person Re-Identification(https://arxiv.org/abs/2409.05277)
• Keywords: generative
• Abstract: We address the problem of person re-identification (reID), that is, retrieving person images from a large dataset, given a query image of the person of interest. A key challenge is to learn person representations robust to intra-class variations, as different persons could have the same attribute, and persons' appearances look different, e.g., with viewpoint changes. Recent reID methods focus on learning person features discriminative only for a particular factor of variations (e.g., human pose), which also requires corresponding supervisory signals (e.g., pose annotations). To tackle this problem, we propose to factorize person images into identity-related and unrelated features. Identity-related features contain information useful for specifying a particular person (e.g., clothing), while identity-unrelated ones hold other factors (e.g., human pose). To this end, we propose a new generative adversarial network, dubbed identity shuffle GAN (IS-GAN). It disentangles identity-related and unrelated features from person images through an identity-shuffling technique that exploits identification labels alone without any auxiliary supervisory signals. We restrict the distribution of identity-unrelated features or encourage the identity-related and unrelated features to be uncorrelated, facilitating the disentanglement process. Experimental results validate the effectiveness of IS-GAN, showing state-of-the-art performance on standard reID benchmarks, including Market-1501, CUHK03, and DukeMTMC-reID. We further demonstrate the advantages of disentangling person representations on a long-term reID task, setting a new state of the art on a Celeb-reID dataset.

Title: Seek and Solve Reasoning for Table Question Answering

• Authors: Ruya Jiang, Chun Wang, Weihong Deng
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05286
• Pdf URL: https://arxiv.org/pdf/2409.05286
• Copy Paste: [[2409.05286]] Seek and Solve Reasoning for Table Question Answering(https://arxiv.org/abs/2409.05286)
• Keywords: in-context
• Abstract: Table-based Question Answering (TQA) involves answering questions based on tabular data. The complexity of table structures and question logic makes this task difficult even for Large Language Models (LLMs). This paper improves TQA performance by leveraging LLMs' reasoning capabilities. Inspired by how humans solve TQA tasks, we propose a Seek-and-Solve pipeline that instructs the LLM to first seek relevant information and then answer questions. The two stages are integrated at the reasoning level, and their Chain of Thought (CoT) paths are integrated into a coherent Seek-and-Solve CoT (SS-CoT). Furthermore, we present a compact single-stage TQA-solving prompt distilled from the pipeline. Experiments demonstrate that under In-Context Learning settings, using samples with SS-CoT paths as demonstrations, the TQA-solving prompt can effectively guide the LLM to solve complex TQA tasks, resulting in improved performance and reliability. Our results highlight the importance of properly eliciting LLMs' reasoning capabilities in solving complex TQA tasks.

Title: TERD: A Unified Framework for Safeguarding Diffusion Models Against Backdoors

• Authors: Yichuan Mo, Hui Huang, Mingjie Li, Ang Li, Yisen Wang
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05294
• Pdf URL: https://arxiv.org/pdf/2409.05294
• Copy Paste: [[2409.05294]] TERD: A Unified Framework for Safeguarding Diffusion Models Against Backdoors(https://arxiv.org/abs/2409.05294)
• Keywords: diffusion
• Abstract: Diffusion models have achieved notable success in image generation, but they remain highly vulnerable to backdoor attacks, which compromise their integrity by producing specific undesirable outputs when presented with a pre-defined trigger. In this paper, we investigate how to protect diffusion models from this dangerous threat. Specifically, we propose TERD, a backdoor defense framework that builds unified modeling for current attacks, which enables us to derive an accessible reversed loss. A trigger reversion strategy is further employed: an initial approximation of the trigger through noise sampled from a prior distribution, followed by refinement through differential multi-step samplers. Additionally, with the reversed trigger, we propose backdoor detection from the noise space, introducing the first backdoor input detection approach for diffusion models and a novel model detection algorithm that calculates the KL divergence between reversed and benign distributions. Extensive evaluations demonstrate that TERD secures a 100% True Positive Rate (TPR) and True Negative Rate (TNR) across datasets of varying resolutions. TERD also demonstrates nice adaptability to other Stochastic Differential Equation (SDE)-based models. Our code is available at this https URL.

Title: Resource-Efficient Generative AI Model Deployment in Mobile Edge Networks

• Authors: Yuxin Liang, Peng Yang, Yuanyuan He, Feng Lyu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05303
• Pdf URL: https://arxiv.org/pdf/2409.05303
• Copy Paste: [[2409.05303]] Resource-Efficient Generative AI Model Deployment in Mobile Edge Networks(https://arxiv.org/abs/2409.05303)
• Keywords: generative
• Abstract: The surging development of Artificial Intelligence-Generated Content (AIGC) marks a transformative era of the content creation and production. Edge servers promise attractive benefits, e.g., reduced service delay and backhaul traffic load, for hosting AIGC services compared to cloud-based solutions. However, the scarcity of available resources on the edge pose significant challenges in deploying generative AI models. In this paper, by characterizing the resource and delay demands of typical generative AI models, we find that the consumption of storage and GPU memory, as well as the model switching delay represented by I/O delay during the preloading phase, are significant and vary across models. These multidimensional coupling factors render it difficult to make efficient edge model deployment decisions. Hence, we present a collaborative edge-cloud framework aiming to properly manage generative AI model deployment on the edge. Specifically, we formulate edge model deployment problem considering heterogeneous features of models as an optimization problem, and propose a model-level decision selection algorithm to solve it. It enables pooled resource sharing and optimizes the trade-off between resource consumption and delay in edge generative AI model deployment. Simulation results validate the efficacy of the proposed algorithm compared with baselines, demonstrating its potential to reduce overall costs by providing feature-aware model deployment decisions.

Title: GDFlow: Anomaly Detection with NCDE-based Normalizing Flow for Advanced Driver Assistance System

• Authors: Kangjun Lee, Minha Kim, Youngho Jun, Simon S. Woo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05346
• Pdf URL: https://arxiv.org/pdf/2409.05346
• Copy Paste: [[2409.05346]] GDFlow: Anomaly Detection with NCDE-based Normalizing Flow for Advanced Driver Assistance System(https://arxiv.org/abs/2409.05346)
• Keywords: anomaly
• Abstract: For electric vehicles, the Adaptive Cruise Control (ACC) in Advanced Driver Assistance Systems (ADAS) is designed to assist braking based on driving conditions, road inclines, predefined deceleration strengths, and user braking patterns. However, the driving data collected during the development of ADAS are generally limited and lack diversity. This deficiency leads to late or aggressive braking for different users. Crucially, it is necessary to effectively identify anomalies, such as unexpected or inconsistent braking patterns in ADAS, especially given the challenge of working with unlabelled, limited, and noisy datasets from real-world electric vehicles. In order to tackle the aforementioned challenges in ADAS, we propose Graph Neural Controlled Differential Equation Normalizing Flow (GDFlow), a model that leverages Normalizing Flow (NF) with Neural Controlled Differential Equations (NCDE) to learn the distribution of normal driving patterns continuously. Compared to the traditional clustering or anomaly detection algorithms, our approach effectively captures the spatio-temporal information from different sensor data and more accurately models continuous changes in driving patterns. Additionally, we introduce a quantile-based maximum likelihood objective to improve the likelihood estimate of the normal data near the boundary of the distribution, enhancing the model's ability to distinguish between normal and anomalous patterns. We validate GDFlow using real-world electric vehicle driving data that we collected from Hyundai IONIQ5 and GV80EV, achieving state-of-the-art performance compared to six baselines across four dataset configurations of different vehicle types and drivers. Furthermore, our model outperforms the latest anomaly detection methods across four time series benchmark datasets. Our approach demonstrates superior efficiency in inference time compared to existing methods.

Title: TriplePlay: Enhancing Federated Learning with CLIP for Non-IID Data and Resource Efficiency

• Authors: Ahmed Imteaj, Md Zarif Hossain, Saika Zaman, Abdur R. Shahid
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05347
• Pdf URL: https://arxiv.org/pdf/2409.05347
• Copy Paste: [[2409.05347]] TriplePlay: Enhancing Federated Learning with CLIP for Non-IID Data and Resource Efficiency(https://arxiv.org/abs/2409.05347)
• Keywords: foundation model
• Abstract: The rapid advancement and increasing complexity of pretrained models, exemplified by CLIP, offer significant opportunities as well as challenges for Federated Learning (FL), a critical component of privacy-preserving artificial intelligence. This research delves into the intricacies of integrating large foundation models like CLIP within FL frameworks to enhance privacy, efficiency, and adaptability across heterogeneous data landscapes. It specifically addresses the challenges posed by non-IID data distributions, the computational and communication overheads of leveraging such complex models, and the skewed representation of classes within datasets. We propose TriplePlay, a framework that integrates CLIP as an adapter to enhance FL's adaptability and performance across diverse data distributions. This approach addresses the long-tail distribution challenge to ensure fairness while reducing resource demands through quantization and low-rank adaptation techniques.Our simulation results demonstrate that TriplePlay effectively decreases GPU usage costs and speeds up the learning process, achieving convergence with reduced communication overhead.

Title: On the Convergence Analysis of Over-Parameterized Variational Autoencoders: A Neural Tangent Kernel Perspective

• Authors: Li Wang, Wei Huang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05349
• Pdf URL: https://arxiv.org/pdf/2409.05349
• Copy Paste: [[2409.05349]] On the Convergence Analysis of Over-Parameterized Variational Autoencoders: A Neural Tangent Kernel Perspective(https://arxiv.org/abs/2409.05349)
• Keywords: generative
• Abstract: Variational Auto-Encoders (VAEs) have emerged as powerful probabilistic models for generative tasks. However, their convergence properties have not been rigorously proven. The challenge of proving convergence is inherently difficult due to the highly non-convex nature of the training objective and the implementation of a Stochastic Neural Network (SNN) within VAE architectures. This paper addresses these challenges by characterizing the optimization trajectory of SNNs utilized in VAEs through the lens of Neural Tangent Kernel (NTK) techniques. These techniques govern the optimization and generalization behaviors of ultra-wide neural networks. We provide a mathematical proof of VAE convergence under mild assumptions, thus advancing the theoretical understanding of VAE optimization dynamics. Furthermore, we establish a novel connection between the optimization problem faced by over-parameterized SNNs and the Kernel Ridge Regression (KRR) problem. Our findings not only contribute to the theoretical foundation of VAEs but also open new avenues for investigating the optimization of generative models using advanced kernel methods. Our theoretical claims are verified by experimental simulations.

Title: Memoryless Multimodal Anomaly Detection via Student-Teacher Network and Signed Distance Learning

• Authors: Zhongbin Sun, Xiaolong Li, Yiran Li, Yue Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05378
• Pdf URL: https://arxiv.org/pdf/2409.05378
• Copy Paste: [[2409.05378]] Memoryless Multimodal Anomaly Detection via Student-Teacher Network and Signed Distance Learning(https://arxiv.org/abs/2409.05378)
• Keywords: anomaly
• Abstract: Unsupervised anomaly detection is a challenging computer vision task, in which 2D-based anomaly detection methods have been extensively studied. However, multimodal anomaly detection based on RGB images and 3D point clouds requires further investigation. The existing methods are mainly inspired by memory bank based methods commonly used in 2D-based anomaly detection, which may cost extra memory for storing mutimodal features. In present study, a novel memoryless method MDSS is proposed for multimodal anomaly detection, which employs a light-weighted student-teacher network and a signed distance function to learn from RGB images and 3D point clouds respectively, and complements the anomaly information from the two modalities. Specifically, a student-teacher network is trained with normal RGB images and masks generated from point clouds by a dynamic loss, and the anomaly score map could be obtained from the discrepancy between the output of student and teacher. Furthermore, the signed distance function learns from normal point clouds to predict the signed distances between points and surface, and the obtained signed distances are used to generate anomaly score map. Subsequently, the anomaly score maps are aligned to generate the final anomaly score map for detection. The experimental results indicate that MDSS is comparable but more stable than the SOTA memory bank based method Shape-guided, and furthermore performs better than other baseline methods.

Title: Deep Learning for Video Anomaly Detection: A Review

• Authors: Peng Wu, Chengyu Pan, Yuting Yan, Guansong Pang, Peng Wang, Yanning Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05383
• Pdf URL: https://arxiv.org/pdf/2409.05383
• Copy Paste: [[2409.05383]] Deep Learning for Video Anomaly Detection: A Review(https://arxiv.org/abs/2409.05383)
• Keywords: anomaly
• Abstract: Video anomaly detection (VAD) aims to discover behaviors or events deviating from the normality in videos. As a long-standing task in the field of computer vision, VAD has witnessed much good progress. In the era of deep learning, with the explosion of architectures of continuously growing capability and capacity, a great variety of deep learning based methods are constantly emerging for the VAD task, greatly improving the generalization ability of detection algorithms and broadening the application scenarios. Therefore, such a multitude of methods and a large body of literature make a comprehensive survey a pressing necessity. In this paper, we present an extensive and comprehensive research review, covering the spectrum of five different categories, namely, semi-supervised, weakly supervised, fully supervised, unsupervised and open-set supervised VAD, and we also delve into the latest VAD works based on pre-trained large models, remedying the limitations of past reviews in terms of only focusing on semi-supervised VAD and small model based methods. For the VAD task with different levels of supervision, we construct a well-organized taxonomy, profoundly discuss the characteristics of different types of methods, and show their performance comparisons. In addition, this review involves the public datasets, open-source codes, and evaluation metrics covering all the aforementioned VAD tasks. Finally, we provide several important research directions for the VAD community.

Title: A Novel Representation of Periodic Pattern and Its Application to Untrained Anomaly Detection

• Authors: Peng Ye, Chengyu Tao, Juan Du
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05389
• Pdf URL: https://arxiv.org/pdf/2409.05389
• Copy Paste: [[2409.05389]] A Novel Representation of Periodic Pattern and Its Application to Untrained Anomaly Detection(https://arxiv.org/abs/2409.05389)
• Keywords: anomaly
• Abstract: There are a variety of industrial products that possess periodic textures or surfaces, such as carbon fiber textiles and display panels. Traditional image-based quality inspection methods for these products require identifying the periodic patterns from normal images (without anomaly and noise) and subsequently detecting anomaly pixels with inconsistent appearances. However, it remains challenging to accurately extract the periodic pattern from a single image in the presence of unknown anomalies and measurement noise. To deal with this challenge, this paper proposes a novel self-representation of the periodic image defined on a set of continuous parameters. In this way, periodic pattern learning can be embedded into a joint optimization framework, which is named periodic-sparse decomposition, with simultaneously modeling the sparse anomalies and Gaussian noise. Finally, for the real-world industrial images that may not strictly satisfy the periodic assumption, we propose a novel pixel-level anomaly scoring strategy to enhance the performance of anomaly detection. Both simulated and real-world case studies demonstrate the effectiveness of the proposed methodology for periodic pattern learning and anomaly detection.

Title: TAVP: Task-Adaptive Visual Prompt for Cross-domain Few-shot Segmentation

• Authors: Jiaqi Yang, Ye Huang, Xiangjian He, Linlin Shen, Guoping Qiu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05393
• Pdf URL: https://arxiv.org/pdf/2409.05393
• Copy Paste: [[2409.05393]] TAVP: Task-Adaptive Visual Prompt for Cross-domain Few-shot Segmentation(https://arxiv.org/abs/2409.05393)
• Keywords: foundation model, generative
• Abstract: Under the backdrop of large-scale pre-training, large visual models (LVM) have demonstrated significant potential in image understanding. The recent emergence of the Segment Anything Model (SAM) has brought a qualitative shift in the field of image segmentation, supporting flexible interactive cues and strong learning capabilities. However, its performance often falls short in cross-domain and few-shot applications. Transferring prior knowledge from foundation models to new applications while preserving learning capabilities is worth exploring. This work proposes a task-adaptive prompt framework based on SAM, a new paradigm for Cross-dominan few-shot segmentation (CD-FSS). First, a Multi-level Feature Fusion (MFF) was used for integrated feature extraction. Besides, an additional Class Domain Task-Adaptive Auto-Prompt (CDTAP) module was combined with the segmentation branch for class-domain agnostic feature extraction and high-quality learnable prompt production. This significant advancement uses a unique generative approach to prompts alongside a comprehensive model structure and specialized prototype computation. While ensuring that the prior knowledge of SAM is not discarded, the new branch disentangles category and domain information through prototypes, guiding it in adapting the CD-FSS. We have achieved the best results on three benchmarks compared to the recent state-of-the-art (SOTA) methods. Comprehensive experiments showed that after task-specific and weighted guidance, the abundant feature information of SAM can be better learned for CD-FSS.

Title: Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling

• Authors: Georgios Pantazopoulos, Malvina Nikandrou, Alessandro Suglia, Oliver Lemon, Arash Eshghi
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05395
• Pdf URL: https://arxiv.org/pdf/2409.05395
• Copy Paste: [[2409.05395]] Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling(https://arxiv.org/abs/2409.05395)
• Keywords: in-context
• Abstract: This study explores replacing Transformers in Visual Language Models (VLMs) with Mamba, a recent structured state space model (SSM) that demonstrates promising performance in sequence modeling. We test models up to 3B parameters under controlled conditions, showing that Mamba-based VLMs outperforms Transformers-based VLMs in captioning, question answering, and reading comprehension. However, we find that Transformers achieve greater performance in visual grounding and the performance gap widens with scale. We explore two hypotheses to explain this phenomenon: 1) the effect of task-agnostic visual encoding on the updates of the hidden states, and 2) the difficulty in performing visual grounding from the perspective of in-context multimodal retrieval. Our results indicate that a task-aware encoding yields minimal performance gains on grounding, however, Transformers significantly outperform Mamba at in-context multimodal retrieval. Overall, Mamba shows promising performance on tasks where the correct output relies on a summary of the image but struggles when retrieval of explicit information from the context is required.

Title: Sequential Posterior Sampling with Diffusion Models

• Authors: Tristan S.W. Stevens, Oisín Nolan, Jean-Luc Robert, Ruud J.G. van Sloun
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05399
• Pdf URL: https://arxiv.org/pdf/2409.05399
• Copy Paste: [[2409.05399]] Sequential Posterior Sampling with Diffusion Models(https://arxiv.org/abs/2409.05399)
• Keywords: diffusion, generative
• Abstract: Diffusion models have quickly risen in popularity for their ability to model complex distributions and perform effective posterior sampling. Unfortunately, the iterative nature of these generative models makes them computationally expensive and unsuitable for real-time sequential inverse problems such as ultrasound imaging. Considering the strong temporal structure across sequences of frames, we propose a novel approach that models the transition dynamics to improve the efficiency of sequential diffusion posterior sampling in conditional image synthesis. Through modeling sequence data using a video vision transformer (ViViT) transition model based on previous diffusion outputs, we can initialize the reverse diffusion trajectory at a lower noise scale, greatly reducing the number of iterations required for convergence. We demonstrate the effectiveness of our approach on a real-world dataset of high frame rate cardiac ultrasound images and show that it achieves the same performance as a full diffusion trajectory while accelerating inference 25$\times$, enabling real-time posterior sampling. Furthermore, we show that the addition of a transition model improves the PSNR up to 8\% in cases with severe motion. Our method opens up new possibilities for real-time applications of diffusion models in imaging and other domains requiring real-time inference.

Title: KRONC: Keypoint-based Robust Camera Optimization for 3D Car Reconstruction

• Authors: Davide Di Nucci, Alessandro Simoni, Matteo Tomei, Luca Ciuffreda, Roberto Vezzani, Rita Cucchiara
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05407
• Pdf URL: https://arxiv.org/pdf/2409.05407
• Copy Paste: [[2409.05407]] KRONC: Keypoint-based Robust Camera Optimization for 3D Car Reconstruction(https://arxiv.org/abs/2409.05407)
• Keywords: diffusion
• Abstract: The three-dimensional representation of objects or scenes starting from a set of images has been a widely discussed topic for years and has gained additional attention after the diffusion of NeRF-based approaches. However, an underestimated prerequisite is the knowledge of camera poses or, more specifically, the estimation of the extrinsic calibration parameters. Although excellent general-purpose Structure-from-Motion methods are available as a pre-processing step, their computational load is high and they require a lot of frames to guarantee sufficient overlapping among the views. This paper introduces KRONC, a novel approach aimed at inferring view poses by leveraging prior knowledge about the object to reconstruct and its representation through semantic keypoints. With a focus on vehicle scenes, KRONC is able to estimate the position of the views as a solution to a light optimization problem targeting the convergence of keypoints' back-projections to a singular point. To validate the method, a specific dataset of real-world car scenes has been collected. Experiments confirm KRONC's ability to generate excellent estimates of camera poses starting from very coarse initialization. Results are comparable with Structure-from-Motion methods with huge savings in computation. Code and data will be made publicly available.

Title: CipherDM: Secure Three-Party Inference for Diffusion Model Sampling

• Authors: Xin Zhao, Xiaojun Chen, Xudong Chen, He Li, Tingyu Fan, Zhendong Zhao
• Subjects: cs.CR, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05414
• Pdf URL: https://arxiv.org/pdf/2409.05414
• Copy Paste: [[2409.05414]] CipherDM: Secure Three-Party Inference for Diffusion Model Sampling(https://arxiv.org/abs/2409.05414)
• Keywords: diffusion
• Abstract: Diffusion Models (DMs) achieve state-of-the-art synthesis results in image generation and have been applied to various fields. However, DMs sometimes seriously violate user privacy during usage, making the protection of privacy an urgent issue. Using traditional privacy computing schemes like Secure Multi-Party Computation (MPC) directly in DMs faces significant computation and communication challenges. To address these issues, we propose CipherDM, the first novel, versatile and universal framework applying MPC technology to DMs for secure sampling, which can be widely implemented on multiple DM based tasks. We thoroughly analyze sampling latency breakdown, find time-consuming parts and design corresponding secure MPC protocols for computing nonlinear activations including SoftMax, SiLU and Mish. CipherDM is evaluated on popular architectures (DDPM, DDIM) using MNIST dataset and on SD deployed by diffusers. Compared to direct implementation on SPU, our approach improves running time by approximately 1.084\times \sim 2.328\times, and reduces communication costs by approximately 1.212\times \sim 1.791\times.

Title: TextToucher: Fine-Grained Text-to-Touch Generation

• Authors: Jiahang Tu, Hao Fu, Fengyu Yang, Hanbin Zhao, Chao Zhang, Hui Qian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05427
• Pdf URL: https://arxiv.org/pdf/2409.05427
• Copy Paste: [[2409.05427]] TextToucher: Fine-Grained Text-to-Touch Generation(https://arxiv.org/abs/2409.05427)
• Keywords: diffusion
• Abstract: Tactile sensation plays a crucial role in the development of multi-modal large models and embodied intelligence. To collect tactile data with minimal cost as possible, a series of studies have attempted to generate tactile images by vision-to-touch image translation. However, compared to text modality, visual modality-driven tactile generation cannot accurately depict human tactile sensation. In this work, we analyze the characteristics of tactile images in detail from two granularities: object-level (tactile texture, tactile shape), and sensor-level (gel status). We model these granularities of information through text descriptions and propose a fine-grained Text-to-Touch generation method (TextToucher) to generate high-quality tactile samples. Specifically, we introduce a multimodal large language model to build the text sentences about object-level tactile information and employ a set of learnable text prompts to represent the sensor-level tactile information. To better guide the tactile generation process with the built text information, we fuse the dual grains of text information and explore various dual-grain text conditioning methods within the diffusion transformer architecture. Furthermore, we propose a Contrastive Text-Touch Pre-training (CTTP) metric to precisely evaluate the quality of text-driven generated tactile data. Extensive experiments demonstrate the superiority of our TextToucher method. The source codes will be available at \url{this https URL}.

Title: EndoOmni: Zero-Shot Cross-Dataset Depth Estimation in Endoscopy by Robust Self-Learning from Noisy Labels

• Authors: Qingyao Tian, Zhen Chen, Huai Liao, Xinyan Huang, Lujie Li, Sebastien Ourselin, Hongbin Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05442
• Pdf URL: https://arxiv.org/pdf/2409.05442
• Copy Paste: [[2409.05442]] EndoOmni: Zero-Shot Cross-Dataset Depth Estimation in Endoscopy by Robust Self-Learning from Noisy Labels(https://arxiv.org/abs/2409.05442)
• Keywords: foundation model
• Abstract: Single-image depth estimation is essential for endoscopy tasks such as localization, reconstruction, and augmented reality. Most existing methods in surgical scenes focus on in-domain depth estimation, limiting their real-world applicability. This constraint stems from the scarcity and inferior labeling quality of medical data for training. In this work, we present EndoOmni, the first foundation model for zero-shot cross-domain depth estimation for endoscopy. To harness the potential of diverse training data, we refine the advanced self-learning paradigm that employs a teacher model to generate pseudo-labels, guiding a student model trained on large-scale labeled and unlabeled data. To address training disturbance caused by inherent noise in depth labels, we propose a robust training framework that leverages both depth labels and estimated confidence from the teacher model to jointly guide the student model training. Moreover, we propose a weighted scale-and-shift invariant loss to adaptively adjust learning weights based on label confidence, thus imposing learning bias towards cleaner label pixels while reducing the influence of highly noisy pixels. Experiments on zero-shot relative depth estimation show that our EndoOmni improves state-of-the-art methods in medical imaging for 41\% and existing foundation models for 25\% in terms of absolute relative error on specific dataset. Furthermore, our model provides strong initialization for fine-tuning to metric depth estimation, maintaining superior performance in both in-domain and out-of-domain scenarios. The source code will be publicly available.

Title: Representational Analysis of Binding in Large Language Models

• Authors: Qin Dai, Benjamin Heinzerling, Kentaro Inui
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.05448
• Pdf URL: https://arxiv.org/pdf/2409.05448
• Copy Paste: [[2409.05448]] Representational Analysis of Binding in Large Language Models(https://arxiv.org/abs/2409.05448)
• Keywords: in-context
• Abstract: Entity tracking is essential for complex reasoning. To perform in-context entity tracking, language models (LMs) must bind an entity to its attribute (e.g., bind a container to its content) to recall attribute for a given entity. For example, given a context mentioning ``The coffee is in Box Z, the stone is in Box M, the map is in Box H'', to infer ``Box Z contains the coffee'' later, LMs must bind ``Box Z'' to ``coffee''. To explain the binding behaviour of LMs, Feng and Steinhardt (2023) introduce a Binding ID mechanism and state that LMs use a abstract concept called Binding ID (BI) to internally mark entity-attribute pairs. However, they have not directly captured the BI determinant information from entity activations. In this work, we provide a novel view of the Binding ID mechanism by localizing the prototype of BI information. Specifically, we discover that there exists a low-rank subspace in the hidden state (or activation) of LMs, that primarily encodes the order of entity and attribute and which is used as the prototype of BI to causally determine the binding. To identify this subspace, we choose principle component analysis as our first attempt and it is empirically proven to be effective. Moreover, we also discover that when editing representations along directions in the subspace, LMs tend to bind a given entity to other attributes accordingly. For example, by patching activations along the BI encoding direction we can make the LM to infer ``Box Z contains the stone'' and ``Box Z contains the map''.

Title: DriveScape: Towards High-Resolution Controllable Multi-View Driving Video Generation

• Authors: Wei Wu, Xi Guo, Weixuan Tang, Tingxuan Huang, Chiyu Wang, Dongyue Chen, Chenjing Ding
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05463
• Pdf URL: https://arxiv.org/pdf/2409.05463
• Copy Paste: [[2409.05463]] DriveScape: Towards High-Resolution Controllable Multi-View Driving Video Generation(https://arxiv.org/abs/2409.05463)
• Keywords: generative
• Abstract: Recent advancements in generative models have provided promising solutions for synthesizing realistic driving videos, which are crucial for training autonomous driving perception models. However, existing approaches often struggle with multi-view video generation due to the challenges of integrating 3D information while maintaining spatial-temporal consistency and effectively learning from a unified model. In this paper, we propose an end-to-end framework named DriveScape for multi-view, 3D condition-guided video generation. DriveScape not only streamlines the process by integrating camera data to ensure comprehensive spatial-temporal coverage, but also introduces a Bi-Directional Modulated Transformer module to effectively align 3D road structural information. As a result, our approach enables precise control over video generation, significantly enhancing realism and providing a robust solution for generating multi-view driving videos. Our framework achieves state-of-the-art results on the nuScenes dataset, demonstrating impressive generative quality metrics with an FID score of 8.34 and an FVD score of 76.39, as well as superior performance across various perception tasks. This paves the way for more accurate environmental simulations in autonomous driving. Code will be available at our project homepage.

Title: CRADLE-VAE: Enhancing Single-Cell Gene Perturbation Modeling with Counterfactual Reasoning-based Artifact Disentanglement

• Authors: Seungheun Baek, Soyon Park, Yan Ting Chok, Junhyun Lee, Jueon Park, Mogan Gim, Jaewoo Kang
• Subjects: cs.LG, cs.AI, q-bio.GN, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2409.05484
• Pdf URL: https://arxiv.org/pdf/2409.05484
• Copy Paste: [[2409.05484]] CRADLE-VAE: Enhancing Single-Cell Gene Perturbation Modeling with Counterfactual Reasoning-based Artifact Disentanglement(https://arxiv.org/abs/2409.05484)
• Keywords: generative
• Abstract: Predicting cellular responses to various perturbations is a critical focus in drug discovery and personalized therapeutics, with deep learning models playing a significant role in this endeavor. Single-cell datasets contain technical artifacts that may hinder the predictability of such models, which poses quality control issues highly regarded in this area. To address this, we propose CRADLE-VAE, a causal generative framework tailored for single-cell gene perturbation modeling, enhanced with counterfactual reasoning-based artifact disentanglement. Throughout training, CRADLE-VAE models the underlying latent distribution of technical artifacts and perturbation effects present in single-cell datasets. It employs counterfactual reasoning to effectively disentangle such artifacts by modulating the latent basal spaces and learns robust features for generating cellular response data with improved quality. Experimental results demonstrate that this approach improves not only treatment effect estimation performance but also generative quality as well. The CRADLE-VAE codebase is publicly available at this https URL.

Title: A Taxonomy of Miscompressions: Preparing Image Forensics for Neural Compression

• Authors: Nora Hofer, Rainer Böhme
• Subjects: cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05490
• Pdf URL: https://arxiv.org/pdf/2409.05490
• Copy Paste: [[2409.05490]] A Taxonomy of Miscompressions: Preparing Image Forensics for Neural Compression(https://arxiv.org/abs/2409.05490)
• Keywords: generative
• Abstract: Neural compression has the potential to revolutionize lossy image compression. Based on generative models, recent schemes achieve unprecedented compression rates at high perceptual quality but compromise semantic fidelity. Details of decompressed images may appear optically flawless but semantically different from the originals, making compression errors difficult or impossible to detect. We explore the problem space and propose a provisional taxonomy of miscompressions. It defines three types of 'what happens' and has a binary 'high impact' flag indicating miscompressions that alter symbols. We discuss how the taxonomy can facilitate risk communication and research into mitigations.

Title: Latent 3D Brain MRI Counterfactual

• Authors: Wei Peng, Tian Xia, Fabio De Sousa Ribeiro, Tomas Bosschieter, Ehsan Adeli, Qingyu Zhao, Ben Glocker, Kilian M. Pohl
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05585
• Pdf URL: https://arxiv.org/pdf/2409.05585
• Copy Paste: [[2409.05585]] Latent 3D Brain MRI Counterfactual(https://arxiv.org/abs/2409.05585)
• Keywords: generative
• Abstract: The number of samples in structural brain MRI studies is often too small to properly train deep learning models. Generative models show promise in addressing this issue by effectively learning the data distribution and generating high-fidelity MRI. However, they struggle to produce diverse, high-quality data outside the distribution defined by the training data. One way to address the issue is using causal models developed for 3D volume counterfactuals. However, accurately modeling causality in high-dimensional spaces is a challenge so that these models generally generate 3D brain MRIS of lower quality. To address these challenges, we propose a two-stage method that constructs a Structural Causal Model (SCM) within the latent space. In the first stage, we employ a VQ-VAE to learn a compact embedding of the MRI volume. Subsequently, we integrate our causal model into this latent space and execute a three-step counterfactual procedure using a closed-form Generalized Linear Model (GLM). Our experiments conducted on real-world high-resolution MRI data (1mm) demonstrate that our method can generate high-quality 3D MRI counterfactuals.

Title: Adapted-MoE: Mixture of Experts with Test-Time Adaption for Anomaly Detection

• Authors: Tianwu Lei, Silin Chen, Bohan Wang, Zhengkai Jiang, Ningmu Zou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05611
• Pdf URL: https://arxiv.org/pdf/2409.05611
• Copy Paste: [[2409.05611]] Adapted-MoE: Mixture of Experts with Test-Time Adaption for Anomaly Detection(https://arxiv.org/abs/2409.05611)
• Keywords: anomaly
• Abstract: Most unsupervised anomaly detection methods based on representations of normal samples to distinguish anomalies have recently made remarkable progress. However, existing methods only learn a single decision boundary for distinguishing the samples within the training dataset, neglecting the variation in feature distribution for normal samples even in the same category in the real world. Furthermore, it was not considered that a distribution bias still exists between the test set and the train set. Therefore, we propose an Adapted-MoE which contains a routing network and a series of expert models to handle multiple distributions of same-category samples by divide and conquer. Specifically, we propose a routing network based on representation learning to route same-category samples into the subclasses feature space. Then, a series of expert models are utilized to learn the representation of various normal samples and construct several independent decision boundaries. We propose the test-time adaption to eliminate the bias between the unseen test sample representation and the feature distribution learned by the expert model. Our experiments are conducted on a dataset that provides multiple subclasses from three categories, namely Texture AD benchmark. The Adapted-MoE significantly improves the performance of the baseline model, achieving 2.18%-7.20% and 1.57%-16.30% increase in I-AUROC and P-AUROC, which outperforms the current state-of-the-art methods. Our code is available at this https URL.

Title: Forward KL Regularized Preference Optimization for Aligning Diffusion Policies

• Authors: Zhao Shan, Chenyou Fan, Shuang Qiu, Jiyuan Shi, Chenjia Bai
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05622
• Pdf URL: https://arxiv.org/pdf/2409.05622
• Copy Paste: [[2409.05622]] Forward KL Regularized Preference Optimization for Aligning Diffusion Policies(https://arxiv.org/abs/2409.05622)
• Keywords: diffusion
• Abstract: Diffusion models have achieved remarkable success in sequential decision-making by leveraging the highly expressive model capabilities in policy learning. A central problem for learning diffusion policies is to align the policy output with human intents in various tasks. To achieve this, previous methods conduct return-conditioned policy generation or Reinforcement Learning (RL)-based policy optimization, while they both rely on pre-defined reward functions. In this work, we propose a novel framework, Forward KL regularized Preference optimization for aligning Diffusion policies, to align the diffusion policy with preferences directly. We first train a diffusion policy from the offline dataset without considering the preference, and then align the policy to the preference data via direct preference optimization. During the alignment phase, we formulate direct preference learning in a diffusion policy, where the forward KL regularization is employed in preference optimization to avoid generating out-of-distribution actions. We conduct extensive experiments for MetaWorld manipulation and D4RL tasks. The results show our method exhibits superior alignment with preferences and outperforms previous state-of-the-art algorithms.

Title: Unlearning or Concealment? A Critical Analysis and Evaluation Metrics for Unlearning in Diffusion Models

• Authors: Aakash Sen Sharma, Niladri Sarkar, Vikram Chundawat, Ankur A Mali, Murari Mandal
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05668
• Pdf URL: https://arxiv.org/pdf/2409.05668
• Copy Paste: [[2409.05668]] Unlearning or Concealment? A Critical Analysis and Evaluation Metrics for Unlearning in Diffusion Models(https://arxiv.org/abs/2409.05668)
• Keywords: diffusion
• Abstract: Recent research has seen significant interest in methods for concept removal and targeted forgetting in diffusion models. In this paper, we conduct a comprehensive white-box analysis to expose significant vulnerabilities in existing diffusion model unlearning methods. We show that the objective functions used for unlearning in the existing methods lead to decoupling of the targeted concepts (meant to be forgotten) for the corresponding prompts. This is concealment and not actual unlearning, which was the original goal. The ineffectiveness of current methods stems primarily from their narrow focus on reducing generation probabilities for specific prompt sets, neglecting the diverse modalities of intermediate guidance employed during the inference process. The paper presents a rigorous theoretical and empirical examination of four commonly used techniques for unlearning in diffusion models. We introduce two new evaluation metrics: Concept Retrieval Score (CRS) and Concept Confidence Score (CCS). These metrics are based on a successful adversarial attack setup that can recover forgotten concepts from unlearned diffusion models. The CRS measures the similarity between the latent representations of the unlearned and fully trained models after unlearning. It reports the extent of retrieval of the forgotten concepts with increasing amount of guidance. The CCS quantifies the confidence of the model in assigning the target concept to the manipulated data. It reports the probability of the unlearned model's generations to be aligned with the original domain knowledge with increasing amount of guidance. Evaluating existing unlearning methods with our proposed stringent metrics for diffusion models reveals significant shortcomings in their ability to truly unlearn concepts. Source Code: this https URL

Title: Zero-shot Outlier Detection via Prior-data Fitted Networks: Model Selection Bygone!

• Authors: Yuchen Shen, Haomin Wen, Leman Akoglu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05672
• Pdf URL: https://arxiv.org/pdf/2409.05672
• Copy Paste: [[2409.05672]] Zero-shot Outlier Detection via Prior-data Fitted Networks: Model Selection Bygone!(https://arxiv.org/abs/2409.05672)
• Keywords: foundation model
• Abstract: Outlier detection (OD) has a vast literature as it finds numerous applications in environmental monitoring, cybersecurity, finance, and medicine to name a few. Being an inherently unsupervised task, model selection is a key bottleneck for OD (both algorithm and hyperparameter selection) without label supervision. There is a long list of techniques to choose from -- both classical algorithms and deep neural architectures -- and while several studies report their hyperparameter sensitivity, the literature is quite slim on unsupervised model selection -- limiting the effective use of OD in practice. In this paper we present FoMo-0D, for zero/0-shot OD exploring a transformative new direction that bypasses the hurdle of model selection altogether (!), thus breaking new ground. The fundamental idea behind FoMo-0D is the Prior-data Fitted Networks, recently introduced by Muller et al.(2022), which trains a Transformer model on a large body of synthetically generated data from a prior data distribution. In essence, FoMo-0D is a pretrained Foundation Model for zero/0-shot OD on tabular data, which can directly predict the (outlier/inlier) label of any test data at inference time, by merely a single forward pass -- making obsolete the need for choosing an algorithm/architecture, tuning its associated hyperparameters, and even training any model parameters when given a new OD dataset. Extensive experiments on 57 public benchmark datasets against 26 baseline methods show that FoMo-0D performs statistically no different from the top 2nd baseline, while significantly outperforming the majority of the baselines, with an average inference time of 7.7 ms per test sample.

Title: AnomalyCD: A benchmark for Earth anomaly change detection with high-resolution and time-series observations

• Authors: Jingtao Li, Qian Zhu, Xinyu Wang, Hengwei Zhao, Yanfei Zhong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.05679
• Pdf URL: https://arxiv.org/pdf/2409.05679
• Copy Paste: [[2409.05679]] AnomalyCD: A benchmark for Earth anomaly change detection with high-resolution and time-series observations(https://arxiv.org/abs/2409.05679)
• Keywords: anomaly
• Abstract: Various Earth anomalies have destroyed the stable, balanced state, resulting in fatalities and serious destruction of property. With the advantages of large-scale and precise observation, high-resolution remote sensing images have been widely used for anomaly monitoring and localization. Powered by the deep representation, the existing methods have achieved remarkable advances, primarily in classification and change detection techniques. However, labeled samples are difficult to acquire due to the low probability of anomaly occurrence, and the trained models are limited to fixed anomaly categories, which hinders the application for anomalies with few samples or unknown anomalies. In this paper, to tackle this problem, we propose the anomaly change detection (AnomalyCD) technique, which accepts time-series observations and learns to identify anomalous changes by learning from the historical normal change pattern. Compared to the existing techniques, AnomalyCD processes an unfixed number of time steps and can localize the various anomalies in a unified manner, without human supervision. To benchmark AnomalyCD, we constructed a high-resolution dataset with time-series images dedicated to various Earth anomalies (the AnomalyCDD dataset). AnomalyCDD contains high-resolution (from 0.15 to 2.39 m/pixel), time-series (from 3 to 7 time steps), and large-scale images (1927.93 km2 in total) collected globally Furthermore, we developed a zero-shot baseline model (AnomalyCDM), which implements the AnomalyCD technique by extracting a general representation from the segment anything model (SAM) and conducting temporal comparison to distinguish the anomalous changes from normal changes. AnomalyCDM is designed as a two-stage workflow to enhance the efficiency, and has the ability to process the unseen images directly, without retraining for each scene.

Title: Segmentation by Factorization: Unsupervised Semantic Segmentation for Pathology by Factorizing Foundation Model Features

• Authors: Jacob Gildenblat, Ofir Hadar
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05697
• Pdf URL: https://arxiv.org/pdf/2409.05697
• Copy Paste: [[2409.05697]] Segmentation by Factorization: Unsupervised Semantic Segmentation for Pathology by Factorizing Foundation Model Features(https://arxiv.org/abs/2409.05697)
• Keywords: foundation model
• Abstract: We introduce Segmentation by Factorization (F-SEG), an unsupervised segmentation method for pathology that generates segmentation masks from pre-trained deep learning models. F-SEG allows the use of pre-trained deep neural networks, including recently developed pathology foundation models, for semantic segmentation. It achieves this without requiring additional training or finetuning, by factorizing the spatial features extracted by the models into segmentation masks and their associated concept features. We create generic tissue phenotypes for H&E images by training clustering models for multiple numbers of clusters on features extracted from several deep learning models on The Cancer Genome Atlas Program (TCGA), and then show how the clusters can be used for factorizing corresponding segmentation masks using off-the-shelf deep learning models. Our results show that F-SEG provides robust unsupervised segmentation capabilities for H&E pathology images, and that the segmentation quality is greatly improved by utilizing pathology foundation models. We discuss and propose methods for evaluating the performance of unsupervised segmentation in pathology.

Title: pFedGPA: Diffusion-based Generative Parameter Aggregation for Personalized Federated Learning

• Authors: Jiahao Lai, Jiaqi Li, Jian Xu, Yanru Wu, Boshi Tang, Siqi Chen, Yongfeng Huang, Wenbo Ding, Yang Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.05701
• Pdf URL: https://arxiv.org/pdf/2409.05701
• Copy Paste: [[2409.05701]] pFedGPA: Diffusion-based Generative Parameter Aggregation for Personalized Federated Learning(https://arxiv.org/abs/2409.05701)
• Keywords: diffusion, generative
• Abstract: Federated Learning (FL) offers a decentralized approach to model training, where data remains local and only model parameters are shared between the clients and the central server. Traditional methods, such as Federated Averaging (FedAvg), linearly aggregate these parameters which are usually trained on heterogeneous data distributions, potentially overlooking the complex, high-dimensional nature of the parameter space. This can result in degraded performance of the aggregated model. While personalized FL approaches can mitigate the heterogeneous data issue to some extent, the limitation of linear aggregation remains unresolved. To alleviate this issue, we investigate the generative approach of diffusion model and propose a novel generative parameter aggregation framework for personalized FL, \texttt{pFedGPA}. In this framework, we deploy a diffusion model on the server to integrate the diverse parameter distributions and propose a parameter inversion method to efficiently generate a set of personalized parameters for each client. This inversion method transforms the uploaded parameters into a latent code, which is then aggregated through denoising sampling to produce the final personalized parameters. By encoding the dependence of a client's model parameters on the specific data distribution using the high-capacity diffusion model, \texttt{pFedGPA} can effectively decouple the complexity of the overall distribution of all clients' model parameters from the complexity of each individual client's parameter distribution. Our experimental results consistently demonstrate the superior performance of the proposed method across multiple datasets, surpassing baseline approaches.

Title: Predicting Critical Heat Flux with Uncertainty Quantification and Domain Generalization Using Conditional Variational Autoencoders and Deep Neural Networks

• Authors: Farah Alsafadi, Aidan Furlong, Xu Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05790
• Pdf URL: https://arxiv.org/pdf/2409.05790
• Copy Paste: [[2409.05790]] Predicting Critical Heat Flux with Uncertainty Quantification and Domain Generalization Using Conditional Variational Autoencoders and Deep Neural Networks(https://arxiv.org/abs/2409.05790)
• Keywords: generative
• Abstract: Deep generative models (DGMs) have proven to be powerful in generating realistic data samples. Their capability to learn the underlying distribution of a dataset enable them to generate synthetic data samples that closely resemble the original training dataset, thus addressing the challenge of data scarcity. In this work, we investigated the capabilities of DGMs by developing a conditional variational autoencoder (CVAE) model to augment the critical heat flux (CHF) measurement data that was used to generate the 2006 Groeneveld lookup table. To determine how this approach compared to traditional methods, a fine-tuned deep neural network (DNN) regression model was created and evaluated with the same dataset. Both the CVAE and DNN models achieved small mean absolute relative errors, with the CVAE model maintaining more favorable results. To quantify the uncertainty in the model's predictions, uncertainty quantification (UQ) was performed with repeated sampling of the CVAE model and ensembling of the DNN model. Following UQ, the DNN ensemble notably improved performance when compared to the baseline DNN model, while the CVAE model achieved similar results to its non-UQ results. The CVAE model was shown to have significantly less variability and a higher confidence after assessment of the prediction-wise relative standard deviations. Evaluating domain generalization, both models achieved small mean error values when predicting both inside and outside the training domain, with predictions outside the training domain showing slightly larger errors. Overall, the CVAE model was comparable to the DNN regression model in predicting CHF values but with better uncertainty behavior.

Title: Enhancing Preference-based Linear Bandits via Human Response Time

• Authors: Shen Li, Yuyang Zhang, Zhaolin Ren, Claire Liang, Na Li, Julie A. Shah
• Subjects: cs.LG, cs.AI, cs.HC, econ.EM, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.05798
• Pdf URL: https://arxiv.org/pdf/2409.05798
• Copy Paste: [[2409.05798]] Enhancing Preference-based Linear Bandits via Human Response Time(https://arxiv.org/abs/2409.05798)
• Keywords: diffusion
• Abstract: Binary human choice feedback is widely used in interactive preference learning for its simplicity, but it provides limited information about preference strength. To overcome this limitation, we leverage human response times, which inversely correlate with preference strength, as complementary information. Our work integrates the EZ-diffusion model, which jointly models human choices and response times, into preference-based linear bandits. We introduce a computationally efficient utility estimator that reformulates the utility estimation problem using both choices and response times as a linear regression problem. Theoretical and empirical comparisons with traditional choice-only estimators reveal that for queries with strong preferences ("easy" queries), choices alone provide limited information, while response times offer valuable complementary information about preference strength. As a result, incorporating response times makes easy queries more useful. We demonstrate this advantage in the fixed-budget best-arm identification problem, with simulations based on three real-world datasets, consistently showing accelerated learning when response times are incorporated.

Title: Celcomen: spatial causal disentanglement for single-cell and tissue perturbation modeling

• Authors: Stathis Megas, Daniel G. Chen, Krzysztof Polanski, Moshe Eliasof, Carola-Bibiane Schonlieb, Sarah A. Teichmann
• Subjects: cs.LG, q-bio.TO
• Abstract URL: https://arxiv.org/abs/2409.05804
• Pdf URL: https://arxiv.org/pdf/2409.05804
• Copy Paste: [[2409.05804]] Celcomen: spatial causal disentanglement for single-cell and tissue perturbation modeling(https://arxiv.org/abs/2409.05804)
• Keywords: generative
• Abstract: Celcomen leverages a mathematical causality framework to disentangle intra- and inter- cellular gene regulation programs in spatial transcriptomics and single-cell data through a generative graph neural network. It can learn gene-gene interactions, as well as generate post-perturbation counterfactual spatial transcriptomics, thereby offering access to experimentally inaccessible samples. We validated its disentanglement, identifiability, and counterfactual prediction capabilities through simulations and in clinically relevant human glioblastoma, human fetal spleen, and mouse lung cancer samples. Celcomen provides the means to model disease and therapy induced changes allowing for new insights into single-cell spatially resolved tissue responses relevant to human health.

Title: Benchmarking Chinese Knowledge Rectification in Large Language Models

• Authors: Tianhe Lu, Jizhan Fang, Yunzhi Yao, Xin Xu, Ningyu Zhang, Huajun Chen
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.05806
• Pdf URL: https://arxiv.org/pdf/2409.05806
• Copy Paste: [[2409.05806]] Benchmarking Chinese Knowledge Rectification in Large Language Models(https://arxiv.org/abs/2409.05806)
• Keywords: generative
• Abstract: While Large Language Models (LLMs) exhibit remarkable generative capabilities, they are not without flaws, particularly in the form of hallucinations. This issue is even more pronounced when LLMs are applied to specific languages and domains. For example, LLMs may generate nonsense information when handling Chinese ancient poetry, proverbs, or idioms, owing to the lack of specific knowledge. To this end, this paper introduces a benchmark for rectifying Chinese knowledge in LLMs via knowledge editing. Specifically, we introduce a new Chinese dataset, CKnowEdit, by collecting seven type of knowledge from various sources, including classical texts, idioms, and content from Baidu Tieba Ruozhiba, thereby accounting for the unique polyphony, antithesis, and logical constructs inherent in the Chinese language. Through the analysis of this dataset, we uncover the challenges faced by current LLMs in mastering Chinese. Furthermore, our evaluation of state-of-the-art knowledge editing techniques on this dataset unveil the substantial scope for advancement in the rectification of Chinese knowledge. Code and dataset are available at this https URL.

Title: VFA: Vision Frequency Analysis of Foundation Models and Human

• Authors: Mohammad-Javad Darvishi-Bayazi, Md Rifat Arefin, Jocelyn Faubert, Irina Rish
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2409.05817
• Pdf URL: https://arxiv.org/pdf/2409.05817
• Copy Paste: [[2409.05817]] VFA: Vision Frequency Analysis of Foundation Models and Human(https://arxiv.org/abs/2409.05817)
• Keywords: foundation model
• Abstract: Machine learning models often struggle with distribution shifts in real-world scenarios, whereas humans exhibit robust adaptation. Models that better align with human perception may achieve higher out-of-distribution generalization. In this study, we investigate how various characteristics of large-scale computer vision models influence their alignment with human capabilities and robustness. Our findings indicate that increasing model and data size and incorporating rich semantic information and multiple modalities enhance models' alignment with human perception and their overall robustness. Our empirical analysis demonstrates a strong correlation between out-of-distribution accuracy and human alignment.