2025-02-26

Title: A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models

• Authors: Zihao Lin, Samyadeep Basu, Mohammad Beigi, Varun Manjunatha, Ryan A. Rossi, Zichao Wang, Yufan Zhou, Sriram Balasubramanian, Arman Zarei, Keivan Rezaei, Ying Shen, Barry Menglong Yao, Zhiyang Xu, Qin Liu, Yuxiang Zhang, Yan Sun, Shilong Liu, Li Shen, Hongxuan Li, Soheil Feizi, Lifu Huang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.17516
• Pdf URL: https://arxiv.org/pdf/2502.17516
• Copy Paste: [[2502.17516]] A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models(https://arxiv.org/abs/2502.17516)
• Keywords: foundation model, generative
• Abstract: The rise of foundation models has transformed machine learning research, prompting efforts to uncover their inner workings and develop more efficient and reliable applications for better control. While significant progress has been made in interpreting Large Language Models (LLMs), multimodal foundation models (MMFMs) - such as contrastive vision-language models, generative vision-language models, and text-to-image models - pose unique interpretability challenges beyond unimodal frameworks. Despite initial studies, a substantial gap remains between the interpretability of LLMs and MMFMs. This survey explores two key aspects: (1) the adaptation of LLM interpretability methods to multimodal models and (2) understanding the mechanistic differences between unimodal language models and crossmodal systems. By systematically reviewing current MMFM analysis techniques, we propose a structured taxonomy of interpretability methods, compare insights across unimodal and multimodal architectures, and highlight critical research gaps.

Title: On the Vulnerability of Concept Erasure in Diffusion Models

• Authors: Lucas Beerens, Alex D. Richardson, Kaicheng Zhang, Dongdong Chen
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2502.17537
• Pdf URL: https://arxiv.org/pdf/2502.17537
• Copy Paste: [[2502.17537]] On the Vulnerability of Concept Erasure in Diffusion Models(https://arxiv.org/abs/2502.17537)
• Keywords: diffusion
• Abstract: The proliferation of text-to-image diffusion models has raised significant privacy and security concerns, particularly regarding the generation of copyrighted or harmful images. To address these issues, research on machine unlearning has developed various concept erasure methods, which aim to remove the effect of unwanted data through post-hoc training. However, we show these erasure techniques are vulnerable, where images of supposedly erased concepts can still be generated using adversarially crafted prompts. We introduce RECORD, a coordinate-descent-based algorithm that discovers prompts capable of eliciting the generation of erased content. We demonstrate that RECORD significantly beats the attack success rate of current state-of-the-art attack methods. Furthermore, our findings reveal that models subjected to concept erasure are more susceptible to adversarial attacks than previously anticipated, highlighting the urgency for more robust unlearning approaches. We open source all our code at this https URL

Title: Training a Generally Curious Agent

• Authors: Fahim Tajwar, Yiding Jiang, Abitha Thankaraj, Sumaita Sadia Rahman, J Zico Kolter, Jeff Schneider, Ruslan Salakhutdinov
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2502.17543
• Pdf URL: https://arxiv.org/pdf/2502.17543
• Copy Paste: [[2502.17543]] Training a Generally Curious Agent(https://arxiv.org/abs/2502.17543)
• Keywords: in-context
• Abstract: Efficient exploration is essential for intelligent systems interacting with their environment, but existing language models often fall short in scenarios that require strategic information gathering. In this paper, we present PAPRIKA, a fine-tuning approach that enables language models to develop general decision-making capabilities that are not confined to particular environments. By training on synthetic interaction data from different tasks that require diverse strategies, PAPRIKA teaches models to explore and adapt their behavior on a new task based on environment feedback in-context without more gradient updates. Experimental results show that models fine-tuned with PAPRIKA can effectively transfer their learned decision-making capabilities to entirely unseen tasks without additional training. Unlike traditional training, our approach's primary bottleneck lies in sampling useful interaction data instead of model updates. To improve sample efficiency, we propose a curriculum learning strategy that prioritizes sampling trajectories from tasks with high learning potential. These results suggest a promising path towards AI systems that can autonomously solve novel sequential decision-making problems that require interactions with the external world.

Title: Flexible Counterfactual Explanations with Generative Models

• Authors: Stig Hellemans, Andres Algaba, Sam Verboven, Vincent Ginis
• Subjects: cs.LG, cs.AI, stat.ME
• Abstract URL: https://arxiv.org/abs/2502.17613
• Pdf URL: https://arxiv.org/pdf/2502.17613
• Copy Paste: [[2502.17613]] Flexible Counterfactual Explanations with Generative Models(https://arxiv.org/abs/2502.17613)
• Keywords: generative
• Abstract: Counterfactual explanations provide actionable insights to achieve desired outcomes by suggesting minimal changes to input features. However, existing methods rely on fixed sets of mutable features, which makes counterfactual explanations inflexible for users with heterogeneous real-world constraints. Here, we introduce Flexible Counterfactual Explanations, a framework incorporating counterfactual templates, which allows users to dynamically specify mutable features at inference time. In our implementation, we use Generative Adversarial Networks (FCEGAN), which align explanations with user-defined constraints without requiring model retraining or additional optimization. Furthermore, FCEGAN is designed for black-box scenarios, leveraging historical prediction datasets to generate explanations without direct access to model internals. Experiments across economic and healthcare datasets demonstrate that FCEGAN significantly improves counterfactual explanations' validity compared to traditional benchmark methods. By integrating user-driven flexibility and black-box compatibility, counterfactual templates support personalized explanations tailored to user constraints.

Title: CalibRefine: Deep Learning-Based Online Automatic Targetless LiDAR-Camera Calibration with Iterative and Attention-Driven Post-Refinement

• Authors: Lei Chenga, Lihao Guoa, Tianya Zhangb, Tam Bangb, Austin Harrisb, Mustafa Hajijc, Mina Sartipib, Siyang Cao
• Subjects: cs.CV, eess.SY
• Abstract URL: https://arxiv.org/abs/2502.17648
• Pdf URL: https://arxiv.org/pdf/2502.17648
• Copy Paste: [[2502.17648]] CalibRefine: Deep Learning-Based Online Automatic Targetless LiDAR-Camera Calibration with Iterative and Attention-Driven Post-Refinement(https://arxiv.org/abs/2502.17648)
• Keywords: self-supervised
• Abstract: Accurate multi-sensor calibration is essential for deploying robust perception systems in applications such as autonomous driving, robotics, and intelligent transportation. Existing LiDAR-camera calibration methods often rely on manually placed targets, preliminary parameter estimates, or intensive data preprocessing, limiting their scalability and adaptability in real-world settings. In this work, we propose a fully automatic, targetless, and online calibration framework, CalibRefine, which directly processes raw LiDAR point clouds and camera images. Our approach is divided into four stages: (1) a Common Feature Discriminator that trains on automatically detected objects--using relative positions, appearance embeddings, and semantic classes--to generate reliable LiDAR-camera correspondences, (2) a coarse homography-based calibration, (3) an iterative refinement to incrementally improve alignment as additional data frames become available, and (4) an attention-based refinement that addresses non-planar distortions by leveraging a Vision Transformer and cross-attention mechanisms. Through extensive experiments on two urban traffic datasets, we show that CalibRefine delivers high-precision calibration results with minimal human involvement, outperforming state-of-the-art targetless methods and remaining competitive with, or surpassing, manually tuned baselines. Our findings highlight how robust object-level feature matching, together with iterative and self-supervised attention-based adjustments, enables consistent sensor fusion in complex, real-world conditions without requiring ground-truth calibration matrices or elaborate data preprocessing.

Title: Yes, Q-learning Helps Offline In-Context RL

• Authors: Denis Tarasov, Alexander Nikulin, Ilya Zisman, Albina Klepach, Andrei Polubarov, Nikita Lyubaykin, Alexander Derevyagin, Igor Kiselev, Vladislav Kurenkov
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.17666
• Pdf URL: https://arxiv.org/pdf/2502.17666
• Copy Paste: [[2502.17666]] Yes, Q-learning Helps Offline In-Context RL(https://arxiv.org/abs/2502.17666)
• Keywords: in-context
• Abstract: In this work, we explore the integration of Reinforcement Learning (RL) approaches within a scalable offline In-Context RL (ICRL) framework. Through experiments across more than 150 datasets derived from GridWorld and MuJoCo environments, we demonstrate that optimizing RL objectives improves performance by approximately 40% on average compared to the widely established Algorithm Distillation (AD) baseline across various dataset coverages, structures, expertise levels, and environmental complexities. Our results also reveal that offline RL-based methods outperform online approaches, which are not specifically designed for offline scenarios. These findings underscore the importance of aligning the learning objectives with RL's reward-maximization goal and demonstrate that offline RL is a promising direction for application in ICRL settings.

Title: Contrastive Visual Data Augmentation

• Authors: Yu Zhou, Bingxuan Li, Mohan Tang, Xiaomeng Jin, Te-Lin Wu, Kuan-Hao Huang, Heng Ji, Kai-Wei Chang, Nanyun Peng
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG, cs.MM
• Abstract URL: https://arxiv.org/abs/2502.17709
• Pdf URL: https://arxiv.org/pdf/2502.17709
• Copy Paste: [[2502.17709]] Contrastive Visual Data Augmentation(https://arxiv.org/abs/2502.17709)
• Keywords: generative
• Abstract: Large multimodal models (LMMs) often struggle to recognize novel concepts, as they rely on pre-trained knowledge and have limited ability to capture subtle visual details. Domain-specific knowledge gaps in training also make them prone to confusing visually similar, commonly misrepresented, or low-resource concepts. To help LMMs better align nuanced visual features with language, improving their ability to recognize and reason about novel or rare concepts, we propose a Contrastive visual Data Augmentation (CoDA) strategy. CoDA extracts key contrastive textual and visual features of target concepts against the known concepts they are misrecognized as, and then uses multimodal generative models to produce targeted synthetic data. Automatic filtering of extracted features and augmented images is implemented to guarantee their quality, as verified by human annotators. We show the effectiveness and efficiency of CoDA on low-resource concept and diverse scene recognition datasets including INaturalist and SUN. We additionally collect NovelSpecies, a benchmark dataset consisting of newly discovered animal species that are guaranteed to be unseen by LMMs. LLaVA-1.6 1-shot updating results on these three datasets show CoDA significantly improves SOTA visual data augmentation strategies by 12.3% (NovelSpecies), 5.1% (SUN), and 6.0% (iNat) absolute gains in accuracy.

Title: Knowledge Distillation with Training Wheels

• Authors: Guanlin Liu, Anand Ramachandran, Tanmay Gangwani, Yan Fu, Abhinav Sethy
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.17717
• Pdf URL: https://arxiv.org/pdf/2502.17717
• Copy Paste: [[2502.17717]] Knowledge Distillation with Training Wheels(https://arxiv.org/abs/2502.17717)
• Keywords: generative
• Abstract: Knowledge distillation is used, in generative language modeling, to train a smaller student model using the help of a larger teacher model, resulting in improved capabilities for the student model. In this paper, we formulate a more general framework for knowledge distillation where the student learns from the teacher during training, and also learns to ask for the teacher's help at test-time following rules specifying test-time restrictions. Towards this, we first formulate knowledge distillation as an entropy-regularized value optimization problem. Adopting Path Consistency Learning to solve this, leads to a new knowledge distillation algorithm using on-policy and off-policy demonstrations. We extend this using constrained reinforcement learning to a framework that incorporates the use of the teacher model as a test-time reference, within constraints. In this situation, akin to a human learner, the model needs to learn not only the learning material, but also the relative difficulty of different sections to prioritize for seeking teacher help. We examine the efficacy of our method through experiments in translation and summarization tasks, observing trends in accuracy and teacher use, noting that our approach unlocks operating points not available to the popular Speculative Decoding approach.

Title: Aligning Compound AI Systems via System-level DPO

• Authors: Xiangwen Wang, Yibo Jacky Zhang, Zhoujie Ding, Katherine Tsai, Sanmi Koyejo
• Subjects: cs.LG, cs.AI, cs.MA
• Abstract URL: https://arxiv.org/abs/2502.17721
• Pdf URL: https://arxiv.org/pdf/2502.17721
• Copy Paste: [[2502.17721]] Aligning Compound AI Systems via System-level DPO(https://arxiv.org/abs/2502.17721)
• Keywords: diffusion
• Abstract: Compound AI systems, comprising multiple interacting components such as LLM agents and external tools, demonstrate state-of-the-art results across diverse tasks. It is hence crucial to align components within the system to produce consistent results that match human expectations. However, conventional alignment methods, such as Direct Preference Optimization (DPO), are not directly applicable to compound AI systems. These challenges include the non-differentiable interactions between components, making end-to-end gradient optimization infeasible. Additionally, system-level preferences cannot be directly translated into component-level preferences, further complicating alignment. We address the issues by formulating compound AI systems as Directed Acyclic Graphs (DAGs), capturing the connections between agents and the data generation processes. We propose a system-level DPO (SysDPO) to jointly align compound systems by adapting the DPO to operate on these DAGs. We study the joint alignment of an LLM and a diffusion model to demonstrate the effectiveness of our approach. Our exploration provides insights into the alignment of compound AI systems and lays a foundation for future advancements.

Title: Can Score-Based Generative Modeling Effectively Handle Medical Image Classification?

• Authors: Sushmita Sarker, Prithul Sarker, George Bebis, Alireza Tavakkoli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.17727
• Pdf URL: https://arxiv.org/pdf/2502.17727
• Copy Paste: [[2502.17727]] Can Score-Based Generative Modeling Effectively Handle Medical Image Classification?(https://arxiv.org/abs/2502.17727)
• Keywords: generative
• Abstract: The remarkable success of deep learning in recent years has prompted applications in medical image classification and diagnosis tasks. While classification models have demonstrated robustness in classifying simpler datasets like MNIST or natural images such as ImageNet, this resilience is not consistently observed in complex medical image datasets where data is more scarce and lacks diversity. Moreover, previous findings on natural image datasets have indicated a potential trade-off between data likelihood and classification accuracy. In this study, we explore the use of score-based generative models as classifiers for medical images, specifically mammographic images. Our findings suggest that our proposed generative classifier model not only achieves superior classification results on CBIS-DDSM, INbreast and Vin-Dr Mammo datasets, but also introduces a novel approach to image classification in a broader context. Our code is publicly available at this https URL

Title: A digital eye-fixation biomarker using a deep anomaly scheme to classify Parkisonian patterns

• Authors: Juan Niño, Luis Guayacán, Santiago Gómez, Fabio Martínez
• Subjects: cs.CV, eess.IV, q-bio.NC
• Abstract URL: https://arxiv.org/abs/2502.17762
• Pdf URL: https://arxiv.org/pdf/2502.17762
• Copy Paste: [[2502.17762]] A digital eye-fixation biomarker using a deep anomaly scheme to classify Parkisonian patterns(https://arxiv.org/abs/2502.17762)
• Keywords: anomaly
• Abstract: Oculomotor alterations constitute a promising biomarker to detect and characterize Parkinson's disease (PD), even in prodromal stages. Currently, only global and simplified eye movement trajectories are employed to approximate the complex and hidden kinematic relationships of the oculomotor function. Recent advances on machine learning and video analysis have encouraged novel characterizations of eye movement patterns to quantify PD. These schemes enable the identification of spatiotemporal segments primarily associated with PD. However, they rely on discriminative models that require large training datasets and depend on balanced class distributions. This work introduces a novel video analysis scheme to quantify Parkinsonian eye fixation patterns with an anomaly detection framework. Contrary to classical deep discriminative schemes that learn differences among labeled classes, the proposed approach is focused on one-class learning, avoiding the necessity of a significant amount of data. The proposed approach focuses only on Parkinson's representation, considering any other class sample as an anomaly of the distribution. This approach was evaluated for an ocular fixation task, in a total of 13 control subjects and 13 patients on different stages of the disease. The proposed digital biomarker achieved an average sensitivity and specificity of 0.97 and 0.63, respectively, yielding an AUC-ROC of 0.95. A statistical test shows significant differences (p < 0.05) among predicted classes, evidencing a discrimination between patients and control subjects.

Title: Can Multimodal LLMs Perform Time Series Anomaly Detection?

• Authors: Xiongxiao Xu, Haoran Wang, Yueqing Liang, Philip S. Yu, Yue Zhao, Kai Shu
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.17812
• Pdf URL: https://arxiv.org/pdf/2502.17812
• Copy Paste: [[2502.17812]] Can Multimodal LLMs Perform Time Series Anomaly Detection?(https://arxiv.org/abs/2502.17812)
• Keywords: anomaly
• Abstract: Large language models (LLMs) have been increasingly used in time series analysis. However, the potential of multimodal LLMs (MLLMs), particularly vision-language models, for time series remains largely under-explored. One natural way for humans to detect time series anomalies is through visualization and textual description. Motivated by this, we raise a critical and practical research question: Can multimodal LLMs perform time series anomaly detection? To answer this, we propose VisualTimeAnomaly benchmark to evaluate MLLMs in time series anomaly detection (TSAD). Our approach transforms time series numerical data into the image format and feed these images into various MLLMs, including proprietary models (GPT-4o and Gemini-1.5) and open-source models (LLaVA-NeXT and Qwen2-VL), each with one larger and one smaller variant. In total, VisualTimeAnomaly contains 12.4k time series images spanning 3 scenarios and 3 anomaly granularities with 9 anomaly types across 8 MLLMs. Starting with the univariate case (point- and range-wise anomalies), we extend our evaluation to more practical scenarios, including multivariate and irregular time series scenarios, and variate-wise anomalies. Our study reveals several key insights: 1) MLLMs detect range- and variate-wise anomalies more effectively than point-wise anomalies. 2) MLLMs are highly robust to irregular time series, even with 25% of the data missing. 3) Open-source MLLMs perform comparably to proprietary models in TSAD. While open-source MLLMs excel on univariate time series, proprietary MLLMs demonstrate superior effectiveness on multivariate time series. To the best of our knowledge, this is the first work to comprehensively investigate MLLMs for TSAD, particularly for multivariate and irregular time series scenarios. We release our dataset and code at this https URL to support future research.

Title: HRR: Hierarchical Retrospection Refinement for Generated Image Detection

• Authors: Peipei Yuan, Zijing Xie, Shuo Ye, Hong Chen, Yulong Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.17862
• Pdf URL: https://arxiv.org/pdf/2502.17862
• Copy Paste: [[2502.17862]] HRR: Hierarchical Retrospection Refinement for Generated Image Detection(https://arxiv.org/abs/2502.17862)
• Keywords: diffusion, generative
• Abstract: Generative artificial intelligence holds significant potential for abuse, and generative image detection has become a key focus of research. However, existing methods primarily focused on detecting a specific generative model and emphasizing the localization of synthetic regions, while neglecting the interference caused by image size and style on model learning. Our goal is to reach a fundamental conclusion: Is the image real or generated? To this end, we propose a diffusion model-based generative image detection framework termed Hierarchical Retrospection Refinement~(HRR). It designs a multi-scale style retrospection module that encourages the model to generate detailed and realistic multi-scale representations, while alleviating the learning biases introduced by dataset styles and generative models. Additionally, based on the principle of correntropy sparse additive machine, a feature refinement module is designed to reduce the impact of redundant features on learning and capture the intrinsic structure and patterns of the data, thereby improving the model's generalization ability. Extensive experiments demonstrate the HRR framework consistently delivers significant performance improvements, outperforming state-of-the-art methods in generated image detection task.

Title: ASurvey: Spatiotemporal Consistency in Video Generation

• Authors: Zhiyu Yin, Kehai Chen, Xuefeng Bai, Ruili Jiang, Juntao Li, Hongdong Li, Jin Liu, Yang Xiang, Jun Yu, Min Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.17863
• Pdf URL: https://arxiv.org/pdf/2502.17863
• Copy Paste: [[2502.17863]] ASurvey: Spatiotemporal Consistency in Video Generation(https://arxiv.org/abs/2502.17863)
• Keywords: foundation model
• Abstract: Video generation, by leveraging a dynamic visual generation method, pushes the boundaries of Artificial Intelligence Generated Content (AIGC). Video generation presents unique challenges beyond static image generation, requiring both high-quality individual frames and temporal coherence to maintain consistency across the spatiotemporal sequence. Recent works have aimed at addressing the spatiotemporal consistency issue in video generation, while few literature review has been organized from this perspective. This gap hinders a deeper understanding of the underlying mechanisms for high-quality video generation. In this survey, we systematically review the recent advances in video generation, covering five key aspects: foundation models, information representations, generation schemes, post-processing techniques, and evaluation metrics. We particularly focus on their contributions to maintaining spatiotemporal consistency. Finally, we discuss the future directions and challenges in this field, hoping to inspire further efforts to advance the development of video generation.

Title: Contrastive Learning with Nasty Noise

• Authors: Ziruo Zhao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.17872
• Pdf URL: https://arxiv.org/pdf/2502.17872
• Copy Paste: [[2502.17872]] Contrastive Learning with Nasty Noise(https://arxiv.org/abs/2502.17872)
• Keywords: self-supervised
• Abstract: Contrastive learning has emerged as a powerful paradigm for self-supervised representation learning. This work analyzes the theoretical limits of contrastive learning under nasty noise, where an adversary modifies or replaces training samples. Using PAC learning and VC-dimension analysis, lower and upper bounds on sample complexity in adversarial settings are established. Additionally, data-dependent sample complexity bounds based on the l2-distance function are derived.

Title: EEGM2: An Efficient Mamba-2-Based Self-Supervised Framework for Long-Sequence EEG Modeling

• Authors: Jiazhen Hong, Geoffrey Mackellar, Soheila Ghane
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2502.17873
• Pdf URL: https://arxiv.org/pdf/2502.17873
• Copy Paste: [[2502.17873]] EEGM2: An Efficient Mamba-2-Based Self-Supervised Framework for Long-Sequence EEG Modeling(https://arxiv.org/abs/2502.17873)
• Keywords: self-supervised, foundation model
• Abstract: Deep learning has achieved significant progress in the development of electroencephalogram (EEG) foundation models, with Transformer-based architectures excelling at capturing long-range dependencies. However, their quadratic computational complexity presents challenges in memory efficiency, training, and inference speed, limiting their scalability and generalizability as a foundation model. In this paper, we propose EEGM2, a self-supervised framework based on structured state space duality (SSD) that overcomes these limitations. EEGM2 introduces three key innovations: (1) a reconstruction-based framework that captures both local and global EEG features through Mamba-2 structured state space models, (2) a spatiotemporal-aware loss function that enhances robustness to noise and preserves spectral information, and (3) a multi-branch receptive field input embedding strategy that improves cross-subject generalization and stability for EEG sequences of varying lengths. In comparison to traditional pretraining methods, on raw EEG or latent representation spaces, EEGM2 shows superior performance on long-sequence tasks, where conventional models struggle. Our experimental results on six EEG datasets validate that EEGM2 not only achieves state-of-the-art cross-domain accuracy but also reduces computational overhead, making it a more efficient solution for deployment on resource-constrained BCI devices.

Title: VVRec: Reconstruction Attacks on DL-based Volumetric Video Upstreaming via Latent Diffusion Model with Gamma Distribution

• Authors: Rui Lu, Bihai Zhang, Dan Wang
• Subjects: cs.CR, cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2502.17880
• Pdf URL: https://arxiv.org/pdf/2502.17880
• Copy Paste: [[2502.17880]] VVRec: Reconstruction Attacks on DL-based Volumetric Video Upstreaming via Latent Diffusion Model with Gamma Distribution(https://arxiv.org/abs/2502.17880)
• Keywords: diffusion
• Abstract: With the popularity of 3D volumetric video applications, such as Autonomous Driving, Virtual Reality, and Mixed Reality, current developers have turned to deep learning for compressing volumetric video frames, i.e., point clouds for video upstreaming. The latest deep learning-based solutions offer higher efficiency, lower distortion, and better hardware support compared to traditional ones like MPEG and JPEG. However, privacy threats arise, especially reconstruction attacks targeting to recover the original input point cloud from the intermediate results. In this paper, we design VVRec, to the best of our knowledge, which is the first targeting DL-based Volumetric Video Reconstruction attack scheme. VVRec demonstrates the ability to reconstruct high-quality point clouds from intercepted transmission intermediate results using four well-trained neural network modules we design. Leveraging the latest latent diffusion models with Gamma distribution and a refinement algorithm, VVRec excels in reconstruction quality, color recovery, and surpasses existing defenses. We evaluate VVRec using three volumetric video datasets. The results demonstrate that VVRec achieves 64.70dB reconstruction accuracy, with an impressive 46.39% reduction of distortion over baselines.

Title: Robust Polyp Detection and Diagnosis through Compositional Prompt-Guided Diffusion Models

• Authors: Jia Yu, Yan Zhu, Peiyao Fu, Tianyi Chen, Junbo Huang, Quanlin Li, Pinghong Zhou, Zhihua Wang, Fei Wu, Shuo Wang, Xian Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.17951
• Pdf URL: https://arxiv.org/pdf/2502.17951
• Copy Paste: [[2502.17951]] Robust Polyp Detection and Diagnosis through Compositional Prompt-Guided Diffusion Models(https://arxiv.org/abs/2502.17951)
• Keywords: diffusion
• Abstract: Colorectal cancer (CRC) is a significant global health concern, and early detection through screening plays a critical role in reducing mortality. While deep learning models have shown promise in improving polyp detection, classification, and segmentation, their generalization across diverse clinical environments, particularly with out-of-distribution (OOD) data, remains a challenge. Multi-center datasets like PolypGen have been developed to address these issues, but their collection is costly and time-consuming. Traditional data augmentation techniques provide limited variability, failing to capture the complexity of medical images. Diffusion models have emerged as a promising solution for generating synthetic polyp images, but the image generation process in current models mainly relies on segmentation masks as the condition, limiting their ability to capture the full clinical context. To overcome these limitations, we propose a Progressive Spectrum Diffusion Model (PSDM) that integrates diverse clinical annotations-such as segmentation masks, bounding boxes, and colonoscopy reports-by transforming them into compositional prompts. These prompts are organized into coarse and fine components, allowing the model to capture both broad spatial structures and fine details, generating clinically accurate synthetic images. By augmenting training data with PSDM-generated samples, our model significantly improves polyp detection, classification, and segmentation. For instance, on the PolypGen dataset, PSDM increases the F1 score by 2.12% and the mean average precision by 3.09%, demonstrating superior performance in OOD scenarios and enhanced generalization.

Title: On Synthetic Data Strategies for Domain-Specific Generative Retrieval

• Authors: Haoyang Wen, Jiang Guo, Yi Zhang, Jiarong Jiang, Zhiguo Wang
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2502.17957
• Pdf URL: https://arxiv.org/pdf/2502.17957
• Copy Paste: [[2502.17957]] On Synthetic Data Strategies for Domain-Specific Generative Retrieval(https://arxiv.org/abs/2502.17957)
• Keywords: generative
• Abstract: This paper investigates synthetic data generation strategies in developing generative retrieval models for domain-specific corpora, thereby addressing the scalability challenges inherent in manually annotating in-domain queries. We study the data strategies for a two-stage training framework: in the first stage, which focuses on learning to decode document identifiers from queries, we investigate LLM-generated queries across multiple granularity (e.g. chunks, sentences) and domain-relevant search constraints that can better capture nuanced relevancy signals. In the second stage, which aims to refine document ranking through preference learning, we explore the strategies for mining hard negatives based on the initial model's predictions. Experiments on public datasets over diverse domains demonstrate the effectiveness of our synthetic data generation and hard negative sampling approach.

Title: Improved YOLOv7x-Based Defect Detection Algorithm for Power Equipment

• Authors: Jin Hou, Hao Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.17961
• Pdf URL: https://arxiv.org/pdf/2502.17961
• Copy Paste: [[2502.17961]] Improved YOLOv7x-Based Defect Detection Algorithm for Power Equipment(https://arxiv.org/abs/2502.17961)
• Keywords: anomaly
• Abstract: The normal operation of power equipment plays a critical role in the power system, making anomaly detection for power equipment highly significant. This paper proposes an improved YOLOv7x-based anomaly detection algorithm for power equipment. First, the ACmix convolutional mixed attention mechanism module is introduced to effectively suppress background noise and irrelevant features, thereby enhancing the network's feature extraction capability. Second, the Biformer attention mechanism is added to the network to strengthen the focus on key features, improving the network's ability to flexibly recognize feature images. Finally, to more comprehensively evaluate the relationship between predicted and ground truth bounding boxes, the original loss function is replaced with the MPDIoU function, addressing the issue of mismatched predicted bounding boxes. The improved algorithm enhances detection accuracy, achieving a mAP@0.5/% of 93.5% for all target categories, a precision of 97.1%, and a recall of 97%.

Title: Model-Free Adversarial Purification via Coarse-To-Fine Tensor Network Representation

• Authors: Guang Lin, Duc Thien Nguyen, Zerui Tao, Konstantinos Slavakis, Toshihisa Tanaka, Qibin Zhao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.17972
• Pdf URL: https://arxiv.org/pdf/2502.17972
• Copy Paste: [[2502.17972]] Model-Free Adversarial Purification via Coarse-To-Fine Tensor Network Representation(https://arxiv.org/abs/2502.17972)
• Keywords: generative
• Abstract: Deep neural networks are known to be vulnerable to well-designed adversarial attacks. Although numerous defense strategies have been proposed, many are tailored to the specific attacks or tasks and often fail to generalize across diverse scenarios. In this paper, we propose Tensor Network Purification (TNP), a novel model-free adversarial purification method by a specially designed tensor network decomposition algorithm. TNP depends neither on the pre-trained generative model nor the specific dataset, resulting in strong robustness across diverse adversarial scenarios. To this end, the key challenge lies in relaxing Gaussian-noise assumptions of classical decompositions and accommodating the unknown distribution of adversarial perturbations. Unlike the low-rank representation of classical decompositions, TNP aims to reconstruct the unobserved clean examples from an adversarial example. Specifically, TNP leverages progressive downsampling and introduces a novel adversarial optimization objective to address the challenge of minimizing reconstruction error but without inadvertently restoring adversarial perturbations. Extensive experiments conducted on CIFAR-10, CIFAR-100, and ImageNet demonstrate that our method generalizes effectively across various norm threats, attack types, and tasks, providing a versatile and promising adversarial purification technique.

Title: Radon-Nikodým Derivative: Re-imagining Anomaly Detection from a Measure Theoretic Perspective

• Authors: Shlok Mehendale, Aditya Challa, Rahul Yedida, Sravan Danda, Santonu Sarkar, Snehanshu Saha
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.18002
• Pdf URL: https://arxiv.org/pdf/2502.18002
• Copy Paste: [[2502.18002]] Radon-Nikodým Derivative: Re-imagining Anomaly Detection from a Measure Theoretic Perspective(https://arxiv.org/abs/2502.18002)
• Keywords: anomaly
• Abstract: Which principle underpins the design of an effective anomaly detection loss function? The answer lies in the concept of \rnthm{} theorem, a fundamental concept in measure theory. The key insight is -- Multiplying the vanilla loss function with the \rnthm{} derivative improves the performance across the board. We refer to this as RN-Loss. This is established using PAC learnability of anomaly detection. We further show that the \rnthm{} derivative offers important insights into unsupervised clustering based anomaly detections as well. We evaluate our algorithm on 96 datasets, including univariate and multivariate data from diverse domains, including healthcare, cybersecurity, and finance. We show that RN-Derivative algorithms outperform state-of-the-art methods on 68\% of Multivariate datasets (based on F-1 scores) and also achieves peak F1-scores on 72\% of time series (Univariate) datasets.

Title: Escaping The Big Data Paradigm in Self-Supervised Representation Learning

• Authors: Carlos Vélez García, Miguel Cazorla, Jorge Pomares
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.18056
• Pdf URL: https://arxiv.org/pdf/2502.18056
• Copy Paste: [[2502.18056]] Escaping The Big Data Paradigm in Self-Supervised Representation Learning(https://arxiv.org/abs/2502.18056)
• Keywords: self-supervised
• Abstract: The reliance on large-scale datasets and extensive computational resources has become a major barrier to advancing representation learning in vision, especially in data-scarce domains. In this paper, we address the critical question: Can we escape the big data paradigm in self-supervised representation learning from images? We introduce SCOTT (Sparse Convolutional Tokenizer for Transformers), a shallow tokenization architecture that is compatible with Masked Image Modeling (MIM) tasks. SCOTT injects convolutional inductive biases into Vision Transformers (ViTs), enhancing their efficacy in small-scale data regimes. Alongside, we propose to train on a Joint-Embedding Predictive Architecture within a MIM framework (MIM-JEPA), operating in latent representation space to capture more semantic features. Our approach enables ViTs to be trained from scratch on datasets orders of magnitude smaller than traditionally required --without relying on massive external datasets for pretraining. We validate our method on three small-size, standard-resoultion, fine-grained datasets: Oxford Flowers-102, Oxford IIIT Pets-37, and ImageNet-100. Despite the challenges of limited data and high intra-class similarity, frozen SCOTT models pretrained with MIM-JEPA significantly outperform fully supervised methods and achieve competitive results with SOTA approaches that rely on large-scale pretraining, complex image augmentations and bigger model sizes. By demonstrating that robust off-the-shelf representations can be learned with limited data, compute, and model sizes, our work paves the way for computer applications in resource constrained environments such as medical imaging or robotics. Our findings challenge the prevailing notion that vast amounts of data are indispensable for effective representation learning in vision, offering a new pathway toward more accessible and inclusive advancements in the field.

Title: Examining the Threat Landscape: Foundation Models and Model Stealing

• Authors: Ankita Raj, Deepankar Varma, Chetan Arora
• Subjects: cs.CV, cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2502.18077
• Pdf URL: https://arxiv.org/pdf/2502.18077
• Copy Paste: [[2502.18077]] Examining the Threat Landscape: Foundation Models and Model Stealing(https://arxiv.org/abs/2502.18077)
• Keywords: foundation model
• Abstract: Foundation models (FMs) for computer vision learn rich and robust representations, enabling their adaptation to task/domain-specific deployments with little to no fine-tuning. However, we posit that the very same strength can make applications based on FMs vulnerable to model stealing attacks. Through empirical analysis, we reveal that models fine-tuned from FMs harbor heightened susceptibility to model stealing, compared to conventional vision architectures like ResNets. We hypothesize that this behavior is due to the comprehensive encoding of visual patterns and features learned by FMs during pre-training, which are accessible to both the attacker and the victim. We report that an attacker is able to obtain 94.28% agreement (matched predictions with victim) for a Vision Transformer based victim model (ViT-L/16) trained on CIFAR-10 dataset, compared to only 73.20% agreement for a ResNet-18 victim, when using ViT-L/16 as the thief model. We arguably show, for the first time, that utilizing FMs for downstream tasks may not be the best choice for deployment in commercial APIs due to their susceptibility to model theft. We thereby alert model owners towards the associated security risks, and highlight the need for robust security measures to safeguard such models against theft. Code is available at this https URL.

Title: PromptMID: Modal Invariant Descriptors Based on Diffusion and Vision Foundation Models for Optical-SAR Image Matching

• Authors: Han Nie, Bin Luo, Jun Liu, Zhitao Fu, Huan Zhou, Shuo Zhang, Weixing Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.18104
• Pdf URL: https://arxiv.org/pdf/2502.18104
• Copy Paste: [[2502.18104]] PromptMID: Modal Invariant Descriptors Based on Diffusion and Vision Foundation Models for Optical-SAR Image Matching(https://arxiv.org/abs/2502.18104)
• Keywords: diffusion, foundation model
• Abstract: The ideal goal of image matching is to achieve stable and efficient performance in unseen domains. However, many existing learning-based optical-SAR image matching methods, despite their effectiveness in specific scenarios, exhibit limited generalization and struggle to adapt to practical applications. Repeatedly training or fine-tuning matching models to address domain differences is not only not elegant enough but also introduces additional computational overhead and data production costs. In recent years, general foundation models have shown great potential for enhancing generalization. However, the disparity in visual domains between natural and remote sensing images poses challenges for their direct application. Therefore, effectively leveraging foundation models to improve the generalization of optical-SAR image matching remains challenge. To address the above challenges, we propose PromptMID, a novel approach that constructs modality-invariant descriptors using text prompts based on land use classification as priors information for optical and SAR image matching. PromptMID extracts multi-scale modality-invariant features by leveraging pre-trained diffusion models and visual foundation models (VFMs), while specially designed feature aggregation modules effectively fuse features across different granularities. Extensive experiments on optical-SAR image datasets from four diverse regions demonstrate that PromptMID outperforms state-of-the-art matching methods, achieving superior results in both seen and unseen domains and exhibiting strong cross-domain generalization capabilities. The source code will be made publicly available this https URL.

Title: Actively Inferring Optimal Measurement Sequences

• Authors: Catherine F. Higham, Paul Henderson, Roderick Murray-Smith
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.18142
• Pdf URL: https://arxiv.org/pdf/2502.18142
• Copy Paste: [[2502.18142]] Actively Inferring Optimal Measurement Sequences(https://arxiv.org/abs/2502.18142)
• Keywords: generative
• Abstract: Measurement of a physical quantity such as light intensity is an integral part of many reconstruction and decision scenarios but can be costly in terms of acquisition time, invasion of or damage to the environment and storage. Data minimisation and compliance with data protection laws is also an important consideration. Where there are a range of measurements that can be made, some may be more informative and compliant with the overall measurement objective than others. We develop an active sequential inference algorithm that uses the low dimensional representational latent space from a variational autoencoder (VAE) to choose which measurement to make next. Our aim is to recover high dimensional data by making as few measurements as possible. We adapt the VAE encoder to map partial data measurements on to the latent space of the complete data. The algorithm draws samples from this latent space and uses the VAE decoder to generate data conditional on the partial measurements. Estimated measurements are made on the generated data and fed back through the partial VAE encoder to the latent space where they can be evaluated prior to making a measurement. Starting from no measurements and a normal prior on the latent space, we consider alternative strategies for choosing the next measurement and updating the predictive posterior prior for the next step. The algorithm is illustrated using the Fashion MNIST dataset and a novel convolutional Hadamard pattern measurement basis. We see that useful patterns are chosen within 10 steps, leading to the convergence of the guiding generative images. Compared with using stochastic variational inference to infer the parameters of the posterior distribution for each generated data point individually, the partial VAE framework can efficiently process batches of generated data and obtains superior results with minimal measurements.

Title: Joint Reconstruction of Spatially-Coherent and Realistic Clothed Humans and Objects from a Single Image

• Authors: Ayushi Dutta, Marco Pesavento, Marco Volino, Adrian Hilton, Armin Mustafa
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.18150
• Pdf URL: https://arxiv.org/pdf/2502.18150
• Copy Paste: [[2502.18150]] Joint Reconstruction of Spatially-Coherent and Realistic Clothed Humans and Objects from a Single Image(https://arxiv.org/abs/2502.18150)
• Keywords: diffusion, generative
• Abstract: Recent advances in human shape learning have focused on achieving accurate human reconstruction from single-view images. However, in the real world, humans share space with other objects. Reconstructing images with humans and objects is challenging due to the occlusions and lack of 3D spatial awareness, which leads to depth ambiguity in the reconstruction. Existing methods in monocular human-object reconstruction fail to capture intricate details of clothed human bodies and object surfaces due to their template-based nature. In this paper, we jointly reconstruct clothed humans and objects in a spatially coherent manner from single-view images, while addressing human-object occlusions. A novel attention-based neural implicit model is proposed that leverages image pixel alignment to retrieve high-quality details, and incorporates semantic features extracted from the human-object pose to enable 3D spatial awareness. A generative diffusion model is used to handle human-object occlusions. For training and evaluation, we introduce a synthetic dataset with rendered scenes of inter-occluded 3D human scans and diverse objects. Extensive evaluation on both synthetic and real datasets demonstrates the superior quality of proposed human-object reconstructions over competitive methods.

Title: CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classification

• Authors: Mingkun Zhang, Keping Bi, Wei Chen, Jiafeng Guo, Xueqi Cheng
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2502.18176
• Pdf URL: https://arxiv.org/pdf/2502.18176
• Copy Paste: [[2502.18176]] CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classification(https://arxiv.org/abs/2502.18176)
• Keywords: diffusion, generative
• Abstract: In this paper, we aim to build an adversarially robust zero-shot image classifier. We ground our work on CLIP, a vision-language pre-trained encoder model that can perform zero-shot classification by matching an image with text prompts ``a photo of a .''. Purification is the path we choose since it does not require adversarial training on specific attack types and thus can cope with any foreseen attacks. We then formulate purification risk as the KL divergence between the joint distributions of the purification process of denoising the adversarial samples and the attack process of adding perturbations to benign samples, through bidirectional Stochastic Differential Equations (SDEs). The final derived results inspire us to explore purification in the multi-modal latent space of CLIP. We propose two variants for our CLIPure approach: CLIPure-Diff which models the likelihood of images' latent vectors with the DiffusionPrior module in DaLLE-2 (modeling the generation process of CLIP's latent vectors), and CLIPure-Cos which models the likelihood with the cosine similarity between the embeddings of an image and ``a photo of a.''. As far as we know, CLIPure is the first purification method in multi-modal latent space and CLIPure-Cos is the first purification method that is not based on generative models, which substantially improves defense efficiency. We conducted extensive experiments on CIFAR-10, ImageNet, and 13 datasets that previous CLIP-based defense methods used for evaluating zero-shot classification robustness. Results show that CLIPure boosts the SOTA robustness by a large margin, e.g., from 71.7% to 91.1% on CIFAR10, from 59.6% to 72.6% on ImageNet, and 108% relative improvements of average robustness on the 13 datasets over previous SOTA. The code is available at this https URL.

Title: Multi-Perspective Data Augmentation for Few-shot Object Detection

• Authors: Anh-Khoa Nguyen Vu, Quoc-Truong Truong, Vinh-Tiep Nguyen, Thanh Duc Ngo, Thanh-Toan Do, Tam V. Nguyen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.18195
• Pdf URL: https://arxiv.org/pdf/2502.18195
• Copy Paste: [[2502.18195]] Multi-Perspective Data Augmentation for Few-shot Object Detection(https://arxiv.org/abs/2502.18195)
• Keywords: diffusion, in-context
• Abstract: Recent few-shot object detection (FSOD) methods have focused on augmenting synthetic samples for novel classes, show promising results to the rise of diffusion models. However, the diversity of such datasets is often limited in representativeness because they lack awareness of typical and hard samples, especially in the context of foreground and background relationships. To tackle this issue, we propose a Multi-Perspective Data Augmentation (MPAD) framework. In terms of foreground-foreground relationships, we propose in-context learning for object synthesis (ICOS) with bounding box adjustments to enhance the detail and spatial information of synthetic samples. Inspired by the large margin principle, support samples play a vital role in defining class boundaries. Therefore, we design a Harmonic Prompt Aggregation Scheduler (HPAS) to mix prompt embeddings at each time step of the generation process in diffusion models, producing hard novel samples. For foreground-background relationships, we introduce a Background Proposal method (BAP) to sample typical and hard backgrounds. Extensive experiments on multiple FSOD benchmarks demonstrate the effectiveness of our approach. Our framework significantly outperforms traditional methods, achieving an average increase of $17.5\%$ in nAP50 over the baseline on PASCAL VOC. Code is available at this https URL.

Title: Training Consistency Models with Variational Noise Coupling

• Authors: Gianluigi Silvestri, Luca Ambrogioni, Chieh-Hsin Lai, Yuhta Takida, Yuki Mitsufuji
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2502.18197
• Pdf URL: https://arxiv.org/pdf/2502.18197
• Copy Paste: [[2502.18197]] Training Consistency Models with Variational Noise Coupling(https://arxiv.org/abs/2502.18197)
• Keywords: diffusion, generative
• Abstract: Consistency Training (CT) has recently emerged as a promising alternative to diffusion models, achieving competitive performance in image generation tasks. However, non-distillation consistency training often suffers from high variance and instability, and analyzing and improving its training dynamics is an active area of research. In this work, we propose a novel CT training approach based on the Flow Matching framework. Our main contribution is a trained noise-coupling scheme inspired by the architecture of Variational Autoencoders (VAE). By training a data-dependent noise emission model implemented as an encoder architecture, our method can indirectly learn the geometry of the noise-to-data mapping, which is instead fixed by the choice of the forward process in classical CT. Empirical results across diverse image datasets show significant generative improvements, with our model outperforming baselines and achieving the state-of-the-art (SoTA) non-distillation CT FID on CIFAR-10, and attaining FID on par with SoTA on ImageNet at $64 \times 64$ resolution in 2-step generation. Our code is available at this https URL .

Title: Synthesizing Consistent Novel Views via 3D Epipolar Attention without Re-Training

• Authors: Botao Ye, Sifei Liu, Xueting Li, Marc Pollefeys, Ming-Hsuan Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.18219
• Pdf URL: https://arxiv.org/pdf/2502.18219
• Copy Paste: [[2502.18219]] Synthesizing Consistent Novel Views via 3D Epipolar Attention without Re-Training(https://arxiv.org/abs/2502.18219)
• Keywords: diffusion
• Abstract: Large diffusion models demonstrate remarkable zero-shot capabilities in novel view synthesis from a single image. However, these models often face challenges in maintaining consistency across novel and reference views. A crucial factor leading to this issue is the limited utilization of contextual information from reference views. Specifically, when there is an overlap in the viewing frustum between two views, it is essential to ensure that the corresponding regions maintain consistency in both geometry and appearance. This observation leads to a simple yet effective approach, where we propose to use epipolar geometry to locate and retrieve overlapping information from the input view. This information is then incorporated into the generation of target views, eliminating the need for training or fine-tuning, as the process requires no learnable parameters. Furthermore, to enhance the overall consistency of generated views, we extend the utilization of epipolar attention to a multi-view setting, allowing retrieval of overlapping information from the input view and other target views. Qualitative and quantitative experimental results demonstrate the effectiveness of our method in significantly improving the consistency of synthesized views without the need for any fine-tuning. Moreover, This enhancement also boosts the performance of downstream applications such as 3D reconstruction. The code is available at this https URL.

Title: Beyond the convexity assumption: Realistic tabular data generation under quantifier-free real linear constraints

• Authors: Mihaela Cătălina Stoian, Eleonora Giunchiglia
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.18237
• Pdf URL: https://arxiv.org/pdf/2502.18237
• Copy Paste: [[2502.18237]] Beyond the convexity assumption: Realistic tabular data generation under quantifier-free real linear constraints(https://arxiv.org/abs/2502.18237)
• Keywords: generative
• Abstract: Synthetic tabular data generation has traditionally been a challenging problem due to the high complexity of the underlying distributions that characterise this type of data. Despite recent advances in deep generative models (DGMs), existing methods often fail to produce realistic datapoints that are well-aligned with available background knowledge. In this paper, we address this limitation by introducing Disjunctive Refinement Layer (DRL), a novel layer designed to enforce the alignment of generated data with the background knowledge specified in user-defined constraints. DRL is the first method able to automatically make deep learning models inherently compliant with constraints as expressive as quantifier-free linear formulas, which can define non-convex and even disconnected spaces. Our experimental analysis shows that DRL not only guarantees constraint satisfaction but also improves efficacy in downstream tasks. Notably, when applied to DGMs that frequently violate constraints, DRL eliminates violations entirely. Further, it improves performance metrics by up to 21.4% in F1-score and 20.9% in Area Under the ROC Curve, thus demonstrating its practical impact on data generation.

Title: Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Models

• Authors: Zhaoyi Liu, Huan Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.18290
• Pdf URL: https://arxiv.org/pdf/2502.18290
• Copy Paste: [[2502.18290]] Stealthy Backdoor Attack in Self-Supervised Learning Vision Encoders for Large Vision Language Models(https://arxiv.org/abs/2502.18290)
• Keywords: self-supervised
• Abstract: Self-supervised learning (SSL) vision encoders learn high-quality image representations and thus have become a vital part of developing vision modality of large vision language models (LVLMs). Due to the high cost of training such encoders, pre-trained encoders are widely shared and deployed into many LVLMs, which are security-critical or bear societal significance. Under this practical scenario, we reveal a new backdoor threat that significant visual hallucinations can be induced into these LVLMs by merely compromising vision encoders. Because of the sharing and reuse of these encoders, many downstream LVLMs may inherit backdoor behaviors from encoders, leading to widespread backdoors. In this work, we propose BadVision, the first method to exploit this vulnerability in SSL vision encoders for LVLMs with novel trigger optimization and backdoor learning techniques. We evaluate BadVision on two types of SSL encoders and LVLMs across eight benchmarks. We show that BadVision effectively drives the LVLMs to attacker-chosen hallucination with over 99% attack success rate, causing a 77.6% relative visual understanding error while maintaining the stealthiness. SoTA backdoor detection methods cannot detect our attack effectively.

Title: Bayesian Computation in Deep Learning

• Authors: Wenlong Chen, Bolian Li, Ruqi Zhang, Yingzhen Li
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2502.18300
• Pdf URL: https://arxiv.org/pdf/2502.18300
• Copy Paste: [[2502.18300]] Bayesian Computation in Deep Learning(https://arxiv.org/abs/2502.18300)
• Keywords: generative
• Abstract: This review paper is intended for the 2nd edition of the Handbook of Markov chain Monte this http URL provide an introduction to approximate inference techniques as Bayesian computation methods applied to deep learning models. We organize the chapter by presenting popular computational methods for (1) Bayesian neural networks and (2) deep generative models, explaining their unique challenges in posterior inference as well as the solutions.

Title: LDGen: Enhancing Text-to-Image Synthesis via Large Language Model-Driven Language Representation

• Authors: Pengzhi Li, Pengfei Yu, Zide Liu, Wei He, Xuhao Pan, Xudong Rao, Tao Wei, Wei Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.18302
• Pdf URL: https://arxiv.org/pdf/2502.18302
• Copy Paste: [[2502.18302]] LDGen: Enhancing Text-to-Image Synthesis via Large Language Model-Driven Language Representation(https://arxiv.org/abs/2502.18302)
• Keywords: diffusion
• Abstract: In this paper, we introduce LDGen, a novel method for integrating large language models (LLMs) into existing text-to-image diffusion models while minimizing computational demands. Traditional text encoders, such as CLIP and T5, exhibit limitations in multilingual processing, hindering image generation across diverse languages. We address these challenges by leveraging the advanced capabilities of LLMs. Our approach employs a language representation strategy that applies hierarchical caption optimization and human instruction techniques to derive precise semantic information,. Subsequently, we incorporate a lightweight adapter and a cross-modal refiner to facilitate efficient feature alignment and interaction between LLMs and image features. LDGen reduces training time and enables zero-shot multilingual image generation. Experimental results indicate that our method surpasses baseline models in both prompt adherence and image aesthetic quality, while seamlessly supporting multiple languages. Project page: this https URL.

Title: Self-Supervised Data Generation for Precision Agriculture: Blending Simulated Environments with Real Imagery

• Authors: Leonardo Saraceni, Ionut Marian Motoi, Daniele Nardi, Thomas Alessandro Ciarfuglia
• Subjects: cs.CV, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2502.18320
• Pdf URL: https://arxiv.org/pdf/2502.18320
• Copy Paste: [[2502.18320]] Self-Supervised Data Generation for Precision Agriculture: Blending Simulated Environments with Real Imagery(https://arxiv.org/abs/2502.18320)
• Keywords: self-supervised
• Abstract: In precision agriculture, the scarcity of labeled data and significant covariate shifts pose unique challenges for training machine learning models. This scarcity is particularly problematic due to the dynamic nature of the environment and the evolving appearance of agricultural subjects as living things. We propose a novel system for generating realistic synthetic data to address these challenges. Utilizing a vineyard simulator based on the Unity engine, our system employs a cut-and-paste technique with geometrical consistency considerations to produce accurate photo-realistic images and labels from synthetic environments to train detection algorithms. This approach generates diverse data samples across various viewpoints and lighting conditions. We demonstrate considerable performance improvements in training a state-of-the-art detector by applying our method to table grapes cultivation. The combination of techniques can be easily automated, an increasingly important consideration for adoption in agricultural practice.

Title: ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation

• Authors: Yifan Pu, Yiming Zhao, Zhicong Tang, Ruihong Yin, Haoxing Ye, Yuhui Yuan, Dong Chen, Jianmin Bao, Sirui Zhang, Yanbin Wang, Lin Liang, Lijuan Wang, Ji Li, Xiu Li, Zhouhui Lian, Gao Huang, Baining Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.18364
• Pdf URL: https://arxiv.org/pdf/2502.18364
• Copy Paste: [[2502.18364]] ART: Anonymous Region Transformer for Variable Multi-Layer Transparent Image Generation(https://arxiv.org/abs/2502.18364)
• Keywords: generative
• Abstract: Multi-layer image generation is a fundamental task that enables users to isolate, select, and edit specific image layers, thereby revolutionizing interactions with generative models. In this paper, we introduce the Anonymous Region Transformer (ART), which facilitates the direct generation of variable multi-layer transparent images based on a global text prompt and an anonymous region layout. Inspired by Schema theory suggests that knowledge is organized in frameworks (schemas) that enable people to interpret and learn from new information by linking it to prior knowledge.}, this anonymous region layout allows the generative model to autonomously determine which set of visual tokens should align with which text tokens, which is in contrast to the previously dominant semantic layout for the image generation task. In addition, the layer-wise region crop mechanism, which only selects the visual tokens belonging to each anonymous region, significantly reduces attention computation costs and enables the efficient generation of images with numerous distinct layers (e.g., 50+). When compared to the full attention approach, our method is over 12 times faster and exhibits fewer layer conflicts. Furthermore, we propose a high-quality multi-layer transparent image autoencoder that supports the direct encoding and decoding of the transparency of variable multi-layer images in a joint manner. By enabling precise control and scalable layer generation, ART establishes a new paradigm for interactive content creation.

Title: Mechanistic PDE Networks for Discovery of Governing Equations

• Authors: Adeel Pervez, Efstratios Gavves, Francesco Locatello
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.18377
• Pdf URL: https://arxiv.org/pdf/2502.18377
• Copy Paste: [[2502.18377]] Mechanistic PDE Networks for Discovery of Governing Equations(https://arxiv.org/abs/2502.18377)
• Keywords: diffusion
• Abstract: We present Mechanistic PDE Networks -- a model for discovery of governing partial differential equations from data. Mechanistic PDE Networks represent spatiotemporal data as space-time dependent linear partial differential equations in neural network hidden representations. The represented PDEs are then solved and decoded for specific tasks. The learned PDE representations naturally express the spatiotemporal dynamics in data in neural network hidden space, enabling increased power for dynamical modeling. Solving the PDE representations in a compute and memory-efficient way, however, is a significant challenge. We develop a native, GPU-capable, parallel, sparse, and differentiable multigrid solver specialized for linear partial differential equations that acts as a module in Mechanistic PDE Networks. Leveraging the PDE solver, we propose a discovery architecture that can discover nonlinear PDEs in complex settings while also being robust to noise. We validate PDE discovery on a number of PDEs, including reaction-diffusion and Navier-Stokes equations.

Title: Enhancing DNA Foundation Models to Address Masking Inefficiencies

• Authors: Monireh Safari, Pablo Millan Arias, Scott C. Lowe, Lila Kari, Angel X. Chang, Graham W. Taylor
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2502.18405
• Pdf URL: https://arxiv.org/pdf/2502.18405
• Copy Paste: [[2502.18405]] Enhancing DNA Foundation Models to Address Masking Inefficiencies(https://arxiv.org/abs/2502.18405)
• Keywords: foundation model
• Abstract: Masked language modelling (MLM) as a pretraining objective has been widely adopted in genomic sequence modelling. While pretrained models can successfully serve as encoders for various downstream tasks, the distribution shift between pretraining and inference detrimentally impacts performance, as the pretraining task is to map [MASK] tokens to predictions, yet the [MASK] is absent during downstream applications. This means the encoder does not prioritize its encodings of non-[MASK] tokens, and expends parameters and compute on work only relevant to the MLM task, despite this being irrelevant at deployment time. In this work, we propose a modified encoder-decoder architecture based on the masked autoencoder framework, designed to address this inefficiency within a BERT-based transformer. We empirically show that the resulting mismatch is particularly detrimental in genomic pipelines where models are often used for feature extraction without fine-tuning. We evaluate our approach on the BIOSCAN-5M dataset, comprising over 2 million unique DNA barcodes. We achieve substantial performance gains in both closed-world and open-world classification tasks when compared against causal models and bidirectional architectures pretrained with MLM tasks.

Title: GHOST 2.0: generative high-fidelity one shot transfer of heads

• Authors: Alexander Groshev (1), Anastasiia Iashchenko (1), Pavel Paramonov (1), Denis Dimitrov (1 and 2), Andrey Kuznetsov (1 and 2) ((1) SberAI, (2) AIRI)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.18417
• Pdf URL: https://arxiv.org/pdf/2502.18417
• Copy Paste: [[2502.18417]] GHOST 2.0: generative high-fidelity one shot transfer of heads(https://arxiv.org/abs/2502.18417)
• Keywords: generative
• Abstract: While the task of face swapping has recently gained attention in the research community, a related problem of head swapping remains largely unexplored. In addition to skin color transfer, head swap poses extra challenges, such as the need to preserve structural information of the whole head during synthesis and inpaint gaps between swapped head and background. In this paper, we address these concerns with GHOST 2.0, which consists of two problem-specific modules. First, we introduce enhanced Aligner model for head reenactment, which preserves identity information at multiple scales and is robust to extreme pose variations. Secondly, we use a Blender module that seamlessly integrates the reenacted head into the target background by transferring skin color and inpainting mismatched regions. Both modules outperform the baselines on the corresponding tasks, allowing to achieve state of the art results in head swapping. We also tackle complex cases, such as large difference in hair styles of source and target.

Title: K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAs

• Authors: Ziheng Ouyang, Zhen Li, Qibin Hou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2502.18461
• Pdf URL: https://arxiv.org/pdf/2502.18461
• Copy Paste: [[2502.18461]] K-LoRA: Unlocking Training-Free Fusion of Any Subject and Style LoRAs(https://arxiv.org/abs/2502.18461)
• Keywords: diffusion
• Abstract: Recent studies have explored combining different LoRAs to jointly generate learned style and content. However, existing methods either fail to effectively preserve both the original subject and style simultaneously or require additional training. In this paper, we argue that the intrinsic properties of LoRA can effectively guide diffusion models in merging learned subject and style. Building on this insight, we propose K-LoRA, a simple yet effective training-free LoRA fusion approach. In each attention layer, K-LoRA compares the Top-K elements in each LoRA to be fused, determining which LoRA to select for optimal fusion. This selection mechanism ensures that the most representative features of both subject and style are retained during the fusion process, effectively balancing their contributions. Experimental results demonstrate that the proposed method effectively integrates the subject and style information learned by the original LoRAs, outperforming state-of-the-art training-based approaches in both qualitative and quantitative results.