2024-09-30

Title: SSP-RACL: Classification of Noisy Fundus Images with Self-Supervised Pretraining and Robust Adaptive Credal Loss

• Authors: Mengwen Ye, Yingzi Huangfu, You Li, Zekuan Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18147
• Pdf URL: https://arxiv.org/pdf/2409.18147
• Copy Paste: [[2409.18147]] SSP-RACL: Classification of Noisy Fundus Images with Self-Supervised Pretraining and Robust Adaptive Credal Loss(https://arxiv.org/abs/2409.18147)
• Keywords: self-supervised
• Abstract: Fundus image classification is crucial in the computer aided diagnosis tasks, but label noise significantly impairs the performance of deep neural networks. To address this challenge, we propose a robust framework, Self-Supervised Pre-training with Robust Adaptive Credal Loss (SSP-RACL), for handling label noise in fundus image datasets. First, we use Masked Autoencoders (MAE) for pre-training to extract features, unaffected by label noise. Subsequently, RACL employ a superset learning framework, setting confidence thresholds and adaptive label relaxation parameter to construct possibility distributions and provide more reliable ground-truth estimates, thus effectively suppressing the memorization effect. Additionally, we introduce clinical knowledge-based asymmetric noise generation to simulate real-world noisy fundus image datasets. Experimental results demonstrate that our proposed method outperforms existing approaches in handling label noise, showing superior performance.

Title: Jump Diffusion-Informed Neural Networks with Transfer Learning for Accurate American Option Pricing under Data Scarcity

• Authors: Qiguo Sun, Hanyue Huang, XiBei Yang, Yuwei Zhang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18168
• Pdf URL: https://arxiv.org/pdf/2409.18168
• Copy Paste: [[2409.18168]] Jump Diffusion-Informed Neural Networks with Transfer Learning for Accurate American Option Pricing under Data Scarcity(https://arxiv.org/abs/2409.18168)
• Keywords: diffusion
• Abstract: Option pricing models, essential in financial mathematics and risk management, have been extensively studied and recently advanced by AI methodologies. However, American option pricing remains challenging due to the complexity of determining optimal exercise times and modeling non-linear payoffs resulting from stochastic paths. Moreover, the prevalent use of the Black-Scholes formula in hybrid models fails to accurately capture the discontinuity in the price process, limiting model performance, especially under scarce data conditions. To address these issues, this study presents a comprehensive framework for American option pricing consisting of six interrelated modules, which combine nonlinear optimization algorithms, analytical and numerical models, and neural networks to improve pricing performance. Additionally, to handle the scarce data challenge, this framework integrates the transfer learning through numerical data augmentation and a physically constrained, jump diffusion process-informed neural network to capture the leptokurtosis of the log return distribution. To increase training efficiency, a warm-up period using Bayesian optimization is designed to provide optimal data loss and physical loss coefficients. Experimental results of six case studies demonstrate the accuracy, convergence, physical effectiveness, and generalization of the framework. Moreover, the proposed model shows superior performance in pricing deep out-of-the-money options.

Title: Evaluation of Security of ML-based Watermarking: Copy and Removal Attacks

• Authors: Vitaliy Kinakh, Brian Pulfer, Yury Belousov, Pierre Fernandez, Teddy Furon, Slava Voloshynovskiy
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18211
• Pdf URL: https://arxiv.org/pdf/2409.18211
• Copy Paste: [[2409.18211]] Evaluation of Security of ML-based Watermarking: Copy and Removal Attacks(https://arxiv.org/abs/2409.18211)
• Keywords: foundation model
• Abstract: The vast amounts of digital content captured from the real world or AI-generated media necessitate methods for copyright protection, traceability, or data provenance verification. Digital watermarking serves as a crucial approach to address these challenges. Its evolution spans three generations: handcrafted, autoencoder-based, and foundation model based methods. %Its evolution spans three generations: handcrafted methods, autoencoder-based schemes, and methods based on foundation models. While the robustness of these systems is well-documented, the security against adversarial attacks remains underexplored. This paper evaluates the security of foundation models' latent space digital watermarking systems that utilize adversarial embedding techniques. A series of experiments investigate the security dimensions under copy and removal attacks, providing empirical insights into these systems' vulnerabilities. All experimental codes and results are available at this https URL}{repository

Title: Trustworthy Text-to-Image Diffusion Models: A Timely and Focused Survey

• Authors: Yi Zhang, Zhen Chen, Chih-Hong Cheng, Wenjie Ruan, Xiaowei Huang, Dezong Zhao, David Flynn, Siddartha Khastgir, Xingyu Zhao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18214
• Pdf URL: https://arxiv.org/pdf/2409.18214
• Copy Paste: [[2409.18214]] Trustworthy Text-to-Image Diffusion Models: A Timely and Focused Survey(https://arxiv.org/abs/2409.18214)
• Keywords: diffusion
• Abstract: Text-to-Image (T2I) Diffusion Models (DMs) have garnered widespread attention for their impressive advancements in image generation. However, their growing popularity has raised ethical and social concerns related to key non-functional properties of trustworthiness, such as robustness, fairness, security, privacy, factuality, and explainability, similar to those in traditional deep learning (DL) tasks. Conventional approaches for studying trustworthiness in DL tasks often fall short due to the unique characteristics of T2I DMs, e.g., the multi-modal nature. Given the challenge, recent efforts have been made to develop new methods for investigating trustworthiness in T2I DMs via various means, including falsification, enhancement, verification \& validation and assessment. However, there is a notable lack of in-depth analysis concerning those non-functional properties and means. In this survey, we provide a timely and focused review of the literature on trustworthy T2I DMs, covering a concise-structured taxonomy from the perspectives of property, means, benchmarks and applications. Our review begins with an introduction to essential preliminaries of T2I DMs, and then we summarise key definitions/metrics specific to T2I tasks and analyses the means proposed in recent literature based on these definitions/metrics. Additionally, we review benchmarks and domain applications of T2I DMs. Finally, we highlight the gaps in current research, discuss the limitations of existing methods, and propose future research directions to advance the development of trustworthy T2I DMs. Furthermore, we keep up-to-date updates in this field to track the latest developments and maintain our GitHub repository at: this https URL

Title: Revolutionizing Payload Inspection: A Self-Supervised Journey to Precision with Few Shots

• Authors: Kyle Stein, Arash Mahyari, Guillermo Francia III, Eman El-Sheikh
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18219
• Pdf URL: https://arxiv.org/pdf/2409.18219
• Copy Paste: [[2409.18219]] Revolutionizing Payload Inspection: A Self-Supervised Journey to Precision with Few Shots(https://arxiv.org/abs/2409.18219)
• Keywords: self-supervised
• Abstract: As networks continue to expand and become more interconnected, the need for novel malware detection methods becomes more pronounced. Traditional security measures are increasingly inadequate against the sophistication of modern cyber attacks. Deep Packet Inspection (DPI) has been pivotal in enhancing network security, offering an in-depth analysis of network traffic that surpasses conventional monitoring techniques. DPI not only examines the metadata of network packets, but also dives into the actual content being carried within the packet payloads, providing a comprehensive view of the data flowing through networks. The integration of advanced deep learning techniques with DPI has introduced modern methodologies into malware detection. However, the challenge with the state-of-the-art supervised learning approaches is that they prevent the generalization to unseen attacks embedded in the payloads, prohibiting them from accurately detecting new attacks and transferring knowledge learned from previous attacks to the new attacks with small labeled sample sizes. This paper leverages the recent advancements in self-supervised learning and few-shot learning. Our proposed self-supervised approach trains a transformer to learn the embedding of the payloads from a vast amount of unlabeled datasets by masking portions of payloads, leading to a learnt representation that well generalizes to various downstream tasks. Once the representation is extracted from payloads, they are used to train a malware detection algorithm. The representation obtained from the transformer is then used to adapt the malware detector to novel types of attacks using few-shot learning approaches. Our experimental results across several datasets show the great success and generalization of the proposed approach to novel scenarios.

Title: Analysis of Spatial augmentation in Self-supervised models in the purview of training and test distributions

• Authors: Abhishek Jha, Tinne Tuytelaars
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18228
• Pdf URL: https://arxiv.org/pdf/2409.18228
• Copy Paste: [[2409.18228]] Analysis of Spatial augmentation in Self-supervised models in the purview of training and test distributions(https://arxiv.org/abs/2409.18228)
• Keywords: self-supervised
• Abstract: In this paper, we present an empirical study of typical spatial augmentation techniques used in self-supervised representation learning methods (both contrastive and non-contrastive), namely random crop and cutout. Our contributions are: (a) we dissociate random cropping into two separate augmentations, overlap and patch, and provide a detailed analysis on the effect of area of overlap and patch size to the accuracy on down stream tasks. (b) We offer an insight into why cutout augmentation does not learn good representation, as reported in earlier literature. Finally, based on these analysis, (c) we propose a distance-based margin to the invariance loss for learning scene-centric representations for the downstream task on object-centric distribution, showing that as simple as a margin proportional to the pixel distance between the two spatial views in the scence-centric images can improve the learned representation. Our study furthers the understanding of the spatial augmentations, and the effect of the domain-gap between the training augmentations and the test distribution.

Title: Bridging the Protection Gap: Innovative Approaches to Shield Older Adults from AI-Enhanced Scams

• Authors: LD Herrera, London Van Sickle, Ashley Podhradsky
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.18249
• Pdf URL: https://arxiv.org/pdf/2409.18249
• Copy Paste: [[2409.18249]] Bridging the Protection Gap: Innovative Approaches to Shield Older Adults from AI-Enhanced Scams(https://arxiv.org/abs/2409.18249)
• Keywords: generative
• Abstract: Artificial Intelligence (AI) is rapidly gaining popularity as individuals, groups, and organizations discover and apply its expanding capabilities. Generative AI creates or alters various content types including text, image, audio, and video that are realistic and challenging to identify as AI-generated constructs. However, guardrails preventing malicious use of AI are easily bypassed. Numerous indications suggest that scammers are already using AI to enhance already successful scams, improving scam effectiveness, speed and credibility, while reducing detectability of scams that target older adults, who are known to be slow to adopt new technologies. Through hypothetical cases analysis of two leading scams, the tech support scams and the romance scams, this paper explores the future of AI in scams affecting older adults by identifying current vulnerabilities and recommending updated defensive measures focusing the establishment of a reliable support network offering elevated support to increase confidence and ability to defend against AI-enhanced scams.

Title: Amodal Instance Segmentation with Diffusion Shape Prior Estimation

• Authors: Minh Tran, Khoa Vo, Tri Nguyen, Ngan Le
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18256
• Pdf URL: https://arxiv.org/pdf/2409.18256
• Copy Paste: [[2409.18256]] Amodal Instance Segmentation with Diffusion Shape Prior Estimation(https://arxiv.org/abs/2409.18256)
• Keywords: diffusion
• Abstract: Amodal Instance Segmentation (AIS) presents an intriguing challenge, including the segmentation prediction of both visible and occluded parts of objects within images. Previous methods have often relied on shape prior information gleaned from training data to enhance amodal segmentation. However, these approaches are susceptible to overfitting and disregard object category details. Recent advancements highlight the potential of conditioned diffusion models, pretrained on extensive datasets, to generate images from latent space. Drawing inspiration from this, we propose AISDiff with a Diffusion Shape Prior Estimation (DiffSP) module. AISDiff begins with the prediction of the visible segmentation mask and object category, alongside occlusion-aware processing through the prediction of occluding masks. Subsequently, these elements are inputted into our DiffSP module to infer the shape prior of the object. DiffSP utilizes conditioned diffusion models pretrained on extensive datasets to extract rich visual features for shape prior estimation. Additionally, we introduce the Shape Prior Amodal Predictor, which utilizes attention-based feature maps from the shape prior to refine amodal segmentation. Experiments across various AIS benchmarks demonstrate the effectiveness of our AISDiff.

Title: Causality-based Subject and Task Fingerprints using fMRI Time-series Data

• Authors: Dachuan Song, Li Shen, Duy Duong-Tran, Xuan Wang
• Subjects: cs.LG, eess.SY
• Abstract URL: https://arxiv.org/abs/2409.18298
• Pdf URL: https://arxiv.org/pdf/2409.18298
• Copy Paste: [[2409.18298]] Causality-based Subject and Task Fingerprints using fMRI Time-series Data(https://arxiv.org/abs/2409.18298)
• Keywords: generative
• Abstract: Recently, there has been a revived interest in system neuroscience causation models due to their unique capability to unravel complex relationships in multi-scale brain networks. In this paper, our goal is to verify the feasibility and effectiveness of using a causality-based approach for fMRI fingerprinting. Specifically, we propose an innovative method that utilizes the causal dynamics activities of the brain to identify the unique cognitive patterns of individuals (e.g., subject fingerprint) and fMRI tasks (e.g., task fingerprint). The key novelty of our approach stems from the development of a two-timescale linear state-space model to extract 'spatio-temporal' (aka causal) signatures from an individual's fMRI time series data. To the best of our knowledge, we pioneer and subsequently quantify, in this paper, the concept of 'causal fingerprint.' Our method is well-separated from other fingerprint studies as we quantify fingerprints from a cause-and-effect perspective, which are then incorporated with a modal decomposition and projection method to perform subject identification and a GNN-based (Graph Neural Network) model to perform task identification. Finally, we show that the experimental results and comparisons with non-causality-based methods demonstrate the effectiveness of the proposed methods. We visualize the obtained causal signatures and discuss their biological relevance in light of the existing understanding of brain functionalities. Collectively, our work paves the way for further studies on causal fingerprints with potential applications in both healthy controls and neurodegenerative diseases.

Title: SOAR: Self-supervision Optimized UAV Action Recognition with Efficient Object-Aware Pretraining

• Authors: Ruiqi Xian, Xiyang Wu, Tianrui Guan, Xijun Wang, Boqing Gong, Dinesh Manocha
• Subjects: cs.CV, cs.AI, cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2409.18300
• Pdf URL: https://arxiv.org/pdf/2409.18300
• Copy Paste: [[2409.18300]] SOAR: Self-supervision Optimized UAV Action Recognition with Efficient Object-Aware Pretraining(https://arxiv.org/abs/2409.18300)
• Keywords: self-supervised
• Abstract: We introduce SOAR, a novel Self-supervised pretraining algorithm for aerial footage captured by Unmanned Aerial Vehicles (UAVs). We incorporate human object knowledge throughout the pretraining process to enhance UAV video pretraining efficiency and downstream action recognition performance. This is in contrast to prior works that primarily incorporate object information during the fine-tuning stage. Specifically, we first propose a novel object-aware masking strategy designed to retain the visibility of certain patches related to objects throughout the pretraining phase. Second, we introduce an object-aware loss function that utilizes object information to adjust the reconstruction loss, preventing bias towards less informative background patches. In practice, SOAR with a vanilla ViT backbone, outperforms best UAV action recognition models, recording a 9.7% and 21.4% boost in top-1 accuracy on the NEC-Drone and UAV-Human datasets, while delivering an inference speed of 18.7ms per video, making it 2x to 5x faster. Additionally, SOAR obtains comparable accuracy to prior self-supervised learning (SSL) methods while requiring 87.5% less pretraining time and 25% less memory usage

Title: Harnessing Wavelet Transformations for Generalizable Deepfake Forgery Detection

• Authors: Lalith Bharadwaj Baru, Shilhora Akshay Patel, Rohit Boddeda
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18301
• Pdf URL: https://arxiv.org/pdf/2409.18301
• Copy Paste: [[2409.18301]] Harnessing Wavelet Transformations for Generalizable Deepfake Forgery Detection(https://arxiv.org/abs/2409.18301)
• Keywords: diffusion, generative
• Abstract: The evolution of digital image manipulation, particularly with the advancement of deep generative models, significantly challenges existing deepfake detection methods, especially when the origin of the deepfake is obscure. To tackle the increasing complexity of these forgeries, we propose \textbf{Wavelet-CLIP}, a deepfake detection framework that integrates wavelet transforms with features derived from the ViT-L/14 architecture, pre-trained in the CLIP fashion. Wavelet-CLIP utilizes Wavelet Transforms to deeply analyze both spatial and frequency features from images, thus enhancing the model's capability to detect sophisticated deepfakes. To verify the effectiveness of our approach, we conducted extensive evaluations against existing state-of-the-art methods for cross-dataset generalization and detection of unseen images generated by standard diffusion models. Our method showcases outstanding performance, achieving an average AUC of 0.749 for cross-data generalization and 0.893 for robustness against unseen deepfakes, outperforming all compared methods. The code can be reproduced from the repo: \url{this https URL}

Title: DeBaRA: Denoising-Based 3D Room Arrangement Generation

• Authors: Léopold Maillard, Nicolas Sereyjol-Garros, Tom Durand, Maks Ovsjanikov
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18336
• Pdf URL: https://arxiv.org/pdf/2409.18336
• Copy Paste: [[2409.18336]] DeBaRA: Denoising-Based 3D Room Arrangement Generation(https://arxiv.org/abs/2409.18336)
• Keywords: diffusion, generative
• Abstract: Generating realistic and diverse layouts of furnished indoor 3D scenes unlocks multiple interactive applications impacting a wide range of industries. The inherent complexity of object interactions, the limited amount of available data and the requirement to fulfill spatial constraints all make generative modeling for 3D scene synthesis and arrangement challenging. Current methods address these challenges autoregressively or by using off-the-shelf diffusion objectives by simultaneously predicting all attributes without 3D reasoning considerations. In this paper, we introduce DeBaRA, a score-based model specifically tailored for precise, controllable and flexible arrangement generation in a bounded environment. We argue that the most critical component of a scene synthesis system is to accurately establish the size and position of various objects within a restricted area. Based on this insight, we propose a lightweight conditional score-based model designed with 3D spatial awareness at its core. We demonstrate that by focusing on spatial attributes of objects, a single trained DeBaRA model can be leveraged at test time to perform several downstream applications such as scene synthesis, completion and re-arrangement. Further, we introduce a novel Self Score Evaluation procedure so it can be optimally employed alongside external LLM models. We evaluate our approach through extensive experiments and demonstrate significant improvement upon state-of-the-art approaches in a range of scenarios.

Title: AER-LLM: Ambiguity-aware Emotion Recognition Leveraging Large Language Models

• Authors: Xin Hong, Yuan Gong, Vidhyasaharan Sethu, Ting Dang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18339
• Pdf URL: https://arxiv.org/pdf/2409.18339
• Copy Paste: [[2409.18339]] AER-LLM: Ambiguity-aware Emotion Recognition Leveraging Large Language Models(https://arxiv.org/abs/2409.18339)
• Keywords: in-context
• Abstract: Recent advancements in Large Language Models (LLMs) have demonstrated great success in many Natural Language Processing (NLP) tasks. In addition to their cognitive intelligence, exploring their capabilities in emotional intelligence is also crucial, as it enables more natural and empathetic conversational AI. Recent studies have shown LLMs' capability in recognizing emotions, but they often focus on single emotion labels and overlook the complex and ambiguous nature of human emotions. This study is the first to address this gap by exploring the potential of LLMs in recognizing ambiguous emotions, leveraging their strong generalization capabilities and in-context learning. We design zero-shot and few-shot prompting and incorporate past dialogue as context information for ambiguous emotion recognition. Experiments conducted using three datasets indicate significant potential for LLMs in recognizing ambiguous emotions, and highlight the substantial benefits of including context information. Furthermore, our findings indicate that LLMs demonstrate a high degree of effectiveness in recognizing less ambiguous emotions and exhibit potential for identifying more ambiguous emotions, paralleling human perceptual capabilities.

Title: SinoSynth: A Physics-based Domain Randomization Approach for Generalizable CBCT Image Enhancement

• Authors: Yunkui Pang, Yilin Liu, Xu Chen, Pew-Thian Yap, Jun Lian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18355
• Pdf URL: https://arxiv.org/pdf/2409.18355
• Copy Paste: [[2409.18355]] SinoSynth: A Physics-based Domain Randomization Approach for Generalizable CBCT Image Enhancement(https://arxiv.org/abs/2409.18355)
• Keywords: generative
• Abstract: Cone Beam Computed Tomography (CBCT) finds diverse applications in medicine. Ensuring high image quality in CBCT scans is essential for accurate diagnosis and treatment delivery. Yet, the susceptibility of CBCT images to noise and artifacts undermines both their usefulness and reliability. Existing methods typically address CBCT artifacts through image-to-image translation approaches. These methods, however, are limited by the artifact types present in the training data, which may not cover the complete spectrum of CBCT degradations stemming from variations in imaging protocols. Gathering additional data to encompass all possible scenarios can often pose a challenge. To address this, we present SinoSynth, a physics-based degradation model that simulates various CBCT-specific artifacts to generate a diverse set of synthetic CBCT images from high-quality CT images without requiring pre-aligned data. Through extensive experiments, we demonstrate that several different generative networks trained on our synthesized data achieve remarkable results on heterogeneous multi-institutional datasets, outperforming even the same networks trained on actual data. We further show that our degradation model conveniently provides an avenue to enforce anatomical constraints in conditional generative models, yielding high-quality and structure-preserving synthetic CT images.

Title: Generative AI for fast and accurate Statistical Computation of Fluids

• Authors: Roberto Molinaro, Samuel Lanthaler, Bogdan Raonić, Tobias Rohner, Victor Armegioiu, Zhong Yi Wan, Fei Sha, Siddhartha Mishra, Leonardo Zepeda-Núñez
• Subjects: cs.LG, math.NA, physics.flu-dyn
• Abstract URL: https://arxiv.org/abs/2409.18359
• Pdf URL: https://arxiv.org/pdf/2409.18359
• Copy Paste: [[2409.18359]] Generative AI for fast and accurate Statistical Computation of Fluids(https://arxiv.org/abs/2409.18359)
• Keywords: diffusion, generative
• Abstract: We present a generative AI algorithm for addressing the challenging task of fast, accurate and robust statistical computation of three-dimensional turbulent fluid flows. Our algorithm, termed as GenCFD, is based on a conditional score-based diffusion model. Through extensive numerical experimentation with both incompressible and compressible fluid flows, we demonstrate that GenCFD provides very accurate approximation of statistical quantities of interest such as mean, variance, point pdfs, higher-order moments, while also generating high quality realistic samples of turbulent fluid flows and ensuring excellent spectral resolution. In contrast, ensembles of operator learning baselines which are trained to minimize mean (absolute) square errors regress to the mean flow. We present rigorous theoretical results uncovering the surprising mechanisms through which diffusion models accurately generate fluid flows. These mechanisms are illustrated with solvable toy models that exhibit the relevant features of turbulent fluid flows while being amenable to explicit analytical formulas.

Title: Multi-hypotheses Conditioned Point Cloud Diffusion for 3D Human Reconstruction from Occluded Images

• Authors: Donghwan Kim, Tae-Kyun Kim
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18364
• Pdf URL: https://arxiv.org/pdf/2409.18364
• Copy Paste: [[2409.18364]] Multi-hypotheses Conditioned Point Cloud Diffusion for 3D Human Reconstruction from Occluded Images(https://arxiv.org/abs/2409.18364)
• Keywords: diffusion
• Abstract: 3D human shape reconstruction under severe occlusion due to human-object or human-human interaction is a challenging problem. Parametric models i.e., SMPL(-X), which are based on the statistics across human shapes, can represent whole human body shapes but are limited to minimally-clothed human shapes. Implicit-function-based methods extract features from the parametric models to employ prior knowledge of human bodies and can capture geometric details such as clothing and hair. However, they often struggle to handle misaligned parametric models and inpaint occluded regions given a single RGB image. In this work, we propose a novel pipeline, MHCDIFF, Multi-hypotheses Conditioned Point Cloud Diffusion, composed of point cloud diffusion conditioned on probabilistic distributions for pixel-aligned detailed 3D human reconstruction under occlusion. Compared to previous implicit-function-based methods, the point cloud diffusion model can capture the global consistent features to generate the occluded regions, and the denoising process corrects the misaligned SMPL meshes. The core of MHCDIFF is extracting local features from multiple hypothesized SMPL(-X) meshes and aggregating the set of features to condition the diffusion model. In the experiments on CAPE and MultiHuman datasets, the proposed method outperforms various SOTA methods based on SMPL, implicit functions, point cloud diffusion, and their combined, under synthetic and real occlusions.

Title: GenesisTex2: Stable, Consistent and High-Quality Text-to-Texture Generation

• Authors: Jiawei Lu, Yingpeng Zhang, Zengjun Zhao, He Wang, Kun Zhou, Tianjia Shao
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18401
• Pdf URL: https://arxiv.org/pdf/2409.18401
• Copy Paste: [[2409.18401]] GenesisTex2: Stable, Consistent and High-Quality Text-to-Texture Generation(https://arxiv.org/abs/2409.18401)
• Keywords: diffusion
• Abstract: Large-scale text-guided image diffusion models have shown astonishing results in text-to-image (T2I) generation. However, applying these models to synthesize textures for 3D geometries remains challenging due to the domain gap between 2D images and textures on a 3D surface. Early works that used a projecting-and-inpainting approach managed to preserve generation diversity but often resulted in noticeable artifacts and style inconsistencies. While recent methods have attempted to address these inconsistencies, they often introduce other issues, such as blurring, over-saturation, or over-smoothing. To overcome these challenges, we propose a novel text-to-texture synthesis framework that leverages pretrained diffusion models. We first introduce a local attention reweighing mechanism in the self-attention layers to guide the model in concentrating on spatial-correlated patches across different views, thereby enhancing local details while preserving cross-view consistency. Additionally, we propose a novel latent space merge pipeline, which further ensures consistency across different viewpoints without sacrificing too much diversity. Our method significantly outperforms existing state-of-the-art techniques regarding texture consistency and visual quality, while delivering results much faster than distillation-based methods. Importantly, our framework does not require additional training or fine-tuning, making it highly adaptable to a wide range of models available on public platforms.

Title: A3: Active Adversarial Alignment for Source-Free Domain Adaptation

• Authors: Chrisantus Eze, Christopher Crick
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18418
• Pdf URL: https://arxiv.org/pdf/2409.18418
• Copy Paste: [[2409.18418]] A3: Active Adversarial Alignment for Source-Free Domain Adaptation(https://arxiv.org/abs/2409.18418)
• Keywords: self-supervised
• Abstract: Unsupervised domain adaptation (UDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain. Recent works have focused on source-free UDA, where only target data is available. This is challenging as models rely on noisy pseudo-labels and struggle with distribution shifts. We propose Active Adversarial Alignment (A3), a novel framework combining self-supervised learning, adversarial training, and active learning for robust source-free UDA. A3 actively samples informative and diverse data using an acquisition function for training. It adapts models via adversarial losses and consistency regularization, aligning distributions without source data access. A3 advances source-free UDA through its synergistic integration of active and adversarial learning for effective domain alignment and noise reduction.

Title: Neural Collaborative Filtering to Detect Anomalies in Human Semantic Trajectories

• Authors: Yueyang Liu, Lance Kennedy, Hossein Amiri, Andreas Züfle
• Subjects: cs.LG, cs.IR, cs.SI
• Abstract URL: https://arxiv.org/abs/2409.18427
• Pdf URL: https://arxiv.org/pdf/2409.18427
• Copy Paste: [[2409.18427]] Neural Collaborative Filtering to Detect Anomalies in Human Semantic Trajectories(https://arxiv.org/abs/2409.18427)
• Keywords: anomaly
• Abstract: Human trajectory anomaly detection has become increasingly important across a wide range of applications, including security surveillance and public health. However, existing trajectory anomaly detection methods are primarily focused on vehicle-level traffic, while human-level trajectory anomaly detection remains under-explored. Since human trajectory data is often very sparse, machine learning methods have become the preferred approach for identifying complex patterns. However, concerns regarding potential biases and the robustness of these models have intensified the demand for more transparent and explainable alternatives. In response to these challenges, our research focuses on developing a lightweight anomaly detection model specifically designed to detect anomalies in human trajectories. We propose a Neural Collaborative Filtering approach to model and predict normal mobility. Our method is designed to model users' daily patterns of life without requiring prior knowledge, thereby enhancing performance in scenarios where data is sparse or incomplete, such as in cold start situations. Our algorithm consists of two main modules. The first is the collaborative filtering module, which applies collaborative filtering to model normal mobility of individual humans to places of interest. The second is the neural module, responsible for interpreting the complex spatio-temporal relationships inherent in human trajectory data. To validate our approach, we conducted extensive experiments using simulated and real-world datasets comparing to numerous state-of-the-art trajectory anomaly detection approaches.

Title: Gradient-free Decoder Inversion in Latent Diffusion Models

• Authors: Seongmin Hong, Suh Yoon Jeon, Kyeonghyun Lee, Ernest K. Ryu, Se Young Chun
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18442
• Pdf URL: https://arxiv.org/pdf/2409.18442
• Copy Paste: [[2409.18442]] Gradient-free Decoder Inversion in Latent Diffusion Models(https://arxiv.org/abs/2409.18442)
• Keywords: diffusion, generative
• Abstract: In latent diffusion models (LDMs), denoising diffusion process efficiently takes place on latent space whose dimension is lower than that of pixel space. Decoder is typically used to transform the representation in latent space to that in pixel space. While a decoder is assumed to have an encoder as an accurate inverse, exact encoder-decoder pair rarely exists in practice even though applications often require precise inversion of decoder. Prior works for decoder inversion in LDMs employed gradient descent inspired by inversions of generative adversarial networks. However, gradient-based methods require larger GPU memory and longer computation time for larger latent space. For example, recent video LDMs can generate more than 16 frames, but GPUs with 24 GB memory can only perform gradient-based decoder inversion for 4 frames. Here, we propose an efficient gradient-free decoder inversion for LDMs, which can be applied to diverse latent models. Theoretical convergence property of our proposed inversion has been investigated not only for the forward step method, but also for the inertial Krasnoselskii-Mann (KM) iterations under mild assumption on cocoercivity that is satisfied by recent LDMs. Our proposed gradient-free method with Adam optimizer and learning rate scheduling significantly reduced computation time and memory usage over prior gradient-based methods and enabled efficient computation in applications such as noise-space watermarking while achieving comparable error levels.

Title: Underwater Image Enhancement with Physical-based Denoising Diffusion Implicit Models

• Authors: Nguyen Gia Bach, Chanh Minh Tran, Eiji Kamioka, Phan Xuan Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18476
• Pdf URL: https://arxiv.org/pdf/2409.18476
• Copy Paste: [[2409.18476]] Underwater Image Enhancement with Physical-based Denoising Diffusion Implicit Models(https://arxiv.org/abs/2409.18476)
• Keywords: diffusion, generative
• Abstract: Underwater vision is crucial for autonomous underwater vehicles (AUVs), and enhancing degraded underwater images in real-time on a resource-constrained AUV is a key challenge due to factors like light absorption and scattering, or the sufficient model computational complexity to resolve such factors. Traditional image enhancement techniques lack adaptability to varying underwater conditions, while learning-based methods, particularly those using convolutional neural networks (CNNs) and generative adversarial networks (GANs), offer more robust solutions but face limitations such as inadequate enhancement, unstable training, or mode collapse. Denoising diffusion probabilistic models (DDPMs) have emerged as a state-of-the-art approach in image-to-image tasks but require intensive computational complexity to achieve the desired underwater image enhancement (UIE) using the recent UW-DDPM solution. To address these challenges, this paper introduces UW-DiffPhys, a novel physical-based and diffusion-based UIE approach. UW-DiffPhys combines light-computation physical-based UIE network components with a denoising U-Net to replace the computationally intensive distribution transformation U-Net in the existing UW-DDPM framework, reducing complexity while maintaining performance. Additionally, the Denoising Diffusion Implicit Model (DDIM) is employed to accelerate the inference process through non-Markovian sampling. Experimental results demonstrate that UW-DiffPhys achieved a substantial reduction in computational complexity and inference time compared to UW-DDPM, with competitive performance in key metrics such as PSNR, SSIM, UCIQE, and an improvement in the overall underwater image quality UIQM metric. The implementation code can be found at the following repository: this https URL

Title: Treating Brain-inspired Memories as Priors for Diffusion Model to Forecast Multivariate Time Series

• Authors: Muyao Wang, Wenchao Chen, Zhibin Duan, Bo Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18491
• Pdf URL: https://arxiv.org/pdf/2409.18491
• Copy Paste: [[2409.18491]] Treating Brain-inspired Memories as Priors for Diffusion Model to Forecast Multivariate Time Series(https://arxiv.org/abs/2409.18491)
• Keywords: diffusion
• Abstract: Forecasting Multivariate Time Series (MTS) involves significant challenges in various application domains. One immediate challenge is modeling temporal patterns with the finite length of the input. These temporal patterns usually involve periodic and sudden events that recur across different channels. To better capture temporal patterns, we get inspiration from humans' memory mechanisms and propose a channel-shared, brain-inspired memory module for MTS. Specifically, brain-inspired memory comprises semantic and episodic memory, where the former is used to capture general patterns, such as periodic events, and the latter is employed to capture special patterns, such as sudden events, respectively. Meanwhile, we design corresponding recall and update mechanisms to better utilize these patterns. Furthermore, acknowledging the capacity of diffusion models to leverage memory as a prior, we present a brain-inspired memory-augmented diffusion model. This innovative model retrieves relevant memories for different channels, utilizing them as distinct priors for MTS predictions. This incorporation significantly enhances the accuracy and robustness of predictions. Experimental results on eight datasets consistently validate the superiority of our approach in capturing and leveraging diverse recurrent temporal patterns across different channels.

Title: Token Caching for Diffusion Transformer Acceleration

• Authors: Jinming Lou, Wenyang Luo, Yufan Liu, Bing Li, Xinmiao Ding, Weiming Hu, Jiajiong Cao, Yuming Li, Chenguang Ma
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18523
• Pdf URL: https://arxiv.org/pdf/2409.18523
• Copy Paste: [[2409.18523]] Token Caching for Diffusion Transformer Acceleration(https://arxiv.org/abs/2409.18523)
• Keywords: diffusion, generative
• Abstract: Diffusion transformers have gained substantial interest in diffusion generative modeling due to their outstanding performance. However, their high computational cost, arising from the quadratic computational complexity of attention mechanisms and multi-step inference, presents a significant bottleneck. To address this challenge, we propose TokenCache, a novel post-training acceleration method that leverages the token-based multi-block architecture of transformers to reduce redundant computations among tokens across inference steps. TokenCache specifically addresses three critical questions in the context of diffusion transformers: (1) which tokens should be pruned to eliminate redundancy, (2) which blocks should be targeted for efficient pruning, and (3) at which time steps caching should be applied to balance speed and quality. In response to these challenges, TokenCache introduces a Cache Predictor that assigns importance scores to tokens, enabling selective pruning without compromising model performance. Furthermore, we propose an adaptive block selection strategy to focus on blocks with minimal impact on the network's output, along with a Two-Phase Round-Robin (TPRR) scheduling policy to optimize caching intervals throughout the denoising process. Experimental results across various models demonstrate that TokenCache achieves an effective trade-off between generation quality and inference speed for diffusion transformers. Our code will be publicly available.

Title: How Effective is Pre-training of Large Masked Autoencoders for Downstream Earth Observation Tasks?

• Authors: Jose Sosa, Mohamed Aloulou, Danila Rukhovich, Rim Sleimi, Boonyarit Changaival, Anis Kacem, Djamila Aouada
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18536
• Pdf URL: https://arxiv.org/pdf/2409.18536
• Copy Paste: [[2409.18536]] How Effective is Pre-training of Large Masked Autoencoders for Downstream Earth Observation Tasks?(https://arxiv.org/abs/2409.18536)
• Keywords: self-supervised, foundation model
• Abstract: Self-supervised pre-training has proven highly effective for many computer vision tasks, particularly when labelled data are scarce. In the context of Earth Observation (EO), foundation models and various other Vision Transformer (ViT)-based approaches have been successfully applied for transfer learning to downstream tasks. However, it remains unclear under which conditions pre-trained models offer significant advantages over training from scratch. In this study, we investigate the effectiveness of pre-training ViT-based Masked Autoencoders (MAE) for downstream EO tasks, focusing on reconstruction, segmentation, and classification. We consider two large ViT-based MAE pre-trained models: a foundation model (Prithvi) and SatMAE. We evaluate Prithvi on reconstruction and segmentation-based downstream tasks, and for SatMAE we assess its performance on a classification downstream task. Our findings suggest that pre-training is particularly beneficial when the fine-tuning task closely resembles the pre-training task, e.g. reconstruction. In contrast, for tasks such as segmentation or classification, training from scratch with specific hyperparameter adjustments proved to be equally or more effective.

Title: AL-GTD: Deep Active Learning for Gaze Target Detection

• Authors: Francesco Tonini, Nicola Dall'Asen, Lorenzo Vaquero, Cigdem Beyan, Elisa Ricci
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18561
• Pdf URL: https://arxiv.org/pdf/2409.18561
• Copy Paste: [[2409.18561]] AL-GTD: Deep Active Learning for Gaze Target Detection(https://arxiv.org/abs/2409.18561)
• Keywords: self-supervised
• Abstract: Gaze target detection aims at determining the image location where a person is looking. While existing studies have made significant progress in this area by regressing accurate gaze heatmaps, these achievements have largely relied on access to extensive labeled datasets, which demands substantial human labor. In this paper, our goal is to reduce the reliance on the size of labeled training data for gaze target detection. To achieve this, we propose AL-GTD, an innovative approach that integrates supervised and self-supervised losses within a novel sample acquisition function to perform active learning (AL). Additionally, it utilizes pseudo-labeling to mitigate distribution shifts during the training phase. AL-GTD achieves the best of all AUC results by utilizing only 40-50% of the training data, in contrast to state-of-the-art (SOTA) gaze target detectors requiring the entire training dataset to achieve the same performance. Importantly, AL-GTD quickly reaches satisfactory performance with 10-20% of the training data, showing the effectiveness of our acquisition function, which is able to acquire the most informative samples. We provide a comprehensive experimental analysis by adapting several AL methods for the task. AL-GTD outperforms AL competitors, simultaneously exhibiting superior performance compared to SOTA gaze target detectors when all are trained within a low-data regime. Code is available at this https URL.

Title: Unsupervised Fingerphoto Presentation Attack Detection With Diffusion Models

• Authors: Hailin Li, Raghavendra Ramachandra, Mohamed Ragab, Soumik Mondal, Yong Kiam Tan, Khin Mi Mi Aung
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18636
• Pdf URL: https://arxiv.org/pdf/2409.18636
• Copy Paste: [[2409.18636]] Unsupervised Fingerphoto Presentation Attack Detection With Diffusion Models(https://arxiv.org/abs/2409.18636)
• Keywords: diffusion
• Abstract: Smartphone-based contactless fingerphoto authentication has become a reliable alternative to traditional contact-based fingerprint biometric systems owing to rapid advances in smartphone camera technology. Despite its convenience, fingerprint authentication through fingerphotos is more vulnerable to presentation attacks, which has motivated recent research efforts towards developing fingerphoto Presentation Attack Detection (PAD) techniques. However, prior PAD approaches utilized supervised learning methods that require labeled training data for both bona fide and attack samples. This can suffer from two key issues, namely (i) generalization:the detection of novel presentation attack instruments (PAIs) unseen in the training data, and (ii) scalability:the collection of a large dataset of attack samples using different PAIs. To address these challenges, we propose a novel unsupervised approach based on a state-of-the-art deep-learning-based diffusion model, the Denoising Diffusion Probabilistic Model (DDPM), which is trained solely on bona fide samples. The proposed approach detects Presentation Attacks (PA) by calculating the reconstruction similarity between the input and output pairs of the DDPM. We present extensive experiments across three PAI datasets to test the accuracy and generalization capability of our approach. The results show that the proposed DDPM-based PAD method achieves significantly better detection error rates on several PAI classes compared to other baseline unsupervised approaches.

Title: When SAM2 Meets Video Camouflaged Object Segmentation: A Comprehensive Evaluation and Adaptation

• Authors: Yuli Zhou, Guolei Sun, Yawei Li, Luca Benini, Ender Konukoglu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18653
• Pdf URL: https://arxiv.org/pdf/2409.18653
• Copy Paste: [[2409.18653]] When SAM2 Meets Video Camouflaged Object Segmentation: A Comprehensive Evaluation and Adaptation(https://arxiv.org/abs/2409.18653)
• Keywords: foundation model
• Abstract: This study investigates the application and performance of the Segment Anything Model 2 (SAM2) in the challenging task of video camouflaged object segmentation (VCOS). VCOS involves detecting objects that blend seamlessly in the surroundings for videos, due to similar colors and textures, poor light conditions, etc. Compared to the objects in normal scenes, camouflaged objects are much more difficult to detect. SAM2, a video foundation model, has shown potential in various tasks. But its effectiveness in dynamic camouflaged scenarios remains under-explored. This study presents a comprehensive study on SAM2's ability in VCOS. First, we assess SAM2's performance on camouflaged video datasets using different models and prompts (click, box, and mask). Second, we explore the integration of SAM2 with existing multimodal large language models (MLLMs) and VCOS methods. Third, we specifically adapt SAM2 by fine-tuning it on the video camouflaged dataset. Our comprehensive experiments demonstrate that SAM2 has excellent zero-shot ability of detecting camouflaged objects in videos. We also show that this ability could be further improved by specifically adjusting SAM2's parameters for VCOS. The code will be available at this https URL

Title: How green is continual learning, really? Analyzing the energy consumption in continual training of vision foundation models

• Authors: Tomaso Trinci, Simone Magistri, Roberto Verdecchia, Andrew D. Bagdanov
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18664
• Pdf URL: https://arxiv.org/pdf/2409.18664
• Copy Paste: [[2409.18664]] How green is continual learning, really? Analyzing the energy consumption in continual training of vision foundation models(https://arxiv.org/abs/2409.18664)
• Keywords: foundation model
• Abstract: With the ever-growing adoption of AI, its impact on the environment is no longer negligible. Despite the potential that continual learning could have towards Green AI, its environmental sustainability remains relatively uncharted. In this work we aim to gain a systematic understanding of the energy efficiency of continual learning algorithms. To that end, we conducted an extensive set of empirical experiments comparing the energy consumption of recent representation-, prompt-, and exemplar-based continual learning algorithms and two standard baseline (fine tuning and joint training) when used to continually adapt a pre-trained ViT-B/16 foundation model. We performed our experiments on three standard datasets: CIFAR-100, ImageNet-R, and DomainNet. Additionally, we propose a novel metric, the Energy NetScore, which we use measure the algorithm efficiency in terms of energy-accuracy trade-off. Through numerous evaluations varying the number and size of the incremental learning steps, our experiments demonstrate that different types of continual learning algorithms have very different impacts on energy consumption during both training and inference. Although often overlooked in the continual learning literature, we found that the energy consumed during the inference phase is crucial for evaluating the environmental sustainability of continual learning models.

Title: Learning from Pattern Completion: Self-supervised Controllable Generation

• Authors: Zhiqiang Chen, Guofan Fan, Jinying Gao, Lei Ma, Bo Lei, Tiejun Huang, Shan Yu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18694
• Pdf URL: https://arxiv.org/pdf/2409.18694
• Copy Paste: [[2409.18694]] Learning from Pattern Completion: Self-supervised Controllable Generation(https://arxiv.org/abs/2409.18694)
• Keywords: self-supervised
• Abstract: The human brain exhibits a strong ability to spontaneously associate different visual attributes of the same or similar visual scene, such as associating sketches and graffiti with real-world visual objects, usually without supervising information. In contrast, in the field of artificial intelligence, controllable generation methods like ControlNet heavily rely on annotated training datasets such as depth maps, semantic segmentation maps, and poses, which limits the method's scalability. Inspired by the neural mechanisms that may contribute to the brain's associative power, specifically the cortical modularization and hippocampal pattern completion, here we propose a self-supervised controllable generation (SCG) framework. Firstly, we introduce an equivariant constraint to promote inter-module independence and intra-module correlation in a modular autoencoder network, thereby achieving functional specialization. Subsequently, based on these specialized modules, we employ a self-supervised pattern completion approach for controllable generation training. Experimental results demonstrate that the proposed modular autoencoder effectively achieves functional specialization, including the modular processing of color, brightness, and edge detection, and exhibits brain-like features including orientation selectivity, color antagonism, and center-surround receptive fields. Through self-supervised training, associative generation capabilities spontaneously emerge in SCG, demonstrating excellent generalization ability to various tasks such as associative generation on painting, sketches, and ancient graffiti. Compared to the previous representative method ControlNet, our proposed approach not only demonstrates superior robustness in more challenging high-noise scenarios but also possesses more promising scalability potential due to its self-supervised manner.

Title: HardCore Generation: Generating Hard UNSAT Problems for Data Augmentation

• Authors: Joseph Cotnareanu, Zhanguang Zhang, Hui-Ling Zhen, Yingxue Zhang, Mark Coates
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18778
• Pdf URL: https://arxiv.org/pdf/2409.18778
• Copy Paste: [[2409.18778]] HardCore Generation: Generating Hard UNSAT Problems for Data Augmentation(https://arxiv.org/abs/2409.18778)
• Keywords: generative
• Abstract: Efficiently determining the satisfiability of a boolean equation -- known as the SAT problem for brevity -- is crucial in various industrial problems. Recently, the advent of deep learning methods has introduced significant potential for enhancing SAT solving. However, a major barrier to the advancement of this field has been the scarcity of large, realistic datasets. The majority of current public datasets are either randomly generated or extremely limited, containing only a few examples from unrelated problem families. These datasets are inadequate for meaningful training of deep learning methods. In light of this, researchers have started exploring generative techniques to create data that more accurately reflect SAT problems encountered in practical situations. These methods have so far suffered from either the inability to produce challenging SAT problems or time-scalability obstacles. In this paper we address both by identifying and manipulating the key contributors to a problem's ``hardness'', known as cores. Although some previous work has addressed cores, the time costs are unacceptably high due to the expense of traditional heuristic core detection techniques. We introduce a fast core detection procedure that uses a graph neural network. Our empirical results demonstrate that we can efficiently generate problems that remain hard to solve and retain key attributes of the original example problems. We show via experiment that the generated synthetic SAT problems can be used in a data augmentation setting to provide improved prediction of solver runtimes.

Title: Challenges of Generating Structurally Diverse Graphs

• Authors: Fedor Velikonivtsev, Mikhail Mironov, Liudmila Prokhorenkova
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18859
• Pdf URL: https://arxiv.org/pdf/2409.18859
• Copy Paste: [[2409.18859]] Challenges of Generating Structurally Diverse Graphs(https://arxiv.org/abs/2409.18859)
• Keywords: generative
• Abstract: For many graph-related problems, it can be essential to have a set of structurally diverse graphs. For instance, such graphs can be used for testing graph algorithms or their neural approximations. However, to the best of our knowledge, the problem of generating structurally diverse graphs has not been explored in the literature. In this paper, we fill this gap. First, we discuss how to define diversity for a set of graphs, why this task is non-trivial, and how one can choose a proper diversity measure. Then, for a given diversity measure, we propose and compare several algorithms optimizing it: we consider approaches based on standard random graph models, local graph optimization, genetic algorithms, and neural generative models. We show that it is possible to significantly improve diversity over basic random graph generators. Additionally, our analysis of generated graphs allows us to better understand the properties of graph distances: depending on which diversity measure is used for optimization, the obtained graphs may possess very different structural properties which gives insights about the sensitivity of the graph distance underlying the diversity measure.

Title: Emu3: Next-Token Prediction is All You Need

• Authors: Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang, Jingjing Liu, Yonghua Lin, Tiejun Huang, Zhongyuan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18869
• Pdf URL: https://arxiv.org/pdf/2409.18869
• Copy Paste: [[2409.18869]] Emu3: Next-Token Prediction is All You Need(https://arxiv.org/abs/2409.18869)
• Keywords: diffusion
• Abstract: While next-token prediction is considered a promising path towards artificial general intelligence, it has struggled to excel in multimodal tasks, which are still dominated by diffusion models (e.g., Stable Diffusion) and compositional approaches (e.g., CLIP combined with LLMs). In this paper, we introduce Emu3, a new suite of state-of-the-art multimodal models trained solely with next-token prediction. By tokenizing images, text, and videos into a discrete space, we train a single transformer from scratch on a mixture of multimodal sequences. Emu3 outperforms several well-established task-specific models in both generation and perception tasks, surpassing flagship models such as SDXL and LLaVA-1.6, while eliminating the need for diffusion or compositional architectures. Emu3 is also capable of generating high-fidelity video via predicting the next token in a video sequence. We simplify complex multimodal model designs by converging on a singular focus: tokens, unlocking great potential for scaling both during training and inference. Our results demonstrate that next-token prediction is a promising path towards building general multimodal intelligence beyond language. We open-source key techniques and models to support further research in this direction.

Title: CESNET-TimeSeries24: Time Series Dataset for Network Traffic Anomaly Detection and Forecasting

• Authors: Josef Koumar, Karel Hynek, Tomáš Čejka, Pavel Šiška
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18874
• Pdf URL: https://arxiv.org/pdf/2409.18874
• Copy Paste: [[2409.18874]] CESNET-TimeSeries24: Time Series Dataset for Network Traffic Anomaly Detection and Forecasting(https://arxiv.org/abs/2409.18874)
• Keywords: anomaly
• Abstract: Anomaly detection in network traffic is crucial for maintaining the security of computer networks and identifying malicious activities. One of the primary approaches to anomaly detection are methods based on forecasting. Nevertheless, extensive real-world network datasets for forecasting and anomaly detection techniques are missing, potentially causing performance overestimation of anomaly detection algorithms. This manuscript addresses this gap by introducing a dataset comprising time series data of network entities' behavior, collected from the CESNET3 network. The dataset was created from 40 weeks of network traffic of 275 thousand active IP addresses. The ISP origin of the presented data ensures a high level of variability among network entities, which forms a unique and authentic challenge for forecasting and anomaly detection models. It provides valuable insights into the practical deployment of forecast-based anomaly detection approaches.

Title: CemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognition

• Authors: Zhonglin Sun, Siyang Song, Ioannis Patras, Georgios Tzimiropoulos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18876
• Pdf URL: https://arxiv.org/pdf/2409.18876
• Copy Paste: [[2409.18876]] CemiFace: Center-based Semi-hard Synthetic Face Generation for Face Recognition(https://arxiv.org/abs/2409.18876)
• Keywords: diffusion, generative
• Abstract: Privacy issue is a main concern in developing face recognition techniques. Although synthetic face images can partially mitigate potential legal risks while maintaining effective face recognition (FR) performance, FR models trained by face images synthesized by existing generative approaches frequently suffer from performance degradation problems due to the insufficient discriminative quality of these synthesized samples. In this paper, we systematically investigate what contributes to solid face recognition model training, and reveal that face images with certain degree of similarities to their identity centers show great effectiveness in the performance of trained FR models. Inspired by this, we propose a novel diffusion-based approach (namely Center-based Semi-hard Synthetic Face Generation (CemiFace)) which produces facial samples with various levels of similarity to the subject center, thus allowing to generate face datasets containing effective discriminative samples for training face recognition. Experimental results show that with a modest degree of similarity, training on the generated dataset can produce competitive performance compared to previous generation methods.

Title: Explainable Artifacts for Synthetic Western Blot Source Attribution

• Authors: João Phillipe Cardenuto, Sara Mandelli, Daniel Moreira, Paolo Bestagini, Edward Delp, Anderson Rocha
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18881
• Pdf URL: https://arxiv.org/pdf/2409.18881
• Copy Paste: [[2409.18881]] Explainable Artifacts for Synthetic Western Blot Source Attribution(https://arxiv.org/abs/2409.18881)
• Keywords: diffusion, generative
• Abstract: Recent advancements in artificial intelligence have enabled generative models to produce synthetic scientific images that are indistinguishable from pristine ones, posing a challenge even for expert scientists habituated to working with such content. When exploited by organizations known as paper mills, which systematically generate fraudulent articles, these technologies can significantly contribute to the spread of misinformation about ungrounded science, potentially undermining trust in scientific research. While previous studies have explored black-box solutions, such as Convolutional Neural Networks, for identifying synthetic content, only some have addressed the challenge of generalizing across different models and providing insight into the artifacts in synthetic images that inform the detection process. This study aims to identify explainable artifacts generated by state-of-the-art generative models (e.g., Generative Adversarial Networks and Diffusion Models) and leverage them for open-set identification and source attribution (i.e., pointing to the model that created the image).

Title: Detecting Dataset Abuse in Fine-Tuning Stable Diffusion Models for Text-to-Image Synthesis

• Authors: Songrui Wang, Yubo Zhu, Wei Tong, Sheng Zhong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18897
• Pdf URL: https://arxiv.org/pdf/2409.18897
• Copy Paste: [[2409.18897]] Detecting Dataset Abuse in Fine-Tuning Stable Diffusion Models for Text-to-Image Synthesis(https://arxiv.org/abs/2409.18897)
• Keywords: diffusion, generative
• Abstract: Text-to-image synthesis has become highly popular for generating realistic and stylized images, often requiring fine-tuning generative models with domain-specific datasets for specialized tasks. However, these valuable datasets face risks of unauthorized usage and unapproved sharing, compromising the rights of the owners. In this paper, we address the issue of dataset abuse during the fine-tuning of Stable Diffusion models for text-to-image synthesis. We present a dataset watermarking framework designed to detect unauthorized usage and trace data leaks. The framework employs two key strategies across multiple watermarking schemes and is effective for large-scale dataset authorization. Extensive experiments demonstrate the framework's effectiveness, minimal impact on the dataset (only 2% of the data required to be modified for high detection accuracy), and ability to trace data leaks. Our results also highlight the robustness and transferability of the framework, proving its practical applicability in detecting dataset abuse.

Title: Unsupervised Low-light Image Enhancement with Lookup Tables and Diffusion Priors

• Authors: Yunlong Lin, Zhenqi Fu, Kairun Wen, Tian Ye, Sixiang Chen, Ge Meng, Yingying Wang, Yue Huang, Xiaotong Tu, Xinghao Ding
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.18899
• Pdf URL: https://arxiv.org/pdf/2409.18899
• Copy Paste: [[2409.18899]] Unsupervised Low-light Image Enhancement with Lookup Tables and Diffusion Priors(https://arxiv.org/abs/2409.18899)
• Keywords: diffusion
• Abstract: Low-light image enhancement (LIE) aims at precisely and efficiently recovering an image degraded in poor illumination environments. Recent advanced LIE techniques are using deep neural networks, which require lots of low-normal light image pairs, network parameters, and computational resources. As a result, their practicality is limited. In this work, we devise a novel unsupervised LIE framework based on diffusion priors and lookup tables (DPLUT) to achieve efficient low-light image recovery. The proposed approach comprises two critical components: a light adjustment lookup table (LLUT) and a noise suppression lookup table (NLUT). LLUT is optimized with a set of unsupervised losses. It aims at predicting pixel-wise curve parameters for the dynamic range adjustment of a specific image. NLUT is designed to remove the amplified noise after the light brightens. As diffusion models are sensitive to noise, diffusion priors are introduced to achieve high-performance noise suppression. Extensive experiments demonstrate that our approach outperforms state-of-the-art methods in terms of visual quality and efficiency.

Title: Improving Visual Object Tracking through Visual Prompting

• Authors: Shih-Fang Chen, Jun-Cheng Chen, I-Hong Jhuo, Yen-Yu Lin
• Subjects: cs.CV, cs.AI, cs.MM, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.18901
• Pdf URL: https://arxiv.org/pdf/2409.18901
• Copy Paste: [[2409.18901]] Improving Visual Object Tracking through Visual Prompting(https://arxiv.org/abs/2409.18901)
• Keywords: foundation model
• Abstract: Learning a discriminative model to distinguish a target from its surrounding distractors is essential to generic visual object tracking. Dynamic target representation adaptation against distractors is challenging due to the limited discriminative capabilities of prevailing trackers. We present a new visual Prompting mechanism for generic Visual Object Tracking (PiVOT) to address this issue. PiVOT proposes a prompt generation network with the pre-trained foundation model CLIP to automatically generate and refine visual prompts, enabling the transfer of foundation model knowledge for tracking. While CLIP offers broad category-level knowledge, the tracker, trained on instance-specific data, excels at recognizing unique object instances. Thus, PiVOT first compiles a visual prompt highlighting potential target locations. To transfer the knowledge of CLIP to the tracker, PiVOT leverages CLIP to refine the visual prompt based on the similarities between candidate objects and the reference templates across potential targets. Once the visual prompt is refined, it can better highlight potential target locations, thereby reducing irrelevant prompt information. With the proposed prompting mechanism, the tracker can generate improved instance-aware feature maps through the guidance of the visual prompt, thus effectively reducing distractors. The proposed method does not involve CLIP during training, thereby keeping the same training complexity and preserving the generalization capability of the pretrained foundation model. Extensive experiments across multiple benchmarks indicate that PiVOT, using the proposed prompting method can suppress distracting objects and enhance the tracker.

Title: AIPatient: Simulating Patients with EHRs and LLM Powered Agentic Workflow

• Authors: Huizi Yu, Jiayan Zhou, Lingyao Li, Shan Chen, Jack Gallifant, Anye Shi, Xiang Li, Wenyue Hua, Mingyu Jin, Guang Chen, Yang Zhou, Zhao Li, Trisha Gupte, Ming-Li Chen, Zahra Azizi, Yongfeng Zhang, Themistocles L. Assimes, Xin Ma, Danielle S. Bitterman, Lin Lu, Lizhou Fan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18924
• Pdf URL: https://arxiv.org/pdf/2409.18924
• Copy Paste: [[2409.18924]] AIPatient: Simulating Patients with EHRs and LLM Powered Agentic Workflow(https://arxiv.org/abs/2409.18924)
• Keywords: diffusion
• Abstract: Simulated patient systems play a crucial role in modern medical education and research, providing safe, integrative learning environments and enabling clinical decision-making simulations. Large Language Models (LLM) could advance simulated patient systems by replicating medical conditions and patient-doctor interactions with high fidelity and low cost. However, ensuring the effectiveness and trustworthiness of these systems remains a challenge, as they require a large, diverse, and precise patient knowledgebase, along with a robust and stable knowledge diffusion to users. Here, we developed AIPatient, an advanced simulated patient system with AIPatient Knowledge Graph (AIPatient KG) as the input and the Reasoning Retrieval-Augmented Generation (Reasoning RAG) agentic workflow as the generation backbone. AIPatient KG samples data from Electronic Health Records (EHRs) in the Medical Information Mart for Intensive Care (MIMIC)-III database, producing a clinically diverse and relevant cohort of 1,495 patients with high knowledgebase validity (F1 0.89). Reasoning RAG leverages six LLM powered agents spanning tasks including retrieval, KG query generation, abstraction, checker, rewrite, and summarization. This agentic framework reaches an overall accuracy of 94.15% in EHR-based medical Question Answering (QA), outperforming benchmarks that use either no agent or only partial agent integration. Our system also presents high readability (median Flesch Reading Ease 77.23; median Flesch Kincaid Grade 5.6), robustness (ANOVA F-value 0.6126, p<0.1), and stability (ANOVA F-value 0.782, p<0.1). The promising performance of the AIPatient system highlights its potential to support a wide range of applications, including medical education, model evaluation, and system integration.

Title: ReviveDiff: A Universal Diffusion Model for Restoring Images in Adverse Weather Conditions

• Authors: Wenfeng Huang, Guoan Xu, Wenjing Jia, Stuart Perry, Guangwei Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.18932
• Pdf URL: https://arxiv.org/pdf/2409.18932
• Copy Paste: [[2409.18932]] ReviveDiff: A Universal Diffusion Model for Restoring Images in Adverse Weather Conditions(https://arxiv.org/abs/2409.18932)
• Keywords: diffusion
• Abstract: Images captured in challenging environments--such as nighttime, foggy, rainy weather, and underwater--often suffer from significant degradation, resulting in a substantial loss of visual quality. Effective restoration of these degraded images is critical for the subsequent vision tasks. While many existing approaches have successfully incorporated specific priors for individual tasks, these tailored solutions limit their applicability to other degradations. In this work, we propose a universal network architecture, dubbed "ReviveDiff", which can address a wide range of degradations and bring images back to life by enhancing and restoring their quality. Our approach is inspired by the observation that, unlike degradation caused by movement or electronic issues, quality degradation under adverse conditions primarily stems from natural media (such as fog, water, and low luminance), which generally preserves the original structures of objects. To restore the quality of such images, we leveraged the latest advancements in diffusion models and developed ReviveDiff to restore image quality from both macro and micro levels across some key factors determining image quality, such as sharpness, distortion, noise level, dynamic range, and color accuracy. We rigorously evaluated ReviveDiff on seven benchmark datasets covering five types of degrading conditions: Rainy, Underwater, Low-light, Smoke, and Nighttime Hazy. Our experimental results demonstrate that ReviveDiff outperforms the state-of-the-art methods both quantitatively and visually.

Title: $O(d/T)$ Convergence Theory for Diffusion Probabilistic Models under Minimal Assumptions

• Authors: Gen Li, Yuling Yan
• Subjects: cs.LG, cs.AI, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2409.18959
• Pdf URL: https://arxiv.org/pdf/2409.18959
• Copy Paste: [[2409.18959]] $O(d/T)$ Convergence Theory for Diffusion Probabilistic Models under Minimal Assumptions(https://arxiv.org/abs/2409.18959)
• Keywords: diffusion, generative
• Abstract: Score-based diffusion models, which generate new data by learning to reverse a diffusion process that perturbs data from the target distribution into noise, have achieved remarkable success across various generative tasks. Despite their superior empirical performance, existing theoretical guarantees are often constrained by stringent assumptions or suboptimal convergence rates. In this paper, we establish a fast convergence theory for a popular SDE-based sampler under minimal assumptions. Our analysis shows that, provided $\ell_{2}$-accurate estimates of the score functions, the total variation distance between the target and generated distributions is upper bounded by $O(d/T)$ (ignoring logarithmic factors), where $d$ is the data dimensionality and $T$ is the number of steps. This result holds for any target distribution with finite first-order moment. To our knowledge, this improves upon existing convergence theory for both the SDE-based sampler and another ODE-based sampler, while imposing minimal assumptions on the target data distribution and score estimates. This is achieved through a novel set of analytical tools that provides a fine-grained characterization of how the error propagates at each step of the reverse process.

Title: ProMerge: Prompt and Merge for Unsupervised Instance Segmentation

• Authors: Dylan Li, Gyungin Shin
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.18961
• Pdf URL: https://arxiv.org/pdf/2409.18961
• Copy Paste: [[2409.18961]] ProMerge: Prompt and Merge for Unsupervised Instance Segmentation(https://arxiv.org/abs/2409.18961)
• Keywords: self-supervised
• Abstract: Unsupervised instance segmentation aims to segment distinct object instances in an image without relying on human-labeled data. This field has recently seen significant advancements, partly due to the strong local correspondences afforded by rich visual feature representations from self-supervised models (e.g., DINO). Recent state-of-the-art approaches use self-supervised features to represent images as graphs and solve a generalized eigenvalue system (i.e., normalized-cut) to generate foreground masks. While effective, this strategy is limited by its attendant computational demands, leading to slow inference speeds. In this paper, we propose Prompt and Merge (ProMerge), which leverages self-supervised visual features to obtain initial groupings of patches and applies a strategic merging to these segments, aided by a sophisticated background-based mask pruning technique. ProMerge not only yields competitive results but also offers a significant reduction in inference time compared to state-of-the-art normalized-cut-based approaches. Furthermore, when training an object detector using our mask predictions as pseudo-labels, the resulting detector surpasses the current leading unsupervised model on various challenging instance segmentation benchmarks.

Title: Exploring Token Pruning in Vision State Space Models

• Authors: Zheng Zhan, Zhenglun Kong, Yifan Gong, Yushu Wu, Zichong Meng, Hangyu Zheng, Xuan Shen, Stratis Ioannidis, Wei Niu, Pu Zhao, Yanzhi Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18962
• Pdf URL: https://arxiv.org/pdf/2409.18962
• Copy Paste: [[2409.18962]] Exploring Token Pruning in Vision State Space Models(https://arxiv.org/abs/2409.18962)
• Keywords: foundation model
• Abstract: State Space Models (SSMs) have the advantage of keeping linear computational complexity compared to attention modules in transformers, and have been applied to vision tasks as a new type of powerful vision foundation model. Inspired by the observations that the final prediction in vision transformers (ViTs) is only based on a subset of most informative tokens, we take the novel step of enhancing the efficiency of SSM-based vision models through token-based pruning. However, direct applications of existing token pruning techniques designed for ViTs fail to deliver good performance, even with extensive fine-tuning. To address this issue, we revisit the unique computational characteristics of SSMs and discover that naive application disrupts the sequential token positions. This insight motivates us to design a novel and general token pruning method specifically for SSM-based vision models. We first introduce a pruning-aware hidden state alignment method to stabilize the neighborhood of remaining tokens for performance enhancement. Besides, based on our detailed analysis, we propose a token importance evaluation method adapted for SSM models, to guide the token pruning. With efficient implementation and practical acceleration methods, our method brings actual speedup. Extensive experiments demonstrate that our approach can achieve significant computation reduction with minimal impact on performance across different tasks. Notably, we achieve 81.7\% accuracy on ImageNet with a 41.6\% reduction in the FLOPs for pruned PlainMamba-L3. Furthermore, our work provides deeper insights into understanding the behavior of SSM-based vision models for future research.

Title: PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation

• Authors: Shaowei Liu, Zhongzheng Ren, Saurabh Gupta, Shenlong Wang
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.18964
• Pdf URL: https://arxiv.org/pdf/2409.18964
• Copy Paste: [[2409.18964]] PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation(https://arxiv.org/abs/2409.18964)
• Keywords: diffusion, generative
• Abstract: We present PhysGen, a novel image-to-video generation method that converts a single image and an input condition (e.g., force and torque applied to an object in the image) to produce a realistic, physically plausible, and temporally consistent video. Our key insight is to integrate model-based physical simulation with a data-driven video generation process, enabling plausible image-space dynamics. At the heart of our system are three core components: (i) an image understanding module that effectively captures the geometry, materials, and physical parameters of the image; (ii) an image-space dynamics simulation model that utilizes rigid-body physics and inferred parameters to simulate realistic behaviors; and (iii) an image-based rendering and refinement module that leverages generative video diffusion to produce realistic video footage featuring the simulated motion. The resulting videos are realistic in both physics and appearance and are even precisely controllable, showcasing superior results over existing data-driven image-to-video generation works through quantitative comparison and comprehensive user study. PhysGen's resulting videos can be used for various downstream applications, such as turning an image into a realistic animation or allowing users to interact with the image and create various dynamics. Project page: this https URL