2024-04-15

Title: GRANP: A Graph Recurrent Attentive Neural Process Model for Vehicle Trajectory Prediction

• Authors: Yuhao Luo, Kehua Chen, Meixin Zhu
• Subjects: cs.LG, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.08004
• Pdf URL: https://arxiv.org/pdf/2404.08004
• Copy Paste: [[2404.08004]] GRANP: A Graph Recurrent Attentive Neural Process Model for Vehicle Trajectory Prediction(https://arxiv.org/abs/2404.08004)
• Keywords: generative
• Abstract: As a vital component in autonomous driving, accurate trajectory prediction effectively prevents traffic accidents and improves driving efficiency. To capture complex spatial-temporal dynamics and social interactions, recent studies developed models based on advanced deep-learning methods. On the other hand, recent studies have explored the use of deep generative models to further account for trajectory uncertainties. However, the current approaches demonstrating indeterminacy involve inefficient and time-consuming practices such as sampling from trained models. To fill this gap, we proposed a novel model named Graph Recurrent Attentive Neural Process (GRANP) for vehicle trajectory prediction while efficiently quantifying prediction uncertainty. In particular, GRANP contains an encoder with deterministic and latent paths, and a decoder for prediction. The encoder, including stacked Graph Attention Networks, LSTM and 1D convolutional layers, is employed to extract spatial-temporal relationships. The decoder is used to learn a latent distribution and thus quantify prediction uncertainty. To reveal the effectiveness of our model, we evaluate the performance of GRANP on the highD dataset. Extensive experiments show that GRANP achieves state-of-the-art results and can efficiently quantify uncertainties. Additionally, we undertake an intuitive case study that showcases the interpretability of the proposed approach. The code is available at https://github.com/joy-driven/GRANP.

Title: Rethinking Artistic Copyright Infringements in the Era of Text-to-Image Generative Models

• Authors: Mazda Moayeri, Samyadeep Basu, Sriram Balasubramanian, Priyatham Kattakinda, Atoosa Chengini, Robert Brauneis, Soheil Feizi
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.08030
• Pdf URL: https://arxiv.org/pdf/2404.08030
• Copy Paste: [[2404.08030]] Rethinking Artistic Copyright Infringements in the Era of Text-to-Image Generative Models(https://arxiv.org/abs/2404.08030)
• Keywords: diffusion, generative
• Abstract: Recent text-to-image generative models such as Stable Diffusion are extremely adept at mimicking and generating copyrighted content, raising concerns amongst artists that their unique styles may be improperly copied. Understanding how generative models copy "artistic style" is more complex than duplicating a single image, as style is comprised by a set of elements (or signature) that frequently co-occurs across a body of work, where each individual work may vary significantly. In our paper, we first reformulate the problem of "artistic copyright infringement" to a classification problem over image sets, instead of probing image-wise similarities. We then introduce ArtSavant, a practical (i.e., efficient and easy to understand) tool to (i) determine the unique style of an artist by comparing it to a reference dataset of works from 372 artists curated from WikiArt, and (ii) recognize if the identified style reappears in generated images. We leverage two complementary methods to perform artistic style classification over image sets, includingTagMatch, which is a novel inherently interpretable and attributable method, making it more suitable for broader use by non-technical stake holders (artists, lawyers, judges, etc). Leveraging ArtSavant, we then perform a large-scale empirical study to provide quantitative insight on the prevalence of artistic style copying across 3 popular text-to-image generative models. Namely, amongst a dataset of prolific artists (including many famous ones), only 20% of them appear to have their styles be at a risk of copying via simple prompting of today's popular text-to-image generative models.

Title: Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models

• Authors: Tanmay Gautam, Youngsuk Park, Hao Zhou, Parameswaran Raman, Wooseok Ha
• Subjects: cs.LG, cs.AI, cs.CL, math.OC
• Abstract URL: https://arxiv.org/abs/2404.08080
• Pdf URL: https://arxiv.org/pdf/2404.08080
• Copy Paste: [[2404.08080]] Variance-reduced Zeroth-Order Methods for Fine-Tuning Language Models(https://arxiv.org/abs/2404.08080)
• Keywords: in-context
• Abstract: Fine-tuning language models (LMs) has demonstrated success in a wide array of downstream tasks. However, as LMs are scaled up, the memory requirements for backpropagation become prohibitively high. Zeroth-order (ZO) optimization methods can leverage memory-efficient forward passes to estimate gradients. More recently, MeZO, an adaptation of ZO-SGD, has been shown to consistently outperform zero-shot and in-context learning when combined with suitable task prompts. In this work, we couple ZO methods with variance reduction techniques to enhance stability and convergence for inference-based LM fine-tuning. We introduce Memory-Efficient Zeroth-Order Stochastic Variance-Reduced Gradient (MeZO-SVRG) and demonstrate its efficacy across multiple LM fine-tuning tasks, eliminating the reliance on task-specific prompts. Evaluated across a range of both masked and autoregressive LMs on benchmark GLUE tasks, MeZO-SVRG outperforms MeZO with up to 20% increase in test accuracies in both full- and partial-parameter fine-tuning settings. MeZO-SVRG benefits from reduced computation time as it often surpasses MeZO's peak test accuracy with a $2\times$ reduction in GPU-hours. MeZO-SVRG significantly reduces the required memory footprint compared to first-order SGD, i.e. by $2\times$ for autoregressive models. Our experiments highlight that MeZO-SVRG's memory savings progressively improve compared to SGD with larger batch sizes.

Title: Self-Supervised Learning of Color Constancy

• Authors: Markus R. Ernst, Francisco M. López, Arthur Aubret, Roland W. Fleming, Jochen Triesch
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08127
• Pdf URL: https://arxiv.org/pdf/2404.08127
• Copy Paste: [[2404.08127]] Self-Supervised Learning of Color Constancy(https://arxiv.org/abs/2404.08127)
• Keywords: self-supervised
• Abstract: Color constancy (CC) describes the ability of the visual system to perceive an object as having a relatively constant color despite changes in lighting conditions. While CC and its limitations have been carefully characterized in humans, it is still unclear how the visual system acquires this ability during development. Here, we present a first study showing that CC develops in a neural network trained in a self-supervised manner through an invariance learning objective. During learning, objects are presented under changing illuminations, while the network aims to map subsequent views of the same object onto close-by latent representations. This gives rise to representations that are largely invariant to the illumination conditions, offering a plausible example of how CC could emerge during human cognitive development via a form of self-supervised learning.

Title: Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentation

• Authors: Sina Hajimiri, Ismail Ben Ayed, Jose Dolz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.08181
• Pdf URL: https://arxiv.org/pdf/2404.08181
• Copy Paste: [[2404.08181]] Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentation(https://arxiv.org/abs/2404.08181)
• Keywords: foundation model
• Abstract: Despite the significant progress in deep learning for dense visual recognition problems, such as semantic segmentation, traditional methods are constrained by fixed class sets. Meanwhile, vision-language foundation models, such as CLIP, have showcased remarkable effectiveness in numerous zero-shot image-level tasks, owing to their robust generalizability. Recently, a body of work has investigated utilizing these models in open-vocabulary semantic segmentation (OVSS). However, existing approaches often rely on impractical supervised pre-training or access to additional pre-trained networks. In this work, we propose a strong baseline for training-free OVSS, termed Neighbour-Aware CLIP (NACLIP), representing a straightforward adaptation of CLIP tailored for this scenario. Our method enforces localization of patches in the self-attention of CLIP's vision transformer which, despite being crucial for dense prediction tasks, has been overlooked in the OVSS literature. By incorporating design choices favouring segmentation, our approach significantly improves performance without requiring additional data, auxiliary pre-trained networks, or extensive hyperparameter tuning, making it highly practical for real-world applications. Experiments are performed on 8 popular semantic segmentation benchmarks, yielding state-of-the-art performance on most scenarios. Our code is publicly available at https://github.com/sinahmr/NACLIP .

Title: Reducing hallucination in structured outputs via Retrieval-Augmented Generation

• Authors: Patrice Béchard, Orlando Marquez Ayala
• Subjects: cs.LG, cs.AI, cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2404.08189
• Pdf URL: https://arxiv.org/pdf/2404.08189
• Copy Paste: [[2404.08189]] Reducing hallucination in structured outputs via Retrieval-Augmented Generation(https://arxiv.org/abs/2404.08189)
• Keywords: generative
• Abstract: A common and fundamental limitation of Generative AI (GenAI) is its propensity to hallucinate. While large language models (LLM) have taken the world by storm, without eliminating or at least reducing hallucinations, real-world GenAI systems may face challenges in user adoption. In the process of deploying an enterprise application that produces workflows based on natural language requirements, we devised a system leveraging Retrieval Augmented Generation (RAG) to greatly improve the quality of the structured output that represents such workflows. Thanks to our implementation of RAG, our proposed system significantly reduces hallucinations in the output and improves the generalization of our LLM in out-of-domain settings. In addition, we show that using a small, well-trained retriever encoder can reduce the size of the accompanying LLM, thereby making deployments of LLM-based systems less resource-intensive.

Title: Uncertain Boundaries: Multidisciplinary Approaches to Copyright Issues in Generative AI

• Authors: Jocelyn Dzuong, Zichong Wang, Wenbin Zhang
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2404.08221
• Pdf URL: https://arxiv.org/pdf/2404.08221
• Copy Paste: [[2404.08221]] Uncertain Boundaries: Multidisciplinary Approaches to Copyright Issues in Generative AI(https://arxiv.org/abs/2404.08221)
• Keywords: generative
• Abstract: In the rapidly evolving landscape of generative artificial intelligence (AI), the increasingly pertinent issue of copyright infringement arises as AI advances to generate content from scraped copyrighted data, prompting questions about ownership and protection that impact professionals across various careers. With this in mind, this survey provides an extensive examination of copyright infringement as it pertains to generative AI, aiming to stay abreast of the latest developments and open problems. Specifically, it will first outline methods of detecting copyright infringement in mediums such as text, image, and video. Next, it will delve an exploration of existing techniques aimed at safeguarding copyrighted works from generative models. Furthermore, this survey will discuss resources and tools for users to evaluate copyright violations. Finally, insights into ongoing regulations and proposals for AI will be explored and compared. Through combining these disciplines, the implications of AI-driven content and copyright are thoroughly illustrated and brought into question.

Title: HCL-MTSAD: Hierarchical Contrastive Consistency Learning for Accurate Detection of Industrial Multivariate Time Series Anomalies

• Authors: Haili Sun, Yan Huang, Lansheng Han, Cai Fu, Chunjie Zhou
• Subjects: cs.LG, cs.AI, cs.CR, cs.IT, eess.SY
• Abstract URL: https://arxiv.org/abs/2404.08224
• Pdf URL: https://arxiv.org/pdf/2404.08224
• Copy Paste: [[2404.08224]] HCL-MTSAD: Hierarchical Contrastive Consistency Learning for Accurate Detection of Industrial Multivariate Time Series Anomalies(https://arxiv.org/abs/2404.08224)
• Keywords: self-supervised, anomaly
• Abstract: Multivariate Time Series (MTS) anomaly detection focuses on pinpointing samples that diverge from standard operational patterns, which is crucial for ensuring the safety and security of industrial applications. The primary challenge in this domain is to develop representations capable of discerning anomalies effectively. The prevalent methods for anomaly detection in the literature are predominantly reconstruction-based and predictive in nature. However, they typically concentrate on a single-dimensional instance level, thereby not fully harnessing the complex associations inherent in industrial MTS. To address this issue, we propose a novel self-supervised hierarchical contrastive consistency learning method for detecting anomalies in MTS, named HCL-MTSAD. It innovatively leverages data consistency at multiple levels inherent in industrial MTS, systematically capturing consistent associations across four latent levels-measurement, sample, channel, and process. By developing a multi-layer contrastive loss, HCL-MTSAD can extensively mine data consistency and spatio-temporal association, resulting in more informative representations. Subsequently, an anomaly discrimination module, grounded in self-supervised hierarchical contrastive learning, is designed to detect timestamp-level anomalies by calculating multi-scale data consistency. Extensive experiments conducted on six diverse MTS datasets retrieved from real cyber-physical systems and server machines, in comparison with 20 baselines, indicate that HCL-MTSAD's anomaly detection capability outperforms the state-of-the-art benchmark models by an average of 1.8\% in terms of F1 score.

Title: Balanced Mixed-Type Tabular Data Synthesis with Diffusion Models

• Authors: Zeyu Yang, Peikun Guo, Khadija Zanna, Akane Sano
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.08254
• Pdf URL: https://arxiv.org/pdf/2404.08254
• Copy Paste: [[2404.08254]] Balanced Mixed-Type Tabular Data Synthesis with Diffusion Models(https://arxiv.org/abs/2404.08254)
• Keywords: diffusion, generative
• Abstract: Diffusion models have emerged as a robust framework for various generative tasks, such as image and audio synthesis, and have also demonstrated a remarkable ability to generate mixed-type tabular data comprising both continuous and discrete variables. However, current approaches to training diffusion models on mixed-type tabular data tend to inherit the imbalanced distributions of features present in the training dataset, which can result in biased sampling. In this research, we introduce a fair diffusion model designed to generate balanced data on sensitive attributes. We present empirical evidence demonstrating that our method effectively mitigates the class imbalance in training data while maintaining the quality of the generated samples. Furthermore, we provide evidence that our approach outperforms existing methods for synthesizing tabular data in terms of performance and fairness.

Title: Struggle with Adversarial Defense? Try Diffusion

• Authors: Yujie Li, Yanbin Wang, Haitao xu, Bin Liu, Jianguo Sun, Zhenhao Guo, Wenrui Ma
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2404.08273
• Pdf URL: https://arxiv.org/pdf/2404.08273
• Copy Paste: [[2404.08273]] Struggle with Adversarial Defense? Try Diffusion(https://arxiv.org/abs/2404.08273)
• Keywords: diffusion, generative
• Abstract: Adversarial attacks induce misclassification by introducing subtle perturbations. Recently, diffusion models are applied to the image classifiers to improve adversarial robustness through adversarial training or by purifying adversarial noise. However, diffusion-based adversarial training often encounters convergence challenges and high computational expenses. Additionally, diffusion-based purification inevitably causes data shift and is deemed susceptible to stronger adaptive attacks. To tackle these issues, we propose the Truth Maximization Diffusion Classifier (TMDC), a generative Bayesian classifier that builds upon pre-trained diffusion models and the Bayesian theorem. Unlike data-driven classifiers, TMDC, guided by Bayesian principles, utilizes the conditional likelihood from diffusion models to determine the class probabilities of input images, thereby insulating against the influences of data shift and the limitations of adversarial training. Moreover, to enhance TMDC's resilience against more potent adversarial attacks, we propose an optimization strategy for diffusion classifiers. This strategy involves post-training the diffusion model on perturbed datasets with ground-truth labels as conditions, guiding the diffusion model to learn the data distribution and maximizing the likelihood under the ground-truth labels. The proposed method achieves state-of-the-art performance on the CIFAR10 dataset against heavy white-box attacks and strong adaptive attacks. Specifically, TMDC achieves robust accuracies of 82.81% against $l_{\infty}$ norm-bounded perturbations and 86.05% against $l_{2}$ norm-bounded perturbations, respectively, with $\epsilon=0.05$.

Title: GPN: Generative Point-based NeRF

• Authors: Haipeng Wang
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2404.08312
• Pdf URL: https://arxiv.org/pdf/2404.08312
• Copy Paste: [[2404.08312]] GPN: Generative Point-based NeRF(https://arxiv.org/abs/2404.08312)
• Keywords: generative
• Abstract: Scanning real-life scenes with modern registration devices typically gives incomplete point cloud representations, primarily due to the limitations of partial scanning, 3D occlusions, and dynamic light conditions. Recent works on processing incomplete point clouds have always focused on point cloud completion. However, these approaches do not ensure consistency between the completed point cloud and the captured images regarding color and geometry. We propose using Generative Point-based NeRF (GPN) to reconstruct and repair a partial cloud by fully utilizing the scanning images and the corresponding reconstructed cloud. The repaired point cloud can achieve multi-view consistency with the captured images at high spatial resolution. For the finetunes of a single scene, we optimize the global latent condition by incorporating an Auto-Decoder architecture while retaining multi-view consistency. As a result, the generated point clouds are smooth, plausible, and geometrically consistent with the partial scanning images. Extensive experiments on ShapeNet demonstrate that our works achieve competitive performances to the other state-of-the-art point cloud-based neural scene rendering and editing performances.

Title: Emerging Property of Masked Token for Effective Pre-training

• Authors: Hyesong Choi, Hunsang Lee, Seyoung Joung, Hyejin Park, Jiyeong Kim, Dongbo Min
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.08330
• Pdf URL: https://arxiv.org/pdf/2404.08330
• Copy Paste: [[2404.08330]] Emerging Property of Masked Token for Effective Pre-training(https://arxiv.org/abs/2404.08330)
• Keywords: self-supervised
• Abstract: Driven by the success of Masked Language Modeling (MLM), the realm of self-supervised learning for computer vision has been invigorated by the central role of Masked Image Modeling (MIM) in driving recent breakthroughs. Notwithstanding the achievements of MIM across various downstream tasks, its overall efficiency is occasionally hampered by the lengthy duration of the pre-training phase. This paper presents a perspective that the optimization of masked tokens as a means of addressing the prevailing issue. Initially, we delve into an exploration of the inherent properties that a masked token ought to possess. Within the properties, we principally dedicated to articulating and emphasizing the `data singularity' attribute inherent in masked tokens. Through a comprehensive analysis of the heterogeneity between masked tokens and visible tokens within pre-trained models, we propose a novel approach termed masked token optimization (MTO), specifically designed to improve model efficiency through weight recalibration and the enhancement of the key property of masked tokens. The proposed method serves as an adaptable solution that seamlessly integrates into any MIM approach that leverages masked tokens. As a result, MTO achieves a considerable improvement in pre-training efficiency, resulting in an approximately 50% reduction in pre-training epochs required to attain converged performance of the recent approaches.

Title: Counterfactual Explanations for Face Forgery Detection via Adversarial Removal of Artifacts

• Authors: Yang Li, Songlin Yang, Wei Wang, Ziwen He, Bo Peng, Jing Dong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.08341
• Pdf URL: https://arxiv.org/pdf/2404.08341
• Copy Paste: [[2404.08341]] Counterfactual Explanations for Face Forgery Detection via Adversarial Removal of Artifacts(https://arxiv.org/abs/2404.08341)
• Keywords: generative
• Abstract: Highly realistic AI generated face forgeries known as deepfakes have raised serious social concerns. Although DNN-based face forgery detection models have achieved good performance, they are vulnerable to latest generative methods that have less forgery traces and adversarial attacks. This limitation of generalization and robustness hinders the credibility of detection results and requires more explanations. In this work, we provide counterfactual explanations for face forgery detection from an artifact removal perspective. Specifically, we first invert the forgery images into the StyleGAN latent space, and then adversarially optimize their latent representations with the discrimination supervision from the target detection model. We verify the effectiveness of the proposed explanations from two aspects: (1) Counterfactual Trace Visualization: the enhanced forgery images are useful to reveal artifacts by visually contrasting the original images and two different visualization methods; (2) Transferable Adversarial Attacks: the adversarial forgery images generated by attacking the detection model are able to mislead other detection models, implying the removed artifacts are general. Extensive experiments demonstrate that our method achieves over 90% attack success rate and superior attack transferability. Compared with naive adversarial noise methods, our method adopts both generative and discriminative model priors, and optimize the latent representations in a synthesis-by-analysis way, which forces the search of counterfactual explanations on the natural face manifold. Thus, more general counterfactual traces can be found and better adversarial attack transferability can be achieved.

Title: OmniSat: Self-Supervised Modality Fusion for Earth Observation

• Authors: Guillaume Astruc, Nicolas Gonthier, Clement Mallet, Loic Landrieu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.08351
• Pdf URL: https://arxiv.org/pdf/2404.08351
• Copy Paste: [[2404.08351]] OmniSat: Self-Supervised Modality Fusion for Earth Observation(https://arxiv.org/abs/2404.08351)
• Keywords: self-supervised
• Abstract: The field of Earth Observations (EO) offers a wealth of data from diverse sensors, presenting a great opportunity for advancing self-supervised multimodal learning. However, current multimodal EO datasets and models focus on a single data type, either mono-date images or time series, which limits their expressivity. We introduce OmniSat, a novel architecture that exploits the spatial alignment between multiple EO modalities to learn expressive multimodal representations without labels. To demonstrate the advantages of combining modalities of different natures, we augment two existing datasets with new modalities. As demonstrated on three downstream tasks: forestry, land cover classification, and crop mapping. OmniSat can learn rich representations in an unsupervised manner, leading to improved performance in the semi- and fully-supervised settings, even when only one modality is available for inference. The code and dataset are available at github.com/gastruc/OmniSat.

Title: Let It Flow: Simultaneous Optimization of 3D Flow and Object Clustering

• Authors: Patrik Vacek, David Hurych, Tomáš Svoboda, Karel Zimmermann
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.08363
• Pdf URL: https://arxiv.org/pdf/2404.08363
• Copy Paste: [[2404.08363]] Let It Flow: Simultaneous Optimization of 3D Flow and Object Clustering(https://arxiv.org/abs/2404.08363)
• Keywords: self-supervised
• Abstract: We study the problem of self-supervised 3D scene flow estimation from real large-scale raw point cloud sequences, which is crucial to various tasks like trajectory prediction or instance segmentation. In the absence of ground truth scene flow labels, contemporary approaches concentrate on deducing optimizing flow across sequential pairs of point clouds by incorporating structure based regularization on flow and object rigidity. The rigid objects are estimated by a variety of 3D spatial clustering methods. While state-of-the-art methods successfully capture overall scene motion using the Neural Prior structure, they encounter challenges in discerning multi-object motions. We identified the structural constraints and the use of large and strict rigid clusters as the main pitfall of the current approaches and we propose a novel clustering approach that allows for combination of overlapping soft clusters as well as non-overlapping rigid clusters representation. Flow is then jointly estimated with progressively growing non-overlapping rigid clusters together with fixed size overlapping soft clusters. We evaluate our method on multiple datasets with LiDAR point clouds, demonstrating the superior performance over the self-supervised baselines reaching new state of the art results. Our method especially excels in resolving flow in complicated dynamic scenes with multiple independently moving objects close to each other which includes pedestrians, cyclists and other vulnerable road users. Our codes will be publicly available.

Title: Graph data augmentation with Gromow-Wasserstein Barycenters

• Authors: Andrea Ponti
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.08376
• Pdf URL: https://arxiv.org/pdf/2404.08376
• Copy Paste: [[2404.08376]] Graph data augmentation with Gromow-Wasserstein Barycenters(https://arxiv.org/abs/2404.08376)
• Keywords: generative
• Abstract: Graphs are ubiquitous in various fields, and deep learning methods have been successful applied in graph classification tasks. However, building large and diverse graph datasets for training can be expensive. While augmentation techniques exist for structured data like images or numerical data, the augmentation of graph data remains challenging. This is primarily due to the complex and non-Euclidean nature of graph data. In this paper, it has been proposed a novel augmentation strategy for graphs that operates in a non-Euclidean space. This approach leverages graphon estimation, which models the generative mechanism of networks sequences. Computational results demonstrate the effectiveness of the proposed augmentation framework in improving the performance of graph classification models. Additionally, using a non-Euclidean distance, specifically the Gromow-Wasserstein distance, results in better approximations of the graphon. This framework also provides a means to validate different graphon estimation approaches, particularly in real-world scenarios where the true graphon is unknown.

Title: Adapting the Segment Anything Model During Usage in Novel Situations

• Authors: Robin Schön, Julian Lorenz, Katja Ludwig, Rainer Lienhart
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.08421
• Pdf URL: https://arxiv.org/pdf/2404.08421
• Copy Paste: [[2404.08421]] Adapting the Segment Anything Model During Usage in Novel Situations(https://arxiv.org/abs/2404.08421)
• Keywords: foundation model
• Abstract: The interactive segmentation task consists in the creation of object segmentation masks based on user interactions. The most common way to guide a model towards producing a correct segmentation consists in clicks on the object and background. The recently published Segment Anything Model (SAM) supports a generalized version of the interactive segmentation problem and has been trained on an object segmentation dataset which contains 1.1B masks. Though being trained extensively and with the explicit purpose of serving as a foundation model, we show significant limitations of SAM when being applied for interactive segmentation on novel domains or object types. On the used datasets, SAM displays a failure rate $\text{FR}_{30}@90$ of up to $72.6 \%$. Since we still want such foundation models to be immediately applicable, we present a framework that can adapt SAM during immediate usage. For this we will leverage the user interactions and masks, which are constructed during the interactive segmentation process. We use this information to generate pseudo-labels, which we use to compute a loss function and optimize a part of the SAM model. The presented method causes a relative reduction of up to $48.1 \%$ in the $\text{FR}_{20}@85$ and $46.6 \%$ in the $\text{FR}_{30}@90$ metrics.

Title: An improved tabular data generator with VAE-GMM integration

• Authors: Patricia A. Apellániz, Juan Parras, Santiago Zazo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.08434
• Pdf URL: https://arxiv.org/pdf/2404.08434
• Copy Paste: [[2404.08434]] An improved tabular data generator with VAE-GMM integration(https://arxiv.org/abs/2404.08434)
• Keywords: generative
• Abstract: The rising use of machine learning in various fields requires robust methods to create synthetic tabular data. Data should preserve key characteristics while addressing data scarcity challenges. Current approaches based on Generative Adversarial Networks, such as the state-of-the-art CTGAN model, struggle with the complex structures inherent in tabular data. These data often contain both continuous and discrete features with non-Gaussian distributions. Therefore, we propose a novel Variational Autoencoder (VAE)-based model that addresses these limitations. Inspired by the TVAE model, our approach incorporates a Bayesian Gaussian Mixture model (BGM) within the VAE architecture. This avoids the limitations imposed by assuming a strictly Gaussian latent space, allowing for a more accurate representation of the underlying data distribution during data generation. Furthermore, our model offers enhanced flexibility by allowing the use of various differentiable distributions for individual features, making it possible to handle both continuous and discrete data types. We thoroughly validate our model on three real-world datasets with mixed data types, including two medically relevant ones, based on their resemblance and utility. This evaluation demonstrates significant outperformance against CTGAN and TVAE, establishing its potential as a valuable tool for generating synthetic tabular data in various domains, particularly in healthcare.

Title: TSLANet: Rethinking Transformers for Time Series Representation Learning

• Authors: Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Xiaoli Li
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2404.08472
• Pdf URL: https://arxiv.org/pdf/2404.08472
• Copy Paste: [[2404.08472]] TSLANet: Rethinking Transformers for Time Series Representation Learning(https://arxiv.org/abs/2404.08472)
• Keywords: self-supervised, anomaly
• Abstract: Time series data, characterized by its intrinsic long and short-range dependencies, poses a unique challenge across analytical applications. While Transformer-based models excel at capturing long-range dependencies, they face limitations in noise sensitivity, computational efficiency, and overfitting with smaller datasets. In response, we introduce a novel Time Series Lightweight Adaptive Network (TSLANet), as a universal convolutional model for diverse time series tasks. Specifically, we propose an Adaptive Spectral Block, harnessing Fourier analysis to enhance feature representation and to capture both long-term and short-term interactions while mitigating noise via adaptive thresholding. Additionally, we introduce an Interactive Convolution Block and leverage self-supervised learning to refine the capacity of TSLANet for decoding complex temporal patterns and improve its robustness on different datasets. Our comprehensive experiments demonstrate that TSLANet outperforms state-of-the-art models in various tasks spanning classification, forecasting, and anomaly detection, showcasing its resilience and adaptability across a spectrum of noise levels and data sizes. The code is available at \url{https://github.com/emadeldeen24/TSLANet}

Title: Decoding AI: The inside story of data analysis in ChatGPT

• Authors: Ozan Evkaya, Miguel de Carvalho
• Subjects: cs.LG, cs.CL, stat.CO
• Abstract URL: https://arxiv.org/abs/2404.08480
• Pdf URL: https://arxiv.org/pdf/2404.08480
• Copy Paste: [[2404.08480]] Decoding AI: The inside story of data analysis in ChatGPT(https://arxiv.org/abs/2404.08480)
• Keywords: generative
• Abstract: As a result of recent advancements in generative AI, the field of Data Science is prone to various changes. This review critically examines the Data Analysis (DA) capabilities of ChatGPT assessing its performance across a wide range of tasks. While DA provides researchers and practitioners with unprecedented analytical capabilities, it is far from being perfect, and it is important to recognize and address its limitations.

Title: Dataset Reset Policy Optimization for RLHF

• Authors: Jonathan D. Chang, Wenhao Shan, Owen Oertell, Kianté Brantley, Dipendra Misra, Jason D. Lee, Wen Sun
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.08495
• Pdf URL: https://arxiv.org/pdf/2404.08495
• Copy Paste: [[2404.08495]] Dataset Reset Policy Optimization for RLHF(https://arxiv.org/abs/2404.08495)
• Keywords: generative
• Abstract: Reinforcement Learning (RL) from Human Preference-based feedback is a popular paradigm for fine-tuning generative models, which has produced impressive models such as GPT-4 and Claude3 Opus. This framework often consists of two steps: learning a reward model from an offline preference dataset followed by running online RL to optimize the learned reward model. In this work, leveraging the idea of reset, we propose a new RLHF algorithm with provable guarantees. Motivated by the fact that offline preference dataset provides informative states (i.e., data that is preferred by the labelers), our new algorithm, Dataset Reset Policy Optimization (DR-PO), integrates the existing offline preference dataset into the online policy training procedure via dataset reset: it directly resets the policy optimizer to the states in the offline dataset, instead of always starting from the initial state distribution. In theory, we show that DR-PO learns to perform at least as good as any policy that is covered by the offline dataset under general function approximation with finite sample complexity. In experiments, we demonstrate that on both the TL;DR summarization and the Anthropic Helpful Harmful (HH) dataset, the generation from DR-PO is better than that from Proximal Policy Optimization (PPO) and Direction Preference Optimization (DPO), under the metric of GPT4 win-rate. Code for this work can be found at https://github.com/Cornell-RL/drpo.

Title: ChatGPT and general-purpose AI count fruits in pictures surprisingly well

• Authors: Konlavach Mengsuwan, Juan Camilo Rivera Palacio, Masahiro Ryo
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.08515
• Pdf URL: https://arxiv.org/pdf/2404.08515
• Copy Paste: [[2404.08515]] ChatGPT and general-purpose AI count fruits in pictures surprisingly well(https://arxiv.org/abs/2404.08515)
• Keywords: foundation model
• Abstract: Object counting is a popular task in deep learning applications in various domains, including agriculture. A conventional deep learning approach requires a large amount of training data, often a logistic problem in a real-world application. To address this issue, we examined how well ChatGPT (GPT4V) and a general-purpose AI (foundation model for object counting, T-Rex) can count the number of fruit bodies (coffee cherries) in 100 images. The foundation model with few-shot learning outperformed the trained YOLOv8 model (R2 = 0.923 and 0.900, respectively). ChatGPT also showed some interesting potential, especially when few-shot learning with human feedback was applied (R2 = 0.360 and 0.460, respectively). Moreover, we examined the time required for implementation as a practical question. Obtaining the results with the foundation model and ChatGPT were much shorter than the YOLOv8 model (0.83 hrs, 1.75 hrs, and 161 hrs). We interpret these results as two surprises for deep learning users in applied domains: a foundation model with few-shot domain-specific learning can drastically save time and effort compared to the conventional approach, and ChatGPT can reveal a relatively good performance. Both approaches do not need coding skills, which can foster AI education and dissemination.

Title: Masked Image Modeling as a Framework for Self-Supervised Learning across Eye Movements

• Authors: Robin Weiler, Matthias Brucklacher, Cyriel M. A. Pennartz, Sander M. Bohté
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.08526
• Pdf URL: https://arxiv.org/pdf/2404.08526
• Copy Paste: [[2404.08526]] Masked Image Modeling as a Framework for Self-Supervised Learning across Eye Movements(https://arxiv.org/abs/2404.08526)
• Keywords: self-supervised, generative
• Abstract: To make sense of their surroundings, intelligent systems must transform complex sensory inputs to structured codes that are reduced to task-relevant information such as object category. Biological agents achieve this in a largely autonomous manner, presumably via self-\allowbreak super-\allowbreak vised learning. Whereas previous attempts to model the underlying mechanisms were largely discriminative in nature, there is ample evidence that the brain employs a generative model of the world. Here, we propose that eye movements, in combination with the focused nature of primate vision, constitute a generative, self-supervised task of predicting and revealing visual information. We construct a proof-of-principle model starting from the framework of masked image modeling (MIM), a common approach in deep representation learning. To do so, we analyze how core components of MIM such as masking technique and data augmentation influence the formation of category-specific representations. This allows us not only to better understand the principles behind MIM, but to then reassemble a MIM more in line with the focused nature of biological perception. From a theoretical angle, we find that MIM disentangles neurons in latent space, a property that has been suggested to structure visual representations in primates, without explicit regulation. Together with previous findings of invariance learning, this highlights an interesting connection of MIM to latent regularization approaches for self-supervised learning. The source code is available under https://github.com/RobinWeiler/FocusMIM

Title: Text Prompt with Normality Guidance for Weakly Supervised Video Anomaly Detection

• Authors: Zhiwei Yang, Jing Liu, Peng Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.08531
• Pdf URL: https://arxiv.org/pdf/2404.08531
• Copy Paste: [[2404.08531]] Text Prompt with Normality Guidance for Weakly Supervised Video Anomaly Detection(https://arxiv.org/abs/2404.08531)
• Keywords: anomaly
• Abstract: Weakly supervised video anomaly detection (WSVAD) is a challenging task. Generating fine-grained pseudo-labels based on weak-label and then self-training a classifier is currently a promising solution. However, since the existing methods use only RGB visual modality and the utilization of category text information is neglected, thus limiting the generation of more accurate pseudo-labels and affecting the performance of self-training. Inspired by the manual labeling process based on the event description, in this paper, we propose a novel pseudo-label generation and self-training framework based on Text Prompt with Normality Guidance (TPWNG) for WSVAD. Our idea is to transfer the rich language-visual knowledge of the contrastive language-image pre-training (CLIP) model for aligning the video event description text and corresponding video frames to generate pseudo-labels. Specifically, We first fine-tune the CLIP for domain adaptation by designing two ranking losses and a distributional inconsistency loss. Further, we propose a learnable text prompt mechanism with the assist of a normality visual prompt to further improve the matching accuracy of video event description text and video frames. Then, we design a pseudo-label generation module based on the normality guidance to infer reliable frame-level pseudo-labels. Finally, we introduce a temporal context self-adaptive learning module to learn the temporal dependencies of different video events more flexibly and accurately. Extensive experiments show that our method achieves state-of-the-art performance on two benchmark datasets, UCF-Crime and XD-Viole

Title: FashionFail: Addressing Failure Cases in Fashion Object Detection and Segmentation

• Authors: Riza Velioglu, Robin Chan, Barbara Hammer
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.08582
• Pdf URL: https://arxiv.org/pdf/2404.08582
• Copy Paste: [[2404.08582]] FashionFail: Addressing Failure Cases in Fashion Object Detection and Segmentation(https://arxiv.org/abs/2404.08582)
• Keywords: foundation model
• Abstract: In the realm of fashion object detection and segmentation for online shopping images, existing state-of-the-art fashion parsing models encounter limitations, particularly when exposed to non-model-worn apparel and close-up shots. To address these failures, we introduce FashionFail; a new fashion dataset with e-commerce images for object detection and segmentation. The dataset is efficiently curated using our novel annotation tool that leverages recent foundation models. The primary objective of FashionFail is to serve as a test bed for evaluating the robustness of models. Our analysis reveals the shortcomings of leading models, such as Attribute-Mask R-CNN and Fashionformer. Additionally, we propose a baseline approach using naive data augmentation to mitigate common failure cases and improve model robustness. Through this work, we aim to inspire and support further research in fashion item detection and segmentation for industrial applications. The dataset, annotation tool, code, and models are available at \url{https://rizavelioglu.github.io/fashionfail/}.

Title: Pathological Primitive Segmentation Based on Visual Foundation Model with Zero-Shot Mask Generation

• Authors: Abu Bakor Hayat Arnob, Xiangxue Wang, Yiping Jiao, Xiao Gan, Wenlong Ming, Jun Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.08584
• Pdf URL: https://arxiv.org/pdf/2404.08584
• Copy Paste: [[2404.08584]] Pathological Primitive Segmentation Based on Visual Foundation Model with Zero-Shot Mask Generation(https://arxiv.org/abs/2404.08584)
• Keywords: foundation model
• Abstract: Medical image processing usually requires a model trained with carefully crafted datasets due to unique image characteristics and domain-specific challenges, especially in pathology. Primitive detection and segmentation in digitized tissue samples are essential for objective and automated diagnosis and prognosis of cancer. SAM (Segment Anything Model) has recently been developed to segment general objects from natural images with high accuracy, but it requires human prompts to generate masks. In this work, we present a novel approach that adapts pre-trained natural image encoders of SAM for detection-based region proposals. Regions proposed by a pre-trained encoder are sent to cascaded feature propagation layers for projection. Then, local semantic and global context is aggregated from multi-scale for bounding box localization and classification. Finally, the SAM decoder uses the identified bounding boxes as essential prompts to generate a comprehensive primitive segmentation map. The entire base framework, SAM, requires no additional training or fine-tuning but could produce an end-to-end result for two fundamental segmentation tasks in pathology. Our method compares with state-of-the-art models in F1 score for nuclei detection and binary/multiclass panoptic(bPQ/mPQ) and mask quality(dice) for segmentation quality on the PanNuke dataset while offering end-to-end efficiency. Our model also achieves remarkable Average Precision (+4.5%) on the secondary dataset (HuBMAP Kidney) compared to Faster RCNN. The code is publicly available at https://github.com/learner-codec/autoprom_sam.

Title: FCert: Certifiably Robust Few-Shot Classification in the Era of Foundation Models

• Authors: Yanting Wang, Wei Zou, Jinyuan Jia
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2404.08631
• Pdf URL: https://arxiv.org/pdf/2404.08631
• Copy Paste: [[2404.08631]] FCert: Certifiably Robust Few-Shot Classification in the Era of Foundation Models(https://arxiv.org/abs/2404.08631)
• Keywords: foundation model
• Abstract: Few-shot classification with foundation models (e.g., CLIP, DINOv2, PaLM-2) enables users to build an accurate classifier with a few labeled training samples (called support samples) for a classification task. However, an attacker could perform data poisoning attacks by manipulating some support samples such that the classifier makes the attacker-desired, arbitrary prediction for a testing input. Empirical defenses cannot provide formal robustness guarantees, leading to a cat-and-mouse game between the attacker and defender. Existing certified defenses are designed for traditional supervised learning, resulting in sub-optimal performance when extended to few-shot classification. In our work, we propose FCert, the first certified defense against data poisoning attacks to few-shot classification. We show our FCert provably predicts the same label for a testing input under arbitrary data poisoning attacks when the total number of poisoned support samples is bounded. We perform extensive experiments on benchmark few-shot classification datasets with foundation models released by OpenAI, Meta, and Google in both vision and text domains. Our experimental results show our FCert: 1) maintains classification accuracy without attacks, 2) outperforms existing state-of-the-art certified defenses for data poisoning attacks, and 3) is efficient and general.

Title: Probing the 3D Awareness of Visual Foundation Models

• Authors: Mohamed El Banani, Amit Raj, Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, Varun Jampani
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.08636
• Pdf URL: https://arxiv.org/pdf/2404.08636
• Copy Paste: [[2404.08636]] Probing the 3D Awareness of Visual Foundation Models(https://arxiv.org/abs/2404.08636)
• Keywords: foundation model
• Abstract: Recent advances in large-scale pretraining have yielded visual foundation models with strong capabilities. Not only can recent models generalize to arbitrary images for their training task, their intermediate representations are useful for other visual tasks such as detection and segmentation. Given that such models can classify, delineate, and localize objects in 2D, we ask whether they also represent their 3D structure? In this work, we analyze the 3D awareness of visual foundation models. We posit that 3D awareness implies that representations (1) encode the 3D structure of the scene and (2) consistently represent the surface across views. We conduct a series of experiments using task-specific probes and zero-shot inference procedures on frozen features. Our experiments reveal several limitations of the current models. Our code and analysis can be found at https://github.com/mbanani/probe3d.