2024-05-07

Title: Prospective Role of Foundation Models in Advancing Autonomous Vehicles

• Authors: Jianhua Wu, Bingzhao Gao, Jincheng Gao, Jianhao Yu, Hongqing Chu, Qiankun Yu, Xun Gong, Yi Chang, H. Eric Tseng, Hong Chen, Jie Chen
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2405.02288
• Pdf URL: https://arxiv.org/pdf/2405.02288
• Copy Paste: [[2405.02288]] Prospective Role of Foundation Models in Advancing Autonomous Vehicles(https://arxiv.org/abs/2405.02288)
• Keywords: self-supervised, foundation model
• Abstract: With the development of artificial intelligence and breakthroughs in deep learning, large-scale Foundation Models (FMs), such as GPT, CLIP, etc., have achieved remarkable results in many fields including natural language processing and computer vision. The application of FMs in autonomous driving holds considerable promise. For example, they can contribute to enhance scene understanding and reasoning. By pre-training on rich linguistic and visual data, FMs can understand and interpret various elements in a driving scene, and provide cognitive reasoning to give linguistic and action commands for driving decisions and planning. Furthermore, FMs can augment data based on its understanding of driving scenarios to provide feasible scenes of those rare occurrences in the long tail distribution that are unlikely to be encountered during routine driving and data collection. The enhancement can subsequently lead to the improvement in the accuracy and reliability of autonomous driving systems. Another testament to the potential of FMs applications lies in the development of World Models, exemplified by the DREAMER series, which showcase the ability to comprehend physical laws and dynamics. Learning from massive data under the paradigm of self-supervised learning, World Model can generate unseen yet plausible driving environment, facilitating the enhancement in the prediction of road users behavior and the off-line training of driving strategies. In this paper, we synthesize the applications and future trends of FMs in autonomous driving. By utilizing the powerful capabilities of FMs, we strive to tackle the potential issues stemming from the long-tail distribution in autonomous driving, consequently advancing overall safety in this domain.

Title: Neural Additive Image Model: Interpretation through Interpolation

• Authors: Arik Reuter, Anton Thielmann, Benjamin Saefken
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.02295
• Pdf URL: https://arxiv.org/pdf/2405.02295
• Copy Paste: [[2405.02295]] Neural Additive Image Model: Interpretation through Interpolation(https://arxiv.org/abs/2405.02295)
• Keywords: diffusion
• Abstract: Understanding how images influence the world, interpreting which effects their semantics have on various quantities and exploring the reasons behind changes in image-based predictions are highly difficult yet extremely interesting problems. By adopting a holistic modeling approach utilizing Neural Additive Models in combination with Diffusion Autoencoders, we can effectively identify the latent hidden semantics of image effects and achieve full intelligibility of additional tabular effects. Our approach offers a high degree of flexibility, empowering us to comprehensively explore the impact of various image characteristics. We demonstrate that the proposed method can precisely identify complex image effects in an ablation study. To further showcase the practical applicability of our proposed model, we conduct a case study in which we investigate how the distinctive features and attributes captured within host images exert influence on the pricing of Airbnb rentals.

Title: TFCounter:Polishing Gems for Training-Free Object Counting

• Authors: Pan Ting, Jianfeng Lin, Wenhao Yu, Wenlong Zhang, Xiaoying Chen, Jinlu Zhang, Binqiang Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.02301
• Pdf URL: https://arxiv.org/pdf/2405.02301
• Copy Paste: [[2405.02301]] TFCounter:Polishing Gems for Training-Free Object Counting(https://arxiv.org/abs/2405.02301)
• Keywords: foundation model
• Abstract: Object counting is a challenging task with broad application prospects in security surveillance, traffic management, and disease diagnosis. Existing object counting methods face a tri-fold challenge: achieving superior performance, maintaining high generalizability, and minimizing annotation costs. We develop a novel training-free class-agnostic object counter, TFCounter, which is prompt-context-aware via the cascade of the essential elements in large-scale foundation models. This approach employs an iterative counting framework with a dual prompt system to recognize a broader spectrum of objects varying in shape, appearance, and size. Besides, it introduces an innovative context-aware similarity module incorporating background context to enhance accuracy within messy scenes. To demonstrate cross-domain generalizability, we collect a novel counting dataset named BIKE-1000, including exclusive 1000 images of shared bicycles from Meituan. Extensive experiments on FSC-147, CARPK, and BIKE-1000 datasets demonstrate that TFCounter outperforms existing leading training-free methods and exhibits competitive results compared to trained counterparts.

Title: Efficient Exploration of Image Classifier Failures with Bayesian Optimization and Text-to-Image Models

• Authors: Adrien Le Coz, Houssem Ouertatani, Stéphane Herbin, Faouzi Adjed
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.02332
• Pdf URL: https://arxiv.org/pdf/2405.02332
• Copy Paste: [[2405.02332]] Efficient Exploration of Image Classifier Failures with Bayesian Optimization and Text-to-Image Models(https://arxiv.org/abs/2405.02332)
• Keywords: generative
• Abstract: Image classifiers should be used with caution in the real world. Performance evaluated on a validation set may not reflect performance in the real world. In particular, classifiers may perform well for conditions that are frequently encountered during training, but poorly for other infrequent conditions. In this study, we hypothesize that recent advances in text-to-image generative models make them valuable for benchmarking computer vision models such as image classifiers: they can generate images conditioned by textual prompts that cause classifier failures, allowing failure conditions to be described with textual attributes. However, their generation cost becomes an issue when a large number of synthetic images need to be generated, which is the case when many different attribute combinations need to be tested. We propose an image classifier benchmarking method as an iterative process that alternates image generation, classifier evaluation, and attribute selection. This method efficiently explores the attributes that ultimately lead to poor behavior detection.

Title: COPAL: Continual Pruning in Large Language Generative Models

• Authors: Srikanth Malla, Joon Hee Choi, Chiho Choi
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2405.02347
• Pdf URL: https://arxiv.org/pdf/2405.02347
• Copy Paste: [[2405.02347]] COPAL: Continual Pruning in Large Language Generative Models(https://arxiv.org/abs/2405.02347)
• Keywords: generative
• Abstract: Adapting pre-trained large language models to different domains in natural language processing requires two key considerations: high computational demands and model's inability to continual adaptation. To simultaneously address both issues, this paper presents COPAL (COntinual Pruning in Adaptive Language settings), an algorithm developed for pruning large language generative models under a continual model adaptation setting. While avoiding resource-heavy finetuning or retraining, our pruning process is guided by the proposed sensitivity analysis. The sensitivity effectively measures model's ability to withstand perturbations introduced by the new dataset and finds model's weights that are relevant for all encountered datasets. As a result, COPAL allows seamless model adaptation to new domains while enhancing the resource efficiency. Our empirical evaluation on a various size of LLMs show that COPAL outperforms baseline models, demonstrating its efficacy in efficiency and adaptability.

Title: A Survey of Time Series Foundation Models: Generalizing Time Series Representation with Large Language Mode

• Authors: Jiexia Ye, Weiqi Zhang, Ke Yi, Yongzi Yu, Ziyue Li, Jia Li, Fugee Tsung
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.02358
• Pdf URL: https://arxiv.org/pdf/2405.02358
• Copy Paste: [[2405.02358]] A Survey of Time Series Foundation Models: Generalizing Time Series Representation with Large Language Mode(https://arxiv.org/abs/2405.02358)
• Keywords: foundation model
• Abstract: Time series data are ubiquitous across various domains, making time series analysis critically important. Traditional time series models are task-specific, featuring singular functionality and limited generalization capacity. Recently, large language foundation models have unveiled their remarkable capabilities for cross-task transferability, zero-shot/few-shot learning, and decision-making explainability. This success has sparked interest in the exploration of foundation models to solve multiple time series challenges simultaneously. There are two main research lines, namely \textbf{pre-training foundation models from scratch for time series} and \textbf{adapting large language foundation models for time series}. They both contribute to the development of a unified model that is highly generalizable, versatile, and comprehensible for time series analysis. This survey offers a 3E analytical framework for comprehensive examination of related research. Specifically, we examine existing works from three dimensions, namely \textbf{Effectiveness}, \textbf{Efficiency} and \textbf{Explainability}. In each dimension, we focus on discussing how related works devise tailored solution by considering unique challenges in the realm of time series.Furthermore, we provide a domain taxonomy to help followers keep up with the domain-specific advancements. In addition, we introduce extensive resources to facilitate the field's development, including datasets, open-source, time series libraries. A GitHub repository is also maintained for resource updates (https://github.com/start2020/Awesome-TimeSeries-LLM-FM).

Title: CVTGAD: Simplified Transformer with Cross-View Attention for Unsupervised Graph-level Anomaly Detection

• Authors: Jindong Li, Qianli Xing, Qi Wang, Yi Chang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.02359
• Pdf URL: https://arxiv.org/pdf/2405.02359
• Copy Paste: [[2405.02359]] CVTGAD: Simplified Transformer with Cross-View Attention for Unsupervised Graph-level Anomaly Detection(https://arxiv.org/abs/2405.02359)
• Keywords: anomaly
• Abstract: Unsupervised graph-level anomaly detection (UGAD) has received remarkable performance in various critical disciplines, such as chemistry analysis and bioinformatics. Existing UGAD paradigms often adopt data augmentation techniques to construct multiple views, and then employ different strategies to obtain representations from different views for jointly conducting UGAD. However, most previous works only considered the relationship between nodes/graphs from a limited receptive field, resulting in some key structure patterns and feature information being neglected. In addition, most existing methods consider different views separately in a parallel manner, which is not able to explore the inter-relationship across different views directly. Thus, a method with a larger receptive field that can explore the inter-relationship across different views directly is in need. In this paper, we propose a novel Simplified Transformer with Cross-View Attention for Unsupervised Graph-level Anomaly Detection, namely, CVTGAD. To increase the receptive field, we construct a simplified transformer-based module, exploiting the relationship between nodes/graphs from both intra-graph and inter-graph perspectives. Furthermore, we design a cross-view attention mechanism to directly exploit the view co-occurrence between different views, bridging the inter-view gap at node level and graph level. To the best of our knowledge, this is the first work to apply transformer and cross attention to UGAD, which realizes graph neural network and transformer working collaboratively. Extensive experiments on 15 real-world datasets of 3 fields demonstrate the superiority of CVTGAD on the UGAD task. The code is available at \url{https://github.com/jindongli-Ai/CVTGAD}.

Title: Continuous Learned Primal Dual

• Authors: Christina Runkel, Ander Biguri, Carola-Bibiane Schönlieb
• Subjects: cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2405.02478
• Pdf URL: https://arxiv.org/pdf/2405.02478
• Copy Paste: [[2405.02478]] Continuous Learned Primal Dual(https://arxiv.org/abs/2405.02478)
• Keywords: diffusion
• Abstract: Neural ordinary differential equations (Neural ODEs) propose the idea that a sequence of layers in a neural network is just a discretisation of an ODE, and thus can instead be directly modelled by a parameterised ODE. This idea has had resounding success in the deep learning literature, with direct or indirect influence in many state of the art ideas, such as diffusion models or time dependant models. Recently, a continuous version of the U-net architecture has been proposed, showing increased performance over its discrete counterpart in many imaging applications and wrapped with theoretical guarantees around its performance and robustness. In this work, we explore the use of Neural ODEs for learned inverse problems, in particular with the well-known Learned Primal Dual algorithm, and apply it to computed tomography (CT) reconstruction.

Title: Beyond Helpfulness and Harmlessness: Eliciting Diverse Behaviors from Large Language Models with Persona In-Context Learning

• Authors: Hyeong Kyu Choi, Yixuan Li
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.02501
• Pdf URL: https://arxiv.org/pdf/2405.02501
• Copy Paste: [[2405.02501]] Beyond Helpfulness and Harmlessness: Eliciting Diverse Behaviors from Large Language Models with Persona In-Context Learning(https://arxiv.org/abs/2405.02501)
• Keywords: in-context
• Abstract: Large Language Models (LLMs) are trained on massive text corpora, which are encoded with diverse personality traits. This triggers an interesting goal of eliciting a desired personality trait from the LLM, and probing its behavioral preferences. Accordingly, we formalize the persona elicitation task, aiming to customize LLM behaviors to align with a target persona. We present Persona In-Context Learning (PICLe), a novel persona elicitation framework grounded in Bayesian inference. At the core, PICLe introduces a new ICL example selection criterion based on likelihood ratio, which is designed to optimally guide the model in eliciting a specific target persona. We demonstrate the effectiveness of PICLe through extensive comparisons against baseline methods across three contemporary LLMs. Code is available at https://github.com/deeplearning-wisc/picle.

Title: Spatio-Temporal SwinMAE: A Swin Transformer based Multiscale Representation Learner for Temporal Satellite Imagery

• Authors: Yohei Nakayama, Jiawei Su
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.02512
• Pdf URL: https://arxiv.org/pdf/2405.02512
• Copy Paste: [[2405.02512]] Spatio-Temporal SwinMAE: A Swin Transformer based Multiscale Representation Learner for Temporal Satellite Imagery(https://arxiv.org/abs/2405.02512)
• Keywords: foundation model
• Abstract: Currently, the foundation models represented by large language models have made dramatic progress and are used in a very wide range of domains including 2D and 3D vision. As one of the important application domains of foundation models, earth observation has attracted attention and various approaches have been developed. When considering earth observation as a single image capture, earth observation imagery can be processed as an image with three or more channels, and when it comes with multiple image captures of different timestamps at one location, the temporal observation can be considered as a set of continuous image resembling video frames or medical SCAN slices. This paper presents Spatio-Temporal SwinMAE (ST-SwinMAE), an architecture which particularly focuses on representation learning for spatio-temporal image processing. Specifically, it uses a hierarchical Masked Auto-encoder (MAE) with Video Swin Transformer blocks. With the architecture, we present a pretrained model named Degas 100M as a geospatial foundation model. Also, we propose an approach for transfer learning with Degas 100M, which both pretrained encoder and decoder of MAE are utilized with skip connections added between them to achieve multi-scale information communication, forms an architecture named Spatio-Temporal SwinUNet (ST-SwinUNet). Our approach shows significant improvements of performance over existing state-of-the-art of foundation models. Specifically, for transfer learning of the land cover downstream task on the PhilEO Bench dataset, it shows 10.4\% higher accuracy compared with other geospatial foundation models on average.

Title: SR4ZCT: Self-supervised Through-plane Resolution Enhancement for CT Images with Arbitrary Resolution and Overlap

• Authors: Jiayang Shi, Daniel M. Pelt, K. Joost Batenburg
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.02515
• Pdf URL: https://arxiv.org/pdf/2405.02515
• Copy Paste: [[2405.02515]] SR4ZCT: Self-supervised Through-plane Resolution Enhancement for CT Images with Arbitrary Resolution and Overlap(https://arxiv.org/abs/2405.02515)
• Keywords: self-supervised
• Abstract: Computed tomography (CT) is a widely used non-invasive medical imaging technique for disease diagnosis. The diagnostic accuracy is often affected by image resolution, which can be insufficient in practice. For medical CT images, the through-plane resolution is often worse than the in-plane resolution and there can be overlap between slices, causing difficulties in diagnoses. Self-supervised methods for through-plane resolution enhancement, which train on in-plane images and infer on through-plane images, have shown promise for both CT and MRI imaging. However, existing self-supervised methods either neglect overlap or can only handle specific cases with fixed combinations of resolution and overlap. To address these limitations, we propose a self-supervised method called SR4ZCT. It employs the same off-axis training approach while being capable of handling arbitrary combinations of resolution and overlap. Our method explicitly models the relationship between resolutions and voxel spacings of different planes to accurately simulate training images that match the original through-plane images. We highlight the significance of accurate modeling in self-supervised off-axis training and demonstrate the effectiveness of SR4ZCT using a real-world dataset.

Title: A Literature Review and Framework for Human Evaluation of Generative Large Language Models in Healthcare

• Authors: Thomas Yu Chow Tam, Sonish Sivarajkumar, Sumit Kapoor, Alisa V Stolyar, Katelyn Polanska, Karleigh R McCarthy, Hunter Osterhoudt, Xizhi Wu, Shyam Visweswaran, Sunyang Fu, Piyush Mathur, Giovanni E. Cacciamani, Cong Sun, Yifan Peng, Yanshan Wang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.02559
• Pdf URL: https://arxiv.org/pdf/2405.02559
• Copy Paste: [[2405.02559]] A Literature Review and Framework for Human Evaluation of Generative Large Language Models in Healthcare(https://arxiv.org/abs/2405.02559)
• Keywords: generative
• Abstract: As generative artificial intelligence (AI), particularly Large Language Models (LLMs), continues to permeate healthcare, it remains crucial to supplement traditional automated evaluations with human expert evaluation. Understanding and evaluating the generated texts is vital for ensuring safety, reliability, and effectiveness. However, the cumbersome, time-consuming, and non-standardized nature of human evaluation presents significant obstacles to the widespread adoption of LLMs in practice. This study reviews existing literature on human evaluation methodologies for LLMs within healthcare. We highlight a notable need for a standardized and consistent human evaluation approach. Our extensive literature search, adhering to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, spans publications from January 2018 to February 2024. This review provides a comprehensive overview of the human evaluation approaches used in diverse healthcare applications.This analysis examines the human evaluation of LLMs across various medical specialties, addressing factors such as evaluation dimensions, sample types, and sizes, the selection and recruitment of evaluators, frameworks and metrics, the evaluation process, and statistical analysis of the results. Drawing from diverse evaluation strategies highlighted in these studies, we propose a comprehensive and practical framework for human evaluation of generative LLMs, named QUEST: Quality of Information, Understanding and Reasoning, Expression Style and Persona, Safety and Harm, and Trust and Confidence. This framework aims to improve the reliability, generalizability, and applicability of human evaluation of generative LLMs in different healthcare applications by defining clear evaluation dimensions and offering detailed guidelines.

Title: A Data Mining-Based Dynamical Anomaly Detection Method for Integrating with an Advance Metering System

• Authors: Sarit Maitra
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2405.02574
• Pdf URL: https://arxiv.org/pdf/2405.02574
• Copy Paste: [[2405.02574]] A Data Mining-Based Dynamical Anomaly Detection Method for Integrating with an Advance Metering System(https://arxiv.org/abs/2405.02574)
• Keywords: anomaly
• Abstract: Building operations consume 30% of total power consumption and contribute 26% of global power-related emissions. Therefore, monitoring, and early detection of anomalies at the meter level are essential for residential and commercial buildings. This work investigates both supervised and unsupervised approaches and introduces a dynamic anomaly detection system. The system introduces a supervised Light Gradient Boosting machine and an unsupervised autoencoder with a dynamic threshold. This system is designed to provide real-time detection of anomalies at the meter level. The proposed dynamical system comes with a dynamic threshold based on the Mahalanobis distance and moving averages. This approach allows the system to adapt to changes in the data distribution over time. The effectiveness of the proposed system is evaluated using real-life power consumption data collected from smart metering systems. This empirical testing ensures that the system's performance is validated under real-world conditions. By detecting unusual data movements and providing early warnings, the proposed system contributes significantly to visual analytics and decision science. Early detection of anomalies enables timely troubleshooting, preventing financial losses and potential disasters such as fire incidents.

Title: Generalizing CLIP to Unseen Domain via Text-Guided Diverse Novel Feature Synthesis

• Authors: Siyuan Yan, Cheng Luo, Zhen Yu, Zongyuan Ge
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.02586
• Pdf URL: https://arxiv.org/pdf/2405.02586
• Copy Paste: [[2405.02586]] Generalizing CLIP to Unseen Domain via Text-Guided Diverse Novel Feature Synthesis(https://arxiv.org/abs/2405.02586)
• Keywords: foundation model
• Abstract: Vision-language foundation models like CLIP have shown impressive zero-shot generalization, but finetuning on downstream datasets can cause overfitting and loss of its generalization ability on unseen domains. Although collecting additional data from new domains of interest is possible, this method is often impractical due to the challenges in obtaining annotated data. To address this, we propose a plug-and-play feature augmentation method called LDFS (Language-Guided Diverse Feature Synthesis) to synthesize new domain features and improve existing CLIP fine-tuning strategies. LDFS has three main contributions: 1) To synthesize novel domain features and promote diversity, we propose an instance-conditional feature augmentation strategy based on a textguided feature augmentation loss. 2) To maintain feature quality after augmenting, we introduce a pairwise regularizer to preserve augmented feature coherence within the CLIP feature space. 3) We propose to use stochastic text feature augmentation to reduce the modality gap and further facilitate the process of text-guided feature synthesis. Extensive experiments show LDFS superiority in improving CLIP generalization ability on unseen domains without collecting data from those domains. The code will be made publicly available.

Title: UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model

• Authors: Shuai Yuan, Lei Luo, Zhuo Hui, Can Pu, Xiaoyu Xiang, Rakesh Ranjan, Denis Demandolx
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2405.02608
• Pdf URL: https://arxiv.org/pdf/2405.02608
• Copy Paste: [[2405.02608]] UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model(https://arxiv.org/abs/2405.02608)
• Keywords: self-supervised, foundation model
• Abstract: Traditional unsupervised optical flow methods are vulnerable to occlusions and motion boundaries due to lack of object-level information. Therefore, we propose UnSAMFlow, an unsupervised flow network that also leverages object information from the latest foundation model Segment Anything Model (SAM). We first include a self-supervised semantic augmentation module tailored to SAM masks. We also analyze the poor gradient landscapes of traditional smoothness losses and propose a new smoothness definition based on homography instead. A simple yet effective mask feature module has also been added to further aggregate features on the object level. With all these adaptations, our method produces clear optical flow estimation with sharp boundaries around objects, which outperforms state-of-the-art methods on both KITTI and Sintel datasets. Our method also generalizes well across domains and runs very efficiently.

Title: Contrastive Dual-Interaction Graph Neural Network for Molecular Property Prediction

• Authors: Zexing Zhao, Guangsi Shi, Xiaopeng Wu, Ruohua Ren, Xiaojun Gao, Fuyi Li
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.02628
• Pdf URL: https://arxiv.org/pdf/2405.02628
• Copy Paste: [[2405.02628]] Contrastive Dual-Interaction Graph Neural Network for Molecular Property Prediction(https://arxiv.org/abs/2405.02628)
• Keywords: self-supervised
• Abstract: Molecular property prediction is a key component of AI-driven drug discovery and molecular characterization learning. Despite recent advances, existing methods still face challenges such as limited ability to generalize, and inadequate representation of learning from unlabeled data, especially for tasks specific to molecular structures. To address these limitations, we introduce DIG-Mol, a novel self-supervised graph neural network framework for molecular property prediction. This architecture leverages the power of contrast learning with dual interaction mechanisms and unique molecular graph enhancement strategies. DIG-Mol integrates a momentum distillation network with two interconnected networks to efficiently improve molecular characterization. The framework's ability to extract key information about molecular structure and higher-order semantics is supported by minimizing loss of contrast. We have established DIG-Mol's state-of-the-art performance through extensive experimental evaluation in a variety of molecular property prediction tasks. In addition to demonstrating superior transferability in a small number of learning scenarios, our visualizations highlight DIG-Mol's enhanced interpretability and representation capabilities. These findings confirm the effectiveness of our approach in overcoming challenges faced by traditional methods and mark a significant advance in molecular property prediction.

Title: Generic Multi-modal Representation Learning for Network Traffic Analysis

• Authors: Luca Gioacchini, Idilio Drago, Marco Mellia, Zied Ben Houidi, Dario Rossi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.02649
• Pdf URL: https://arxiv.org/pdf/2405.02649
• Copy Paste: [[2405.02649]] Generic Multi-modal Representation Learning for Network Traffic Analysis(https://arxiv.org/abs/2405.02649)
• Keywords: anomaly
• Abstract: Network traffic analysis is fundamental for network management, troubleshooting, and security. Tasks such as traffic classification, anomaly detection, and novelty discovery are fundamental for extracting operational information from network data and measurements. We witness the shift from deep packet inspection and basic machine learning to Deep Learning (DL) approaches where researchers define and test a custom DL architecture designed for each specific problem. We here advocate the need for a general DL architecture flexible enough to solve different traffic analysis tasks. We test this idea by proposing a DL architecture based on generic data adaptation modules, followed by an integration module that summarises the extracted information into a compact and rich intermediate representation (i.e. embeddings). The result is a flexible Multi-modal Autoencoder (MAE) pipeline that can solve different use cases. We demonstrate the architecture with traffic classification (TC) tasks since they allow us to quantitatively compare results with state-of-the-art solutions. However, we argue that the MAE architecture is generic and can be used to learn representations useful in multiple scenarios. On TC, the MAE performs on par or better than alternatives while avoiding cumbersome feature engineering, thus streamlining the adoption of DL solutions for traffic analysis.

Title: From Generalization Analysis to Optimization Designs for State Space Models

• Authors: Fusheng Liu, Qianxiao Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2405.02670
• Pdf URL: https://arxiv.org/pdf/2405.02670
• Copy Paste: [[2405.02670]] From Generalization Analysis to Optimization Designs for State Space Models(https://arxiv.org/abs/2405.02670)
• Keywords: foundation model
• Abstract: A State Space Model (SSM) is a foundation model in time series analysis, which has recently been shown as an alternative to transformers in sequence modeling. In this paper, we theoretically study the generalization of SSMs and propose improvements to training algorithms based on the generalization results. Specifically, we give a \textit{data-dependent} generalization bound for SSMs, showing an interplay between the SSM parameters and the temporal dependencies of the training sequences. Leveraging the generalization bound, we (1) set up a scaling rule for model initialization based on the proposed generalization measure, which significantly improves the robustness of the output value scales on SSMs to different temporal patterns in the sequence data; (2) introduce a new regularization method for training SSMs to enhance the generalization performance. Numerical results are conducted to validate our results.

Title: Position Paper: Quo Vadis, Unsupervised Time Series Anomaly Detection?

• Authors: M. Saquib Sarfraz, Mei-Yen Chen, Lukas Layer, Kunyu Peng, Marios Koulakis
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2405.02678
• Pdf URL: https://arxiv.org/pdf/2405.02678
• Copy Paste: [[2405.02678]] Position Paper: Quo Vadis, Unsupervised Time Series Anomaly Detection?(https://arxiv.org/abs/2405.02678)
• Keywords: anomaly
• Abstract: The current state of machine learning scholarship in Timeseries Anomaly Detection (TAD) is plagued by the persistent use of flawed evaluation metrics, inconsistent benchmarking practices, and a lack of proper justification for the choices made in novel deep learning-based model designs. Our paper presents a critical analysis of the status quo in TAD, revealing the misleading track of current research and highlighting problematic methods, and evaluation practices. Our position advocates for a shift in focus from pursuing only the novelty in model design to improving benchmarking practices, creating non-trivial datasets, and placing renewed emphasis on studying the utility of model architectures for specific tasks. Our findings demonstrate the need for rigorous evaluation protocols, the creation of simple baselines, and the revelation that state-of-the-art deep anomaly detection models effectively learn linear mappings. These findings suggest the need for more exploration and development of simple and interpretable TAD methods. The increment of model complexity in the state-of-the-art deep-learning based models unfortunately offers very little improvement. We offer insights and suggestions for the field to move forward.

Title: DiffuseTrace: A Transparent and Flexible Watermarking Scheme for Latent Diffusion Model

• Authors: Liangqi Lei, Keke Gai, Jing Yu, Liehuang Zhu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2405.02696
• Pdf URL: https://arxiv.org/pdf/2405.02696
• Copy Paste: [[2405.02696]] DiffuseTrace: A Transparent and Flexible Watermarking Scheme for Latent Diffusion Model(https://arxiv.org/abs/2405.02696)
• Keywords: diffusion, generative
• Abstract: Latent Diffusion Models (LDMs) enable a wide range of applications but raise ethical concerns regarding illegal utilization.Adding watermarks to generative model outputs is a vital technique employed for copyright tracking and mitigating potential risks associated with AI-generated content. However, post-hoc watermarking techniques are susceptible to evasion. Existing watermarking methods for LDMs can only embed fixed messages. Watermark message alteration requires model retraining. The stability of the watermark is influenced by model updates and iterations. Furthermore, the current reconstruction-based watermark removal techniques utilizing variational autoencoders (VAE) and diffusion models have the capability to remove a significant portion of watermarks. Therefore, we propose a novel technique called DiffuseTrace. The goal is to embed invisible watermarks in all generated images for future detection semantically. The method establishes a unified representation of the initial latent variables and the watermark information through training an encoder-decoder model. The watermark information is embedded into the initial latent variables through the encoder and integrated into the sampling process. The watermark information is extracted by reversing the diffusion process and utilizing the decoder. DiffuseTrace does not rely on fine-tuning of the diffusion model components. The watermark is embedded into the image space semantically without compromising image quality. The encoder-decoder can be utilized as a plug-in in arbitrary diffusion models. We validate through experiments the effectiveness and flexibility of DiffuseTrace. DiffuseTrace holds an unprecedented advantage in combating the latest attacks based on variational autoencoders and Diffusion Models.

Title: Stable Diffusion Dataset Generation for Downstream Classification Tasks

• Authors: Eugenio Lomurno, Matteo D'Oria, Matteo Matteucci
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2405.02698
• Pdf URL: https://arxiv.org/pdf/2405.02698
• Copy Paste: [[2405.02698]] Stable Diffusion Dataset Generation for Downstream Classification Tasks(https://arxiv.org/abs/2405.02698)
• Keywords: diffusion, generative
• Abstract: Recent advances in generative artificial intelligence have enabled the creation of high-quality synthetic data that closely mimics real-world data. This paper explores the adaptation of the Stable Diffusion 2.0 model for generating synthetic datasets, using Transfer Learning, Fine-Tuning and generation parameter optimisation techniques to improve the utility of the dataset for downstream classification tasks. We present a class-conditional version of the model that exploits a Class-Encoder and optimisation of key generation parameters. Our methodology led to synthetic datasets that, in a third of cases, produced models that outperformed those trained on real datasets.

Title: Towards a Scalable Identification of Novel Modes in Generative Models

• Authors: Jingwei Zhang, Mohammad Jalali, Cheuk Ting Li, Farzan Farnia
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2405.02700
• Pdf URL: https://arxiv.org/pdf/2405.02700
• Copy Paste: [[2405.02700]] Towards a Scalable Identification of Novel Modes in Generative Models(https://arxiv.org/abs/2405.02700)
• Keywords: generative
• Abstract: An interpretable comparison of generative models requires the identification of sample types produced more frequently by each of the involved models. While several quantitative scores have been proposed in the literature to rank different generative models, such score-based evaluations do not reveal the nuanced differences between the generative models in capturing various sample types. In this work, we propose a method called Fourier-based Identification of Novel Clusters (FINC) to identify modes produced by a generative model with a higher frequency in comparison to a reference distribution. FINC provides a scalable stochastic algorithm based on random Fourier features to estimate the eigenspace of kernel covariance matrices of two generative models and utilize the principal eigendirections to detect the sample types present more dominantly in each model. We demonstrate the application of the FINC method to standard computer vision datasets and generative model frameworks. Our numerical results suggest the scalability and efficiency of the developed Fourier-based method in highlighting the sample types captured with different frequencies by widely-used generative models.

Title: Enhancing News Summarization with ELearnFit through Efficient In-Context Learning and Efficient Fine-Tuning

• Authors: Che Guan, Andrew Chin, Puya Vahabi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.02710
• Pdf URL: https://arxiv.org/pdf/2405.02710
• Copy Paste: [[2405.02710]] Enhancing News Summarization with ELearnFit through Efficient In-Context Learning and Efficient Fine-Tuning(https://arxiv.org/abs/2405.02710)
• Keywords: generative, in-context
• Abstract: With the deluge of information delivered by the daily news cycle, there is a growing need to effectively and efficiently summarize news feeds for quick consumption. We leverage large language models (LLMs), with their advanced learning and generative abilities as compared to conventional language models, to generate concise and coherent summaries for news articles from the XSum dataset. Our paper focuses on two key aspects of LLMs: Efficient in-context Learning (ELearn) and Parameter Efficient Fine-tuning (EFit). Under ELearn, we find that increasing the number of shots in prompts and utilizing simple templates generally improve the quality of summaries. We also find that utilizing relevant examples in few-shot learning for ELearn does not improve model performance. In addition, we studied EFit using different methods and demonstrate that fine-tuning the first layer of LLMs produces better outcomes as compared to fine-tuning other layers or utilizing LoRA. We also find that leveraging more relevant training samples using selective layers does not result in better performance. By combining ELearn and EFit, we create a new model (ELearnFit) that leverages the benefits of both few-shot learning and fine-tuning and produces superior performance to either model alone. We also use ELearnFit to highlight the trade-offs between prompting and fine-tuning, especially for situations where only a limited number of annotated samples are available. Ultimately, our research provides practical techniques to optimize news summarization during the prompting and fine-tuning stages and enhances the synthesis of news articles.

Title: CoE-SQL: In-Context Learning for Multi-Turn Text-to-SQL with Chain-of-Editions

• Authors: Hanchong Zhang, Ruisheng Cao, Hongshen Xu, Lu Chen, Kai Yu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.02712
• Pdf URL: https://arxiv.org/pdf/2405.02712
• Copy Paste: [[2405.02712]] CoE-SQL: In-Context Learning for Multi-Turn Text-to-SQL with Chain-of-Editions(https://arxiv.org/abs/2405.02712)
• Keywords: in-context
• Abstract: Recently, Large Language Models (LLMs) have been demonstrated to possess impressive capabilities in a variety of domains and tasks. We investigate the issue of prompt design in the multi-turn text-to-SQL task and attempt to enhance the LLMs' reasoning capacity when generating SQL queries. In the conversational context, the current SQL query can be modified from the preceding SQL query with only a few operations due to the context dependency. We introduce our method called CoE-SQL which can prompt LLMs to generate the SQL query based on the previously generated SQL query with an edition chain. We also conduct extensive ablation studies to determine the optimal configuration of our approach. Our approach outperforms different in-context learning baselines stably and achieves state-of-the-art performances on two benchmarks SParC and CoSQL using LLMs, which is also competitive to the SOTA fine-tuned models.

Title: U-DiTs: Downsample Tokens in U-Shaped Diffusion Transformers

• Authors: Yuchuan Tian, Zhijun Tu, Hanting Chen, Jie Hu, Chao Xu, Yunhe Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.02730
• Pdf URL: https://arxiv.org/pdf/2405.02730
• Copy Paste: [[2405.02730]] U-DiTs: Downsample Tokens in U-Shaped Diffusion Transformers(https://arxiv.org/abs/2405.02730)
• Keywords: diffusion
• Abstract: Diffusion Transformers (DiTs) introduce the transformer architecture to diffusion tasks for latent-space image generation. With an isotropic architecture that chains a series of transformer blocks, DiTs demonstrate competitive performance and good scalability; but meanwhile, the abandonment of U-Net by DiTs and their following improvements is worth rethinking. To this end, we conduct a simple toy experiment by comparing a U-Net architectured DiT with an isotropic one. It turns out that the U-Net architecture only gain a slight advantage amid the U-Net inductive bias, indicating potential redundancies within the U-Net-style DiT. Inspired by the discovery that U-Net backbone features are low-frequency-dominated, we perform token downsampling on the query-key-value tuple for self-attention and bring further improvements despite a considerable amount of reduction in computation. Based on self-attention with downsampled tokens, we propose a series of U-shaped DiTs (U-DiTs) in the paper and conduct extensive experiments to demonstrate the extraordinary performance of U-DiT models. The proposed U-DiT could outperform DiT-XL/2 with only 1/6 of its computation cost. Codes are available at https://github.com/YuchuanTian/U-DiT.

Title: Systematic Review: Anomaly Detection in Connected and Autonomous Vehicles

• Authors: J. R. V. Solaas, N. Tuptuk, E. Mariconti
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2405.02731
• Pdf URL: https://arxiv.org/pdf/2405.02731
• Copy Paste: [[2405.02731]] Systematic Review: Anomaly Detection in Connected and Autonomous Vehicles(https://arxiv.org/abs/2405.02731)
• Keywords: anomaly
• Abstract: This systematic review focuses on anomaly detection for connected and autonomous vehicles. The initial database search identified 2160 articles, of which 203 were included in this review after rigorous screening and assessment. This study revealed that the most commonly used Artificial Intelligence (AI) algorithms employed in anomaly detection are neural networks like LSTM, CNN, and autoencoders, alongside one-class SVM. Most anomaly-based models were trained using real-world operational vehicle data, although anomalies, such as attacks and faults, were often injected artificially into the datasets. These models were evaluated mostly using five key evaluation metrics: recall, accuracy, precision, F1-score, and false positive rate. The most frequently used selection of evaluation metrics used for anomaly detection models were accuracy, precision, recall, and F1-score. This systematic review presents several recommendations. First, there is a need to incorporate multiple evaluation metrics to provide a comprehensive assessment of the anomaly detection models. Second, only a small proportion of the studies have made their models open source, indicating a need to share models publicly to facilitate collaboration within the research community, and to validate and compare findings effectively. Third, there is a need for benchmarking datasets with predefined anomalies or cyberattacks to test and improve the effectiveness of the proposed anomaly-based detection models. Furthermore, there is a need for future research to investigate the deployment of anomaly detection to a vehicle to assess its performance on the road. There is a notable lack of research done on intrusion detection systems using different protocols to CAN, such as Ethernet and FlexRay.

Title: A self-supervised text-vision framework for automated brain abnormality detection

• Authors: David A. Wood, Emily Guilhem, Sina Kafiabadi, Ayisha Al Busaidi, Kishan Dissanayake, Ahmed Hammam, Nina Mansoor, Matthew Townend, Siddharth Agarwal, Yiran Wei, Asif Mazumder, Gareth J. Barker, Peter Sasieni, Sebastien Ourselin, James H. Cole, Thomas C. Booth
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.02782
• Pdf URL: https://arxiv.org/pdf/2405.02782
• Copy Paste: [[2405.02782]] A self-supervised text-vision framework for automated brain abnormality detection(https://arxiv.org/abs/2405.02782)
• Keywords: self-supervised
• Abstract: Artificial neural networks trained on large, expert-labelled datasets are considered state-of-the-art for a range of medical image recognition tasks. However, categorically labelled datasets are time-consuming to generate and constrain classification to a pre-defined, fixed set of classes. For neuroradiological applications in particular, this represents a barrier to clinical adoption. To address these challenges, we present a self-supervised text-vision framework that learns to detect clinically relevant abnormalities in brain MRI scans by directly leveraging the rich information contained in accompanying free-text neuroradiology reports. Our training approach consisted of two-steps. First, a dedicated neuroradiological language model - NeuroBERT - was trained to generate fixed-dimensional vector representations of neuroradiology reports (N = 50,523) via domain-specific self-supervised learning tasks. Next, convolutional neural networks (one per MRI sequence) learnt to map individual brain scans to their corresponding text vector representations by optimising a mean square error loss. Once trained, our text-vision framework can be used to detect abnormalities in unreported brain MRI examinations by scoring scans against suitable query sentences (e.g., 'there is an acute stroke', 'there is hydrocephalus' etc.), enabling a range of classification-based applications including automated triage. Potentially, our framework could also serve as a clinical decision support tool, not only by suggesting findings to radiologists and detecting errors in provisional reports, but also by retrieving and displaying examples of pathologies from historical examinations that could be relevant to the current case based on textual descriptors.

Title: Efficient Text-driven Motion Generation via Latent Consistency Training

• Authors: Mengxian Hu, Minghao Zhu, Xun Zhou, Qingqing Yan, Shu Li, Chengju Liu, Qijun Chen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.02791
• Pdf URL: https://arxiv.org/pdf/2405.02791
• Copy Paste: [[2405.02791]] Efficient Text-driven Motion Generation via Latent Consistency Training(https://arxiv.org/abs/2405.02791)
• Keywords: diffusion
• Abstract: Motion diffusion models have recently proven successful for text-driven human motion generation. Despite their excellent generation performance, they are challenging to infer in real time due to the multi-step sampling mechanism that involves tens or hundreds of repeat function evaluation iterations. To this end, we investigate a motion latent consistency Training (MLCT) for motion generation to alleviate the computation and time consumption during iteration inference. It applies diffusion pipelines to low-dimensional motion latent spaces to mitigate the computational burden of each function evaluation. Explaining the diffusion process with probabilistic flow ordinary differential equation (PF-ODE) theory, the MLCT allows extremely few steps infer between the prior distribution to the motion latent representation distribution via maintaining consistency of the outputs over the trajectory of PF-ODE. Especially, we introduce a quantization constraint to optimize motion latent representations that are bounded, regular, and well-reconstructed compared to traditional variational constraints. Furthermore, we propose a conditional PF-ODE trajectory simulation method, which improves the conditional generation performance with minimal additional training costs. Extensive experiments on two human motion generation benchmarks show that the proposed model achieves state-of-the-art performance with less than 10\% time cost.

Title: Adapting to Distribution Shift by Visual Domain Prompt Generation

• Authors: Zhixiang Chi, Li Gu, Tao Zhong, Huan Liu, Yuanhao Yu, Konstantinos N Plataniotis, Yang Wang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.02797
• Pdf URL: https://arxiv.org/pdf/2405.02797
• Copy Paste: [[2405.02797]] Adapting to Distribution Shift by Visual Domain Prompt Generation(https://arxiv.org/abs/2405.02797)
• Keywords: foundation model
• Abstract: In this paper, we aim to adapt a model at test-time using a few unlabeled data to address distribution shifts. To tackle the challenges of extracting domain knowledge from a limited amount of data, it is crucial to utilize correlated information from pre-trained backbones and source domains. Previous studies fail to utilize recent foundation models with strong out-of-distribution generalization. Additionally, domain-centric designs are not flavored in their works. Furthermore, they employ the process of modelling source domains and the process of learning to adapt independently into disjoint training stages. In this work, we propose an approach on top of the pre-computed features of the foundation model. Specifically, we build a knowledge bank to learn the transferable knowledge from source domains. Conditioned on few-shot target data, we introduce a domain prompt generator to condense the knowledge bank into a domain-specific prompt. The domain prompt then directs the visual features towards a particular domain via a guidance module. Moreover, we propose a domain-aware contrastive loss and employ meta-learning to facilitate domain knowledge extraction. Extensive experiments are conducted to validate the domain knowledge extraction. The proposed method outperforms previous work on 5 large-scale benchmarks including WILDS and DomainNet.

Title: Is Flash Attention Stable?

• Authors: Alicia Golden, Samuel Hsia, Fei Sun, Bilge Acun, Basil Hosmer, Yejin Lee, Zachary DeVito, Jeff Johnson, Gu-Yeon Wei, David Brooks, Carole-Jean Wu
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2405.02803
• Pdf URL: https://arxiv.org/pdf/2405.02803
• Copy Paste: [[2405.02803]] Is Flash Attention Stable?(https://arxiv.org/abs/2405.02803)
• Keywords: generative
• Abstract: Training large-scale machine learning models poses distinct system challenges, given both the size and complexity of today's workloads. Recently, many organizations training state-of-the-art Generative AI models have reported cases of instability during training, often taking the form of loss spikes. Numeric deviation has emerged as a potential cause of this training instability, although quantifying this is especially challenging given the costly nature of training runs. In this work, we develop a principled approach to understanding the effects of numeric deviation, and construct proxies to put observations into context when downstream effects are difficult to quantify. As a case study, we apply this framework to analyze the widely-adopted Flash Attention optimization. We find that Flash Attention sees roughly an order of magnitude more numeric deviation as compared to Baseline Attention at BF16 when measured during an isolated forward pass. We then use a data-driven analysis based on the Wasserstein Distance to provide upper bounds on how this numeric deviation impacts model weights during training, finding that the numerical deviation present in Flash Attention is 2-5 times less significant than low-precision training.

Title: Verlet Flows: Exact-Likelihood Integrators for Flow-Based Generative Models

• Authors: Ezra Erives, Bowen Jing, Tommi Jaakkola
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2405.02805
• Pdf URL: https://arxiv.org/pdf/2405.02805
• Copy Paste: [[2405.02805]] Verlet Flows: Exact-Likelihood Integrators for Flow-Based Generative Models(https://arxiv.org/abs/2405.02805)
• Keywords: generative
• Abstract: Approximations in computing model likelihoods with continuous normalizing flows (CNFs) hinder the use of these models for importance sampling of Boltzmann distributions, where exact likelihoods are required. In this work, we present Verlet flows, a class of CNFs on an augmented state-space inspired by symplectic integrators from Hamiltonian dynamics. When used with carefully constructed Taylor-Verlet integrators, Verlet flows provide exact-likelihood generative models which generalize coupled flow architectures from a non-continuous setting while imposing minimal expressivity constraints. On experiments over toy densities, we demonstrate that the variance of the commonly used Hutchinson trace estimator is unsuitable for importance sampling, whereas Verlet flows perform comparably to full autograd trace computations while being significantly faster.

Title: SMCD: High Realism Motion Style Transfer via Mamba-based Diffusion

• Authors: Ziyun Qian, Zeyu Xiao, Zhenyi Wu, Dingkang Yang, Mingcheng Li, Shunli Wang, Shuaibing Wang, Dongliang Kou, Lihua Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.02844
• Pdf URL: https://arxiv.org/pdf/2405.02844
• Copy Paste: [[2405.02844]] SMCD: High Realism Motion Style Transfer via Mamba-based Diffusion(https://arxiv.org/abs/2405.02844)
• Keywords: diffusion
• Abstract: Motion style transfer is a significant research direction in multimedia applications. It enables the rapid switching of different styles of the same motion for virtual digital humans, thus vastly increasing the diversity and realism of movements. It is widely applied in multimedia scenarios such as movies, games, and the Metaverse. However, most of the current work in this field adopts the GAN, which may lead to instability and convergence issues, making the final generated motion sequence somewhat chaotic and unable to reflect a highly realistic and natural style. To address these problems, we consider style motion as a condition and propose the Style Motion Conditioned Diffusion (SMCD) framework for the first time, which can more comprehensively learn the style features of motion. Moreover, we apply Mamba model for the first time in the motion style transfer field, introducing the Motion Style Mamba (MSM) module to handle longer motion sequences. Thirdly, aiming at the SMCD framework, we propose Diffusion-based Content Consistency Loss and Content Consistency Loss to assist the overall framework's training. Finally, we conduct extensive experiments. The results reveal that our method surpasses state-of-the-art methods in both qualitative and quantitative comparisons, capable of generating more realistic motion sequences.

Title: MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior

• Authors: Honghua Chen, Chen Change Loy, Xingang Pan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.02859
• Pdf URL: https://arxiv.org/pdf/2405.02859
• Copy Paste: [[2405.02859]] MVIP-NeRF: Multi-view 3D Inpainting on NeRF Scenes via Diffusion Prior(https://arxiv.org/abs/2405.02859)
• Keywords: diffusion, generative
• Abstract: Despite the emergence of successful NeRF inpainting methods built upon explicit RGB and depth 2D inpainting supervisions, these methods are inherently constrained by the capabilities of their underlying 2D inpainters. This is due to two key reasons: (i) independently inpainting constituent images results in view-inconsistent imagery, and (ii) 2D inpainters struggle to ensure high-quality geometry completion and alignment with inpainted RGB images. To overcome these limitations, we propose a novel approach called MVIP-NeRF that harnesses the potential of diffusion priors for NeRF inpainting, addressing both appearance and geometry aspects. MVIP-NeRF performs joint inpainting across multiple views to reach a consistent solution, which is achieved via an iterative optimization process based on Score Distillation Sampling (SDS). Apart from recovering the rendered RGB images, we also extract normal maps as a geometric representation and define a normal SDS loss that motivates accurate geometry inpainting and alignment with the appearance. Additionally, we formulate a multi-view SDS score function to distill generative priors simultaneously from different view images, ensuring consistent visual completion when dealing with large view variations. Our experimental results show better appearance and geometry recovery than previous NeRF inpainting methods.

Title: Boundary-aware Decoupled Flow Networks for Realistic Extreme Rescaling

• Authors: Jinmin Li, Tao Dai, Jingyun Zhang, Kang Liu, Jun Wang, Shaoming Wang, Shu-Tao Xia, rizen guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.02941
• Pdf URL: https://arxiv.org/pdf/2405.02941
• Copy Paste: [[2405.02941]] Boundary-aware Decoupled Flow Networks for Realistic Extreme Rescaling(https://arxiv.org/abs/2405.02941)
• Keywords: generative
• Abstract: Recently developed generative methods, including invertible rescaling network (IRN) based and generative adversarial network (GAN) based methods, have demonstrated exceptional performance in image rescaling. However, IRN-based methods tend to produce over-smoothed results, while GAN-based methods easily generate fake details, which thus hinders their real applications. To address this issue, we propose Boundary-aware Decoupled Flow Networks (BDFlow) to generate realistic and visually pleasing results. Unlike previous methods that model high-frequency information as standard Gaussian distribution directly, our BDFlow first decouples the high-frequency information into \textit{semantic high-frequency} that adheres to a Boundary distribution and \textit{non-semantic high-frequency} counterpart that adheres to a Gaussian distribution. Specifically, to capture semantic high-frequency parts accurately, we use Boundary-aware Mask (BAM) to constrain the model to produce rich textures, while non-semantic high-frequency part is randomly sampled from a Gaussian distribution.Comprehensive experiments demonstrate that our BDFlow significantly outperforms other state-of-the-art methods while maintaining lower complexity. Notably, our BDFlow improves the PSNR by $4.4$ dB and the SSIM by $0.1$ on average over GRAIN, utilizing only 74\% of the parameters and 20\% of the computation. The code will be available at https://github.com/THU-Kingmin/BAFlow.

Title: Score-based Generative Priors Guided Model-driven Network for MRI Reconstruction

• Authors: Xiaoyu Qiao, Weisheng Li, Yuping Huang, Lijian Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.02958
• Pdf URL: https://arxiv.org/pdf/2405.02958
• Copy Paste: [[2405.02958]] Score-based Generative Priors Guided Model-driven Network for MRI Reconstruction(https://arxiv.org/abs/2405.02958)
• Keywords: generative
• Abstract: Score matching with Langevin dynamics (SMLD) method has been successfully applied to accelerated MRI. However, the hyperparameters in the sampling process require subtle tuning, otherwise the results can be severely corrupted by hallucination artifacts, particularly with out-of-distribution test data. In this study, we propose a novel workflow in which SMLD results are regarded as additional priors to guide model-driven network training. First, we adopted a pretrained score network to obtain samples as preliminary guidance images (PGI) without the need for network retraining, parameter tuning and in-distribution test data. Although PGIs are corrupted by hallucination artifacts, we believe that they can provide extra information through effective denoising steps to facilitate reconstruction. Therefore, we designed a denoising module (DM) in the second step to improve the quality of PGIs. The features are extracted from the components of Langevin dynamics and the same score network with fine-tuning; hence, we can directly learn the artifact patterns. Third, we designed a model-driven network whose training is guided by denoised PGIs (DGIs). DGIs are densely connected with intermediate reconstructions in each cascade to enrich the features and are periodically updated to provide more accurate guidance. Our experiments on different sequences revealed that despite the low average quality of PGIs, the proposed workflow can effectively extract valuable information to guide the network training, even with severely reduced training data and sampling steps. Our method outperforms other cutting-edge techniques by effectively mitigating hallucination artifacts, yielding robust and high-quality reconstruction results.

Title: JOSENet: A Joint Stream Embedding Network for Violence Detection in Surveillance Videos

• Authors: Pietro Nardelli, Danilo Comminiello
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2405.02961
• Pdf URL: https://arxiv.org/pdf/2405.02961
• Copy Paste: [[2405.02961]] JOSENet: A Joint Stream Embedding Network for Violence Detection in Surveillance Videos(https://arxiv.org/abs/2405.02961)
• Keywords: self-supervised
• Abstract: Due to the ever-increasing availability of video surveillance cameras and the growing need for crime prevention, the violence detection task is attracting greater attention from the research community. With respect to other action recognition tasks, violence detection in surveillance videos shows additional issues, such as the presence of a significant variety of real fight scenes. Unfortunately, available datasets seem to be very small compared with other action recognition datasets. Moreover, in surveillance applications, people in the scenes always differ for each video and the background of the footage differs for each camera. Also, violent actions in real-life surveillance videos must be detected quickly to prevent unwanted consequences, thus models would definitely benefit from a reduction in memory usage and computational costs. Such problems make classical action recognition methods difficult to be adopted. To tackle all these issues, we introduce JOSENet, a novel self-supervised framework that provides outstanding performance for violence detection in surveillance videos. The proposed model receives two spatiotemporal video streams, i.e., RGB frames and optical flows, and involves a new regularized self-supervised learning approach for videos. JOSENet provides improved performance compared to self-supervised state-of-the-art methods, while requiring one-fourth of the number of frames per video segment and a reduced frame rate. The source code and the instructions to reproduce our experiments are available at https://github.com/ispamm/JOSENet.

Title: Parameter-Efficient Fine-Tuning with Discrete Fourier Transform

• Authors: Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, Jia Li
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2405.03003
• Pdf URL: https://arxiv.org/pdf/2405.03003
• Copy Paste: [[2405.03003]] Parameter-Efficient Fine-Tuning with Discrete Fourier Transform(https://arxiv.org/abs/2405.03003)
• Keywords: foundation model
• Abstract: Low-rank adaptation~(LoRA) has recently gained much interest in fine-tuning foundation models. It effectively reduces the number of trainable parameters by incorporating low-rank matrices $A$ and $B$ to represent the weight change, i.e., $\Delta W=BA$. Despite LoRA's progress, it faces storage challenges when handling extensive customization adaptations or larger base models. In this work, we aim to further compress trainable parameters by enjoying the powerful expressiveness of the Fourier transform. Specifically, we introduce FourierFT, which treats $\Delta W$ as a matrix in the spatial domain and learns only a small fraction of its spectral coefficients. With the trained spectral coefficients, we implement the inverse discrete Fourier transform to recover $\Delta W$. Empirically, our FourierFT method shows comparable or better performance with fewer parameters than LoRA on various tasks, including natural language understanding, natural language generation, instruction tuning, and image classification. For example, when performing instruction tuning on the LLaMA2-7B model, FourierFT surpasses LoRA with only 0.064M trainable parameters, compared to LoRA's 33.5M. Our code is released at \url{https://github.com/Chaos96/fourierft}.

Title: Matten: Video Generation with Mamba-Attention

• Authors: Yu Gao, Jiancheng Huang, Xiaopeng Sun, Zequn Jie, Yujie Zhong, Lin Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.03025
• Pdf URL: https://arxiv.org/pdf/2405.03025
• Copy Paste: [[2405.03025]] Matten: Video Generation with Mamba-Attention(https://arxiv.org/abs/2405.03025)
• Keywords: diffusion
• Abstract: In this paper, we introduce Matten, a cutting-edge latent diffusion model with Mamba-Attention architecture for video generation. With minimal computational cost, Matten employs spatial-temporal attention for local video content modeling and bidirectional Mamba for global video content modeling. Our comprehensive experimental evaluation demonstrates that Matten has competitive performance with the current Transformer-based and GAN-based models in benchmark performance, achieving superior FVD scores and efficiency. Additionally, we observe a direct positive correlation between the complexity of our designed model and the improvement in video quality, indicating the excellent scalability of Matten.

Title: Convolutional Learning on Directed Acyclic Graphs

• Authors: Samuel Rey, Hamed Ajorlou, Gonzalo Mateos
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2405.03056
• Pdf URL: https://arxiv.org/pdf/2405.03056
• Copy Paste: [[2405.03056]] Convolutional Learning on Directed Acyclic Graphs(https://arxiv.org/abs/2405.03056)
• Keywords: diffusion
• Abstract: We develop a novel convolutional architecture tailored for learning from data defined over directed acyclic graphs (DAGs). DAGs can be used to model causal relationships among variables, but their nilpotent adjacency matrices pose unique challenges towards developing DAG signal processing and machine learning tools. To address this limitation, we harness recent advances offering alternative definitions of causal shifts and convolutions for signals on DAGs. We develop a novel convolutional graph neural network that integrates learnable DAG filters to account for the partial ordering induced by the graph topology, thus providing valuable inductive bias to learn effective representations of DAG-supported data. We discuss the salient advantages and potential limitations of the proposed DAG convolutional network (DCN) and evaluate its performance on two learning tasks using synthetic data: network diffusion estimation and source identification. DCN compares favorably relative to several baselines, showcasing its promising potential.

Title: AnoGAN for Tabular Data: A Novel Approach to Anomaly Detection

• Authors: Aditya Singh, Pavan Reddy
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2405.03075
• Pdf URL: https://arxiv.org/pdf/2405.03075
• Copy Paste: [[2405.03075]] AnoGAN for Tabular Data: A Novel Approach to Anomaly Detection(https://arxiv.org/abs/2405.03075)
• Keywords: generative, anomaly
• Abstract: Anomaly detection, a critical facet in data analysis, involves identifying patterns that deviate from expected behavior. This research addresses the complexities inherent in anomaly detection, exploring challenges and adapting to sophisticated malicious activities. With applications spanning cybersecurity, healthcare, finance, and surveillance, anomalies often signify critical information or potential threats. Inspired by the success of Anomaly Generative Adversarial Network (AnoGAN) in image domains, our research extends its principles to tabular data. Our contributions include adapting AnoGAN's principles to a new domain and promising advancements in detecting previously undetectable anomalies. This paper delves into the multifaceted nature of anomaly detection, considering the dynamic evolution of normal behavior, context-dependent anomaly definitions, and data-related challenges like noise and imbalances.

Title: Intra-task Mutual Attention based Vision Transformer for Few-Shot Learning

• Authors: Weihao Jiang, Chang Liu, Kun He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.03109
• Pdf URL: https://arxiv.org/pdf/2405.03109
• Copy Paste: [[2405.03109]] Intra-task Mutual Attention based Vision Transformer for Few-Shot Learning(https://arxiv.org/abs/2405.03109)
• Keywords: self-supervised
• Abstract: Humans possess remarkable ability to accurately classify new, unseen images after being exposed to only a few examples. Such ability stems from their capacity to identify common features shared between new and previously seen images while disregarding distractions such as background variations. However, for artificial neural network models, determining the most relevant features for distinguishing between two images with limited samples presents a challenge. In this paper, we propose an intra-task mutual attention method for few-shot learning, that involves splitting the support and query samples into patches and encoding them using the pre-trained Vision Transformer (ViT) architecture. Specifically, we swap the class (CLS) token and patch tokens between the support and query sets to have the mutual attention, which enables each set to focus on the most useful information. This facilitates the strengthening of intra-class representations and promotes closer proximity between instances of the same class. For implementation, we adopt the ViT-based network architecture and utilize pre-trained model parameters obtained through self-supervision. By leveraging Masked Image Modeling as a self-supervised training task for pre-training, the pre-trained model yields semantically meaningful representations while successfully avoiding supervision collapse. We then employ a meta-learning method to fine-tune the last several layers and CLS token modules. Our strategy significantly reduces the num- ber of parameters that require fine-tuning while effectively uti- lizing the capability of pre-trained model. Extensive experiments show that our framework is simple, effective and computationally efficient, achieving superior performance as compared to the state-of-the-art baselines on five popular few-shot classification benchmarks under the 5-shot and 1-shot scenarios

Title: AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encoding

• Authors: Tao Liu, Feilong Chen, Shuai Fan, Chenpeng Du, Qi Chen, Xie Chen, Kai Yu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.03121
• Pdf URL: https://arxiv.org/pdf/2405.03121
• Copy Paste: [[2405.03121]] AniTalker: Animate Vivid and Diverse Talking Faces through Identity-Decoupled Facial Motion Encoding(https://arxiv.org/abs/2405.03121)
• Keywords: diffusion, self-supervised
• Abstract: The paper introduces AniTalker, an innovative framework designed to generate lifelike talking faces from a single portrait. Unlike existing models that primarily focus on verbal cues such as lip synchronization and fail to capture the complex dynamics of facial expressions and nonverbal cues, AniTalker employs a universal motion representation. This innovative representation effectively captures a wide range of facial dynamics, including subtle expressions and head movements. AniTalker enhances motion depiction through two self-supervised learning strategies: the first involves reconstructing target video frames from source frames within the same identity to learn subtle motion representations, and the second develops an identity encoder using metric learning while actively minimizing mutual information between the identity and motion encoders. This approach ensures that the motion representation is dynamic and devoid of identity-specific details, significantly reducing the need for labeled data. Additionally, the integration of a diffusion model with a variance adapter allows for the generation of diverse and controllable facial animations. This method not only demonstrates AniTalker's capability to create detailed and realistic facial movements but also underscores its potential in crafting dynamic avatars for real-world applications. Synthetic results can be viewed at https://github.com/X-LANCE/AniTalker.

Title: Video Diffusion Models: A Survey

• Authors: Andrew Melnik, Michal Ljubljanac, Cong Lu, Qi Yan, Weiming Ren, Helge Ritter
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.03150
• Pdf URL: https://arxiv.org/pdf/2405.03150
• Copy Paste: [[2405.03150]] Video Diffusion Models: A Survey(https://arxiv.org/abs/2405.03150)
• Keywords: diffusion, generative
• Abstract: Diffusion generative models have recently become a robust technique for producing and modifying coherent, high-quality video. This survey offers a systematic overview of critical elements of diffusion models for video generation, covering applications, architectural choices, and the modeling of temporal dynamics. Recent advancements in the field are summarized and grouped into development trends. The survey concludes with an overview of remaining challenges and an outlook on the future of the field. Website: https://github.com/ndrwmlnk/Awesome-Video-Diffusion-Models

Title: DeepMpMRI: Tensor-decomposition Regularized Learning for Fast and High-Fidelity Multi-Parametric Microstructural MR Imaging

• Authors: Wenxin Fan, Jian Cheng, Cheng Li, Xinrui Ma, Jing Yang, Juan Zou, Ruoyou Wu, Zan Chen, Yuanjing Feng, Hairong Zheng, Shanshan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.03159
• Pdf URL: https://arxiv.org/pdf/2405.03159
• Copy Paste: [[2405.03159]] DeepMpMRI: Tensor-decomposition Regularized Learning for Fast and High-Fidelity Multi-Parametric Microstructural MR Imaging(https://arxiv.org/abs/2405.03159)
• Keywords: diffusion
• Abstract: Deep learning has emerged as a promising approach for learning the nonlinear mapping between diffusion-weighted MR images and tissue parameters, which enables automatic and deep understanding of the brain microstructures. However, the efficiency and accuracy in the multi-parametric estimations are still limited since previous studies tend to estimate multi-parametric maps with dense sampling and isolated signal modeling. This paper proposes DeepMpMRI, a unified framework for fast and high-fidelity multi-parametric estimation from various diffusion models using sparsely sampled q-space data. DeepMpMRI is equipped with a newly designed tensor-decomposition-based regularizer to effectively capture fine details by exploiting the correlation across parameters. In addition, we introduce a Nesterov-based adaptive learning algorithm that optimizes the regularization parameter dynamically to enhance the performance. DeepMpMRI is an extendable framework capable of incorporating flexible network architecture. Experimental results demonstrate the superiority of our approach over 5 state-of-the-art methods in simultaneously estimating multi-parametric maps for various diffusion models with fine-grained details both quantitatively and qualitatively, achieving 4.5 - 22.5$\times$ acceleration compared to the dense sampling of a total of 270 diffusion gradients.

Title: Oracle-Checker Scheme for Evaluating a Generative Large Language Model

• Authors: Yueling Jenny Zeng, Li-C. Wang, Thomas Ibbetson
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2405.03170
• Pdf URL: https://arxiv.org/pdf/2405.03170
• Copy Paste: [[2405.03170]] Oracle-Checker Scheme for Evaluating a Generative Large Language Model(https://arxiv.org/abs/2405.03170)
• Keywords: generative
• Abstract: This work presents a novel approach called oracle-checker scheme for evaluating the answer given by a generative large language model (LLM). Two types of checkers are presented. The first type of checker follows the idea of property testing. The second type of checker follows the idea of program checking. Their applications are demonstrated in two separate contexts, entity extraction and paraphrase decision, respectively.

Title: Hyperbolic Geometric Latent Diffusion Model for Graph Generation

• Authors: Xingcheng Fu, Yisen Gao, Yuecen Wei, Qingyun Sun, Hao Peng, Jianxin Li, Xianxian Li
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2405.03188
• Pdf URL: https://arxiv.org/pdf/2405.03188
• Copy Paste: [[2405.03188]] Hyperbolic Geometric Latent Diffusion Model for Graph Generation(https://arxiv.org/abs/2405.03188)
• Keywords: diffusion, generative
• Abstract: Diffusion models have made significant contributions to computer vision, sparking a growing interest in the community recently regarding the application of them to graph generation. Existing discrete graph diffusion models exhibit heightened computational complexity and diminished training efficiency. A preferable and natural way is to directly diffuse the graph within the latent space. However, due to the non-Euclidean structure of graphs is not isotropic in the latent space, the existing latent diffusion models effectively make it difficult to capture and preserve the topological information of graphs. To address the above challenges, we propose a novel geometrically latent diffusion framework HypDiff. Specifically, we first establish a geometrically latent space with interpretability measures based on hyperbolic geometry, to define anisotropic latent diffusion processes for graphs. Then, we propose a geometrically latent diffusion process that is constrained by both radial and angular geometric properties, thereby ensuring the preservation of the original topological properties in the generative graphs. Extensive experimental results demonstrate the superior effectiveness of HypDiff for graph generation with various topologies.

Title: Mind the Gap Between Synthetic and Real: Utilizing Transfer Learning to Probe the Boundaries of Stable Diffusion Generated Data

• Authors: Leonhard Hennicke, Christian Medeiros Adriano, Holger Giese, Jan Mathias Koehler, Lukas Schott
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.03243
• Pdf URL: https://arxiv.org/pdf/2405.03243
• Copy Paste: [[2405.03243]] Mind the Gap Between Synthetic and Real: Utilizing Transfer Learning to Probe the Boundaries of Stable Diffusion Generated Data(https://arxiv.org/abs/2405.03243)
• Keywords: diffusion, foundation model, generative
• Abstract: Generative foundation models like Stable Diffusion comprise a diverse spectrum of knowledge in computer vision with the potential for transfer learning, e.g., via generating data to train student models for downstream tasks. This could circumvent the necessity of collecting labeled real-world data, thereby presenting a form of data-free knowledge distillation. However, the resultant student models show a significant drop in accuracy compared to models trained on real data. We investigate possible causes for this drop and focus on the role of the different layers of the student model. By training these layers using either real or synthetic data, we reveal that the drop mainly stems from the model's final layers. Further, we briefly investigate other factors, such as differences in data-normalization between synthetic and real, the impact of data augmentations, texture vs.\ shape learning, and assuming oracle prompts. While we find that some of those factors can have an impact, they are not sufficient to close the gap towards real data. Building upon our insights that mainly later layers are responsible for the drop, we investigate the data-efficiency of fine-tuning a synthetically trained model with real data applied to only those last layers. Our results suggest an improved trade-off between the amount of real training data used and the model's accuracy. Our findings contribute to the understanding of the gap between synthetic and real data and indicate solutions to mitigate the scarcity of labeled real data.

Title: Exploring the Frontiers of Softmax: Provable Optimization, Applications in Diffusion Model, and Beyond

• Authors: Jiuxiang Gu, Chenyang Li, Yingyu Liang, Zhenmei Shi, Zhao Song
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.03251
• Pdf URL: https://arxiv.org/pdf/2405.03251
• Copy Paste: [[2405.03251]] Exploring the Frontiers of Softmax: Provable Optimization, Applications in Diffusion Model, and Beyond(https://arxiv.org/abs/2405.03251)
• Keywords: diffusion, generative
• Abstract: The softmax activation function plays a crucial role in the success of large language models (LLMs), particularly in the self-attention mechanism of the widely adopted Transformer architecture. However, the underlying learning dynamics that contribute to the effectiveness of softmax remain largely unexplored. As a step towards better understanding, this paper provides a theoretical study of the optimization and generalization properties of two-layer softmax neural networks, providing theoretical insights into their superior performance as other activation functions, such as ReLU and exponential. Leveraging the Neural Tangent Kernel (NTK) framework, our analysis reveals that the normalization effect of the softmax function leads to a good perturbation property of the induced NTK matrix, resulting in a good convex region of the loss landscape. Consequently, softmax neural networks can learn the target function in the over-parametrization regime. To demonstrate the broad applicability of our theoretical findings, we apply them to the task of learning score estimation functions in diffusion models, a promising approach for generative modeling. Our analysis shows that gradient-based algorithms can learn the score function with a provable accuracy. Our work provides a deeper understanding of the effectiveness of softmax neural networks and their potential in various domains, paving the way for further advancements in natural language processing and beyond.

Title: Multi-Modality Spatio-Temporal Forecasting via Self-Supervised Learning

• Authors: Jiewen Deng, Renhe Jiang, Jiaqi Zhang, Xuan Song
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2405.03255
• Pdf URL: https://arxiv.org/pdf/2405.03255
• Copy Paste: [[2405.03255]] Multi-Modality Spatio-Temporal Forecasting via Self-Supervised Learning(https://arxiv.org/abs/2405.03255)
• Keywords: self-supervised
• Abstract: Multi-modality spatio-temporal (MoST) data extends spatio-temporal (ST) data by incorporating multiple modalities, which is prevalent in monitoring systems, encompassing diverse traffic demands and air quality assessments. Despite significant strides in ST modeling in recent years, there remains a need to emphasize harnessing the potential of information from different modalities. Robust MoST forecasting is more challenging because it possesses (i) high-dimensional and complex internal structures and (ii) dynamic heterogeneity caused by temporal, spatial, and modality variations. In this study, we propose a novel MoST learning framework via Self-Supervised Learning, namely MoSSL, which aims to uncover latent patterns from temporal, spatial, and modality perspectives while quantifying dynamic heterogeneity. Experiment results on two real-world MoST datasets verify the superiority of our approach compared with the state-of-the-art baselines. Model implementation is available at https://github.com/beginner-sketch/MoSSL.

Title: Animate Your Thoughts: Decoupled Reconstruction of Dynamic Natural Vision from Slow Brain Activity

• Authors: Yizhuo Lu, Changde Du, Chong Wang, Xuanliu Zhu, Liuyun Jiang, Huiguang He
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.03280
• Pdf URL: https://arxiv.org/pdf/2405.03280
• Copy Paste: [[2405.03280]] Animate Your Thoughts: Decoupled Reconstruction of Dynamic Natural Vision from Slow Brain Activity(https://arxiv.org/abs/2405.03280)
• Keywords: diffusion, generative
• Abstract: Reconstructing human dynamic vision from brain activity is a challenging task with great scientific significance. The difficulty stems from two primary issues: (1) vision-processing mechanisms in the brain are highly intricate and not fully revealed, making it challenging to directly learn a mapping between fMRI and video; (2) the temporal resolution of fMRI is significantly lower than that of natural videos. To overcome these issues, this paper propose a two-stage model named Mind-Animator, which achieves state-of-the-art performance on three public datasets. Specifically, during the fMRI-to-feature stage, we decouple semantic, structural, and motion features from fMRI through fMRI-vision-language tri-modal contrastive learning and sparse causal attention. In the feature-to-video stage, these features are merged to videos by an inflated Stable Diffusion. We substantiate that the reconstructed video dynamics are indeed derived from fMRI, rather than hallucinations of the generative model, through permutation tests. Additionally, the visualization of voxel-wise and ROI-wise importance maps confirms the neurobiological interpretability of our model.

Title: Enhancing Spatiotemporal Disease Progression Models via Latent Diffusion and Prior Knowledge

• Authors: Lemuel Puglisi, Daniel C. Alexander, Daniele Ravì
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.03328
• Pdf URL: https://arxiv.org/pdf/2405.03328
• Copy Paste: [[2405.03328]] Enhancing Spatiotemporal Disease Progression Models via Latent Diffusion and Prior Knowledge(https://arxiv.org/abs/2405.03328)
• Keywords: diffusion, generative
• Abstract: In this work, we introduce Brain Latent Progression (BrLP), a novel spatiotemporal disease progression model based on latent diffusion. BrLP is designed to predict the evolution of diseases at the individual level on 3D brain MRIs. Existing deep generative models developed for this task are primarily data-driven and face challenges in learning disease progressions. BrLP addresses these challenges by incorporating prior knowledge from disease models to enhance the accuracy of predictions. To implement this, we propose to integrate an auxiliary model that infers volumetric changes in various brain regions. Additionally, we introduce Latent Average Stabilization (LAS), a novel technique to improve spatiotemporal consistency of the predicted progression. BrLP is trained and evaluated on a large dataset comprising 11,730 T1-weighted brain MRIs from 2,805 subjects, collected from three publicly available, longitudinal Alzheimer's Disease (AD) studies. In our experiments, we compare the MRI scans generated by BrLP with the actual follow-up MRIs available from the subjects, in both cross-sectional and longitudinal settings. BrLP demonstrates significant improvements over existing methods, with an increase of 22% in volumetric accuracy across AD-related brain regions and 43% in image similarity to the ground-truth scans. The ability of BrLP to generate conditioned 3D scans at the subject level, along with the novelty of integrating prior knowledge to enhance accuracy, represents a significant advancement in disease progression modeling, opening new avenues for precision medicine. The code of BrLP is available at the following link: https://github.com/LemuelPuglisi/BrLP.

Title: GLIP: Electromagnetic Field Exposure Map Completion by Deep Generative Networks

• Authors: Mohammed Mallik, Davy P. Gaillot, Laurent Clavier
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2405.03384
• Pdf URL: https://arxiv.org/pdf/2405.03384
• Copy Paste: [[2405.03384]] GLIP: Electromagnetic Field Exposure Map Completion by Deep Generative Networks(https://arxiv.org/abs/2405.03384)
• Keywords: generative
• Abstract: In Spectrum cartography (SC), the generation of exposure maps for radio frequency electromagnetic fields (RF-EMF) spans dimensions of frequency, space, and time, which relies on a sparse collection of sensor data, posing a challenging ill-posed inverse problem. Cartography methods based on models integrate designed priors, such as sparsity and low-rank structures, to refine the solution of this inverse problem. In our previous work, EMF exposure map reconstruction was achieved by Generative Adversarial Networks (GANs) where physical laws or structural constraints were employed as a prior, but they require a large amount of labeled data or simulated full maps for training to produce efficient results. In this paper, we present a method to reconstruct EMF exposure maps using only the generator network in GANs which does not require explicit training, thus overcoming the limitations of GANs, such as using reference full exposure maps. This approach uses a prior from sensor data as Local Image Prior (LIP) captured by deep convolutional generative networks independent of learning the network parameters from images in an urban environment. Experimental results show that, even when only sparse sensor data are available, our method can produce accurate estimates.

Title: LGTM: Local-to-Global Text-Driven Human Motion Diffusion Model

• Authors: Haowen Sun, Ruikun Zheng, Haibin Huang, Chongyang Ma, Hui Huang, Ruizhen Hu
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2405.03485
• Pdf URL: https://arxiv.org/pdf/2405.03485
• Copy Paste: [[2405.03485]] LGTM: Local-to-Global Text-Driven Human Motion Diffusion Model(https://arxiv.org/abs/2405.03485)
• Keywords: diffusion
• Abstract: In this paper, we introduce LGTM, a novel Local-to-Global pipeline for Text-to-Motion generation. LGTM utilizes a diffusion-based architecture and aims to address the challenge of accurately translating textual descriptions into semantically coherent human motion in computer animation. Specifically, traditional methods often struggle with semantic discrepancies, particularly in aligning specific motions to the correct body parts. To address this issue, we propose a two-stage pipeline to overcome this challenge: it first employs large language models (LLMs) to decompose global motion descriptions into part-specific narratives, which are then processed by independent body-part motion encoders to ensure precise local semantic alignment. Finally, an attention-based full-body optimizer refines the motion generation results and guarantees the overall coherence. Our experiments demonstrate that LGTM gains significant improvements in generating locally accurate, semantically-aligned human motion, marking a notable advancement in text-to-motion applications. Code and data for this paper are available at https://github.com/L-Sun/LGTM

Title: UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Images

• Authors: Yiting Qu, Xinyue Shen, Yixin Wu, Michael Backes, Savvas Zannettou, Yang Zhang
• Subjects: cs.CR, cs.CV, cs.SI
• Abstract URL: https://arxiv.org/abs/2405.03486
• Pdf URL: https://arxiv.org/pdf/2405.03486
• Copy Paste: [[2405.03486]] UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Images(https://arxiv.org/abs/2405.03486)
• Keywords: generative
• Abstract: Image safety classifiers play an important role in identifying and mitigating the spread of unsafe images online (e.g., images including violence, hateful rhetoric, etc.). At the same time, with the advent of text-to-image models and increasing concerns about the safety of AI models, developers are increasingly relying on image safety classifiers to safeguard their models. Yet, the performance of current image safety classifiers remains unknown for real-world and AI-generated images. To bridge this research gap, in this work, we propose UnsafeBench, a benchmarking framework that evaluates the effectiveness and robustness of image safety classifiers. First, we curate a large dataset of 10K real-world and AI-generated images that are annotated as safe or unsafe based on a set of 11 unsafe categories of images (sexual, violent, hateful, etc.). Then, we evaluate the effectiveness and robustness of five popular image safety classifiers, as well as three classifiers that are powered by general-purpose visual language models. Our assessment indicates that existing image safety classifiers are not comprehensive and effective enough in mitigating the multifaceted problem of unsafe images. Also, we find that classifiers trained only on real-world images tend to have degraded performance when applied to AI-generated images. Motivated by these findings, we design and implement a comprehensive image moderation tool called PerspectiveVision, which effectively identifies 11 categories of real-world and AI-generated unsafe images. The best PerspectiveVision model achieves an overall F1-Score of 0.810 on six evaluation datasets, which is comparable with closed-source and expensive state-of-the-art models like GPT-4V. UnsafeBench and PerspectiveVision can aid the research community in better understanding the landscape of image safety classification in the era of generative AI.

Title: Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond

• Authors: Zheng Zhu, Xiaofeng Wang, Wangbo Zhao, Chen Min, Nianchen Deng, Min Dou, Yuqi Wang, Botian Shi, Kai Wang, Chi Zhang, Yang You, Zhaoxiang Zhang, Dawei Zhao, Liang Xiao, Jian Zhao, Jiwen Lu, Guan Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.03520
• Pdf URL: https://arxiv.org/pdf/2405.03520
• Copy Paste: [[2405.03520]] Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond(https://arxiv.org/abs/2405.03520)
• Keywords: generative
• Abstract: General world models represent a crucial pathway toward achieving Artificial General Intelligence (AGI), serving as the cornerstone for various applications ranging from virtual environments to decision-making systems. Recently, the emergence of the Sora model has attained significant attention due to its remarkable simulation capabilities, which exhibits an incipient comprehension of physical laws. In this survey, we embark on a comprehensive exploration of the latest advancements in world models. Our analysis navigates through the forefront of generative methodologies in video generation, where world models stand as pivotal constructs facilitating the synthesis of highly realistic visual content. Additionally, we scrutinize the burgeoning field of autonomous-driving world models, meticulously delineating their indispensable role in reshaping transportation and urban mobility. Furthermore, we delve into the intricacies inherent in world models deployed within autonomous agents, shedding light on their profound significance in enabling intelligent interactions within dynamic environmental contexts. At last, we examine challenges and limitations of world models, and discuss their potential future directions. We hope this survey can serve as a foundational reference for the research community and inspire continued innovation. This survey will be regularly updated at: https://github.com/GigaAI-research/General-World-Models-Survey.

Title: CCDM: Continuous Conditional Diffusion Models for Image Generation

• Authors: Xin Ding, Yongwei Wang, Kao Zhang, Z. Jane Wang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.03546
• Pdf URL: https://arxiv.org/pdf/2405.03546
• Copy Paste: [[2405.03546]] CCDM: Continuous Conditional Diffusion Models for Image Generation(https://arxiv.org/abs/2405.03546)
• Keywords: diffusion, generative
• Abstract: Continuous Conditional Generative Modeling (CCGM) aims to estimate the distribution of high-dimensional data, typically images, conditioned on scalar continuous variables known as regression labels. While Continuous conditional Generative Adversarial Networks (CcGANs) were initially designed for this task, their adversarial training mechanism remains vulnerable to extremely sparse or imbalanced data, resulting in suboptimal outcomes. To enhance the quality of generated images, a promising alternative is to replace CcGANs with Conditional Diffusion Models (CDMs), renowned for their stable training process and ability to produce more realistic images. However, existing CDMs encounter challenges when applied to CCGM tasks due to several limitations such as inadequate U-Net architectures and deficient model fitting mechanisms for handling regression labels. In this paper, we introduce Continuous Conditional Diffusion Models (CCDMs), the first CDM designed specifically for the CCGM task. CCDMs address the limitations of existing CDMs by introducing specially designed conditional diffusion processes, a modified denoising U-Net with a custom-made conditioning mechanism, a novel hard vicinal loss for model fitting, and an efficient conditional sampling procedure. With comprehensive experiments on four datasets with varying resolutions ranging from 64x64 to 192x192, we demonstrate the superiority of the proposed CCDM over state-of-the-art CCGM models, establishing new benchmarks in CCGM. Extensive ablation studies validate the model design and implementation configuration of the proposed CCDM. Our code is publicly available at https://github.com/UBCDingXin/CCDM.

Title: Position Paper: Leveraging Foundational Models for Black-Box Optimization: Benefits, Challenges, and Future Directions

• Authors: Xingyou Song, Yingtao Tian, Robert Tjarko Lange, Chansoo Lee, Yujin Tang, Yutian Chen
• Subjects: cs.LG, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2405.03547
• Pdf URL: https://arxiv.org/pdf/2405.03547
• Copy Paste: [[2405.03547]] Position Paper: Leveraging Foundational Models for Black-Box Optimization: Benefits, Challenges, and Future Directions(https://arxiv.org/abs/2405.03547)
• Keywords: foundation model
• Abstract: Undeniably, Large Language Models (LLMs) have stirred an extraordinary wave of innovation in the machine learning research domain, resulting in substantial impact across diverse fields such as reinforcement learning, robotics, and computer vision. Their incorporation has been rapid and transformative, marking a significant paradigm shift in the field of machine learning research. However, the field of experimental design, grounded on black-box optimization, has been much less affected by such a paradigm shift, even though integrating LLMs with optimization presents a unique landscape ripe for exploration. In this position paper, we frame the field of black-box optimization around sequence-based foundation models and organize their relationship with previous literature. We discuss the most promising ways foundational language models can revolutionize optimization, which include harnessing the vast wealth of information encapsulated in free-form text to enrich task comprehension, utilizing highly flexible sequence models such as Transformers to engineer superior optimization strategies, and enhancing performance prediction over previously unseen search spaces.

Title: GREEN: Generative Radiology Report Evaluation and Error Notation

• Authors: Sophie Ostmeier, Justin Xu, Zhihong Chen, Maya Varma, Louis Blankemeier, Christian Bluethgen, Arne Edward Michalson, Michael Moseley, Curtis Langlotz, Akshay S Chaudhari, Jean-Benoit Delbrouck
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2405.03595
• Pdf URL: https://arxiv.org/pdf/2405.03595
• Copy Paste: [[2405.03595]] GREEN: Generative Radiology Report Evaluation and Error Notation(https://arxiv.org/abs/2405.03595)
• Keywords: generative
• Abstract: Evaluating radiology reports is a challenging problem as factual correctness is extremely important due to the need for accurate medical communication about medical images. Existing automatic evaluation metrics either suffer from failing to consider factual correctness (e.g., BLEU and ROUGE) or are limited in their interpretability (e.g., F1CheXpert and F1RadGraph). In this paper, we introduce GREEN (Generative Radiology Report Evaluation and Error Notation), a radiology report generation metric that leverages the natural language understanding of language models to identify and explain clinically significant errors in candidate reports, both quantitatively and qualitatively. Compared to current metrics, GREEN offers: 1) a score aligned with expert preferences, 2) human interpretable explanations of clinically significant errors, enabling feedback loops with end-users, and 3) a lightweight open-source method that reaches the performance of commercial counterparts. We validate our GREEN metric by comparing it to GPT-4, as well as to error counts of 6 experts and preferences of 2 experts. Our method demonstrates not only higher correlation with expert error counts, but simultaneously higher alignment with expert preferences when compared to previous approaches."

Title: Classification of Breast Cancer Histopathology Images using a Modified Supervised Contrastive Learning Method

• Authors: Matina Mahdizadeh Sani, Ali Royat, Mahdieh Soleymani Baghshah
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2405.03642
• Pdf URL: https://arxiv.org/pdf/2405.03642
• Copy Paste: [[2405.03642]] Classification of Breast Cancer Histopathology Images using a Modified Supervised Contrastive Learning Method(https://arxiv.org/abs/2405.03642)
• Keywords: self-supervised
• Abstract: Deep neural networks have reached remarkable achievements in medical image processing tasks, specifically classifying and detecting various diseases. However, when confronted with limited data, these networks face a critical vulnerability, often succumbing to overfitting by excessively memorizing the limited information available. This work addresses the challenge mentioned above by improving the supervised contrastive learning method to reduce the impact of false positives. Unlike most existing methods that rely predominantly on fully supervised learning, our approach leverages the advantages of self-supervised learning in conjunction with employing the available labeled data. We evaluate our method on the BreakHis dataset, which consists of breast cancer histopathology images, and demonstrate an increase in classification accuracy by 1.45% at the image level and 1.42% at the patient level compared to the state-of-the-art method. This improvement corresponds to 93.63% absolute accuracy, highlighting our approach's effectiveness in leveraging data properties to learn more appropriate representation space.

Title: Collecting Consistently High Quality Object Tracks with Minimal Human Involvement by Using Self-Supervised Learning to Detect Tracker Errors

• Authors: Samreen Anjum, Suyog Jain, Danna Gurari
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.03643
• Pdf URL: https://arxiv.org/pdf/2405.03643
• Copy Paste: [[2405.03643]] Collecting Consistently High Quality Object Tracks with Minimal Human Involvement by Using Self-Supervised Learning to Detect Tracker Errors(https://arxiv.org/abs/2405.03643)
• Keywords: self-supervised
• Abstract: We propose a hybrid framework for consistently producing high-quality object tracks by combining an automated object tracker with little human input. The key idea is to tailor a module for each dataset to intelligently decide when an object tracker is failing and so humans should be brought in to re-localize an object for continued tracking. Our approach leverages self-supervised learning on unlabeled videos to learn a tailored representation for a target object that is then used to actively monitor its tracked region and decide when the tracker fails. Since labeled data is not needed, our approach can be applied to novel object categories. Experiments on three datasets demonstrate our method outperforms existing approaches, especially for small, fast moving, or occluded objects.

Title: Field-of-View Extension for Diffusion MRI via Deep Generative Models

• Authors: Chenyu Gao, Shunxing Bao, Michael Kim, Nancy Newlin, Praitayini Kanakaraj, Tianyuan Yao, Gaurav Rudravaram, Yuankai Huo, Daniel Moyer, Kurt Schilling, Walter Kukull, Arthur Toga, Derek Archer, Timothy Hohman, Bennett Landman, Zhiyuan Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.03652
• Pdf URL: https://arxiv.org/pdf/2405.03652
• Copy Paste: [[2405.03652]] Field-of-View Extension for Diffusion MRI via Deep Generative Models(https://arxiv.org/abs/2405.03652)
• Keywords: diffusion, generative
• Abstract: Purpose: In diffusion MRI (dMRI), the volumetric and bundle analyses of whole-brain tissue microstructure and connectivity can be severely impeded by an incomplete field-of-view (FOV). This work aims to develop a method for imputing the missing slices directly from existing dMRI scans with an incomplete FOV. We hypothesize that the imputed image with complete FOV can improve the whole-brain tractography for corrupted data with incomplete FOV. Therefore, our approach provides a desirable alternative to discarding the valuable dMRI data, enabling subsequent tractography analyses that would otherwise be challenging or unattainable with corrupted data. Approach: We propose a framework based on a deep generative model that estimates the absent brain regions in dMRI scans with incomplete FOV. The model is capable of learning both the diffusion characteristics in diffusion-weighted images (DWI) and the anatomical features evident in the corresponding structural images for efficiently imputing missing slices of DWI outside of incomplete FOV. Results: For evaluating the imputed slices, on the WRAP dataset the proposed framework achieved PSNRb0=22.397, SSIMb0=0.905, PSNRb1300=22.479, SSIMb1300=0.893; on the NACC dataset it achieved PSNRb0=21.304, SSIMb0=0.892, PSNRb1300=21.599, SSIMb1300= 0.877. The proposed framework improved the tractography accuracy, as demonstrated by an increased average Dice score for 72 tracts (p < 0.001) on both the WRAP and NACC datasets. Conclusions: Results suggest that the proposed framework achieved sufficient imputation performance in dMRI data with incomplete FOV for improving whole-brain tractography, thereby repairing the corrupted data. Our approach achieved more accurate whole-brain tractography results with extended and complete FOV and reduced the uncertainty when analyzing bundles associated with Alzheimer's Disease.

Title: An Empty Room is All We Want: Automatic Defurnishing of Indoor Panoramas

• Authors: Mira Slavcheva, Dave Gausebeck, Kevin Chen, David Buchhofer, Azwad Sabik, Chen Ma, Sachal Dhillon, Olaf Brandt, Alan Dolhasz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2405.03682
• Pdf URL: https://arxiv.org/pdf/2405.03682
• Copy Paste: [[2405.03682]] An Empty Room is All We Want: Automatic Defurnishing of Indoor Panoramas(https://arxiv.org/abs/2405.03682)
• Keywords: diffusion
• Abstract: We propose a pipeline that leverages Stable Diffusion to improve inpainting results in the context of defurnishing -- the removal of furniture items from indoor panorama images. Specifically, we illustrate how increased context, domain-specific model fine-tuning, and improved image blending can produce high-fidelity inpaints that are geometrically plausible without needing to rely on room layout estimation. We demonstrate qualitative and quantitative improvements over other furniture removal techniques.