2024-01-30

Title: Context-driven self-supervised visual learning: Harnessing the environment as a data source

• Authors: Lizhen Zhu, James Z. Wang, Wonseuk Lee, Brad Wyble
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15120
• Pdf URL: https://arxiv.org/pdf/2401.15120
• Copy Paste: [[2401.15120]] Context-driven self-supervised visual learning: Harnessing the environment as a data source(https://arxiv.org/abs/2401.15120)
• Keywords: self-supervised
• Abstract: Visual learning often occurs in a specific context, where an agent acquires skills through exploration and tracking of its location in a consistent environment. The historical spatial context of the agent provides a similarity signal for self-supervised contrastive learning. We present a unique approach, termed Environmental Spatial Similarity (ESS), that complements existing contrastive learning methods. Using images from simulated, photorealistic environments as an experimental setting, we demonstrate that ESS outperforms traditional instance discrimination approaches. Moreover, sampling additional data from the same environment substantially improves accuracy and provides new augmentations. ESS allows remarkable proficiency in room classification and spatial prediction tasks, especially in unfamiliar environments. This learning paradigm has the potential to enable rapid visual learning in agents operating in new environments with unique visual characteristics. Potentially transformative applications span from robotics to space exploration. Our proof of concept demonstrates improved efficiency over methods that rely on extensive, disconnected datasets.

Title: Large Language Model Guided Knowledge Distillation for Time Series Anomaly Detection

• Authors: Chen Liu, Shibo He, Qihang Zhou, Shizhong Li, Wenchao Meng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15123
• Pdf URL: https://arxiv.org/pdf/2401.15123
• Copy Paste: [[2401.15123]] Large Language Model Guided Knowledge Distillation for Time Series Anomaly Detection(https://arxiv.org/abs/2401.15123)
• Keywords: self-supervised, anomaly
• Abstract: Self-supervised methods have gained prominence in time series anomaly detection due to the scarcity of available annotations. Nevertheless, they typically demand extensive training data to acquire a generalizable representation map, which conflicts with scenarios of a few available samples, thereby limiting their performance. To overcome the limitation, we propose \textbf{AnomalyLLM}, a knowledge distillation-based time series anomaly detection approach where the student network is trained to mimic the features of the large language model (LLM)-based teacher network that is pretrained on large-scale datasets. During the testing phase, anomalies are detected when the discrepancy between the features of the teacher and student networks is large. To circumvent the student network from learning the teacher network's feature of anomalous samples, we devise two key strategies. 1) Prototypical signals are incorporated into the student network to consolidate the normal feature extraction. 2) We use synthetic anomalies to enlarge the representation gap between the two networks. AnomalyLLM demonstrates state-of-the-art performance on 15 datasets, improving accuracy by at least 14.5\% in the UCR dataset.

Title: SCANIA Component X Dataset: A Real-World Multivariate Time Series Dataset for Predictive Maintenance

• Authors: Zahra Kharazian, Tony Lindgren, Sindri Magnússon, Olof Steinert, Oskar Andersson Reyna
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15199
• Pdf URL: https://arxiv.org/pdf/2401.15199
• Copy Paste: [[2401.15199]] SCANIA Component X Dataset: A Real-World Multivariate Time Series Dataset for Predictive Maintenance(https://arxiv.org/abs/2401.15199)
• Keywords: anomaly
• Abstract: This paper presents a description of a real-world, multivariate time series dataset collected from an anonymized engine component (called Component X) of a fleet of trucks from SCANIA, Sweden. This dataset includes diverse variables capturing detailed operational data, repair records, and specifications of trucks while maintaining confidentiality by anonymization. It is well-suited for a range of machine learning applications, such as classification, regression, survival analysis, and anomaly detection, particularly when applied to predictive maintenance scenarios. The large population size and variety of features in the format of histograms and numerical counters, along with the inclusion of temporal information, make this real-world dataset unique in the field. The objective of releasing this dataset is to give a broad range of researchers the possibility of working with real-world data from an internationally well-known company and introduce a standard benchmark to the predictive maintenance field, fostering reproducible research.

Title: Deep Learning with Tabular Data: A Self-supervised Approach

• Authors: Tirth Kiranbhai Vyas
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15238
• Pdf URL: https://arxiv.org/pdf/2401.15238
• Copy Paste: [[2401.15238]] Deep Learning with Tabular Data: A Self-supervised Approach(https://arxiv.org/abs/2401.15238)
• Keywords: self-supervised
• Abstract: We have described a novel approach for training tabular data using the TabTransformer model with self-supervised learning. Traditional machine learning models for tabular data, such as GBDT are being widely used though our paper examines the effectiveness of the TabTransformer which is a Transformer based model optimised specifically for tabular data. The TabTransformer captures intricate relationships and dependencies among features in tabular data by leveraging the self-attention mechanism of Transformers. We have used a self-supervised learning approach in this study, where the TabTransformer learns from unlabelled data by creating surrogate supervised tasks, eliminating the need for the labelled data. The aim is to find the most effective TabTransformer model representation of categorical and numerical features. To address the challenges faced during the construction of various input settings into the Transformers. Furthermore, a comparative analysis is also been conducted to examine performance of the TabTransformer model against baseline models such as MLP and supervised TabTransformer. The research has presented with a novel approach by creating various variants of TabTransformer model namely, Binned-TT, Vanilla-MLP-TT, MLP- based-TT which has helped to increase the effective capturing of the underlying relationship between various features of the tabular dataset by constructing optimal inputs. And further we have employed a self-supervised learning approach in the form of a masking-based unsupervised setting for tabular data. The findings shed light on the best way to represent categorical and numerical features, emphasizing the TabTransormer performance when compared to established machine learning models and other self-supervised learning methods.

Title: MEA-Defender: A Robust Watermark against Model Extraction Attack

• Authors: Peizhuo Lv, Hualong Ma, Kai Chen, Jiachen Zhou, Shengzhi Zhang, Ruigang Liang, Shenchen Zhu, Pan Li, Yingjun Zhang
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15239
• Pdf URL: https://arxiv.org/pdf/2401.15239
• Copy Paste: [[2401.15239]] MEA-Defender: A Robust Watermark against Model Extraction Attack(https://arxiv.org/abs/2401.15239)
• Keywords: self-supervised
• Abstract: Recently, numerous highly-valuable Deep Neural Networks (DNNs) have been trained using deep learning algorithms. To protect the Intellectual Property (IP) of the original owners over such DNN models, backdoor-based watermarks have been extensively studied. However, most of such watermarks fail upon model extraction attack, which utilizes input samples to query the target model and obtains the corresponding outputs, thus training a substitute model using such input-output pairs. In this paper, we propose a novel watermark to protect IP of DNN models against model extraction, named MEA-Defender. In particular, we obtain the watermark by combining two samples from two source classes in the input domain and design a watermark loss function that makes the output domain of the watermark within that of the main task samples. Since both the input domain and the output domain of our watermark are indispensable parts of those of the main task samples, the watermark will be extracted into the stolen model along with the main task during model extraction. We conduct extensive experiments on four model extraction attacks, using five datasets and six models trained based on supervised learning and self-supervised learning algorithms. The experimental results demonstrate that MEA-Defender is highly robust against different model extraction attacks, and various watermark removal/detection approaches.

Title: Improving Medical Reasoning through Retrieval and Self-Reflection with Retrieval-Augmented Large Language Models

• Authors: Minbyul Jeong, Jiwoong Sohn, Mujeen Sung, Jaewoo Kang
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2401.15269
• Pdf URL: https://arxiv.org/pdf/2401.15269
• Copy Paste: [[2401.15269]] Improving Medical Reasoning through Retrieval and Self-Reflection with Retrieval-Augmented Large Language Models(https://arxiv.org/abs/2401.15269)
• Keywords: foundation model
• Abstract: Recent proprietary large language models (LLMs), such as GPT-4, have achieved a milestone in tackling diverse challenges in the biomedical domain, ranging from multiple-choice questions to long-form generations. To address challenges that still cannot be handled with the encoded knowledge of LLMs, various retrieval-augmented generation (RAG) methods have been developed by searching documents from the knowledge corpus and appending them unconditionally or selectively to the input of LLMs for generation. However, when applying existing methods to different domain-specific problems, poor generalization becomes apparent, leading to fetching incorrect documents or making inaccurate judgments. In this paper, we introduce Self-BioRAG, a framework reliable for biomedical text that specializes in generating explanations, retrieving domain-specific documents, and self-reflecting generated responses. We utilize 84k filtered biomedical instruction sets to train Self-BioRAG that can assess its generated explanations with customized reflective tokens. Our work proves that domain-specific components, such as a retriever, domain-related document corpus, and instruction sets are necessary for adhering to domain-related instructions. Using three major medical question-answering benchmark datasets, experimental results of Self-BioRAG demonstrate significant performance gains by achieving a 7.2% absolute improvement on average over the state-of-the-art open-foundation model with a parameter size of 7B or less. Overall, we analyze that Self-BioRAG finds the clues in the question, retrieves relevant documents if needed, and understands how to answer with information from retrieved documents and encoded knowledge as a medical expert does. We release our data and code for training our framework components and model weights (7B and 13B) to enhance capabilities in biomedical and clinical domains.

Title: GEM: Boost Simple Network for Glass Surface Segmentation via Segment Anything Model and Data Synthesis

• Authors: Jing Hao, Moyun Liu, Kuo Feng Hung
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15282
• Pdf URL: https://arxiv.org/pdf/2401.15282
• Copy Paste: [[2401.15282]] GEM: Boost Simple Network for Glass Surface Segmentation via Segment Anything Model and Data Synthesis(https://arxiv.org/abs/2401.15282)
• Keywords: diffusion, foundation model
• Abstract: Detecting glass regions is a challenging task due to the ambiguity of their transparency and reflection properties. These transparent glasses share the visual appearance of both transmitted arbitrary background scenes and reflected objects, thus having no fixed patterns.Recent visual foundation models, which are trained on vast amounts of data, have manifested stunning performance in terms of image perception and image generation. To segment glass surfaces with higher accuracy, we make full use of two visual foundation models: Segment Anything (SAM) and Stable Diffusion.Specifically, we devise a simple glass surface segmentor named GEM, which only consists of a SAM backbone, a simple feature pyramid, a discerning query selection module, and a mask decoder. The discerning query selection can adaptively identify glass surface features, assigning them as initialized queries in the mask decoder. We also propose a Synthetic but photorealistic large-scale Glass Surface Detection dataset dubbed S-GSD via diffusion model with four different scales, which contain 1x, 5x, 10x, and 20x of the original real data size. This dataset is a feasible source for transfer learning. The scale of synthetic data has positive impacts on transfer learning, while the improvement will gradually saturate as the amount of data increases. Extensive experiments demonstrate that GEM achieves a new state-of-the-art on the GSD-S validation set (IoU +2.1%). Codes and datasets are available at: https://github.com/isbrycee/GEM-Glass-Segmentor.

Title: L-AutoDA: Leveraging Large Language Models for Automated Decision-based Adversarial Attacks

• Authors: Ping Guo, Fei Liu, Xi Lin, Qingchuan Zhao, Qingfu Zhang
• Subjects: cs.CR, cs.AI, cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15335
• Pdf URL: https://arxiv.org/pdf/2401.15335
• Copy Paste: [[2401.15335]] L-AutoDA: Leveraging Large Language Models for Automated Decision-based Adversarial Attacks(https://arxiv.org/abs/2401.15335)
• Keywords: generative
• Abstract: In the rapidly evolving field of machine learning, adversarial attacks present a significant challenge to model robustness and security. Decision-based attacks, which only require feedback on the decision of a model rather than detailed probabilities or scores, are particularly insidious and difficult to defend against. This work introduces L-AutoDA (Large Language Model-based Automated Decision-based Adversarial Attacks), a novel approach leveraging the generative capabilities of Large Language Models (LLMs) to automate the design of these attacks. By iteratively interacting with LLMs in an evolutionary framework, L-AutoDA automatically designs competitive attack algorithms efficiently without much human effort. We demonstrate the efficacy of L-AutoDA on CIFAR-10 dataset, showing significant improvements over baseline methods in both success rate and computational efficiency. Our findings underscore the potential of language models as tools for adversarial attack generation and highlight new avenues for the development of robust AI systems.

Title: Transformer-based Clipped Contrastive Quantization Learning for Unsupervised Image Retrieval

• Authors: Ayush Dubey, Shiv Ram Dubey, Satish Kumar Singh, Wei-Ta Chu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15362
• Pdf URL: https://arxiv.org/pdf/2401.15362
• Copy Paste: [[2401.15362]] Transformer-based Clipped Contrastive Quantization Learning for Unsupervised Image Retrieval(https://arxiv.org/abs/2401.15362)
• Keywords: self-supervised
• Abstract: Unsupervised image retrieval aims to learn the important visual characteristics without any given level to retrieve the similar images for a given query image. The Convolutional Neural Network (CNN)-based approaches have been extensively exploited with self-supervised contrastive learning for image hashing. However, the existing approaches suffer due to lack of effective utilization of global features by CNNs and biased-ness created by false negative pairs in the contrastive learning. In this paper, we propose a TransClippedCLR model by encoding the global context of an image using Transformer having local context through patch based processing, by generating the hash codes through product quantization and by avoiding the potential false negative pairs through clipped contrastive learning. The proposed model is tested with superior performance for unsupervised image retrieval on benchmark datasets, including CIFAR10, NUS-Wide and Flickr25K, as compared to the recent state-of-the-art deep models. The results using the proposed clipped contrastive learning are greatly improved on all datasets as compared to same backbone network with vanilla contrastive learning.

Title: Face to Cartoon Incremental Super-Resolution using Knowledge Distillation

• Authors: Trinetra Devkatte, Shiv Ram Dubey, Satish Kumar Singh, Abdenour Hadid
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.15366
• Pdf URL: https://arxiv.org/pdf/2401.15366
• Copy Paste: [[2401.15366]] Face to Cartoon Incremental Super-Resolution using Knowledge Distillation(https://arxiv.org/abs/2401.15366)
• Keywords: generative
• Abstract: Facial super-resolution/hallucination is an important area of research that seeks to enhance low-resolution facial images for a variety of applications. While Generative Adversarial Networks (GANs) have shown promise in this area, their ability to adapt to new, unseen data remains a challenge. This paper addresses this problem by proposing an incremental super-resolution using GANs with knowledge distillation (ISR-KD) for face to cartoon. Previous research in this area has not investigated incremental learning, which is critical for real-world applications where new data is continually being generated. The proposed ISR-KD aims to develop a novel unified framework for facial super-resolution that can handle different settings, including different types of faces such as cartoon face and various levels of detail. To achieve this, a GAN-based super-resolution network was pre-trained on the CelebA dataset and then incrementally trained on the iCartoonFace dataset, using knowledge distillation to retain performance on the CelebA test set while improving the performance on iCartoonFace test set. Our experiments demonstrate the effectiveness of knowledge distillation in incrementally adding capability to the model for cartoon face super-resolution while retaining the learned knowledge for facial hallucination tasks in GANs.

Title: A Survey on Data Augmentation in Large Model Era

• Authors: Yue Zhou, Chenlu Guo, Xu Wang, Yi Chang, Yuan Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15422
• Pdf URL: https://arxiv.org/pdf/2401.15422
• Copy Paste: [[2401.15422]] A Survey on Data Augmentation in Large Model Era(https://arxiv.org/abs/2401.15422)
• Keywords: diffusion
• Abstract: Large models, encompassing large language and diffusion models, have shown exceptional promise in approximating human-level intelligence, garnering significant interest from both academic and industrial spheres. However, the training of these large models necessitates vast quantities of high-quality data, and with continuous updates to these models, the existing reservoir of high-quality data may soon be depleted. This challenge has catalyzed a surge in research focused on data augmentation methods. Leveraging large models, these data augmentation techniques have outperformed traditional approaches. This paper offers an exhaustive review of large model-driven data augmentation methods, adopting a comprehensive perspective. We begin by establishing a classification of relevant studies into three main categories: image augmentation, text augmentation, and paired data augmentation. Following this, we delve into various data post-processing techniques pertinent to large model-based data augmentation. Our discussion then expands to encompass the array of applications for these data augmentation methods within natural language processing, computer vision, and audio signal processing. We proceed to evaluate the successes and limitations of large model-based data augmentation across different scenarios. Concluding our review, we highlight prospective challenges and avenues for future exploration in the field of data augmentation. Our objective is to furnish researchers with critical insights, ultimately contributing to the advancement of more sophisticated large models. We consistently maintain the related open-source materials at: https://github.com/MLGroup-JLU/LLM-data-aug-survey.

Title: Wind speed super-resolution and validation: from ERA5 to CERRA via diffusion models

• Authors: Fabio Merizzi, Andrea Asperti, Stefano Colamonaco
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15469
• Pdf URL: https://arxiv.org/pdf/2401.15469
• Copy Paste: [[2401.15469]] Wind speed super-resolution and validation: from ERA5 to CERRA via diffusion models(https://arxiv.org/abs/2401.15469)
• Keywords: diffusion
• Abstract: The Copernicus Regional Reanalysis for Europe, CERRA, is a high-resolution regional reanalysis dataset for the European domain. In recent years it has shown significant utility across various climate-related tasks, ranging from forecasting and climate change research to renewable energy prediction, resource management, air quality risk assessment, and the forecasting of rare events, among others. Unfortunately, the availability of CERRA is lagging two years behind the current date, due to constraints in acquiring the requisite external data and the intensive computational demands inherent in its generation. As a solution, this paper introduces a novel method using diffusion models to approximate CERRA downscaling in a data-driven manner, without additional informations. By leveraging the lower resolution ERA5 dataset, which provides boundary conditions for CERRA, we approach this as a super-resolution task. Focusing on wind speed around Italy, our model, trained on existing CERRA data, shows promising results, closely mirroring original CERRA data. Validation with in-situ observations further confirms the model's accuracy in approximating ground measurements.

Title: ConvoSense: Overcoming Monotonous Commonsense Inferences for Conversational AI

• Authors: Sarah E. Finch, Jinho D. Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.15471
• Pdf URL: https://arxiv.org/pdf/2401.15471
• Copy Paste: [[2401.15471]] ConvoSense: Overcoming Monotonous Commonsense Inferences for Conversational AI(https://arxiv.org/abs/2401.15471)
• Keywords: generative
• Abstract: Mastering commonsense understanding and reasoning is a pivotal skill essential for conducting engaging conversations. While there have been several attempts to create datasets that facilitate commonsense inferences in dialogue contexts, existing datasets tend to lack in-depth details, restate information already present in the conversation, and often fail to capture the multifaceted nature of commonsense reasoning. In response to these limitations, we compile a new synthetic dataset for commonsense reasoning in dialogue contexts using GPT, ConvoSense, that boasts greater contextual novelty, offers a higher volume of inferences per example, and substantially enriches the detail conveyed by the inferences. Our dataset contains over 500,000 inferences across 12,000 dialogues with 10 popular inference types, which empowers the training of generative commonsense models for dialogue that are superior in producing plausible inferences with high novelty when compared to models trained on the previous datasets. To the best of our knowledge, ConvoSense is the first of its kind to provide such a multitude of novel inferences at such a large scale.

Title: Style-News: Incorporating Stylized News Generation and Adversarial Verification for Neural Fake News Detection

• Authors: Wei-Yao Wang, Yu-Chieh Chang, Wen-Chih Peng
• Subjects: cs.CL, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2401.15509
• Pdf URL: https://arxiv.org/pdf/2401.15509
• Copy Paste: [[2401.15509]] Style-News: Incorporating Stylized News Generation and Adversarial Verification for Neural Fake News Detection(https://arxiv.org/abs/2401.15509)
• Keywords: generative
• Abstract: With the improvements in generative models, the issues of producing hallucinations in various domains (e.g., law, writing) have been brought to people's attention due to concerns about misinformation. In this paper, we focus on neural fake news, which refers to content generated by neural networks aiming to mimic the style of real news to deceive people. To prevent harmful disinformation spreading fallaciously from malicious social media (e.g., content farms), we propose a novel verification framework, Style-News, using publisher metadata to imply a publisher's template with the corresponding text types, political stance, and credibility. Based on threat modeling aspects, a style-aware neural news generator is introduced as an adversary for generating news content conditioning for a specific publisher, and style and source discriminators are trained to defend against this attack by identifying which publisher the style corresponds with, and discriminating whether the source of the given news is human-written or machine-generated. To evaluate the quality of the generated content, we integrate various dimensional metrics (language fluency, content preservation, and style adherence) and demonstrate that Style-News significantly outperforms the previous approaches by a margin of 0.35 for fluency, 15.24 for content, and 0.38 for style at most. Moreover, our discriminative model outperforms state-of-the-art baselines in terms of publisher prediction (up to 4.64%) and neural fake news detection (+6.94% $\sim$ 31.72%).

Title: Exploring the Transferability of a Foundation Model for Fundus Images: Application to Hypertensive Retinopathy

• Authors: Julio Silva-Rodriguez, Jihed Chelbi, Waziha Kabir, Hadi Chakor, Jose Dolz, Ismail Ben Ayed, Riadh Kobbi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15526
• Pdf URL: https://arxiv.org/pdf/2401.15526
• Copy Paste: [[2401.15526]] Exploring the Transferability of a Foundation Model for Fundus Images: Application to Hypertensive Retinopathy(https://arxiv.org/abs/2401.15526)
• Keywords: foundation model
• Abstract: Using deep learning models pre-trained on Imagenet is the traditional solution for medical image classification to deal with data scarcity. Nevertheless, relevant literature supports that this strategy may offer limited gains due to the high dissimilarity between domains. Currently, the paradigm of adapting domain-specialized foundation models is proving to be a promising alternative. However, how to perform such knowledge transfer, and the benefits and limitations it presents, are under study. The CGI-HRDC challenge for Hypertensive Retinopathy diagnosis on fundus images introduces an appealing opportunity to evaluate the transferability of a recently released vision-language foundation model of the retina, FLAIR. In this work, we explore the potential of using FLAIR features as starting point for fundus image classification, and we compare its performance with regard to Imagenet initialization on two popular transfer learning methods: Linear Probing (LP) and Fine-Tuning (FP). Our empirical observations suggest that, in any case, the use of the traditional strategy provides performance gains. In contrast, direct transferability from FLAIR model allows gains of 2.5%. When fine-tuning the whole network, the performance gap increases up to 4%. In this case, we show that avoiding feature deterioration via LP initialization of the classifier allows the best re-use of the rich pre-trained features. Although direct transferability using LP still offers limited performance, we believe that foundation models such as FLAIR will drive the evolution of deep-learning-based fundus image analysis.

Title: An Information-Theoretic Analysis of In-Context Learning

• Authors: Hong Jun Jeon, Jason D. Lee, Qi Lei, Benjamin Van Roy
• Subjects: cs.LG, cs.IT
• Abstract URL: https://arxiv.org/abs/2401.15530
• Pdf URL: https://arxiv.org/pdf/2401.15530
• Copy Paste: [[2401.15530]] An Information-Theoretic Analysis of In-Context Learning(https://arxiv.org/abs/2401.15530)
• Keywords: in-context
• Abstract: Previous theoretical results pertaining to meta-learning on sequences build on contrived assumptions and are somewhat convoluted. We introduce new information-theoretic tools that lead to an elegant and very general decomposition of error into three components: irreducible error, meta-learning error, and intra-task error. These tools unify analyses across many meta-learning challenges. To illustrate, we apply them to establish new results about in-context learning with transformers. Our theoretical results characterizes how error decays in both the number of training sequences and sequence lengths. Our results are very general; for example, they avoid contrived mixing time assumptions made by all prior results that establish decay of error with sequence length.

Title: Anomaly Detection of Particle Orbit in Accelerator using LSTM Deep Learning Technology

• Authors: Zhiyuan Chen, Wei Lu, Radhika Bhong, Yimin Hu, Brian Freeman, Adam Carpenter
• Subjects: cs.LG, physics.acc-ph
• Abstract URL: https://arxiv.org/abs/2401.15543
• Pdf URL: https://arxiv.org/pdf/2401.15543
• Copy Paste: [[2401.15543]] Anomaly Detection of Particle Orbit in Accelerator using LSTM Deep Learning Technology(https://arxiv.org/abs/2401.15543)
• Keywords: anomaly
• Abstract: A stable, reliable, and controllable orbit lock system is crucial to an electron (or ion) accelerator because the beam orbit and beam energy instability strongly affect the quality of the beam delivered to experimental halls. Currently, when the orbit lock system fails operators must manually intervene. This paper develops a Machine Learning based fault detection methodology to identify orbit lock anomalies and notify accelerator operations staff of the off-normal behavior. Our method is unsupervised, so it does not require labeled data. It uses Long-Short Memory Networks (LSTM) Auto Encoder to capture normal patterns and predict future values of monitoring sensors in the orbit lock system. Anomalies are detected when the prediction error exceeds a threshold. We conducted experiments using monitoring data from Jefferson Lab's Continuous Electron Beam Accelerator Facility (CEBAF). The results are promising: the percentage of real anomalies identified by our solution is 68.6%-89.3% using monitoring data of a single component in the orbit lock control system. The accuracy can be as high as 82%.

Title: BrepGen: A B-rep Generative Diffusion Model with Structured Latent Geometry

• Authors: Xiang Xu, Joseph G. Lambourne, Pradeep Kumar Jayaraman, Zhengqing Wang, Karl D.D. Willis, Yasutaka Furukawa
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15563
• Pdf URL: https://arxiv.org/pdf/2401.15563
• Copy Paste: [[2401.15563]] BrepGen: A B-rep Generative Diffusion Model with Structured Latent Geometry(https://arxiv.org/abs/2401.15563)
• Keywords: diffusion, generative
• Abstract: This paper presents BrepGen, a diffusion-based generative approach that directly outputs a Boundary representation (B-rep) Computer-Aided Design (CAD) model. BrepGen represents a B-rep model as a novel structured latent geometry in a hierarchical tree. With the root node representing a whole CAD solid, each element of a B-rep model (i.e., a face, an edge, or a vertex) progressively turns into a child-node from top to bottom. B-rep geometry information goes into the nodes as the global bounding box of each primitive along with a latent code describing the local geometric shape. The B-rep topology information is implicitly represented by node duplication. When two faces share an edge, the edge curve will appear twice in the tree, and a T-junction vertex with three incident edges appears six times in the tree with identical node features. Starting from the root and progressing to the leaf, BrepGen employs Transformer-based diffusion models to sequentially denoise node features while duplicated nodes are detected and merged, recovering the B-Rep topology information. Extensive experiments show that BrepGen sets a new milestone in CAD B-rep generation, surpassing existing methods on various benchmarks. Results on our newly collected furniture dataset further showcase its exceptional capability in generating complicated geometry. While previous methods were limited to generating simple prismatic shapes, BrepGen incorporates free-form and doubly-curved surfaces for the first time. Additional applications of BrepGen include CAD autocomplete and design interpolation. The code, pretrained models, and dataset will be released.

Title: Intriguing Equivalence Structures of the Embedding Space of Vision Transformers

• Authors: Shaeke Salman, Md Montasir Bin Shams, Xiuwen Liu
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2401.15568
• Pdf URL: https://arxiv.org/pdf/2401.15568
• Copy Paste: [[2401.15568]] Intriguing Equivalence Structures of the Embedding Space of Vision Transformers(https://arxiv.org/abs/2401.15568)
• Keywords: foundation model
• Abstract: Pre-trained large foundation models play a central role in the recent surge of artificial intelligence, resulting in fine-tuned models with remarkable abilities when measured on benchmark datasets, standard exams, and applications. Due to their inherent complexity, these models are not well understood. While small adversarial inputs to such models are well known, the structures of the representation space are not well characterized despite their fundamental importance. In this paper, using the vision transformers as an example due to the continuous nature of their input space, we show via analyses and systematic experiments that the representation space consists of large piecewise linear subspaces where there exist very different inputs sharing the same representations, and at the same time, local normal spaces where there are visually indistinguishable inputs having very different representations. The empirical results are further verified using the local directional estimations of the Lipschitz constants of the underlying models. Consequently, the resulting representations change the results of downstream models, and such models are subject to overgeneralization and with limited semantically meaningful generalization capability.

Title: Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalization

• Authors: Yinbin Han, Meisam Razaviyayn, Renyuan Xu
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2401.15604
• Pdf URL: https://arxiv.org/pdf/2401.15604
• Copy Paste: [[2401.15604]] Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalization(https://arxiv.org/abs/2401.15604)
• Keywords: diffusion
• Abstract: Diffusion models have emerged as a powerful tool rivaling GANs in generating high-quality samples with improved fidelity, flexibility, and robustness. A key component of these models is to learn the score function through score matching. Despite empirical success on various tasks, it remains unclear whether gradient-based algorithms can learn the score function with a provable accuracy. As a first step toward answering this question, this paper establishes a mathematical framework for analyzing score estimation using neural networks trained by gradient descent. Our analysis covers both the optimization and the generalization aspects of the learning procedure. In particular, we propose a parametric form to formulate the denoising score-matching problem as a regression with noisy labels. Compared to the standard supervised learning setup, the score-matching problem introduces distinct challenges, including unbounded input, vector-valued output, and an additional time variable, preventing existing techniques from being applied directly. In this paper, we show that with a properly designed neural network architecture, the score function can be accurately approximated by a reproducing kernel Hilbert space induced by neural tangent kernels. Furthermore, by applying an early-stopping rule for gradient descent and leveraging certain coupling arguments between neural network training and kernel regression, we establish the first generalization error (sample complexity) bounds for learning the score function despite the presence of noise in the observations. Our analysis is grounded in a novel parametric form of the neural network and an innovative connection between score matching and regression analysis, facilitating the application of advanced statistical and optimization techniques.

Title: Diffusion-based graph generative methods

• Authors: Hongyang Chen, Can Xu, Lingyu Zheng, Qiang Zhang, Xuemin Lin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15617
• Pdf URL: https://arxiv.org/pdf/2401.15617
• Copy Paste: [[2401.15617]] Diffusion-based graph generative methods(https://arxiv.org/abs/2401.15617)
• Keywords: diffusion, generative
• Abstract: Being the most cutting-edge generative methods, diffusion methods have shown great advances in wide generation tasks. Among them, graph generation attracts significant research attention for its broad application in real life. In our survey, we systematically and comprehensively review on diffusion-based graph generative methods. We first make a review on three mainstream paradigms of diffusion methods, which are denoising diffusion probabilistic models, score-based genrative models, and stochastic differential equations. Then we further categorize and introduce the latest applications of diffusion models on graphs. In the end, we point out some limitations of current studies and future directions of future explorations. The summary of existing methods metioned in this survey is in https://github.com/zhejiangzhuque/Diffusion-based-Graph-Generative-Methods.

Title: Generative AI-enabled Blockchain Networks: Fundamentals, Applications, and Case Study

• Authors: Cong T. Nguyen, Yinqiu Liu, Hongyang Du, Dinh Thai Hoang, Dusit Niyato, Diep N. Nguyen, Shiwen Mao
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15625
• Pdf URL: https://arxiv.org/pdf/2401.15625
• Copy Paste: [[2401.15625]] Generative AI-enabled Blockchain Networks: Fundamentals, Applications, and Case Study(https://arxiv.org/abs/2401.15625)
• Keywords: diffusion, generative
• Abstract: Generative Artificial Intelligence (GAI) has recently emerged as a promising solution to address critical challenges of blockchain technology, including scalability, security, privacy, and interoperability. In this paper, we first introduce GAI techniques, outline their applications, and discuss existing solutions for integrating GAI into blockchains. Then, we discuss emerging solutions that demonstrate the effectiveness of GAI in addressing various challenges of blockchain, such as detecting unknown blockchain attacks and smart contract vulnerabilities, designing key secret sharing schemes, and enhancing privacy. Moreover, we present a case study to demonstrate that GAI, specifically the generative diffusion model, can be employed to optimize blockchain network performance metrics. Experimental results clearly show that, compared to a baseline traditional AI approach, the proposed generative diffusion model approach can converge faster, achieve higher rewards, and significantly improve the throughput and latency of the blockchain network. Additionally, we highlight future research directions for GAI in blockchain applications, including personalized GAI-enabled blockchains, GAI-blockchain synergy, and privacy and security considerations within blockchain ecosystems.

Title: FreeStyle: Free Lunch for Text-guided Style Transfer using Diffusion Models

• Authors: Feihong He, Gang Li, Mengyuan Zhang, Leilei Yan, Lingyu Si, Fanzhang Li
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.15636
• Pdf URL: https://arxiv.org/pdf/2401.15636
• Copy Paste: [[2401.15636]] FreeStyle: Free Lunch for Text-guided Style Transfer using Diffusion Models(https://arxiv.org/abs/2401.15636)
• Keywords: diffusion, generative
• Abstract: The rapid development of generative diffusion models has significantly advanced the field of style transfer. However, most current style transfer methods based on diffusion models typically involve a slow iterative optimization process, e.g., model fine-tuning and textual inversion of style concept. In this paper, we introduce FreeStyle, an innovative style transfer method built upon a pre-trained large diffusion model, requiring no further optimization. Besides, our method enables style transfer only through a text description of the desired style, eliminating the necessity of style images. Specifically, we propose a dual-stream encoder and single-stream decoder architecture, replacing the conventional U-Net in diffusion models. In the dual-stream encoder, two distinct branches take the content image and style text prompt as inputs, achieving content and style decoupling. In the decoder, we further modulate features from the dual streams based on a given content image and the corresponding style text prompt for precise style transfer. Our experimental results demonstrate high-quality synthesis and fidelity of our method across various content images and style text prompts. The code and more results are available at our project website:https://freestylefreelunch.github.io/.

Title: UP-CrackNet: Unsupervised Pixel-Wise Road Crack Detection via Adversarial Image Restoration

• Authors: Nachuan Ma, Rui Fan, Lihua Xie
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2401.15647
• Pdf URL: https://arxiv.org/pdf/2401.15647
• Copy Paste: [[2401.15647]] UP-CrackNet: Unsupervised Pixel-Wise Road Crack Detection via Adversarial Image Restoration(https://arxiv.org/abs/2401.15647)
• Keywords: generative, anomaly
• Abstract: Over the past decade, automated methods have been developed to detect cracks more efficiently, accurately, and objectively, with the ultimate goal of replacing conventional manual visual inspection techniques. Among these methods, semantic segmentation algorithms have demonstrated promising results in pixel-wise crack detection tasks. However, training such data-driven algorithms requires a large amount of human-annotated datasets with pixel-level annotations, which is a highly labor-intensive and time-consuming process. Moreover, supervised learning-based methods often struggle with poor generalization ability in unseen datasets. Therefore, we propose an unsupervised pixel-wise road crack detection network, known as UP-CrackNet. Our approach first generates multi-scale square masks and randomly selects them to corrupt undamaged road images by removing certain regions. Subsequently, a generative adversarial network is trained to restore the corrupted regions by leveraging the semantic context learned from surrounding uncorrupted regions. During the testing phase, an error map is generated by calculating the difference between the input and restored images, which allows for pixel-wise crack detection. Our comprehensive experimental results demonstrate that UP-CrackNet outperforms other general-purpose unsupervised anomaly detection algorithms, and exhibits comparable performance and superior generalizability when compared with state-of-the-art supervised crack segmentation algorithms. Our source code is publicly available at mias.group/UP-CrackNet.

Title: CPDM: Content-Preserving Diffusion Model for Underwater Image Enhancement

• Authors: Xiaowen Shi, Yuan-Gen Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15649
• Pdf URL: https://arxiv.org/pdf/2401.15649
• Copy Paste: [[2401.15649]] CPDM: Content-Preserving Diffusion Model for Underwater Image Enhancement(https://arxiv.org/abs/2401.15649)
• Keywords: diffusion
• Abstract: Underwater image enhancement (UIE) is challenging since image degradation in aquatic environments is complicated and changing over time. Existing mainstream methods rely on either physical-model or data-driven, suffering from performance bottlenecks due to changes in imaging conditions or training instability. In this article, we make the first attempt to adapt the diffusion model to the UIE task and propose a Content-Preserving Diffusion Model (CPDM) to address the above challenges. CPDM first leverages a diffusion model as its fundamental model for stable training and then designs a content-preserving framework to deal with changes in imaging conditions. Specifically, we construct a conditional input module by adopting both the raw image and the difference between the raw and noisy images as the input, which can enhance the model's adaptability by considering the changes involving the raw images in underwater environments. To preserve the essential content of the raw images, we construct a content compensation module for content-aware training by extracting low-level features from the raw images. Extensive experimental results validate the effectiveness of our CPDM, surpassing the state-of-the-art methods in terms of both subjective and objective metrics.

Title: Continuous-Multiple Image Outpainting in One-Step via Positional Query and A Diffusion-based Approach

• Authors: Shaofeng Zhang, Jinfa Huang, Qiang Zhou, Zhibin Wang, Fan Wang, Jiebo Luo, Junchi Yan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15652
• Pdf URL: https://arxiv.org/pdf/2401.15652
• Copy Paste: [[2401.15652]] Continuous-Multiple Image Outpainting in One-Step via Positional Query and A Diffusion-based Approach(https://arxiv.org/abs/2401.15652)
• Keywords: diffusion, generative
• Abstract: Image outpainting aims to generate the content of an input sub-image beyond its original boundaries. It is an important task in content generation yet remains an open problem for generative models. This paper pushes the technical frontier of image outpainting in two directions that have not been resolved in literature: 1) outpainting with arbitrary and continuous multiples (without restriction), and 2) outpainting in a single step (even for large expansion multiples). Moreover, we develop a method that does not depend on a pre-trained backbone network, which is in contrast commonly required by the previous SOTA outpainting methods. The arbitrary multiple outpainting is achieved by utilizing randomly cropped views from the same image during training to capture arbitrary relative positional information. Specifically, by feeding one view and positional embeddings as queries, we can reconstruct another view. At inference, we generate images with arbitrary expansion multiples by inputting an anchor image and its corresponding positional embeddings. The one-step outpainting ability here is particularly noteworthy in contrast to previous methods that need to be performed for $N$ times to obtain a final multiple which is $N$ times of its basic and fixed multiple. We evaluate the proposed approach (called PQDiff as we adopt a diffusion-based generator as our embodiment, under our proposed \textbf{P}ositional \textbf{Q}uery scheme) on public benchmarks, demonstrating its superior performance over state-of-the-art approaches. Specifically, PQDiff achieves state-of-the-art FID scores on the Scenery (\textbf{21.512}), Building Facades (\textbf{25.310}), and WikiArts (\textbf{36.212}) datasets. Furthermore, under the 2.25x, 5x and 11.7x outpainting settings, PQDiff only takes \textbf{40.6\%}, \textbf{20.3\%} and \textbf{10.2\%} of the time of the benchmark state-of-the-art (SOTA) method.

Title: LLsM: Generative Linguistic Steganography with Large Language Model

• Authors: Yihao Wang, Ruiqi Song, Ru Zhang, Jianyi Liu, Lingxiao Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2401.15656
• Pdf URL: https://arxiv.org/pdf/2401.15656
• Copy Paste: [[2401.15656]] LLsM: Generative Linguistic Steganography with Large Language Model(https://arxiv.org/abs/2401.15656)
• Keywords: generative
• Abstract: Linguistic Steganography (LS) tasks aim to generate steganographic texts (stego) based on secret information. Only authorized recipients can perceive the existence of secret information in the texts and accurately extract it, thereby preserving privacy. However, the controllability of the stego generated by existing schemes is poor, and the generated stego is difficult to contain specific discourse characteristics such as style, genre, and theme. As a result, the stego are often easily detectable, compromising covert communication. To address these problems, this paper proposes a novel scheme named LLsM, a generative LS based on a Large Language Model (LLM). We fine-tuned the LLM LLaMA2 with a large-scale constructed dataset encompassing rich discourse characteristics, which enables the fine-tuned LLM to generate texts with specific discourse in a controllable manner. Then the discourse characteristics are used as guiding information and inputted into the fine-tuned LLM in the form of Prompt together with secret information. The candidate pool, derived from sampling and truncation, undergoes range encoding to ensure the stego imitate natural text distribution. Experiments demonstrate that LLsM performs superior to prevalent baselines regarding text quality, statistical analysis, discourse matching, and anti-steganalysis. In particular, LLsM's MAUVE surpasses that of some baselines by 70%-80%, and its anti-steganalysis performance is 30%-40% higher. Notably, we also present the long stego generated by LLsM, showing its potential superiority in long LS tasks.

Title: Data-Free Generalized Zero-Shot Learning

• Authors: Bowen Tang, Long Yan, Jing Zhang, Qian Yu, Lu Sheng, Dong Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15657
• Pdf URL: https://arxiv.org/pdf/2401.15657
• Copy Paste: [[2401.15657]] Data-Free Generalized Zero-Shot Learning(https://arxiv.org/abs/2401.15657)
• Keywords: generative
• Abstract: Deep learning models have the ability to extract rich knowledge from large-scale datasets. However, the sharing of data has become increasingly challenging due to concerns regarding data copyright and privacy. Consequently, this hampers the effective transfer of knowledge from existing data to novel downstream tasks and concepts. Zero-shot learning (ZSL) approaches aim to recognize new classes by transferring semantic knowledge learned from base classes. However, traditional generative ZSL methods often require access to real images from base classes and rely on manually annotated attributes, which presents challenges in terms of data restrictions and model scalability. To this end, this paper tackles a challenging and practical problem dubbed as data-free zero-shot learning (DFZSL), where only the CLIP-based base classes data pre-trained classifier is available for zero-shot classification. Specifically, we propose a generic framework for DFZSL, which consists of three main components. Firstly, to recover the virtual features of the base data, we model the CLIP features of base class images as samples from a von Mises-Fisher (vMF) distribution based on the pre-trained classifier. Secondly, we leverage the text features of CLIP as low-cost semantic information and propose a feature-language prompt tuning (FLPT) method to further align the virtual image features and textual features. Thirdly, we train a conditional generative model using the well-aligned virtual image features and corresponding semantic text features, enabling the generation of new classes features and achieve better zero-shot generalization. Our framework has been evaluated on five commonly used benchmarks for generalized ZSL, as well as 11 benchmarks for the base-to-new ZSL. The results demonstrate the superiority and effectiveness of our approach. Our code is available in https://github.com/ylong4/DFZSL

Title: Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidance

• Authors: Qingcheng Zhao, Pengyu Long, Qixuan Zhang, Dafei Qin, Han Liang, Longwen Zhang, Yingliang Zhang, Jingyi Yu, Lan Xu
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2401.15687
• Pdf URL: https://arxiv.org/pdf/2401.15687
• Copy Paste: [[2401.15687]] Media2Face: Co-speech Facial Animation Generation With Multi-Modality Guidance(https://arxiv.org/abs/2401.15687)
• Keywords: diffusion
• Abstract: The synthesis of 3D facial animations from speech has garnered considerable attention. Due to the scarcity of high-quality 4D facial data and well-annotated abundant multi-modality labels, previous methods often suffer from limited realism and a lack of lexible conditioning. We address this challenge through a trilogy. We first introduce Generalized Neural Parametric Facial Asset (GNPFA), an efficient variational auto-encoder mapping facial geometry and images to a highly generalized expression latent space, decoupling expressions and identities. Then, we utilize GNPFA to extract high-quality expressions and accurate head poses from a large array of videos. This presents the M2F-D dataset, a large, diverse, and scan-level co-speech 3D facial animation dataset with well-annotated emotional and style labels. Finally, we propose Media2Face, a diffusion model in GNPFA latent space for co-speech facial animation generation, accepting rich multi-modality guidances from audio, text, and image. Extensive experiments demonstrate that our model not only achieves high fidelity in facial animation synthesis but also broadens the scope of expressiveness and style adaptability in 3D facial animation.

Title: Object-Driven One-Shot Fine-tuning of Text-to-Image Diffusion with Prototypical Embedding

• Authors: Jianxiang Lu, Cong Xie, Hui Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15708
• Pdf URL: https://arxiv.org/pdf/2401.15708
• Copy Paste: [[2401.15708]] Object-Driven One-Shot Fine-tuning of Text-to-Image Diffusion with Prototypical Embedding(https://arxiv.org/abs/2401.15708)
• Keywords: diffusion
• Abstract: As large-scale text-to-image generation models have made remarkable progress in the field of text-to-image generation, many fine-tuning methods have been proposed. However, these models often struggle with novel objects, especially with one-shot scenarios. Our proposed method aims to address the challenges of generalizability and fidelity in an object-driven way, using only a single input image and the object-specific regions of interest. To improve generalizability and mitigate overfitting, in our paradigm, a prototypical embedding is initialized based on the object's appearance and its class, before fine-tuning the diffusion model. And during fine-tuning, we propose a class-characterizing regularization to preserve prior knowledge of object classes. To further improve fidelity, we introduce object-specific loss, which can also use to implant multiple objects. Overall, our proposed object-driven method for implanting new objects can integrate seamlessly with existing concepts as well as with high fidelity and generalization. Our method outperforms several existing works. The code will be released.

Title: Cross-Scale MAE: A Tale of Multi-Scale Exploitation in Remote Sensing

• Authors: Maofeng Tang, Andrei Cozma, Konstantinos Georgiou, Hairong Qi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15855
• Pdf URL: https://arxiv.org/pdf/2401.15855
• Copy Paste: [[2401.15855]] Cross-Scale MAE: A Tale of Multi-Scale Exploitation in Remote Sensing(https://arxiv.org/abs/2401.15855)
• Keywords: self-supervised, generative
• Abstract: Remote sensing images present unique challenges to image analysis due to the extensive geographic coverage, hardware limitations, and misaligned multi-scale images. This paper revisits the classical multi-scale representation learning problem but under the general framework of self-supervised learning for remote sensing image understanding. We present Cross-Scale MAE, a self-supervised model built upon the Masked Auto-Encoder (MAE).During pre-training, Cross-Scale MAE employs scale augmentation techniques and enforces cross-scale consistency constraints through both contrastive and generative losses to ensure consistent and meaningful representations well-suited for a wide range of downstream tasks. Further, our implementation leverages the xFormers library to accelerate network pre-training on a single GPU while maintaining the quality of learned representations. Experimental evaluations demonstrate that Cross-Scale MAE exhibits superior performance compared to standard MAE and other state-of-the-art remote sensing MAE methods.

Title: Diffusion Facial Forgery Detection

• Authors: Harry Cheng, Yangyang Guo, Tianyi Wang, Liqiang Nie, Mohan Kankanhalli
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15859
• Pdf URL: https://arxiv.org/pdf/2401.15859
• Copy Paste: [[2401.15859]] Diffusion Facial Forgery Detection(https://arxiv.org/abs/2401.15859)
• Keywords: diffusion
• Abstract: Detecting diffusion-generated images has recently grown into an emerging research area. Existing diffusion-based datasets predominantly focus on general image generation. However, facial forgeries, which pose a more severe social risk, have remained less explored thus far. To address this gap, this paper introduces DiFF, a comprehensive dataset dedicated to face-focused diffusion-generated images. DiFF comprises over 500,000 images that are synthesized using thirteen distinct generation methods under four conditions. In particular, this dataset leverages 30,000 carefully collected textual and visual prompts, ensuring the synthesis of images with both high fidelity and semantic consistency. We conduct extensive experiments on the DiFF dataset via a human test and several representative forgery detection methods. The results demonstrate that the binary detection accuracy of both human observers and automated detectors often falls below 30%, shedding light on the challenges in detecting diffusion-generated facial forgeries. Furthermore, we propose an edge graph regularization approach to effectively enhance the generalization capability of existing detectors.

Title: $\boldsymbol{M^2}$-Encoder: Advancing Bilingual Image-Text Understanding by Large-scale Efficient Pretraining

• Authors: Qingpei Guo, Furong Xu, Hanxiao Zhang, Wang Ren, Ziping Ma, Lin Ju, Jian Wang, Jingdong Chen, Ming Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15896
• Pdf URL: https://arxiv.org/pdf/2401.15896
• Copy Paste: [[2401.15896]] $\boldsymbol{M^2}$-Encoder: Advancing Bilingual Image-Text Understanding by Large-scale Efficient Pretraining(https://arxiv.org/abs/2401.15896)
• Keywords: foundation model
• Abstract: Vision-language foundation models like CLIP have revolutionized the field of artificial intelligence. Nevertheless, VLM models supporting multi-language, e.g., in both Chinese and English, have lagged due to the relative scarcity of large-scale pretraining datasets. Toward this end, we introduce a comprehensive bilingual (Chinese-English) dataset BM-6B with over 6 billion image-text pairs, aimed at enhancing multimodal foundation models to well understand images in both languages. To handle such a scale of dataset, we propose a novel grouped aggregation approach for image-text contrastive loss computation, which reduces the communication overhead and GPU memory demands significantly, facilitating a 60% increase in training speed. We pretrain a series of bilingual image-text foundation models with an enhanced fine-grained understanding ability on BM-6B, the resulting models, dubbed as $M^2$-Encoders (pronounced "M-Square"), set new benchmarks in both languages for multimodal retrieval and classification tasks. Notably, Our largest $M^2$-Encoder-10B model has achieved top-1 accuracies of 88.5% on ImageNet and 80.7% on ImageNet-CN under a zero-shot classification setting, surpassing previously reported SoTA methods by 2.2% and 21.1%, respectively. The $M^2$-Encoder series represents one of the most comprehensive bilingual image-text foundation models to date, so we are making it available to the research community for further exploration and development.

Title: MV2MAE: Multi-View Video Masked Autoencoders

• Authors: Ketul Shah, Robert Crandall, Jie Xu, Peng Zhou, Marian George, Mayank Bansal, Rama Chellappa
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15900
• Pdf URL: https://arxiv.org/pdf/2401.15900
• Copy Paste: [[2401.15900]] MV2MAE: Multi-View Video Masked Autoencoders(https://arxiv.org/abs/2401.15900)
• Keywords: self-supervised
• Abstract: Videos captured from multiple viewpoints can help in perceiving the 3D structure of the world and benefit computer vision tasks such as action recognition, tracking, etc. In this paper, we present a method for self-supervised learning from synchronized multi-view videos. We use a cross-view reconstruction task to inject geometry information in the model. Our approach is based on the masked autoencoder (MAE) framework. In addition to the same-view decoder, we introduce a separate cross-view decoder which leverages cross-attention mechanism to reconstruct a target viewpoint video using a video from source viewpoint, to help representations robust to viewpoint changes. For videos, static regions can be reconstructed trivially which hinders learning meaningful representations. To tackle this, we introduce a motion-weighted reconstruction loss which improves temporal modeling. We report state-of-the-art results on the NTU-60, NTU-120 and ETRI datasets, as well as in the transfer learning setting on NUCLA, PKU-MMD-II and ROCOG-v2 datasets, demonstrating the robustness of our approach. Code will be made available.

Title: Toward the Identifiability of Comparative Deep Generative Models

• Authors: Romain Lopez, Jan-Christian Huetter, Ehsan Hajiramezanali, Jonathan Pritchard, Aviv Regev
• Subjects: cs.LG, q-bio.GN, stat.ME
• Abstract URL: https://arxiv.org/abs/2401.15903
• Pdf URL: https://arxiv.org/pdf/2401.15903
• Copy Paste: [[2401.15903]] Toward the Identifiability of Comparative Deep Generative Models(https://arxiv.org/abs/2401.15903)
• Keywords: generative
• Abstract: Deep Generative Models (DGMs) are versatile tools for learning data representations while adequately incorporating domain knowledge such as the specification of conditional probability distributions. Recently proposed DGMs tackle the important task of comparing data sets from different sources. One such example is the setting of contrastive analysis that focuses on describing patterns that are enriched in a target data set compared to a background data set. The practical deployment of those models often assumes that DGMs naturally infer interpretable and modular latent representations, which is known to be an issue in practice. Consequently, existing methods often rely on ad-hoc regularization schemes, although without any theoretical grounding. Here, we propose a theory of identifiability for comparative DGMs by extending recent advances in the field of non-linear independent component analysis. We show that, while these models lack identifiability across a general class of mixing functions, they surprisingly become identifiable when the mixing function is piece-wise affine (e.g., parameterized by a ReLU neural network). We also investigate the impact of model misspecification, and empirically show that previously proposed regularization techniques for fitting comparative DGMs help with identifiability when the number of latent variables is not known in advance. Finally, we introduce a novel methodology for fitting comparative DGMs that improves the treatment of multiple data sources via multi-objective optimization and that helps adjust the hyperparameter for the regularization in an interpretable manner, using constrained optimization. We empirically validate our theory and new methodology using simulated data as well as a recent data set of genetic perturbations in cells profiled via single-cell RNA sequencing.

Title: Self-Supervised Learning in Event Sequences: A Comparative Study and Hybrid Approach of Generative Modeling and Contrastive Learning

• Authors: Viktor Moskvoretskii, Dmitry Osin, Egor Shvetsov, Igor Udovichenko, Maxim Zhelnin, Andrey Dukhovny, Anna Zhimerikina, Albert Efimov, Evgeny Burnaev
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.15935
• Pdf URL: https://arxiv.org/pdf/2401.15935
• Copy Paste: [[2401.15935]] Self-Supervised Learning in Event Sequences: A Comparative Study and Hybrid Approach of Generative Modeling and Contrastive Learning(https://arxiv.org/abs/2401.15935)
• Keywords: self-supervised, generative
• Abstract: This study investigates self-supervised learning techniques to obtain representations of Event Sequences. It is a key modality in various applications, including but not limited to banking, e-commerce, and healthcare. We perform a comprehensive study of generative and contrastive approaches in self-supervised learning, applying them both independently. We find that there is no single supreme method. Consequently, we explore the potential benefits of combining these approaches. To achieve this goal, we introduce a novel method that aligns generative and contrastive embeddings as distinct modalities, drawing inspiration from contemporary multimodal research. Generative and contrastive approaches are often treated as mutually exclusive, leaving a gap for their combined exploration. Our results demonstrate that this aligned model performs at least on par with, and mostly surpasses, existing methods and is more universal across a variety of tasks. Furthermore, we demonstrate that self-supervised methods consistently outperform the supervised approach on our datasets.

Title: AdvNF: Reducing Mode Collapse in Conditional Normalising Flows using Adversarial Learning

• Authors: Vikas Kanaujia, Mathias S. Scheurer, Vipul Arora
• Subjects: cs.LG, cond-mat.stat-mech, physics.comp-ph
• Abstract URL: https://arxiv.org/abs/2401.15948
• Pdf URL: https://arxiv.org/pdf/2401.15948
• Copy Paste: [[2401.15948]] AdvNF: Reducing Mode Collapse in Conditional Normalising Flows using Adversarial Learning(https://arxiv.org/abs/2401.15948)
• Keywords: generative
• Abstract: Deep generative models complement Markov-chain-Monte-Carlo methods for efficiently sampling from high-dimensional distributions. Among these methods, explicit generators, such as Normalising Flows (NFs), in combination with the Metropolis Hastings algorithm have been extensively applied to get unbiased samples from target distributions. We systematically study central problems in conditional NFs, such as high variance, mode collapse and data efficiency. We propose adversarial training for NFs to ameliorate these problems. Experiments are conducted with low-dimensional synthetic datasets and XY spin models in two spatial dimensions.

Title: StableIdentity: Inserting Anybody into Anywhere at First Sight

• Authors: Qinghe Wang, Xu Jia, Xiaomin Li, Taiqing Li, Liqian Ma, Yunzhi Zhuge, Huchuan Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15975
• Pdf URL: https://arxiv.org/pdf/2401.15975
• Copy Paste: [[2401.15975]] StableIdentity: Inserting Anybody into Anywhere at First Sight(https://arxiv.org/abs/2401.15975)
• Keywords: diffusion
• Abstract: Recent advances in large pretrained text-to-image models have shown unprecedented capabilities for high-quality human-centric generation, however, customizing face identity is still an intractable problem. Existing methods cannot ensure stable identity preservation and flexible editability, even with several images for each subject during training. In this work, we propose StableIdentity, which allows identity-consistent recontextualization with just one face image. More specifically, we employ a face encoder with an identity prior to encode the input face, and then land the face representation into a space with an editable prior, which is constructed from celeb names. By incorporating identity prior and editability prior, the learned identity can be injected anywhere with various contexts. In addition, we design a masked two-phase diffusion loss to boost the pixel-level perception of the input face and maintain the diversity of generation. Extensive experiments demonstrate our method outperforms previous customization methods. In addition, the learned identity can be flexibly combined with the off-the-shelf modules such as ControlNet. Notably, to the best knowledge, we are the first to directly inject the identity learned from a single image into video/3D generation without finetuning. We believe that the proposed StableIdentity is an important step to unify image, video, and 3D customized generation models.

Title: Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling

• Authors: Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Da, Hongsheng Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.15977
• Pdf URL: https://arxiv.org/pdf/2401.15977
• Copy Paste: [[2401.15977]] Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling(https://arxiv.org/abs/2401.15977)
• Keywords: diffusion
• Abstract: We introduce Motion-I2V, a novel framework for consistent and controllable image-to-video generation (I2V). In contrast to previous methods that directly learn the complicated image-to-video mapping, Motion-I2V factorizes I2V into two stages with explicit motion modeling. For the first stage, we propose a diffusion-based motion field predictor, which focuses on deducing the trajectories of the reference image's pixels. For the second stage, we propose motion-augmented temporal attention to enhance the limited 1-D temporal attention in video latent diffusion models. This module can effectively propagate reference image's feature to synthesized frames with the guidance of predicted trajectories from the first stage. Compared with existing methods, Motion-I2V can generate more consistent videos even at the presence of large motion and viewpoint variation. By training a sparse trajectory ControlNet for the first stage, Motion-I2V can support users to precisely control motion trajectories and motion regions with sparse trajectory and region annotations. This offers more controllability of the I2V process than solely relying on textual instructions. Additionally, Motion-I2V's second stage naturally supports zero-shot video-to-video translation. Both qualitative and quantitative comparisons demonstrate the advantages of Motion-I2V over prior approaches in consistent and controllable image-to-video generation.

Title: GPS: Graph Contrastive Learning via Multi-scale Augmented Views from Adversarial Pooling

• Authors: Wei Ju, Yiyang Gu, Zhengyang Mao, Ziyue Qiao, Yifang Qin, Xiao Luo, Hui Xiong, Ming Zhang
• Subjects: cs.LG, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2401.16011
• Pdf URL: https://arxiv.org/pdf/2401.16011
• Copy Paste: [[2401.16011]] GPS: Graph Contrastive Learning via Multi-scale Augmented Views from Adversarial Pooling(https://arxiv.org/abs/2401.16011)
• Keywords: self-supervised
• Abstract: Self-supervised graph representation learning has recently shown considerable promise in a range of fields, including bioinformatics and social networks. A large number of graph contrastive learning approaches have shown promising performance for representation learning on graphs, which train models by maximizing agreement between original graphs and their augmented views (i.e., positive views). Unfortunately, these methods usually involve pre-defined augmentation strategies based on the knowledge of human experts. Moreover, these strategies may fail to generate challenging positive views to provide sufficient supervision signals. In this paper, we present a novel approach named Graph Pooling ContraSt (GPS) to address these issues. Motivated by the fact that graph pooling can adaptively coarsen the graph with the removal of redundancy, we rethink graph pooling and leverage it to automatically generate multi-scale positive views with varying emphasis on providing challenging positives and preserving semantics, i.e., strongly-augmented view and weakly-augmented view. Then, we incorporate both views into a joint contrastive learning framework with similarity learning and consistency learning, where our pooling module is adversarially trained with respect to the encoder for adversarial robustness. Experiments on twelve datasets on both graph classification and transfer learning tasks verify the superiority of the proposed method over its counterparts.

Title: Spatial-Aware Latent Initialization for Controllable Image Generation

• Authors: Wenqiang Sun, Teng Li, Zehong Lin, Jun Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16157
• Pdf URL: https://arxiv.org/pdf/2401.16157
• Copy Paste: [[2401.16157]] Spatial-Aware Latent Initialization for Controllable Image Generation(https://arxiv.org/abs/2401.16157)
• Keywords: diffusion
• Abstract: Recently, text-to-image diffusion models have demonstrated impressive ability to generate high-quality images conditioned on the textual input. However, these models struggle to accurately adhere to textual instructions regarding spatial layout information. While previous research has primarily focused on aligning cross-attention maps with layout conditions, they overlook the impact of the initialization noise on the layout guidance. To achieve better layout control, we propose leveraging a spatial-aware initialization noise during the denoising process. Specifically, we find that the inverted reference image with finite inversion steps contains valuable spatial awareness regarding the object's position, resulting in similar layouts in the generated images. Based on this observation, we develop an open-vocabulary framework to customize a spatial-aware initialization noise for each layout condition. Without modifying other modules except the initialization noise, our approach can be seamlessly integrated as a plug-and-play module within other training-free layout guidance frameworks. We evaluate our approach quantitatively and qualitatively on the available Stable Diffusion model and COCO dataset. Equipped with the spatial-aware latent initialization, our method significantly improves the effectiveness of layout guidance while preserving high-quality content.

Title: Diffutoon: High-Resolution Editable Toon Shading via Diffusion Models

• Authors: Zhongjie Duan, Chengyu Wang, Cen Chen, Weining Qian, Jun Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16224
• Pdf URL: https://arxiv.org/pdf/2401.16224
• Copy Paste: [[2401.16224]] Diffutoon: High-Resolution Editable Toon Shading via Diffusion Models(https://arxiv.org/abs/2401.16224)
• Keywords: diffusion
• Abstract: Toon shading is a type of non-photorealistic rendering task of animation. Its primary purpose is to render objects with a flat and stylized appearance. As diffusion models have ascended to the forefront of image synthesis methodologies, this paper delves into an innovative form of toon shading based on diffusion models, aiming to directly render photorealistic videos into anime styles. In video stylization, extant methods encounter persistent challenges, notably in maintaining consistency and achieving high visual quality. In this paper, we model the toon shading problem as four subproblems: stylization, consistency enhancement, structure guidance, and colorization. To address the challenges in video stylization, we propose an effective toon shading approach called \textit{Diffutoon}. Diffutoon is capable of rendering remarkably detailed, high-resolution, and extended-duration videos in anime style. It can also edit the content according to prompts via an additional branch. The efficacy of Diffutoon is evaluated through quantitive metrics and human evaluation. Notably, Diffutoon surpasses both open-source and closed-source baseline approaches in our experiments. Our work is accompanied by the release of both the source code and example videos on Github (Project page: https://ecnu-cilab.github.io/DiffutoonProjectPage/).

Title: Towards Red Teaming in Multimodal and Multilingual Translation

• Authors: Christophe Ropers, David Dale, Prangthip Hansanti, Gabriel Mejia Gonzalez, Ivan Evtimov, Corinne Wong, Christophe Touret, Kristina Pereyra, Seohyun Sonia Kim, Cristian Canton Ferrer, Pierre Andrews, Marta R. Costa-jussà
• Subjects: cs.CL, cs.CY
• Abstract URL: https://arxiv.org/abs/2401.16247
• Pdf URL: https://arxiv.org/pdf/2401.16247
• Copy Paste: [[2401.16247]] Towards Red Teaming in Multimodal and Multilingual Translation(https://arxiv.org/abs/2401.16247)
• Keywords: generative
• Abstract: Assessing performance in Natural Language Processing is becoming increasingly complex. One particular challenge is the potential for evaluation datasets to overlap with training data, either directly or indirectly, which can lead to skewed results and overestimation of model performance. As a consequence, human evaluation is gaining increasing interest as a means to assess the performance and reliability of models. One such method is the red teaming approach, which aims to generate edge cases where a model will produce critical errors. While this methodology is becoming standard practice for generative AI, its application to the realm of conditional AI remains largely unexplored. This paper presents the first study on human-based red teaming for Machine Translation (MT), marking a significant step towards understanding and improving the performance of translation models. We delve into both human-based red teaming and a study on automation, reporting lessons learned and providing recommendations for both translation models and red teaming drills. This pioneering work opens up new avenues for research and development in the field of MT.

Title: Cutup and Detect: Human Fall Detection on Cutup Untrimmed Videos Using a Large Foundational Video Understanding Model

• Authors: Till Grutschus, Ola Karrar, Emir Esenov, Ekta Vats
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2401.16280
• Pdf URL: https://arxiv.org/pdf/2401.16280
• Copy Paste: [[2401.16280]] Cutup and Detect: Human Fall Detection on Cutup Untrimmed Videos Using a Large Foundational Video Understanding Model(https://arxiv.org/abs/2401.16280)
• Keywords: foundation model
• Abstract: This work explores the performance of a large video understanding foundation model on the downstream task of human fall detection on untrimmed video and leverages a pretrained vision transformer for multi-class action detection, with classes: "Fall", "Lying" and "Other/Activities of daily living (ADL)". A method for temporal action localization that relies on a simple cutup of untrimmed videos is demonstrated. The methodology includes a preprocessing pipeline that converts datasets with timestamp action annotations into labeled datasets of short action clips. Simple and effective clip-sampling strategies are introduced. The effectiveness of the proposed method has been empirically evaluated on the publicly available High-Quality Fall Simulation Dataset (HQFSD). The experimental results validate the performance of the proposed pipeline. The results are promising for real-time application, and the falls are detected on video level with a state-of-the-art 0.96 F1 score on the HQFSD dataset under the given experimental settings. The source code will be made available on GitHub.

Title: A Survey on Visual Anomaly Detection: Challenge, Approach, and Prospect

• Authors: Yunkang Cao, Xiaohao Xu, Jiangning Zhang, Yuqi Cheng, Xiaonan Huang, Guansong Pang, Weiming Shen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2401.16402
• Pdf URL: https://arxiv.org/pdf/2401.16402
• Copy Paste: [[2401.16402]] A Survey on Visual Anomaly Detection: Challenge, Approach, and Prospect(https://arxiv.org/abs/2401.16402)
• Keywords: anomaly
• Abstract: Visual Anomaly Detection (VAD) endeavors to pinpoint deviations from the concept of normality in visual data, widely applied across diverse domains, e.g., industrial defect inspection, and medical lesion detection. This survey comprehensively examines recent advancements in VAD by identifying three primary challenges: 1) scarcity of training data, 2) diversity of visual modalities, and 3) complexity of hierarchical anomalies. Starting with a brief overview of the VAD background and its generic concept definitions, we progressively categorize, emphasize, and discuss the latest VAD progress from the perspective of sample number, data modality, and anomaly hierarchy. Through an in-depth analysis of the VAD field, we finally summarize future developments for VAD and conclude the key findings and contributions of this survey.