2024-03-01

Title: Exploring Privacy and Fairness Risks in Sharing Diffusion Models: An Adversarial Perspective

• Authors: Xinjian Luo, Yangfan Jiang, Fei Wei, Yuncheng Wu, Xiaokui Xiao, Beng Chin Ooi
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.18607
• Pdf URL: https://arxiv.org/pdf/2402.18607
• Copy Paste: [[2402.18607]] Exploring Privacy and Fairness Risks in Sharing Diffusion Models: An Adversarial Perspective(https://arxiv.org/abs/2402.18607)
• Keywords: diffusion, generative
• Abstract: Diffusion models have recently gained significant attention in both academia and industry due to their impressive generative performance in terms of both sampling quality and distribution coverage. Accordingly, proposals are made for sharing pre-trained diffusion models across different organizations, as a way of improving data utilization while enhancing privacy protection by avoiding sharing private data directly. However, the potential risks associated with such an approach have not been comprehensively examined. In this paper, we take an adversarial perspective to investigate the potential privacy and fairness risks associated with the sharing of diffusion models. Specifically, we investigate the circumstances in which one party (the sharer) trains a diffusion model using private data and provides another party (the receiver) black-box access to the pre-trained model for downstream tasks. We demonstrate that the sharer can execute fairness poisoning attacks to undermine the receiver's downstream models by manipulating the training data distribution of the diffusion model. Meanwhile, the receiver can perform property inference attacks to reveal the distribution of sensitive features in the sharer's dataset. Our experiments conducted on real-world datasets demonstrate remarkable attack performance on different types of diffusion models, which highlights the critical importance of robust data auditing and privacy protection protocols in pertinent applications.

Title: ICE-SEARCH: A Language Model-Driven Feature Selection Approach

• Authors: Tianze (Tom)Yang, Tianyi (Tim)Yang, Shaoshan Liu, Fuyuan Lvu, Xue Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.18609
• Pdf URL: https://arxiv.org/pdf/2402.18609
• Copy Paste: [[2402.18609]] ICE-SEARCH: A Language Model-Driven Feature Selection Approach(https://arxiv.org/abs/2402.18609)
• Keywords: in-context
• Abstract: This study unveils the In-Context Evolutionary Search (ICE-SEARCH) method, the first work that melds language models (LMs) with evolutionary algorithms for feature selection (FS) tasks and demonstrates its effectiveness in Medical Predictive Analytics (MPA) applications. ICE-SEARCH harnesses the crossover and mutation capabilities inherent in LMs within an evolutionary framework, significantly improving FS through the model's comprehensive world knowledge and its adaptability to a variety of roles. Our evaluation of this methodology spans three crucial MPA tasks: stroke, cardiovascular disease, and diabetes, where ICE-SEARCH outperforms traditional FS methods in pinpointing essential features for medical applications. ICE-SEARCH achieves State-of-the-Art (SOTA) performance in stroke prediction and diabetes prediction; the Decision-Randomized ICE-SEARCH ranks as SOTA in cardiovascular disease prediction. Our results not only demonstrate the efficacy of ICE-SEARCH in medical FS but also underscore the versatility, efficiency, and scalability of integrating LMs in FS tasks. The study emphasizes the critical role of incorporating domain-specific insights, illustrating ICE-SEARCH's robustness, generalizability, and swift convergence. This opens avenues for further research into comprehensive and intricate FS landscapes, marking a significant stride in the application of artificial intelligence in medical predictive analytics.

Title: Pre-training Differentially Private Models with Limited Public Data

• Authors: Zhiqi Bu, Xinwei Zhang, Mingyi Hong, Sheng Zha, George Karypis
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.18752
• Pdf URL: https://arxiv.org/pdf/2402.18752
• Copy Paste: [[2402.18752]] Pre-training Differentially Private Models with Limited Public Data(https://arxiv.org/abs/2402.18752)
• Keywords: foundation model
• Abstract: The superior performance of large foundation models relies on the use of massive amounts of high-quality data, which often contain sensitive, private and copyrighted material that requires formal protection. While differential privacy (DP) is a prominent method to gauge the degree of security provided to the models, its application is commonly limited to the model fine-tuning stage, due to the performance degradation when applying DP during the pre-training stage. Consequently, DP is yet not capable of protecting a substantial portion of the data used during the initial pre-training process. In this work, we first provide a theoretical understanding of the efficacy of DP training by analyzing the per-iteration loss improvement. We make a key observation that DP optimizers' performance degradation can be significantly mitigated by the use of limited public data, which leads to a novel DP continual pre-training strategy. Empirically, using only 10\% of public data, our strategy can achieve DP accuracy of 41.5\% on ImageNet-21k (with $\epsilon=8$), as well as non-DP accuracy of 55.7\% and and 60.0\% on downstream tasks Places365 and iNaturalist-2021, respectively, on par with state-of-the-art standard pre-training and substantially outperforming existing DP pre-trained models.

Title: Advancing Generative AI for Portuguese with Open Decoder Gervásio PT*

• Authors: Rodrigo Santos, João Silva, Luís Gomes, João Rodrigues, António Branco
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.18766
• Pdf URL: https://arxiv.org/pdf/2402.18766
• Copy Paste: [[2402.18766]] Advancing Generative AI for Portuguese with Open Decoder Gervásio PT*(https://arxiv.org/abs/2402.18766)
• Keywords: generative
• Abstract: To advance the neural decoding of Portuguese, in this paper we present a fully open Transformer-based, instruction-tuned decoder model that sets a new state of the art in this respect. To develop this decoder, which we named Gerv\'asio PT*, a strong LLaMA~2 7B model was used as a starting point, and its further improvement through additional training was done over language resources that include new instruction data sets of Portuguese prepared for this purpose, which are also contributed in this paper. All versions of Gerv\'asio are open source and distributed for free under an open license, including for either research or commercial usage, and can be run on consumer-grade hardware, thus seeking to contribute to the advancement of research and innovation in language technology for Portuguese.

Title: A Quantitative Evaluation of Score Distillation Sampling Based Text-to-3D

• Authors: Xiaohan Fei, Chethan Parameshwara, Jiawei Mo, Xiaolong Li, Ashwin Swaminathan, CJ Taylor, Paolo Favaro, Stefano Soatto
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18780
• Pdf URL: https://arxiv.org/pdf/2402.18780
• Copy Paste: [[2402.18780]] A Quantitative Evaluation of Score Distillation Sampling Based Text-to-3D(https://arxiv.org/abs/2402.18780)
• Keywords: diffusion, generative
• Abstract: The development of generative models that create 3D content from a text prompt has made considerable strides thanks to the use of the score distillation sampling (SDS) method on pre-trained diffusion models for image generation. However, the SDS method is also the source of several artifacts, such as the Janus problem, the misalignment between the text prompt and the generated 3D model, and 3D model inaccuracies. While existing methods heavily rely on the qualitative assessment of these artifacts through visual inspection of a limited set of samples, in this work we propose more objective quantitative evaluation metrics, which we cross-validate via human ratings, and show analysis of the failure cases of the SDS technique. We demonstrate the effectiveness of this analysis by designing a novel computationally efficient baseline model that achieves state-of-the-art performance on the proposed metrics while addressing all the above-mentioned artifacts.

Title: BlockEcho: Retaining Long-Range Dependencies for Imputing Block-Wise Missing Data

• Authors: Qiao Han, Mingqian Li, Yao Yang, Yiteng Zhai
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.18800
• Pdf URL: https://arxiv.org/pdf/2402.18800
• Copy Paste: [[2402.18800]] BlockEcho: Retaining Long-Range Dependencies for Imputing Block-Wise Missing Data(https://arxiv.org/abs/2402.18800)
• Keywords: generative
• Abstract: Block-wise missing data poses significant challenges in real-world data imputation tasks. Compared to scattered missing data, block-wise gaps exacerbate adverse effects on subsequent analytic and machine learning tasks, as the lack of local neighboring elements significantly reduces the interpolation capability and predictive power. However, this issue has not received adequate attention. Most SOTA matrix completion methods appeared less effective, primarily due to overreliance on neighboring elements for predictions. We systematically analyze the issue and propose a novel matrix completion method ``BlockEcho" for a more comprehensive solution. This method creatively integrates Matrix Factorization (MF) within Generative Adversarial Networks (GAN) to explicitly retain long-distance inter-element relationships in the original matrix. Besides, we incorporate an additional discriminator for GAN, comparing the generator's intermediate progress with pre-trained MF results to constrain high-order feature distributions. Subsequently, we evaluate BlockEcho on public datasets across three domains. Results demonstrate superior performance over both traditional and SOTA methods when imputing block-wise missing data, especially at higher missing rates. The advantage also holds for scattered missing data at high missing rates. We also contribute on the analyses in providing theoretical justification on the optimality and convergence of fusing MF and GAN for missing block data.

Title: Dual Operating Modes of In-Context Learning

• Authors: Ziqian Lin, Kangwook Lee
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.18819
• Pdf URL: https://arxiv.org/pdf/2402.18819
• Copy Paste: [[2402.18819]] Dual Operating Modes of In-Context Learning(https://arxiv.org/abs/2402.18819)
• Keywords: in-context
• Abstract: In-context learning (ICL) exhibits dual operating modes: task learning, i.e., acquiring a new skill from in-context samples, and task retrieval, i.e., locating and activating a relevant pretrained skill. Recent theoretical work investigates various mathematical models to analyze ICL, but existing models explain only one operating mode at a time. We introduce a probabilistic model, with which one can explain the dual operating modes of ICL simultaneously. Focusing on in-context learning of linear functions, we extend existing models for pretraining data by introducing multiple task groups and task-dependent input distributions. We then analyze the behavior of the optimally pretrained model under the squared loss, i.e., the MMSE estimator of the label given in-context examples. Regarding pretraining task distribution as prior and in-context examples as the observation, we derive the closed-form expression of the task posterior distribution. With the closed-form expression, we obtain a quantitative understanding of the two operating modes of ICL. Furthermore, we shed light on an unexplained phenomenon observed in practice: under certain settings, the ICL risk initially increases and then decreases with more in-context examples. Our model offers a plausible explanation for this "early ascent" phenomenon: a limited number of in-context samples may lead to the retrieval of an incorrect skill, thereby increasing the risk, which will eventually diminish as task learning takes effect with more in-context samples. We also theoretically analyze ICL with biased labels, e.g., zero-shot ICL, where in-context examples are assigned random labels. Lastly, we validate our findings and predictions via experiments involving Transformers and large language models.

Title: Extended Flow Matching: a Method of Conditional Generation with Generalized Continuity Equation

• Authors: Noboru Isobe, Masanori Koyama, Kohei Hayashi, Kenji Fukumizu
• Subjects: cs.LG, math.AP, math.FA, math.OC, math.PR
• Abstract URL: https://arxiv.org/abs/2402.18839
• Pdf URL: https://arxiv.org/pdf/2402.18839
• Copy Paste: [[2402.18839]] Extended Flow Matching: a Method of Conditional Generation with Generalized Continuity Equation(https://arxiv.org/abs/2402.18839)
• Keywords: diffusion, generative
• Abstract: The task of conditional generation is one of the most important applications of generative models, and numerous methods have been developed to date based on the celebrated diffusion models, with the guidance-based classifier-free method taking the lead. However, the theory of the guidance-based method not only requires the user to fine-tune the "guidance strength," but its target vector field does not necessarily correspond to the conditional distribution used in training. In this paper, we develop the theory of conditional generation based on Flow Matching, a current strong contender of diffusion methods. Motivated by the interpretation of a probability path as a distribution on path space, we establish a novel theory of flow-based generation of conditional distribution by employing the mathematical framework of generalized continuity equation instead of the continuity equation in flow matching. This theory naturally derives a method that aims to match the matrix field as opposed to the vector field. Our framework ensures the continuity of the generated conditional distribution through the existence of flow between conditional distributions. We will present our theory through experiments and mathematical results.

Title: ViewFusion: Towards Multi-View Consistency via Interpolated Denoising

• Authors: Xianghui Yang, Yan Zuo, Sameera Ramasinghe, Loris Bazzani, Gil Avraham, Anton van den Hengel
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18842
• Pdf URL: https://arxiv.org/pdf/2402.18842
• Copy Paste: [[2402.18842]] ViewFusion: Towards Multi-View Consistency via Interpolated Denoising(https://arxiv.org/abs/2402.18842)
• Keywords: diffusion
• Abstract: Novel-view synthesis through diffusion models has demonstrated remarkable potential for generating diverse and high-quality images. Yet, the independent process of image generation in these prevailing methods leads to challenges in maintaining multiple-view consistency. To address this, we introduce ViewFusion, a novel, training-free algorithm that can be seamlessly integrated into existing pre-trained diffusion models. Our approach adopts an auto-regressive method that implicitly leverages previously generated views as context for the next view generation, ensuring robust multi-view consistency during the novel-view generation process. Through a diffusion process that fuses known-view information via interpolated denoising, our framework successfully extends single-view conditioned models to work in multiple-view conditional settings without any additional fine-tuning. Extensive experimental results demonstrate the effectiveness of ViewFusion in generating consistent and detailed novel views.

Title: SwitchLight: Co-design of Physics-driven Architecture and Pre-training Framework for Human Portrait Relighting

• Authors: Hoon Kim, Minje Jang, Wonjun Yoon, Jisoo Lee, Donghyun Na, Sanghyun Woo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18848
• Pdf URL: https://arxiv.org/pdf/2402.18848
• Copy Paste: [[2402.18848]] SwitchLight: Co-design of Physics-driven Architecture and Pre-training Framework for Human Portrait Relighting(https://arxiv.org/abs/2402.18848)
• Keywords: self-supervised
• Abstract: We introduce a co-designed approach for human portrait relighting that combines a physics-guided architecture with a pre-training framework. Drawing on the Cook-Torrance reflectance model, we have meticulously configured the architecture design to precisely simulate light-surface interactions. Furthermore, to overcome the limitation of scarce high-quality lightstage data, we have developed a self-supervised pre-training strategy. This novel combination of accurate physical modeling and expanded training dataset establishes a new benchmark in relighting realism.

Title: Percept, Chat, and then Adapt: Multimodal Knowledge Transfer of Foundation Models for Open-World Video Recognition

• Authors: Boyu Chen, Siran Chen, Kunchang Li, Qinglin Xu, Yu Qiao, Yali Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18951
• Pdf URL: https://arxiv.org/pdf/2402.18951
• Copy Paste: [[2402.18951]] Percept, Chat, and then Adapt: Multimodal Knowledge Transfer of Foundation Models for Open-World Video Recognition(https://arxiv.org/abs/2402.18951)
• Keywords: foundation model
• Abstract: Open-world video recognition is challenging since traditional networks are not generalized well on complex environment variations. Alternatively, foundation models with rich knowledge have recently shown their generalization power. However, how to apply such knowledge has not been fully explored for open-world video recognition. To this end, we propose a generic knowledge transfer pipeline, which progressively exploits and integrates external multimodal knowledge from foundation models to boost open-world video recognition. We name it PCA, based on three stages of Percept, Chat, and Adapt. First, we perform Percept process to reduce the video domain gap and obtain external visual knowledge. Second, we generate rich linguistic semantics as external textual knowledge in Chat stage. Finally, we blend external multimodal knowledge in Adapt stage, by inserting multimodal knowledge adaptation modules into networks. We conduct extensive experiments on three challenging open-world video benchmarks, i.e., TinyVIRAT, ARID, and QV-Pipe. Our approach achieves state-of-the-art performance on all three datasets.

Title: Graph Generation via Spectral Diffusion

• Authors: Giorgia Minello, Alessandro Bicciato, Luca Rossi, Andrea Torsello, Luca Cosmo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.18974
• Pdf URL: https://arxiv.org/pdf/2402.18974
• Copy Paste: [[2402.18974]] Graph Generation via Spectral Diffusion(https://arxiv.org/abs/2402.18974)
• Keywords: diffusion, generative
• Abstract: In this paper, we present GRASP, a novel graph generative model based on 1) the spectral decomposition of the graph Laplacian matrix and 2) a diffusion process. Specifically, we propose to use a denoising model to sample eigenvectors and eigenvalues from which we can reconstruct the graph Laplacian and adjacency matrix. Our permutation invariant model can also handle node features by concatenating them to the eigenvectors of each node. Using the Laplacian spectrum allows us to naturally capture the structural characteristics of the graph and work directly in the node space while avoiding the quadratic complexity bottleneck that limits the applicability of other methods. This is achieved by truncating the spectrum, which as we show in our experiments results in a faster yet accurate generative process. An extensive set of experiments on both synthetic and real world graphs demonstrates the strengths of our model against state-of-the-art alternatives.

Title: Always be Pre-Training: Representation Learning for Network Intrusion Detection with GNNs

• Authors: Zhengyao Gu, Diego Troy Lopez, Lilas Alrahis, Ozgur Sinanoglu
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.18986
• Pdf URL: https://arxiv.org/pdf/2402.18986
• Copy Paste: [[2402.18986]] Always be Pre-Training: Representation Learning for Network Intrusion Detection with GNNs(https://arxiv.org/abs/2402.18986)
• Keywords: in-context
• Abstract: Graph neural network-based network intrusion detection systems have recently demonstrated state-of-the-art performance on benchmark datasets. Nevertheless, these methods suffer from a reliance on target encoding for data pre-processing, limiting widespread adoption due to the associated need for annotated labels--a cost-prohibitive requirement. In this work, we propose a solution involving in-context pre-training and the utilization of dense representations for categorical features to jointly overcome the label-dependency limitation. Our approach exhibits remarkable data efficiency, achieving over 98% of the performance of the supervised state-of-the-art with less than 4% labeled data on the NF-UQ-NIDS-V2 dataset.

Title: COFT-AD: COntrastive Fine-Tuning for Few-Shot Anomaly Detection

• Authors: Jingyi Liao, Xun Xu, Manh Cuong Nguyen, Adam Goodge, Chuan Sheng Foo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.18998
• Pdf URL: https://arxiv.org/pdf/2402.18998
• Copy Paste: [[2402.18998]] COFT-AD: COntrastive Fine-Tuning for Few-Shot Anomaly Detection(https://arxiv.org/abs/2402.18998)
• Keywords: anomaly
• Abstract: Existing approaches towards anomaly detection~(AD) often rely on a substantial amount of anomaly-free data to train representation and density models. However, large anomaly-free datasets may not always be available before the inference stage; in which case an anomaly detection model must be trained with only a handful of normal samples, a.k.a. few-shot anomaly detection (FSAD). In this paper, we propose a novel methodology to address the challenge of FSAD which incorporates two important techniques. Firstly, we employ a model pre-trained on a large source dataset to initialize model weights. Secondly, to ameliorate the covariate shift between source and target domains, we adopt contrastive training to fine-tune on the few-shot target domain data. To learn suitable representations for the downstream AD task, we additionally incorporate cross-instance positive pairs to encourage a tight cluster of the normal samples, and negative pairs for better separation between normal and synthesized negative samples. We evaluate few-shot anomaly detection on on 3 controlled AD tasks and 4 real-world AD tasks to demonstrate the effectiveness of the proposed method.

Title: Generating, Reconstructing, and Representing Discrete and Continuous Data: Generalized Diffusion with Learnable Encoding-Decoding

• Authors: Guangyi Liu, Yu Wang, Zeyu Feng, Qiyu Wu, Liping Tang, Yuan Gao, Zhen Li, Shuguang Cui, Julian McAuley, Eric P. Xing, Zichao Yang, Zhiting Hu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.19009
• Pdf URL: https://arxiv.org/pdf/2402.19009
• Copy Paste: [[2402.19009]] Generating, Reconstructing, and Representing Discrete and Continuous Data: Generalized Diffusion with Learnable Encoding-Decoding(https://arxiv.org/abs/2402.19009)
• Keywords: diffusion, generative
• Abstract: The vast applications of deep generative models are anchored in three core capabilities -- generating new instances, reconstructing inputs, and learning compact representations -- across various data types, such as discrete text/protein sequences and continuous images. Existing model families, like Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), autoregressive models, and diffusion models, generally excel in specific capabilities and data types but fall short in others. We introduce generalized diffusion with learnable encoder-decoder (DiLED), that seamlessly integrates the core capabilities for broad applicability and enhanced performance. DiLED generalizes the Gaussian noising-denoising in standard diffusion by introducing parameterized encoding-decoding. Crucially, DiLED is compatible with the well-established diffusion model objective and training recipes, allowing effective learning of the encoder-decoder parameters jointly with diffusion. By choosing appropriate encoder/decoder (e.g., large language models), DiLED naturally applies to different data types. Extensive experiments on text, proteins, and images demonstrate DiLED's flexibility to handle diverse data and tasks and its strong improvement over various existing models.

Title: Atmospheric Turbulence Removal with Video Sequence Deep Visual Priors

• Authors: P. Hill, N. Anantrasirichai, A. Achim, D.R. Bull
• Subjects: cs.CV, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.19041
• Pdf URL: https://arxiv.org/pdf/2402.19041
• Copy Paste: [[2402.19041]] Atmospheric Turbulence Removal with Video Sequence Deep Visual Priors(https://arxiv.org/abs/2402.19041)
• Keywords: self-supervised
• Abstract: Atmospheric turbulence poses a challenge for the interpretation and visual perception of visual imagery due to its distortion effects. Model-based approaches have been used to address this, but such methods often suffer from artefacts associated with moving content. Conversely, deep learning based methods are dependent on large and diverse datasets that may not effectively represent any specific content. In this paper, we address these problems with a self-supervised learning method that does not require ground truth. The proposed method is not dependent on any dataset outside of the single data sequence being processed but is also able to improve the quality of any input raw sequences or pre-processed sequences. Specifically, our method is based on an accelerated Deep Image Prior (DIP), but integrates temporal information using pixel shuffling and a temporal sliding window. This efficiently learns spatio-temporal priors leading to a system that effectively mitigates atmospheric turbulence distortions. The experiments show that our method improves visual quality results qualitatively and quantitatively.

Title: Theoretical Foundations of Deep Selective State-Space Models

• Authors: Nicola Muca Cirone, Antonio Orvieto, Benjamin Walker, Cristopher Salvi, Terry Lyons
• Subjects: cs.LG, math.DS
• Abstract URL: https://arxiv.org/abs/2402.19047
• Pdf URL: https://arxiv.org/pdf/2402.19047
• Copy Paste: [[2402.19047]] Theoretical Foundations of Deep Selective State-Space Models(https://arxiv.org/abs/2402.19047)
• Keywords: foundation model
• Abstract: Structured state-space models (SSMs) such as S4, stemming from the seminal work of Gu et al., are gaining popularity as effective approaches for modeling sequential data. Deep SSMs demonstrate outstanding performance across a diverse set of domains, at a reduced training and inference cost compared to attention-based transformers. Recent developments show that if the linear recurrence powering SSMs allows for multiplicative interactions between inputs and hidden states (e.g. GateLoop, Mamba, GLA), then the resulting architecture can surpass in both in accuracy and efficiency attention-powered foundation models trained on text, at scales of billion parameters. In this paper, we give theoretical grounding to this recent finding using tools from Rough Path Theory: we show that when random linear recurrences are equipped with simple input-controlled transitions (selectivity mechanism), then the hidden state is provably a low-dimensional projection of a powerful mathematical object called the signature of the input -- capturing non-linear interactions between tokens at distinct timescales. Our theory not only motivates the success of modern selective state-space models such as Mamba but also provides a solid framework to understand the expressive power of future SSM variants.

Title: VEnvision3D: A Synthetic Perception Dataset for 3D Multi-Task Model Research

• Authors: Jiahao Zhou, Chen Long, Yue Xie, Jialiang Wang, Boheng Li, Haiping Wang, Zhe Chen, Zhen Dong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.19059
• Pdf URL: https://arxiv.org/pdf/2402.19059
• Copy Paste: [[2402.19059]] VEnvision3D: A Synthetic Perception Dataset for 3D Multi-Task Model Research(https://arxiv.org/abs/2402.19059)
• Keywords: foundation model
• Abstract: Developing a unified multi-task foundation model has become a critical challenge in computer vision research. In the current field of 3D computer vision, most datasets solely focus on a relatively limited set of tasks, which complicates the concurrent training requirements of various downstream tasks. This makes the training of multi-objective networks difficult to proceed with, which further hinders the development of foundation models in the 3D vision field. In this paper, we introduce VEnvision3D, a large 3D synthetic perception dataset for multi-task learning, including depth completion, segmentation, upsampling, place recognition, and 3D reconstruction. Since the data for each task was collected in the same scenarios, tasks are inherently aligned in terms of the utilized data. Therefore, such a unique attribute can assist in exploring the potential for the multi-task model and even the foundation model without separate training methods. Several new benchmarks based on the characteristics of the proposed dataset were presented. Extensive studies were performed on end-to-end models, revealing new observations, challenges, and opportunities for future research. In addition, we designed a straightfoward multi-task network to uncover the ability that VEnvision3D can offer for the foundation model. Our dataset and code will be open-sourced upon acceptance.

Title: VideoMAC: Video Masked Autoencoders Meet ConvNets

• Authors: Gensheng Pei, Tao Chen, Xiruo Jiang, Huafeng Liu, Zeren Sun, Yazhou Yao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.19082
• Pdf URL: https://arxiv.org/pdf/2402.19082
• Copy Paste: [[2402.19082]] VideoMAC: Video Masked Autoencoders Meet ConvNets(https://arxiv.org/abs/2402.19082)
• Keywords: self-supervised
• Abstract: Recently, the advancement of self-supervised learning techniques, like masked autoencoders (MAE), has greatly influenced visual representation learning for images and videos. Nevertheless, it is worth noting that the predominant approaches in existing masked image / video modeling rely excessively on resource-intensive vision transformers (ViTs) as the feature encoder. In this paper, we propose a new approach termed as \textbf{VideoMAC}, which combines video masked autoencoders with resource-friendly ConvNets. Specifically, VideoMAC employs symmetric masking on randomly sampled pairs of video frames. To prevent the issue of mask pattern dissipation, we utilize ConvNets which are implemented with sparse convolutional operators as encoders. Simultaneously, we present a simple yet effective masked video modeling (MVM) approach, a dual encoder architecture comprising an online encoder and an exponential moving average target encoder, aimed to facilitate inter-frame reconstruction consistency in videos. Additionally, we demonstrate that VideoMAC, empowering classical (ResNet) / modern (ConvNeXt) convolutional encoders to harness the benefits of MVM, outperforms ViT-based approaches on downstream tasks, including video object segmentation (+\textbf{5.2\%} / \textbf{6.4\%} $\mathcal{J}\&\mathcal{F}$), body part propagation (+\textbf{6.3\%} / \textbf{3.1\%} mIoU), and human pose tracking (+\textbf{10.2\%} / \textbf{11.1\%} PCK@0.1).

Title: Leveraging Representations from Intermediate Encoder-blocks for Synthetic Image Detection

• Authors: Christos Koutlis, Symeon Papadopoulos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.19091
• Pdf URL: https://arxiv.org/pdf/2402.19091
• Copy Paste: [[2402.19091]] Leveraging Representations from Intermediate Encoder-blocks for Synthetic Image Detection(https://arxiv.org/abs/2402.19091)
• Keywords: foundation model
• Abstract: The recently developed and publicly available synthetic image generation methods and services make it possible to create extremely realistic imagery on demand, raising great risks for the integrity and safety of online information. State-of-the-art Synthetic Image Detection (SID) research has led to strong evidence on the advantages of feature extraction from foundation models. However, such extracted features mostly encapsulate high-level visual semantics instead of fine-grained details, which are more important for the SID task. On the contrary, shallow layers encode low-level visual information. In this work, we leverage the image representations extracted by intermediate Transformer blocks of CLIP's image-encoder via a lightweight network that maps them to a learnable forgery-aware vector space capable of generalizing exceptionally well. We also employ a trainable module to incorporate the importance of each Transformer block to the final prediction. Our method is compared against the state-of-the-art by evaluating it on 20 test datasets and exhibits an average +10.6% absolute performance improvement. Notably, the best performing models require just a single epoch for training (~8 minutes). Code available at https://github.com/mever-team/rine.

Title: TEncDM: Understanding the Properties of Diffusion Model in the Space of Language Model Encodings

• Authors: Alexander Shabalin, Viacheslav Meshchaninov, Tingir Badmaev, Dmitry Molchanov, Grigory Bartosh, Sergey Markov, Dmitry Vetrov
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.19097
• Pdf URL: https://arxiv.org/pdf/2402.19097
• Copy Paste: [[2402.19097]] TEncDM: Understanding the Properties of Diffusion Model in the Space of Language Model Encodings(https://arxiv.org/abs/2402.19097)
• Keywords: diffusion
• Abstract: Drawing inspiration from the success of diffusion models in various domains, numerous research papers proposed methods for adapting them to text data. Despite these efforts, none of them has managed to achieve the quality of the large language models. In this paper, we conduct a comprehensive analysis of key components of the text diffusion models and introduce a novel approach named Text Encoding Diffusion Model (TEncDM). Instead of the commonly used token embedding space, we train our model in the space of the language model encodings. Additionally, we propose to use a Transformer-based decoder that utilizes contextual information for text reconstruction. We also analyse self-conditioning and find that it increases the magnitude of the model outputs, allowing the reduction of the number of denoising steps at the inference stage. Evaluation of TEncDM on two downstream text generation tasks, QQP and XSum, demonstrates its superiority over existing non-autoregressive models.

Title: CollaFuse: Navigating Limited Resources and Privacy in Collaborative Generative AI

• Authors: Domenique Zipperling, Simeon Allmendinger, Lukas Struppek, Niklas Kühl
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.19105
• Pdf URL: https://arxiv.org/pdf/2402.19105
• Copy Paste: [[2402.19105]] CollaFuse: Navigating Limited Resources and Privacy in Collaborative Generative AI(https://arxiv.org/abs/2402.19105)
• Keywords: diffusion, generative
• Abstract: In the landscape of generative artificial intelligence, diffusion-based models present challenges for socio-technical systems in data requirements and privacy. Traditional approaches like federated learning distribute the learning process but strain individual clients, especially with constrained resources (e.g., edge devices). In response to these challenges, we introduce CollaFuse, a novel framework inspired by split learning. Tailored for efficient and collaborative use of denoising diffusion probabilistic models, CollaFuse enables shared server training and inference, alleviating client computational burdens. This is achieved by retaining data and computationally inexpensive GPU processes locally at each client while outsourcing the computationally expensive processes to the shared server. Demonstrated in a healthcare context, CollaFuse enhances privacy by highly reducing the need for sensitive information sharing. These capabilities hold the potential to impact various application areas, such as the design of edge computing solutions, healthcare research, or autonomous driving. In essence, our work advances distributed machine learning, shaping the future of collaborative GenAI networks.

Title: A SAM-guided Two-stream Lightweight Model for Anomaly Detection

• Authors: Chenghao Li, Lei Qi, Xin Geng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.19145
• Pdf URL: https://arxiv.org/pdf/2402.19145
• Copy Paste: [[2402.19145]] A SAM-guided Two-stream Lightweight Model for Anomaly Detection(https://arxiv.org/abs/2402.19145)
• Keywords: anomaly
• Abstract: In industrial anomaly detection, model efficiency and mobile-friendliness become the primary concerns in real-world applications. Simultaneously, the impressive generalization capabilities of Segment Anything (SAM) have garnered broad academic attention, making it an ideal choice for localizing unseen anomalies and diverse real-world patterns. In this paper, considering these two critical factors, we propose a SAM-guided Two-stream Lightweight Model for unsupervised anomaly detection (STLM) that not only aligns with the two practical application requirements but also harnesses the robust generalization capabilities of SAM. We employ two lightweight image encoders, i.e., our two-stream lightweight module, guided by SAM's knowledge. To be specific, one stream is trained to generate discriminative and general feature representations in both normal and anomalous regions, while the other stream reconstructs the same images without anomalies, which effectively enhances the differentiation of two-stream representations when facing anomalous regions. Furthermore, we employ a shared mask decoder and a feature aggregation module to generate anomaly maps. Our experiments conducted on MVTec AD benchmark show that STLM, with about 16M parameters and achieving an inference time in 20ms, competes effectively with state-of-the-art methods in terms of performance, 98.26% on pixel-level AUC and 94.92% on PRO. We further experiment on more difficult datasets, e.g., VisA and DAGM, to demonstrate the effectiveness and generalizability of STLM.

Title: Teaching Large Language Models an Unseen Language on the Fly

• Authors: Chen Zhang, Xiao Liu, Jiuheng Lin, Yansong Feng
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.19167
• Pdf URL: https://arxiv.org/pdf/2402.19167
• Copy Paste: [[2402.19167]] Teaching Large Language Models an Unseen Language on the Fly(https://arxiv.org/abs/2402.19167)
• Keywords: in-context
• Abstract: Existing large language models struggle to support numerous low-resource languages, particularly the extremely low-resource ones where there is minimal training data available for effective parameter updating. We thus investigate whether LLMs can learn a new language on the fly solely through prompting. To study this question, we collect a research suite for Zhuang, a language supported by no LLMs currently. We introduce \textsc{DiPMT++}, a framework for adapting LLMs to unseen languages by in-context learning. Using a dictionary and only 5K parallel sentences, \textsc{DiPMT++} significantly enhances the performance of GPT-4 from 0 to 16 BLEU for Chinese-to-Zhuang translation and achieves 32 BLEU for Zhuang-to-Chinese translation. Furthermore, we demonstrate the practical utility of this framework in aiding humans to translate completely unseen languages, which could contribute to the preservation of linguistic diversity.

Title: Disentangling representations of retinal images with generative models

• Authors: Sarah Müller, Lisa M. Koch, Hendrik P. A. Lensch, Philipp Berens
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.19186
• Pdf URL: https://arxiv.org/pdf/2402.19186
• Copy Paste: [[2402.19186]] Disentangling representations of retinal images with generative models(https://arxiv.org/abs/2402.19186)
• Keywords: generative
• Abstract: Retinal fundus images play a crucial role in the early detection of eye diseases and, using deep learning approaches, recent studies have even demonstrated their potential for detecting cardiovascular risk factors and neurological disorders. However, the impact of technical factors on these images can pose challenges for reliable AI applications in ophthalmology. For example, large fundus cohorts are often confounded by factors like camera type, image quality or illumination level, bearing the risk of learning shortcuts rather than the causal relationships behind the image generation process. Here, we introduce a novel population model for retinal fundus images that effectively disentangles patient attributes from camera effects, thus enabling controllable and highly realistic image generation. To achieve this, we propose a novel disentanglement loss based on distance correlation. Through qualitative and quantitative analyses, we demonstrate the effectiveness of this novel loss function in disentangling the learned subspaces. Our results show that our model provides a new perspective on the complex relationship between patient attributes and technical confounders in retinal fundus image generation.

Title: Memory-Augmented Generative Adversarial Transformers

• Authors: Stephan Raaijmakers, Roos Bakker, Anita Cremers, Roy de Kleijn, Tom Kouwenhoven, Tessa Verhoef
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.19218
• Pdf URL: https://arxiv.org/pdf/2402.19218
• Copy Paste: [[2402.19218]] Memory-Augmented Generative Adversarial Transformers(https://arxiv.org/abs/2402.19218)
• Keywords: generative
• Abstract: Conversational AI systems that rely on Large Language Models, like Transformers, have difficulty interweaving external data (like facts) with the language they generate. Vanilla Transformer architectures are not designed for answering factual questions with high accuracy. This paper investigates a possible route for addressing this problem. We propose to extend the standard Transformer architecture with an additional memory bank holding extra information (such as facts drawn from a knowledge base), and an extra attention layer for addressing this memory. We add this augmented memory to a Generative Adversarial Network-inspired Transformer architecture. This setup allows for implementing arbitrary felicity conditions on the generated language of the Transformer. We first demonstrate how this machinery can be deployed for handling factual questions in goal-oriented dialogues. Secondly, we demonstrate that our approach can be useful for applications like {\it style adaptation} as well: the adaptation of utterances according to certain stylistic (external) constraints, like social properties of human interlocutors in dialogues.

Title: CricaVPR: Cross-image Correlation-aware Representation Learning for Visual Place Recognition

• Authors: Feng Lu, Xiangyuan Lan, Lijun Zhang, Dongmei Jiang, Yaowei Wang, Chun Yuan
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.19231
• Pdf URL: https://arxiv.org/pdf/2402.19231
• Copy Paste: [[2402.19231]] CricaVPR: Cross-image Correlation-aware Representation Learning for Visual Place Recognition(https://arxiv.org/abs/2402.19231)
• Keywords: foundation model
• Abstract: Over the past decade, most methods in visual place recognition (VPR) have used neural networks to produce feature representations. These networks typically produce a global representation of a place image using only this image itself and neglect the cross-image variations (e.g. viewpoint and illumination), which limits their robustness in challenging scenes. In this paper, we propose a robust global representation method with cross-image correlation awareness for VPR, named CricaVPR. Our method uses the self-attention mechanism to correlate multiple images within a batch. These images can be taken in the same place with different conditions or viewpoints, or even captured from different places. Therefore, our method can utilize the cross-image variations as a cue to guide the representation learning, which ensures more robust features are produced. To further facilitate the robustness, we propose a multi-scale convolution-enhanced adaptation method to adapt pre-trained visual foundation models to the VPR task, which introduces the multi-scale local information to further enhance the cross-image correlation-aware representation. Experimental results show that our method outperforms state-of-the-art methods by a large margin with significantly less training time. Our method achieves 94.5% R@1 on Pitts30k using 512-dim global features. The code is released at https://github.com/Lu-Feng/CricaVPR.

Title: StiefelGen: A Simple, Model Agnostic Approach for Time Series Data Augmentation over Riemannian Manifolds

• Authors: Prasad Cheema, Mahito Sugiyama
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.19287
• Pdf URL: https://arxiv.org/pdf/2402.19287
• Copy Paste: [[2402.19287]] StiefelGen: A Simple, Model Agnostic Approach for Time Series Data Augmentation over Riemannian Manifolds(https://arxiv.org/abs/2402.19287)
• Keywords: generative
• Abstract: Data augmentation is an area of research which has seen active development in many machine learning fields, such as in image-based learning models, reinforcement learning for self driving vehicles, and general noise injection for point cloud data. However, convincing methods for general time series data augmentation still leaves much to be desired, especially since the methods developed for these models do not readily cross-over. Three common approaches for time series data augmentation include: (i) Constructing a physics-based model and then imbuing uncertainty over the coefficient space (for example), (ii) Adding noise to the observed data set(s), and, (iii) Having access to ample amounts of time series data sets from which a robust generative neural network model can be trained. However, for many practical problems that work with time series data in the industry: (i) One usually does not have access to a robust physical model, (ii) The addition of noise can in of itself require large or difficult assumptions (for example, what probability distribution should be used? Or, how large should the noise variance be?), and, (iii) In practice, it can be difficult to source a large representative time series data base with which to train the neural network model for the underlying problem. In this paper, we propose a methodology which attempts to simultaneously tackle all three of these previous limitations to a large extent. The method relies upon the well-studied matrix differential geometry of the Stiefel manifold, as it proposes a simple way in which time series signals can placed on, and then smoothly perturbed over the manifold. We attempt to clarify how this method works by showcasing several potential use cases which in particular work to take advantage of the unique properties of this underlying manifold.

Title: Anomaly Detection in Offshore Wind Turbine Structures using Hierarchical Bayesian Modelling

• Authors: S. M. Smith, A. J. Hughes, T. A. Dardeno, L. A. Bull, N. Dervilis, K. Worden
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.19295
• Pdf URL: https://arxiv.org/pdf/2402.19295
• Copy Paste: [[2402.19295]] Anomaly Detection in Offshore Wind Turbine Structures using Hierarchical Bayesian Modelling(https://arxiv.org/abs/2402.19295)
• Keywords: anomaly
• Abstract: Population-based structural health monitoring (PBSHM), aims to share information between members of a population. An offshore wind (OW) farm could be considered as a population of nominally-identical wind-turbine structures. However, benign variations exist among members, such as geometry, sea-bed conditions and temperature differences. These factors could influence structural properties and therefore the dynamic response, making it more difficult to detect structural problems via traditional SHM techniques. This paper explores the use of a hierarchical Bayesian model to infer expected soil stiffness distributions at both population and local levels, as a basis to perform anomaly detection, in the form of scour, for new and existing turbines. To do this, observations of natural frequency will be generated as though they are from a small population of wind turbines. Differences between individual observations will be introduced by postulating distributions over the soil stiffness and measurement noise, as well as reducing soil depth (to represent scour), in the case of anomaly detection.

Title: DiffAssemble: A Unified Graph-Diffusion Model for 2D and 3D Reassembly

• Authors: Gianluca Scarpellini, Stefano Fiorini, Francesco Giuliari, Pietro Morerio, Alessio Del Bue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.19302
• Pdf URL: https://arxiv.org/pdf/2402.19302
• Copy Paste: [[2402.19302]] DiffAssemble: A Unified Graph-Diffusion Model for 2D and 3D Reassembly(https://arxiv.org/abs/2402.19302)
• Keywords: diffusion
• Abstract: Reassembly tasks play a fundamental role in many fields and multiple approaches exist to solve specific reassembly problems. In this context, we posit that a general unified model can effectively address them all, irrespective of the input data type (images, 3D, etc.). We introduce DiffAssemble, a Graph Neural Network (GNN)-based architecture that learns to solve reassembly tasks using a diffusion model formulation. Our method treats the elements of a set, whether pieces of 2D patch or 3D object fragments, as nodes of a spatial graph. Training is performed by introducing noise into the position and rotation of the elements and iteratively denoising them to reconstruct the coherent initial pose. DiffAssemble achieves state-of-the-art (SOTA) results in most 2D and 3D reassembly tasks and is the first learning-based approach that solves 2D puzzles for both rotation and translation. Furthermore, we highlight its remarkable reduction in run-time, performing 11 times faster than the quickest optimization-based method for puzzle solving. Code available at https://github.com/IIT-PAVIS/DiffAssemble

Title: A Novel Approach to Industrial Defect Generation through Blended Latent Diffusion Model with Online Adaptation

• Authors: Hanxi Li, Zhengxun Zhang, Hao Chen, Lin Wu, Bo Li, Deyin Liu, Mingwen Wang
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2402.19330
• Pdf URL: https://arxiv.org/pdf/2402.19330
• Copy Paste: [[2402.19330]] A Novel Approach to Industrial Defect Generation through Blended Latent Diffusion Model with Online Adaptation(https://arxiv.org/abs/2402.19330)
• Keywords: diffusion, anomaly
• Abstract: Effectively addressing the challenge of industrial Anomaly Detection (AD) necessitates an ample supply of defective samples, a constraint often hindered by their scarcity in industrial contexts. This paper introduces a novel algorithm designed to augment defective samples, thereby enhancing AD performance. The proposed method tailors the blended latent diffusion model for defect sample generation, employing a diffusion model to generate defective samples in the latent space. A feature editing process, controlled by a "trimap" mask and text prompts, refines the generated samples. The image generation inference process is structured into three stages: a free diffusion stage, an editing diffusion stage, and an online decoder adaptation stage. This sophisticated inference strategy yields high-quality synthetic defective samples with diverse pattern variations, leading to significantly improved AD accuracies based on the augmented training set. Specifically, on the widely recognized MVTec AD dataset, the proposed method elevates the state-of-the-art (SOTA) performance of AD with augmented data by 1.5%, 1.9%, and 3.1% for AD metrics AP, IAP, and IAP90, respectively. The implementation code of this work can be found at the GitHub repository https://github.com/GrandpaXun242/AdaBLDM.git

Title: Compact Speech Translation Models via Discrete Speech Units Pretraining

• Authors: Tsz Kin Lam, Alexandra Birch, Barry Haddow
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2402.19333
• Pdf URL: https://arxiv.org/pdf/2402.19333
• Copy Paste: [[2402.19333]] Compact Speech Translation Models via Discrete Speech Units Pretraining(https://arxiv.org/abs/2402.19333)
• Keywords: self-supervised
• Abstract: Using Self-Supervised Learning (SSL) as model initialization is now common to obtain strong results in Speech Translation (ST). However, they also impose a large memory footprint, hindering on-device deployment. In this paper, we leverage the SSL models by pretraining smaller models on their Discrete Speech Units (DSU). We pretrain encoder-decoder models on 1) Filterbank-to-DSU and 2) DSU-to-Translation data, and take the encoder from 1) and the decoder from 2) to initialise a new model, finetuning this on limited speech-translation data. The final model becomes compact by using the DSU pretraining to distil the knowledge of the SSL model. Our method has several benefits over using DSU as model inputs, such as shorter inference pipeline and robustness over (DSU) tokenization. In contrast to ASR pretraining, it does not require transcripts, making it applicable to low-resource settings. Evaluation on CoVoST-2 X-En shows that our method is >$0.5$ BLEU better than a ST model that directly finetune the SSL model, given only half the model size, and on a par with ASR pretraining.

Title: Structure Preserving Diffusion Models

• Authors: Haoye Lu, Spencer Szabados, Yaoliang Yu
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.19369
• Pdf URL: https://arxiv.org/pdf/2402.19369
• Copy Paste: [[2402.19369]] Structure Preserving Diffusion Models(https://arxiv.org/abs/2402.19369)
• Keywords: diffusion
• Abstract: Diffusion models have become the leading distribution-learning method in recent years. Herein, we introduce structure-preserving diffusion processes, a family of diffusion processes for learning distributions that possess additional structure, such as group symmetries, by developing theoretical conditions under which the diffusion transition steps preserve said symmetry. While also enabling equivariant data sampling trajectories, we exemplify these results by developing a collection of different symmetry equivariant diffusion models capable of learning distributions that are inherently symmetric. Empirical studies, over both synthetic and real-world datasets, are used to validate the developed models adhere to the proposed theory and are capable of achieving improved performance over existing methods in terms of sample equality. We also show how the proposed models can be used to achieve theoretically guaranteed equivariant image noise reduction without prior knowledge of the image orientation.

Title: Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality

• Authors: Siyu Chen, Heejune Sheen, Tianhao Wang, Zhuoran Yang
• Subjects: cs.LG, cs.AI, math.OC, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.19442
• Pdf URL: https://arxiv.org/pdf/2402.19442
• Copy Paste: [[2402.19442]] Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality(https://arxiv.org/abs/2402.19442)
• Keywords: in-context
• Abstract: We study the dynamics of gradient flow for training a multi-head softmax attention model for in-context learning of multi-task linear regression. We establish the global convergence of gradient flow under suitable choices of initialization. In addition, we prove that an interesting "task allocation" phenomenon emerges during the gradient flow dynamics, where each attention head focuses on solving a single task of the multi-task model. Specifically, we prove that the gradient flow dynamics can be split into three phases -- a warm-up phase where the loss decreases rather slowly and the attention heads gradually build up their inclination towards individual tasks, an emergence phase where each head selects a single task and the loss rapidly decreases, and a convergence phase where the attention parameters converge to a limit. Furthermore, we prove the optimality of gradient flow in the sense that the limiting model learned by gradient flow is on par with the best possible multi-head softmax attention model up to a constant factor. Our analysis also delineates a strict separation in terms of the prediction accuracy of ICL between single-head and multi-head attention models. The key technique for our convergence analysis is to map the gradient flow dynamics in the parameter space to a set of ordinary differential equations in the spectral domain, where the relative magnitudes of the semi-singular values of the attention weights determines task allocation. To our best knowledge, our work provides the first convergence result for the multi-head softmax attention model.

Title: Retrieval-Augmented Generation for AI-Generated Content: A Survey

• Authors: Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang, Wentao Zhang, Bin Cui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.19473
• Pdf URL: https://arxiv.org/pdf/2402.19473
• Copy Paste: [[2402.19473]] Retrieval-Augmented Generation for AI-Generated Content: A Survey(https://arxiv.org/abs/2402.19473)
• Keywords: foundation model
• Abstract: The development of Artificial Intelligence Generated Content (AIGC) has been facilitated by advancements in model algorithms, scalable foundation model architectures, and the availability of ample high-quality datasets. While AIGC has achieved remarkable performance, it still faces challenges, such as the difficulty of maintaining up-to-date and long-tail knowledge, the risk of data leakage, and the high costs associated with training and inference. Retrieval-Augmented Generation (RAG) has recently emerged as a paradigm to address such challenges. In particular, RAG introduces the information retrieval process, which enhances AIGC results by retrieving relevant objects from available data stores, leading to greater accuracy and robustness. In this paper, we comprehensively review existing efforts that integrate RAG technique into AIGC scenarios. We first classify RAG foundations according to how the retriever augments the generator. We distill the fundamental abstractions of the augmentation methodologies for various retrievers and generators. This unified perspective encompasses all RAG scenarios, illuminating advancements and pivotal technologies that help with potential future progress. We also summarize additional enhancements methods for RAG, facilitating effective engineering and implementation of RAG systems. Then from another view, we survey on practical applications of RAG across different modalities and tasks, offering valuable references for researchers and practitioners. Furthermore, we introduce the benchmarks for RAG, discuss the limitations of current RAG systems, and suggest potential directions for future research. Project: https://github.com/hymie122/RAG-Survey

Title: DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models

• Authors: Muyang Li, Tianle Cai, Jiaxin Cao, Qinsheng Zhang, Han Cai, Junjie Bai, Yangqing Jia, Ming-Yu Liu, Kai Li, Song Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.19481
• Pdf URL: https://arxiv.org/pdf/2402.19481
• Copy Paste: [[2402.19481]] DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models(https://arxiv.org/abs/2402.19481)
• Keywords: diffusion
• Abstract: Diffusion models have achieved great success in synthesizing high-quality images. However, generating high-resolution images with diffusion models is still challenging due to the enormous computational costs, resulting in a prohibitive latency for interactive applications. In this paper, we propose DistriFusion to tackle this problem by leveraging parallelism across multiple GPUs. Our method splits the model input into multiple patches and assigns each patch to a GPU. However, na\"{\i}vely implementing such an algorithm breaks the interaction between patches and loses fidelity, while incorporating such an interaction will incur tremendous communication overhead. To overcome this dilemma, we observe the high similarity between the input from adjacent diffusion steps and propose displaced patch parallelism, which takes advantage of the sequential nature of the diffusion process by reusing the pre-computed feature maps from the previous timestep to provide context for the current step. Therefore, our method supports asynchronous communication, which can be pipelined by computation. Extensive experiments show that our method can be applied to recent Stable Diffusion XL with no quality degradation and achieve up to a 6.1$\times$ speedup on eight NVIDIA A100s compared to one. Our code is publicly available at https://github.com/mit-han-lab/distrifuser.