2024-02-08

Title: CasCast: Skillful High-resolution Precipitation Nowcasting via Cascaded Modelling

• Authors: Junchao Gong, Lei Bai, Peng Ye, Wanghan Xu, Na Liu, Jianhua Dai, Xiaokang Yang, Wanli Ouyang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.04290
• Pdf URL: https://arxiv.org/pdf/2402.04290
• Copy Paste: [[2402.04290]] CasCast: Skillful High-resolution Precipitation Nowcasting via Cascaded Modelling(https://arxiv.org/abs/2402.04290)
• Keywords: diffusion
• Abstract: Precipitation nowcasting based on radar data plays a crucial role in extreme weather prediction and has broad implications for disaster management. Despite progresses have been made based on deep learning, two key challenges of precipitation nowcasting are not well-solved: (i) the modeling of complex precipitation system evolutions with different scales, and (ii) accurate forecasts for extreme precipitation. In this work, we propose CasCast, a cascaded framework composed of a deterministic and a probabilistic part to decouple the predictions for mesoscale precipitation distributions and small-scale patterns. Then, we explore training the cascaded framework at the high resolution and conducting the probabilistic modeling in a low dimensional latent space with a frame-wise-guided diffusion transformer for enhancing the optimization of extreme events while reducing computational costs. Extensive experiments on three benchmark radar precipitation datasets show that CasCast achieves competitive performance. Especially, CasCast significantly surpasses the baseline (up to +91.8%) for regional extreme-precipitation nowcasting.

Title: AdaFlow: Imitation Learning with Variance-Adaptive Flow-Based Policies

• Authors: Xixi Hu, Bo Liu, Xingchao Liu, Qiang Liu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.04292
• Pdf URL: https://arxiv.org/pdf/2402.04292
• Copy Paste: [[2402.04292]] AdaFlow: Imitation Learning with Variance-Adaptive Flow-Based Policies(https://arxiv.org/abs/2402.04292)
• Keywords: diffusion, generative
• Abstract: Diffusion-based imitation learning improves Behavioral Cloning (BC) on multi-modal decision-making, but comes at the cost of significantly slower inference due to the recursion in the diffusion process. It urges us to design efficient policy generators while keeping the ability to generate diverse actions. To address this challenge, we propose AdaFlow, an imitation learning framework based on flow-based generative modeling. AdaFlow represents the policy with state-conditioned ordinary differential equations (ODEs), which are known as probability flows. We reveal an intriguing connection between the conditional variance of their training loss and the discretization error of the ODEs. With this insight, we propose a variance-adaptive ODE solver that can adjust its step size in the inference stage, making AdaFlow an adaptive decision-maker, offering rapid inference without sacrificing diversity. Interestingly, it automatically reduces to a one-step generator when the action distribution is uni-modal. Our comprehensive empirical evaluation shows that AdaFlow achieves high performance across all dimensions, including success rate, behavioral diversity, and inference speed. The code is available at https://github.com/hxixixh/AdaFlow

Title: ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation

• Authors: Weiming Ren, Harry Yang, Ge Zhang, Cong Wei, Xinrun Du, Stephen Huang, Wenhu Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04324
• Pdf URL: https://arxiv.org/pdf/2402.04324
• Copy Paste: [[2402.04324]] ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation(https://arxiv.org/abs/2402.04324)
• Keywords: diffusion
• Abstract: Image-to-video (I2V) generation aims to use the initial frame (alongside a text prompt) to create a video sequence. A grand challenge in I2V generation is to maintain visual consistency throughout the video: existing methods often struggle to preserve the integrity of the subject, background, and style from the first frame, as well as ensure a fluid and logical progression within the video narrative. To mitigate these issues, we propose ConsistI2V, a diffusion-based method to enhance visual consistency for I2V generation. Specifically, we introduce (1) spatiotemporal attention over the first frame to maintain spatial and motion consistency, (2) noise initialization from the low-frequency band of the first frame to enhance layout consistency. These two approaches enable ConsistI2V to generate highly consistent videos. We also extend the proposed approaches to show their potential to improve consistency in auto-regressive long video generation and camera motion control. To verify the effectiveness of our method, we propose I2V-Bench, a comprehensive evaluation benchmark for I2V generation. Our automatic and human evaluation results demonstrate the superiority of ConsistI2V over existing methods.

Title: Scaling laws for learning with real and surrogate data

• Authors: Ayush Jain, Andrea Montanari, Eren Sasoglu
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.04376
• Pdf URL: https://arxiv.org/pdf/2402.04376
• Copy Paste: [[2402.04376]] Scaling laws for learning with real and surrogate data(https://arxiv.org/abs/2402.04376)
• Keywords: generative
• Abstract: Collecting large quantities of high-quality data is often prohibitively expensive or impractical, and a crucial bottleneck in machine learning. One may instead augment a small set of $n$ data points from the target distribution with data from more accessible sources like public datasets, data collected under different circumstances, or synthesized by generative models. Blurring distinctions, we refer to such data as `surrogate data'. We define a simple scheme for integrating surrogate data into training and use both theoretical models and empirical studies to explore its behavior. Our main findings are: $(i)$ Integrating surrogate data can significantly reduce the test error on the original distribution; $(ii)$ In order to reap this benefit, it is crucial to use optimally weighted empirical risk minimization; $(iii)$ The test error of models trained on mixtures of real and surrogate data is well described by a scaling law. This can be used to predict the optimal weighting and the gain from surrogate data.

Title: Fine-Tuned Language Models Generate Stable Inorganic Materials as Text

• Authors: Nate Gruver, Anuroop Sriram, Andrea Madotto, Andrew Gordon Wilson, C. Lawrence Zitnick, Zachary Ulissi
• Subjects: cs.LG, cond-mat.mtrl-sci
• Abstract URL: https://arxiv.org/abs/2402.04379
• Pdf URL: https://arxiv.org/pdf/2402.04379
• Copy Paste: [[2402.04379]] Fine-Tuned Language Models Generate Stable Inorganic Materials as Text(https://arxiv.org/abs/2402.04379)
• Keywords: diffusion
• Abstract: We propose fine-tuning large language models for generation of stable materials. While unorthodox, fine-tuning large language models on text-encoded atomistic data is simple to implement yet reliable, with around 90% of sampled structures obeying physical constraints on atom positions and charges. Using energy above hull calculations from both learned ML potentials and gold-standard DFT calculations, we show that our strongest model (fine-tuned LLaMA-2 70B) can generate materials predicted to be metastable at about twice the rate (49% vs 28%) of CDVAE, a competing diffusion model. Because of text prompting's inherent flexibility, our models can simultaneously be used for unconditional generation of stable material, infilling of partial structures and text-conditional generation. Finally, we show that language models' ability to capture key symmetries of crystal structures improves with model scale, suggesting that the biases of pretrained LLMs are surprisingly well-suited for atomistic data.

Title: FairWire: Fair Graph Generation

• Authors: O. Deniz Kose, Yanning Shen
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.04383
• Pdf URL: https://arxiv.org/pdf/2402.04383
• Copy Paste: [[2402.04383]] FairWire: Fair Graph Generation(https://arxiv.org/abs/2402.04383)
• Keywords: generative
• Abstract: Machine learning over graphs has recently attracted growing attention due to its ability to analyze and learn complex relations within critical interconnected systems. However, the disparate impact that is amplified by the use of biased graph structures in these algorithms has raised significant concerns for the deployment of them in real-world decision systems. In addition, while synthetic graph generation has become pivotal for privacy and scalability considerations, the impact of generative learning algorithms on the structural bias has not yet been investigated. Motivated by this, this work focuses on the analysis and mitigation of structural bias for both real and synthetic graphs. Specifically, we first theoretically analyze the sources of structural bias that result in disparity for the predictions of dyadic relations. To alleviate the identified bias factors, we design a novel fairness regularizer that offers a versatile use. Faced with the bias amplification in graph generation models that is brought to light in this work, we further propose a fair graph generation framework, FairWire, by leveraging our fair regularizer design in a generative model. Experimental results on real-world networks validate that the proposed tools herein deliver effective structural bias mitigation for both real and synthetic graphs.

Title: Denoising Diffusion Probabilistic Models in Six Simple Steps

• Authors: Richard E. Turner, Cristiana-Diana Diaconu, Stratis Markou, Aliaksandra Shysheya, Andrew Y. K. Foong, Bruno Mlodozeniec
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.04384
• Pdf URL: https://arxiv.org/pdf/2402.04384
• Copy Paste: [[2402.04384]] Denoising Diffusion Probabilistic Models in Six Simple Steps(https://arxiv.org/abs/2402.04384)
• Keywords: diffusion, generative
• Abstract: Denoising Diffusion Probabilistic Models (DDPMs) are a very popular class of deep generative model that have been successfully applied to a diverse range of problems including image and video generation, protein and material synthesis, weather forecasting, and neural surrogates of partial differential equations. Despite their ubiquity it is hard to find an introduction to DDPMs which is simple, comprehensive, clean and clear. The compact explanations necessary in research papers are not able to elucidate all of the different design steps taken to formulate the DDPM and the rationale of the steps that are presented is often omitted to save space. Moreover, the expositions are typically presented from the variational lower bound perspective which is unnecessary and arguably harmful as it obfuscates why the method is working and suggests generalisations that do not perform well in practice. On the other hand, perspectives that take the continuous time-limit are beautiful and general, but they have a high barrier-to-entry as they require background knowledge of stochastic differential equations and probability flow. In this note, we distill down the formulation of the DDPM into six simple steps each of which comes with a clear rationale. We assume that the reader is familiar with fundamental topics in machine learning including basic probabilistic modelling, Gaussian distributions, maximum likelihood estimation, and deep learning.

Title: CEHR-GPT: Generating Electronic Health Records with Chronological Patient Timelines

• Authors: Chao Pang, Xinzhuo Jiang, Nishanth Parameshwar Pavinkurve, Krishna S. Kalluri, Elise L. Minto, Jason Patterson, Linying Zhang, George Hripcsak, Noémie Elhadad, Karthik Natarajan
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.04400
• Pdf URL: https://arxiv.org/pdf/2402.04400
• Copy Paste: [[2402.04400]] CEHR-GPT: Generating Electronic Health Records with Chronological Patient Timelines(https://arxiv.org/abs/2402.04400)
• Keywords: generative
• Abstract: Synthetic Electronic Health Records (EHR) have emerged as a pivotal tool in advancing healthcare applications and machine learning models, particularly for researchers without direct access to healthcare data. Although existing methods, like rule-based approaches and generative adversarial networks (GANs), generate synthetic data that resembles real-world EHR data, these methods often use a tabular format, disregarding temporal dependencies in patient histories and limiting data replication. Recently, there has been a growing interest in leveraging Generative Pre-trained Transformers (GPT) for EHR data. This enables applications like disease progression analysis, population estimation, counterfactual reasoning, and synthetic data generation. In this work, we focus on synthetic data generation and demonstrate the capability of training a GPT model using a particular patient representation derived from CEHR-BERT, enabling us to generate patient sequences that can be seamlessly converted to the Observational Medical Outcomes Partnership (OMOP) data format.

Title: Detection Transformer for Teeth Detection, Segmentation, and Numbering in Oral Rare Diseases: Focus on Data Augmentation and Inpainting Techniques

• Authors: Hocine Kadi, Théo Sourget, Marzena Kawczynski, Sara Bendjama, Bruno Grollemund, Agnès Bloch-Zupan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04408
• Pdf URL: https://arxiv.org/pdf/2402.04408
• Copy Paste: [[2402.04408]] Detection Transformer for Teeth Detection, Segmentation, and Numbering in Oral Rare Diseases: Focus on Data Augmentation and Inpainting Techniques(https://arxiv.org/abs/2402.04408)
• Keywords: diffusion
• Abstract: In this work, we focused on deep learning image processing in the context of oral rare diseases, which pose challenges due to limited data availability. A crucial step involves teeth detection, segmentation and numbering in panoramic radiographs. To this end, we used a dataset consisting of 156 panoramic radiographs from individuals with rare oral diseases and labeled by experts. We trained the Detection Transformer (DETR) neural network for teeth detection, segmentation, and numbering the 52 teeth classes. In addition, we used data augmentation techniques, including geometric transformations. Finally, we generated new panoramic images using inpainting techniques with stable diffusion, by removing teeth from a panoramic radiograph and integrating teeth into it. The results showed a mAP exceeding 0,69 for DETR without data augmentation. The mAP was improved to 0,82 when data augmentation techniques are used. Furthermore, we observed promising performances when using new panoramic radiographs generated with inpainting technique, with mAP of 0,76.

Title: PreGIP: Watermarking the Pretraining of Graph Neural Networks for Deep Intellectual Property Protection

• Authors: Enyan Dai, Minhua Lin, Suhang Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.04435
• Pdf URL: https://arxiv.org/pdf/2402.04435
• Copy Paste: [[2402.04435]] PreGIP: Watermarking the Pretraining of Graph Neural Networks for Deep Intellectual Property Protection(https://arxiv.org/abs/2402.04435)
• Keywords: self-supervised
• Abstract: Pretraining on Graph Neural Networks (GNNs) has shown great power in facilitating various downstream tasks. As pretraining generally requires huge amount of data and computational resources, the pretrained GNNs are high-value Intellectual Properties (IP) of the legitimate owner. However, adversaries may illegally copy and deploy the pretrained GNN models for their downstream tasks. Though initial efforts have been made to watermark GNN classifiers for IP protection, these methods require the target classification task for watermarking, and thus are not applicable to self-supervised pretraining of GNN models. Hence, in this work, we propose a novel framework named PreGIP to watermark the pretraining of GNN encoder for IP protection while maintain the high-quality of the embedding space. PreGIP incorporates a task-free watermarking loss to watermark the embedding space of pretrained GNN encoder. A finetuning-resistant watermark injection is further deployed. Theoretical analysis and extensive experiments show the effectiveness of {\method} in IP protection and maintaining high-performance for downstream tasks.

Title: IoT Network Traffic Analysis with Deep Learning

• Authors: Mei Liu, Leon Yang
• Subjects: cs.LG, cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2402.04469
• Pdf URL: https://arxiv.org/pdf/2402.04469
• Copy Paste: [[2402.04469]] IoT Network Traffic Analysis with Deep Learning(https://arxiv.org/abs/2402.04469)
• Keywords: anomaly
• Abstract: As IoT networks become more complex and generate massive amounts of dynamic data, it is difficult to monitor and detect anomalies using traditional statistical methods and machine learning methods. Deep learning algorithms can process and learn from large amounts of data and can also be trained using unsupervised learning techniques, meaning they don't require labelled data to detect anomalies. This makes it possible to detect new and unknown anomalies that may not have been detected before. Also, deep learning algorithms can be automated and highly scalable; thereby, they can run continuously in the backend and make it achievable to monitor large IoT networks instantly. In this work, we conduct a literature review on the most recent works using deep learning techniques and implement a model using ensemble techniques on the KDD Cup 99 dataset. The experimental results showcase the impressive performance of our deep anomaly detection model, achieving an accuracy of over 98\%.

Title: Text2Street: Controllable Text-to-image Generation for Street Views

• Authors: Jinming Su, Songen Gu, Yiting Duan, Xingyue Chen, Junfeng Luo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04504
• Pdf URL: https://arxiv.org/pdf/2402.04504
• Copy Paste: [[2402.04504]] Text2Street: Controllable Text-to-image Generation for Street Views(https://arxiv.org/abs/2402.04504)
• Keywords: diffusion
• Abstract: Text-to-image generation has made remarkable progress with the emergence of diffusion models. However, it is still a difficult task to generate images for street views based on text, mainly because the road topology of street scenes is complex, the traffic status is diverse and the weather condition is various, which makes conventional text-to-image models difficult to deal with. To address these challenges, we propose a novel controllable text-to-image framework, named \textbf{Text2Street}. In the framework, we first introduce the lane-aware road topology generator, which achieves text-to-map generation with the accurate road structure and lane lines armed with the counting adapter, realizing the controllable road topology generation. Then, the position-based object layout generator is proposed to obtain text-to-layout generation through an object-level bounding box diffusion strategy, realizing the controllable traffic object layout generation. Finally, the multiple control image generator is designed to integrate the road topology, object layout and weather description to realize controllable street-view image generation. Extensive experiments show that the proposed approach achieves controllable street-view text-to-image generation and validates the effectiveness of the Text2Street framework for street views.

Title: BRI3L: A Brightness Illusion Image Dataset for Identification and Localization of Regions of Illusory Perception

• Authors: Aniket Roy, Anirban Roy, Soma Mitra, Kuntal Ghosh
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04541
• Pdf URL: https://arxiv.org/pdf/2402.04541
• Copy Paste: [[2402.04541]] BRI3L: A Brightness Illusion Image Dataset for Identification and Localization of Regions of Illusory Perception(https://arxiv.org/abs/2402.04541)
• Keywords: diffusion
• Abstract: Visual illusions play a significant role in understanding visual perception. Current methods in understanding and evaluating visual illusions are mostly deterministic filtering based approach and they evaluate on a handful of visual illusions, and the conclusions therefore, are not generic. To this end, we generate a large-scale dataset of 22,366 images (BRI3L: BRightness Illusion Image dataset for Identification and Localization of illusory perception) of the five types of brightness illusions and benchmark the dataset using data-driven neural network based approaches. The dataset contains label information - (1) whether a particular image is illusory/nonillusory, (2) the segmentation mask of the illusory region of the image. Hence, both the classification and segmentation task can be evaluated using this dataset. We follow the standard psychophysical experiments involving human subjects to validate the dataset. To the best of our knowledge, this is the first attempt to develop a dataset of visual illusions and benchmark using data-driven approach for illusion classification and localization. We consider five well-studied types of brightness illusions: 1) Hermann grid, 2) Simultaneous Brightness Contrast, 3) White illusion, 4) Grid illusion, and 5) Induced Grating illusion. Benchmarking on the dataset achieves 99.56% accuracy in illusion identification and 84.37% pixel accuracy in illusion localization. The application of deep learning model, it is shown, also generalizes over unseen brightness illusions like brightness assimilation to contrast transitions. We also test the ability of state-of-theart diffusion models to generate brightness illusions. We have provided all the code, dataset, instructions etc in the github repo: https://github.com/aniket004/BRI3L

Title: FM-Fusion: Instance-aware Semantic Mapping Boosted by Vision-Language Foundation Models

• Authors: Chuhao Liu, Ke Wang, Jieqi Shi, Zhijian Qiao, Shaojie Shen
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.04555
• Pdf URL: https://arxiv.org/pdf/2402.04555
• Copy Paste: [[2402.04555]] FM-Fusion: Instance-aware Semantic Mapping Boosted by Vision-Language Foundation Models(https://arxiv.org/abs/2402.04555)
• Keywords: foundation model
• Abstract: Semantic mapping based on the supervised object detectors is sensitive to image distribution. In real-world environments, the object detection and segmentation performance can lead to a major drop, preventing the use of semantic mapping in a wider domain. On the other hand, the development of vision-language foundation models demonstrates a strong zero-shot transferability across data distribution. It provides an opportunity to construct generalizable instance-aware semantic maps. Hence, this work explores how to boost instance-aware semantic mapping from object detection generated from foundation models. We propose a probabilistic label fusion method to predict close-set semantic classes from open-set label measurements. An instance refinement module merges the over-segmented instances caused by inconsistent segmentation. We integrate all the modules into a unified semantic mapping system. Reading a sequence of RGB-D input, our work incrementally reconstructs an instance-aware semantic map. We evaluate the zero-shot performance of our method in ScanNet and SceneNN datasets. Our method achieves 40.3 mean average precision (mAP) on the ScanNet semantic instance segmentation task. It outperforms the traditional semantic mapping method significantly.

Title: OIL-AD: An Anomaly Detection Framework for Sequential Decision Sequences

• Authors: Chen Wang, Sarah Erfani, Tansu Alpcan, Christopher Leckie
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.04567
• Pdf URL: https://arxiv.org/pdf/2402.04567
• Copy Paste: [[2402.04567]] OIL-AD: An Anomaly Detection Framework for Sequential Decision Sequences(https://arxiv.org/abs/2402.04567)
• Keywords: anomaly
• Abstract: Anomaly detection in decision-making sequences is a challenging problem due to the complexity of normality representation learning and the sequential nature of the task. Most existing methods based on Reinforcement Learning (RL) are difficult to implement in the real world due to unrealistic assumptions, such as having access to environment dynamics, reward signals, and online interactions with the environment. To address these limitations, we propose an unsupervised method named Offline Imitation Learning based Anomaly Detection (OIL-AD), which detects anomalies in decision-making sequences using two extracted behaviour features: action optimality and sequential association. Our offline learning model is an adaptation of behavioural cloning with a transformer policy network, where we modify the training process to learn a Q function and a state value function from normal trajectories. We propose that the Q function and the state value function can provide sufficient information about agents' behavioural data, from which we derive two features for anomaly detection. The intuition behind our method is that the action optimality feature derived from the Q function can differentiate the optimal action from others at each local state, and the sequential association feature derived from the state value function has the potential to maintain the temporal correlations between decisions (state-action pairs). Our experiments show that OIL-AD can achieve outstanding online anomaly detection performance with up to 34.8% improvement in F1 score over comparable baselines.

Title: Sparse Anatomical Prompt Semi-Supervised Learning with Masked Image Modeling for CBCT Tooth Segmentation

• Authors: Pengyu Dai, Yafei Ou, Yang Liu, Yue Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04587
• Pdf URL: https://arxiv.org/pdf/2402.04587
• Copy Paste: [[2402.04587]] Sparse Anatomical Prompt Semi-Supervised Learning with Masked Image Modeling for CBCT Tooth Segmentation(https://arxiv.org/abs/2402.04587)
• Keywords: self-supervised
• Abstract: Accurate tooth identification and segmentation in Cone Beam Computed Tomography (CBCT) dental images can significantly enhance the efficiency and precision of manual diagnoses performed by dentists. However, existing segmentation methods are mainly developed based on large data volumes training, on which their annotations are extremely time-consuming. Meanwhile, the teeth of each class in CBCT dental images being closely positioned, coupled with subtle inter-class differences, gives rise to the challenge of indistinct boundaries when training model with limited data. To address these challenges, this study aims to propose a tasked-oriented Masked Auto-Encoder paradigm to effectively utilize large amounts of unlabeled data to achieve accurate tooth segmentation with limited labeled data. Specifically, we first construct a self-supervised pre-training framework of masked auto encoder to efficiently utilize unlabeled data to enhance the network performance. Subsequently, we introduce a sparse masked prompt mechanism based on graph attention to incorporate boundary information of the teeth, aiding the network in learning the anatomical structural features of teeth. To the best of our knowledge, we are pioneering the integration of the mask pre-training paradigm into the CBCT tooth segmentation task. Extensive experiments demonstrate both the feasibility of our proposed method and the potential of the boundary prompt mechanism.

Title: Alirector: Alignment-Enhanced Chinese Grammatical Error Corrector

• Authors: Haihui Yang, Xiaojun Quan
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.04601
• Pdf URL: https://arxiv.org/pdf/2402.04601
• Copy Paste: [[2402.04601]] Alirector: Alignment-Enhanced Chinese Grammatical Error Corrector(https://arxiv.org/abs/2402.04601)
• Keywords: generative
• Abstract: Chinese grammatical error correction (CGEC) faces serious overcorrection challenges when employing autoregressive generative models such as sequence-to-sequence (Seq2Seq) models and decoder-only large language models (LLMs). While previous methods aim to address overcorrection in Seq2Seq models, they are difficult to adapt to decoder-only LLMs. In this paper, we propose an alignment-enhanced corrector for the overcorrection problem that applies to both Seq2Seq models and decoder-only LLMs. Our method first trains a correction model to generate an initial correction of the source sentence. Then, we combine the source sentence with the initial correction and feed it through an alignment model for another round of correction, aiming to enforce the alignment model to focus on potential overcorrection. Moreover, to enhance the model's ability to identify nuances, we further explore the reverse alignment of the source sentence and the initial correction. Finally, we transfer the alignment knowledge from two alignment models to the correction model, instructing it on how to avoid overcorrection. Experimental results on three CGEC datasets demonstrate the effectiveness of our approach in alleviating overcorrection and improving overall performance.

Title: TinyLLM: Learning a Small Student from Multiple Large Language Models

• Authors: Yijun Tian, Yikun Han, Xiusi Chen, Wei Wang, Nitesh V. Chawla
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04616
• Pdf URL: https://arxiv.org/pdf/2402.04616
• Copy Paste: [[2402.04616]] TinyLLM: Learning a Small Student from Multiple Large Language Models(https://arxiv.org/abs/2402.04616)
• Keywords: in-context
• Abstract: Transferring the reasoning capability from stronger large language models (LLMs) to smaller ones has been quite appealing, as smaller LLMs are more flexible to deploy with less expense. Among the existing solutions, knowledge distillation stands out due to its outstanding efficiency and generalization. However, existing methods suffer from several drawbacks, including limited knowledge diversity and the lack of rich contextual information. To solve the problems and facilitate the learning of compact language models, we propose TinyLLM, a novel knowledge distillation paradigm to learn a small student LLM from multiple large teacher LLMs. In particular, we encourage the student LLM to not only generate the correct answers but also understand the rationales behind these answers. Given that different LLMs possess diverse reasoning skills, we guide the student model to assimilate knowledge from various teacher LLMs. We further introduce an in-context example generator and a teacher-forcing Chain-of-Thought strategy to ensure that the rationales are accurate and grounded in contextually appropriate scenarios. Extensive experiments on six datasets across two reasoning tasks demonstrate the superiority of our method. Results show that TinyLLM can outperform large teacher LLMs significantly, despite having a considerably smaller model size.

Title: Noise Map Guidance: Inversion with Spatial Context for Real Image Editing

• Authors: Hansam Cho, Jonghyun Lee, Seoung Bum Kim, Tae-Hyun Oh, Yonghyun Jeong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04625
• Pdf URL: https://arxiv.org/pdf/2402.04625
• Copy Paste: [[2402.04625]] Noise Map Guidance: Inversion with Spatial Context for Real Image Editing(https://arxiv.org/abs/2402.04625)
• Keywords: diffusion
• Abstract: Text-guided diffusion models have become a popular tool in image synthesis, known for producing high-quality and diverse images. However, their application to editing real images often encounters hurdles primarily due to the text condition deteriorating the reconstruction quality and subsequently affecting editing fidelity. Null-text Inversion (NTI) has made strides in this area, but it fails to capture spatial context and requires computationally intensive per-timestep optimization. Addressing these challenges, we present Noise Map Guidance (NMG), an inversion method rich in a spatial context, tailored for real-image editing. Significantly, NMG achieves this without necessitating optimization, yet preserves the editing quality. Our empirical investigations highlight NMG's adaptability across various editing techniques and its robustness to variants of DDIM inversions.

Title: Domain Bridge: Generative model-based domain forensic for black-box models

• Authors: Jiyi Zhang, Han Fang, Ee-Chien Chang
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04640
• Pdf URL: https://arxiv.org/pdf/2402.04640
• Copy Paste: [[2402.04640]] Domain Bridge: Generative model-based domain forensic for black-box models(https://arxiv.org/abs/2402.04640)
• Keywords: diffusion, generative
• Abstract: In forensic investigations of machine learning models, techniques that determine a model's data domain play an essential role, with prior work relying on large-scale corpora like ImageNet to approximate the target model's domain. Although such methods are effective in finding broad domains, they often struggle in identifying finer-grained classes within those domains. In this paper, we introduce an enhanced approach to determine not just the general data domain (e.g., human face) but also its specific attributes (e.g., wearing glasses). Our approach uses an image embedding model as the encoder and a generative model as the decoder. Beginning with a coarse-grained description, the decoder generates a set of images, which are then presented to the unknown target model. Successful classifications by the model guide the encoder to refine the description, which in turn, are used to produce a more specific set of images in the subsequent iteration. This iterative refinement narrows down the exact class of interest. A key strength of our approach lies in leveraging the expansive dataset, LAION-5B, on which the generative model Stable Diffusion is trained. This enlarges our search space beyond traditional corpora, such as ImageNet. Empirical results showcase our method's performance in identifying specific attributes of a model's input domain, paving the way for more detailed forensic analyses of deep learning models.

Title: LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views

• Authors: Yuji Roh, Qingyun Liu, Huan Gui, Zhe Yuan, Yujin Tang, Steven Euijong Whang, Liang Liu, Shuchao Bi, Lichan Hong, Ed H. Chi, Zhe Zhao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.04644
• Pdf URL: https://arxiv.org/pdf/2402.04644
• Copy Paste: [[2402.04644]] LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views(https://arxiv.org/abs/2402.04644)
• Keywords: foundation model
• Abstract: Fine-tuning is becoming widely used for leveraging the power of pre-trained foundation models in new downstream tasks. While there are many successes of fine-tuning on various tasks, recent studies have observed challenges in the generalization of fine-tuned models to unseen distributions (i.e., out-of-distribution; OOD). To improve OOD generalization, some previous studies identify the limitations of fine-tuning data and regulate fine-tuning to preserve the general representation learned from pre-training data. However, potential limitations in the pre-training data and models are often ignored. In this paper, we contend that overly relying on the pre-trained representation may hinder fine-tuning from learning essential representations for downstream tasks and thus hurt its OOD generalization. It can be especially catastrophic when new tasks are from different (sub)domains compared to pre-training data. To address the issues in both pre-training and fine-tuning data, we propose a novel generalizable fine-tuning method LEVI, where the pre-trained model is adaptively ensembled layer-wise with a small task-specific model, while preserving training and inference efficiencies. By combining two complementing models, LEVI effectively suppresses problematic features in both the fine-tuning data and pre-trained model and preserves useful features for new tasks. Broad experiments with large language and vision models show that LEVI greatly improves fine-tuning generalization via emphasizing different views from fine-tuning data and pre-trained features.

Title: Latent Plan Transformer: Planning as Latent Variable Inference

• Authors: Deqian Kong, Dehong Xu, Minglu Zhao, Bo Pang, Jianwen Xie, Andrew Lizarraga, Yuhao Huang, Sirui Xie, Ying Nian Wu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04647
• Pdf URL: https://arxiv.org/pdf/2402.04647
• Copy Paste: [[2402.04647]] Latent Plan Transformer: Planning as Latent Variable Inference(https://arxiv.org/abs/2402.04647)
• Keywords: generative
• Abstract: In tasks aiming for long-term returns, planning becomes necessary. We study generative modeling for planning with datasets repurposed from offline reinforcement learning. Specifically, we identify temporal consistency in the absence of step-wise rewards as one key technical challenge. We introduce the Latent Plan Transformer (LPT), a novel model that leverages a latent space to connect a Transformer-based trajectory generator and the final return. LPT can be learned with maximum likelihood estimation on trajectory-return pairs. In learning, posterior sampling of the latent variable naturally gathers sub-trajectories to form a consistent abstraction despite the finite context. During test time, the latent variable is inferred from an expected return before policy execution, realizing the idea of planning as inference. It then guides the autoregressive policy throughout the episode, functioning as a plan. Our experiments demonstrate that LPT can discover improved decisions from suboptimal trajectories. It achieves competitive performance across several benchmarks, including Gym-Mujoco, Maze2D, and Connect Four, exhibiting capabilities of nuanced credit assignments, trajectory stitching, and adaptation to environmental contingencies. These results validate that latent variable inference can be a strong alternative to step-wise reward prompting.

Title: OV-NeRF: Open-vocabulary Neural Radiance Fields with Vision and Language Foundation Models for 3D Semantic Understanding

• Authors: Guibiao Liao, Kaichen Zhou, Zhenyu Bao, Kanglin Liu, Qing Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04648
• Pdf URL: https://arxiv.org/pdf/2402.04648
• Copy Paste: [[2402.04648]] OV-NeRF: Open-vocabulary Neural Radiance Fields with Vision and Language Foundation Models for 3D Semantic Understanding(https://arxiv.org/abs/2402.04648)
• Keywords: foundation model
• Abstract: The development of Neural Radiance Fields (NeRFs) has provided a potent representation for encapsulating the geometric and appearance characteristics of 3D scenes. Enhancing the capabilities of NeRFs in open-vocabulary 3D semantic perception tasks has been a recent focus. However, current methods that extract semantics directly from Contrastive Language-Image Pretraining (CLIP) for semantic field learning encounter difficulties due to noisy and view-inconsistent semantics provided by CLIP. To tackle these limitations, we propose OV-NeRF, which exploits the potential of pre-trained vision and language foundation models to enhance semantic field learning through proposed single-view and cross-view strategies. First, from the single-view perspective, we introduce Region Semantic Ranking (RSR) regularization by leveraging 2D mask proposals derived from SAM to rectify the noisy semantics of each training view, facilitating accurate semantic field learning. Second, from the cross-view perspective, we propose a Cross-view Self-enhancement (CSE) strategy to address the challenge raised by view-inconsistent semantics. Rather than invariably utilizing the 2D inconsistent semantics from CLIP, CSE leverages the 3D consistent semantics generated from the well-trained semantic field itself for semantic field training, aiming to reduce ambiguity and enhance overall semantic consistency across different views. Extensive experiments validate our OV-NeRF outperforms current state-of-the-art methods, achieving a significant improvement of 20.31% and 18.42% in mIoU metric on Replica and Scannet, respectively. Furthermore, our approach exhibits consistent superior results across various CLIP configurations, further verifying its robustness.

Title: An Over Complete Deep Learning Method for Inverse Problems

• Authors: Moshe Eliasof, Eldad Haber, Eran Treister
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04653
• Pdf URL: https://arxiv.org/pdf/2402.04653
• Copy Paste: [[2402.04653]] An Over Complete Deep Learning Method for Inverse Problems(https://arxiv.org/abs/2402.04653)
• Keywords: diffusion
• Abstract: Obtaining meaningful solutions for inverse problems has been a major challenge with many applications in science and engineering. Recent machine learning techniques based on proximal and diffusion-based methods have shown promising results. However, as we show in this work, they can also face challenges when applied to some exemplary problems. We show that similar to previous works on over-complete dictionaries, it is possible to overcome these shortcomings by embedding the solution into higher dimensions. The novelty of the work proposed is that we jointly design and learn the embedding and the regularizer for the embedding vector. We demonstrate the merit of this approach on several exemplary and common inverse problems.

Title: Large Language Models As Faithful Explainers

• Authors: Yu-Neng Chuang, Guanchu Wang, Chia-Yuan Chang, Ruixiang Tang, Fan Yang, Mengnan Du, Xuanting Cai, Xia Hu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04678
• Pdf URL: https://arxiv.org/pdf/2402.04678
• Copy Paste: [[2402.04678]] Large Language Models As Faithful Explainers(https://arxiv.org/abs/2402.04678)
• Keywords: generative
• Abstract: Large Language Models (LLMs) have recently become proficient in addressing complex tasks by utilizing their rich internal knowledge and reasoning ability. Consequently, this complexity hinders traditional input-focused explanation algorithms for explaining the complex decision-making processes of LLMs. Recent advancements have thus emerged for self-explaining their predictions through a single feed-forward inference in a natural language format. However, natural language explanations are often criticized for lack of faithfulness since these explanations may not accurately reflect the decision-making behaviors of the LLMs. In this work, we introduce a generative explanation framework, xLLM, to improve the faithfulness of the explanations provided in natural language formats for LLMs. Specifically, we propose an evaluator to quantify the faithfulness of natural language explanation and enhance the faithfulness by an iterative optimization process of xLLM, with the goal of maximizing the faithfulness scores. Experiments conducted on three NLU datasets demonstrate that xLLM can significantly improve the faithfulness of generated explanations, which are in alignment with the behaviors of LLMs.

Title: EvoSeed: Unveiling the Threat on Deep Neural Networks with Real-World Illusions

• Authors: Shashank Kotyan, PoYuan Mao, Danilo Vasconcellos Vargas
• Subjects: cs.CV, cs.AI, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2402.04699
• Pdf URL: https://arxiv.org/pdf/2402.04699
• Copy Paste: [[2402.04699]] EvoSeed: Unveiling the Threat on Deep Neural Networks with Real-World Illusions(https://arxiv.org/abs/2402.04699)
• Keywords: diffusion
• Abstract: Deep neural networks are exploited using natural adversarial samples, which have no impact on human perception but are misclassified. Current approaches often rely on the white-box nature of deep neural networks to generate these adversarial samples or alter the distribution of adversarial samples compared to training distribution. To alleviate the limitations of current approaches, we propose EvoSeed, a novel evolutionary strategy-based search algorithmic framework to generate natural adversarial samples. Our EvoSeed framework uses auxiliary Diffusion and Classifier models to operate in a model-agnostic black-box setting. We employ CMA-ES to optimize the search for an adversarial seed vector, which, when processed by the Conditional Diffusion Model, results in an unrestricted natural adversarial sample misclassified by the Classifier Model. Experiments show that generated adversarial images are of high image quality and are transferable to different classifiers. Our approach demonstrates promise in enhancing the quality of adversarial samples using evolutionary algorithms. We hope our research opens new avenues to enhance the robustness of deep neural networks in real-world scenarios. Project Website can be accessed at \url{https://shashankkotyan.github.io/EvoSeed}.

Title: InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior

• Authors: Chenguo Lin, Yadong Mu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04717
• Pdf URL: https://arxiv.org/pdf/2402.04717
• Copy Paste: [[2402.04717]] InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior(https://arxiv.org/abs/2402.04717)
• Keywords: generative
• Abstract: Comprehending natural language instructions is a charming property for 3D indoor scene synthesis systems. Existing methods directly model object joint distributions and express object relations implicitly within a scene, thereby hindering the controllability of generation. We introduce InstructScene, a novel generative framework that integrates a semantic graph prior and a layout decoder to improve controllability and fidelity for 3D scene synthesis. The proposed semantic graph prior jointly learns scene appearances and layout distributions, exhibiting versatility across various downstream tasks in a zero-shot manner. To facilitate the benchmarking for text-driven 3D scene synthesis, we curate a high-quality dataset of scene-instruction pairs with large language and multimodal models. Extensive experimental results reveal that the proposed method surpasses existing state-of-the-art approaches by a large margin. Thorough ablation studies confirm the efficacy of crucial design components. Project page: https://chenguolin.github.io/projects/InstructScene.

Title: Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints

• Authors: Jian Chen, Ruiyi Zhang, Yufan Zhou, Changyou Chen
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04754
• Pdf URL: https://arxiv.org/pdf/2402.04754
• Copy Paste: [[2402.04754]] Towards Aligned Layout Generation via Diffusion Model with Aesthetic Constraints(https://arxiv.org/abs/2402.04754)
• Keywords: diffusion
• Abstract: Controllable layout generation refers to the process of creating a plausible visual arrangement of elements within a graphic design (e.g., document and web designs) with constraints representing design intentions. Although recent diffusion-based models have achieved state-of-the-art FID scores, they tend to exhibit more pronounced misalignment compared to earlier transformer-based models. In this work, we propose the $\textbf{LA}$yout $\textbf{C}$onstraint diffusion mod$\textbf{E}$l (LACE), a unified model to handle a broad range of layout generation tasks, such as arranging elements with specified attributes and refining or completing a coarse layout design. The model is based on continuous diffusion models. Compared with existing methods that use discrete diffusion models, continuous state-space design can enable the incorporation of differentiable aesthetic constraint functions in training. For conditional generation, we introduce conditions via masked input. Extensive experiment results show that LACE produces high-quality layouts and outperforms existing state-of-the-art baselines.

Title: How Realistic Is Your Synthetic Data? Constraining Deep Generative Models for Tabular Data

• Authors: Mihaela Cătălina Stoian, Salijona Dyrmishi, Maxime Cordy, Thomas Lukasiewicz, Eleonora Giunchiglia
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04823
• Pdf URL: https://arxiv.org/pdf/2402.04823
• Copy Paste: [[2402.04823]] How Realistic Is Your Synthetic Data? Constraining Deep Generative Models for Tabular Data(https://arxiv.org/abs/2402.04823)
• Keywords: generative
• Abstract: Deep Generative Models (DGMs) have been shown to be powerful tools for generating tabular data, as they have been increasingly able to capture the complex distributions that characterize them. However, to generate realistic synthetic data, it is often not enough to have a good approximation of their distribution, as it also requires compliance with constraints that encode essential background knowledge on the problem at hand. In this paper, we address this limitation and show how DGMs for tabular data can be transformed into Constrained Deep Generative Models (C-DGMs), whose generated samples are guaranteed to be compliant with the given constraints. This is achieved by automatically parsing the constraints and transforming them into a Constraint Layer (CL) seamlessly integrated with the DGM. Our extensive experimental analysis with various DGMs and tasks reveals that standard DGMs often violate constraints, some exceeding $95\%$ non-compliance, while their corresponding C-DGMs are never non-compliant. Then, we quantitatively demonstrate that, at training time, C-DGMs are able to exploit the background knowledge expressed by the constraints to outperform their standard counterparts with up to $6.5\%$ improvement in utility and detection. Further, we show how our CL does not necessarily need to be integrated at training time, as it can be also used as a guardrail at inference time, still producing some improvements in the overall performance of the models. Finally, we show that our CL does not hinder the sample generation time of the models.

Title: PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognition

• Authors: Jinghui Lu, Ziwei Yang, Yanjie Wang, Xuejing Liu, Can Huang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.04838
• Pdf URL: https://arxiv.org/pdf/2402.04838
• Copy Paste: [[2402.04838]] PaDeLLM-NER: Parallel Decoding in Large Language Models for Named Entity Recognition(https://arxiv.org/abs/2402.04838)
• Keywords: generative
• Abstract: In this study, we aim to reduce generation latency for Named Entity Recognition (NER) with Large Language Models (LLMs). The main cause of high latency in LLMs is the sequential decoding process, which autoregressively generates all labels and mentions for NER, significantly increase the sequence length. To this end, we introduce Parallel Decoding in LLM for NE} (PaDeLLM-NER), a approach that integrates seamlessly into existing generative model frameworks without necessitating additional modules or architectural modifications. PaDeLLM-NER allows for the simultaneous decoding of all mentions, thereby reducing generation latency. Experiments reveal that PaDeLLM-NER significantly increases inference speed that is 1.76 to 10.22 times faster than the autoregressive approach for both English and Chinese. Simultaneously it maintains the quality of predictions as evidenced by the performance that is on par with the state-of-the-art across various datasets.

Title: Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learning

• Authors: Yuxuan Bian, Xuan Ju, Jiangtong Li, Zhijian Xu, Dawei Cheng, Qiang Xu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04852
• Pdf URL: https://arxiv.org/pdf/2402.04852
• Copy Paste: [[2402.04852]] Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learning(https://arxiv.org/abs/2402.04852)
• Keywords: self-supervised
• Abstract: In this study, we present aLLM4TS, an innovative framework that adapts Large Language Models (LLMs) for time-series representation learning. Central to our approach is that we reconceive time-series forecasting as a self-supervised, multi-patch prediction task, which, compared to traditional mask-and-reconstruction methods, captures temporal dynamics in patch representations more effectively. Our strategy encompasses two-stage training: (i). a causal continual pre-training phase on various time-series datasets, anchored on next patch prediction, effectively syncing LLM capabilities with the intricacies of time-series data; (ii). fine-tuning for multi-patch prediction in the targeted time-series context. A distinctive element of our framework is the patch-wise decoding layer, which departs from previous methods reliant on sequence-level decoding. Such a design directly transposes individual patches into temporal sequences, thereby significantly bolstering the model's proficiency in mastering temporal patch-based representations. aLLM4TS demonstrates superior performance in several downstream tasks, proving its effectiveness in deriving temporal representations with enhanced transferability and marking a pivotal advancement in the adaptation of LLMs for time-series analysis.

Title: Advancing Anomaly Detection: An Adaptation Model and a New Dataset

• Authors: Liyun Zhu, Arjun Raj, Lei Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.04857
• Pdf URL: https://arxiv.org/pdf/2402.04857
• Copy Paste: [[2402.04857]] Advancing Anomaly Detection: An Adaptation Model and a New Dataset(https://arxiv.org/abs/2402.04857)
• Keywords: anomaly
• Abstract: Industry surveillance is widely applicable in sectors like retail, manufacturing, education, and smart cities, each presenting unique anomalies requiring specialized detection. However, adapting anomaly detection models to novel viewpoints within the same scenario poses challenges. Extending these models to entirely new scenarios necessitates retraining or fine-tuning, a process that can be time consuming. To address these challenges, we propose the Scenario-Adaptive Anomaly Detection (SA2D) method, leveraging the few-shot learning framework for faster adaptation of pre-trained models to new concepts. Despite this approach, a significant challenge emerges from the absence of a comprehensive dataset with diverse scenarios and camera views. In response, we introduce the Multi-Scenario Anomaly Detection (MSAD) dataset, encompassing 14 distinct scenarios captured from various camera views. This real-world dataset is the first high-resolution anomaly detection dataset, offering a solid foundation for training superior models. MSAD includes diverse normal motion patterns, incorporating challenging variations like different lighting and weather conditions. Through experimentation, we validate the efficacy of SA2D, particularly when trained on the MSAD dataset. Our results show that SA2D not only excels under novel viewpoints within the same scenario but also demonstrates competitive performance when faced with entirely new scenarios. This highlights our method's potential in addressing challenges in detecting anomalies across diverse and evolving surveillance scenarios.

Title: L4Q: Parameter Efficient Quantization-Aware Training on Large Language Models via LoRA-wise LSQ

• Authors: Hyesung Jeon, Yulhwa Kim, Jae-joon Kim
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.04902
• Pdf URL: https://arxiv.org/pdf/2402.04902
• Copy Paste: [[2402.04902]] L4Q: Parameter Efficient Quantization-Aware Training on Large Language Models via LoRA-wise LSQ(https://arxiv.org/abs/2402.04902)
• Keywords: in-context
• Abstract: Post-training quantization (PTQ) and quantization-aware training (QAT) methods are gaining popularity in mitigating the high memory and computational costs associated with Large Language Models (LLMs). In resource-constrained scenarios, PTQ, with its reduced training overhead, is often preferred over QAT, despite the latter's potential for higher accuracy. Meanwhile, parameter-efficient fine-tuning (PEFT) methods like low-rank adaptation (LoRA) have been introduced, and recent efforts have explored quantization-aware PEFT techniques. However, these approaches may lack generality due to their reliance on the pre-quantized model's configuration. Their effectiveness may be compromised by non-linearly quantized or mixed-precision weights, and the retraining of specific quantization parameters might impede optimal performance. To address these challenges, we propose L4Q, an algorithm for parameter-efficient quantization-aware training. L4Q leverages LoRA-wise learned quantization step size for LLMs, aiming to enhance generality. The simultaneous quantization-and-fine-tuning process of L4Q is applicable to high-precision models, yielding linearly quantized weights with superior accuracy. Our experiments, conducted on the LLaMA and LLaMA2 model families using an instructional dataset, showcase L4Q's capabilities in language comprehension and few-shot in-context learning, achieving sub-4-bit precision while maintaining comparable training times to applying PEFT on a quantized model.

Title: Towards Biologically Plausible and Private Gene Expression Data Generation

• Authors: Dingfan Chen, Marie Oestreich, Tejumade Afonja, Raouf Kerkouche, Matthias Becker, Mario Fritz
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04912
• Pdf URL: https://arxiv.org/pdf/2402.04912
• Copy Paste: [[2402.04912]] Towards Biologically Plausible and Private Gene Expression Data Generation(https://arxiv.org/abs/2402.04912)
• Keywords: generative
• Abstract: Generative models trained with Differential Privacy (DP) are becoming increasingly prominent in the creation of synthetic data for downstream applications. Existing literature, however, primarily focuses on basic benchmarking datasets and tends to report promising results only for elementary metrics and relatively simple data distributions. In this paper, we initiate a systematic analysis of how DP generative models perform in their natural application scenarios, specifically focusing on real-world gene expression data. We conduct a comprehensive analysis of five representative DP generation methods, examining them from various angles, such as downstream utility, statistical properties, and biological plausibility. Our extensive evaluation illuminates the unique characteristics of each DP generation method, offering critical insights into the strengths and weaknesses of each approach, and uncovering intriguing possibilities for future developments. Perhaps surprisingly, our analysis reveals that most methods are capable of achieving seemingly reasonable downstream utility, according to the standard evaluation metrics considered in existing literature. Nevertheless, we find that none of the DP methods are able to accurately capture the biological characteristics of the real dataset. This observation suggests a potential over-optimistic assessment of current methodologies in this field and underscores a pressing need for future enhancements in model design.

Title: Personalized Text Generation with Fine-Grained Linguistic Control

• Authors: Bashar Alhafni, Vivek Kulkarni, Dhruv Kumar, Vipul Raheja
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.04914
• Pdf URL: https://arxiv.org/pdf/2402.04914
• Copy Paste: [[2402.04914]] Personalized Text Generation with Fine-Grained Linguistic Control(https://arxiv.org/abs/2402.04914)
• Keywords: generative
• Abstract: As the text generation capabilities of large language models become increasingly prominent, recent studies have focused on controlling particular aspects of the generated text to make it more personalized. However, most research on controllable text generation focuses on controlling the content or modeling specific high-level/coarse-grained attributes that reflect authors' writing styles, such as formality, domain, or sentiment. In this paper, we focus on controlling fine-grained attributes spanning multiple linguistic dimensions, such as lexical and syntactic attributes. We introduce a novel benchmark to train generative models and evaluate their ability to generate personalized text based on multiple fine-grained linguistic attributes. We systematically investigate the performance of various large language models on our benchmark and draw insights from the factors that impact their performance. We make our code, data, and pretrained models publicly available.

Title: Source-Free Domain Adaptation with Diffusion-Guided Source Data Generation

• Authors: Shivang Chopra, Suraj Kothawade, Houda Aynaou, Aman Chadha
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04929
• Pdf URL: https://arxiv.org/pdf/2402.04929
• Copy Paste: [[2402.04929]] Source-Free Domain Adaptation with Diffusion-Guided Source Data Generation(https://arxiv.org/abs/2402.04929)
• Keywords: diffusion
• Abstract: This paper introduces a novel approach to leverage the generalizability capability of Diffusion Models for Source-Free Domain Adaptation (DM-SFDA). Our proposed DM-SFDA method involves fine-tuning a pre-trained text-to-image diffusion model to generate source domain images using features from the target images to guide the diffusion process. Specifically, the pre-trained diffusion model is fine-tuned to generate source samples that minimize entropy and maximize confidence for the pre-trained source model. We then apply established unsupervised domain adaptation techniques to align the generated source images with target domain data. We validate our approach through comprehensive experiments across a range of datasets, including Office-31, Office-Home, and VisDA. The results highlight significant improvements in SFDA performance, showcasing the potential of diffusion models in generating contextually relevant, domain-specific images.

Title: Blue noise for diffusion models

• Authors: Xingchang Huang, Corentin Salaün, Cristina Vasconcelos, Christian Theobalt, Cengiz Öztireli, Gurprit Singh
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.04930
• Pdf URL: https://arxiv.org/pdf/2402.04930
• Copy Paste: [[2402.04930]] Blue noise for diffusion models(https://arxiv.org/abs/2402.04930)
• Keywords: diffusion
• Abstract: Most of the existing diffusion models use Gaussian noise for training and sampling across all time steps, which may not optimally account for the frequency contents reconstructed by the denoising network. Despite the diverse applications of correlated noise in computer graphics, its potential for improving the training process has been underexplored. In this paper, we introduce a novel and general class of diffusion models taking correlated noise within and across images into account. More specifically, we propose a time-varying noise model to incorporate correlated noise into the training process, as well as a method for fast generation of correlated noise mask. Our model is built upon deterministic diffusion models and utilizes blue noise to help improve the generation quality compared to using Gaussian white (random) noise only. Further, our framework allows introducing correlation across images within a single mini-batch to improve gradient flow. We perform both qualitative and quantitative evaluations on a variety of datasets using our method, achieving improvements on different tasks over existing deterministic diffusion models in terms of FID metric.

Title: LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation

• Authors: Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, Ziwei Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.05054
• Pdf URL: https://arxiv.org/pdf/2402.05054
• Copy Paste: [[2402.05054]] LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation(https://arxiv.org/abs/2402.05054)
• Keywords: diffusion
• Abstract: 3D content creation has achieved significant progress in terms of both quality and speed. Although current feed-forward models can produce 3D objects in seconds, their resolution is constrained by the intensive computation required during training. In this paper, we introduce Large Multi-View Gaussian Model (LGM), a novel framework designed to generate high-resolution 3D models from text prompts or single-view images. Our key insights are two-fold: 1) 3D Representation: We propose multi-view Gaussian features as an efficient yet powerful representation, which can then be fused together for differentiable rendering. 2) 3D Backbone: We present an asymmetric U-Net as a high-throughput backbone operating on multi-view images, which can be produced from text or single-view image input by leveraging multi-view diffusion models. Extensive experiments demonstrate the high fidelity and efficiency of our approach. Notably, we maintain the fast speed to generate 3D objects within 5 seconds while boosting the training resolution to 512, thereby achieving high-resolution 3D content generation.

Title: NITO: Neural Implicit Fields for Resolution-free Topology Optimization

• Authors: Amin Heyrani Nobari, Giorgio Giannone, Lyle Regenwetter, Faez Ahmed
• Subjects: cs.LG, cs.CE
• Abstract URL: https://arxiv.org/abs/2402.05073
• Pdf URL: https://arxiv.org/pdf/2402.05073
• Copy Paste: [[2402.05073]] NITO: Neural Implicit Fields for Resolution-free Topology Optimization(https://arxiv.org/abs/2402.05073)
• Keywords: diffusion
• Abstract: Topology optimization is a critical task in engineering design, where the goal is to optimally distribute material in a given space for maximum performance. We introduce Neural Implicit Topology Optimization (NITO), a novel approach to accelerate topology optimization problems using deep learning. NITO stands out as one of the first frameworks to offer a resolution-free and domain-agnostic solution in deep learning-based topology optimization. NITO synthesizes structures with up to seven times better structural efficiency compared to SOTA diffusion models and does so in a tenth of the time. In the NITO framework, we introduce a novel method, the Boundary Point Order-Invariant MLP (BPOM), to represent boundary conditions in a sparse and domain-agnostic manner, moving away from expensive simulation-based approaches. Crucially, NITO circumvents the domain and resolution limitations that restrict Convolutional Neural Network (CNN) models to a structured domain of fixed size -- limitations that hinder the widespread adoption of CNNs in engineering applications. This generalizability allows a single NITO model to train and generate solutions in countless domains, eliminating the need for numerous domain-specific CNNs and their extensive datasets. Despite its generalizability, NITO outperforms SOTA models even in specialized tasks, is an order of magnitude smaller, and is practically trainable at high resolutions that would be restrictive for CNNs. This combination of versatility, efficiency, and performance underlines NITO's potential to transform the landscape of engineering design optimization problems through implicit fields.

Title: On diffusion models for amortized inference: Benchmarking and improving stochastic control and sampling

• Authors: Marcin Sendera, Minsu Kim, Sarthak Mittal, Pablo Lemos, Luca Scimeca, Jarrid Rector-Brooks, Alexandre Adam, Yoshua Bengio, Nikolay Malkin
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.05098
• Pdf URL: https://arxiv.org/pdf/2402.05098
• Copy Paste: [[2402.05098]] On diffusion models for amortized inference: Benchmarking and improving stochastic control and sampling(https://arxiv.org/abs/2402.05098)
• Keywords: diffusion, generative
• Abstract: We study the problem of training diffusion models to sample from a distribution with a given unnormalized density or energy function. We benchmark several diffusion-structured inference methods, including simulation-based variational approaches and off-policy methods (continuous generative flow networks). Our results shed light on the relative advantages of existing algorithms while bringing into question some claims from past work. We also propose a novel exploration strategy for off-policy methods, based on local search in the target space with the use of a replay buffer, and show that it improves the quality of samples on a variety of target distributions. Our code for the sampling methods and benchmarks studied is made public at https://github.com/GFNOrg/gfn-diffusion as a base for future work on diffusion models for amortized inference.