2024-03-06

Title: Towards efficient deep autoencoders for multivariate time series anomaly detection

• Authors: Marcin Pietroń, Dominik Żurek, Kamil Faber, Roberto Corizzo
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02429
• Pdf URL: https://arxiv.org/pdf/2403.02429
• Copy Paste: [[2403.02429]] Towards efficient deep autoencoders for multivariate time series anomaly detection(https://arxiv.org/abs/2403.02429)
• Keywords: anomaly
• Abstract: Multivariate time series anomaly detection is a crucial problem in many industrial and research applications. Timely detection of anomalies allows, for instance, to prevent defects in manufacturing processes and failures in cyberphysical systems. Deep learning methods are preferred among others for their accuracy and robustness for the analysis of complex multivariate data. However, a key aspect is being able to extract predictions in a timely manner, to accommodate real-time requirements in different applications. In the case of deep learning models, model reduction is extremely important to achieve optimal results in real-time systems with limited time and memory constraints. In this paper, we address this issue by proposing a novel compression method for deep autoencoders that involves three key factors. First, pruning reduces the number of weights, while preventing catastrophic drops in accuracy by means of a fast search process that identifies high sparsity levels. Second, linear and non-linear quantization reduces model complexity by reducing the number of bits for every single weight. The combined contribution of these three aspects allow the model size to be reduced, by removing a subset of the weights (pruning), and decreasing their bit-width (quantization). As a result, the compressed model is faster and easier to adopt in highly constrained hardware environments. Experiments performed on popular multivariate anomaly detection benchmarks, show that our method is capable of achieving significant model compression ratio (between 80% and 95%) without a significant reduction in the anomaly detection performance.

Title: Root Causing Prediction Anomalies Using Explainable AI

• Authors: Ramanathan Vishnampet, Rajesh Shenoy, Jianhui Chen, Anuj Gupta
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02439
• Pdf URL: https://arxiv.org/pdf/2403.02439
• Copy Paste: [[2403.02439]] Root Causing Prediction Anomalies Using Explainable AI(https://arxiv.org/abs/2403.02439)
• Keywords: anomaly
• Abstract: This paper presents a novel application of explainable AI (XAI) for root-causing performance degradation in machine learning models that learn continuously from user engagement data. In such systems a single feature corruption can cause cascading feature, label and concept drifts. We have successfully applied this technique to improve the reliability of models used in personalized advertising. Performance degradation in such systems manifest as prediction anomalies in the models. These models are typically trained continuously using features that are produced by hundreds of real time data processing pipelines or derived from other upstream models. A failure in any of these pipelines or an instability in any of the upstream models can cause feature corruption, causing the model's predicted output to deviate from the actual output and the training data to become corrupted. The causal relationship between the features and the predicted output is complex, and root-causing is challenging due to the scale and dynamism of the system. We demonstrate how temporal shifts in the global feature importance distribution can effectively isolate the cause of a prediction anomaly, with better recall than model-to-feature correlation methods. The technique appears to be effective even when approximating the local feature importance using a simple perturbation-based method, and aggregating over a few thousand examples. We have found this technique to be a model-agnostic, cheap and effective way to monitor complex data pipelines in production and have deployed a system for continuously analyzing the global feature importance distribution of continuously trained models.

Title: Anatomically Constrained Tractography of the Fetal Brain

• Authors: Camilo Calixto, Camilo Jaimes, Matheus D. Soldatelli, Simon K. Warfield, Ali Gholipour, Davood Karimi
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02444
• Pdf URL: https://arxiv.org/pdf/2403.02444
• Copy Paste: [[2403.02444]] Anatomically Constrained Tractography of the Fetal Brain(https://arxiv.org/abs/2403.02444)
• Keywords: diffusion
• Abstract: Diffusion-weighted Magnetic Resonance Imaging (dMRI) is increasingly used to study the fetal brain in utero. An important computation enabled by dMRI is streamline tractography, which has unique applications such as tract-specific analysis of the brain white matter and structural connectivity assessment. However, due to the low fetal dMRI data quality and the challenging nature of tractography, existing methods tend to produce highly inaccurate results. They generate many false streamlines while failing to reconstruct streamlines that constitute the major white matter tracts. In this paper, we advocate for anatomically constrained tractography based on an accurate segmentation of the fetal brain tissue directly in the dMRI space. We develop a deep learning method to compute the segmentation automatically. Experiments on independent test data show that this method can accurately segment the fetal brain tissue and drastically improve tractography results. It enables the reconstruction of highly curved tracts such as optic radiations. Importantly, our method infers the tissue segmentation and streamline propagation direction from a diffusion tensor fit to the dMRI data, making it applicable to routine fetal dMRI scans. The proposed method can lead to significant improvements in the accuracy and reproducibility of quantitative assessment of the fetal brain with dMRI.

Title: HeAR -- Health Acoustic Representations

• Authors: Sebastien Baur, Zaid Nabulsi, Wei-Hung Weng, Jake Garrison, Louis Blankemeier, Sam Fishman, Christina Chen, Sujay Kakarmath, Minyoi Maimbolwa, Nsala Sanjase, Brian Shuma, Yossi Matias, Greg S. Corrado, Shwetak Patel, Shravya Shetty, Shruthi Prabhakara, Monde Muyoyeta, Diego Ardila
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02522
• Pdf URL: https://arxiv.org/pdf/2403.02522
• Copy Paste: [[2403.02522]] HeAR -- Health Acoustic Representations(https://arxiv.org/abs/2403.02522)
• Keywords: self-supervised
• Abstract: Health acoustic sounds such as coughs and breaths are known to contain useful health signals with significant potential for monitoring health and disease, yet are underexplored in the medical machine learning community. The existing deep learning systems for health acoustics are often narrowly trained and evaluated on a single task, which is limited by data and may hinder generalization to other tasks. To mitigate these gaps, we develop HeAR, a scalable self-supervised learning-based deep learning system using masked autoencoders trained on a large dataset of 313 million two-second long audio clips. Through linear probes, we establish HeAR as a state-of-the-art health audio embedding model on a benchmark of 33 health acoustic tasks across 6 datasets. By introducing this work, we hope to enable and accelerate further health acoustics research.

Title: Towards Foundation Time Series Model: To Synthesize Or Not To Synthesize?

• Authors: Kseniia Kuvshinova, Olga Tsymboi, Alina Kostromina, Dmitry Simakov, Elizaveta Kovtun
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02534
• Pdf URL: https://arxiv.org/pdf/2403.02534
• Copy Paste: [[2403.02534]] Towards Foundation Time Series Model: To Synthesize Or Not To Synthesize?(https://arxiv.org/abs/2403.02534)
• Keywords: foundation model
• Abstract: The industry is rich in cases when we are required to make forecasting for large amounts of time series at once. However, we might be in a situation where we can not afford to train a separate model for each of them. Such issue in time series modeling remains without due attention. The remedy for this setting is the establishment of a foundation model. Such a model is expected to work in zero-shot and few-shot regimes. However, what should we take as a training dataset for such kind of model? Witnessing the benefits from the enrichment of NLP datasets with artificially-generated data, we might want to adopt their experience for time series. In contrast to natural language, the process of generation of synthetic time series data is even more favorable because it provides full control of series patterns, time horizons, and number of samples. In this work, we consider the essential question if it is advantageous to train a foundation model on synthetic data or it is better to utilize only a limited number of real-life examples. Our experiments are conducted only for regular time series and speak in favor of leveraging solely the real time series. Moreover, the choice of the proper source dataset strongly influences the performance during inference. When provided access even to a limited quantity of short time series data, employing it within a supervised framework yields more favorable results than training on a larger volume of synthetic data. The code for our experiments is publicly available on Github \url{https://github.com/sb-ai-lab/synthesize_or_not}.

Title: Updating the Minimum Information about CLinical Artificial Intelligence (MI-CLAIM) checklist for generative modeling research

• Authors: Brenda Y. Miao, Irene Y. Chen, Christopher YK Williams, Jaysón Davidson, Augusto Garcia-Agundez, Harry Sun, Travis Zack, Atul J. Butte, Madhumita Sushil
• Subjects: cs.CL, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02558
• Pdf URL: https://arxiv.org/pdf/2403.02558
• Copy Paste: [[2403.02558]] Updating the Minimum Information about CLinical Artificial Intelligence (MI-CLAIM) checklist for generative modeling research(https://arxiv.org/abs/2403.02558)
• Keywords: diffusion, generative
• Abstract: Recent advances in generative models, including large language models (LLMs), vision language models (VLMs), and diffusion models, have accelerated the field of natural language and image processing in medicine and marked a significant paradigm shift in how biomedical models can be developed and deployed. While these models are highly adaptable to new tasks, scaling and evaluating their usage presents new challenges not addressed in previous frameworks. In particular, the ability of these models to produce useful outputs with little to no specialized training data ("zero-" or "few-shot" approaches), as well as the open-ended nature of their outputs, necessitate the development of updated guidelines in using and evaluating these models. In response to gaps in standards and best practices for the development of clinical AI tools identified by US Executive Order 141103 and several emerging national networks for clinical AI evaluation, we begin to formalize some of these guidelines by building on the "Minimum information about clinical artificial intelligence modeling" (MI-CLAIM) checklist. The MI-CLAIM checklist, originally developed in 2020, provided a set of six steps with guidelines on the minimum information necessary to encourage transparent, reproducible research for artificial intelligence (AI) in medicine. Here, we propose modifications to the original checklist that highlight differences in training, evaluation, interpretability, and reproducibility of generative models compared to traditional AI models for clinical research. This updated checklist also seeks to clarify cohort selection reporting and adds additional items on alignment with ethical standards.

Title: Semantic Human Mesh Reconstruction with Textures

• Authors: Xiaoyu Zhan, Jianxin Yang, Yuanqi Li, Jie Guo, Yanwen Guo, Wenping Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02561
• Pdf URL: https://arxiv.org/pdf/2403.02561
• Copy Paste: [[2403.02561]] Semantic Human Mesh Reconstruction with Textures(https://arxiv.org/abs/2403.02561)
• Keywords: diffusion, self-supervised
• Abstract: The field of 3D detailed human mesh reconstruction has made significant progress in recent years. However, current methods still face challenges when used in industrial applications due to unstable results, low-quality meshes, and a lack of UV unwrapping and skinning weights. In this paper, we present SHERT, a novel pipeline that can reconstruct semantic human meshes with textures and high-precision details. SHERT applies semantic- and normal-based sampling between the detailed surface (eg mesh and SDF) and the corresponding SMPL-X model to obtain a partially sampled semantic mesh and then generates the complete semantic mesh by our specifically designed self-supervised completion and refinement networks. Using the complete semantic mesh as a basis, we employ a texture diffusion model to create human textures that are driven by both images and texts. Our reconstructed meshes have stable UV unwrapping, high-quality triangle meshes, and consistent semantic information. The given SMPL-X model provides semantic information and shape priors, allowing SHERT to perform well even with incorrect and incomplete inputs. The semantic information also makes it easy to substitute and animate different body parts such as the face, body, and hands. Quantitative and qualitative experiments demonstrate that SHERT is capable of producing high-fidelity and robust semantic meshes that outperform state-of-the-art methods.

Title: Pooling Image Datasets With Multiple Covariate Shift and Imbalance

• Authors: Sotirios Panagiotis Chytas, Vishnu Suresh Lokhande, Peiran Li, Vikas Singh
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02598
• Pdf URL: https://arxiv.org/pdf/2403.02598
• Copy Paste: [[2403.02598]] Pooling Image Datasets With Multiple Covariate Shift and Imbalance(https://arxiv.org/abs/2403.02598)
• Keywords: self-supervised
• Abstract: Small sample sizes are common in many disciplines, which necessitates pooling roughly similar datasets across multiple institutions to study weak but relevant associations between images and disease outcomes. Such data often manifest shift/imbalance in covariates (i.e., secondary non-imaging data). Controlling for such nuisance variables is common within standard statistical analysis, but the ideas do not directly apply to overparameterized models. Consequently, recent work has shown how strategies from invariant representation learning provides a meaningful starting point, but the current repertoire of methods is limited to accounting for shifts/imbalances in just a couple of covariates at a time. In this paper, we show how viewing this problem from the perspective of Category theory provides a simple and effective solution that completely avoids elaborate multi-stage training pipelines that would otherwise be needed. We show the effectiveness of this approach via extensive experiments on real datasets. Further, we discuss how this style of formulation offers a unified perspective on at least 5+ distinct problem settings, from self-supervised learning to matching problems in 3D reconstruction.

Title: Unsupervised Spatio-Temporal State Estimation for Fine-grained Adaptive Anomaly Diagnosis of Industrial Cyber-physical Systems

• Authors: Haili Sun, Yan Huang, Lansheng Han, Cai Fu, Chunjie Zhou
• Subjects: cs.LG, cs.AI, cs.CR, cs.NI, eess.SY
• Abstract URL: https://arxiv.org/abs/2403.02616
• Pdf URL: https://arxiv.org/pdf/2403.02616
• Copy Paste: [[2403.02616]] Unsupervised Spatio-Temporal State Estimation for Fine-grained Adaptive Anomaly Diagnosis of Industrial Cyber-physical Systems(https://arxiv.org/abs/2403.02616)
• Keywords: anomaly
• Abstract: Accurate detection and diagnosis of abnormal behaviors such as network attacks from multivariate time series (MTS) are crucial for ensuring the stable and effective operation of industrial cyber-physical systems (CPS). However, existing researches pay little attention to the logical dependencies among system working states, and have difficulties in explaining the evolution mechanisms of abnormal signals. To reveal the spatio-temporal association relationships and evolution mechanisms of the working states of industrial CPS, this paper proposes a fine-grained adaptive anomaly diagnosis method (i.e. MAD-Transformer) to identify and diagnose anomalies in MTS. MAD-Transformer first constructs a temporal state matrix to characterize and estimate the change patterns of the system states in the temporal dimension. Then, to better locate the anomalies, a spatial state matrix is also constructed to capture the inter-sensor state correlation relationships within the system. Subsequently, based on these two types of state matrices, a three-branch structure of series-temporal-spatial attention module is designed to simultaneously capture the series, temporal, and space dependencies among MTS. Afterwards, three associated alignment loss functions and a reconstruction loss are constructed to jointly optimize the model. Finally, anomalies are determined and diagnosed by comparing the residual matrices with the original matrices. We conducted comparative experiments on five publicly datasets spanning three application domains (service monitoring, spatial and earth exploration, and water treatment), along with a petroleum refining simulation dataset collected by ourselves. The results demonstrate that MAD-Transformer can adaptively detect fine-grained anomalies with short duration, and outperforms the state-of-the-art baselines in terms of noise robustness and localization performance.

Title: Modeling Collaborator: Enabling Subjective Vision Classification With Minimal Human Effort via LLM Tool-Use

• Authors: Imad Eddine Toubal, Aditya Avinash, Neil Gordon Alldrin, Jan Dlabal, Wenlei Zhou, Enming Luo, Otilia Stretcu, Hao Xiong, Chun-Ta Lu, Howard Zhou, Ranjay Krishna, Ariel Fuxman, Tom Duerig
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02626
• Pdf URL: https://arxiv.org/pdf/2403.02626
• Copy Paste: [[2403.02626]] Modeling Collaborator: Enabling Subjective Vision Classification With Minimal Human Effort via LLM Tool-Use(https://arxiv.org/abs/2403.02626)
• Keywords: foundation model
• Abstract: From content moderation to wildlife conservation, the number of applications that require models to recognize nuanced or subjective visual concepts is growing. Traditionally, developing classifiers for such concepts requires substantial manual effort measured in hours, days, or even months to identify and annotate data needed for training. Even with recently proposed Agile Modeling techniques, which enable rapid bootstrapping of image classifiers, users are still required to spend 30 minutes or more of monotonous, repetitive data labeling just to train a single classifier. Drawing on Fiske's Cognitive Miser theory, we propose a new framework that alleviates manual effort by replacing human labeling with natural language interactions, reducing the total effort required to define a concept by an order of magnitude: from labeling 2,000 images to only 100 plus some natural language interactions. Our framework leverages recent advances in foundation models, both large language models and vision-language models, to carve out the concept space through conversation and by automatically labeling training data points. Most importantly, our framework eliminates the need for crowd-sourced annotations. Moreover, our framework ultimately produces lightweight classification models that are deployable in cost-sensitive scenarios. Across 15 subjective concepts and across 2 public image classification datasets, our trained models outperform traditional Agile Modeling as well as state-of-the-art zero-shot classification models like ALIGN, CLIP, CuPL, and large visual question-answering models like PaLI-X.

Title: Few-shot Learner Parameterization by Diffusion Time-steps

• Authors: Zhongqi Yue, Pan Zhou, Richang Hong, Hanwang Zhang, Qianru Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02649
• Pdf URL: https://arxiv.org/pdf/2403.02649
• Copy Paste: [[2403.02649]] Few-shot Learner Parameterization by Diffusion Time-steps(https://arxiv.org/abs/2403.02649)
• Keywords: diffusion, foundation model
• Abstract: Even when using large multi-modal foundation models, few-shot learning is still challenging -- if there is no proper inductive bias, it is nearly impossible to keep the nuanced class attributes while removing the visually prominent attributes that spuriously correlate with class labels. To this end, we find an inductive bias that the time-steps of a Diffusion Model (DM) can isolate the nuanced class attributes, i.e., as the forward diffusion adds noise to an image at each time-step, nuanced attributes are usually lost at an earlier time-step than the spurious attributes that are visually prominent. Building on this, we propose Time-step Few-shot (TiF) learner. We train class-specific low-rank adapters for a text-conditioned DM to make up for the lost attributes, such that images can be accurately reconstructed from their noisy ones given a prompt. Hence, at a small time-step, the adapter and prompt are essentially a parameterization of only the nuanced class attributes. For a test image, we can use the parameterization to only extract the nuanced class attributes for classification. TiF learner significantly outperforms OpenCLIP and its adapters on a variety of fine-grained and customized few-shot learning tasks. Codes are in https://github.com/yue-zhongqi/tif.

Title: Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters

• Authors: Weizhi Wang, Khalil Mrini, Linjie Yang, Sateesh Kumar, Yu Tian, Xifeng Yan, Heng Wang
• Subjects: cs.CV, cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02677
• Pdf URL: https://arxiv.org/pdf/2403.02677
• Copy Paste: [[2403.02677]] Finetuned Multimodal Language Models Are High-Quality Image-Text Data Filters(https://arxiv.org/abs/2403.02677)
• Keywords: foundation model
• Abstract: We propose a novel framework for filtering image-text data by leveraging fine-tuned Multimodal Language Models (MLMs). Our approach outperforms predominant filtering methods (e.g., CLIPScore) via integrating the recent advances in MLMs. We design four distinct yet complementary metrics to holistically measure the quality of image-text data. A new pipeline is established to construct high-quality instruction data for fine-tuning MLMs as data filters. Comparing with CLIPScore, our MLM filters produce more precise and comprehensive scores that directly improve the quality of filtered data and boost the performance of pre-trained models. We achieve significant improvements over CLIPScore on popular foundation models (i.e., CLIP and BLIP2) and various downstream tasks. Our MLM filter can generalize to different models and tasks, and be used as a drop-in replacement for CLIPScore. An additional ablation study is provided to verify our design choices for the MLM filter.

Title: Time Weaver: A Conditional Time Series Generation Model

• Authors: Sai Shankar Narasimhan, Shubhankar Agarwal, Oguzhan Akcin, Sujay Sanghavi, Sandeep Chinchali
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2403.02682
• Pdf URL: https://arxiv.org/pdf/2403.02682
• Copy Paste: [[2403.02682]] Time Weaver: A Conditional Time Series Generation Model(https://arxiv.org/abs/2403.02682)
• Keywords: diffusion, generative
• Abstract: Imagine generating a city's electricity demand pattern based on weather, the presence of an electric vehicle, and location, which could be used for capacity planning during a winter freeze. Such real-world time series are often enriched with paired heterogeneous contextual metadata (weather, location, etc.). Current approaches to time series generation often ignore this paired metadata, and its heterogeneity poses several practical challenges in adapting existing conditional generation approaches from the image, audio, and video domains to the time series domain. To address this gap, we introduce Time Weaver, a novel diffusion-based model that leverages the heterogeneous metadata in the form of categorical, continuous, and even time-variant variables to significantly improve time series generation. Additionally, we show that naive extensions of standard evaluation metrics from the image to the time series domain are insufficient. These metrics do not penalize conditional generation approaches for their poor specificity in reproducing the metadata-specific features in the generated time series. Thus, we innovate a novel evaluation metric that accurately captures the specificity of conditional generation and the realism of the generated time series. We show that Time Weaver outperforms state-of-the-art benchmarks, such as Generative Adversarial Networks (GANs), by up to 27% in downstream classification tasks on real-world energy, medical, air quality, and traffic data sets.

Title: Deep Common Feature Mining for Efficient Video Semantic Segmentation

• Authors: Yaoyan Zheng, Hongyu Yang, Di Huang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02689
• Pdf URL: https://arxiv.org/pdf/2403.02689
• Copy Paste: [[2403.02689]] Deep Common Feature Mining for Efficient Video Semantic Segmentation(https://arxiv.org/abs/2403.02689)
• Keywords: self-supervised
• Abstract: Recent advancements in video semantic segmentation have made substantial progress by exploiting temporal correlations. Nevertheless, persistent challenges, including redundant computation and the reliability of the feature propagation process, underscore the need for further innovation. In response, we present Deep Common Feature Mining (DCFM), a novel approach strategically designed to address these challenges by leveraging the concept of feature sharing. DCFM explicitly decomposes features into two complementary components. The common representation extracted from a key-frame furnishes essential high-level information to neighboring non-key frames, allowing for direct re-utilization without feature propagation. Simultaneously, the independent feature, derived from each video frame, captures rapidly changing information, providing frame-specific clues crucial for segmentation. To achieve such decomposition, we employ a symmetric training strategy tailored for sparsely annotated data, empowering the backbone to learn a robust high-level representation enriched with common information. Additionally, we incorporate a self-supervised loss function to reinforce intra-class feature similarity and enhance temporal consistency. Experimental evaluations on the VSPW and Cityscapes datasets demonstrate the effectiveness of our method, showing a superior balance between accuracy and efficiency.

Title: Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models

• Authors: Sang T. Truong, Duc Q. Nguyen, Toan Nguyen, Dong D. Le, Nhi N. Truong, Tho Quan, Sanmi Koyejo
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02715
• Pdf URL: https://arxiv.org/pdf/2403.02715
• Copy Paste: [[2403.02715]] Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models(https://arxiv.org/abs/2403.02715)
• Keywords: generative
• Abstract: Recent advancements in large language models (LLMs) have underscored their importance in the evolution of artificial intelligence. However, despite extensive pretraining on multilingual datasets, available open-sourced LLMs exhibit limited effectiveness in processing Vietnamese. The challenge is exacerbated by the absence of systematic benchmark datasets and metrics tailored for Vietnamese LLM evaluation. To mitigate these issues, we have finetuned LLMs specifically for Vietnamese and developed a comprehensive evaluation framework encompassing 10 common tasks and 31 metrics. Our evaluation results reveal that the fine-tuned LLMs exhibit enhanced comprehension and generative capabilities in Vietnamese. Moreover, our analysis indicates that models with more parameters can introduce more biases and uncalibrated outputs and the key factor influencing LLM performance is the quality of the training or fine-tuning datasets. These insights underscore the significance of meticulous fine-tuning with high-quality datasets in enhancing LLM performance.

Title: Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustment

• Authors: Congzhi Zhang, Linhai Zhang, Deyu Zhou, Guoqiang Xu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02738
• Pdf URL: https://arxiv.org/pdf/2403.02738
• Copy Paste: [[2403.02738]] Causal Prompting: Debiasing Large Language Model Prompting based on Front-Door Adjustment(https://arxiv.org/abs/2403.02738)
• Keywords: in-context
• Abstract: Despite the significant achievements of existing prompting methods such as in-context learning and chain-of-thought for large language models (LLMs), they still face challenges of various biases. Traditional debiasing methods primarily focus on the model training stage, including data augmentation-based and reweight-based approaches, with the limitations of addressing the complex biases of LLMs. To address such limitations, the causal relationship behind the prompting methods is uncovered using a structural causal model, and a novel causal prompting method based on front-door adjustment is proposed to effectively mitigate the bias of LLMs. In specific, causal intervention is implemented by designing the prompts without accessing the parameters and logits of LLMs.The chain-of-thoughts generated by LLMs are employed as the mediator variable and the causal effect between the input prompt and the output answers is calculated through front-door adjustment to mitigate model biases. Moreover, to obtain the representation of the samples precisely and estimate the causal effect more accurately, contrastive learning is used to fine-tune the encoder of the samples by aligning the space of the encoder with the LLM. Experimental results show that the proposed causal prompting approach achieves excellent performance on 3 natural language processing datasets on both open-source and closed-source LLMs.

Title: Self-adaptive Traffic Anomaly Detection System for IoT Smart Home Environments

• Authors: Naoto Watanabe (1), Taku Yamazaki (1), Takumi Miyoshi (1), Ryo Yamamoto (2), Masataka Nakahara (3), Norihiro Okui (3), Ayumu Kubota (3) ((1) Shibaura Institute of Technology, (2) The University of Electro-Communications, (3) KDDI Research, Inc.)
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.02744
• Pdf URL: https://arxiv.org/pdf/2403.02744
• Copy Paste: [[2403.02744]] Self-adaptive Traffic Anomaly Detection System for IoT Smart Home Environments(https://arxiv.org/abs/2403.02744)
• Keywords: anomaly
• Abstract: With the growth of internet of things (IoT) devices, cyberattacks, such as distributed denial of service, that exploit vulnerable devices infected with malware have increased. Therefore, vendors and users must keep their device firmware updated to eliminate vulnerabilities and quickly handle unknown cyberattacks. However, it is difficult for both vendors and users to continually keep the devices safe because vendors must provide updates quickly and the users must continuously manage the conditions of all deployed devices. Therefore, to ensure security, it is necessary for a system to adapt autonomously to changes in cyberattacks. In addition, it is important to consider network-side security that detects and filters anomalous traffic at the gateway to comprehensively protect those devices. This paper proposes a self-adaptive anomaly detection system for IoT traffic, including unknown attacks. The proposed system comprises a honeypot server and a gateway. The honeypot server continuously captures traffic and adaptively generates an anomaly detection model using real-time captured traffic. Thereafter, the gateway uses the generated model to detect anomalous traffic. Thus, the proposed system can adapt to unknown attacks to reflect pattern changes in anomalous traffic based on real-time captured traffic. Three experiments were conducted to evaluate the proposed system: a virtual experiment using pre-captured traffic from various regions across the world, a demonstration experiment using real-time captured traffic, and a virtual experiment using a public dataset containing the traffic generated by malware. The experimental results indicate that a system adaptable in real time to evolving cyberattacks is a novel approach for ensuring the comprehensive security of IoT devices against both known and unknown attacks.

Title: Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications

• Authors: Stav Cohen, Ron Bitton, Ben Nassi
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2403.02817
• Pdf URL: https://arxiv.org/pdf/2403.02817
• Copy Paste: [[2403.02817]] Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications(https://arxiv.org/abs/2403.02817)
• Keywords: generative
• Abstract: In the past year, numerous companies have incorporated Generative AI (GenAI) capabilities into new and existing applications, forming interconnected Generative AI (GenAI) ecosystems consisting of semi/fully autonomous agents powered by GenAI services. While ongoing research highlighted risks associated with the GenAI layer of agents (e.g., dialog poisoning, membership inference, prompt leaking, jailbreaking), a critical question emerges: Can attackers develop malware to exploit the GenAI component of an agent and launch cyber-attacks on the entire GenAI ecosystem? This paper introduces Morris II, the first worm designed to target GenAI ecosystems through the use of adversarial self-replicating prompts. The study demonstrates that attackers can insert such prompts into inputs that, when processed by GenAI models, prompt the model to replicate the input as output (replication), engaging in malicious activities (payload). Additionally, these inputs compel the agent to deliver them (propagate) to new agents by exploiting the connectivity within the GenAI ecosystem. We demonstrate the application of Morris II against GenAIpowered email assistants in two use cases (spamming and exfiltrating personal data), under two settings (black-box and white-box accesses), using two types of input data (text and images). The worm is tested against three different GenAI models (Gemini Pro, ChatGPT 4.0, and LLaVA), and various factors (e.g., propagation rate, replication, malicious activity) influencing the performance of the worm are evaluated.

Title: Tuning-Free Noise Rectification for High Fidelity Image-to-Video Generation

• Authors: Weijie Li, Litong Gong, Yiran Zhu, Fanda Fan, Biao Wang, Tiezheng Ge, Bo Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02827
• Pdf URL: https://arxiv.org/pdf/2403.02827
• Copy Paste: [[2403.02827]] Tuning-Free Noise Rectification for High Fidelity Image-to-Video Generation(https://arxiv.org/abs/2403.02827)
• Keywords: diffusion
• Abstract: Image-to-video (I2V) generation tasks always suffer from keeping high fidelity in the open domains. Traditional image animation techniques primarily focus on specific domains such as faces or human poses, making them difficult to generalize to open domains. Several recent I2V frameworks based on diffusion models can generate dynamic content for open domain images but fail to maintain fidelity. We found that two main factors of low fidelity are the loss of image details and the noise prediction biases during the denoising process. To this end, we propose an effective method that can be applied to mainstream video diffusion models. This method achieves high fidelity based on supplementing more precise image information and noise rectification. Specifically, given a specified image, our method first adds noise to the input image latent to keep more details, then denoises the noisy latent with proper rectification to alleviate the noise prediction biases. Our method is tuning-free and plug-and-play. The experimental results demonstrate the effectiveness of our approach in improving the fidelity of generated videos. For more image-to-video generated results, please refer to the project website: https://noise-rectification.github.io.

Title: FLGuard: Byzantine-Robust Federated Learning via Ensemble of Contrastive Models

• Authors: Younghan Lee, Yungi Cho, Woorim Han, Ho Bae, Yunheung Paek
• Subjects: cs.LG, cs.AI, cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.02846
• Pdf URL: https://arxiv.org/pdf/2403.02846
• Copy Paste: [[2403.02846]] FLGuard: Byzantine-Robust Federated Learning via Ensemble of Contrastive Models(https://arxiv.org/abs/2403.02846)
• Keywords: self-supervised
• Abstract: Federated Learning (FL) thrives in training a global model with numerous clients by only sharing the parameters of their local models trained with their private training datasets. Therefore, without revealing the private dataset, the clients can obtain a deep learning (DL) model with high performance. However, recent research proposed poisoning attacks that cause a catastrophic loss in the accuracy of the global model when adversaries, posed as benign clients, are present in a group of clients. Therefore, recent studies suggested byzantine-robust FL methods that allow the server to train an accurate global model even with the adversaries present in the system. However, many existing methods require the knowledge of the number of malicious clients or the auxiliary (clean) dataset or the effectiveness reportedly decreased hugely when the private dataset was non-independently and identically distributed (non-IID). In this work, we propose FLGuard, a novel byzantine-robust FL method that detects malicious clients and discards malicious local updates by utilizing the contrastive learning technique, which showed a tremendous improvement as a self-supervised learning method. With contrastive models, we design FLGuard as an ensemble scheme to maximize the defensive capability. We evaluate FLGuard extensively under various poisoning attacks and compare the accuracy of the global model with existing byzantine-robust FL methods. FLGuard outperforms the state-of-the-art defense methods in most cases and shows drastic improvement, especially in non-IID settings. https://github.com/201younghanlee/FLGuard

Title: Enhancing Conceptual Understanding in Multimodal Contrastive Learning through Hard Negative Samples

• Authors: Philipp J. Rösch, Norbert Oswald, Michaela Geierhos, Jindřich Libovický
• Subjects: cs.CV, cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.02875
• Pdf URL: https://arxiv.org/pdf/2403.02875
• Copy Paste: [[2403.02875]] Enhancing Conceptual Understanding in Multimodal Contrastive Learning through Hard Negative Samples(https://arxiv.org/abs/2403.02875)
• Keywords: generative
• Abstract: Current multimodal models leveraging contrastive learning often face limitations in developing fine-grained conceptual understanding. This is due to random negative samples during pretraining, causing almost exclusively very dissimilar concepts to be compared in the loss function. Consequently, the models struggle with fine-grained semantic differences. To address this problem, we introduce a novel pretraining method incorporating synthetic hard negative text examples. The hard negatives permute terms corresponding to visual concepts, leading to a more fine-grained visual and textual concept alignment. Further, we introduce InpaintCOCO, a new challenging dataset for assessing the fine-grained alignment of colors, objects, and sizes in vision-language models. We created the dataset using generative inpainting from COCO images by changing the visual concepts so that the images no longer match their original captions. Our results show significant improvements in fine-grained concept understanding across a wide range of vision-language datasets, including our InpaintCOCO dataset.

Title: Zero-LED: Zero-Reference Lighting Estimation Diffusion Model for Low-Light Image Enhancement

• Authors: Jinhong He, Minglong Xue, Zhipu Liu, Chengyun Song, Senming Zhong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02879
• Pdf URL: https://arxiv.org/pdf/2403.02879
• Copy Paste: [[2403.02879]] Zero-LED: Zero-Reference Lighting Estimation Diffusion Model for Low-Light Image Enhancement(https://arxiv.org/abs/2403.02879)
• Keywords: diffusion
• Abstract: Diffusion model-based low-light image enhancement methods rely heavily on paired training data, leading to limited extensive application. Meanwhile, existing unsupervised methods lack effective bridging capabilities for unknown degradation. To address these limitations, we propose a novel zero-reference lighting estimation diffusion model for low-light image enhancement called Zero-LED. It utilizes the stable convergence ability of diffusion models to bridge the gap between low-light domains and real normal-light domains and successfully alleviates the dependence on pairwise training data via zero-reference learning. Specifically, we first design the initial optimization network to preprocess the input image and implement bidirectional constraints between the diffusion model and the initial optimization network through multiple objective functions. Subsequently, the degradation factors of the real-world scene are optimized iteratively to achieve effective light enhancement. In addition, we explore a frequency-domain based and semantically guided appearance reconstruction module that encourages feature alignment of the recovered image at a fine-grained level and satisfies subjective expectations. Finally, extensive experiments demonstrate the superiority of our approach to other state-of-the-art methods and more significant generalization capabilities. We will open the source code upon acceptance of the paper.

Title: Enhancing the Rate-Distortion-Perception Flexibility of Learned Image Codecs with Conditional Diffusion Decoders

• Authors: Daniele Mari, Simone Milani
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.02887
• Pdf URL: https://arxiv.org/pdf/2403.02887
• Copy Paste: [[2403.02887]] Enhancing the Rate-Distortion-Perception Flexibility of Learned Image Codecs with Conditional Diffusion Decoders(https://arxiv.org/abs/2403.02887)
• Keywords: diffusion, generative
• Abstract: Learned image compression codecs have recently achieved impressive compression performances surpassing the most efficient image coding architectures. However, most approaches are trained to minimize rate and distortion which often leads to unsatisfactory visual results at low bitrates since perceptual metrics are not taken into account. In this paper, we show that conditional diffusion models can lead to promising results in the generative compression task when used as a decoder, and that, given a compressed representation, they allow creating new tradeoff points between distortion and perception at the decoder side based on the sampling method.

Title: Cross-Domain Image Conversion by CycleDM

• Authors: Sho Shimotsumagari, Shumpei Takezaki, Daichi Haraguchi, Seiichi Uchida
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02919
• Pdf URL: https://arxiv.org/pdf/2403.02919
• Copy Paste: [[2403.02919]] Cross-Domain Image Conversion by CycleDM(https://arxiv.org/abs/2403.02919)
• Keywords: diffusion
• Abstract: The purpose of this paper is to enable the conversion between machine-printed character images (i.e., font images) and handwritten character images through machine learning. For this purpose, we propose a novel unpaired image-to-image domain conversion method, CycleDM, which incorporates the concept of CycleGAN into the diffusion model. Specifically, CycleDM has two internal conversion models that bridge the denoising processes of two image domains. These conversion models are efficiently trained without explicit correspondence between the domains. By applying machine-printed and handwritten character images to the two modalities, CycleDM realizes the conversion between them. Our experiments for evaluating the converted images quantitatively and qualitatively found that ours performs better than other comparable approaches.

Title: RulePrompt: Weakly Supervised Text Classification with Prompting PLMs and Self-Iterative Logical Rules

• Authors: Miaomiao Li, Jiaqi Zhu, Yang Wang, Yi Yang, Yilin Li, Hongan Wang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02932
• Pdf URL: https://arxiv.org/pdf/2403.02932
• Copy Paste: [[2403.02932]] RulePrompt: Weakly Supervised Text Classification with Prompting PLMs and Self-Iterative Logical Rules(https://arxiv.org/abs/2403.02932)
• Keywords: self-supervised
• Abstract: Weakly supervised text classification (WSTC), also called zero-shot or dataless text classification, has attracted increasing attention due to its applicability in classifying a mass of texts within the dynamic and open Web environment, since it requires only a limited set of seed words (label names) for each category instead of labeled data. With the help of recently popular prompting Pre-trained Language Models (PLMs), many studies leveraged manually crafted and/or automatically identified verbalizers to estimate the likelihood of categories, but they failed to differentiate the effects of these category-indicative words, let alone capture their correlations and realize adaptive adjustments according to the unlabeled corpus. In this paper, in order to let the PLM effectively understand each category, we at first propose a novel form of rule-based knowledge using logical expressions to characterize the meanings of categories. Then, we develop a prompting PLM-based approach named RulePrompt for the WSTC task, consisting of a rule mining module and a rule-enhanced pseudo label generation module, plus a self-supervised fine-tuning module to make the PLM align with this task. Within this framework, the inaccurate pseudo labels assigned to texts and the imprecise logical rules associated with categories mutually enhance each other in an alternative manner. That establishes a self-iterative closed loop of knowledge (rule) acquisition and utilization, with seed words serving as the starting point. Extensive experiments validate the effectiveness and robustness of our approach, which markedly outperforms state-of-the-art weakly supervised methods. What is more, our approach yields interpretable category rules, proving its advantage in disambiguating easily-confused categories.

Title: Neural Image Compression with Text-guided Encoding for both Pixel-level and Perceptual Fidelity

• Authors: Hagyeong Lee, Minkyu Kim, Jun-Hyuk Kim, Seungeon Kim, Dokwan Oh, Jaeho Lee
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02944
• Pdf URL: https://arxiv.org/pdf/2403.02944
• Copy Paste: [[2403.02944]] Neural Image Compression with Text-guided Encoding for both Pixel-level and Perceptual Fidelity(https://arxiv.org/abs/2403.02944)
• Keywords: generative
• Abstract: Recent advances in text-guided image compression have shown great potential to enhance the perceptual quality of reconstructed images. These methods, however, tend to have significantly degraded pixel-wise fidelity, limiting their practicality. To fill this gap, we develop a new text-guided image compression algorithm that achieves both high perceptual and pixel-wise fidelity. In particular, we propose a compression framework that leverages text information mainly by text-adaptive encoding and training with joint image-text loss. By doing so, we avoid decoding based on text-guided generative models -- known for high generative diversity -- and effectively utilize the semantic information of text at a global level. Experimental results on various datasets show that our method can achieve high pixel-level and perceptual quality, with either human- or machine-generated captions. In particular, our method outperforms all baselines in terms of LPIPS, with some room for even more improvements when we use more carefully generated captions.

Title: Benchmarking the Text-to-SQL Capability of Large Language Models: A Comprehensive Evaluation

• Authors: Bin Zhang, Yuxiao Ye, Guoqing Du, Xiaoru Hu, Zhishuai Li, Sun Yang, Chi Harold Liu, Rui Zhao, Ziyue Li, Hangyu Mao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02951
• Pdf URL: https://arxiv.org/pdf/2403.02951
• Copy Paste: [[2403.02951]] Benchmarking the Text-to-SQL Capability of Large Language Models: A Comprehensive Evaluation(https://arxiv.org/abs/2403.02951)
• Keywords: in-context
• Abstract: Large Language Models (LLMs) have emerged as a powerful tool in advancing the Text-to-SQL task, significantly outperforming traditional methods. Nevertheless, as a nascent research field, there is still no consensus on the optimal prompt templates and design frameworks. Additionally, existing benchmarks inadequately explore the performance of LLMs across the various sub-tasks of the Text-to-SQL process, which hinders the assessment of LLMs' cognitive capabilities and the optimization of LLM-based solutions.To address the aforementioned issues, we firstly construct a new dataset designed to mitigate the risk of overfitting in LLMs. Then we formulate five evaluation tasks to comprehensively assess the performance of diverse methods across various LLMs throughout the Text-to-SQL process.Our study highlights the performance disparities among LLMs and proposes optimal in-context learning solutions tailored to each task. These findings offer valuable insights for enhancing the development of LLM-based Text-to-SQL systems.

Title: On the Asymptotic Mean Square Error Optimality of Diffusion Probabilistic Models

• Authors: Benedikt Fesl, Benedikt Böck, Florian Strasser, Michael Baur, Michael Joham, Wolfgang Utschick
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.02957
• Pdf URL: https://arxiv.org/pdf/2403.02957
• Copy Paste: [[2403.02957]] On the Asymptotic Mean Square Error Optimality of Diffusion Probabilistic Models(https://arxiv.org/abs/2403.02957)
• Keywords: diffusion
• Abstract: Diffusion probabilistic models (DPMs) have recently shown great potential for denoising tasks. Despite their practical utility, there is a notable gap in their theoretical understanding. This paper contributes novel theoretical insights by rigorously proving the asymptotic convergence of a specific DPM denoising strategy to the mean square error (MSE)-optimal conditional mean estimator (CME) over a large number of diffusion steps. The studied DPM-based denoiser shares the training procedure of DPMs but distinguishes itself by forwarding only the conditional mean during the reverse inference process after training. We highlight the unique perspective that DPMs are composed of an asymptotically optimal denoiser while simultaneously inheriting a powerful generator by switching re-sampling in the reverse process on and off. The theoretical findings are validated by numerical results.

Title: ChatGPT and biometrics: an assessment of face recognition, gender detection, and age estimation capabilities

• Authors: Ahmad Hassanpour, Yasamin Kowsari, Hatef Otroshi Shahreza, Bian Yang, Sebastien Marcel
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02965
• Pdf URL: https://arxiv.org/pdf/2403.02965
• Copy Paste: [[2403.02965]] ChatGPT and biometrics: an assessment of face recognition, gender detection, and age estimation capabilities(https://arxiv.org/abs/2403.02965)
• Keywords: foundation model
• Abstract: This paper explores the application of large language models (LLMs), like ChatGPT, for biometric tasks. We specifically examine the capabilities of ChatGPT in performing biometric-related tasks, with an emphasis on face recognition, gender detection, and age estimation. Since biometrics are considered as sensitive information, ChatGPT avoids answering direct prompts, and thus we crafted a prompting strategy to bypass its safeguard and evaluate the capabilities for biometrics tasks. Our study reveals that ChatGPT recognizes facial identities and differentiates between two facial images with considerable accuracy. Additionally, experimental results demonstrate remarkable performance in gender detection and reasonable accuracy for the age estimation tasks. Our findings shed light on the promising potentials in the application of LLMs and foundation models for biometrics.

Title: Recall-Oriented Continual Learning with Generative Adversarial Meta-Model

• Authors: Haneol Kang, Dong-Wan Choi
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.03082
• Pdf URL: https://arxiv.org/pdf/2403.03082
• Copy Paste: [[2403.03082]] Recall-Oriented Continual Learning with Generative Adversarial Meta-Model(https://arxiv.org/abs/2403.03082)
• Keywords: generative
• Abstract: The stability-plasticity dilemma is a major challenge in continual learning, as it involves balancing the conflicting objectives of maintaining performance on previous tasks while learning new tasks. In this paper, we propose the recall-oriented continual learning framework to address this challenge. Inspired by the human brain's ability to separate the mechanisms responsible for stability and plasticity, our framework consists of a two-level architecture where an inference network effectively acquires new knowledge and a generative network recalls past knowledge when necessary. In particular, to maximize the stability of past knowledge, we investigate the complexity of knowledge depending on different representations, and thereby introducing generative adversarial meta-model (GAMM) that incrementally learns task-specific parameters instead of input data samples of the task. Through our experiments, we show that our framework not only effectively learns new knowledge without any disruption but also achieves high stability of previous knowledge in both task-aware and task-agnostic learning scenarios. Our code is available at: https://github.com/bigdata-inha/recall-oriented-cl-framework.

Title: NRDF: Neural Riemannian Distance Fields for Learning Articulated Pose Priors

• Authors: Yannan He, Garvita Tiwari, Tolga Birdal, Jan Eric Lenssen, Gerard Pons-Moll
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.03122
• Pdf URL: https://arxiv.org/pdf/2403.03122
• Copy Paste: [[2403.03122]] NRDF: Neural Riemannian Distance Fields for Learning Articulated Pose Priors(https://arxiv.org/abs/2403.03122)
• Keywords: generative
• Abstract: Faithfully modeling the space of articulations is a crucial task that allows recovery and generation of realistic poses, and remains a notorious challenge. To this end, we introduce Neural Riemannian Distance Fields (NRDFs), data-driven priors modeling the space of plausible articulations, represented as the zero-level-set of a neural field in a high-dimensional product-quaternion space. To train NRDFs only on positive examples, we introduce a new sampling algorithm, ensuring that the geodesic distances follow a desired distribution, yielding a principled distance field learning paradigm. We then devise a projection algorithm to map any random pose onto the level-set by an adaptive-step Riemannian optimizer, adhering to the product manifold of joint rotations at all times. NRDFs can compute the Riemannian gradient via backpropagation and by mathematical analogy, are related to Riemannian flow matching, a recent generative model. We conduct a comprehensive evaluation of NRDF against other pose priors in various downstream tasks, i.e., pose generation, image-based pose estimation, and solving inverse kinematics, highlighting NRDF's superior performance. Besides humans, NRDF's versatility extends to hand and animal poses, as it can effectively represent any articulation.

Title: Dual Mean-Teacher: An Unbiased Semi-Supervised Framework for Audio-Visual Source Localization

• Authors: Yuxin Guo, Shijie Ma, Hu Su, Zhiqing Wang, Yuhao Zhao, Wei Zou, Siyang Sun, Yun Zheng
• Subjects: cs.CV, cs.LG, cs.MM, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2403.03145
• Pdf URL: https://arxiv.org/pdf/2403.03145
• Copy Paste: [[2403.03145]] Dual Mean-Teacher: An Unbiased Semi-Supervised Framework for Audio-Visual Source Localization(https://arxiv.org/abs/2403.03145)
• Keywords: self-supervised
• Abstract: Audio-Visual Source Localization (AVSL) aims to locate sounding objects within video frames given the paired audio clips. Existing methods predominantly rely on self-supervised contrastive learning of audio-visual correspondence. Without any bounding-box annotations, they struggle to achieve precise localization, especially for small objects, and suffer from blurry boundaries and false positives. Moreover, the naive semi-supervised method is poor in fully leveraging the information of abundant unlabeled data. In this paper, we propose a novel semi-supervised learning framework for AVSL, namely Dual Mean-Teacher (DMT), comprising two teacher-student structures to circumvent the confirmation bias issue. Specifically, two teachers, pre-trained on limited labeled data, are employed to filter out noisy samples via the consensus between their predictions, and then generate high-quality pseudo-labels by intersecting their confidence maps. The sufficient utilization of both labeled and unlabeled data and the proposed unbiased framework enable DMT to outperform current state-of-the-art methods by a large margin, with CIoU of 90.4% and 48.8% on Flickr-SoundNet and VGG-Sound Source, obtaining 8.9%, 9.6% and 4.6%, 6.4% improvements over self- and semi-supervised methods respectively, given only 3% positional-annotations. We also extend our framework to some existing AVSL methods and consistently boost their performance.

Title: Design2Code: How Far Are We From Automating Front-End Engineering?

• Authors: Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, Diyi Yang
• Subjects: cs.CL, cs.CV, cs.CY
• Abstract URL: https://arxiv.org/abs/2403.03163
• Pdf URL: https://arxiv.org/pdf/2403.03163
• Copy Paste: [[2403.03163]] Design2Code: How Far Are We From Automating Front-End Engineering?(https://arxiv.org/abs/2403.03163)
• Keywords: generative
• Abstract: Generative AI has made rapid advancements in recent years, achieving unprecedented capabilities in multimodal understanding and code generation. This can enable a new paradigm of front-end development, in which multimodal LLMs might directly convert visual designs into code implementations. In this work, we formalize this as a Design2Code task and conduct comprehensive benchmarking. Specifically, we manually curate a benchmark of 484 diverse real-world webpages as test cases and develop a set of automatic evaluation metrics to assess how well current multimodal LLMs can generate the code implementations that directly render into the given reference webpages, given the screenshots as input. We also complement automatic metrics with comprehensive human evaluations. We develop a suite of multimodal prompting methods and show their effectiveness on GPT-4V and Gemini Pro Vision. We further finetune an open-source Design2Code-18B model that successfully matches the performance of Gemini Pro Vision. Both human evaluation and automatic metrics show that GPT-4V performs the best on this task compared to other models. Moreover, annotators think GPT-4V generated webpages can replace the original reference webpages in 49% of cases in terms of visual appearance and content; and perhaps surprisingly, in 64% of cases GPT-4V generated webpages are considered better than the original reference webpages. Our fine-grained break-down metrics indicate that open-source models mostly lag in recalling visual elements from the input webpages and in generating correct layout designs, while aspects like text content and coloring can be drastically improved with proper finetuning.

Title: PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset

• Authors: Arda Uzunoğlu, Abdalfatah Rashid Safa, Gözde Gül Şahin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.03167
• Pdf URL: https://arxiv.org/pdf/2403.03167
• Copy Paste: [[2403.03167]] PARADISE: Evaluating Implicit Planning Skills of Language Models with Procedural Warnings and Tips Dataset(https://arxiv.org/abs/2403.03167)
• Keywords: generative
• Abstract: Recently, there has been growing interest within the community regarding whether large language models are capable of planning or executing plans. However, most prior studies use LLMs to generate high-level plans for simplified scenarios lacking linguistic complexity and domain diversity, limiting analysis of their planning abilities. These setups constrain evaluation methods (e.g., predefined action space), architectural choices (e.g., only generative models), and overlook the linguistic nuances essential for realistic analysis. To tackle this, we present PARADISE, an abductive reasoning task using Q\&A format on practical procedural text sourced from wikiHow. It involves warning and tip inference tasks directly associated with goals, excluding intermediary steps, with the aim of testing the ability of the models to infer implicit knowledge of the plan solely from the given goal. Our experiments, utilizing fine-tuned language models and zero-shot prompting, reveal the effectiveness of task-specific small models over large language models in most scenarios. Despite advancements, all models fall short of human performance. Notably, our analysis uncovers intriguing insights, such as variations in model behavior with dropped keywords, struggles of BERT-family and GPT-4 with physical and abstract goals, and the proposed tasks offering valuable prior knowledge for other unseen procedural tasks. The PARADISE dataset and associated resources are publicly available for further research exploration with https://github.com/GGLAB-KU/paradise.

Title: Behavior Generation with Latent Actions

• Authors: Seungjae Lee, Yibin Wang, Haritheja Etukuru, H. Jin Kim, Nur Muhammad Mahi Shafiullah, Lerrel Pinto
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.03181
• Pdf URL: https://arxiv.org/pdf/2403.03181
• Copy Paste: [[2403.03181]] Behavior Generation with Latent Actions(https://arxiv.org/abs/2403.03181)
• Keywords: diffusion, generative
• Abstract: Generative modeling of complex behaviors from labeled datasets has been a longstanding problem in decision making. Unlike language or image generation, decision making requires modeling actions - continuous-valued vectors that are multimodal in their distribution, potentially drawn from uncurated sources, where generation errors can compound in sequential prediction. A recent class of models called Behavior Transformers (BeT) addresses this by discretizing actions using k-means clustering to capture different modes. However, k-means struggles to scale for high-dimensional action spaces or long sequences, and lacks gradient information, and thus BeT suffers in modeling long-range actions. In this work, we present Vector-Quantized Behavior Transformer (VQ-BeT), a versatile model for behavior generation that handles multimodal action prediction, conditional generation, and partial observations. VQ-BeT augments BeT by tokenizing continuous actions with a hierarchical vector quantization module. Across seven environments including simulated manipulation, autonomous driving, and robotics, VQ-BeT improves on state-of-the-art models such as BeT and Diffusion Policies. Importantly, we demonstrate VQ-BeT's improved ability to capture behavior modes while accelerating inference speed 5x over Diffusion Policies. Videos and code can be found https://sjlee.cc/vq-bet

Title: How Well Can Transformers Emulate In-context Newton's Method?

• Authors: Angeliki Giannou, Liu Yang, Tianhao Wang, Dimitris Papailiopoulos, Jason D. Lee
• Subjects: cs.LG, cs.AI, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.03183
• Pdf URL: https://arxiv.org/pdf/2403.03183
• Copy Paste: [[2403.03183]] How Well Can Transformers Emulate In-context Newton's Method?(https://arxiv.org/abs/2403.03183)
• Keywords: in-context
• Abstract: Transformer-based models have demonstrated remarkable in-context learning capabilities, prompting extensive research into its underlying mechanisms. Recent studies have suggested that Transformers can implement first-order optimization algorithms for in-context learning and even second order ones for the case of linear regression. In this work, we study whether Transformers can perform higher order optimization methods, beyond the case of linear regression. We establish that linear attention Transformers with ReLU layers can approximate second order optimization algorithms for the task of logistic regression and achieve $\epsilon$ error with only a logarithmic to the error more layers. As a by-product we demonstrate the ability of even linear attention-only Transformers in implementing a single step of Newton's iteration for matrix inversion with merely two layers. These results suggest the ability of the Transformer architecture to implement complex algorithms, beyond gradient descent.

Title: MAGID: An Automated Pipeline for Generating Synthetic Multi-modal Datasets

• Authors: Hossein Aboutalebi, Hwanjun Song, Yusheng Xie, Arshit Gupta, Justin Sun, Hang Su, Igor Shalyminov, Nikolaos Pappas, Siffi Singh, Saab Mansour
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.03194
• Pdf URL: https://arxiv.org/pdf/2403.03194
• Copy Paste: [[2403.03194]] MAGID: An Automated Pipeline for Generating Synthetic Multi-modal Datasets(https://arxiv.org/abs/2403.03194)
• Keywords: diffusion
• Abstract: Development of multimodal interactive systems is hindered by the lack of rich, multimodal (text, images) conversational data, which is needed in large quantities for LLMs. Previous approaches augment textual dialogues with retrieved images, posing privacy, diversity, and quality constraints. In this work, we introduce \textbf{M}ultimodal \textbf{A}ugmented \textbf{G}enerative \textbf{I}mages \textbf{D}ialogues (MAGID), a framework to augment text-only dialogues with diverse and high-quality images. Subsequently, a diffusion model is applied to craft corresponding images, ensuring alignment with the identified text. Finally, MAGID incorporates an innovative feedback loop between an image description generation module (textual LLM) and image quality modules (addressing aesthetics, image-text matching, and safety), that work in tandem to generate high-quality and multi-modal dialogues. We compare MAGID to other SOTA baselines on three dialogue datasets, using automated and human evaluation. Our results show that MAGID is comparable to or better than baselines, with significant improvements in human evaluation, especially against retrieval baselines where the image database is small.

Title: Scaling Rectified Flow Transformers for High-Resolution Image Synthesis

• Authors: Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, Robin Rombach
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.03206
• Pdf URL: https://arxiv.org/pdf/2403.03206
• Copy Paste: [[2403.03206]] Scaling Rectified Flow Transformers for High-Resolution Image Synthesis(https://arxiv.org/abs/2403.03206)
• Keywords: diffusion, generative
• Abstract: Diffusion models create data from noise by inverting the forward paths of data towards noise and have emerged as a powerful generative modeling technique for high-dimensional, perceptual data such as images and videos. Rectified flow is a recent generative model formulation that connects data and noise in a straight line. Despite its better theoretical properties and conceptual simplicity, it is not yet decisively established as standard practice. In this work, we improve existing noise sampling techniques for training rectified flow models by biasing them towards perceptually relevant scales. Through a large-scale study, we demonstrate the superior performance of this approach compared to established diffusion formulations for high-resolution text-to-image synthesis. Additionally, we present a novel transformer-based architecture for text-to-image generation that uses separate weights for the two modalities and enables a bidirectional flow of information between image and text tokens, improving text comprehension, typography, and human preference ratings. We demonstrate that this architecture follows predictable scaling trends and correlates lower validation loss to improved text-to-image synthesis as measured by various metrics and human evaluations. Our largest models outperform state-of-the-art models, and we will make our experimental data, code, and model weights publicly available.

Title: Self-supervised 3D Patient Modeling with Multi-modal Attentive Fusion

• Authors: Meng Zheng, Benjamin Planche, Xuan Gong, Fan Yang, Terrence Chen, Ziyan Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.03217
• Pdf URL: https://arxiv.org/pdf/2403.03217
• Copy Paste: [[2403.03217]] Self-supervised 3D Patient Modeling with Multi-modal Attentive Fusion(https://arxiv.org/abs/2403.03217)
• Keywords: self-supervised
• Abstract: 3D patient body modeling is critical to the success of automated patient positioning for smart medical scanning and operating rooms. Existing CNN-based end-to-end patient modeling solutions typically require a) customized network designs demanding large amount of relevant training data, covering extensive realistic clinical scenarios (e.g., patient covered by sheets), which leads to suboptimal generalizability in practical deployment, b) expensive 3D human model annotations, i.e., requiring huge amount of manual effort, resulting in systems that scale poorly. To address these issues, we propose a generic modularized 3D patient modeling method consists of (a) a multi-modal keypoint detection module with attentive fusion for 2D patient joint localization, to learn complementary cross-modality patient body information, leading to improved keypoint localization robustness and generalizability in a wide variety of imaging (e.g., CT, MRI etc.) and clinical scenarios (e.g., heavy occlusions); and (b) a self-supervised 3D mesh regression module which does not require expensive 3D mesh parameter annotations to train, bringing immediate cost benefits for clinical deployment. We demonstrate the efficacy of the proposed method by extensive patient positioning experiments on both public and clinical data. Our evaluation results achieve superior patient positioning performance across various imaging modalities in real clinical scenarios.