2024-04-18

Title: Advancing Network Intrusion Detection: Integrating Graph Neural Networks with Scattering Transform and Node2Vec for Enhanced Anomaly Detection

• Authors: Abdeljalil Zoubir, Badr Missaoui
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.10800
• Pdf URL: https://arxiv.org/pdf/2404.10800
• Copy Paste: [[2404.10800]] Advancing Network Intrusion Detection: Integrating Graph Neural Networks with Scattering Transform and Node2Vec for Enhanced Anomaly Detection(https://arxiv.org/abs/2404.10800)
• Keywords: anomaly
• Abstract: In this paper, we present two novel methods in Network Intrusion Detection Systems (NIDS) using Graph Neural Networks (GNNs). The first approach, Scattering Transform with E-GraphSAGE (STEG), utilizes the scattering transform to conduct multi-resolution analysis of edge feature vectors. This provides a detailed representation that is essential for identifying subtle anomalies in network traffic. The second approach improves node representation by initiating with Node2Vec, diverging from standard methods of using uniform values, thereby capturing a more accurate and holistic network picture. Our methods have shown significant improvements in performance compared to existing state-of-the-art methods in benchmark NIDS datasets.

Title: Incubating Text Classifiers Following User Instruction with Nothing but LLM

• Authors: Letian Peng, Jingbo Shang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.10877
• Pdf URL: https://arxiv.org/pdf/2404.10877
• Copy Paste: [[2404.10877]] Incubating Text Classifiers Following User Instruction with Nothing but LLM(https://arxiv.org/abs/2404.10877)
• Keywords: in-context
• Abstract: In this paper, we aim to generate text classification data given arbitrary class definitions (i.e., user instruction), so one can train a small text classifier without any human annotation or raw corpus. Compared with pioneer attempts, our proposed Incubator is the first framework that can handle complicated and even mutually dependent classes (e.g., "TED Talk given by Educator" and "Other"). Specifically, Incubator is an LLM firstly tuned on the instruction-to-data mappings that we obtained from classification datasets and descriptions on HuggingFace together with in-context augmentation by GPT-4. We then refine Incubator by learning on the cluster centers of semantic textual embeddings to emphasize the uniformity and semantic diversity in generations. We compare Incubator on various classification tasks with strong baselines such as direct LLM-based inference and training data generation by prompt engineering. Experiments show Incubator is able to (1) perform well on traditional benchmarks, (2) take label dependency and user preference into consideration, and (3) enable logical text mining by incubating multiple classifiers.

Title: Search Beyond Queries: Training Smaller Language Models for Web Interactions via Reinforcement Learning

• Authors: Moghis Fereidouni, A.B. Siddique
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.10887
• Pdf URL: https://arxiv.org/pdf/2404.10887
• Copy Paste: [[2404.10887]] Search Beyond Queries: Training Smaller Language Models for Web Interactions via Reinforcement Learning(https://arxiv.org/abs/2404.10887)
• Keywords: in-context
• Abstract: Traditional search systems focus on query formulation for effective results but face challenges in scenarios such as product searches where crucial product details (e.g., size, color) remain concealed until users visit specific product pages. This highlights the need for intelligent web navigation agents capable of formulating queries and navigating web pages according to users' high-level intents. In response to this need, this work introduces a Grounded Language Agent for Intelligent Web Interactions, called GLAINTEL. Drawing upon advancements in language modeling and reinforcement learning, GLAINTEL investigates the efficacy of transformer-based models in enhancing the search capabilities of interactive web environments. Given the dynamic action space for each state in web navigation, GLAINTEL employs the Flan-T5 architecture and incorporates language modeling and value estimation heads. This work focuses on training smaller language models as agents across various scenarios, systematically evaluating the impact of human demonstrations on the training process. Specifically, we investigate scenarios where no human demonstrations are available and subsequently assess the effective utilization of such demonstrations. We also explore unsupervised domain adaptation for situations where demonstrations are confined to a specific domain. Experimental evaluations across diverse setups demonstrate the effectiveness of training agents in unsupervised settings, outperforming in-context learning-based approaches that employ larger models with up to 540 billion parameters. Surprisingly, behavioral cloning-based methods that straightforwardly use human demonstrations do not outperform unsupervised learning-based methods. Additionally, combining human demonstrations with Reinforcement Learning-based training yields results comparable to models utilizing GPT-4.

Title: Multi-Task Multi-Modal Self-Supervised Learning for Facial Expression Recognition

• Authors: Marah Halawa, Florian Blume, Pia Bideau, Martin Maier, Rasha Abdel Rahman, Olaf Hellwich
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.10904
• Pdf URL: https://arxiv.org/pdf/2404.10904
• Copy Paste: [[2404.10904]] Multi-Task Multi-Modal Self-Supervised Learning for Facial Expression Recognition(https://arxiv.org/abs/2404.10904)
• Keywords: self-supervised
• Abstract: Human communication is multi-modal; e.g., face-to-face interaction involves auditory signals (speech) and visual signals (face movements and hand gestures). Hence, it is essential to exploit multiple modalities when designing machine learning-based facial expression recognition systems. In addition, given the ever-growing quantities of video data that capture human facial expressions, such systems should utilize raw unlabeled videos without requiring expensive annotations. Therefore, in this work, we employ a multitask multi-modal self-supervised learning method for facial expression recognition from in-the-wild video data. Our model combines three self-supervised objective functions: First, a multi-modal contrastive loss, that pulls diverse data modalities of the same video together in the representation space. Second, a multi-modal clustering loss that preserves the semantic structure of input data in the representation space. Finally, a multi-modal data reconstruction loss. We conduct a comprehensive study on this multimodal multi-task self-supervised learning method on three facial expression recognition benchmarks. To that end, we examine the performance of learning through different combinations of self-supervised tasks on the facial expression recognition downstream task. Our model ConCluGen outperforms several multi-modal self-supervised and fully supervised baselines on the CMU-MOSEI dataset. Our results generally show that multi-modal self-supervision tasks offer large performance gains for challenging tasks such as facial expression recognition, while also reducing the amount of manual annotations required. We release our pre-trained models as well as source code publicly

Title: Residual Connections Harm Self-Supervised Abstract Feature Learning

• Authors: Xiao Zhang, Ruoxi Jiang, William Gao, Rebecca Willett, Michael Maire
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.10947
• Pdf URL: https://arxiv.org/pdf/2404.10947
• Copy Paste: [[2404.10947]] Residual Connections Harm Self-Supervised Abstract Feature Learning(https://arxiv.org/abs/2404.10947)
• Keywords: self-supervised
• Abstract: We demonstrate that adding a weighting factor to decay the strength of identity shortcuts within residual networks substantially improves semantic feature learning in the state-of-the-art self-supervised masked autoencoding (MAE) paradigm. Our modification to the identity shortcuts within a VIT-B/16 backbone of an MAE boosts linear probing accuracy on ImageNet from 67.3% to 72.3%. This significant gap suggests that, while residual connection structure serves an essential role in facilitating gradient propagation, it may have a harmful side effect of reducing capacity for abstract learning by virtue of injecting an echo of shallower representations into deeper layers. We ameliorate this downside via a fixed formula for monotonically decreasing the contribution of identity connections as layer depth increases. Our design promotes the gradual development of feature abstractions, without impacting network trainability. Analyzing the representations learned by our modified residual networks, we find correlation between low effective feature rank and downstream task performance.

Title: Many-Shot In-Context Learning

• Authors: Rishabh Agarwal, Avi Singh, Lei M. Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John D. Co-Reyes, Eric Chu, Feryal Behbahani, Aleksandra Faust, Hugo Larochelle
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.11018
• Pdf URL: https://arxiv.org/pdf/2404.11018
• Copy Paste: [[2404.11018]] Many-Shot In-Context Learning(https://arxiv.org/abs/2404.11018)
• Keywords: generative, in-context
• Abstract: Large language models (LLMs) excel at few-shot in-context learning (ICL) -- learning from a few examples provided in context at inference, without any weight updates. Newly expanded context windows allow us to investigate ICL with hundreds or thousands of examples -- the many-shot regime. Going from few-shot to many-shot, we observe significant performance gains across a wide variety of generative and discriminative tasks. While promising, many-shot ICL can be bottlenecked by the available amount of human-generated examples. To mitigate this limitation, we explore two new settings: Reinforced and Unsupervised ICL. Reinforced ICL uses model-generated chain-of-thought rationales in place of human examples. Unsupervised ICL removes rationales from the prompt altogether, and prompts the model only with domain-specific questions. We find that both Reinforced and Unsupervised ICL can be quite effective in the many-shot regime, particularly on complex reasoning tasks. Finally, we demonstrate that, unlike few-shot learning, many-shot learning is effective at overriding pretraining biases and can learn high-dimensional functions with numerical inputs. Our analysis also reveals the limitations of next-token prediction loss as an indicator of downstream ICL performance.

Title: ViLLM-Eval: A Comprehensive Evaluation Suite for Vietnamese Large Language Models

• Authors: Trong-Hieu Nguyen, Anh-Cuong Le, Viet-Cuong Nguyen
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11086
• Pdf URL: https://arxiv.org/pdf/2404.11086
• Copy Paste: [[2404.11086]] ViLLM-Eval: A Comprehensive Evaluation Suite for Vietnamese Large Language Models(https://arxiv.org/abs/2404.11086)
• Keywords: foundation model
• Abstract: The rapid advancement of large language models (LLMs) necessitates the development of new benchmarks to accurately assess their capabilities. To address this need for Vietnamese, this work aims to introduce ViLLM-Eval, the comprehensive evaluation suite designed to measure the advanced knowledge and reasoning abilities of foundation models within a Vietnamese context. ViLLM-Eval consists of multiple-choice questions and predict next word tasks spanning various difficulty levels and diverse disciplines, ranging from humanities to science and engineering. A thorough evaluation of the most advanced LLMs on ViLLM-Eval revealed that even the best performing models have significant room for improvement in understanding and responding to Vietnamese language tasks. ViLLM-Eval is believed to be instrumental in identifying key strengths and weaknesses of foundation models, ultimately promoting their development and enhancing their performance for Vietnamese users.

Title: LAPTOP-Diff: Layer Pruning and Normalized Distillation for Compressing Diffusion Models

• Authors: Dingkun Zhang, Sijia Li, Chen Chen, Qingsong Xie, Haonan Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11098
• Pdf URL: https://arxiv.org/pdf/2404.11098
• Copy Paste: [[2404.11098]] LAPTOP-Diff: Layer Pruning and Normalized Distillation for Compressing Diffusion Models(https://arxiv.org/abs/2404.11098)
• Keywords: diffusion
• Abstract: In the era of AIGC, the demand for low-budget or even on-device applications of diffusion models emerged. In terms of compressing the Stable Diffusion models (SDMs), several approaches have been proposed, and most of them leveraged the handcrafted layer removal methods to obtain smaller U-Nets, along with knowledge distillation to recover the network performance. However, such a handcrafting manner of layer removal is inefficient and lacks scalability and generalization, and the feature distillation employed in the retraining phase faces an imbalance issue that a few numerically significant feature loss terms dominate over others throughout the retraining process. To this end, we proposed the layer pruning and normalized distillation for compressing diffusion models (LAPTOP-Diff). We, 1) introduced the layer pruning method to compress SDM's U-Net automatically and proposed an effective one-shot pruning criterion whose one-shot performance is guaranteed by its good additivity property, surpassing other layer pruning and handcrafted layer removal methods, 2) proposed the normalized feature distillation for retraining, alleviated the imbalance issue. Using the proposed LAPTOP-Diff, we compressed the U-Nets of SDXL and SDM-v1.5 for the most advanced performance, achieving a minimal 4.0% decline in PickScore at a pruning ratio of 50% while the comparative methods' minimal PickScore decline is 8.2%. We will release our code.

Title: TiNO-Edit: Timestep and Noise Optimization for Robust Diffusion-Based Image Editing

• Authors: Sherry X. Chen, Yaron Vaxman, Elad Ben Baruch, David Asulin, Aviad Moreshet, Kuo-Chin Lien, Misha Sra, Pradeep Sen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11120
• Pdf URL: https://arxiv.org/pdf/2404.11120
• Copy Paste: [[2404.11120]] TiNO-Edit: Timestep and Noise Optimization for Robust Diffusion-Based Image Editing(https://arxiv.org/abs/2404.11120)
• Keywords: diffusion
• Abstract: Despite many attempts to leverage pre-trained text-to-image models (T2I) like Stable Diffusion (SD) for controllable image editing, producing good predictable results remains a challenge. Previous approaches have focused on either fine-tuning pre-trained T2I models on specific datasets to generate certain kinds of images (e.g., with a specific object or person), or on optimizing the weights, text prompts, and/or learning features for each input image in an attempt to coax the image generator to produce the desired result. However, these approaches all have shortcomings and fail to produce good results in a predictable and controllable manner. To address this problem, we present TiNO-Edit, an SD-based method that focuses on optimizing the noise patterns and diffusion timesteps during editing, something previously unexplored in the literature. With this simple change, we are able to generate results that both better align with the original images and reflect the desired result. Furthermore, we propose a set of new loss functions that operate in the latent domain of SD, greatly speeding up the optimization when compared to prior approaches, which operate in the pixel domain. Our method can be easily applied to variations of SD including Textual Inversion and DreamBooth that encode new concepts and incorporate them into the edited results. We present a host of image-editing capabilities enabled by our approach. Our code is publicly available at https://github.com/SherryXTChen/TiNO-Edit.

Title: Learning SO(3)-Invariant Semantic Correspondence via Local Shape Transform

• Authors: Chunghyun Park, Seungwook Sim, Jaesik Park, Minsu Cho
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11156
• Pdf URL: https://arxiv.org/pdf/2404.11156
• Copy Paste: [[2404.11156]] Learning SO(3)-Invariant Semantic Correspondence via Local Shape Transform(https://arxiv.org/abs/2404.11156)
• Keywords: self-supervised
• Abstract: Establishing accurate 3D correspondences between shapes stands as a pivotal challenge with profound implications for computer vision and robotics. However, existing self-supervised methods for this problem assume perfect input shape alignment, restricting their real-world applicability. In this work, we introduce a novel self-supervised Rotation-Invariant 3D correspondence learner with Local Shape Transform, dubbed RIST, that learns to establish dense correspondences between shapes even under challenging intra-class variations and arbitrary orientations. Specifically, RIST learns to dynamically formulate an SO(3)-invariant local shape transform for each point, which maps the SO(3)-equivariant global shape descriptor of the input shape to a local shape descriptor. These local shape descriptors are provided as inputs to our decoder to facilitate point cloud self- and cross-reconstruction. Our proposed self-supervised training pipeline encourages semantically corresponding points from different shapes to be mapped to similar local shape descriptors, enabling RIST to establish dense point-wise correspondences. RIST demonstrates state-of-the-art performances on 3D part label transfer and semantic keypoint transfer given arbitrarily rotated point cloud pairs, outperforming existing methods by significant margins.

Title: KI-GAN: Knowledge-Informed Generative Adversarial Networks for Enhanced Multi-Vehicle Trajectory Forecasting at Signalized Intersections

• Authors: Chuheng Wei, Guoyuan Wu, Matthew J. Barth, Amr Abdelraouf, Rohit Gupta, Kyungtae Han
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.11181
• Pdf URL: https://arxiv.org/pdf/2404.11181
• Copy Paste: [[2404.11181]] KI-GAN: Knowledge-Informed Generative Adversarial Networks for Enhanced Multi-Vehicle Trajectory Forecasting at Signalized Intersections(https://arxiv.org/abs/2404.11181)
• Keywords: generative
• Abstract: Reliable prediction of vehicle trajectories at signalized intersections is crucial to urban traffic management and autonomous driving systems. However, it presents unique challenges, due to the complex roadway layout at intersections, involvement of traffic signal controls, and interactions among different types of road users. To address these issues, we present in this paper a novel model called Knowledge-Informed Generative Adversarial Network (KI-GAN), which integrates both traffic signal information and multi-vehicle interactions to predict vehicle trajectories accurately. Additionally, we propose a specialized attention pooling method that accounts for vehicle orientation and proximity at intersections. Based on the SinD dataset, our KI-GAN model is able to achieve an Average Displacement Error (ADE) of 0.05 and a Final Displacement Error (FDE) of 0.12 for a 6-second observation and 6-second prediction cycle. When the prediction window is extended to 9 seconds, the ADE and FDE values are further reduced to 0.11 and 0.26, respectively. These results demonstrate the effectiveness of the proposed KI-GAN model in vehicle trajectory prediction under complex scenarios at signalized intersections, which represents a significant advancement in the target field.

Title: Position Engineering: Boosting Large Language Models through Positional Information Manipulation

• Authors: Zhiyuan He, Huiqiang Jiang, Zilong Wang, Yuqing Yang, Luna Qiu, Lili Qiu
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.11216
• Pdf URL: https://arxiv.org/pdf/2404.11216
• Copy Paste: [[2404.11216]] Position Engineering: Boosting Large Language Models through Positional Information Manipulation(https://arxiv.org/abs/2404.11216)
• Keywords: in-context
• Abstract: The performance of large language models (LLMs) is significantly influenced by the quality of the prompts provided. In response, researchers have developed enormous prompt engineering strategies aimed at modifying the prompt text to enhance task performance. In this paper, we introduce a novel technique termed position engineering, which offers a more efficient way to guide large language models. Unlike prompt engineering, which requires substantial effort to modify the text provided to LLMs, position engineering merely involves altering the positional information in the prompt without modifying the text itself. We have evaluated position engineering in two widely-used LLM scenarios: retrieval-augmented generation (RAG) and in-context learning (ICL). Our findings show that position engineering substantially improves upon the baseline in both cases. Position engineering thus represents a promising new strategy for exploiting the capabilities of large language models.

Title: In-Context Learning State Vector with Inner and Momentum Optimization

• Authors: Dongfang Li, Zhenyu Liu, Xinshuo Hu, Zetian Sun, Baotian Hu, Min Zhang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11225
• Pdf URL: https://arxiv.org/pdf/2404.11225
• Copy Paste: [[2404.11225]] In-Context Learning State Vector with Inner and Momentum Optimization(https://arxiv.org/abs/2404.11225)
• Keywords: in-context
• Abstract: Large Language Models (LLMs) have exhibited an impressive ability to perform In-Context Learning (ICL) from only a few examples. Recent works have indicated that the functions learned by ICL can be represented through compressed vectors derived from the transformer. However, the working mechanisms and optimization of these vectors are yet to be thoroughly explored. In this paper, we address this gap by presenting a comprehensive analysis of these compressed vectors, drawing parallels to the parameters trained with gradient descent, and introduce the concept of state vector. Inspired by the works on model soup and momentum-based gradient descent, we propose inner and momentum optimization methods that are applied to refine the state vector progressively as test-time adaptation. Moreover, we simulate state vector aggregation in the multiple example setting, where demonstrations comprising numerous examples are usually too lengthy for regular ICL, and further propose a divide-and-conquer aggregation method to address this challenge. We conduct extensive experiments using Llama-2 and GPT-J in both zero-shot setting and few-shot setting. The experimental results show that our optimization method effectively enhances the state vector and achieves the state-of-the-art performance on diverse tasks. Code is available at https://github.com/HITsz-TMG/ICL-State-Vector

Title: ONOT: a High-Quality ICAO-compliant Synthetic Mugshot Dataset

• Authors: Nicolò Di Domenico, Guido Borghi, Annalisa Franco, Davide Maltoni
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11236
• Pdf URL: https://arxiv.org/pdf/2404.11236
• Copy Paste: [[2404.11236]] ONOT: a High-Quality ICAO-compliant Synthetic Mugshot Dataset(https://arxiv.org/abs/2404.11236)
• Keywords: generative
• Abstract: Nowadays, state-of-the-art AI-based generative models represent a viable solution to overcome privacy issues and biases in the collection of datasets containing personal information, such as faces. Following this intuition, in this paper we introduce ONOT, a synthetic dataset specifically focused on the generation of high-quality faces in adherence to the requirements of the ISO/IEC 39794-5 standards that, following the guidelines of the International Civil Aviation Organization (ICAO), defines the interchange formats of face images in electronic Machine-Readable Travel Documents (eMRTD). The strictly controlled and varied mugshot images included in ONOT are useful in research fields related to the analysis of face images in eMRTD, such as Morphing Attack Detection and Face Quality Assessment. The dataset is publicly released, in combination with the generation procedure details in order to improve the reproducibility and enable future extensions.

Title: Optical Image-to-Image Translation Using Denoising Diffusion Models: Heterogeneous Change Detection as a Use Case

• Authors: João Gabriel Vinholi, Marco Chini, Anis Amziane, Renato Machado, Danilo Silva, Patrick Matgen
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11243
• Pdf URL: https://arxiv.org/pdf/2404.11243
• Copy Paste: [[2404.11243]] Optical Image-to-Image Translation Using Denoising Diffusion Models: Heterogeneous Change Detection as a Use Case(https://arxiv.org/abs/2404.11243)
• Keywords: diffusion
• Abstract: We introduce an innovative deep learning-based method that uses a denoising diffusion-based model to translate low-resolution images to high-resolution ones from different optical sensors while preserving the contents and avoiding undesired artifacts. The proposed method is trained and tested on a large and diverse data set of paired Sentinel-II and Planet Dove images. We show that it can solve serious image generation issues observed when the popular classifier-free guided Denoising Diffusion Implicit Model (DDIM) framework is used in the task of Image-to-Image Translation of multi-sensor optical remote sensing images and that it can generate large images with highly consistent patches, both in colors and in features. Moreover, we demonstrate how our method improves heterogeneous change detection results in two urban areas: Beirut, Lebanon, and Austin, USA. Our contributions are: i) a new training and testing algorithm based on denoising diffusion models for optical image translation; ii) a comprehensive image quality evaluation and ablation study; iii) a comparison with the classifier-free guided DDIM framework; and iv) change detection experiments on heterogeneous data.

Title: DACAD: Domain Adaptation Contrastive Learning for Anomaly Detection in Multivariate Time Series

• Authors: Zahra Zamanzadeh Darban, Geoffrey I. Webb, Mahsa Salehi
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11269
• Pdf URL: https://arxiv.org/pdf/2404.11269
• Copy Paste: [[2404.11269]] DACAD: Domain Adaptation Contrastive Learning for Anomaly Detection in Multivariate Time Series(https://arxiv.org/abs/2404.11269)
• Keywords: self-supervised, anomaly
• Abstract: Time series anomaly detection (TAD) faces a significant challenge due to the scarcity of labelled data, which hinders the development of accurate detection models. Unsupervised domain adaptation (UDA) addresses this challenge by leveraging a labelled dataset from a related domain to detect anomalies in a target dataset. Existing domain adaptation techniques assume that the number of anomalous classes does not change between the source and target domains. In this paper, we propose a novel Domain Adaptation Contrastive learning for Anomaly Detection in multivariate time series (DACAD) model to address this issue by combining UDA and contrastive representation learning. DACAD's approach includes an anomaly injection mechanism that introduces various types of synthetic anomalies, enhancing the model's ability to generalise across unseen anomalous classes in different domains. This method significantly broadens the model's adaptability and robustness. Additionally, we propose a supervised contrastive loss for the source domain and a self-supervised contrastive triplet loss for the target domain, improving comprehensive feature representation learning and extraction of domain-invariant features. Finally, an effective Centre-based Entropy Classifier (CEC) is proposed specifically for anomaly detection, facilitating accurate learning of normal boundaries in the source domain. Our extensive evaluation across multiple real-world datasets against leading models in time series anomaly detection and UDA underscores DACAD's effectiveness. The results validate DACAD's superiority in transferring knowledge across domains and its potential to mitigate the challenge of limited labelled data in time series anomaly detection.

Title: Closely Interactive Human Reconstruction with Proxemics and Physics-Guided Adaption

• Authors: Buzhen Huang, Chen Li, Chongyang Xu, Liang Pan, Yangang Wang, Gim Hee Lee
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11291
• Pdf URL: https://arxiv.org/pdf/2404.11291
• Copy Paste: [[2404.11291]] Closely Interactive Human Reconstruction with Proxemics and Physics-Guided Adaption(https://arxiv.org/abs/2404.11291)
• Keywords: diffusion
• Abstract: Existing multi-person human reconstruction approaches mainly focus on recovering accurate poses or avoiding penetration, but overlook the modeling of close interactions. In this work, we tackle the task of reconstructing closely interactive humans from a monocular video. The main challenge of this task comes from insufficient visual information caused by depth ambiguity and severe inter-person occlusion. In view of this, we propose to leverage knowledge from proxemic behavior and physics to compensate the lack of visual information. This is based on the observation that human interaction has specific patterns following the social proxemics. Specifically, we first design a latent representation based on Vector Quantised-Variational AutoEncoder (VQ-VAE) to model human interaction. A proxemics and physics guided diffusion model is then introduced to denoise the initial distribution. We design the diffusion model as dual branch with each branch representing one individual such that the interaction can be modeled via cross attention. With the learned priors of VQ-VAE and physical constraint as the additional information, our proposed approach is capable of estimating accurate poses that are also proxemics and physics plausible. Experimental results on Hi4D, 3DPW, and CHI3D demonstrate that our method outperforms existing approaches. The code is available at \url{https://github.com/boycehbz/HumanInteraction}.

Title: Exploring Key Point Analysis with Pairwise Generation and Graph Partitioning

• Authors: Xiao Li, Yong Jiang, Shen Huang, Pengjun Xie, Gong Cheng, Fei Huang
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.11384
• Pdf URL: https://arxiv.org/pdf/2404.11384
• Copy Paste: [[2404.11384]] Exploring Key Point Analysis with Pairwise Generation and Graph Partitioning(https://arxiv.org/abs/2404.11384)
• Keywords: generative
• Abstract: Key Point Analysis (KPA), the summarization of multiple arguments into a concise collection of key points, continues to be a significant and unresolved issue within the field of argument mining. Existing models adapt a two-stage pipeline of clustering arguments or generating key points for argument clusters. This approach rely on semantic similarity instead of measuring the existence of shared key points among arguments. Additionally, it only models the intra-cluster relationship among arguments, disregarding the inter-cluster relationship between arguments that do not share key points. To address these limitations, we propose a novel approach for KPA with pairwise generation and graph partitioning. Our objective is to train a generative model that can simultaneously provide a score indicating the presence of shared key point between a pair of arguments and generate the shared key point. Subsequently, to map generated redundant key points to a concise set of key points, we proceed to construct an arguments graph by considering the arguments as vertices, the generated key points as edges, and the scores as edge weights. We then propose a graph partitioning algorithm to partition all arguments sharing the same key points to the same subgraph. Notably, our experimental findings demonstrate that our proposed model surpasses previous models when evaluated on both the ArgKP and QAM datasets.

Title: Enhancing Data Privacy In Wireless Sensor Networks: Investigating Techniques And Protocols To Protect Privacy Of Data Transmitted Over Wireless Sensor Networks In Critical Applications Of Healthcare And National Security

• Authors: Akinsola Ahmed, Ejiofor Oluomachi, Akinde Abdullah, Njoku Tochukwu
• Subjects: cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2404.11388
• Pdf URL: https://arxiv.org/pdf/2404.11388
• Copy Paste: [[2404.11388]] Enhancing Data Privacy In Wireless Sensor Networks: Investigating Techniques And Protocols To Protect Privacy Of Data Transmitted Over Wireless Sensor Networks In Critical Applications Of Healthcare And National Security(https://arxiv.org/abs/2404.11388)
• Keywords: diffusion
• Abstract: The article discusses the emergence of Wireless Sensor Networks (WSNs) as a groundbreaking technology in data processing and communication. It outlines how WSNs, composed of dispersed autonomous sensors, are utilized to monitor physical and environmental factors, transmitting data wirelessly for analysis. The article explores various applications of WSNs in healthcare, national security, emergency response, and infrastructure monitoring, highlighting their roles in enhancing patient care, public health surveillance, border security, disaster management, and military operations. Additionally, it examines the foundational concepts of data privacy in WSNs, focusing on encryption techniques, authentication mechanisms, anonymization techniques, and access control mechanisms. The article also addresses vulnerabilities, threats, and challenges related to data privacy in healthcare and national security contexts, emphasizing regulatory compliance, ethical considerations, and socio-economic factors. Furthermore, it introduces the Diffusion of Innovation Theory as a framework for understanding the adoption of privacy-enhancing technologies in WSNs. Finally, the article reviews empirical studies demonstrating the efficacy of security solutions in preserving data privacy in WSNs, offering insights into advancements in safeguarding sensitive information.

Title: Neural Shrödinger Bridge Matching for Pansharpening

• Authors: Zihan Cao, Xiao Wu, Liang-Jian Deng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11416
• Pdf URL: https://arxiv.org/pdf/2404.11416
• Copy Paste: [[2404.11416]] Neural Shrödinger Bridge Matching for Pansharpening(https://arxiv.org/abs/2404.11416)
• Keywords: diffusion
• Abstract: Recent diffusion probabilistic models (DPM) in the field of pansharpening have been gradually gaining attention and have achieved state-of-the-art (SOTA) performance. In this paper, we identify shortcomings in directly applying DPMs to the task of pansharpening as an inverse problem: 1) initiating sampling directly from Gaussian noise neglects the low-resolution multispectral image (LRMS) as a prior; 2) low sampling efficiency often necessitates a higher number of sampling steps. We first reformulate pansharpening into the stochastic differential equation (SDE) form of an inverse problem. Building upon this, we propose a Schr\"odinger bridge matching method that addresses both issues. We design an efficient deep neural network architecture tailored for the proposed SB matching. In comparison to the well-established DL-regressive-based framework and the recent DPM framework, our method demonstrates SOTA performance with fewer sampling steps. Moreover, we discuss the relationship between SB matching and other methods based on SDEs and ordinary differential equations (ODEs), as well as its connection with optimal transport. Code will be available.

Title: Towards Highly Realistic Artistic Style Transfer via Stable Diffusion with Step-aware and Layer-aware Prompt

• Authors: Zhanjie Zhang, Quanwei Zhang, Huaizhong Lin, Wei Xing, Juncheng Mo, Shuaicheng Huang, Jinheng Xie, Guangyuan Li, Junsheng Luan, Lei Zhao, Dalong Zhang, Lixia Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11474
• Pdf URL: https://arxiv.org/pdf/2404.11474
• Copy Paste: [[2404.11474]] Towards Highly Realistic Artistic Style Transfer via Stable Diffusion with Step-aware and Layer-aware Prompt(https://arxiv.org/abs/2404.11474)
• Keywords: diffusion, generative
• Abstract: Artistic style transfer aims to transfer the learned artistic style onto an arbitrary content image, generating artistic stylized images. Existing generative adversarial network-based methods fail to generate highly realistic stylized images and always introduce obvious artifacts and disharmonious patterns. Recently, large-scale pre-trained diffusion models opened up a new way for generating highly realistic artistic stylized images. However, diffusion model-based methods generally fail to preserve the content structure of input content images well, introducing some undesired content structure and style patterns. To address the above problems, we propose a novel pre-trained diffusion-based artistic style transfer method, called LSAST, which can generate highly realistic artistic stylized images while preserving the content structure of input content images well, without bringing obvious artifacts and disharmonious style patterns. Specifically, we introduce a Step-aware and Layer-aware Prompt Space, a set of learnable prompts, which can learn the style information from the collection of artworks and dynamically adjusts the input images' content structure and style pattern. To train our prompt space, we propose a novel inversion method, called Step-ware and Layer-aware Prompt Inversion, which allows the prompt space to learn the style information of the artworks collection. In addition, we inject a pre-trained conditional branch of ControlNet into our LSAST, which further improved our framework's ability to maintain content structure. Extensive experiments demonstrate that our proposed method can generate more highly realistic artistic stylized images than the state-of-the-art artistic style transfer methods.

Title: AdaIR: Exploiting Underlying Similarities of Image Restoration Tasks with Adapters

• Authors: Hao-Wei Chen, Yu-Syuan Xu, Kelvin C.K. Chan, Hsien-Kai Kuo, Chun-Yi Lee, Ming-Hsuan Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11475
• Pdf URL: https://arxiv.org/pdf/2404.11475
• Copy Paste: [[2404.11475]] AdaIR: Exploiting Underlying Similarities of Image Restoration Tasks with Adapters(https://arxiv.org/abs/2404.11475)
• Keywords: self-supervised
• Abstract: Existing image restoration approaches typically employ extensive networks specifically trained for designated degradations. Despite being effective, such methods inevitably entail considerable storage costs and computational overheads due to the reliance on task-specific networks. In this work, we go beyond this well-established framework and exploit the inherent commonalities among image restoration tasks. The primary objective is to identify components that are shareable across restoration tasks and augment the shared components with modules specifically trained for individual tasks. Towards this goal, we propose AdaIR, a novel framework that enables low storage cost and efficient training without sacrificing performance. Specifically, a generic restoration network is first constructed through self-supervised pre-training using synthetic degradations. Subsequent to the pre-training phase, adapters are trained to adapt the pre-trained network to specific degradations. AdaIR requires solely the training of lightweight, task-specific modules, ensuring a more efficient storage and training regimen. We have conducted extensive experiments to validate the effectiveness of AdaIR and analyze the influence of the pre-training strategy on discovering shareable components. Extensive experimental results show that AdaIR achieves outstanding results on multi-task restoration while utilizing significantly fewer parameters (1.9 MB) and less training time (7 hours) for each restoration task. The source codes and trained models will be released.

Title: FedPFT: Federated Proxy Fine-Tuning of Foundation Models

• Authors: Zhaopeng Peng, Xiaoliang Fan, Yufan Chen, Zheng Wang, Shirui Pan, Chenglu Wen, Ruisheng Zhang, Cheng Wang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11536
• Pdf URL: https://arxiv.org/pdf/2404.11536
• Copy Paste: [[2404.11536]] FedPFT: Federated Proxy Fine-Tuning of Foundation Models(https://arxiv.org/abs/2404.11536)
• Keywords: foundation model
• Abstract: Adapting Foundation Models (FMs) for downstream tasks through Federated Learning (FL) emerges a promising strategy for protecting data privacy and valuable FMs. Existing methods fine-tune FM by allocating sub-FM to clients in FL, however, leading to suboptimal performance due to insufficient tuning and inevitable error accumulations of gradients. In this paper, we propose Federated Proxy Fine-Tuning (FedPFT), a novel method enhancing FMs adaptation in downstream tasks through FL by two key modules. First, the sub-FM construction module employs a layer-wise compression approach, facilitating comprehensive FM fine-tuning across all layers by emphasizing those crucial neurons. Second, the sub-FM alignment module conducts a two-step distillations-layer-level and neuron-level-before and during FL fine-tuning respectively, to reduce error of gradient by accurately aligning sub-FM with FM under theoretical guarantees. Experimental results on seven commonly used datasets (i.e., four text and three vision) demonstrate the superiority of FedPFT.

Title: SSDiff: Spatial-spectral Integrated Diffusion Model for Remote Sensing Pansharpening

• Authors: Yu Zhong, Xiao Wu, Liang-Jian Deng, Zihan Cao
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.11537
• Pdf URL: https://arxiv.org/pdf/2404.11537
• Copy Paste: [[2404.11537]] SSDiff: Spatial-spectral Integrated Diffusion Model for Remote Sensing Pansharpening(https://arxiv.org/abs/2404.11537)
• Keywords: diffusion
• Abstract: Pansharpening is a significant image fusion technique that merges the spatial content and spectral characteristics of remote sensing images to generate high-resolution multispectral images. Recently, denoising diffusion probabilistic models have been gradually applied to visual tasks, enhancing controllable image generation through low-rank adaptation (LoRA). In this paper, we introduce a spatial-spectral integrated diffusion model for the remote sensing pansharpening task, called SSDiff, which considers the pansharpening process as the fusion process of spatial and spectral components from the perspective of subspace decomposition. Specifically, SSDiff utilizes spatial and spectral branches to learn spatial details and spectral features separately, then employs a designed alternating projection fusion module (APFM) to accomplish the fusion. Furthermore, we propose a frequency modulation inter-branch module (FMIM) to modulate the frequency distribution between branches. The two components of SSDiff can perform favorably against the APFM when utilizing a LoRA-like branch-wise alternative fine-tuning method. It refines SSDiff to capture component-discriminating features more sufficiently. Finally, extensive experiments on four commonly used datasets, i.e., WorldView-3, WorldView-2, GaoFen-2, and QuickBird, demonstrate the superiority of SSDiff both visually and quantitatively. The code will be made open source after possible acceptance.

Title: GenFighter: A Generative and Evolutive Textual Attack Removal

• Authors: Md Athikul Islam, Edoardo Serra, Sushil Jajodia
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2404.11538
• Pdf URL: https://arxiv.org/pdf/2404.11538
• Copy Paste: [[2404.11538]] GenFighter: A Generative and Evolutive Textual Attack Removal(https://arxiv.org/abs/2404.11538)
• Keywords: generative
• Abstract: Adversarial attacks pose significant challenges to deep neural networks (DNNs) such as Transformer models in natural language processing (NLP). This paper introduces a novel defense strategy, called GenFighter, which enhances adversarial robustness by learning and reasoning on the training classification distribution. GenFighter identifies potentially malicious instances deviating from the distribution, transforms them into semantically equivalent instances aligned with the training data, and employs ensemble techniques for a unified and robust response. By conducting extensive experiments, we show that GenFighter outperforms state-of-the-art defenses in accuracy under attack and attack success rate metrics. Additionally, it requires a high number of queries per attack, making the attack more challenging in real scenarios. The ablation study shows that our approach integrates transfer learning, a generative/evolutive procedure, and an ensemble method, providing an effective defense against NLP adversarial attacks.

Title: Evaluating Span Extraction in Generative Paradigm: A Reflection on Aspect-Based Sentiment Analysis

• Authors: Soyoung Yang, Won Ik Cho
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.11539
• Pdf URL: https://arxiv.org/pdf/2404.11539
• Copy Paste: [[2404.11539]] Evaluating Span Extraction in Generative Paradigm: A Reflection on Aspect-Based Sentiment Analysis(https://arxiv.org/abs/2404.11539)
• Keywords: generative
• Abstract: In the era of rapid evolution of generative language models within the realm of natural language processing, there is an imperative call to revisit and reformulate evaluation methodologies, especially in the domain of aspect-based sentiment analysis (ABSA). This paper addresses the emerging challenges introduced by the generative paradigm, which has moderately blurred traditional boundaries between understanding and generation tasks. Building upon prevailing practices in the field, we analyze the advantages and shortcomings associated with the prevalent ABSA evaluation paradigms. Through an in-depth examination, supplemented by illustrative examples, we highlight the intricacies involved in aligning generative outputs with other evaluative metrics, specifically those derived from other tasks, including question answering. While we steer clear of advocating for a singular and definitive metric, our contribution lies in paving the path for a comprehensive guideline tailored for ABSA evaluations in this generative paradigm. In this position paper, we aim to provide practitioners with profound reflections, offering insights and directions that can aid in navigating this evolving landscape, ensuring evaluations that are both accurate and reflective of generative capabilities.

Title: Predicting Long-horizon Futures by Conditioning on Geometry and Time

• Authors: Tarasha Khurana, Deva Ramanan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11554
• Pdf URL: https://arxiv.org/pdf/2404.11554
• Copy Paste: [[2404.11554]] Predicting Long-horizon Futures by Conditioning on Geometry and Time(https://arxiv.org/abs/2404.11554)
• Keywords: diffusion
• Abstract: Our work explores the task of generating future sensor observations conditioned on the past. We are motivated by `predictive coding' concepts from neuroscience as well as robotic applications such as self-driving vehicles. Predictive video modeling is challenging because the future may be multi-modal and learning at scale remains computationally expensive for video processing. To address both challenges, our key insight is to leverage the large-scale pretraining of image diffusion models which can handle multi-modality. We repurpose image models for video prediction by conditioning on new frame timestamps. Such models can be trained with videos of both static and dynamic scenes. To allow them to be trained with modestly-sized datasets, we introduce invariances by factoring out illumination and texture by forcing the model to predict (pseudo) depth, readily obtained for in-the-wild videos via off-the-shelf monocular depth networks. In fact, we show that simply modifying networks to predict grayscale pixels already improves the accuracy of video prediction. Given the extra controllability with timestamp conditioning, we propose sampling schedules that work better than the traditional autoregressive and hierarchical sampling strategies. Motivated by probabilistic metrics from the object forecasting literature, we create a benchmark for video prediction on a diverse set of videos spanning indoor and outdoor scenes and a large vocabulary of objects. Our experiments illustrate the effectiveness of learning to condition on timestamps, and show the importance of predicting the future with invariant modalities.

Title: MoA: Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generation

• Authors: Kuan-Chieh (Jackson)Wang, Daniil Ostashev, Yuwei Fang, Sergey Tulyakov, Kfir Aberman
• Subjects: cs.CV, cs.AI, cs.GR
• Abstract URL: https://arxiv.org/abs/2404.11565
• Pdf URL: https://arxiv.org/pdf/2404.11565
• Copy Paste: [[2404.11565]] MoA: Mixture-of-Attention for Subject-Context Disentanglement in Personalized Image Generation(https://arxiv.org/abs/2404.11565)
• Keywords: diffusion
• Abstract: We introduce a new architecture for personalization of text-to-image diffusion models, coined Mixture-of-Attention (MoA). Inspired by the Mixture-of-Experts mechanism utilized in large language models (LLMs), MoA distributes the generation workload between two attention pathways: a personalized branch and a non-personalized prior branch. MoA is designed to retain the original model's prior by fixing its attention layers in the prior branch, while minimally intervening in the generation process with the personalized branch that learns to embed subjects in the layout and context generated by the prior branch. A novel routing mechanism manages the distribution of pixels in each layer across these branches to optimize the blend of personalized and generic content creation. Once trained, MoA facilitates the creation of high-quality, personalized images featuring multiple subjects with compositions and interactions as diverse as those generated by the original model. Crucially, MoA enhances the distinction between the model's pre-existing capability and the newly augmented personalized intervention, thereby offering a more disentangled subject-context control that was previously unattainable. Project page: https://snap-research.github.io/mixture-of-attention

Title: Prompt Optimizer of Text-to-Image Diffusion Models for Abstract Concept Understanding

• Authors: Zezhong Fan, Xiaohan Li, Chenhao Fang, Topojoy Biswas, Kaushiki Nag, Jianpeng Xu, Kannan Achan
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2404.11589
• Pdf URL: https://arxiv.org/pdf/2404.11589
• Copy Paste: [[2404.11589]] Prompt Optimizer of Text-to-Image Diffusion Models for Abstract Concept Understanding(https://arxiv.org/abs/2404.11589)
• Keywords: diffusion, generative
• Abstract: The rapid evolution of text-to-image diffusion models has opened the door of generative AI, enabling the translation of textual descriptions into visually compelling images with remarkable quality. However, a persistent challenge within this domain is the optimization of prompts to effectively convey abstract concepts into concrete objects. For example, text encoders can hardly express "peace", while can easily illustrate olive branches and white doves. This paper introduces a novel approach named Prompt Optimizer for Abstract Concepts (POAC) specifically designed to enhance the performance of text-to-image diffusion models in interpreting and generating images from abstract concepts. We propose a Prompt Language Model (PLM), which is initialized from a pre-trained language model, and then fine-tuned with a curated dataset of abstract concept prompts. The dataset is created with GPT-4 to extend the abstract concept to a scene and concrete objects. Our framework employs a Reinforcement Learning (RL)-based optimization strategy, focusing on the alignment between the generated images by a stable diffusion model and optimized prompts. Through extensive experiments, we demonstrate that our proposed POAC significantly improves the accuracy and aesthetic quality of generated images, particularly in the description of abstract concepts and alignment with optimized prompts. We also present a comprehensive analysis of our model's performance across diffusion models under different settings, showcasing its versatility and effectiveness in enhancing abstract concept representation.

Title: IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination

• Authors: Xi Chen (1), Sida Peng (1), Dongchen Yang (1), Yuan Liu (2), Bowen Pan (3), Chengfei Lv (3), Xiaowei Zhou (1) ((1) Zhejiang University, (2) The University of Hong Kong, (3) Tao Technology Department, Alibaba Group)
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11593
• Pdf URL: https://arxiv.org/pdf/2404.11593
• Copy Paste: [[2404.11593]] IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination(https://arxiv.org/abs/2404.11593)
• Keywords: diffusion, generative
• Abstract: This paper aims to recover object materials from posed images captured under an unknown static lighting condition. Recent methods solve this task by optimizing material parameters through differentiable physically based rendering. However, due to the coupling between object geometry, materials, and environment lighting, there is inherent ambiguity during the inverse rendering process, preventing previous methods from obtaining accurate results. To overcome this ill-posed problem, our key idea is to learn the material prior with a generative model for regularizing the optimization process. We observe that the general rendering equation can be split into diffuse and specular shading terms, and thus formulate the material prior as diffusion models of albedo and specular. Thanks to this design, our model can be trained using the existing abundant 3D object data, and naturally acts as a versatile tool to resolve the ambiguity when recovering material representations from RGB images. In addition, we develop a coarse-to-fine training strategy that leverages estimated materials to guide diffusion models to satisfy multi-view consistent constraints, leading to more stable and accurate results. Extensive experiments on real-world and synthetic datasets demonstrate that our approach achieves state-of-the-art performance on material recovery. The code will be available at https://zju3dv.github.io/IntrinsicAnything.

Title: Learning to Solve the Constrained Most Probable Explanation Task in Probabilistic Graphical Models

• Authors: Shivvrat Arya, Tahrima Rahman, Vibhav Gogate
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.11606
• Pdf URL: https://arxiv.org/pdf/2404.11606
• Copy Paste: [[2404.11606]] Learning to Solve the Constrained Most Probable Explanation Task in Probabilistic Graphical Models(https://arxiv.org/abs/2404.11606)
• Keywords: self-supervised
• Abstract: We propose a self-supervised learning approach for solving the following constrained optimization task in log-linear models or Markov networks. Let $f$ and $g$ be two log-linear models defined over the sets $\mathbf{X}$ and $\mathbf{Y}$ of random variables respectively. Given an assignment $\mathbf{x}$ to all variables in $\mathbf{X}$ (evidence) and a real number $q$, the constrained most-probable explanation (CMPE) task seeks to find an assignment $\mathbf{y}$ to all variables in $\mathbf{Y}$ such that $f(\mathbf{x}, \mathbf{y})$ is maximized and $g(\mathbf{x}, \mathbf{y})\leq q$. In our proposed self-supervised approach, given assignments $\mathbf{x}$ to $\mathbf{X}$ (data), we train a deep neural network that learns to output near-optimal solutions to the CMPE problem without requiring access to any pre-computed solutions. The key idea in our approach is to use first principles and approximate inference methods for CMPE to derive novel loss functions that seek to push infeasible solutions towards feasible ones and feasible solutions towards optimal ones. We analyze the properties of our proposed method and experimentally demonstrate its efficacy on several benchmark problems.

Title: InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior

• Authors: Zhiheng Liu, Hao Ouyang, Qiuyu Wang, Ka Leong Cheng, Jie Xiao, Kai Zhu, Nan Xue, Yu Liu, Yujun Shen, Yang Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11613
• Pdf URL: https://arxiv.org/pdf/2404.11613
• Copy Paste: [[2404.11613]] InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior(https://arxiv.org/abs/2404.11613)
• Keywords: diffusion
• Abstract: 3D Gaussians have recently emerged as an efficient representation for novel view synthesis. This work studies its editability with a particular focus on the inpainting task, which aims to supplement an incomplete set of 3D Gaussians with additional points for visually harmonious rendering. Compared to 2D inpainting, the crux of inpainting 3D Gaussians is to figure out the rendering-relevant properties of the introduced points, whose optimization largely benefits from their initial 3D positions. To this end, we propose to guide the point initialization with an image-conditioned depth completion model, which learns to directly restore the depth map based on the observed image. Such a design allows our model to fill in depth values at an aligned scale with the original depth, and also to harness strong generalizability from largescale diffusion prior. Thanks to the more accurate depth completion, our approach, dubbed InFusion, surpasses existing alternatives with sufficiently better fidelity and efficiency under various complex scenarios. We further demonstrate the effectiveness of InFusion with several practical applications, such as inpainting with user-specific texture or with novel object insertion.

Title: Factorized Diffusion: Perceptual Illusions by Noise Decomposition

• Authors: Daniel Geng, Inbum Park, Andrew Owens
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.11615
• Pdf URL: https://arxiv.org/pdf/2404.11615
• Copy Paste: [[2404.11615]] Factorized Diffusion: Perceptual Illusions by Noise Decomposition(https://arxiv.org/abs/2404.11615)
• Keywords: diffusion
• Abstract: Given a factorization of an image into a sum of linear components, we present a zero-shot method to control each individual component through diffusion model sampling. For example, we can decompose an image into low and high spatial frequencies and condition these components on different text prompts. This produces hybrid images, which change appearance depending on viewing distance. By decomposing an image into three frequency subbands, we can generate hybrid images with three prompts. We also use a decomposition into grayscale and color components to produce images whose appearance changes when they are viewed in grayscale, a phenomena that naturally occurs under dim lighting. And we explore a decomposition by a motion blur kernel, which produces images that change appearance under motion blurring. Our method works by denoising with a composite noise estimate, built from the components of noise estimates conditioned on different prompts. We also show that for certain decompositions, our method recovers prior approaches to compositional generation and spatial control. Finally, we show that we can extend our approach to generate hybrid images from real images. We do this by holding one component fixed and generating the remaining components, effectively solving an inverse problem.