2024-09-13

Title: A Survey of Anomaly Detection in In-Vehicle Networks

• Authors: Övgü Özdemir, M. Tuğberk İşyapar, Pınar Karagöz, Klaus Werner Schmidt, Demet Demir, N. Alpay Karagöz
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2409.07505
• Pdf URL: https://arxiv.org/pdf/2409.07505
• Copy Paste: [[2409.07505]] A Survey of Anomaly Detection in In-Vehicle Networks(https://arxiv.org/abs/2409.07505)
• Keywords: anomaly
• Abstract: Modern vehicles are equipped with Electronic Control Units (ECU) that are used for controlling important vehicle functions including safety-critical operations. ECUs exchange information via in-vehicle communication buses, of which the Controller Area Network (CAN bus) is by far the most widespread representative. Problems that may occur in the vehicle's physical parts or malicious attacks may cause anomalies in the CAN traffic, impairing the correct vehicle operation. Therefore, the detection of such anomalies is vital for vehicle safety. This paper reviews the research on anomaly detection for in-vehicle networks, more specifically for the CAN bus. Our main focus is the evaluation of methods used for CAN bus anomaly detection together with the datasets used in such analysis. To provide the reader with a more comprehensive understanding of the subject, we first give a brief review of related studies on time series-based anomaly detection. Then, we conduct an extensive survey of recent deep learning-based techniques as well as conventional techniques for CAN bus anomaly detection. Our comprehensive analysis delves into anomaly detection algorithms employed in in-vehicle networks, specifically focusing on their learning paradigms, inherent strengths, and weaknesses, as well as their efficacy when applied to CAN bus datasets. Lastly, we highlight challenges and open research problems in CAN bus anomaly detection.

Title: EchoDFKD: Data-Free Knowledge Distillation for Cardiac Ultrasound Segmentation using Synthetic Data

• Authors: Grégoire Petit, Nathan Palluau, Axel Bauer, Clemens Dlaska
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07566
• Pdf URL: https://arxiv.org/pdf/2409.07566
• Copy Paste: [[2409.07566]] EchoDFKD: Data-Free Knowledge Distillation for Cardiac Ultrasound Segmentation using Synthetic Data(https://arxiv.org/abs/2409.07566)
• Keywords: generative
• Abstract: The application of machine learning to medical ultrasound videos of the heart, i.e., echocardiography, has recently gained traction with the availability of large public datasets. Traditional supervised tasks, such as ejection fraction regression, are now making way for approaches focusing more on the latent structure of data distributions, as well as generative methods. We propose a model trained exclusively by knowledge distillation, either on real or synthetical data, involving retrieving masks suggested by a teacher model. We achieve state-of-the-art (SOTA) values on the task of identifying end-diastolic and end-systolic frames. By training the model only on synthetic data, it reaches segmentation capabilities close to the performance when trained on real data with a significantly reduced number of weights. A comparison with the 5 main existing methods shows that our method outperforms the others in most cases. We also present a new evaluation method that does not require human annotation and instead relies on a large auxiliary model. We show that this method produces scores consistent with those obtained from human annotations. Relying on the integrated knowledge from a vast amount of records, this method overcomes certain inherent limitations of human annotator labeling. Code: this https URL

Title: Self-Masking Networks for Unsupervised Adaptation

• Authors: Alfonso Taboada Warmerdam, Mathilde Caron, Yuki M. Asano
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07577
• Pdf URL: https://arxiv.org/pdf/2409.07577
• Copy Paste: [[2409.07577]] Self-Masking Networks for Unsupervised Adaptation(https://arxiv.org/abs/2409.07577)
• Keywords: self-supervised, foundation model
• Abstract: With the advent of billion-parameter foundation models, efficient fine-tuning has become increasingly important for the adaptation of models to downstream tasks. However, especially in computer vision, it can be hard to achieve good performance when access to quality labeled data is lacking. In this work, we propose a method adapting pretrained generalist models in a self-supervised manner by learning binary masks. These self-supervised masking networks (SMNs) are up to 79x more efficient to store and significantly improve performance on label-efficient downstream tasks. We validate the usefulness of learning binary masks as a fine-tuning method on 8 datasets and 3 model architectures, and we demonstrate the effectiveness of SMNs in 3 label-efficient settings.

Title: New constructions of pseudorandom codes

• Authors: Surendra Ghentiyala, Venkatesan Guruswami
• Subjects: cs.CR, cs.CC
• Abstract URL: https://arxiv.org/abs/2409.07580
• Pdf URL: https://arxiv.org/pdf/2409.07580
• Copy Paste: [[2409.07580]] New constructions of pseudorandom codes(https://arxiv.org/abs/2409.07580)
• Keywords: generative
• Abstract: Introduced in [CG24], pseudorandom error-correcting codes (PRCs) are a new cryptographic primitive with applications in watermarking generative AI models. These are codes where a collection of polynomially many codewords is computationally indistinguishable from random, except to individuals with the decoding key. In this work, we examine the assumptions under which PRCs with robustness to a constant error rate exist. 1. We show that if both the planted hyperloop assumption introduced in [BKR23] and security of a version of Goldreich's PRG hold, then there exist public-key PRCs for which no efficient adversary can distinguish a polynomial number of codewords from random with better than $o(1)$ advantage. 2. We revisit the construction of [CG24] and show that it can be based on a wider range of assumptions than presented in [CG24]. To do this, we introduce a weakened version of the planted XOR assumption which we call the weak planted XOR assumption and which may be of independent interest. 3. We initiate the study of PRCs which are secure against space-bounded adversaries. We show how to construct secret-key PRCs of length $O(n)$ which are $\textit{unconditionally}$ indistinguishable from random by $\text{poly}(n)$ time, $O(n^{1.5-\varepsilon})$ space adversaries.

Title: Zero-Shot Machine-Generated Text Detection Using Mixture of Large Language Models

• Authors: Matthieu Dubois, François Yvon, Pablo Piantanida
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07615
• Pdf URL: https://arxiv.org/pdf/2409.07615
• Copy Paste: [[2409.07615]] Zero-Shot Machine-Generated Text Detection Using Mixture of Large Language Models(https://arxiv.org/abs/2409.07615)
• Keywords: generative
• Abstract: The dissemination of Large Language Models (LLMs), trained at scale, and endowed with powerful text-generating abilities has vastly increased the threats posed by generative AI technologies by reducing the cost of producing harmful, toxic, faked or forged content. In response, various proposals have been made to automatically discriminate artificially generated from human-written texts, typically framing the problem as a classification problem. Most approaches evaluate an input document by a well-chosen detector LLM, assuming that low-perplexity scores reliably signal machine-made content. As using one single detector can induce brittleness of performance, we instead consider several and derive a new, theoretically grounded approach to combine their respective strengths. Our experiments, using a variety of generator LLMs, suggest that our method effectively increases the robustness of detection.

Title: Ensemble Methods for Sequence Classification with Hidden Markov Models

• Authors: Maxime Kawawa-Beaudan, Srijan Sood, Soham Palande, Ganapathy Mani, Tucker Balch, Manuela Veloso
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07619
• Pdf URL: https://arxiv.org/pdf/2409.07619
• Copy Paste: [[2409.07619]] Ensemble Methods for Sequence Classification with Hidden Markov Models(https://arxiv.org/abs/2409.07619)
• Keywords: anomaly
• Abstract: We present a lightweight approach to sequence classification using Ensemble Methods for Hidden Markov Models (HMMs). HMMs offer significant advantages in scenarios with imbalanced or smaller datasets due to their simplicity, interpretability, and efficiency. These models are particularly effective in domains such as finance and biology, where traditional methods struggle with high feature dimensionality and varied sequence lengths. Our ensemble-based scoring method enables the comparison of sequences of any length and improves performance on imbalanced datasets. This study focuses on the binary classification problem, particularly in scenarios with data imbalance, where the negative class is the majority (e.g., normal data) and the positive class is the minority (e.g., anomalous data), often with extreme distribution skews. We propose a novel training approach for HMM Ensembles that generalizes to multi-class problems and supports classification and anomaly detection. Our method fits class-specific groups of diverse models using random data subsets, and compares likelihoods across classes to produce composite scores, achieving high average precisions and AUCs. In addition, we compare our approach with neural network-based methods such as Convolutional Neural Networks (CNNs) and Long Short-Term Memory networks (LSTMs), highlighting the efficiency and robustness of HMMs in data-scarce environments. Motivated by real-world use cases, our method demonstrates robust performance across various benchmarks, offering a flexible framework for diverse applications.

Title: DiffTED: One-shot Audio-driven TED Talk Video Generation with Diffusion-based Co-speech Gestures

• Authors: Steven Hogue, Chenxu Zhang, Hamza Daruger, Yapeng Tian, Xiaohu Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07649
• Pdf URL: https://arxiv.org/pdf/2409.07649
• Copy Paste: [[2409.07649]] DiffTED: One-shot Audio-driven TED Talk Video Generation with Diffusion-based Co-speech Gestures(https://arxiv.org/abs/2409.07649)
• Keywords: diffusion, generative
• Abstract: Audio-driven talking video generation has advanced significantly, but existing methods often depend on video-to-video translation techniques and traditional generative networks like GANs and they typically generate taking heads and co-speech gestures separately, leading to less coherent outputs. Furthermore, the gestures produced by these methods often appear overly smooth or subdued, lacking in diversity, and many gesture-centric approaches do not integrate talking head generation. To address these limitations, we introduce DiffTED, a new approach for one-shot audio-driven TED-style talking video generation from a single image. Specifically, we leverage a diffusion model to generate sequences of keypoints for a Thin-Plate Spline motion model, precisely controlling the avatar's animation while ensuring temporally coherent and diverse gestures. This innovative approach utilizes classifier-free guidance, empowering the gestures to flow naturally with the audio input without relying on pre-trained classifiers. Experiments demonstrate that DiffTED generates temporally coherent talking videos with diverse co-speech gestures.

Title: Foundation Models Boost Low-Level Perceptual Similarity Metrics

• Authors: Abhijay Ghildyal, Nabajeet Barman, Saman Zadtootaghaj
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07650
• Pdf URL: https://arxiv.org/pdf/2409.07650
• Copy Paste: [[2409.07650]] Foundation Models Boost Low-Level Perceptual Similarity Metrics(https://arxiv.org/abs/2409.07650)
• Keywords: foundation model
• Abstract: For full-reference image quality assessment (FR-IQA) using deep-learning approaches, the perceptual similarity score between a distorted image and a reference image is typically computed as a distance measure between features extracted from a pretrained CNN or more recently, a Transformer network. Often, these intermediate features require further fine-tuning or processing with additional neural network layers to align the final similarity scores with human judgments. So far, most IQA models based on foundation models have primarily relied on the final layer or the embedding for the quality score estimation. In contrast, this work explores the potential of utilizing the intermediate features of these foundation models, which have largely been unexplored so far in the design of low-level perceptual similarity metrics. We demonstrate that the intermediate features are comparatively more effective. Moreover, without requiring any training, these metrics can outperform both traditional and state-of-the-art learned metrics by utilizing distance measures between the features.

Title: Experimenting with Legal AI Solutions: The Case of Question-Answering for Access to Justice

• Authors: Jonathan Li, Rohan Bhambhoria, Samuel Dahan, Xiaodan Zhu
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07713
• Pdf URL: https://arxiv.org/pdf/2409.07713
• Copy Paste: [[2409.07713]] Experimenting with Legal AI Solutions: The Case of Question-Answering for Access to Justice(https://arxiv.org/abs/2409.07713)
• Keywords: generative
• Abstract: Generative AI models, such as the GPT and Llama series, have significant potential to assist laypeople in answering legal questions. However, little prior work focuses on the data sourcing, inference, and evaluation of these models in the context of laypersons. To this end, we propose a human-centric legal NLP pipeline, covering data sourcing, inference, and evaluation. We introduce and release a dataset, LegalQA, with real and specific legal questions spanning from employment law to criminal law, corresponding answers written by legal experts, and citations for each answer. We develop an automatic evaluation protocol for this dataset, then show that retrieval-augmented generation from only 850 citations in the train set can match or outperform internet-wide retrieval, despite containing 9 orders of magnitude less data. Finally, we propose future directions for open-sourced efforts, which fall behind closed-sourced models.

Title: Advancing Depth Anything Model for Unsupervised Monocular Depth Estimation in Endoscopy

• Authors: Bojian Li, Bo Liu, Jinghua Yue, Fugen Zhou
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07723
• Pdf URL: https://arxiv.org/pdf/2409.07723
• Copy Paste: [[2409.07723]] Advancing Depth Anything Model for Unsupervised Monocular Depth Estimation in Endoscopy(https://arxiv.org/abs/2409.07723)
• Keywords: foundation model
• Abstract: Depth estimation is a cornerstone of 3D reconstruction and plays a vital role in minimally invasive endoscopic surgeries. However, most current depth estimation networks rely on traditional convolutional neural networks, which are limited in their ability to capture global information. Foundation models offer a promising avenue for enhancing depth estimation, but those currently available are primarily trained on natural images, leading to suboptimal performance when applied to endoscopic images. In this work, we introduce a novel fine-tuning strategy for the Depth Anything Model and integrate it with an intrinsic-based unsupervised monocular depth estimation framework. Our approach includes a low-rank adaptation technique based on random vectors, which improves the model's adaptability to different scales. Additionally, we propose a residual block built on depthwise separable convolution to compensate for the transformer's limited ability to capture high-frequency details, such as edges and textures. Our experimental results on the SCARED dataset show that our method achieves state-of-the-art performance while minimizing the number of trainable parameters. Applying this method in minimally invasive endoscopic surgery could significantly enhance both the precision and safety of these procedures.

Title: LOCKEY: A Novel Approach to Model Authentication and Deepfake Tracking

• Authors: Mayank Kumar Singh, Naoya Takahashi, Wei-Hsiang Liao, Yuki Mitsufuji
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2409.07743
• Pdf URL: https://arxiv.org/pdf/2409.07743
• Copy Paste: [[2409.07743]] LOCKEY: A Novel Approach to Model Authentication and Deepfake Tracking(https://arxiv.org/abs/2409.07743)
• Keywords: generative
• Abstract: This paper presents a novel approach to deter unauthorized deepfakes and enable user tracking in generative models, even when the user has full access to the model parameters, by integrating key-based model authentication with watermarking techniques. Our method involves providing users with model parameters accompanied by a unique, user-specific key. During inference, the model is conditioned upon the key along with the standard input. A valid key results in the expected output, while an invalid key triggers a degraded output, thereby enforcing key-based model authentication. For user tracking, the model embeds the user's unique key as a watermark within the generated content, facilitating the identification of the user's ID. We demonstrate the effectiveness of our approach on two types of models, audio codecs and vocoders, utilizing the SilentCipher watermarking method. Additionally, we assess the robustness of the embedded watermarks against various distortions, validating their reliability in various scenarios.

Title: Learning Brain Tumor Representation in 3D High-Resolution MR Images via Interpretable State Space Models

• Authors: Qingqiao Hu, Daoan Zhang, Jiebo Luo, Zhenyu Gong, Benedikt Wiestler, Jianguo Zhang, Hongwei Bran Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07746
• Pdf URL: https://arxiv.org/pdf/2409.07746
• Copy Paste: [[2409.07746]] Learning Brain Tumor Representation in 3D High-Resolution MR Images via Interpretable State Space Models(https://arxiv.org/abs/2409.07746)
• Keywords: self-supervised
• Abstract: Learning meaningful and interpretable representations from high-dimensional volumetric magnetic resonance (MR) images is essential for advancing personalized medicine. While Vision Transformers (ViTs) have shown promise in handling image data, their application to 3D multi-contrast MR images faces challenges due to computational complexity and interpretability. To address this, we propose a novel state-space-model (SSM)-based masked autoencoder which scales ViT-like models to handle high-resolution data effectively while also enhancing the interpretability of learned representations. We propose a latent-to-spatial mapping technique that enables direct visualization of how latent features correspond to specific regions in the input volumes in the context of SSM. We validate our method on two key neuro-oncology tasks: identification of isocitrate dehydrogenase mutation status and 1p/19q co-deletion classification, achieving state-of-the-art accuracy. Our results highlight the potential of SSM-based self-supervised learning to transform radiomics analysis by combining efficiency and interpretability.

Title: DiTAS: Quantizing Diffusion Transformers via Enhanced Activation Smoothing

• Authors: Zhenyuan Dong, Sai Qian Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07756
• Pdf URL: https://arxiv.org/pdf/2409.07756
• Copy Paste: [[2409.07756]] DiTAS: Quantizing Diffusion Transformers via Enhanced Activation Smoothing(https://arxiv.org/abs/2409.07756)
• Keywords: diffusion
• Abstract: Diffusion Transformers (DiTs) have recently attracted significant interest from both industry and academia due to their enhanced capabilities in visual generation, surpassing the performance of traditional diffusion models that employ U-Net. However, the improved performance of DiTs comes at the expense of higher parameter counts and implementation costs, which significantly limits their deployment on resource-constrained devices like mobile phones. We propose DiTAS, a data-free post-training quantization (PTQ) method for efficient DiT inference. DiTAS relies on the proposed temporal-aggregated smoothing techniques to mitigate the impact of the channel-wise outliers within the input activations, leading to much lower quantization error under extremely low bitwidth. To further enhance the performance of the quantized DiT, we adopt the layer-wise grid search strategy to optimize the smoothing factor. Experimental results demonstrate that our approach enables 4-bit weight, 8-bit activation (W4A8) quantization for DiTs while maintaining comparable performance as the full-precision model.

Title: XMOL: Explainable Multi-property Optimization of Molecules

• Authors: Aye Phyu Phyu Aung, Jay Chaudhary, Ji Wei Yoon, Senthilnath Jayavelu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07786
• Pdf URL: https://arxiv.org/pdf/2409.07786
• Copy Paste: [[2409.07786]] XMOL: Explainable Multi-property Optimization of Molecules(https://arxiv.org/abs/2409.07786)
• Keywords: diffusion
• Abstract: Molecular optimization is a key challenge in drug discovery and material science domain, involving the design of molecules with desired properties. Existing methods focus predominantly on single-property optimization, necessitating repetitive runs to target multiple properties, which is inefficient and computationally expensive. Moreover, these methods often lack transparency, making it difficult for researchers to understand and control the optimization process. To address these issues, we propose a novel framework, Explainable Multi-property Optimization of Molecules (XMOL), to optimize multiple molecular properties simultaneously while incorporating explainability. Our approach builds on state-of-the-art geometric diffusion models, extending them to multi-property optimization through the introduction of spectral normalization and enhanced molecular constraints for stabilized training. Additionally, we integrate interpretive and explainable techniques throughout the optimization process. We evaluated XMOL on the real-world molecular datasets i.e., QM9, demonstrating its effectiveness in both single property and multiple properties optimization while offering interpretable results, paving the way for more efficient and reliable molecular design.

Title: SURGIVID: Annotation-Efficient Surgical Video Object Discovery

• Authors: Çağhan Köksal, Ghazal Ghazaei, Nassir Navab
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.07801
• Pdf URL: https://arxiv.org/pdf/2409.07801
• Copy Paste: [[2409.07801]] SURGIVID: Annotation-Efficient Surgical Video Object Discovery(https://arxiv.org/abs/2409.07801)
• Keywords: self-supervised
• Abstract: Surgical scenes convey crucial information about the quality of surgery. Pixel-wise localization of tools and anatomical structures is the first task towards deeper surgical analysis for microscopic or endoscopic surgical views. This is typically done via fully-supervised methods which are annotation greedy and in several cases, demanding medical expertise. Considering the profusion of surgical videos obtained through standardized surgical workflows, we propose an annotation-efficient framework for the semantic segmentation of surgical scenes. We employ image-based self-supervised object discovery to identify the most salient tools and anatomical structures in surgical videos. These proposals are further refined within a minimally supervised fine-tuning step. Our unsupervised setup reinforced with only 36 annotation labels indicates comparable localization performance with fully-supervised segmentation models. Further, leveraging surgical phase labels as weak labels can better guide model attention towards surgical tools, leading to $\sim 2\%$ improvement in tool localization. Extensive ablation studies on the CaDIS dataset validate the effectiveness of our proposed solution in discovering relevant surgical objects with minimal or no supervision.

Title: FPMT: Enhanced Semi-Supervised Model for Traffic Incident Detection

• Authors: Xinying Lu, Jianli Xiao
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2409.07839
• Pdf URL: https://arxiv.org/pdf/2409.07839
• Copy Paste: [[2409.07839]] FPMT: Enhanced Semi-Supervised Model for Traffic Incident Detection(https://arxiv.org/abs/2409.07839)
• Keywords: generative
• Abstract: For traffic incident detection, the acquisition of data and labels is notably resource-intensive, rendering semi-supervised traffic incident detection both a formidable and consequential challenge. Thus, this paper focuses on traffic incident detection with a semi-supervised learning way. It proposes a semi-supervised learning model named FPMT within the framework of MixText. The data augmentation module introduces Generative Adversarial Networks to balance and expand the dataset. During the mix-up process in the hidden space, it employs a probabilistic pseudo-mixing mechanism to enhance regularization and elevate model precision. In terms of training strategy, it initiates with unsupervised training on all data, followed by supervised fine-tuning on a subset of labeled data, and ultimately completing the goal of semi-supervised training. Through empirical validation on four authentic datasets, our FPMT model exhibits outstanding performance across various metrics. Particularly noteworthy is its robust performance even in scenarios with low label rates.

Title: UGAD: Universal Generative AI Detector utilizing Frequency Fingerprints

• Authors: Inzamamul Alam, Muhammad Shahid Muneer, Simon S. Woo
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07913
• Pdf URL: https://arxiv.org/pdf/2409.07913
• Copy Paste: [[2409.07913]] UGAD: Universal Generative AI Detector utilizing Frequency Fingerprints(https://arxiv.org/abs/2409.07913)
• Keywords: diffusion, generative
• Abstract: In the wake of a fabricated explosion image at the Pentagon, an ability to discern real images from fake counterparts has never been more critical. Our study introduces a novel multi-modal approach to detect AI-generated images amidst the proliferation of new generation methods such as Diffusion models. Our method, UGAD, encompasses three key detection steps: First, we transform the RGB images into YCbCr channels and apply an Integral Radial Operation to emphasize salient radial features. Secondly, the Spatial Fourier Extraction operation is used for a spatial shift, utilizing a pre-trained deep learning network for optimal feature extraction. Finally, the deep neural network classification stage processes the data through dense layers using softmax for classification. Our approach significantly enhances the accuracy of differentiating between real and AI-generated images, as evidenced by a 12.64% increase in accuracy and 28.43% increase in AUC compared to existing state-of-the-art methods.

Title: Control+Shift: Generating Controllable Distribution Shifts

• Authors: Roy Friedman, Rhea Chowers
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07940
• Pdf URL: https://arxiv.org/pdf/2409.07940
• Copy Paste: [[2409.07940]] Control+Shift: Generating Controllable Distribution Shifts(https://arxiv.org/abs/2409.07940)
• Keywords: generative
• Abstract: We propose a new method for generating realistic datasets with distribution shifts using any decoder-based generative model. Our approach systematically creates datasets with varying intensities of distribution shifts, facilitating a comprehensive analysis of model performance degradation. We then use these generated datasets to evaluate the performance of various commonly used networks and observe a consistent decline in performance with increasing shift intensity, even when the effect is almost perceptually unnoticeable to the human eye. We see this degradation even when using data augmentations. We also find that enlarging the training dataset beyond a certain point has no effect on the robustness and that stronger inductive biases increase robustness.

Title: Do Vision Foundation Models Enhance Domain Generalization in Medical Image Segmentation?

• Authors: Kerem Cekmeceli, Meva Himmetoglu, Guney I. Tombak, Anna Susmelj, Ertunc Erdil, Ender Konukoglu
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07960
• Pdf URL: https://arxiv.org/pdf/2409.07960
• Copy Paste: [[2409.07960]] Do Vision Foundation Models Enhance Domain Generalization in Medical Image Segmentation?(https://arxiv.org/abs/2409.07960)
• Keywords: foundation model
• Abstract: Neural networks achieve state-of-the-art performance in many supervised learning tasks when the training data distribution matches the test data distribution. However, their performance drops significantly under domain (covariate) shift, a prevalent issue in medical image segmentation due to varying acquisition settings across different scanner models and protocols. Recently, foundational models (FMs) trained on large datasets have gained attention for their ability to be adapted for downstream tasks and achieve state-of-the-art performance with excellent generalization capabilities on natural images. However, their effectiveness in medical image segmentation remains underexplored. In this paper, we investigate the domain generalization performance of various FMs, including DinoV2, SAM, MedSAM, and MAE, when fine-tuned using various parameter-efficient fine-tuning (PEFT) techniques such as Ladder and Rein (+LoRA) and decoder heads. We introduce a novel decode head architecture, HQHSAM, which simply integrates elements from two state-of-the-art decoder heads, HSAM and HQSAM, to enhance segmentation performance. Our extensive experiments on multiple datasets, encompassing various anatomies and modalities, reveal that FMs, particularly with the HQHSAM decode head, improve domain generalization for medical image segmentation. Moreover, we found that the effectiveness of PEFT techniques varies across different FMs. These findings underscore the potential of FMs to enhance the domain generalization performance of neural networks in medical image segmentation across diverse clinical settings, providing a solid foundation for future research. Code and models are available for research purposes at \url{this https URL}.

Title: Estimating atmospheric variables from Digital Typhoon Satellite Images via Conditional Denoising Diffusion Models

• Authors: Zhangyue Ling, Pritthijit Nath, César Quilodrán-Casas
• Subjects: cs.CV, physics.ao-ph
• Abstract URL: https://arxiv.org/abs/2409.07961
• Pdf URL: https://arxiv.org/pdf/2409.07961
• Copy Paste: [[2409.07961]] Estimating atmospheric variables from Digital Typhoon Satellite Images via Conditional Denoising Diffusion Models(https://arxiv.org/abs/2409.07961)
• Keywords: diffusion
• Abstract: This study explores the application of diffusion models in the field of typhoons, predicting multiple ERA5 meteorological variables simultaneously from Digital Typhoon satellite images. The focus of this study is taken to be Taiwan, an area very vulnerable to typhoons. By comparing the performance of Conditional Denoising Diffusion Probability Model (CDDPM) with Convolutional Neural Networks (CNN) and Squeeze-and-Excitation Networks (SENet), results suggest that the CDDPM performs best in generating accurate and realistic meteorological data. Specifically, CDDPM achieved a PSNR of 32.807, which is approximately 7.9% higher than CNN and 5.5% higher than SENet. Furthermore, CDDPM recorded an RMSE of 0.032, showing a 11.1% improvement over CNN and 8.6% improvement over SENet. A key application of this research can be for imputation purposes in missing meteorological datasets and generate additional high-quality meteorological data using satellite images. It is hoped that the results of this analysis will enable more robust and detailed forecasting, reducing the impact of severe weather events on vulnerable regions. Code accessible at this https URL.

Title: ProbTalk3D: Non-Deterministic Emotion Controllable Speech-Driven 3D Facial Animation Synthesis Using VQ-VAE

• Authors: Sichun Wu, Kazi Injamamul Haque, Zerrin Yumak
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.07966
• Pdf URL: https://arxiv.org/pdf/2409.07966
• Copy Paste: [[2409.07966]] ProbTalk3D: Non-Deterministic Emotion Controllable Speech-Driven 3D Facial Animation Synthesis Using VQ-VAE(https://arxiv.org/abs/2409.07966)
• Keywords: generative
• Abstract: Audio-driven 3D facial animation synthesis has been an active field of research with attention from both academia and industry. While there are promising results in this area, recent approaches largely focus on lip-sync and identity control, neglecting the role of emotions and emotion control in the generative process. That is mainly due to the lack of emotionally rich facial animation data and algorithms that can synthesize speech animations with emotional expressions at the same time. In addition, majority of the models are deterministic, meaning given the same audio input, they produce the same output motion. We argue that emotions and non-determinism are crucial to generate diverse and emotionally-rich facial animations. In this paper, we propose ProbTalk3D a non-deterministic neural network approach for emotion controllable speech-driven 3D facial animation synthesis using a two-stage VQ-VAE model and an emotionally rich facial animation dataset 3DMEAD. We provide an extensive comparative analysis of our model against the recent 3D facial animation synthesis approaches, by evaluating the results objectively, qualitatively, and with a perceptual user study. We highlight several objective metrics that are more suitable for evaluating stochastic outputs and use both in-the-wild and ground truth data for subjective evaluation. To our knowledge, that is the first non-deterministic 3D facial animation synthesis method incorporating a rich emotion dataset and emotion control with emotion labels and intensity levels. Our evaluation demonstrates that the proposed model achieves superior performance compared to state-of-the-art emotion-controlled, deterministic and non-deterministic models. We recommend watching the supplementary video for quality judgement. The entire codebase is publicly available (this https URL).

Title: SPARK: Self-supervised Personalized Real-time Monocular Face Capture

• Authors: Kelian Baert, Shrisha Bharadwaj, Fabien Castan, Benoit Maujean, Marc Christie, Victoria Abrevaya, Adnane Boukhayma
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.07984
• Pdf URL: https://arxiv.org/pdf/2409.07984
• Copy Paste: [[2409.07984]] SPARK: Self-supervised Personalized Real-time Monocular Face Capture(https://arxiv.org/abs/2409.07984)
• Keywords: self-supervised
• Abstract: Feedforward monocular face capture methods seek to reconstruct posed faces from a single image of a person. Current state of the art approaches have the ability to regress parametric 3D face models in real-time across a wide range of identities, lighting conditions and poses by leveraging large image datasets of human faces. These methods however suffer from clear limitations in that the underlying parametric face model only provides a coarse estimation of the face shape, thereby limiting their practical applicability in tasks that require precise 3D reconstruction (aging, face swapping, digital make-up, ...). In this paper, we propose a method for high-precision 3D face capture taking advantage of a collection of unconstrained videos of a subject as prior information. Our proposal builds on a two stage approach. We start with the reconstruction of a detailed 3D face avatar of the person, capturing both precise geometry and appearance from a collection of videos. We then use the encoder from a pre-trained monocular face reconstruction method, substituting its decoder with our personalized model, and proceed with transfer learning on the video collection. Using our pre-estimated image formation model, we obtain a more precise self-supervision objective, enabling improved expression and pose alignment. This results in a trained encoder capable of efficiently regressing pose and expression parameters in real-time from previously unseen images, which combined with our personalized geometry model yields more accurate and high fidelity mesh inference. Through extensive qualitative and quantitative evaluation, we showcase the superiority of our final model as compared to state-of-the-art baselines, and demonstrate its generalization ability to unseen pose, expression and lighting.

Title: Network Anomaly Traffic Detection via Multi-view Feature Fusion

• Authors: Song Hao, Wentao Fu, Xuanze Chen, Chengxiang Jin, Jiajun Zhou, Shanqing Yu, Qi Xuan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08020
• Pdf URL: https://arxiv.org/pdf/2409.08020
• Copy Paste: [[2409.08020]] Network Anomaly Traffic Detection via Multi-view Feature Fusion(https://arxiv.org/abs/2409.08020)
• Keywords: anomaly
• Abstract: Traditional anomalous traffic detection methods are based on single-view analysis, which has obvious limitations in dealing with complex attacks and encrypted communications. In this regard, we propose a Multi-view Feature Fusion (MuFF) method for network anomaly traffic detection. MuFF models the temporal and interactive relationships of packets in network traffic based on the temporal and interactive viewpoints respectively. It learns temporal and interactive features. These features are then fused from different perspectives for anomaly traffic detection. Extensive experiments on six real traffic datasets show that MuFF has excellent performance in network anomalous traffic detection, which makes up for the shortcomings of detection under a single perspective.

Title: Scribble-Guided Diffusion for Training-free Text-to-Image Generation

• Authors: Seonho Lee, Jiho Choi, Seohyun Lim, Jiwook Kim, Hyunjung Shim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08026
• Pdf URL: https://arxiv.org/pdf/2409.08026
• Copy Paste: [[2409.08026]] Scribble-Guided Diffusion for Training-free Text-to-Image Generation(https://arxiv.org/abs/2409.08026)
• Keywords: diffusion
• Abstract: Recent advancements in text-to-image diffusion models have demonstrated remarkable success, yet they often struggle to fully capture the user's intent. Existing approaches using textual inputs combined with bounding boxes or region masks fall short in providing precise spatial guidance, often leading to misaligned or unintended object orientation. To address these limitations, we propose Scribble-Guided Diffusion (ScribbleDiff), a training-free approach that utilizes simple user-provided scribbles as visual prompts to guide image generation. However, incorporating scribbles into diffusion models presents challenges due to their sparse and thin nature, making it difficult to ensure accurate orientation alignment. To overcome these challenges, we introduce moment alignment and scribble propagation, which allow for more effective and flexible alignment between generated images and scribble inputs. Experimental results on the PASCAL-Scribble dataset demonstrate significant improvements in spatial control and consistency, showcasing the effectiveness of scribble-based guidance in diffusion models. Our code is available at this https URL.

Title: Self-Supervised Learning of Iterative Solvers for Constrained Optimization

• Authors: Lukas Lüken, Sergio Lucia
• Subjects: cs.LG, math.OC
• Abstract URL: https://arxiv.org/abs/2409.08066
• Pdf URL: https://arxiv.org/pdf/2409.08066
• Copy Paste: [[2409.08066]] Self-Supervised Learning of Iterative Solvers for Constrained Optimization(https://arxiv.org/abs/2409.08066)
• Keywords: self-supervised
• Abstract: Obtaining the solution of constrained optimization problems as a function of parameters is very important in a multitude of applications, such as control and planning. Solving such parametric optimization problems in real time can present significant challenges, particularly when it is necessary to obtain highly accurate solutions or batches of solutions. To solve these challenges, we propose a learning-based iterative solver for constrained optimization which can obtain very fast and accurate solutions by customizing the solver to a specific parametric optimization problem. For a given set of parameters of the constrained optimization problem, we propose a first step with a neural network predictor that outputs primal-dual solutions of a reasonable degree of accuracy. This primal-dual solution is then improved to a very high degree of accuracy in a second step by a learned iterative solver in the form of a neural network. A novel loss function based on the Karush-Kuhn-Tucker conditions of optimality is introduced, enabling fully self-supervised training of both neural networks without the necessity of prior sampling of optimizer solutions. The evaluation of a variety of quadratic and nonlinear parametric test problems demonstrates that the predictor alone is already competitive with recent self-supervised schemes for approximating optimal solutions. The second step of our proposed learning-based iterative constrained optimizer achieves solutions with orders of magnitude better accuracy than other learning-based approaches, while being faster to evaluate than state-of-the-art solvers and natively allowing for GPU parallelization.

Title: Diffusion-Based Image-to-Image Translation by Noise Correction via Prompt Interpolation

• Authors: Junsung Lee, Minsoo Kang, Bohyung Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08077
• Pdf URL: https://arxiv.org/pdf/2409.08077
• Copy Paste: [[2409.08077]] Diffusion-Based Image-to-Image Translation by Noise Correction via Prompt Interpolation(https://arxiv.org/abs/2409.08077)
• Keywords: diffusion
• Abstract: We propose a simple but effective training-free approach tailored to diffusion-based image-to-image translation. Our approach revises the original noise prediction network of a pretrained diffusion model by introducing a noise correction term. We formulate the noise correction term as the difference between two noise predictions; one is computed from the denoising network with a progressive interpolation of the source and target prompt embeddings, while the other is the noise prediction with the source prompt embedding. The final noise prediction network is given by a linear combination of the standard denoising term and the noise correction term, where the former is designed to reconstruct must-be-preserved regions while the latter aims to effectively edit regions of interest relevant to the target prompt. Our approach can be easily incorporated into existing image-to-image translation methods based on diffusion models. Extensive experiments verify that the proposed technique achieves outstanding performance with low latency and consistently improves existing frameworks when combined with them.

Title: SimMAT: Exploring Transferability from Vision Foundation Models to Any Image Modality

• Authors: Chenyang Lei, Liyi Chen, Jun Cen, Xiao Chen, Zhen Lei, Felix Heide, Ziwei Liu, Qifeng Chen, Zhaoxiang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08083
• Pdf URL: https://arxiv.org/pdf/2409.08083
• Copy Paste: [[2409.08083]] SimMAT: Exploring Transferability from Vision Foundation Models to Any Image Modality(https://arxiv.org/abs/2409.08083)
• Keywords: foundation model
• Abstract: Foundation models like ChatGPT and Sora that are trained on a huge scale of data have made a revolutionary social impact. However, it is extremely challenging for sensors in many different fields to collect similar scales of natural images to train strong foundation models. To this end, this work presents a simple and effective framework SimMAT to study an open problem: the transferability from vision foundation models trained on natural RGB images to other image modalities of different physical properties (e.g., polarization). SimMAT consists of a modality-agnostic transfer layer (MAT) and a pretrained foundation model. We apply SimMAT to a representative vision foundation model Segment Anything Model (SAM) to support any evaluated new image modality. Given the absence of relevant benchmarks, we construct a new benchmark to evaluate the transfer learning performance. Our experiments confirm the intriguing potential of transferring vision foundation models in enhancing other sensors' performance. Specifically, SimMAT can improve the segmentation performance (mIoU) from 22.15% to 53.88% on average for evaluated modalities and consistently outperforms other baselines. We hope that SimMAT can raise awareness of cross-modal transfer learning and benefit various fields for better results with vision foundation models.

Title: EZIGen: Enhancing zero-shot subject-driven image generation with precise subject encoding and decoupled guidance

• Authors: Zicheng Duan, Yuxuan Ding, Chenhui Gou, Ziqin Zhou, Ethan Smith, Lingqiao Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08091
• Pdf URL: https://arxiv.org/pdf/2409.08091
• Copy Paste: [[2409.08091]] EZIGen: Enhancing zero-shot subject-driven image generation with precise subject encoding and decoupled guidance(https://arxiv.org/abs/2409.08091)
• Keywords: diffusion
• Abstract: Zero-shot subject-driven image generation aims to produce images that incorporate a subject from a given example image. The challenge lies in preserving the subject's identity while aligning with the text prompt, which often requires modifying certain aspects of the subject's appearance. Despite advancements in diffusion model based methods, existing approaches still struggle to balance identity preservation with text prompt alignment. In this study, we conducted an in-depth investigation into this issue and uncovered key insights for achieving effective identity preservation while maintaining a strong balance. Our key findings include: (1) the design of the subject image encoder significantly impacts identity preservation quality, and (2) generating an initial layout is crucial for both text alignment and identity preservation. Building on these insights, we introduce a new approach called EZIGen, which employs two main strategies: a carefully crafted subject image Encoder based on the UNet architecture of the pretrained Stable Diffusion model to ensure high-quality identity transfer, following a process that decouples the guidance stages and iteratively refines the initial image layout. Through these strategies, EZIGen achieves state-of-the-art results on multiple subject-driven benchmarks with a unified model and 100 times less training data.

Title: The Faetar Benchmark: Speech Recognition in a Very Under-Resourced Language

• Authors: Michael Ong, Sean Robertson, Leo Peckham, Alba Jorquera Jimenez de Aberasturi, Paula Arkhangorodsky, Robin Huo, Aman Sakhardande, Mark Hallap, Naomi Nagy, Ewan Dunbar
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2409.08103
• Pdf URL: https://arxiv.org/pdf/2409.08103
• Copy Paste: [[2409.08103]] The Faetar Benchmark: Speech Recognition in a Very Under-Resourced Language(https://arxiv.org/abs/2409.08103)
• Keywords: foundation model
• Abstract: We introduce the Faetar Automatic Speech Recognition Benchmark, a benchmark corpus designed to push the limits of current approaches to low-resource speech recognition. Faetar, a Franco-Provençal variety spoken primarily in Italy, has no standard orthography, has virtually no existing textual or speech resources other than what is included in the benchmark, and is quite different from other forms of Franco-Provençal. The corpus comes from field recordings, most of which are noisy, for which only 5 hrs have matching transcriptions, and for which forced alignment is of variable quality. The corpus contains an additional 20 hrs of unlabelled speech. We report baseline results from state-of-the-art multilingual speech foundation models with a best phone error rate of 30.4%, using a pipeline that continues pre-training on the foundation model using the unlabelled set.

Title: Towards a graph-based foundation model for network traffic analysis

• Authors: Louis Van Langendonck, Ismael Castell-Uroz, Pere Barlet-Ros
• Subjects: cs.LG, cs.AI, cs.CR, cs.NI
• Abstract URL: https://arxiv.org/abs/2409.08111
• Pdf URL: https://arxiv.org/pdf/2409.08111
• Copy Paste: [[2409.08111]] Towards a graph-based foundation model for network traffic analysis(https://arxiv.org/abs/2409.08111)
• Keywords: self-supervised, foundation model
• Abstract: Foundation models have shown great promise in various fields of study. A potential application of such models is in computer network traffic analysis, where these models can grasp the complexities of network traffic dynamics and adapt to any specific task or network environment with minimal fine-tuning. Previous approaches have used tokenized hex-level packet data and the model architecture of large language transformer models. We propose a new, efficient graph-based alternative at the flow-level. Our approach represents network traffic as a dynamic spatio-temporal graph, employing a self-supervised link prediction pretraining task to capture the spatial and temporal dynamics in this network graph framework. To evaluate the effectiveness of our approach, we conduct a few-shot learning experiment for three distinct downstream network tasks: intrusion detection, traffic classification, and botnet classification. Models finetuned from our pretrained base achieve an average performance increase of 6.87\% over training from scratch, demonstrating their ability to effectively learn general network traffic dynamics during pretraining. This success suggests the potential for a large-scale version to serve as an operational foundational model.

Title: MagicStyle: Portrait Stylization Based on Reference Image

• Authors: Zhaoli Deng, Kaibin Zhou, Fanyi Wang, Zhenpeng Mi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08156
• Pdf URL: https://arxiv.org/pdf/2409.08156
• Copy Paste: [[2409.08156]] MagicStyle: Portrait Stylization Based on Reference Image(https://arxiv.org/abs/2409.08156)
• Keywords: diffusion
• Abstract: The development of diffusion models has significantly advanced the research on image stylization, particularly in the area of stylizing a content image based on a given style image, which has attracted many scholars. The main challenge in this reference image stylization task lies in how to maintain the details of the content image while incorporating the color and texture features of the style image. This challenge becomes even more pronounced when the content image is a portrait which has complex textural details. To address this challenge, we propose a diffusion model-based reference image stylization method specifically for portraits, called MagicStyle. MagicStyle consists of two phases: Content and Style DDIM Inversion (CSDI) and Feature Fusion Forward (FFF). The CSDI phase involves a reverse denoising process, where DDIM Inversion is performed separately on the content image and the style image, storing the self-attention query, key and value features of both images during the inversion process. The FFF phase executes forward denoising, harmoniously integrating the texture and color information from the pre-stored feature queries, keys and values into the diffusion generation process based on our Well-designed Feature Fusion Attention (FFA). We conducted comprehensive comparative and ablation experiments to validate the effectiveness of our proposed MagicStyle and FFA.

Title: On the Role of Context in Reading Time Prediction

• Authors: Andreas Opedal, Eleanor Chodroff, Ryan Cotterell, Ethan Gotlieb Wilcox
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08160
• Pdf URL: https://arxiv.org/pdf/2409.08160
• Copy Paste: [[2409.08160]] On the Role of Context in Reading Time Prediction(https://arxiv.org/abs/2409.08160)
• Keywords: in-context
• Abstract: We present a new perspective on how readers integrate context during real-time language comprehension. Our proposals build on surprisal theory, which posits that the processing effort of a linguistic unit (e.g., a word) is an affine function of its in-context information content. We first observe that surprisal is only one out of many potential ways that a contextual predictor can be derived from a language model. Another one is the pointwise mutual information (PMI) between a unit and its context, which turns out to yield the same predictive power as surprisal when controlling for unigram frequency. Moreover, both PMI and surprisal are correlated with frequency. This means that neither PMI nor surprisal contains information about context alone. In response to this, we propose a technique where we project surprisal onto the orthogonal complement of frequency, yielding a new contextual predictor that is uncorrelated with frequency. Our experiments show that the proportion of variance in reading times explained by context is a lot smaller when context is represented by the orthogonalized predictor. From an interpretability standpoint, this indicates that previous studies may have overstated the role that context has in predicting reading times.

Title: High-Frequency Anti-DreamBooth: Robust Defense Against Image Synthesis

• Authors: Takuto Onikubo, Yusuke Matsui
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08167
• Pdf URL: https://arxiv.org/pdf/2409.08167
• Copy Paste: [[2409.08167]] High-Frequency Anti-DreamBooth: Robust Defense Against Image Synthesis(https://arxiv.org/abs/2409.08167)
• Keywords: generative
• Abstract: Recently, text-to-image generative models have been misused to create unauthorized malicious images of individuals, posing a growing social problem. Previous solutions, such as Anti-DreamBooth, add adversarial noise to images to protect them from being used as training data for malicious generation. However, we found that the adversarial noise can be removed by adversarial purification methods such as DiffPure. Therefore, we propose a new adversarial attack method that adds strong perturbation on the high-frequency areas of images to make it more robust to adversarial purification. Our experiment showed that the adversarial images retained noise even after adversarial purification, hindering malicious image generation.

Title: Fine-tuning Large Language Models for Entity Matching

• Authors: Aaron Steiner, Ralph Peeters, Christian Bizer
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08185
• Pdf URL: https://arxiv.org/pdf/2409.08185
• Copy Paste: [[2409.08185]] Fine-tuning Large Language Models for Entity Matching(https://arxiv.org/abs/2409.08185)
• Keywords: generative, in-context
• Abstract: Generative large language models (LLMs) are a promising alternative to pre-trained language models for entity matching due to their high zero-shot performance and their ability to generalize to unseen entities. Existing research on using LLMs for entity matching has focused on prompt engineering and in-context learning. This paper explores the potential of fine-tuning LLMs for entity matching. We analyze fine-tuning along two dimensions: 1) The representation of training examples, where we experiment with adding different types of LLM-generated explanations to the training set, and 2) the selection and generation of training examples using LLMs. In addition to the matching performance on the source dataset, we investigate how fine-tuning affects the model's ability to generalize to other in-domain datasets as well as across topical domains. Our experiments show that fine-tuning significantly improves the performance of the smaller models while the results for the larger models are mixed. Fine-tuning also improves the generalization to in-domain datasets while hurting cross-domain transfer. We show that adding structured explanations to the training set has a positive impact on the performance of three out of four LLMs, while the proposed example selection and generation methods only improve the performance of Llama 3.1 8B while decreasing the performance of GPT-4o Mini.

Title: VI3DRM:Towards meticulous 3D Reconstruction from Sparse Views via Photo-Realistic Novel View Synthesis

• Authors: Hao Chen, Jiafu Wu, Ying Jin, Jinlong Peng, Xiaofeng Mao, Mingmin Chi, Mufeng Yao, Bo Peng, Jian Li, Yun Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08207
• Pdf URL: https://arxiv.org/pdf/2409.08207
• Copy Paste: [[2409.08207]] VI3DRM:Towards meticulous 3D Reconstruction from Sparse Views via Photo-Realistic Novel View Synthesis(https://arxiv.org/abs/2409.08207)
• Keywords: diffusion
• Abstract: Recently, methods like Zero-1-2-3 have focused on single-view based 3D reconstruction and have achieved remarkable success. However, their predictions for unseen areas heavily rely on the inductive bias of large-scale pretrained diffusion models. Although subsequent work, such as DreamComposer, attempts to make predictions more controllable by incorporating additional views, the results remain unrealistic due to feature entanglement in the vanilla latent space, including factors such as lighting, material, and structure. To address these issues, we introduce the Visual Isotropy 3D Reconstruction Model (VI3DRM), a diffusion-based sparse views 3D reconstruction model that operates within an ID consistent and perspective-disentangled 3D latent space. By facilitating the disentanglement of semantic information, color, material properties and lighting, VI3DRM is capable of generating highly realistic images that are indistinguishable from real photographs. By leveraging both real and synthesized images, our approach enables the accurate construction of pointmaps, ultimately producing finely textured meshes or point clouds. On the NVS task, tested on the GSO dataset, VI3DRM significantly outperforms state-of-the-art method DreamComposer, achieving a PSNR of 38.61, an SSIM of 0.929, and an LPIPS of 0.027. Code will be made available upon publication.

Title: LT3SD: Latent Trees for 3D Scene Diffusion

• Authors: Quan Meng, Lei Li, Matthias Nießner, Angela Dai
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08215
• Pdf URL: https://arxiv.org/pdf/2409.08215
• Copy Paste: [[2409.08215]] LT3SD: Latent Trees for 3D Scene Diffusion(https://arxiv.org/abs/2409.08215)
• Keywords: diffusion, generative
• Abstract: We present LT3SD, a novel latent diffusion model for large-scale 3D scene generation. Recent advances in diffusion models have shown impressive results in 3D object generation, but are limited in spatial extent and quality when extended to 3D scenes. To generate complex and diverse 3D scene structures, we introduce a latent tree representation to effectively encode both lower-frequency geometry and higher-frequency detail in a coarse-to-fine hierarchy. We can then learn a generative diffusion process in this latent 3D scene space, modeling the latent components of a scene at each resolution level. To synthesize large-scale scenes with varying sizes, we train our diffusion model on scene patches and synthesize arbitrary-sized output 3D scenes through shared diffusion generation across multiple scene patches. Through extensive experiments, we demonstrate the efficacy and benefits of LT3SD for large-scale, high-quality unconditional 3D scene generation and for probabilistic completion for partial scene observations.

Title: IFAdapter: Instance Feature Control for Grounded Text-to-Image Generation

• Authors: Yinwei Wu, Xianpan Zhou, Bing Ma, Xuefeng Su, Kai Ma, Xinchao Wang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2409.08240
• Pdf URL: https://arxiv.org/pdf/2409.08240
• Copy Paste: [[2409.08240]] IFAdapter: Instance Feature Control for Grounded Text-to-Image Generation(https://arxiv.org/abs/2409.08240)
• Keywords: diffusion
• Abstract: While Text-to-Image (T2I) diffusion models excel at generating visually appealing images of individual instances, they struggle to accurately position and control the features generation of multiple instances. The Layout-to-Image (L2I) task was introduced to address the positioning challenges by incorporating bounding boxes as spatial control signals, but it still falls short in generating precise instance features. In response, we propose the Instance Feature Generation (IFG) task, which aims to ensure both positional accuracy and feature fidelity in generated instances. To address the IFG task, we introduce the Instance Feature Adapter (IFAdapter). The IFAdapter enhances feature depiction by incorporating additional appearance tokens and utilizing an Instance Semantic Map to align instance-level features with spatial locations. The IFAdapter guides the diffusion process as a plug-and-play module, making it adaptable to various community models. For evaluation, we contribute an IFG benchmark and develop a verification pipeline to objectively compare models' abilities to generate instances with accurate positioning and features. Experimental results demonstrate that IFAdapter outperforms other models in both quantitative and qualitative evaluations.

Title: Dynamic Prompting of Frozen Text-to-Image Diffusion Models for Panoptic Narrative Grounding

• Authors: Hongyu Li, Tianrui Hui, Zihan Ding, Jing Zhang, Bin Ma, Xiaoming Wei, Jizhong Han, Si Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08251
• Pdf URL: https://arxiv.org/pdf/2409.08251
• Copy Paste: [[2409.08251]] Dynamic Prompting of Frozen Text-to-Image Diffusion Models for Panoptic Narrative Grounding(https://arxiv.org/abs/2409.08251)
• Keywords: diffusion
• Abstract: Panoptic narrative grounding (PNG), whose core target is fine-grained image-text alignment, requires a panoptic segmentation of referred objects given a narrative caption. Previous discriminative methods achieve only weak or coarse-grained alignment by panoptic segmentation pretraining or CLIP model adaptation. Given the recent progress of text-to-image Diffusion models, several works have shown their capability to achieve fine-grained image-text alignment through cross-attention maps and improved general segmentation performance. However, the direct use of phrase features as static prompts to apply frozen Diffusion models to the PNG task still suffers from a large task gap and insufficient vision-language interaction, yielding inferior performance. Therefore, we propose an Extractive-Injective Phrase Adapter (EIPA) bypass within the Diffusion UNet to dynamically update phrase prompts with image features and inject the multimodal cues back, which leverages the fine-grained image-text alignment capability of Diffusion models more sufficiently. In addition, we also design a Multi-Level Mutual Aggregation (MLMA) module to reciprocally fuse multi-level image and phrase features for segmentation refinement. Extensive experiments on the PNG benchmark show that our method achieves new state-of-the-art performance.

Title: LoRID: Low-Rank Iterative Diffusion for Adversarial Purification

• Authors: Geigh Zollicoffer, Minh Vu, Ben Nebgen, Juan Castorena, Boian Alexandrov, Manish Bhattarai
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2409.08255
• Pdf URL: https://arxiv.org/pdf/2409.08255
• Copy Paste: [[2409.08255]] LoRID: Low-Rank Iterative Diffusion for Adversarial Purification(https://arxiv.org/abs/2409.08255)
• Keywords: diffusion
• Abstract: This work presents an information-theoretic examination of diffusion-based purification methods, the state-of-the-art adversarial defenses that utilize diffusion models to remove malicious perturbations in adversarial examples. By theoretically characterizing the inherent purification errors associated with the Markov-based diffusion purifications, we introduce LoRID, a novel Low-Rank Iterative Diffusion purification method designed to remove adversarial perturbation with low intrinsic purification errors. LoRID centers around a multi-stage purification process that leverages multiple rounds of diffusion-denoising loops at the early time-steps of the diffusion models, and the integration of Tucker decomposition, an extension of matrix factorization, to remove adversarial noise at high-noise regimes. Consequently, LoRID increases the effective diffusion time-steps and overcomes strong adversarial attacks, achieving superior robustness performance in CIFAR-10/100, CelebA-HQ, and ImageNet datasets under both white-box and black-box settings.

Title: Improving Virtual Try-On with Garment-focused Diffusion Models

• Authors: Siqi Wan, Yehao Li, Jingwen Chen, Yingwei Pan, Ting Yao, Yang Cao, Tao Mei
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.08258
• Pdf URL: https://arxiv.org/pdf/2409.08258
• Copy Paste: [[2409.08258]] Improving Virtual Try-On with Garment-focused Diffusion Models(https://arxiv.org/abs/2409.08258)
• Keywords: diffusion, generative
• Abstract: Diffusion models have led to the revolutionizing of generative modeling in numerous image synthesis tasks. Nevertheless, it is not trivial to directly apply diffusion models for synthesizing an image of a target person wearing a given in-shop garment, i.e., image-based virtual try-on (VTON) task. The difficulty originates from the aspect that the diffusion process should not only produce holistically high-fidelity photorealistic image of the target person, but also locally preserve every appearance and texture detail of the given garment. To address this, we shape a new Diffusion model, namely GarDiff, which triggers the garment-focused diffusion process with amplified guidance of both basic visual appearance and detailed textures (i.e., high-frequency details) derived from the given garment. GarDiff first remoulds a pre-trained latent diffusion model with additional appearance priors derived from the CLIP and VAE encodings of the reference garment. Meanwhile, a novel garment-focused adapter is integrated into the UNet of diffusion model, pursuing local fine-grained alignment with the visual appearance of reference garment and human pose. We specifically design an appearance loss over the synthesized garment to enhance the crucial, high-frequency details. Extensive experiments on VITON-HD and DressCode datasets demonstrate the superiority of our GarDiff when compared to state-of-the-art VTON approaches. Code is publicly available at: \href{this https URL}{this https URL}.

Title: Improving Text-guided Object Inpainting with Semantic Pre-inpainting

• Authors: Yifu Chen, Jingwen Chen, Yingwei Pan, Yehao Li, Ting Yao, Zhineng Chen, Tao Mei
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2409.08260
• Pdf URL: https://arxiv.org/pdf/2409.08260
• Copy Paste: [[2409.08260]] Improving Text-guided Object Inpainting with Semantic Pre-inpainting(https://arxiv.org/abs/2409.08260)
• Keywords: diffusion
• Abstract: Recent years have witnessed the success of large text-to-image diffusion models and their remarkable potential to generate high-quality images. The further pursuit of enhancing the editability of images has sparked significant interest in the downstream task of inpainting a novel object described by a text prompt within a designated region in the image. Nevertheless, the problem is not trivial from two aspects: 1) Solely relying on one single U-Net to align text prompt and visual object across all the denoising timesteps is insufficient to generate desired objects; 2) The controllability of object generation is not guaranteed in the intricate sampling space of diffusion model. In this paper, we propose to decompose the typical single-stage object inpainting into two cascaded processes: 1) semantic pre-inpainting that infers the semantic features of desired objects in a multi-modal feature space; 2) high-fieldity object generation in diffusion latent space that pivots on such inpainted semantic features. To achieve this, we cascade a Transformer-based semantic inpainter and an object inpainting diffusion model, leading to a novel CAscaded Transformer-Diffusion (CAT-Diffusion) framework for text-guided object inpainting. Technically, the semantic inpainter is trained to predict the semantic features of the target object conditioning on unmasked context and text prompt. The outputs of the semantic inpainter then act as the informative visual prompts to guide high-fieldity object generation through a reference adapter layer, leading to controllable object inpainting. Extensive evaluations on OpenImages-V6 and MSCOCO validate the superiority of CAT-Diffusion against the state-of-the-art methods. Code is available at \url{this https URL}.

Title: DreamBeast: Distilling 3D Fantastical Animals with Part-Aware Knowledge Transfer

• Authors: Runjia Li, Junlin Han, Luke Melas-Kyriazi, Chunyi Sun, Zhaochong An, Zhongrui Gui, Shuyang Sun, Philip Torr, Tomas Jakab
• Subjects: cs.CV, cs.GR, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2409.08271
• Pdf URL: https://arxiv.org/pdf/2409.08271
• Copy Paste: [[2409.08271]] DreamBeast: Distilling 3D Fantastical Animals with Part-Aware Knowledge Transfer(https://arxiv.org/abs/2409.08271)
• Keywords: diffusion
• Abstract: We present DreamBeast, a novel method based on score distillation sampling (SDS) for generating fantastical 3D animal assets composed of distinct parts. Existing SDS methods often struggle with this generation task due to a limited understanding of part-level semantics in text-to-image diffusion models. While recent diffusion models, such as Stable Diffusion 3, demonstrate a better part-level understanding, they are prohibitively slow and exhibit other common problems associated with single-view diffusion models. DreamBeast overcomes this limitation through a novel part-aware knowledge transfer mechanism. For each generated asset, we efficiently extract part-level knowledge from the Stable Diffusion 3 model into a 3D Part-Affinity implicit representation. This enables us to instantly generate Part-Affinity maps from arbitrary camera views, which we then use to modulate the guidance of a multi-view diffusion model during SDS to create 3D assets of fantastical animals. DreamBeast significantly enhances the quality of generated 3D creatures with user-specified part compositions while reducing computational overhead, as demonstrated by extensive quantitative and qualitative evaluations.

Title: Click2Mask: Local Editing with Dynamic Mask Generation

• Authors: Omer Regev, Omri Avrahami, Dani Lischinski
• Subjects: cs.CV, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2409.08272
• Pdf URL: https://arxiv.org/pdf/2409.08272
• Copy Paste: [[2409.08272]] Click2Mask: Local Editing with Dynamic Mask Generation(https://arxiv.org/abs/2409.08272)
• Keywords: diffusion, generative
• Abstract: Recent advancements in generative models have revolutionized image generation and editing, making these tasks accessible to non-experts. This paper focuses on local image editing, particularly the task of adding new content to a loosely specified area. Existing methods often require a precise mask or a detailed description of the location, which can be cumbersome and prone to errors. We propose Click2Mask, a novel approach that simplifies the local editing process by requiring only a single point of reference (in addition to the content description). A mask is dynamically grown around this point during a Blended Latent Diffusion (BLD) process, guided by a masked CLIP-based semantic loss. Click2Mask surpasses the limitations of segmentation-based and fine-tuning dependent methods, offering a more user-friendly and contextually accurate solution. Our experiments demonstrate that Click2Mask not only minimizes user effort but also delivers competitive or superior local image manipulation results compared to SoTA methods, according to both human judgement and automatic metrics. Key contributions include the simplification of user input, the ability to freely add objects unconstrained by existing segments, and the integration potential of our dynamic mask approach within other editing methods.

Title: DreamHOI: Subject-Driven Generation of 3D Human-Object Interactions with Diffusion Priors

• Authors: Thomas Hanwen Zhu, Ruining Li, Tomas Jakab
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2409.08278
• Pdf URL: https://arxiv.org/pdf/2409.08278
• Copy Paste: [[2409.08278]] DreamHOI: Subject-Driven Generation of 3D Human-Object Interactions with Diffusion Priors(https://arxiv.org/abs/2409.08278)
• Keywords: diffusion
• Abstract: We present DreamHOI, a novel method for zero-shot synthesis of human-object interactions (HOIs), enabling a 3D human model to realistically interact with any given object based on a textual description. This task is complicated by the varying categories and geometries of real-world objects and the scarcity of datasets encompassing diverse HOIs. To circumvent the need for extensive data, we leverage text-to-image diffusion models trained on billions of image-caption pairs. We optimize the articulation of a skinned human mesh using Score Distillation Sampling (SDS) gradients obtained from these models, which predict image-space edits. However, directly backpropagating image-space gradients into complex articulation parameters is ineffective due to the local nature of such gradients. To overcome this, we introduce a dual implicit-explicit representation of a skinned mesh, combining (implicit) neural radiance fields (NeRFs) with (explicit) skeleton-driven mesh articulation. During optimization, we transition between implicit and explicit forms, grounding the NeRF generation while refining the mesh articulation. We validate our approach through extensive experiments, demonstrating its effectiveness in generating realistic HOIs.