2024-08-06

Title: Siamese Transformer Networks for Few-shot Image Classification

• Authors: Weihao Jiang, Shuoxi Zhang, Kun He
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.01427
• Pdf URL: https://arxiv.org/pdf/2408.01427
• Copy Paste: [[2408.01427]] Siamese Transformer Networks for Few-shot Image Classification(https://arxiv.org/abs/2408.01427)
• Keywords: self-supervised
• Abstract: Humans exhibit remarkable proficiency in visual classification tasks, accurately recognizing and classifying new images with minimal examples. This ability is attributed to their capacity to focus on details and identify common features between previously seen and new images. In contrast, existing few-shot image classification methods often emphasize either global features or local features, with few studies considering the integration of both. To address this limitation, we propose a novel approach based on the Siamese Transformer Network (STN). Our method employs two parallel branch networks utilizing the pre-trained Vision Transformer (ViT) architecture to extract global and local features, respectively. Specifically, we implement the ViT-Small network architecture and initialize the branch networks with pre-trained model parameters obtained through self-supervised learning. We apply the Euclidean distance measure to the global features and the Kullback-Leibler (KL) divergence measure to the local features. To integrate the two metrics, we first employ L2 normalization and then weight the normalized results to obtain the final similarity score. This strategy leverages the advantages of both global and local features while ensuring their complementary benefits. During the training phase, we adopt a meta-learning approach to fine-tune the entire network. Our strategy effectively harnesses the potential of global and local features in few-shot image classification, circumventing the need for complex feature adaptation modules and enhancing the model's generalization ability. Extensive experiments demonstrate that our framework is simple yet effective, achieving superior performance compared to state-of-the-art baselines on four popular few-shot classification benchmarks in both 5-shot and 1-shot scenarios.

Title: Transferable Adversarial Facial Images for Privacy Protection

• Authors: Minghui Li, Jiangxiong Wang, Hao Zhang, Ziqi Zhou, Shengshan Hu, Xiaobing Pei
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.01428
• Pdf URL: https://arxiv.org/pdf/2408.01428
• Copy Paste: [[2408.01428]] Transferable Adversarial Facial Images for Privacy Protection(https://arxiv.org/abs/2408.01428)
• Keywords: generative
• Abstract: The success of deep face recognition (FR) systems has raised serious privacy concerns due to their ability to enable unauthorized tracking of users in the digital world. Previous studies proposed introducing imperceptible adversarial noises into face images to deceive those face recognition models, thus achieving the goal of enhancing facial privacy protection. Nevertheless, they heavily rely on user-chosen references to guide the generation of adversarial noises, and cannot simultaneously construct natural and highly transferable adversarial face images in black-box scenarios. In light of this, we present a novel face privacy protection scheme with improved transferability while maintain high visual quality. We propose shaping the entire face space directly instead of exploiting one kind of facial characteristic like makeup information to integrate adversarial noises. To achieve this goal, we first exploit global adversarial latent search to traverse the latent space of the generative model, thereby creating natural adversarial face images with high transferability. We then introduce a key landmark regularization module to preserve the visual identity information. Finally, we investigate the impacts of various kinds of latent spaces and find that $\mathcal{F}$ latent space benefits the trade-off between visual naturalness and adversarial transferability. Extensive experiments over two datasets demonstrate that our approach significantly enhances attack transferability while maintaining high visual quality, outperforming state-of-the-art methods by an average 25% improvement in deep FR models and 10% improvement on commercial FR APIs, including Face++, Aliyun, and Tencent.

Title: SUSTechGAN: Image Generation for Object Recognition in Adverse Conditions of Autonomous Driving

• Authors: Gongjin Lan, Yang Peng, Qi Hao, Chengzhong Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.01430
• Pdf URL: https://arxiv.org/pdf/2408.01430
• Copy Paste: [[2408.01430]] SUSTechGAN: Image Generation for Object Recognition in Adverse Conditions of Autonomous Driving(https://arxiv.org/abs/2408.01430)
• Keywords: generative
• Abstract: Autonomous driving significantly benefits from data-driven deep neural networks. However, the data in autonomous driving typically fits the long-tailed distribution, in which the critical driving data in adverse conditions is hard to collect. Although generative adversarial networks (GANs) have been applied to augment data for autonomous driving, generating driving images in adverse conditions is still challenging. In this work, we propose a novel SUSTechGAN with dual attention modules and multi-scale generators to generate driving images for improving object recognition of autonomous driving in adverse conditions. We test the SUSTechGAN and the existing well-known GANs to generate driving images in adverse conditions of rain and night and apply the generated images to retrain object recognition networks. Specifically, we add generated images into the training datasets to retrain the well-known YOLOv5 and evaluate the improvement of the retrained YOLOv5 for object recognition in adverse conditions. The experimental results show that the generated driving images by our SUSTechGAN significantly improved the performance of retrained YOLOv5 in rain and night conditions, which outperforms the well-known GANs. The open-source code, video description and datasets are available on the page 1 to facilitate image generation development in autonomous driving under adverse conditions.

Title: Img2CAD: Reverse Engineering 3D CAD Models from Images through VLM-Assisted Conditional Factorization

• Authors: Yang You, Mikaela Angelina Uy, Jiaqi Han, Rahul Thomas, Haotong Zhang, Suya You, Leonidas Guibas
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2408.01437
• Pdf URL: https://arxiv.org/pdf/2408.01437
• Copy Paste: [[2408.01437]] Img2CAD: Reverse Engineering 3D CAD Models from Images through VLM-Assisted Conditional Factorization(https://arxiv.org/abs/2408.01437)
• Keywords: foundation model
• Abstract: Reverse engineering 3D computer-aided design (CAD) models from images is an important task for many downstream applications including interactive editing, manufacturing, architecture, robotics, etc. The difficulty of the task lies in vast representational disparities between the CAD output and the image input. CAD models are precise, programmatic constructs that involves sequential operations combining discrete command structure with continuous attributes -- making it challenging to learn and optimize in an end-to-end fashion. Concurrently, input images introduce inherent challenges such as photo-metric variability and sensor noise, complicating the reverse engineering process. In this work, we introduce a novel approach that conditionally factorizes the task into two sub-problems. First, we leverage large foundation models, particularly GPT-4V, to predict the global discrete base structure with semantic information. Second, we propose TrAssembler that conditioned on the discrete structure with semantics predicts the continuous attribute values. To support the training of our TrAssembler, we further constructed an annotated CAD dataset of common objects from ShapeNet. Putting all together, our approach and data demonstrate significant first steps towards CAD-ifying images in the wild. Our project page: this https URL

Title: Multi-task SAR Image Processing via GAN-based Unsupervised Manipulation

• Authors: Xuran Hu, Mingzhe Zhu, Ziqiang Xu, Zhenpeng Feng, Ljubisa Stankovic
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.01553
• Pdf URL: https://arxiv.org/pdf/2408.01553
• Copy Paste: [[2408.01553]] Multi-task SAR Image Processing via GAN-based Unsupervised Manipulation(https://arxiv.org/abs/2408.01553)
• Keywords: generative
• Abstract: Generative Adversarial Networks (GANs) have shown tremendous potential in synthesizing a large number of realistic SAR images by learning patterns in the data distribution. Some GANs can achieve image editing by introducing latent codes, demonstrating significant promise in SAR image processing. Compared to traditional SAR image processing methods, editing based on GAN latent space control is entirely unsupervised, allowing image processing to be conducted without any labeled data. Additionally, the information extracted from the data is more interpretable. This paper proposes a novel SAR image processing framework called GAN-based Unsupervised Editing (GUE), aiming to address the following two issues: (1) disentangling semantic directions in the GAN latent space and finding meaningful directions; (2) establishing a comprehensive SAR image processing framework while achieving multiple image processing functions. In the implementation of GUE, we decompose the entangled semantic directions in the GAN latent space by training a carefully designed network. Moreover, we can accomplish multiple SAR image processing tasks (including despeckling, localization, auxiliary identification, and rotation editing) in a single training process without any form of supervision. Extensive experiments validate the effectiveness of the proposed method.

Title: Self-Supervised Depth Estimation Based on Camera Models

• Authors: Jinchang Zhang, Praveen Kumar Reddy, Xue-Iuan Wong, Guoyu Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01565
• Pdf URL: https://arxiv.org/pdf/2408.01565
• Copy Paste: [[2408.01565]] Self-Supervised Depth Estimation Based on Camera Models(https://arxiv.org/abs/2408.01565)
• Keywords: self-supervised
• Abstract: Depth estimationn is a critical topic for robotics and vision-related tasks. In monocular depth estimation, in comparison with supervised learning that requires expensive ground truth labeling, self-supervised methods possess great potential due to no labeling cost. However, self-supervised learning still has a large gap with supervised learning in depth estimation performance. Meanwhile, scaling is also a major issue for monocular unsupervised depth estimation, which commonly still needs ground truth scale from GPS, LiDAR, or existing maps to correct. In deep learning era, while existing methods mainly rely on the exploration of image relationships to train the unsupervised neural networks, fundamental information provided by the camera itself has been generally ignored, which can provide extensive supervision information for free, without the need for any extra equipment to provide supervision signals. Utilizing the camera itself's intrinsics and extrinsics, depth information can be calculated for ground regions and regions connecting ground based on physical principles, providing free supervision information without any other sensors. The method is easy to realize and can be a component to enhance the effects of all the unsupervised methods.

Title: Counterfactual Explanations for Medical Image Classification and Regression using Diffusion Autoencoder

• Authors: Matan Atad, David Schinz, Hendrik Moeller, Robert Graf, Benedikt Wiestler, Daniel Rueckert, Nassir Navab, Jan S. Kirschke, Matthias Keicher
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.01571
• Pdf URL: https://arxiv.org/pdf/2408.01571
• Copy Paste: [[2408.01571]] Counterfactual Explanations for Medical Image Classification and Regression using Diffusion Autoencoder(https://arxiv.org/abs/2408.01571)
• Keywords: diffusion, generative
• Abstract: Counterfactual explanations (CEs) aim to enhance the interpretability of machine learning models by illustrating how alterations in input features would affect the resulting predictions. Common CE approaches require an additional model and are typically constrained to binary counterfactuals. In contrast, we propose a novel method that operates directly on the latent space of a generative model, specifically a Diffusion Autoencoder (DAE). This approach offers inherent interpretability by enabling the generation of CEs and the continuous visualization of the model's internal representation across decision boundaries. Our method leverages the DAE's ability to encode images into a semantically rich latent space in an unsupervised manner, eliminating the need for labeled data or separate feature extraction models. We show that these latent representations are helpful for medical condition classification and the ordinal regression of severity pathologies, such as vertebral compression fractures (VCF) and diabetic retinopathy (DR). Beyond binary CEs, our method supports the visualization of ordinal CEs using a linear model, providing deeper insights into the model's decision-making process and enhancing interpretability. Experiments across various medical imaging datasets demonstrate the method's advantages in interpretability and versatility. The linear manifold of the DAE's latent space allows for meaningful interpolation and manipulation, making it a powerful tool for exploring medical image properties. Our code is available at this https URL.

Title: Transforming Slot Schema Induction with Generative Dialogue State Inference

• Authors: James D. Finch, Boxin Zhao, Jinho D. Choi
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.01638
• Pdf URL: https://arxiv.org/pdf/2408.01638
• Copy Paste: [[2408.01638]] Transforming Slot Schema Induction with Generative Dialogue State Inference(https://arxiv.org/abs/2408.01638)
• Keywords: generative
• Abstract: The challenge of defining a slot schema to represent the state of a task-oriented dialogue system is addressed by Slot Schema Induction (SSI), which aims to automatically induce slots from unlabeled dialogue data. Whereas previous approaches induce slots by clustering value spans extracted directly from the dialogue text, we demonstrate the power of discovering slots using a generative approach. By training a model to generate slot names and values that summarize key dialogue information with no prior task knowledge, our SSI method discovers high-quality candidate information for representing dialogue state. These discovered slot-value candidates can be easily clustered into unified slot schemas that align well with human-authored schemas. Experimental comparisons on the MultiWOZ and SGD datasets demonstrate that Generative Dialogue State Inference (GenDSI) outperforms the previous state-of-the-art on multiple aspects of the SSI task.

Title: SAT3D: Image-driven Semantic Attribute Transfer in 3D

• Authors: Zhijun Zhai, Zengmao Wang, Xiaoxiao Long, Kaixuan Zhou, Bo Du
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.01664
• Pdf URL: https://arxiv.org/pdf/2408.01664
• Copy Paste: [[2408.01664]] SAT3D: Image-driven Semantic Attribute Transfer in 3D(https://arxiv.org/abs/2408.01664)
• Keywords: generative
• Abstract: GAN-based image editing task aims at manipulating image attributes in the latent space of generative models. Most of the previous 2D and 3D-aware approaches mainly focus on editing attributes in images with ambiguous semantics or regions from a reference image, which fail to achieve photographic semantic attribute transfer, such as the beard from a photo of a man. In this paper, we propose an image-driven Semantic Attribute Transfer method in 3D (SAT3D) by editing semantic attributes from a reference image. For the proposed method, the exploration is conducted in the style space of a pre-trained 3D-aware StyleGAN-based generator by learning the correlations between semantic attributes and style code channels. For guidance, we associate each attribute with a set of phrase-based descriptor groups, and develop a Quantitative Measurement Module (QMM) to quantitatively describe the attribute characteristics in images based on descriptor groups, which leverages the image-text comprehension capability of CLIP. During the training process, the QMM is incorporated into attribute losses to calculate attribute similarity between images, guiding target semantic transferring and irrelevant semantics preserving. We present our 3D-aware attribute transfer results across multiple domains and also conduct comparisons with classical 2D image editing methods, demonstrating the effectiveness and customizability of our SAT3D.

Title: Multiple Contexts and Frequencies Aggregation Network forDeepfake Detection

• Authors: Zifeng Li, Wenzhong Tang, Shijun Gao, Shuai Wang, Yanxiang Wang
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.01668
• Pdf URL: https://arxiv.org/pdf/2408.01668
• Copy Paste: [[2408.01668]] Multiple Contexts and Frequencies Aggregation Network forDeepfake Detection(https://arxiv.org/abs/2408.01668)
• Keywords: generative
• Abstract: Deepfake detection faces increasing challenges since the fast growth of generative models in developing massive and diverse Deepfake technologies. Recent advances rely on introducing heuristic features from spatial or frequency domains rather than modeling general forgery features within backbones. To address this issue, we turn to the backbone design with two intuitive priors from spatial and frequency detectors, \textit{i.e.,} learning robust spatial attributes and frequency distributions that are discriminative for real and fake samples. To this end, we propose an efficient network for face forgery detection named MkfaNet, which consists of two core modules. For spatial contexts, we design a Multi-Kernel Aggregator that adaptively selects organ features extracted by multiple convolutions for modeling subtle facial differences between real and fake faces. For the frequency components, we propose a Multi-Frequency Aggregator to process different bands of frequency components by adaptively reweighing high-frequency and low-frequency features. Comprehensive experiments on seven popular deepfake detection benchmarks demonstrate that our proposed MkfaNet variants achieve superior performances in both within-domain and across-domain evaluations with impressive efficiency of parameter usage.

Title: iControl3D: An Interactive System for Controllable 3D Scene Generation

• Authors: Xingyi Li, Yizheng Wu, Jun Cen, Juewen Peng, Kewei Wang, Ke Xian, Zhe Wang, Zhiguo Cao, Guosheng Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01678
• Pdf URL: https://arxiv.org/pdf/2408.01678
• Copy Paste: [[2408.01678]] iControl3D: An Interactive System for Controllable 3D Scene Generation(https://arxiv.org/abs/2408.01678)
• Keywords: diffusion
• Abstract: 3D content creation has long been a complex and time-consuming process, often requiring specialized skills and resources. While recent advancements have allowed for text-guided 3D object and scene generation, they still fall short of providing sufficient control over the generation process, leading to a gap between the user's creative vision and the generated results. In this paper, we present iControl3D, a novel interactive system that empowers users to generate and render customizable 3D scenes with precise control. To this end, a 3D creator interface has been developed to provide users with fine-grained control over the creation process. Technically, we leverage 3D meshes as an intermediary proxy to iteratively merge individual 2D diffusion-generated images into a cohesive and unified 3D scene representation. To ensure seamless integration of 3D meshes, we propose to perform boundary-aware depth alignment before fusing the newly generated mesh with the existing one in 3D space. Additionally, to effectively manage depth discrepancies between remote content and foreground, we propose to model remote content separately with an environment map instead of 3D meshes. Finally, our neural rendering interface enables users to build a radiance field of their scene online and navigate the entire scene. Extensive experiments have been conducted to demonstrate the effectiveness of our system. The code will be made available at this https URL.

Title: Controllable Unlearning for Image-to-Image Generative Models via $\varepsilon$-Constrained Optimization

• Authors: Xiaohua Feng, Chaochao Chen, Yuyuan Li, Li Zhang
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01689
• Pdf URL: https://arxiv.org/pdf/2408.01689
• Copy Paste: [[2408.01689]] Controllable Unlearning for Image-to-Image Generative Models via $\varepsilon$-Constrained Optimization(https://arxiv.org/abs/2408.01689)
• Keywords: generative
• Abstract: While generative models have made significant advancements in recent years, they also raise concerns such as privacy breaches and biases. Machine unlearning has emerged as a viable solution, aiming to remove specific training data, e.g., containing private information and bias, from models. In this paper, we study the machine unlearning problem in Image-to-Image (I2I) generative models. Previous studies mainly treat it as a single objective optimization problem, offering a solitary solution, thereby neglecting the varied user expectations towards the trade-off between complete unlearning and model utility. To address this issue, we propose a controllable unlearning framework that uses a control coefficient $\varepsilon$ to control the trade-off. We reformulate the I2I generative model unlearning problem into a $\varepsilon$-constrained optimization problem and solve it with a gradient-based method to find optimal solutions for unlearning boundaries. These boundaries define the valid range for the control coefficient. Within this range, every yielded solution is theoretically guaranteed with Pareto optimality. We also analyze the convergence rate of our framework under various control functions. Extensive experiments on two benchmark datasets across three mainstream I2I models demonstrate the effectiveness of our controllable unlearning framework.

Title: Downstream Transfer Attack: Adversarial Attacks on Downstream Models with Pre-trained Vision Transformers

• Authors: Weijie Zheng, Xingjun Ma, Hanxun Huang, Zuxuan Wu, Yu-Gang Jiang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.01705
• Pdf URL: https://arxiv.org/pdf/2408.01705
• Copy Paste: [[2408.01705]] Downstream Transfer Attack: Adversarial Attacks on Downstream Models with Pre-trained Vision Transformers(https://arxiv.org/abs/2408.01705)
• Keywords: self-supervised, foundation model
• Abstract: With the advancement of vision transformers (ViTs) and self-supervised learning (SSL) techniques, pre-trained large ViTs have become the new foundation models for computer vision applications. However, studies have shown that, like convolutional neural networks (CNNs), ViTs are also susceptible to adversarial attacks, where subtle perturbations in the input can fool the model into making false predictions. This paper studies the transferability of such an adversarial vulnerability from a pre-trained ViT model to downstream tasks. We focus on \emph{sample-wise} transfer attacks and propose a novel attack method termed \emph{Downstream Transfer Attack (DTA)}. For a given test image, DTA leverages a pre-trained ViT model to craft the adversarial example and then applies the adversarial example to attack a fine-tuned version of the model on a downstream dataset. During the attack, DTA identifies and exploits the most vulnerable layers of the pre-trained model guided by a cosine similarity loss to craft highly transferable attacks. Through extensive experiments with pre-trained ViTs by 3 distinct pre-training methods, 3 fine-tuning schemes, and across 10 diverse downstream datasets, we show that DTA achieves an average attack success rate (ASR) exceeding 90\%, surpassing existing methods by a huge margin. When used with adversarial training, the adversarial examples generated by our DTA can significantly improve the model's robustness to different downstream transfer attacks.

Title: Landmark-guided Diffusion Model for High-fidelity and Temporally Coherent Talking Head Generation

• Authors: Jintao Tan, Xize Cheng, Lingyu Xiong, Lei Zhu, Xiandong Li, Xianjia Wu, Kai Gong, Minglei Li, Yi Cai
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.01732
• Pdf URL: https://arxiv.org/pdf/2408.01732
• Copy Paste: [[2408.01732]] Landmark-guided Diffusion Model for High-fidelity and Temporally Coherent Talking Head Generation(https://arxiv.org/abs/2408.01732)
• Keywords: diffusion
• Abstract: Audio-driven talking head generation is a significant and challenging task applicable to various fields such as virtual avatars, film production, and online conferences. However, the existing GAN-based models emphasize generating well-synchronized lip shapes but overlook the visual quality of generated frames, while diffusion-based models prioritize generating high-quality frames but neglect lip shape matching, resulting in jittery mouth movements. To address the aforementioned problems, we introduce a two-stage diffusion-based model. The first stage involves generating synchronized facial landmarks based on the given speech. In the second stage, these generated landmarks serve as a condition in the denoising process, aiming to optimize mouth jitter issues and generate high-fidelity, well-synchronized, and temporally coherent talking head videos. Extensive experiments demonstrate that our model yields the best performance.

Title: SkyDiffusion: Street-to-Satellite Image Synthesis with Diffusion Models and BEV Paradigm

• Authors: Junyan Ye, Jun He, Weijia Li, Zhutao Lv, Jinhua Yu, Haote Yang, Conghui He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01812
• Pdf URL: https://arxiv.org/pdf/2408.01812
• Copy Paste: [[2408.01812]] SkyDiffusion: Street-to-Satellite Image Synthesis with Diffusion Models and BEV Paradigm(https://arxiv.org/abs/2408.01812)
• Keywords: diffusion
• Abstract: Street-to-satellite image synthesis focuses on generating realistic satellite images from corresponding ground street-view images while maintaining a consistent content layout, similar to looking down from the sky. The significant differences in perspectives create a substantial domain gap between the views, making this cross-view generation task particularly challenging. In this paper, we introduce SkyDiffusion, a novel cross-view generation method for synthesizing satellite images from street-view images, leveraging diffusion models and Bird's Eye View (BEV) paradigm. First, we design a Curved-BEV method to transform street-view images to the satellite view, reformulating the challenging cross-domain image synthesis task into a conditional generation problem. Curved-BEV also includes a "Multi-to-One" mapping strategy for combining multiple street-view images within the same satellite coverage area, effectively solving the occlusion issues in dense urban scenes. Next, we design a BEV-controlled diffusion model to generate satellite images consistent with the street-view content, which also incorporates a light manipulation module to optimize the lighting condition of the synthesized image using a reference satellite. Experimental results demonstrate that SkyDiffusion outperforms state-of-the-art methods on both suburban (CVUSA & CVACT) and urban (VIGOR-Chicago) cross-view datasets, with an average SSIM increase of 14.5% and a FID reduction of 29.6%, achieving realistic and content-consistent satellite image generation. The code and models of this work will be released at this https URL.

Title: GLDiTalker: Speech-Driven 3D Facial Animation with Graph Latent Diffusion Transformer

• Authors: Yihong Lin, Lingyu Xiong, Xiandong Li, Wenxiong Kang, Xianjia Wu, Liang Peng, Songju Lei, Huang Xu, Zhaoxin Fan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01826
• Pdf URL: https://arxiv.org/pdf/2408.01826
• Copy Paste: [[2408.01826]] GLDiTalker: Speech-Driven 3D Facial Animation with Graph Latent Diffusion Transformer(https://arxiv.org/abs/2408.01826)
• Keywords: diffusion
• Abstract: 3D speech-driven facial animation generation has received much attention in both industrial applications and academic research. Since the non-verbal facial cues that exist across the face in reality are non-deterministic, the generated results should be diverse. However, most recent methods are deterministic models that cannot learn a many-to-many mapping between audio and facial motion to generate diverse facial animations. To address this problem, we propose GLDiTalker, which introduces a motion prior along with some stochasticity to reduce the uncertainty of cross-modal mapping while increasing non-determinacy of the non-verbal facial cues that reside throughout the face. Particularly, GLDiTalker uses VQ-VAE to map facial motion mesh sequences into latent space in the first stage, and then iteratively adds and removes noise to the latent facial motion features in the second stage. In order to integrate different levels of spatial information, the Spatial Pyramidal SpiralConv Encoder is also designed to extract multi-scale features. Extensive qualitative and quantitative experiments demonstrate that our method achieves the state-of-the-art performance.

Title: Supervised Image Translation from Visible to Infrared Domain for Object Detection

• Authors: Prahlad Anand, Qiranul Saadiyean, Aniruddh Sikdar, Nalini N, Suresh Sundaram
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01843
• Pdf URL: https://arxiv.org/pdf/2408.01843
• Copy Paste: [[2408.01843]] Supervised Image Translation from Visible to Infrared Domain for Object Detection(https://arxiv.org/abs/2408.01843)
• Keywords: generative
• Abstract: This study aims to learn a translation from visible to infrared imagery, bridging the domain gap between the two modalities so as to improve accuracy on downstream tasks including object detection. Previous approaches attempt to perform bi-domain feature fusion through iterative optimization or end-to-end deep convolutional networks. However, we pose the problem as similar to that of image translation, adopting a two-stage training strategy with a Generative Adversarial Network and an object detection model. The translation model learns a conversion that preserves the structural detail of visible images while preserving the texture and other characteristics of infrared images. Images so generated are used to train standard object detection frameworks including Yolov5, Mask and Faster RCNN. We also investigate the usefulness of integrating a super-resolution step into our pipeline to further improve model accuracy, and achieve an improvement of as high as 5.3% mAP.

Title: Safe Semi-Supervised Contrastive Learning Using In-Distribution Data as Positive Examples

• Authors: Min Gu Kwak, Hyungu Kahng, Seoung Bum Kim
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01872
• Pdf URL: https://arxiv.org/pdf/2408.01872
• Copy Paste: [[2408.01872]] Safe Semi-Supervised Contrastive Learning Using In-Distribution Data as Positive Examples(https://arxiv.org/abs/2408.01872)
• Keywords: self-supervised
• Abstract: Semi-supervised learning methods have shown promising results in solving many practical problems when only a few labels are available. The existing methods assume that the class distributions of labeled and unlabeled data are equal; however, their performances are significantly degraded in class distribution mismatch scenarios where out-of-distribution (OOD) data exist in the unlabeled data. Previous safe semi-supervised learning studies have addressed this problem by making OOD data less likely to affect training based on labeled data. However, even if the studies effectively filter out the unnecessary OOD data, they can lose the basic information that all data share regardless of class. To this end, we propose to apply a self-supervised contrastive learning approach to fully exploit a large amount of unlabeled data. We also propose a contrastive loss function with coefficient schedule to aggregate as an anchor the labeled negative examples of the same class into positive examples. To evaluate the performance of the proposed method, we conduct experiments on image classification datasets - CIFAR-10, CIFAR-100, Tiny ImageNet, and CIFAR-100+Tiny ImageNet - under various mismatch ratios. The results show that self-supervised contrastive learning significantly improves classification accuracy. Moreover, aggregating the in-distribution examples produces better representation and consequently further improves classification accuracy.

Title: Self-Supervised Pretrained Models and Latent Feature Distribution Optimization

• Authors: Qiuyu Zhu, Liheng Hu, Sijin Wang
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01920
• Pdf URL: https://arxiv.org/pdf/2408.01920
• Copy Paste: [[2408.01920]] Self-Supervised Pretrained Models and Latent Feature Distribution Optimization(https://arxiv.org/abs/2408.01920)
• Keywords: self-supervised
• Abstract: In the face of complex natural images, existing deep clustering algorithms fall significantly short in terms of clustering accuracy when compared to supervised classification methods, making them less practical. This paper introduces an image clustering algorithm based on self-supervised pretrained models and latent feature distribution optimization, substantially enhancing clustering performance. It is found that: (1) For complex natural images, we effectively enhance the discriminative power of latent features by leveraging self-supervised pretrained models and their fine-tuning, resulting in improved clustering performance. (2) In the latent feature space, by searching for k-nearest neighbor images for each training sample and shortening the distance between the training sample and its nearest neighbor, the discriminative power of latent features can be further enhanced, and clustering performance can be improved. (3) In the latent feature space, reducing the distance between sample features and the nearest predefined cluster centroids can optimize the distribution of latent features, therefore further improving clustering performance. Through experiments on multiple datasets, our approach outperforms the latest clustering algorithms and achieves state-of-the-art clustering results. When the number of categories in the datasets is small, such as CIFAR-10 and STL-10, and there are significant differences between categories, our clustering algorithm has similar accuracy to supervised methods without using pretrained models, slightly lower than supervised methods using pre-trained models. The code linked algorithm is this https URL.

Title: Defining and Evaluating Decision and Composite Risk in Language Models Applied to Natural Language Inference

• Authors: Ke Shen, Mayank Kejriwal
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.01935
• Pdf URL: https://arxiv.org/pdf/2408.01935
• Copy Paste: [[2408.01935]] Defining and Evaluating Decision and Composite Risk in Language Models Applied to Natural Language Inference(https://arxiv.org/abs/2408.01935)
• Keywords: generative
• Abstract: Despite their impressive performance, large language models (LLMs) such as ChatGPT are known to pose important risks. One such set of risks arises from misplaced confidence, whether over-confidence or under-confidence, that the models have in their inference. While the former is well studied, the latter is not, leading to an asymmetry in understanding the comprehensive risk of the model based on misplaced confidence. In this paper, we address this asymmetry by defining two types of risk (decision and composite risk), and proposing an experimental framework consisting of a two-level inference architecture and appropriate metrics for measuring such risks in both discriminative and generative LLMs. The first level relies on a decision rule that determines whether the underlying language model should abstain from inference. The second level (which applies if the model does not abstain) is the model's inference. Detailed experiments on four natural language commonsense reasoning datasets using both an open-source ensemble-based RoBERTa model and ChatGPT, demonstrate the practical utility of the evaluation framework. For example, our results show that our framework can get an LLM to confidently respond to an extra 20.1% of low-risk inference tasks that other methods might misclassify as high-risk, and skip 19.8% of high-risk tasks, which would have been answered incorrectly.

Title: RobNODDI: Robust NODDI Parameter Estimation with Adaptive Sampling under Continuous Representation

• Authors: Taohui Xiao, Jian Cheng, Wenxin Fan, Jing Yang, Cheng Li, Enqing Dong, Shanshan Wang
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2408.01944
• Pdf URL: https://arxiv.org/pdf/2408.01944
• Copy Paste: [[2408.01944]] RobNODDI: Robust NODDI Parameter Estimation with Adaptive Sampling under Continuous Representation(https://arxiv.org/abs/2408.01944)
• Keywords: diffusion
• Abstract: Neurite Orientation Dispersion and Density Imaging (NODDI) is an important imaging technology used to evaluate the microstructure of brain tissue, which is of great significance for the discovery and treatment of various neurological diseases. Current deep learning-based methods perform parameter estimation through diffusion magnetic resonance imaging (dMRI) with a small number of diffusion gradients. These methods speed up parameter estimation and improve accuracy. However, the diffusion directions used by most existing deep learning models during testing needs to be strictly consistent with the diffusion directions during training. This results in poor generalization and robustness of deep learning models in dMRI parameter estimation. In this work, we verify for the first time that the parameter estimation performance of current mainstream methods will significantly decrease when the testing diffusion directions and the training diffusion directions are inconsistent. A robust NODDI parameter estimation method with adaptive sampling under continuous representation (RobNODDI) is proposed. Furthermore, long short-term memory (LSTM) units and fully connected layers are selected to learn continuous representation signals. To this end, we use a total of 100 subjects to conduct experiments based on the Human Connectome Project (HCP) dataset, of which 60 are used for training, 20 are used for validation, and 20 are used for testing. The test results indicate that RobNODDI improves the generalization performance and robustness of the deep learning model, enhancing the stability and flexibility of deep learning NODDI parameter estimatimation applications.

Title: Masked Angle-Aware Autoencoder for Remote Sensing Images

• Authors: Zhihao Li, Biao Hou, Siteng Ma, Zitong Wu, Xianpeng Guo, Bo Ren, Licheng Jiao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.01946
• Pdf URL: https://arxiv.org/pdf/2408.01946
• Copy Paste: [[2408.01946]] Masked Angle-Aware Autoencoder for Remote Sensing Images(https://arxiv.org/abs/2408.01946)
• Keywords: self-supervised
• Abstract: To overcome the inherent domain gap between remote sensing (RS) images and natural images, some self-supervised representation learning methods have made promising progress. However, they have overlooked the diverse angles present in RS objects. This paper proposes the Masked Angle-Aware Autoencoder (MA3E) to perceive and learn angles during pre-training. We design a \textit{scaling center crop} operation to create the rotated crop with random orientation on each original image, introducing the explicit angle variation. MA3E inputs this composite image while reconstruct the original image, aiming to effectively learn rotation-invariant representations by restoring the angle variation introduced on the rotated crop. To avoid biases caused by directly reconstructing the rotated crop, we propose an Optimal Transport (OT) loss that automatically assigns similar original image patches to each rotated crop patch for reconstruction. MA3E demonstrates more competitive performance than existing pre-training methods on seven different RS image datasets in three downstream tasks.

Title: Dataset Scale and Societal Consistency Mediate Facial Impression Bias in Vision-Language AI

• Authors: Robert Wolfe, Aayushi Dangol, Alexis Hiniker, Bill Howe
• Subjects: cs.CV, cs.AI, cs.CL, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.01959
• Pdf URL: https://arxiv.org/pdf/2408.01959
• Copy Paste: [[2408.01959]] Dataset Scale and Societal Consistency Mediate Facial Impression Bias in Vision-Language AI(https://arxiv.org/abs/2408.01959)
• Keywords: diffusion
• Abstract: Multimodal AI models capable of associating images and text hold promise for numerous domains, ranging from automated image captioning to accessibility applications for blind and low-vision users. However, uncertainty about bias has in some cases limited their adoption and availability. In the present work, we study 43 CLIP vision-language models to determine whether they learn human-like facial impression biases, and we find evidence that such biases are reflected across three distinct CLIP model families. We show for the first time that the the degree to which a bias is shared across a society predicts the degree to which it is reflected in a CLIP model. Human-like impressions of visually unobservable attributes, like trustworthiness and sexuality, emerge only in models trained on the largest dataset, indicating that a better fit to uncurated cultural data results in the reproduction of increasingly subtle social biases. Moreover, we use a hierarchical clustering approach to show that dataset size predicts the extent to which the underlying structure of facial impression bias resembles that of facial impression bias in humans. Finally, we show that Stable Diffusion models employing CLIP as a text encoder learn facial impression biases, and that these biases intersect with racial biases in Stable Diffusion XL-Turbo. While pretrained CLIP models may prove useful for scientific studies of bias, they will also require significant dataset curation when intended for use as general-purpose models in a zero-shot setting.

Title: AnomalySD: Few-Shot Multi-Class Anomaly Detection with Stable Diffusion Model

• Authors: Zhenyu Yan, Qingqing Fang, Wenxi Lv, Qinliang Su
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.01960
• Pdf URL: https://arxiv.org/pdf/2408.01960
• Copy Paste: [[2408.01960]] AnomalySD: Few-Shot Multi-Class Anomaly Detection with Stable Diffusion Model(https://arxiv.org/abs/2408.01960)
• Keywords: diffusion, anomaly
• Abstract: Anomaly detection is a critical task in industrial manufacturing, aiming to identify defective parts of products. Most industrial anomaly detection methods assume the availability of sufficient normal data for training. This assumption may not hold true due to the cost of labeling or data privacy policies. Additionally, mainstream methods require training bespoke models for different objects, which incurs heavy costs and lacks flexibility in practice. To address these issues, we seek help from Stable Diffusion (SD) model due to its capability of zero/few-shot inpainting, which can be leveraged to inpaint anomalous regions as normal. In this paper, a few-shot multi-class anomaly detection framework that adopts Stable Diffusion model is proposed, named AnomalySD. To adapt SD to anomaly detection task, we design different hierarchical text descriptions and the foreground mask mechanism for fine-tuning SD. In the inference stage, to accurately mask anomalous regions for inpainting, we propose multi-scale mask strategy and prototype-guided mask strategy to handle diverse anomalous regions. Hierarchical text prompts are also utilized to guide the process of inpainting in the inference stage. The anomaly score is estimated based on inpainting result of all masks. Extensive experiments on the MVTec-AD and VisA datasets demonstrate the superiority of our approach. We achieved anomaly classification and segmentation results of 93.6%/94.8% AUROC on the MVTec-AD dataset and 86.1%/96.5% AUROC on the VisA dataset under multi-class and one-shot settings.

Title: Unsupervised Representation Learning by Balanced Self Attention Matching

• Authors: Daniel Shalam, Simon Korman
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.02014
• Pdf URL: https://arxiv.org/pdf/2408.02014
• Copy Paste: [[2408.02014]] Unsupervised Representation Learning by Balanced Self Attention Matching(https://arxiv.org/abs/2408.02014)
• Keywords: self-supervised
• Abstract: Many leading self-supervised methods for unsupervised representation learning, in particular those for embedding image features, are built on variants of the instance discrimination task, whose optimization is known to be prone to instabilities that can lead to feature collapse. Different techniques have been devised to circumvent this issue, including the use of negative pairs with different contrastive losses, the use of external memory banks, and breaking of symmetry by using separate encoding networks with possibly different structures. Our method, termed BAM, rather than directly matching features of different views (augmentations) of input images, is based on matching their self-attention vectors, which are the distributions of similarities to the entire set of augmented images of a batch. We obtain rich representations and avoid feature collapse by minimizing a loss that matches these distributions to their globally balanced and entropy regularized version, which is obtained through a simple self-optimal-transport computation. We ablate and verify our method through a wide set of experiments that show competitive performance with leading methods on both semi-supervised and transfer-learning benchmarks. Our implementation and pre-trained models are available at this http URL .

Title: Individualized multi-horizon MRI trajectory prediction for Alzheimer's Disease

• Authors: Rosemary He, Gabriella Ang, Daniel Tward
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02018
• Pdf URL: https://arxiv.org/pdf/2408.02018
• Copy Paste: [[2408.02018]] Individualized multi-horizon MRI trajectory prediction for Alzheimer's Disease(https://arxiv.org/abs/2408.02018)
• Keywords: anomaly
• Abstract: Neurodegeneration as measured through magnetic resonance imaging (MRI) is recognized as a potential biomarker for diagnosing Alzheimer's disease (AD), but is generally considered less specific than amyloid or tau based biomarkers. Due to a large amount of variability in brain anatomy between different individuals, we hypothesize that leveraging MRI time series can help improve specificity, by treating each patient as their own baseline. Here we turn to conditional variational autoencoders to generate individualized MRI predictions given the subject's age, disease status and one previous scan. Using serial imaging data from the Alzheimer's Disease Neuroimaging Initiative, we train a novel architecture to build a latent space distribution which can be sampled from to generate future predictions of changing anatomy. This enables us to extrapolate beyond the dataset and predict MRIs up to 10 years. We evaluated the model on a held-out set from ADNI and an independent dataset (from Open Access Series of Imaging Studies). By comparing to several alternatives, we show that our model produces more individualized images with higher resolution. Further, if an individual already has a follow-up MRI, we demonstrate a usage of our model to compute a likelihood ratio classifier for disease status. In practice, the model may be able to assist in early diagnosis of AD and provide a counterfactual baseline trajectory for treatment effect estimation. Furthermore, it generates a synthetic dataset that can potentially be used for downstream tasks such as anomaly detection and classification.

Title: Faster Diffusion Action Segmentation

• Authors: Shuaibing Wang, Shunli Wang, Mingcheng Li, Dingkang Yang, Haopeng Kuang, Ziyun Qian, Lihua Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02024
• Pdf URL: https://arxiv.org/pdf/2408.02024
• Copy Paste: [[2408.02024]] Faster Diffusion Action Segmentation(https://arxiv.org/abs/2408.02024)
• Keywords: diffusion
• Abstract: Temporal Action Segmentation (TAS) is an essential task in video analysis, aiming to segment and classify continuous frames into distinct action segments. However, the ambiguous boundaries between actions pose a significant challenge for high-precision segmentation. Recent advances in diffusion models have demonstrated substantial success in TAS tasks due to their stable training process and high-quality generation capabilities. However, the heavy sampling steps required by diffusion models pose a substantial computational burden, limiting their practicality in real-time applications. Additionally, most related works utilize Transformer-based encoder architectures. Although these architectures excel at capturing long-range dependencies, they incur high computational costs and face feature-smoothing issues when processing long video sequences. To address these challenges, we propose EffiDiffAct, an efficient and high-performance TAS algorithm. Specifically, we develop a lightweight temporal feature encoder that reduces computational overhead and mitigates the rank collapse phenomenon associated with traditional self-attention mechanisms. Furthermore, we introduce an adaptive skip strategy that allows for dynamic adjustment of timestep lengths based on computed similarity metrics during inference, thereby further enhancing computational efficiency. Comprehensive experiments on the 50Salads, Breakfast, and GTEA datasets demonstrated the effectiveness of the proposed algorithm.

Title: Robustness of Watermarking on Text-to-Image Diffusion Models

• Authors: Xiaodong Wu, Xiangman Li, Jianbing Ni
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2408.02035
• Pdf URL: https://arxiv.org/pdf/2408.02035
• Copy Paste: [[2408.02035]] Robustness of Watermarking on Text-to-Image Diffusion Models(https://arxiv.org/abs/2408.02035)
• Keywords: diffusion, generative
• Abstract: Watermarking has become one of promising techniques to not only aid in identifying AI-generated images but also serve as a deterrent against the unethical use of these models. However, the robustness of watermarking techniques has not been extensively studied recently. In this paper, we investigate the robustness of generative watermarking, which is created from the integration of watermarking embedding and text-to-image generation processing in generative models, e.g., latent diffusion models. Specifically, we propose three attacking methods, i.e., discriminator-based attacks, edge prediction-based attacks, and fine-tune-based attacks, under the scenario where the watermark decoder is not accessible. The model is allowed to be fine-tuned to created AI agents with specific generative tasks for personalizing or specializing. We found that generative watermarking methods are robust to direct evasion attacks, like discriminator-based attacks, or manipulation based on the edge information in edge prediction-based attacks but vulnerable to malicious fine-tuning. Experimental results show that our fine-tune-based attacks can decrease the accuracy of the watermark detection to nearly $67.92\%$. In addition, We conduct an ablation study on the length of fine-tuned messages, encoder/decoder's depth and structure to identify key factors that impact the performance of fine-tune-based attacks.

Title: LEGO: Self-Supervised Representation Learning for Scene Text Images

• Authors: Yujin Ren, Jiaxin Zhang, Lianwen Jin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02036
• Pdf URL: https://arxiv.org/pdf/2408.02036
• Copy Paste: [[2408.02036]] LEGO: Self-Supervised Representation Learning for Scene Text Images(https://arxiv.org/abs/2408.02036)
• Keywords: self-supervised
• Abstract: In recent years, significant progress has been made in scene text recognition by data-driven methods. However, due to the scarcity of annotated real-world data, the training of these methods predominantly relies on synthetic data. The distribution gap between synthetic and real data constrains the further performance improvement of these methods in real-world applications. To tackle this problem, a highly promising approach is to utilize massive amounts of unlabeled real data for self-supervised training, which has been widely proven effective in many NLP and CV tasks. Nevertheless, generic self-supervised methods are unsuitable for scene text images due to their sequential nature. To address this issue, we propose a Local Explicit and Global Order-aware self-supervised representation learning method (LEGO) that accounts for the characteristics of scene text images. Inspired by the human cognitive process of learning words, which involves spelling, reading, and writing, we propose three novel pre-text tasks for LEGO to model sequential, semantic, and structural features, respectively. The entire pre-training process is optimized by using a consistent Text Knowledge Codebook. Extensive experiments validate that LEGO outperforms previous scene text self-supervised methods. The recognizer incorporated with our pre-trained model achieves superior or comparable performance compared to state-of-the-art scene text recognition methods on six benchmarks. Furthermore, we demonstrate that LEGO can achieve superior performance in other text-related tasks.

Title: Deep Spectral Methods for Unsupervised Ultrasound Image Interpretation

• Authors: Oleksandra Tmenova, Yordanka Velikova, Mahdi Saleh, Nassir Navab
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02043
• Pdf URL: https://arxiv.org/pdf/2408.02043
• Copy Paste: [[2408.02043]] Deep Spectral Methods for Unsupervised Ultrasound Image Interpretation(https://arxiv.org/abs/2408.02043)
• Keywords: self-supervised
• Abstract: Ultrasound imaging is challenging to interpret due to non-uniform intensities, low contrast, and inherent artifacts, necessitating extensive training for non-specialists. Advanced representation with clear tissue structure separation could greatly assist clinicians in mapping underlying anatomy and distinguishing between tissue layers. Decomposing an image into semantically meaningful segments is mainly achieved using supervised segmentation algorithms. Unsupervised methods are beneficial, as acquiring large labeled datasets is difficult and costly, but despite their advantages, they still need to be explored in ultrasound. This paper proposes a novel unsupervised deep learning strategy tailored to ultrasound to obtain easily interpretable tissue separations. We integrate key concepts from unsupervised deep spectral methods, which combine spectral graph theory with deep learning methods. We utilize self-supervised transformer features for spectral clustering to generate meaningful segments based on ultrasound-specific metrics and shape and positional priors, ensuring semantic consistency across the dataset. We evaluate our unsupervised deep learning strategy on three ultrasound datasets, showcasing qualitative results across anatomical contexts without label requirements. We also conduct a comparative analysis against other clustering algorithms to demonstrate superior segmentation performance, boundary preservation, and label consistency.

Title: Fine-tuning multilingual language models in Twitter/X sentiment analysis: a study on Eastern-European V4 languages

• Authors: Tomáš Filip, Martin Pavlíček, Petr Sosík
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02044
• Pdf URL: https://arxiv.org/pdf/2408.02044
• Copy Paste: [[2408.02044]] Fine-tuning multilingual language models in Twitter/X sentiment analysis: a study on Eastern-European V4 languages(https://arxiv.org/abs/2408.02044)
• Keywords: in-context
• Abstract: The aspect-based sentiment analysis (ABSA) is a standard NLP task with numerous approaches and benchmarks, where large language models (LLM) represent the current state-of-the-art. We focus on ABSA subtasks based on Twitter/X data in underrepresented languages. On such narrow tasks, small tuned language models can often outperform universal large ones, providing available and cheap solutions. We fine-tune several LLMs (BERT, BERTweet, Llama2, Llama3, Mistral) for classification of sentiment towards Russia and Ukraine in the context of the ongoing military conflict. The training/testing dataset was obtained from the academic API from Twitter/X during 2023, narrowed to the languages of the V4 countries (Czech Republic, Slovakia, Poland, Hungary). Then we measure their performance under a variety of settings including translations, sentiment targets, in-context learning and more, using GPT4 as a reference model. We document several interesting phenomena demonstrating, among others, that some models are much better fine-tunable on multilingual Twitter tasks than others, and that they can reach the SOTA level with a very small training set. Finally we identify combinations of settings providing the best results.

Title: Step Saver: Predicting Minimum Denoising Steps for Diffusion Model Image Generation

• Authors: Jean Yu, Haim Barad
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02054
• Pdf URL: https://arxiv.org/pdf/2408.02054
• Copy Paste: [[2408.02054]] Step Saver: Predicting Minimum Denoising Steps for Diffusion Model Image Generation(https://arxiv.org/abs/2408.02054)
• Keywords: diffusion
• Abstract: In this paper, we introduce an innovative NLP model specifically fine-tuned to determine the minimal number of denoising steps required for any given text prompt. This advanced model serves as a real-time tool that recommends the ideal denoise steps for generating high-quality images efficiently. It is designed to work seamlessly with the Diffusion model, ensuring that images are produced with superior quality in the shortest possible time. Although our explanation focuses on the DDIM scheduler, the methodology is adaptable and can be applied to various other schedulers like Euler, Euler Ancestral, Heun, DPM2 Karras, UniPC, and more. This model allows our customers to conserve costly computing resources by executing the fewest necessary denoising steps to achieve optimal quality in the produced images.

Title: FDiff-Fusion:Denoising diffusion fusion network based on fuzzy learning for 3D medical image segmentation

• Authors: Weiping Ding, Sheng Geng, Haipeng Wang, Jiashuang Huang, Tianyi Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02075
• Pdf URL: https://arxiv.org/pdf/2408.02075
• Copy Paste: [[2408.02075]] FDiff-Fusion:Denoising diffusion fusion network based on fuzzy learning for 3D medical image segmentation(https://arxiv.org/abs/2408.02075)
• Keywords: diffusion
• Abstract: In recent years, the denoising diffusion model has achieved remarkable success in image segmentation modeling. With its powerful nonlinear modeling capabilities and superior generalization performance, denoising diffusion models have gradually been applied to medical image segmentation tasks, bringing new perspectives and methods to this field. However, existing methods overlook the uncertainty of segmentation boundaries and the fuzziness of regions, resulting in the instability and inaccuracy of the segmentation results. To solve this problem, a denoising diffusion fusion network based on fuzzy learning for 3D medical image segmentation (FDiff-Fusion) is proposed in this paper. By integrating the denoising diffusion model into the classical U-Net network, this model can effectively extract rich semantic information from input medical images, thus providing excellent pixel-level representation for medical image segmentation. ... Finally, to validate the effectiveness of FDiff-Fusion, we compare it with existing advanced segmentation networks on the BRATS 2020 brain tumor dataset and the BTCV abdominal multi-organ dataset. The results show that FDiff-Fusion significantly improves the Dice scores and HD95 distance on these two datasets, demonstrating its superiority in medical image segmentation tasks.

Title: LDFaceNet: Latent Diffusion-based Network for High-Fidelity Deepfake Generation

• Authors: Dwij Mehta, Aditya Mehta, Pratik Narang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02078
• Pdf URL: https://arxiv.org/pdf/2408.02078
• Copy Paste: [[2408.02078]] LDFaceNet: Latent Diffusion-based Network for High-Fidelity Deepfake Generation(https://arxiv.org/abs/2408.02078)
• Keywords: diffusion, generative
• Abstract: Over the past decade, there has been tremendous progress in the domain of synthetic media generation. This is mainly due to the powerful methods based on generative adversarial networks (GANs). Very recently, diffusion probabilistic models, which are inspired by non-equilibrium thermodynamics, have taken the spotlight. In the realm of image generation, diffusion models (DMs) have exhibited remarkable proficiency in producing both realistic and heterogeneous imagery through their stochastic sampling procedure. This paper proposes a novel facial swapping module, termed as LDFaceNet (Latent Diffusion based Face Swapping Network), which is based on a guided latent diffusion model that utilizes facial segmentation and facial recognition modules for a conditioned denoising process. The model employs a unique loss function to offer directional guidance to the diffusion process. Notably, LDFaceNet can incorporate supplementary facial guidance for desired outcomes without any retraining. To the best of our knowledge, this represents the first application of the latent diffusion model in the face-swapping task without prior training. The results of this study demonstrate that the proposed method can generate extremely realistic and coherent images by leveraging the potential of the diffusion model for facial swapping, thereby yielding superior visual outcomes and greater diversity.

Title: Past Movements-Guided Motion Representation Learning for Human Motion Prediction

• Authors: Junyu Shi, Baoxuan Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02091
• Pdf URL: https://arxiv.org/pdf/2408.02091
• Copy Paste: [[2408.02091]] Past Movements-Guided Motion Representation Learning for Human Motion Prediction(https://arxiv.org/abs/2408.02091)
• Keywords: self-supervised
• Abstract: Human motion prediction based on 3D skeleton is a significant challenge in computer vision, primarily focusing on the effective representation of motion. In this paper, we propose a self-supervised learning framework designed to enhance motion representation. This framework consists of two stages: first, the network is pretrained through the self-reconstruction of past sequences, and the guided reconstruction of future sequences based on past movements. We design a velocity-based mask strategy to focus on the joints with large-scale moving. Subsequently, the pretrained network undergoes finetuning for specific tasks. Self-reconstruction, guided by patterns of past motion, substantially improves the model's ability to represent the spatiotemporal relationships among joints but also captures the latent relationships between past and future sequences. This capability is crucial for motion prediction tasks that solely depend on historical motion data. By employing this straightforward yet effective training paradigm, our method outperforms existing \textit{state-of-the-art} methods, reducing the average prediction errors by 8.8\% across Human3.6M, 3DPW, and AMASS datasets. The code is available at this https URL.

Title: Effective Demonstration Annotation for In-Context Learning via Language Model-Based Determinantal Point Process

• Authors: Peng Wang, Xiaobin Wang, Chao Lou, Shengyu Mao, Pengjun Xie, Yong Jiang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.02103
• Pdf URL: https://arxiv.org/pdf/2408.02103
• Copy Paste: [[2408.02103]] Effective Demonstration Annotation for In-Context Learning via Language Model-Based Determinantal Point Process(https://arxiv.org/abs/2408.02103)
• Keywords: in-context
• Abstract: In-context learning (ICL) is a few-shot learning paradigm that involves learning mappings through input-output pairs and appropriately applying them to new instances. Despite the remarkable ICL capabilities demonstrated by Large Language Models (LLMs), existing works are highly dependent on large-scale labeled support sets, not always feasible in practical scenarios. To refine this approach, we focus primarily on an innovative selective annotation mechanism, which precedes the standard demonstration retrieval. We introduce the Language Model-based Determinant Point Process (LM-DPP) that simultaneously considers the uncertainty and diversity of unlabeled instances for optimal selection. Consequently, this yields a subset for annotation that strikes a trade-off between the two factors. We apply LM-DPP to various language models, including GPT-J, LlaMA, and GPT-3. Experimental results on 9 NLU and 2 Generation datasets demonstrate that LM-DPP can effectively select canonical examples. Further analysis reveals that LLMs benefit most significantly from subsets that are both low uncertainty and high diversity.

Title: AssemAI: Interpretable Image-Based Anomaly Detection for Manufacturing Pipelines

• Authors: Renjith Prasad, Chathurangi Shyalika, Ramtin Zand, Fadi El Kalach, Revathy Venkataramanan, Ramy Harik, Amit Sheth
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02181
• Pdf URL: https://arxiv.org/pdf/2408.02181
• Copy Paste: [[2408.02181]] AssemAI: Interpretable Image-Based Anomaly Detection for Manufacturing Pipelines(https://arxiv.org/abs/2408.02181)
• Keywords: anomaly
• Abstract: Anomaly detection in manufacturing pipelines remains a critical challenge, intensified by the complexity and variability of industrial environments. This paper introduces AssemAI, an interpretable image-based anomaly detection system tailored for smart manufacturing pipelines. Our primary contributions include the creation of a tailored image dataset and the development of a custom object detection model, YOLO-FF, designed explicitly for anomaly detection in manufacturing assembly environments. Utilizing the preprocessed image dataset derived from an industry-focused rocket assembly pipeline, we address the challenge of imbalanced image data and demonstrate the importance of image-based methods in anomaly detection. The proposed approach leverages domain knowledge in data preparation, model development and reasoning. We compare our method against several baselines, including simple CNN and custom Visual Transformer (ViT) models, showcasing the effectiveness of our custom data preparation and pretrained CNN integration. Additionally, we incorporate explainability techniques at both user and model levels, utilizing ontology for user-friendly explanations and SCORE-CAM for in-depth feature and model analysis. Finally, the model was also deployed in a real-time setting. Our results include ablation studies on the baselines, providing a comprehensive evaluation of the proposed system. This work highlights the broader impact of advanced image-based anomaly detection in enhancing the reliability and efficiency of smart manufacturing processes.

Title: Source-Free Domain-Invariant Performance Prediction

• Authors: Ekaterina Khramtsova, Mahsa Baktashmotlagh, Guido Zuccon, Xi Wang, Mathieu Salzmann
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02209
• Pdf URL: https://arxiv.org/pdf/2408.02209
• Copy Paste: [[2408.02209]] Source-Free Domain-Invariant Performance Prediction(https://arxiv.org/abs/2408.02209)
• Keywords: generative
• Abstract: Accurately estimating model performance poses a significant challenge, particularly in scenarios where the source and target domains follow different data distributions. Most existing performance prediction methods heavily rely on the source data in their estimation process, limiting their applicability in a more realistic setting where only the trained model is accessible. The few methods that do not require source data exhibit considerably inferior performance. In this work, we propose a source-free approach centred on uncertainty-based estimation, using a generative model for calibration in the absence of source data. We establish connections between our approach for unsupervised calibration and temperature scaling. We then employ a gradient-based strategy to evaluate the correctness of the calibrated predictions. Our experiments on benchmark object recognition datasets reveal that existing source-based methods fall short with limited source sample availability. Furthermore, our approach significantly outperforms the current state-of-the-art source-free and source-based methods, affirming its effectiveness in domain-invariant performance estimation.

Title: Climate-Driven Doubling of Maize Loss Probability in U.S. Crop Insurance: Spatiotemporal Prediction and Possible Policy Responses

• Authors: A Samuel Pottinger, Lawson Connor, Brookie Guzder-Williams, Maya Weltman-Fahs, Timothy Bowles
• Subjects: cs.LG, q-fin.RM
• Abstract URL: https://arxiv.org/abs/2408.02217
• Pdf URL: https://arxiv.org/pdf/2408.02217
• Copy Paste: [[2408.02217]] Climate-Driven Doubling of Maize Loss Probability in U.S. Crop Insurance: Spatiotemporal Prediction and Possible Policy Responses(https://arxiv.org/abs/2408.02217)
• Keywords: generative
• Abstract: Climate change not only threatens agricultural producers but also strains financial institutions. These important food system actors include government entities tasked with both insuring grower livelihoods and supporting response to continued global warming. We use an artificial neural network to predict future maize yields in the U.S. Corn Belt, finding alarming changes to institutional risk exposure within the Federal Crop Insurance Program. Specifically, our machine learning method anticipates more frequent and more severe yield losses that would result in the annual probability of Yield Protection (YP) claims to more than double at mid-century relative to simulations without continued climate change. Furthermore, our dual finding of relatively unchanged average yields paired with decreasing yield stability reveals targeted opportunities to adjust coverage formulas to include variability. This important structural shift may help regulators support grower adaptation to continued climate change by recognizing the value of risk-reducing strategies such as regenerative agriculture. Altogether, paired with open source interactive tools for deeper investigation, our risk profile simulations fill an actionable gap in current understanding, bridging granular historic yield estimation and climate-informed prediction of future insurer-relevant loss.

Title: ProCreate, Don\'t Reproduce! Propulsive Energy Diffusion for Creative Generation

• Authors: Jack Lu, Ryan Teehan, Mengye Ren
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02226
• Pdf URL: https://arxiv.org/pdf/2408.02226
• Copy Paste: [[2408.02226]] ProCreate, Don\'t Reproduce! Propulsive Energy Diffusion for Creative Generation(https://arxiv.org/abs/2408.02226)
• Keywords: diffusion, generative
• Abstract: In this paper, we propose ProCreate, a simple and easy-to-implement method to improve sample diversity and creativity of diffusion-based image generative models and to prevent training data reproduction. ProCreate operates on a set of reference images and actively propels the generated image embedding away from the reference embeddings during the generation process. We propose FSCG-8 (Few-Shot Creative Generation 8), a few-shot creative generation dataset on eight different categories -- encompassing different concepts, styles, and settings -- in which ProCreate achieves the highest sample diversity and fidelity. Furthermore, we show that ProCreate is effective at preventing replicating training data in a large-scale evaluation using training text prompts. Code and FSCG-8 are available at this https URL. The project page is available at this https URL.

Title: REVISION: Rendering Tools Enable Spatial Fidelity in Vision-Language Models

• Authors: Agneet Chatterjee, Yiran Luo, Tejas Gokhale, Yezhou Yang, Chitta Baral
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02231
• Pdf URL: https://arxiv.org/pdf/2408.02231
• Copy Paste: [[2408.02231]] REVISION: Rendering Tools Enable Spatial Fidelity in Vision-Language Models(https://arxiv.org/abs/2408.02231)
• Keywords: generative
• Abstract: Text-to-Image (T2I) and multimodal large language models (MLLMs) have been adopted in solutions for several computer vision and multimodal learning tasks. However, it has been found that such vision-language models lack the ability to correctly reason over spatial relationships. To tackle this shortcoming, we develop the REVISION framework which improves spatial fidelity in vision-language models. REVISION is a 3D rendering based pipeline that generates spatially accurate synthetic images, given a textual prompt. REVISION is an extendable framework, which currently supports 100+ 3D assets, 11 spatial relationships, all with diverse camera perspectives and backgrounds. Leveraging images from REVISION as additional guidance in a training-free manner consistently improves the spatial consistency of T2I models across all spatial relationships, achieving competitive performance on the VISOR and T2I-CompBench benchmarks. We also design RevQA, a question-answering benchmark to evaluate the spatial reasoning abilities of MLLMs, and find that state-of-the-art models are not robust to complex spatial reasoning under adversarial settings. Our results and findings indicate that utilizing rendering-based frameworks is an effective approach for developing spatially-aware generative models.

Title: BOTS-LM: Training Large Language Models for Setswana

• Authors: Nathan Brown, Vukosi Marivate
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.02239
• Pdf URL: https://arxiv.org/pdf/2408.02239
• Copy Paste: [[2408.02239]] BOTS-LM: Training Large Language Models for Setswana(https://arxiv.org/abs/2408.02239)
• Keywords: generative
• Abstract: In this work we present BOTS-LM, a series of bilingual language models proficient in both Setswana and English. Leveraging recent advancements in data availability and efficient fine-tuning, BOTS-LM achieves performance similar to models significantly larger than itself while maintaining computational efficiency. Our initial release features an 8 billion parameter generative large language model, with upcoming 0.5 billion and 1 billion parameter large language models and a 278 million parameter encoder-only model soon to be released. We find the 8 billion parameter model significantly outperforms Llama-3-70B and Aya 23 on English-Setswana translation tasks, approaching the performance of dedicated machine translation models, while approaching 70B parameter performance on Setswana reasoning as measured by a machine translated subset of the MMLU benchmark. To accompany the BOTS-LM series of language models, we release the largest Setswana web dataset, SetsText, totalling over 267 million tokens. In addition, we release the largest machine translated Setswana dataset, the first and largest synthetic Setswana dataset, training and evaluation code, training logs, and MMLU-tsn, a machine translated subset of MMLU.

Title: Evaluating Vision-Language Models for Zero-Shot Detection, Classification, and Association of Motorcycles, Passengers, and Helmets

• Authors: Lucas Choi, Ross Greer
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02244
• Pdf URL: https://arxiv.org/pdf/2408.02244
• Copy Paste: [[2408.02244]] Evaluating Vision-Language Models for Zero-Shot Detection, Classification, and Association of Motorcycles, Passengers, and Helmets(https://arxiv.org/abs/2408.02244)
• Keywords: foundation model
• Abstract: Motorcycle accidents pose significant risks, particularly when riders and passengers do not wear helmets. This study evaluates the efficacy of an advanced vision-language foundation model, OWLv2, in detecting and classifying various helmet-wearing statuses of motorcycle occupants using video data. We extend the dataset provided by the CVPR AI City Challenge and employ a cascaded model approach for detection and classification tasks, integrating OWLv2 and CNN models. The results highlight the potential of zero-shot learning to address challenges arising from incomplete and biased training datasets, demonstrating the usage of such models in detecting motorcycles, helmet usage, and occupant positions under varied conditions. We have achieved an average precision of 0.5324 for helmet detection and provided precision-recall curves detailing the detection and classification performance. Despite limitations such as low-resolution data and poor visibility, our research shows promising advancements in automated vehicle safety and traffic safety enforcement systems.

Title: Curriculum learning based pre-training using Multi-Modal Contrastive Masked Autoencoders

• Authors: Muhammad Abdullah Jamal, Omid Mohareri
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02245
• Pdf URL: https://arxiv.org/pdf/2408.02245
• Copy Paste: [[2408.02245]] Curriculum learning based pre-training using Multi-Modal Contrastive Masked Autoencoders(https://arxiv.org/abs/2408.02245)
• Keywords: diffusion
• Abstract: In this paper, we propose a new pre-training method for image understanding tasks under Curriculum Learning (CL) paradigm which leverages RGB-D. The method utilizes Multi-Modal Contrastive Masked Autoencoder and Denoising techniques. Recent approaches either use masked autoencoding (e.g., MultiMAE) or contrastive learning(e.g., Pri3D, or combine them in a single contrastive masked autoencoder architecture such as CMAE and CAV-MAE. However, none of the single contrastive masked autoencoder is applicable to RGB-D datasets. To improve the performance and efficacy of such methods, we propose a new pre-training strategy based on CL. Specifically, in the first stage, we pre-train the model using contrastive learning to learn cross-modal representations. In the second stage, we initialize the modality-specific encoders using the weights from the first stage and then pre-train the model using masked autoencoding and denoising/noise prediction used in diffusion models. Masked autoencoding focuses on reconstructing the missing patches in the input modality using local spatial correlations, while denoising learns high frequency components of the input data. Our approach is scalable, robust and suitable for pre-training with limited RGB-D datasets. Extensive experiments on multiple datasets such as ScanNet, NYUv2 and SUN RGB-D show the efficacy and superior performance of our approach. Specifically, we show an improvement of +1.0% mIoU against Mask3D on ScanNet semantic segmentation. We further demonstrate the effectiveness of our approach in low-data regime by evaluating it for semantic segmentation task against the state-of-the-art methods.

Title: Contrastive Learning and Abstract Concepts: The Case of Natural Numbers

• Authors: Daniel N. Nissani (Nissensohn)
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02247
• Pdf URL: https://arxiv.org/pdf/2408.02247
• Copy Paste: [[2408.02247]] Contrastive Learning and Abstract Concepts: The Case of Natural Numbers(https://arxiv.org/abs/2408.02247)
• Keywords: self-supervised
• Abstract: Contrastive Learning (CL) has been successfully applied to classification and other downstream tasks related to concrete concepts, such as objects contained in the ImageNet dataset. No attempts seem to have been made so far in applying this promising scheme to more abstract entities. A prominent example of these could be the concept of (discrete) Quantity. CL can be frequently interpreted as a self-supervised scheme guided by some profound and ubiquitous conservation principle (e.g. conservation of identity in object classification tasks). In this introductory work we apply a suitable conservation principle to the semi-abstract concept of natural numbers by which discrete quantities can be estimated or predicted. We experimentally show, by means of a toy problem, that contrastive learning can be trained to count at a glance with high accuracy both at human as well as at super-human ranges.. We compare this with the results of a trained-to-count at a glance supervised learning (SL) neural network scheme of similar architecture. We show that both schemes exhibit similar good performance on baseline experiments, where the distributions of the training and testing stages are equal. Importantly, we demonstrate that in some generalization scenarios, where training and testing distributions differ, CL boasts more robust and much better error performance.

Title: SelfGeo: Self-supervised and Geodesic-consistent Estimation of Keypoints on Deformable Shapes

• Authors: Mohammad Zohaib, Luca Cosmo, Alessio Del Bue
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02291
• Pdf URL: https://arxiv.org/pdf/2408.02291
• Copy Paste: [[2408.02291]] SelfGeo: Self-supervised and Geodesic-consistent Estimation of Keypoints on Deformable Shapes(https://arxiv.org/abs/2408.02291)
• Keywords: self-supervised
• Abstract: Unsupervised 3D keypoints estimation from Point Cloud Data (PCD) is a complex task, even more challenging when an object shape is deforming. As keypoints should be semantically and geometrically consistent across all the 3D frames - each keypoint should be anchored to a specific part of the deforming shape irrespective of intrinsic and extrinsic motion. This paper presents, "SelfGeo", a self-supervised method that computes persistent 3D keypoints of non-rigid objects from arbitrary PCDs without the need of human annotations. The gist of SelfGeo is to estimate keypoints between frames that respect invariant properties of deforming bodies. Our main contribution is to enforce that keypoints deform along with the shape while keeping constant geodesic distances among them. This principle is then propagated to the design of a set of losses which minimization let emerge repeatable keypoints in specific semantic locations of the non-rigid shape. We show experimentally that the use of geodesic has a clear advantage in challenging dynamic scenes and with different classes of deforming shapes (humans and animals). Code and data are available at: this https URL

Title: A Lean Transformer Model for Dynamic Malware Analysis and Detection

• Authors: Tony Quertier, Benjamin Marais, Grégoire Barrué, Stéphane Morucci, Sévan Azé, Sébastien Salladin
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.02313
• Pdf URL: https://arxiv.org/pdf/2408.02313
• Copy Paste: [[2408.02313]] A Lean Transformer Model for Dynamic Malware Analysis and Detection(https://arxiv.org/abs/2408.02313)
• Keywords: generative
• Abstract: Malware is a fast-growing threat to the modern computing world and existing lines of defense are not efficient enough to address this issue. This is mainly due to the fact that many prevention solutions rely on signature-based detection methods that can easily be circumvented by hackers. Therefore, there is a recurrent need for behavior-based analysis where a suspicious file is ran in a secured environment and its traces are collected to reports for analysis. Previous works have shown some success leveraging Neural Networks and API calls sequences extracted from these execution reports. Recently, Large Language Models and Generative AI have demonstrated impressive capabilities mainly in Natural Language Processing tasks and promising applications in the cybersecurity field for both attackers and defenders. In this paper, we design an Encoder-Only model, based on the Transformers architecture, to detect malicious files, digesting their API call sequences collected by an execution emulation solution. We are also limiting the size of the model architecture and the number of its parameters since it is often considered that Large Language Models may be overkill for specific tasks such as the one we are dealing with hereafter. In addition to achieving decent detection results, this approach has the advantage of reducing our carbon footprint by limiting training and inference times and facilitating technical operations with less hardware requirements. We also carry out some analysis of our results and highlight the limits and possible improvements when using Transformers to analyze malicious files.

Title: A Sharp Convergence Theory for The Probability Flow ODEs of Diffusion Models

• Authors: Gen Li, Yuting Wei, Yuejie Chi, Yuxin Chen
• Subjects: cs.LG, eess.SP, math.NA, math.ST, stat.ML
• Abstract URL: https://arxiv.org/abs/2408.02320
• Pdf URL: https://arxiv.org/pdf/2408.02320
• Copy Paste: [[2408.02320]] A Sharp Convergence Theory for The Probability Flow ODEs of Diffusion Models(https://arxiv.org/abs/2408.02320)
• Keywords: diffusion, generative
• Abstract: Diffusion models, which convert noise into new data instances by learning to reverse a diffusion process, have become a cornerstone in contemporary generative modeling. In this work, we develop non-asymptotic convergence theory for a popular diffusion-based sampler (i.e., the probability flow ODE sampler) in discrete time, assuming access to $\ell_2$-accurate estimates of the (Stein) score functions. For distributions in $\mathbb{R}^d$, we prove that $d/\varepsilon$ iterations -- modulo some logarithmic and lower-order terms -- are sufficient to approximate the target distribution to within $\varepsilon$ total-variation distance. This is the first result establishing nearly linear dimension-dependency (in $d$) for the probability flow ODE sampler. Imposing only minimal assumptions on the target data distribution (e.g., no smoothness assumption is imposed), our results also characterize how $\ell_2$ score estimation errors affect the quality of the data generation processes. In contrast to prior works, our theory is developed based on an elementary yet versatile non-asymptotic approach without the need of resorting to SDE and ODE toolboxes.

Title: Dialogue Ontology Relation Extraction via Constrained Chain-of-Thought Decoding

• Authors: Renato Vukovic, David Arps, Carel van Niekerk, Benjamin Matthias Ruppik, Hsien-Chin Lin, Michael Heck, Milica Gašić
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.02361
• Pdf URL: https://arxiv.org/pdf/2408.02361
• Copy Paste: [[2408.02361]] Dialogue Ontology Relation Extraction via Constrained Chain-of-Thought Decoding(https://arxiv.org/abs/2408.02361)
• Keywords: generative
• Abstract: State-of-the-art task-oriented dialogue systems typically rely on task-specific ontologies for fulfilling user queries. The majority of task-oriented dialogue data, such as customer service recordings, comes without ontology and annotation. Such ontologies are normally built manually, limiting the application of specialised systems. Dialogue ontology construction is an approach for automating that process and typically consists of two steps: term extraction and relation extraction. In this work, we focus on relation extraction in a transfer learning set-up. To improve the generalisation, we propose an extension to the decoding mechanism of large language models. We adapt Chain-of-Thought (CoT) decoding, recently developed for reasoning problems, to generative relation extraction. Here, we generate multiple branches in the decoding space and select the relations based on a confidence threshold. By constraining the decoding to ontology terms and relations, we aim to decrease the risk of hallucination. We conduct extensive experimentation on two widely used datasets and find improvements in performance on target ontology for source fine-tuned and one-shot prompted large language models.

Title: A Few-Shot Approach for Relation Extraction Domain Adaptation using Large Language Models

• Authors: Vanni Zavarella, Juan Carlos Gamero-Salinas, Sergio Consoli
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02377
• Pdf URL: https://arxiv.org/pdf/2408.02377
• Copy Paste: [[2408.02377]] A Few-Shot Approach for Relation Extraction Domain Adaptation using Large Language Models(https://arxiv.org/abs/2408.02377)
• Keywords: in-context
• Abstract: Knowledge graphs (KGs) have been successfully applied to the analysis of complex scientific and technological domains, with automatic KG generation methods typically building upon relation extraction models capturing fine-grained relations between domain entities in text. While these relations are fully applicable across scientific areas, existing models are trained on few domain-specific datasets such as SciERC and do not perform well on new target domains. In this paper, we experiment with leveraging in-context learning capabilities of Large Language Models to perform schema-constrained data annotation, collecting in-domain training instances for a Transformer-based relation extraction model deployed on titles and abstracts of research papers in the Architecture, Construction, Engineering and Operations (AECO) domain. By assessing the performance gain with respect to a baseline Deep Learning architecture trained on off-domain data, we show that by using a few-shot learning strategy with structured prompts and only minimal expert annotation the presented approach can potentially support domain adaptation of a science KG generation model.

Title: Multi-weather Cross-view Geo-localization Using Denoising Diffusion Models

• Authors: Tongtong Feng, Qing Li, Xin Wang, Mingzi Wang, Guangyao Li, Wenwu Zhu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.02408
• Pdf URL: https://arxiv.org/pdf/2408.02408
• Copy Paste: [[2408.02408]] Multi-weather Cross-view Geo-localization Using Denoising Diffusion Models(https://arxiv.org/abs/2408.02408)
• Keywords: diffusion
• Abstract: Cross-view geo-localization in GNSS-denied environments aims to determine an unknown location by matching drone-view images with the correct geo-tagged satellite-view images from a large gallery. Recent research shows that learning discriminative image representations under specific weather conditions can significantly enhance performance. However, the frequent occurrence of unseen extreme weather conditions hinders progress. This paper introduces MCGF, a Multi-weather Cross-view Geo-localization Framework designed to dynamically adapt to unseen weather conditions. MCGF establishes a joint optimization between image restoration and geo-localization using denoising diffusion models. For image restoration, MCGF incorporates a shared encoder and a lightweight restoration module to help the backbone eliminate weather-specific information. For geo-localization, MCGF uses EVA-02 as a backbone for feature extraction, with cross-entropy loss for training and cosine distance for testing. Extensive experiments on University160k-WX demonstrate that MCGF achieves competitive results for geo-localization in varying weather conditions.

Title: Fairness and Bias Mitigation in Computer Vision: A Survey

• Authors: Sepehr Dehdashtian, Ruozhen He, Yi Li, Guha Balakrishnan, Nuno Vasconcelos, Vicente Ordonez, Vishnu Naresh Boddeti
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02464
• Pdf URL: https://arxiv.org/pdf/2408.02464
• Copy Paste: [[2408.02464]] Fairness and Bias Mitigation in Computer Vision: A Survey(https://arxiv.org/abs/2408.02464)
• Keywords: generative
• Abstract: Computer vision systems have witnessed rapid progress over the past two decades due to multiple advances in the field. As these systems are increasingly being deployed in high-stakes real-world applications, there is a dire need to ensure that they do not propagate or amplify any discriminatory tendencies in historical or human-curated data or inadvertently learn biases from spurious correlations. This paper presents a comprehensive survey on fairness that summarizes and sheds light on ongoing trends and successes in the context of computer vision. The topics we discuss include 1) The origin and technical definitions of fairness drawn from the wider fair machine learning literature and adjacent disciplines. 2) Work that sought to discover and analyze biases in computer vision systems. 3) A summary of methods proposed to mitigate bias in computer vision systems in recent years. 4) A comprehensive summary of resources and datasets produced by researchers to measure, analyze, and mitigate bias and enhance fairness. 5) Discussion of the field's success, continuing trends in the context of multimodal foundation and generative models, and gaps that still need to be addressed. The presented characterization should help researchers understand the importance of identifying and mitigating bias in computer vision and the state of the field and identify potential directions for future research.

Title: Exploring Conditional Multi-Modal Prompts for Zero-shot HOI Detection

• Authors: Ting Lei, Shaofeng Yin, Yuxin Peng, Yang Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02484
• Pdf URL: https://arxiv.org/pdf/2408.02484
• Copy Paste: [[2408.02484]] Exploring Conditional Multi-Modal Prompts for Zero-shot HOI Detection(https://arxiv.org/abs/2408.02484)
• Keywords: foundation model
• Abstract: Zero-shot Human-Object Interaction (HOI) detection has emerged as a frontier topic due to its capability to detect HOIs beyond a predefined set of categories. This task entails not only identifying the interactiveness of human-object pairs and localizing them but also recognizing both seen and unseen interaction categories. In this paper, we introduce a novel framework for zero-shot HOI detection using Conditional Multi-Modal Prompts, namely CMMP. This approach enhances the generalization of large foundation models, such as CLIP, when fine-tuned for HOI detection. Unlike traditional prompt-learning methods, we propose learning decoupled vision and language prompts for interactiveness-aware visual feature extraction and generalizable interaction classification, respectively. Specifically, we integrate prior knowledge of different granularity into conditional vision prompts, including an input-conditioned instance prior and a global spatial pattern prior. The former encourages the image encoder to treat instances belonging to seen or potentially unseen HOI concepts equally while the latter provides representative plausible spatial configuration of the human and object under interaction. Besides, we employ language-aware prompt learning with a consistency constraint to preserve the knowledge of the large foundation model to enable better generalization in the text branch. Extensive experiments demonstrate the efficacy of our detector with conditional multi-modal prompts, outperforming previous state-of-the-art on unseen classes of various zero-shot settings. The code and models are available at \url{this https URL}.

Title: OneLove beyond the field -- A few-shot pipeline for topic and sentiment analysis during the FIFA World Cup in Qatar

• Authors: Christoph Rauchegger, Sonja Mei Wang, Pieter Delobelle
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.02520
• Pdf URL: https://arxiv.org/pdf/2408.02520
• Copy Paste: [[2408.02520]] OneLove beyond the field -- A few-shot pipeline for topic and sentiment analysis during the FIFA World Cup in Qatar(https://arxiv.org/abs/2408.02520)
• Keywords: in-context
• Abstract: The FIFA World Cup in Qatar was discussed extensively in the news and on social media. Due to news reports with allegations of human rights violations, there were calls to boycott it. Wearing a OneLove armband was part of a planned protest activity. Controversy around the armband arose when FIFA threatened to sanction captains who wear it. To understand what topics Twitter users Tweeted about and what the opinion of German Twitter users was towards the OneLove armband, we performed an analysis of German Tweets published during the World Cup using in-context learning with LLMs. We validated the labels on human annotations. We found that Twitter users initially discussed the armband's impact, LGBT rights, and politics; after the ban, the conversation shifted towards politics in sports in general, accompanied by a subtle shift in sentiment towards neutrality. Our evaluation serves as a framework for future research to explore the impact of sports activism and evolving public sentiment. This is especially useful in settings where labeling datasets for specific opinions is unfeasible, such as when events are unfolding.

Title: RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation

• Authors: Daniel Fleischer, Moshe Berchansky, Moshe Wasserblat, Peter Izsak
• Subjects: cs.CL, cs.AI, cs.IR, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.02545
• Pdf URL: https://arxiv.org/pdf/2408.02545
• Copy Paste: [[2408.02545]] RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation(https://arxiv.org/abs/2408.02545)
• Keywords: generative
• Abstract: Implementing Retrieval-Augmented Generation (RAG) systems is inherently complex, requiring deep understanding of data, use cases, and intricate design decisions. Additionally, evaluating these systems presents significant challenges, necessitating assessment of both retrieval accuracy and generative quality through a multi-faceted approach. We introduce RAG Foundry, an open-source framework for augmenting large language models for RAG use cases. RAG Foundry integrates data creation, training, inference and evaluation into a single workflow, facilitating the creation of data-augmented datasets for training and evaluating large language models in RAG settings. This integration enables rapid prototyping and experimentation with various RAG techniques, allowing users to easily generate datasets and train RAG models using internal or specialized knowledge sources. We demonstrate the framework effectiveness by augmenting and fine-tuning Llama-3 and Phi-3 models with diverse RAG configurations, showcasing consistent improvements across three knowledge-intensive datasets. Code is released as open-source in this https URL.

Title: Operational range bounding of spectroscopy models with anomaly detection

• Authors: Luís F. Simões, Pierluigi Casale, Marília Felismino, Kai Hou Yip, Ingo P. Waldmann, Giovanna Tinetti, Theresa Lueftinger
• Subjects: cs.LG, astro-ph.IM
• Abstract URL: https://arxiv.org/abs/2408.02581
• Pdf URL: https://arxiv.org/pdf/2408.02581
• Copy Paste: [[2408.02581]] Operational range bounding of spectroscopy models with anomaly detection(https://arxiv.org/abs/2408.02581)
• Keywords: anomaly
• Abstract: Safe operation of machine learning models requires architectures that explicitly delimit their operational ranges. We evaluate the ability of anomaly detection algorithms to provide indicators correlated with degraded model performance. By placing acceptance thresholds over such indicators, hard boundaries are formed that define the model's coverage. As a use case, we consider the extraction of exoplanetary spectra from transit light curves, specifically within the context of ESA's upcoming Ariel mission. Isolation Forests are shown to effectively identify contexts where prediction models are likely to fail. Coverage/error trade-offs are evaluated under conditions of data and concept drift. The best performance is seen when Isolation Forests model projections of the prediction model's explainability SHAP values.

Title: LaMamba-Diff: Linear-Time High-Fidelity Diffusion Models Based on Local Attention and Mamba

• Authors: Yunxiang Fu, Chaoqi Chen, Yizhou Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02615
• Pdf URL: https://arxiv.org/pdf/2408.02615
• Copy Paste: [[2408.02615]] LaMamba-Diff: Linear-Time High-Fidelity Diffusion Models Based on Local Attention and Mamba(https://arxiv.org/abs/2408.02615)
• Keywords: diffusion, generative
• Abstract: Recent Transformer-based diffusion models have shown remarkable performance, largely attributed to the ability of the self-attention mechanism to accurately capture both global and local contexts by computing all-pair interactions among input tokens. However, their quadratic complexity poses significant computational challenges for long-sequence inputs. Conversely, a recent state space model called Mamba offers linear complexity by compressing a filtered global context into a hidden state. Despite its efficiency, compression inevitably leads to information loss of fine-grained local dependencies among tokens, which are crucial for effective visual generative modeling. Motivated by these observations, we introduce Local Attentional Mamba (LaMamba) blocks that combine the strengths of self-attention and Mamba, capturing both global contexts and local details with linear complexity. Leveraging the efficient U-Net architecture, our model exhibits exceptional scalability and surpasses the performance of DiT across various model scales on ImageNet at 256x256 resolution, all while utilizing substantially fewer GFLOPs and a comparable number of parameters. Compared to state-of-the-art diffusion models on ImageNet 256x256 and 512x512, our largest model presents notable advantages, such as a reduction of up to 62\% GFLOPs compared to DiT-XL/2, while achieving superior performance with comparable or fewer parameters.

Title: Language Model Can Listen While Speaking

• Authors: Ziyang Ma, Yakun Song, Chenpeng Du, Jian Cong, Zhuo Chen, Yuping Wang, Yuxuan Wang, Xie Chen
• Subjects: cs.CL, cs.AI, cs.HC, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.02622
• Pdf URL: https://arxiv.org/pdf/2408.02622
• Copy Paste: [[2408.02622]] Language Model Can Listen While Speaking(https://arxiv.org/abs/2408.02622)
• Keywords: self-supervised
• Abstract: Dialogue serves as the most natural manner of human-computer interaction (HCI). Recent advancements in speech language models (SLM) have significantly enhanced speech-based conversational AI. However, these models are limited to turn-based conversation, lacking the ability to interact with humans in real-time spoken scenarios, for example, being interrupted when the generated content is not satisfactory. To address these limitations, we explore full duplex modeling (FDM) in interactive speech language models (iSLM), focusing on enhancing real-time interaction and, more explicitly, exploring the quintessential ability of interruption. We introduce a novel model design, namely listening-while-speaking language model (LSLM), an end-to-end system equipped with both listening and speaking channels. Our LSLM employs a token-based decoder-only TTS for speech generation and a streaming self-supervised learning (SSL) encoder for real-time audio input. LSLM fuses both channels for autoregressive generation and detects turn-taking in real time. Three fusion strategies -- early fusion, middle fusion, and late fusion -- are explored, with middle fusion achieving an optimal balance between speech generation and real-time interaction. Two experimental settings, command-based FDM and voice-based FDM, demonstrate LSLM's robustness to noise and sensitivity to diverse instructions. Our results highlight LSLM's capability to achieve duplex communication with minimal impact on existing systems. This study aims to advance the development of interactive speech dialogue systems, enhancing their applicability in real-world contexts.

Title: Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining

• Authors: Dongyang Liu, Shitian Zhao, Le Zhuo, Weifeng Lin, Yu Qiao, Hongsheng Li, Peng Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.02657
• Pdf URL: https://arxiv.org/pdf/2408.02657
• Copy Paste: [[2408.02657]] Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining(https://arxiv.org/abs/2408.02657)
• Keywords: diffusion, foundation model, generative
• Abstract: We present Lumina-mGPT, a family of multimodal autoregressive models capable of various vision and language tasks, particularly excelling in generating flexible photorealistic images from text descriptions. Unlike existing autoregressive image generation approaches, Lumina-mGPT employs a pretrained decoder-only transformer as a unified framework for modeling multimodal token sequences. Our key insight is that a simple decoder-only transformer with multimodal Generative PreTraining (mGPT), utilizing the next-token prediction objective on massive interleaved text-image sequences, can learn broad and general multimodal capabilities, thereby illuminating photorealistic text-to-image generation. Building on these pretrained models, we propose Flexible Progressive Supervised Finetuning (FP-SFT) on high-quality image-text pairs to fully unlock their potential for high-aesthetic image synthesis at any resolution while maintaining their general multimodal capabilities. Furthermore, we introduce Ominiponent Supervised Finetuning (Omni-SFT), transforming Lumina-mGPT into a foundation model that seamlessly achieves omnipotent task unification. The resulting model demonstrates versatile multimodal capabilities, including visual generation tasks like flexible text-to-image generation and controllable generation, visual recognition tasks like segmentation and depth estimation, and vision-language tasks like multiturn visual question answering. Additionally, we analyze the differences and similarities between diffusion-based and autoregressive methods in a direct comparison.