2024-08-29

Title: Multi-Slice Spatial Transcriptomics Data Integration Analysis with STG3Net

• Authors: Donghai Fang, Fangfang Zhu, Wenwen Min
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.15246
• Pdf URL: https://arxiv.org/pdf/2408.15246
• Copy Paste: [[2408.15246]] Multi-Slice Spatial Transcriptomics Data Integration Analysis with STG3Net(https://arxiv.org/abs/2408.15246)
• Keywords: generative
• Abstract: With the rapid development of the latest Spatially Resolved Transcriptomics (SRT) technology, which allows for the mapping of gene expression within tissue sections, the integrative analysis of multiple SRT data has become increasingly important. However, batch effects between multiple slices pose significant challenges in analyzing SRT data. To address these challenges, we have developed a plug-and-play batch correction method called Global Nearest Neighbor (G2N) anchor pairs selection. G2N effectively mitigates batch effects by selecting representative anchor pairs across slices. Building upon G2N, we propose STG3Net, which cleverly combines masked graph convolutional autoencoders as backbone modules. These autoencoders, integrated with generative adversarial learning, enable STG3Net to achieve robust multi-slice spatial domain identification and batch correction. We comprehensively evaluate the feasibility of STG3Net on three multiple SRT datasets from different platforms, considering accuracy, consistency, and the F1LISI metric (a measure of batch effect correction efficiency). Compared to existing methods, STG3Net achieves the best overall performance while preserving the biological variability and connectivity between slices. Source code and all public datasets used in this paper are available at this https URL and this https URL.

Title: TrajFM: A Vehicle Trajectory Foundation Model for Region and Task Transferability

• Authors: Yan Lin, Tonglong Wei, Zeyu Zhou, Haomin Wen, Jilin Hu, Shengnan Guo, Youfang Lin, Huaiyu Wan
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.15251
• Pdf URL: https://arxiv.org/pdf/2408.15251
• Copy Paste: [[2408.15251]] TrajFM: A Vehicle Trajectory Foundation Model for Region and Task Transferability(https://arxiv.org/abs/2408.15251)
• Keywords: foundation model
• Abstract: Vehicle trajectories provide valuable movement information that supports various downstream tasks and powers real-world applications. A desirable trajectory learning model should transfer between different regions and tasks without retraining, thus improving computational efficiency and effectiveness with limited training data. However, a model's ability to transfer across regions is limited by the unique spatial features and POI arrangements of each region, which are closely linked to vehicle movement patterns and difficult to generalize. Additionally, achieving task transferability is challenging due to the differing generation schemes required for various tasks. Existing efforts towards transferability primarily involve learning embedding vectors for trajectories, which perform poorly in region transfer and still require retraining of prediction modules for task transfer. To address these challenges, we propose TrajFM, a vehicle trajectory foundation model that excels in both region and task transferability. For region transferability, we introduce STRFormer as the main learnable model within TrajFM. It integrates spatial, temporal, and POI modalities of trajectories to effectively manage variations in POI arrangements across regions and includes a learnable spatio-temporal Rotary position embedding module for handling spatial features. For task transferability, we propose a trajectory masking and recovery scheme. This scheme unifies the generation processes of various tasks into the masking and recovery of modalities and sub-trajectories, allowing TrajFM to be pre-trained once and transferred to different tasks without retraining. Experiments on two real-world vehicle trajectory datasets under various settings demonstrate the effectiveness of TrajFM. Code is available at https://anonymous.4open.science/r/TrajFM-30E4.

Title: Multitask Fine-Tuning and Generative Adversarial Learning for Improved Auxiliary Classification

• Authors: Christopher Sun, Abishek Satish
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.15265
• Pdf URL: https://arxiv.org/pdf/2408.15265
• Copy Paste: [[2408.15265]] Multitask Fine-Tuning and Generative Adversarial Learning for Improved Auxiliary Classification(https://arxiv.org/abs/2408.15265)
• Keywords: generative
• Abstract: In this study, we implement a novel BERT architecture for multitask fine-tuning on three downstream tasks: sentiment classification, paraphrase detection, and semantic textual similarity prediction. Our model, Multitask BERT, incorporates layer sharing and a triplet architecture, custom sentence pair tokenization, loss pairing, and gradient surgery. Such optimizations yield a 0.516 sentiment classification accuracy, 0.886 paraphase detection accuracy, and 0.864 semantic textual similarity correlation on test data. We also apply generative adversarial learning to BERT, constructing a conditional generator model that maps from latent space to create fake embeddings in $\mathbb{R}^{768}$. These fake embeddings are concatenated with real BERT embeddings and passed into a discriminator model for auxiliary classification. Using this framework, which we refer to as AC-GAN-BERT, we conduct semi-supervised sensitivity analyses to investigate the effect of increasing amounts of unlabeled training data on AC-GAN-BERT's test accuracy. Overall, aside from implementing a high-performing multitask classification system, our novelty lies in the application of adversarial learning to construct a generator that mimics BERT. We find that the conditional generator successfully produces rich embeddings with clear spatial correlation with class labels, demonstrating avoidance of mode collapse. Our findings validate the GAN-BERT approach and point to future directions of generator-aided knowledge distillation.

Title: 3D Photon Counting CT Image Super-Resolution Using Conditional Diffusion Model

• Authors: Chuang Niu, Christopher Wiedeman, Mengzhou Li, Jonathan S Maltz, Ge Wang
• Subjects: cs.CV, eess.SP
• Abstract URL: https://arxiv.org/abs/2408.15283
• Pdf URL: https://arxiv.org/pdf/2408.15283
• Copy Paste: [[2408.15283]] 3D Photon Counting CT Image Super-Resolution Using Conditional Diffusion Model(https://arxiv.org/abs/2408.15283)
• Keywords: diffusion
• Abstract: This study aims to improve photon counting CT (PCCT) image resolution using denoising diffusion probabilistic models (DDPM). Although DDPMs have shown superior performance when applied to various computer vision tasks, their effectiveness has yet to be translated to high dimensional CT super-resolution. To train DDPMs in a conditional sampling manner, we first leverage CatSim to simulate realistic lower resolution PCCT images from high-resolution CT scans. Since maximizing DDPM performance is time-consuming for both inference and training, especially on high-dimensional PCCT data, we explore both 2D and 3D networks for conditional DDPM and apply methods to accelerate training. In particular, we decompose the 3D task into efficient 2D DDPMs and design a joint 2D inference in the reverse diffusion process that synergizes 2D results of all three dimensions to make the final 3D prediction. Experimental results show that our DDPM achieves improved results versus baseline reference models in recovering high-frequency structures, suggesting that a framework based on realistic simulation and DDPM shows promise for improving PCCT resolution.

Title: Multi-Feature Aggregation in Diffusion Models for Enhanced Face Super-Resolution

• Authors: Marcelo dos Santos, Rayson Laroca, Rafael O. Ribeiro, João C. Neves, David Menotti
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15386
• Pdf URL: https://arxiv.org/pdf/2408.15386
• Copy Paste: [[2408.15386]] Multi-Feature Aggregation in Diffusion Models for Enhanced Face Super-Resolution(https://arxiv.org/abs/2408.15386)
• Keywords: diffusion
• Abstract: Super-resolution algorithms often struggle with images from surveillance environments due to adverse conditions such as unknown degradation, variations in pose, irregular illumination, and occlusions. However, acquiring multiple images, even of low quality, is possible with surveillance cameras. In this work, we develop an algorithm based on diffusion models that utilize a low-resolution image combined with features extracted from multiple low-quality images to generate a super-resolved image while minimizing distortions in the individual's identity. Unlike other algorithms, our approach recovers facial features without explicitly providing attribute information or without the need to calculate a gradient of a function during the reconstruction process. To the best of our knowledge, this is the first time multi-features combined with low-resolution images are used as conditioners to generate more reliable super-resolution images using stochastic differential equations. The FFHQ dataset was employed for training, resulting in state-of-the-art performance in facial recognition and verification metrics when evaluated on the CelebA and Quis-Campi datasets. Our code is publicly available at this https URL

Title: Avoiding Generative Model Writer's Block With Embedding Nudging

• Authors: Ali Zand, Milad Nasr
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15450
• Pdf URL: https://arxiv.org/pdf/2408.15450
• Copy Paste: [[2408.15450]] Avoiding Generative Model Writer's Block With Embedding Nudging(https://arxiv.org/abs/2408.15450)
• Keywords: diffusion, generative
• Abstract: Generative image models, since introduction, have become a global phenomenon. From new arts becoming possible to new vectors of abuse, many new capabilities have become available. One of the challenging issues with generative models is controlling the generation process specially to prevent specific generations classes or instances . There are several reasons why one may want to control the output of generative models, ranging from privacy and safety concerns to application limitations or user preferences To address memorization and privacy challenges, there has been considerable research dedicated to filtering prompts or filtering the outputs of these models. What all these solutions have in common is that at the end of the day they stop the model from producing anything, hence limiting the usability of the model. In this paper, we propose a method for addressing this usability issue by making it possible to steer away from unwanted concepts (when detected in model's output) and still generating outputs. In particular we focus on the latent diffusion image generative models and how one can prevent them to generate particular images while generating similar images with limited overhead. We focus on mitigating issues like image memorization, demonstrating our technique's effectiveness through qualitative and quantitative evaluations. Our method successfully prevents the generation of memorized training images while maintaining comparable image quality and relevance to the unmodified model.

Title: Hand1000: Generating Realistic Hands from Text with Only 1,000 Images

• Authors: Haozhuo Zhang, Bin Zhu, Yu Cao, Yanbin Hao
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2408.15461
• Pdf URL: https://arxiv.org/pdf/2408.15461
• Copy Paste: [[2408.15461]] Hand1000: Generating Realistic Hands from Text with Only 1,000 Images(https://arxiv.org/abs/2408.15461)
• Keywords: diffusion
• Abstract: Text-to-image generation models have achieved remarkable advancements in recent years, aiming to produce realistic images from textual descriptions. However, these models often struggle with generating anatomically accurate representations of human hands. The resulting images frequently exhibit issues such as incorrect numbers of fingers, unnatural twisting or interlacing of fingers, or blurred and indistinct hands. These issues stem from the inherent complexity of hand structures and the difficulty in aligning textual descriptions with precise visual depictions of hands. To address these challenges, we propose a novel approach named Hand1000 that enables the generation of realistic hand images with target gesture using only 1,000 training samples. The training of Hand1000 is divided into three stages with the first stage aiming to enhance the model's understanding of hand anatomy by using a pre-trained hand gesture recognition model to extract gesture representation. The second stage further optimizes text embedding by incorporating the extracted hand gesture representation, to improve alignment between the textual descriptions and the generated hand images. The third stage utilizes the optimized embedding to fine-tune the Stable Diffusion model to generate realistic hand images. In addition, we construct the first publicly available dataset specifically designed for text-to-hand image generation. Based on the existing hand gesture recognition dataset, we adopt advanced image captioning models and LLaMA3 to generate high-quality textual descriptions enriched with detailed gesture information. Extensive experiments demonstrate that Hand1000 significantly outperforms existing models in producing anatomically correct hand images while faithfully representing other details in the text, such as faces, clothing, and colors.

Title: MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning

• Authors: Yifu Yuan, Zhenrui Zheng, Zibin Dong, Jianye Hao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.15501
• Pdf URL: https://arxiv.org/pdf/2408.15501
• Copy Paste: [[2408.15501]] MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learning(https://arxiv.org/abs/2408.15501)
• Keywords: diffusion
• Abstract: Multi-objective Reinforcement Learning (MORL) seeks to develop policies that simultaneously optimize multiple conflicting objectives, but it requires extensive online interactions. Offline MORL provides a promising solution by training on pre-collected datasets to generalize to any preference upon deployment. However, real-world offline datasets are often conservatively and narrowly distributed, failing to comprehensively cover preferences, leading to the emergence of out-of-distribution (OOD) preference areas. Existing offline MORL algorithms exhibit poor generalization to OOD preferences, resulting in policies that do not align with preferences. Leveraging the excellent expressive and generalization capabilities of diffusion models, we propose MODULI (Multi-objective Diffusion Planner with Sliding Guidance), which employs a preference-conditioned diffusion model as a planner to generate trajectories that align with various preferences and derive action for decision-making. To achieve accurate generation, MODULI introduces two return normalization methods under diverse preferences for refining guidance. To further enhance generalization to OOD preferences, MODULI proposes a novel sliding guidance mechanism, which involves training an additional slider adapter to capture the direction of preference changes. Incorporating the slider, it transitions from in-distribution (ID) preferences to generating OOD preferences, patching, and extending the incomplete Pareto front. Extensive experiments on the D4MORL benchmark demonstrate that our algorithm outperforms state-of-the-art Offline MORL baselines, exhibiting excellent generalization to OOD preferences.

Title: Measuring the Reliability of Causal Probing Methods: Tradeoffs, Limitations, and the Plight of Nullifying Interventions

• Authors: Marc Canby, Adam Davies, Chirag Rastogi, Julia Hockenmaier
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2408.15510
• Pdf URL: https://arxiv.org/pdf/2408.15510
• Copy Paste: [[2408.15510]] Measuring the Reliability of Causal Probing Methods: Tradeoffs, Limitations, and the Plight of Nullifying Interventions(https://arxiv.org/abs/2408.15510)
• Keywords: foundation model
• Abstract: Causal probing is an approach to interpreting foundation models, such as large language models, by training probes to recognize latent properties of interest from embeddings, intervening on probes to modify this representation, and analyzing the resulting changes in the model's behavior. While some recent works have cast doubt on the theoretical basis of several leading causal probing intervention methods, it has been unclear how to systematically and empirically evaluate their effectiveness in practice. To address this problem, we propose a general empirical analysis framework to evaluate the reliability of causal probing interventions, formally defining and quantifying two key causal probing desiderata: completeness (fully transforming the representation of the target property) and selectivity (minimally impacting other properties). Our formalism allows us to make the first direct comparisons between different families of causal probing methods (e.g., linear vs. nonlinear or counterfactual vs. nullifying interventions). We conduct extensive experiments across several leading methods, finding that (1) there is an inherent tradeoff between these criteria, and no method is able to consistently satisfy both at once; and (2) across the board, nullifying interventions are always far less complete than counterfactual interventions, indicating that nullifying methods may not be an effective approach to causal probing.

Title: Ray-Distance Volume Rendering for Neural Scene Reconstruction

• Authors: Ruihong Yin, Yunlu Chen, Sezer Karaoglu, Theo Gevers
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15524
• Pdf URL: https://arxiv.org/pdf/2408.15524
• Copy Paste: [[2408.15524]] Ray-Distance Volume Rendering for Neural Scene Reconstruction(https://arxiv.org/abs/2408.15524)
• Keywords: self-supervised
• Abstract: Existing methods in neural scene reconstruction utilize the Signed Distance Function (SDF) to model the density function. However, in indoor scenes, the density computed from the SDF for a sampled point may not consistently reflect its real importance in volume rendering, often due to the influence of neighboring objects. To tackle this issue, our work proposes a novel approach for indoor scene reconstruction, which instead parameterizes the density function with the Signed Ray Distance Function (SRDF). Firstly, the SRDF is predicted by the network and transformed to a ray-conditioned density function for volume rendering. We argue that the ray-specific SRDF only considers the surface along the camera ray, from which the derived density function is more consistent to the real occupancy than that from the SDF. Secondly, although SRDF and SDF represent different aspects of scene geometries, their values should share the same sign indicating the underlying spatial occupancy. Therefore, this work introduces a SRDF-SDF consistency loss to constrain the signs of the SRDF and SDF outputs. Thirdly, this work proposes a self-supervised visibility task, introducing the physical visibility geometry to the reconstruction task. The visibility task combines prior from predicted SRDF and SDF as pseudo labels, and contributes to generating more accurate 3D geometry. Our method implemented with different representations has been validated on indoor datasets, achieving improved performance in both reconstruction and view synthesis.

Title: ConsistencyTrack: A Robust Multi-Object Tracker with a Generation Strategy of Consistency Model

• Authors: Lifan Jiang, Zhihui Wang, Siqi Yin, Guangxiao Ma, Peng Zhang, Boxi Wu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15548
• Pdf URL: https://arxiv.org/pdf/2408.15548
• Copy Paste: [[2408.15548]] ConsistencyTrack: A Robust Multi-Object Tracker with a Generation Strategy of Consistency Model(https://arxiv.org/abs/2408.15548)
• Keywords: diffusion
• Abstract: Multi-object tracking (MOT) is a critical technology in computer vision, designed to detect multiple targets in video sequences and assign each target a unique ID per frame. Existed MOT methods excel at accurately tracking multiple objects in real-time across various scenarios. However, these methods still face challenges such as poor noise resistance and frequent ID switches. In this research, we propose a novel ConsistencyTrack, joint detection and tracking(JDT) framework that formulates detection and association as a denoising diffusion process on perturbed bounding boxes. This progressive denoising strategy significantly improves the model's noise resistance. During the training phase, paired object boxes within two adjacent frames are diffused from ground-truth boxes to a random distribution, and then the model learns to detect and track by reversing this process. In inference, the model refines randomly generated boxes into detection and tracking results through minimal denoising steps. ConsistencyTrack also introduces an innovative target association strategy to address target occlusion. Experiments on the MOT17 and DanceTrack datasets demonstrate that ConsistencyTrack outperforms other compared methods, especially better than DiffusionTrack in inference speed and other performance metrics. Our code is available at this https URL.

Title: VFLIP: A Backdoor Defense for Vertical Federated Learning via Identification and Purification

• Authors: Yungi Cho, Woorim Han, Miseon Yu, Ho Bae, Yunheung Paek
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.15591
• Pdf URL: https://arxiv.org/pdf/2408.15591
• Copy Paste: [[2408.15591]] VFLIP: A Backdoor Defense for Vertical Federated Learning via Identification and Purification(https://arxiv.org/abs/2408.15591)
• Keywords: anomaly
• Abstract: Vertical Federated Learning (VFL) focuses on handling vertically partitioned data over FL participants. Recent studies have discovered a significant vulnerability in VFL to backdoor attacks which specifically target the distinct characteristics of VFL. Therefore, these attacks may neutralize existing defense mechanisms designed primarily for Horizontal Federated Learning (HFL) and deep neural networks. In this paper, we present the first backdoor defense, called VFLIP, specialized for VFL. VFLIP employs the identification and purification techniques that operate at the inference stage, consequently improving the robustness against backdoor attacks to a great extent. VFLIP first identifies backdoor-triggered embeddings by adopting a participant-wise anomaly detection approach. Subsequently, VFLIP conducts purification which removes the embeddings identified as malicious and reconstructs all the embeddings based on the remaining embeddings. We conduct extensive experiments on CIFAR10, CINIC10, Imagenette, NUS-WIDE, and BankMarketing to demonstrate that VFLIP can effectively mitigate backdoor attacks in VFL. this https URL

Title: Exploring Selective Layer Fine-Tuning in Federated Learning

• Authors: Yuchang Sun, Yuexiang Xie, Bolin Ding, Yaliang Li, Jun Zhang
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2408.15600
• Pdf URL: https://arxiv.org/pdf/2408.15600
• Copy Paste: [[2408.15600]] Exploring Selective Layer Fine-Tuning in Federated Learning(https://arxiv.org/abs/2408.15600)
• Keywords: foundation model
• Abstract: Federated learning (FL) has emerged as a promising paradigm for fine-tuning foundation models using distributed data in a privacy-preserving manner. Under limited computational resources, clients often find it more practical to fine-tune a selected subset of layers, rather than the entire model, based on their task-specific data. In this study, we provide a thorough theoretical exploration of selective layer fine-tuning in FL, emphasizing a flexible approach that allows the clients to adjust their selected layers according to their local data and resources. We theoretically demonstrate that the layer selection strategy has a significant impact on model convergence in two critical aspects: the importance of selected layers and the heterogeneous choices across clients. Drawing from these insights, we further propose a strategic layer selection method that utilizes local gradients and regulates layer selections across clients. The extensive experiments on both image and text datasets demonstrate the effectiveness of the proposed strategy compared with several baselines, highlighting its advances in identifying critical layers that adapt to the client heterogeneity and training dynamics in FL.

Title: Can Visual Language Models Replace OCR-Based Visual Question Answering Pipelines in Production? A Case Study in Retail

• Authors: Bianca Lamm, Janis Keuper
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15626
• Pdf URL: https://arxiv.org/pdf/2408.15626
• Copy Paste: [[2408.15626]] Can Visual Language Models Replace OCR-Based Visual Question Answering Pipelines in Production? A Case Study in Retail(https://arxiv.org/abs/2408.15626)
• Keywords: foundation model
• Abstract: Most production-level deployments for Visual Question Answering (VQA) tasks are still build as processing pipelines of independent steps including image pre-processing, object- and text detection, Optical Character Recognition (OCR) and (mostly supervised) object classification. However, the recent advances in vision Foundation Models [25] and Vision Language Models (VLMs) [23] raise the question if these custom trained, multi-step approaches can be replaced with pre-trained, single-step VLMs. This paper analyzes the performance and limits of various VLMs in the context of VQA and OCR [5, 9, 12] tasks in a production-level scenario. Using data from the Retail-786k [10] dataset, we investigate the capabilities of pre-trained VLMs to answer detailed questions about advertised products in images. Our study includes two commercial models, GPT-4V [16] and GPT-4o [17], as well as four open-source models: InternVL [5], LLaVA 1.5 [12], LLaVA-NeXT [13], and CogAgent [9]. Our initial results show, that there is in general no big performance gap between open-source and commercial models. However, we observe a strong task dependent variance in VLM performance: while most models are able to answer questions regarding the product brand and price with high accuracy, they completely fail at the same time to correctly identity the specific product name or discount. This indicates the problem of VLMs to solve fine-grained classification tasks as well to model the more abstract concept of discounts.

Title: CSAD: Unsupervised Component Segmentation for Logical Anomaly Detection

• Authors: Yu-Hsuan Hsieh, Shang-Hong Lai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15628
• Pdf URL: https://arxiv.org/pdf/2408.15628
• Copy Paste: [[2408.15628]] CSAD: Unsupervised Component Segmentation for Logical Anomaly Detection(https://arxiv.org/abs/2408.15628)
• Keywords: foundation model, anomaly
• Abstract: To improve logical anomaly detection, some previous works have integrated segmentation techniques with conventional anomaly detection methods. Although these methods are effective, they frequently lead to unsatisfactory segmentation results and require manual annotations. To address these drawbacks, we develop an unsupervised component segmentation technique that leverages foundation models to autonomously generate training labels for a lightweight segmentation network without human labeling. Integrating this new segmentation technique with our proposed Patch Histogram module and the Local-Global Student-Teacher (LGST) module, we achieve a detection AUROC of 95.3% in the MVTec LOCO AD dataset, which surpasses previous SOTA methods. Furthermore, our proposed method provides lower latency and higher throughput than most existing approaches.

Title: GANs Conditioning Methods: A Survey

• Authors: Anis Bourou, Auguste Genovesio, Valérie Mezger
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.15640
• Pdf URL: https://arxiv.org/pdf/2408.15640
• Copy Paste: [[2408.15640]] GANs Conditioning Methods: A Survey(https://arxiv.org/abs/2408.15640)
• Keywords: generative
• Abstract: In recent years, Generative Adversarial Networks (GANs) have seen significant advancements, leading to their widespread adoption across various fields. The original GAN architecture enables the generation of images without any specific control over the content, making it an unconditional generation process. However, many practical applications require precise control over the generated output, which has led to the development of conditional GANs (cGANs) that incorporate explicit conditioning to guide the generation process. cGANs extend the original framework by incorporating additional information (conditions), enabling the generation of samples that adhere to that specific criteria. Various conditioning methods have been proposed, each differing in how they integrate the conditioning information into both the generator and the discriminator networks. In this work, we review the conditioning methods proposed for GANs, exploring the characteristics of each method and highlighting their unique mechanisms and theoretical foundations. Furthermore, we conduct a comparative analysis of these methods, evaluating their performance on various image datasets. Through these analyses, we aim to provide insights into the strengths and limitations of various conditioning techniques, guiding future research and application in generative modeling.

Title: Harnessing the Intrinsic Knowledge of Pretrained Language Models for Challenging Text Classification Settings

• Authors: Lingyu Gao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.15650
• Pdf URL: https://arxiv.org/pdf/2408.15650
• Copy Paste: [[2408.15650]] Harnessing the Intrinsic Knowledge of Pretrained Language Models for Challenging Text Classification Settings(https://arxiv.org/abs/2408.15650)
• Keywords: in-context
• Abstract: Text classification is crucial for applications such as sentiment analysis and toxic text filtering, but it still faces challenges due to the complexity and ambiguity of natural language. Recent advancements in deep learning, particularly transformer architectures and large-scale pretraining, have achieved inspiring success in NLP fields. Building on these advancements, this thesis explores three challenging settings in text classification by leveraging the intrinsic knowledge of pretrained language models (PLMs). Firstly, to address the challenge of selecting misleading yet incorrect distractors for cloze questions, we develop models that utilize features based on contextualized word representations from PLMs, achieving performance that rivals or surpasses human accuracy. Secondly, to enhance model generalization to unseen labels, we create small finetuning datasets with domain-independent task label descriptions, improving model performance and robustness. Lastly, we tackle the sensitivity of large language models to in-context learning prompts by selecting effective demonstrations, focusing on misclassified examples and resolving model ambiguity regarding test example labels.

Title: Merging and Splitting Diffusion Paths for Semantically Coherent Panoramas

• Authors: Fabio Quattrini, Vittorio Pippi, Silvia Cascianelli, Rita Cucchiara
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15660
• Pdf URL: https://arxiv.org/pdf/2408.15660
• Copy Paste: [[2408.15660]] Merging and Splitting Diffusion Paths for Semantically Coherent Panoramas(https://arxiv.org/abs/2408.15660)
• Keywords: diffusion
• Abstract: Diffusion models have become the State-of-the-Art for text-to-image generation, and increasing research effort has been dedicated to adapting the inference process of pretrained diffusion models to achieve zero-shot capabilities. An example is the generation of panorama images, which has been tackled in recent works by combining independent diffusion paths over overlapping latent features, which is referred to as joint diffusion, obtaining perceptually aligned panoramas. However, these methods often yield semantically incoherent outputs and trade-off diversity for uniformity. To overcome this limitation, we propose the Merge-Attend-Diffuse operator, which can be plugged into different types of pretrained diffusion models used in a joint diffusion setting to improve the perceptual and semantical coherence of the generated panorama images. Specifically, we merge the diffusion paths, reprogramming self- and cross-attention to operate on the aggregated latent space. Extensive quantitative and qualitative experimental analysis, together with a user study, demonstrate that our method maintains compatibility with the input prompt and visual quality of the generated images while increasing their semantic coherence. We release the code at this https URL.

Title: Towards reliable respiratory disease diagnosis based on cough sounds and vision transformers

• Authors: Qian Wang, Zhaoyang Bu, Jiaxuan Mao, Wenyu Zhu, Jingya Zhao, Wei Du, Guochao Shi, Min Zhou, Si Chen, Jieming Qu
• Subjects: cs.CV, cs.LG, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2408.15667
• Pdf URL: https://arxiv.org/pdf/2408.15667
• Copy Paste: [[2408.15667]] Towards reliable respiratory disease diagnosis based on cough sounds and vision transformers(https://arxiv.org/abs/2408.15667)
• Keywords: self-supervised
• Abstract: Recent advancements in deep learning techniques have sparked performance boosts in various real-world applications including disease diagnosis based on multi-modal medical data. Cough sound data-based respiratory disease (e.g., COVID-19 and Chronic Obstructive Pulmonary Disease) diagnosis has also attracted much attention. However, existing works usually utilise traditional machine learning or deep models of moderate scales. On the other hand, the developed approaches are trained and evaluated on small-scale data due to the difficulty of curating and annotating clinical data on scale. To address these issues in prior works, we create a unified framework to evaluate various deep models from lightweight Convolutional Neural Networks (e.g., ResNet18) to modern vision transformers and compare their performance in respiratory disease classification. Based on the observations from such an extensive empirical study, we propose a novel approach to cough-based disease classification based on both self-supervised and supervised learning on a large-scale cough data set. Experimental results demonstrate our proposed approach outperforms prior arts consistently on two benchmark datasets for COVID-19 diagnosis and a proprietary dataset for COPD/non-COPD classification with an AUROC of 92.5%.

Title: Synthetic Forehead-creases Biometric Generation for Reliable User Verification

• Authors: Abhishek Tandon, Geetanjali Sharma, Gaurav Jaswal, Aditya Nigam, Raghavendra Ramachandra
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15693
• Pdf URL: https://arxiv.org/pdf/2408.15693
• Copy Paste: [[2408.15693]] Synthetic Forehead-creases Biometric Generation for Reliable User Verification(https://arxiv.org/abs/2408.15693)
• Keywords: diffusion
• Abstract: Recent studies have emphasized the potential of forehead-crease patterns as an alternative for face, iris, and periocular recognition, presenting contactless and convenient solutions, particularly in situations where faces are covered by surgical masks. However, collecting forehead data presents challenges, including cost and time constraints, as developing and optimizing forehead verification methods requires a substantial number of high-quality images. To tackle these challenges, the generation of synthetic biometric data has gained traction due to its ability to protect privacy while enabling effective training of deep learning-based biometric verification methods. In this paper, we present a new framework to synthesize forehead-crease image data while maintaining important features, such as uniqueness and realism. The proposed framework consists of two main modules: a Subject-Specific Generation Module (SSGM), based on an image-to-image Brownian Bridge Diffusion Model (BBDM), which learns a one-to-many mapping between image pairs to generate identity-aware synthetic forehead creases corresponding to real subjects, and a Subject-Agnostic Generation Module (SAGM), which samples new synthetic identities with assistance from the SSGM. We evaluate the diversity and realism of the generated forehead-crease images primarily using the Fréchet Inception Distance (FID) and the Structural Similarity Index Measure (SSIM). In addition, we assess the utility of synthetically generated forehead-crease images using a forehead-crease verification system (FHCVS). The results indicate an improvement in the verification accuracy of the FHCVS by utilizing synthetic data.

Title: Autoregressive model path dependence near Ising criticality

• Authors: Yi Hong Teoh, Roger G. Melko
• Subjects: cs.LG, cond-mat.dis-nn
• Abstract URL: https://arxiv.org/abs/2408.15715
• Pdf URL: https://arxiv.org/pdf/2408.15715
• Copy Paste: [[2408.15715]] Autoregressive model path dependence near Ising criticality(https://arxiv.org/abs/2408.15715)
• Keywords: generative
• Abstract: Autoregressive models are a class of generative model that probabilistically predict the next output of a sequence based on previous inputs. The autoregressive sequence is by definition one-dimensional (1D), which is natural for language tasks and hence an important component of modern architectures like recurrent neural networks (RNNs) and transformers. However, when language models are used to predict outputs on physical systems that are not intrinsically 1D, the question arises of which choice of autoregressive sequence -- if any -- is optimal. In this paper, we study the reconstruction of critical correlations in the two-dimensional (2D) Ising model, using RNNs and transformers trained on binary spin data obtained near the thermal phase transition. We compare the training performance for a number of different 1D autoregressive sequences imposed on finite-size 2D lattices. We find that paths with long 1D segments are more efficient at training the autoregressive models compared to space-filling curves that better preserve the 2D locality. Our results illustrate the potential importance in choosing the optimal autoregressive sequence ordering when training modern language models for tasks in physics.

Title: Defending Text-to-image Diffusion Models: Surprising Efficacy of Textual Perturbations Against Backdoor Attacks

• Authors: Oscar Chew, Po-Yi Lu, Jayden Lin, Hsuan-Tien Lin
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15721
• Pdf URL: https://arxiv.org/pdf/2408.15721
• Copy Paste: [[2408.15721]] Defending Text-to-image Diffusion Models: Surprising Efficacy of Textual Perturbations Against Backdoor Attacks(https://arxiv.org/abs/2408.15721)
• Keywords: diffusion
• Abstract: Text-to-image diffusion models have been widely adopted in real-world applications due to their ability to generate realistic images from textual descriptions. However, recent studies have shown that these methods are vulnerable to backdoor attacks. Despite the significant threat posed by backdoor attacks on text-to-image diffusion models, countermeasures remain under-explored. In this paper, we address this research gap by demonstrating that state-of-the-art backdoor attacks against text-to-image diffusion models can be effectively mitigated by a surprisingly simple defense strategy - textual perturbation. Experiments show that textual perturbations are effective in defending against state-of-the-art backdoor attacks with minimal sacrifice to generation quality. We analyze the efficacy of textual perturbation from two angles: text embedding space and cross-attention maps. They further explain how backdoor attacks have compromised text-to-image diffusion models, providing insights for studying future attack and defense strategies. Our code is available at this https URL.

Title: Form and meaning co-determine the realization of tone in Taiwan Mandarin spontaneous speech: the case of Tone 3 sandhi

• Authors: Yuxin Lu, Yu-Ying Chuang, R. Harald Baayen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2408.15747
• Pdf URL: https://arxiv.org/pdf/2408.15747
• Copy Paste: [[2408.15747]] Form and meaning co-determine the realization of tone in Taiwan Mandarin spontaneous speech: the case of Tone 3 sandhi(https://arxiv.org/abs/2408.15747)
• Keywords: generative
• Abstract: In Standard Chinese, Tone 3 (the dipping tone) becomes Tone 2 (rising tone) when followed by another Tone 3. Previous studies have noted that this sandhi process may be incomplete, in the sense that the assimilated Tone 3 is still distinct from a true Tone 2. While Mandarin Tone 3 sandhi is widely studied using carefully controlled laboratory speech (Xu, 1997) and more formal registers of Beijing Mandarin (Yuan and Chen, 2014), less is known about its realization in spontaneous speech, and about the effect of contextual factors on tonal realization. The present study investigates the pitch contours of two-character words with T2-T3 and T3-T3 tone patterns in spontaneous Taiwan Mandarin conversations. Our analysis makes use of the Generative Additive Mixed Model (GAMM, Wood, 2017) to examine fundamental frequency (f0) contours as a function of normalized time. We consider various factors known to influence pitch contours, including gender, speaking rate, speaker, neighboring tones, word position, bigram probability, and also novel predictors, word and word sense (Chuang et al., 2024). Our analyses revealed that in spontaneous Taiwan Mandarin, T3-T3 words become indistinguishable from T2-T3 words, indicating complete sandhi, once the strong effect of word (or word sense) is taken into account. For our data, the shape of f0 contours is not co-determined by word frequency. In contrast, the effect of word meaning on f0 contours is robust, as strong as the effect of adjacent tones, and is present for both T2-T3 and T3-T3 words.

Title: GenDDS: Generating Diverse Driving Video Scenarios with Prompt-to-Video Generative Model

• Authors: Yongjie Fu, Yunlong Li, Xuan Di
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.15868
• Pdf URL: https://arxiv.org/pdf/2408.15868
• Copy Paste: [[2408.15868]] GenDDS: Generating Diverse Driving Video Scenarios with Prompt-to-Video Generative Model(https://arxiv.org/abs/2408.15868)
• Keywords: diffusion, generative
• Abstract: Autonomous driving training requires a diverse range of datasets encompassing various traffic conditions, weather scenarios, and road types. Traditional data augmentation methods often struggle to generate datasets that represent rare occurrences. To address this challenge, we propose GenDDS, a novel approach for generating driving scenarios generation by leveraging the capabilities of Stable Diffusion XL (SDXL), an advanced latent diffusion model. Our methodology involves the use of descriptive prompts to guide the synthesis process, aimed at producing realistic and diverse driving scenarios. With the power of the latest computer vision techniques, such as ControlNet and Hotshot-XL, we have built a complete pipeline for video generation together with SDXL. We employ the KITTI dataset, which includes real-world driving videos, to train the model. Through a series of experiments, we demonstrate that our model can generate high-quality driving videos that closely replicate the complexity and variability of real-world driving scenarios. This research contributes to the development of sophisticated training data for autonomous driving systems and opens new avenues for creating virtual environments for simulation and validation purposes.

Title: Disentangled Diffusion Autoencoder for Harmonization of Multi-site Neuroimaging Data

• Authors: Ayodeji Ijishakin, Ana Lawry Aguila, Elizabeth Levitis, Ahmed Abdulaal, Andre Altmann, James Cole
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15890
• Pdf URL: https://arxiv.org/pdf/2408.15890
• Copy Paste: [[2408.15890]] Disentangled Diffusion Autoencoder for Harmonization of Multi-site Neuroimaging Data(https://arxiv.org/abs/2408.15890)
• Keywords: diffusion, generative
• Abstract: Combining neuroimaging datasets from multiple sites and scanners can help increase statistical power and thus provide greater insight into subtle neuroanatomical effects. However, site-specific effects pose a challenge by potentially obscuring the biological signal and introducing unwanted variance. Existing harmonization techniques, which use statistical models to remove such effects, have been shown to incompletely remove site effects while also failing to preserve biological variability. More recently, generative models using GANs or autoencoder-based approaches, have been proposed for site adjustment. However, such methods are known for instability during training or blurry image generation. In recent years, diffusion models have become increasingly popular for their ability to generate high-quality synthetic images. In this work, we introduce the disentangled diffusion autoencoder (DDAE), a novel diffusion model designed for controlling specific aspects of an image. We apply the DDAE to the task of harmonizing MR images by generating high-quality site-adjusted images that preserve biological variability. We use data from 7 different sites and demonstrate the DDAE's superiority in generating high-resolution, harmonized 2D MR images over previous approaches. As far as we are aware, this work marks the first diffusion-based model for site adjustment of neuroimaging data.

Title: Airfoil Diffusion: Denoising Diffusion Model For Conditional Airfoil Generation

• Authors: Reid Graves, Amir Barati Farimani
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2408.15898
• Pdf URL: https://arxiv.org/pdf/2408.15898
• Copy Paste: [[2408.15898]] Airfoil Diffusion: Denoising Diffusion Model For Conditional Airfoil Generation(https://arxiv.org/abs/2408.15898)
• Keywords: diffusion
• Abstract: The design of aerodynamic shapes, such as airfoils, has traditionally required significant computational resources and relied on predefined design parameters, which limit the potential for novel shape synthesis. In this work, we introduce a data-driven methodology for airfoil generation using a diffusion model. Trained on a dataset of preexisting airfoils, our model can generate an arbitrary number of new airfoils from random vectors, which can be conditioned on specific aerodynamic performance metrics such as lift and drag, or geometric criteria. Our results demonstrate that the diffusion model effectively produces airfoil shapes with realistic aerodynamic properties, offering substantial improvements in efficiency, flexibility, and the potential for discovering innovative airfoil designs. This approach significantly expands the design space, facilitating the synthesis of high-performance aerodynamic shapes that transcend the limitations of traditional methods.

Title: MetaGFN: Exploring Distant Modes with Adapted Metadynamics for Continuous GFlowNets

• Authors: Dominic Phillips, Flaviu Cipcigan
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2408.15905
• Pdf URL: https://arxiv.org/pdf/2408.15905
• Copy Paste: [[2408.15905]] MetaGFN: Exploring Distant Modes with Adapted Metadynamics for Continuous GFlowNets(https://arxiv.org/abs/2408.15905)
• Keywords: generative
• Abstract: Generative Flow Networks (GFlowNets) are a class of generative models that sample objects in proportion to a specified reward function through a learned policy. They can be trained either on-policy or off-policy, needing a balance between exploration and exploitation for fast convergence to a target distribution. While exploration strategies for discrete GFlowNets have been studied, exploration in the continuous case remains to be investigated, despite the potential for novel exploration algorithms due to the local connectedness of continuous domains. Here, we introduce Adapted Metadynamics, a variant of metadynamics that can be applied to arbitrary black-box reward functions on continuous domains. We use Adapted Metadynamics as an exploration strategy for continuous GFlowNets. We show three continuous domains where the resulting algorithm, MetaGFN, accelerates convergence to the target distribution and discovers more distant reward modes than previous off-policy exploration strategies used for GFlowNets.

Title: DiffAge3D: Diffusion-based 3D-aware Face Aging

• Authors: Junaid Wahid, Fangneng Zhan, Pramod Rao, Christian Theobalt
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15922
• Pdf URL: https://arxiv.org/pdf/2408.15922
• Copy Paste: [[2408.15922]] DiffAge3D: Diffusion-based 3D-aware Face Aging(https://arxiv.org/abs/2408.15922)
• Keywords: diffusion
• Abstract: Face aging is the process of converting an individual's appearance to a younger or older version of themselves. Existing face aging techniques have been limited to 2D settings, which often weaken their applications as there is a growing demand for 3D face modeling. Moreover, existing aging methods struggle to perform faithful aging, maintain identity, and retain the fine details of the input images. Given these limitations and the need for a 3D-aware aging method, we propose DiffAge3D, the first 3D-aware aging framework that not only performs faithful aging and identity preservation but also operates in a 3D setting. Our aging framework allows to model the aging and camera pose separately by only taking a single image with a target age. Our framework includes a robust 3D-aware aging dataset generation pipeline by utilizing a pre-trained 3D GAN and the rich text embedding capabilities within CLIP model. Notably, we do not employ any inversion bottleneck in dataset generation. Instead, we randomly generate training samples from the latent space of 3D GAN, allowing us to manipulate the rich latent space of GAN to generate ages even with large gaps. With the generated dataset, we train a viewpoint-aware diffusion-based aging model to control the camera pose and facial age. Through quantitative and qualitative evaluations, we demonstrate that DiffAge3D outperforms existing methods, particularly in multiview-consistent aging and fine details preservation.

Title: Efficient Slice Anomaly Detection Network for 3D Brain MRI Volume

• Authors: Zeduo Zhang, Yalda Mohsenzadeh
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2408.15958
• Pdf URL: https://arxiv.org/pdf/2408.15958
• Copy Paste: [[2408.15958]] Efficient Slice Anomaly Detection Network for 3D Brain MRI Volume(https://arxiv.org/abs/2408.15958)
• Keywords: anomaly
• Abstract: Current anomaly detection methods excel with benchmark industrial data but struggle with natural images and medical data due to varying definitions of 'normal' and 'abnormal.' This makes accurate identification of deviations in these fields particularly challenging. Especially for 3D brain MRI data, all the state-of-the-art models are reconstruction-based with 3D convolutional neural networks which are memory-intensive, time-consuming and producing noisy outputs that require further post-processing. We propose a framework called Simple Slice-based Network (SimpleSliceNet), which utilizes a model pre-trained on ImageNet and fine-tuned on a separate MRI dataset as a 2D slice feature extractor to reduce computational cost. We aggregate the extracted features to perform anomaly detection tasks on 3D brain MRI volumes. Our model integrates a conditional normalizing flow to calculate log likelihood of features and employs the Semi-Push-Pull Mechanism to enhance anomaly detection accuracy. The results indicate improved performance, showcasing our model's remarkable adaptability and effectiveness when addressing the challenges exists in brain MRI data. In addition, for the large-scale 3D brain volumes, our model SimpleSliceNet outperforms the state-of-the-art 2D and 3D models in terms of accuracy, memory usage and time consumption. Code is available at: https://anonymous.4open.science/r/SimpleSliceNet-8EA3.

Title: Distribution Backtracking Builds A Faster Convergence Trajectory for One-step Diffusion Distillation

• Authors: Shengyuan Zhang, Ling Yang, Zejian Li, An Zhao, Chenye Meng, Changyuan Yang, Guang Yang, Zhiyuan Yang, Lingyun Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15991
• Pdf URL: https://arxiv.org/pdf/2408.15991
• Copy Paste: [[2408.15991]] Distribution Backtracking Builds A Faster Convergence Trajectory for One-step Diffusion Distillation(https://arxiv.org/abs/2408.15991)
• Keywords: diffusion
• Abstract: Accelerating the sampling speed of diffusion models remains a significant challenge. Recent score distillation methods distill a heavy teacher model into an one-step student generator, which is optimized by calculating the difference between the two score functions on the samples generated by the student model. However, there is a score mismatch issue in the early stage of the distillation process, because existing methods mainly focus on using the endpoint of pre-trained diffusion models as teacher models, overlooking the importance of the convergence trajectory between the student generator and the teacher model. To address this issue, we extend the score distillation process by introducing the entire convergence trajectory of teacher models and propose Distribution Backtracking Distillation (DisBack) for distilling student generators. DisBask is composed of two stages: Degradation Recording and Distribution Backtracking. Degradation Recording is designed to obtain the convergence trajectory of teacher models, which records the degradation path from the trained teacher model to the untrained initial student generator. The degradation path implicitly represents the intermediate distributions of teacher models. Then Distribution Backtracking trains a student generator to backtrack the intermediate distributions for approximating the convergence trajectory of teacher models. Extensive experiments show that DisBack achieves faster and better convergence than the existing distillation method and accomplishes comparable generation performance. Notably, DisBack is easy to implement and can be generalized to existing distillation methods to boost performance. Our code is publicly available on this https URL.

Title: TEDRA: Text-based Editing of Dynamic and Photoreal Actors

• Authors: Basavaraj Sunagad, Heming Zhu, Mohit Mendiratta, Adam Kortylewski, Christian Theobalt, Marc Habermann
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2408.15995
• Pdf URL: https://arxiv.org/pdf/2408.15995
• Copy Paste: [[2408.15995]] TEDRA: Text-based Editing of Dynamic and Photoreal Actors(https://arxiv.org/abs/2408.15995)
• Keywords: diffusion, generative
• Abstract: Over the past years, significant progress has been made in creating photorealistic and drivable 3D avatars solely from videos of real humans. However, a core remaining challenge is the fine-grained and user-friendly editing of clothing styles by means of textual descriptions. To this end, we present TEDRA, the first method allowing text-based edits of an avatar, which maintains the avatar's high fidelity, space-time coherency, as well as dynamics, and enables skeletal pose and view control. We begin by training a model to create a controllable and high-fidelity digital replica of the real actor. Next, we personalize a pretrained generative diffusion model by fine-tuning it on various frames of the real character captured from different camera angles, ensuring the digital representation faithfully captures the dynamics and movements of the real person. This two-stage process lays the foundation for our approach to dynamic human avatar editing. Utilizing this personalized diffusion model, we modify the dynamic avatar based on a provided text prompt using our Personalized Normal Aligned Score Distillation Sampling (PNA-SDS) within a model-based guidance framework. Additionally, we propose a time step annealing strategy to ensure high-quality edits. Our results demonstrate a clear improvement over prior work in functionality and visual quality.