2024-03-20

Title: A Dataset and Benchmark for Copyright Protection from Text-to-Image Diffusion Models

• Authors: Rui Ma, Qiang Zhou, Bangjun Xiao, Yizhu Jin, Daquan Zhou, Xiuyu Li, Aishani Singh, Yi Qu, Kurt Keutzer, Xiaodong Xie, Jingtong Hu, Zhen Dong, Shanghang Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12052
• Pdf URL: https://arxiv.org/pdf/2403.12052
• Copy Paste: [[2403.12052]] A Dataset and Benchmark for Copyright Protection from Text-to-Image Diffusion Models(https://arxiv.org/abs/2403.12052)
• Keywords: diffusion
• Abstract: Copyright is a legal right that grants creators the exclusive authority to reproduce, distribute, and profit from their creative works. However, the recent advancements in text-to-image generation techniques have posed significant challenges to copyright protection, as these methods have facilitated the learning of unauthorized content, artistic creations, and portraits, which are subsequently utilized to generate and disseminate uncontrolled content. Especially, the use of stable diffusion, an emerging model for text-to-image generation, poses an increased risk of unauthorized copyright infringement and distribution. Currently, there is a lack of systematic studies evaluating the potential correlation between content generated by stable diffusion and those under copyright protection. Conducting such studies faces several challenges, including i) the intrinsic ambiguity related to copyright infringement in text-to-image models, ii) the absence of a comprehensive large-scale dataset, and iii) the lack of standardized metrics for defining copyright infringement. This work provides the first large-scale standardized dataset and benchmark on copyright protection. Specifically, we propose a pipeline to coordinate CLIP, ChatGPT, and diffusion models to generate a dataset that contains anchor images, corresponding prompts, and images generated by text-to-image models, reflecting the potential abuses of copyright. Furthermore, we explore a suite of evaluation metrics to judge the effectiveness of copyright protection methods. The proposed dataset, benchmark library, and evaluation metrics will be open-sourced to facilitate future research and application. The website and dataset can be accessed website dataset.

Title: Consistency Models Improve Diffusion Inverse Solvers

• Authors: Tongda Xu, Ziran Zhu, Dailan He, Yuanyuan Wang, Ming Sun, Ning Li, Hongwei Qin, Yan Wang, Jingjing Liu, Ya-Qin Zhang
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12063
• Pdf URL: https://arxiv.org/pdf/2403.12063
• Copy Paste: [[2403.12063]] Consistency Models Improve Diffusion Inverse Solvers(https://arxiv.org/abs/2403.12063)
• Keywords: diffusion
• Abstract: Diffusion inverse solvers (DIS) aim to find an image $x$ that lives on the diffusion prior while satisfying the constraint $f(x) = y$, given an operator $f(.)$ and measurement $y$. Most non-linear DIS use posterior mean $\hat{x}_{0|t}=\mathbb{E}[x_0|x_t]$ to evaluate $f(.)$ and minimize the distance $||f(\hat{x}_{0|t})-y||^2$. Previous works show that posterior mean-based distance is biased; instead, posterior sample $x_{0|t}\sim p_{\theta}(x_0|x_t)$ promises a better candidate. In this paper, we first clarify when is posterior sample better: $1)$ When $f(.)$ is linear, the distance with posterior mean is as good as single posterior sample, thus preferable as it does not require Monte Carlo; $2)$ When $f(.)$ is non-linear, the distance using posterior sample is better. As previous approximations to posterior sample do not look like a real image, we propose to use consistency model (CM) as a high quality approximation. In addition, we propose a new family of DIS using pure CM. Empirically, we show that replacing posterior mean by CM improves DIS performance on non-linear $f(.)$ (e.g. semantic segmentation, image captioning). Further, our pure CM inversion works well for both linear and non-linear $f(.)$.

Title: Evaluating Robustness of Generative Search Engine on Adversarial Factual Questions

• Authors: Xuming Hu, Xiaochuan Li, Junzhe Chen, Yinghui Li, Yangning Li, Xiaoguang Li, Yasheng Wang, Qun Liu, Lijie Wen, Philip S. Yu, Zhijiang Guo
• Subjects: cs.CL, cs.AI, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.12077
• Pdf URL: https://arxiv.org/pdf/2403.12077
• Copy Paste: [[2403.12077]] Evaluating Robustness of Generative Search Engine on Adversarial Factual Questions(https://arxiv.org/abs/2403.12077)
• Keywords: generative
• Abstract: Generative search engines have the potential to transform how people seek information online, but generated responses from existing large language models (LLMs)-backed generative search engines may not always be accurate. Nonetheless, retrieval-augmented generation exacerbates safety concerns, since adversaries may successfully evade the entire system by subtly manipulating the most vulnerable part of a claim. To this end, we propose evaluating the robustness of generative search engines in the realistic and high-risk setting, where adversaries have only black-box system access and seek to deceive the model into returning incorrect responses. Through a comprehensive human evaluation of various generative search engines, such as Bing Chat, PerplexityAI, and YouChat across diverse queries, we demonstrate the effectiveness of adversarial factual questions in inducing incorrect responses. Moreover, retrieval-augmented generation exhibits a higher susceptibility to factual errors compared to LLMs without retrieval. These findings highlight the potential security risks of these systems and emphasize the need for rigorous evaluation before deployment.

Title: Deep Generative Design for Mass Production

• Authors: Jihoon Kim, Yongmin Kwon, Namwoo Kang
• Subjects: cs.CV, cs.AI, cs.LG, eess.IV
• Abstract URL: https://arxiv.org/abs/2403.12098
• Pdf URL: https://arxiv.org/pdf/2403.12098
• Copy Paste: [[2403.12098]] Deep Generative Design for Mass Production(https://arxiv.org/abs/2403.12098)
• Keywords: generative
• Abstract: Generative Design (GD) has evolved as a transformative design approach, employing advanced algorithms and AI to create diverse and innovative solutions beyond traditional constraints. Despite its success, GD faces significant challenges regarding the manufacturability of complex designs, often necessitating extensive manual modifications due to limitations in standard manufacturing processes and the reliance on additive manufacturing, which is not ideal for mass production. Our research introduces an innovative framework addressing these manufacturability concerns by integrating constraints pertinent to die casting and injection molding into GD, through the utilization of 2D depth images. This method simplifies intricate 3D geometries into manufacturable profiles, removing unfeasible features such as non-manufacturable overhangs and allowing for the direct consideration of essential manufacturing aspects like thickness and rib design. Consequently, designs previously unsuitable for mass production are transformed into viable solutions. We further enhance this approach by adopting an advanced 2D generative model, which offer a more efficient alternative to traditional 3D shape generation methods. Our results substantiate the efficacy of this framework, demonstrating the production of innovative, and, importantly, manufacturable designs. This shift towards integrating practical manufacturing considerations into GD represents a pivotal advancement, transitioning from purely inspirational concepts to actionable, production-ready solutions. Our findings underscore usefulness and potential of GD for broader industry adoption, marking a significant step forward in aligning GD with the demands of manufacturing challenges.

Title: Syn-QA2: Evaluating False Assumptions in Long-tail Questions with Synthetic QA Datasets

• Authors: Ashwin Daswani, Rohan Sawant, Najoung Kim
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12145
• Pdf URL: https://arxiv.org/pdf/2403.12145
• Copy Paste: [[2403.12145]] Syn-QA2: Evaluating False Assumptions in Long-tail Questions with Synthetic QA Datasets(https://arxiv.org/abs/2403.12145)
• Keywords: generative
• Abstract: Sensitivity to false assumptions (or false premises) in information-seeking questions is critical for robust question-answering (QA) systems. Recent work has shown that false assumptions in naturally occurring questions pose challenges to current models, with low performance on both generative QA and simple detection tasks (Kim et al. 2023). However, the focus of existing work on naturally occurring questions leads to a gap in the analysis of model behavior on the long tail of the distribution of possible questions. To this end, we introduce Syn-(QA)$^2$, a set of two synthetically generated QA datasets: one generated using perturbed relations from Wikidata, and the other by perturbing HotpotQA (Yang et al. 2018). Our findings from evaluating a range of large language models are threefold: (1) false assumptions in QA are challenging, echoing the findings of prior work, (2) the binary detection task is challenging even compared to the difficulty of generative QA itself, possibly due to the linguistic structure of the problem, and (3) the detection task is more challenging with long-tail questions compared to naturally occurring questions, highlighting the utility of our synthetic datasets and generation method.

Title: Graph-Jigsaw Conditioned Diffusion Model for Skeleton-based Video Anomaly Detection

• Authors: Ali Karami, Thi Kieu Khanh Ho, Narges Armanfard
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12172
• Pdf URL: https://arxiv.org/pdf/2403.12172
• Copy Paste: [[2403.12172]] Graph-Jigsaw Conditioned Diffusion Model for Skeleton-based Video Anomaly Detection(https://arxiv.org/abs/2403.12172)
• Keywords: diffusion, anomaly
• Abstract: Skeleton-based video anomaly detection (SVAD) is a crucial task in computer vision. Accurately identifying abnormal patterns or events enables operators to promptly detect suspicious activities, thereby enhancing safety. Achieving this demands a comprehensive understanding of human motions, both at body and region levels, while also accounting for the wide variations of performing a single action. However, existing studies fail to simultaneously address these crucial properties. This paper introduces a novel, practical and lightweight framework, namely Graph-Jigsaw Conditioned Diffusion Model for Skeleton-based Video Anomaly Detection (GiCiSAD) to overcome the challenges associated with SVAD. GiCiSAD consists of three novel modules: the Graph Attention-based Forecasting module to capture the spatio-temporal dependencies inherent in the data, the Graph-level Jigsaw Puzzle Maker module to distinguish subtle region-level discrepancies between normal and abnormal motions, and the Graph-based Conditional Diffusion model to generate a wide spectrum of human motions. Extensive experiments on four widely used skeleton-based video datasets show that GiCiSAD outperforms existing methods with significantly fewer training parameters, establishing it as the new state-of-the-art.

Title: E2F-Net: Eyes-to-Face Inpainting via StyleGAN Latent Space

• Authors: Ahmad Hassanpour, Fatemeh Jamalbafrani, Bian Yang, Kiran Raja, Raymond Veldhuis, Julian Fierrez
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12197
• Pdf URL: https://arxiv.org/pdf/2403.12197
• Copy Paste: [[2403.12197]] E2F-Net: Eyes-to-Face Inpainting via StyleGAN Latent Space(https://arxiv.org/abs/2403.12197)
• Keywords: generative
• Abstract: Face inpainting, the technique of restoring missing or damaged regions in facial images, is pivotal for applications like face recognition in occluded scenarios and image analysis with poor-quality captures. This process not only needs to produce realistic visuals but also preserve individual identity characteristics. The aim of this paper is to inpaint a face given periocular region (eyes-to-face) through a proposed new Generative Adversarial Network (GAN)-based model called Eyes-to-Face Network (E2F-Net). The proposed approach extracts identity and non-identity features from the periocular region using two dedicated encoders have been used. The extracted features are then mapped to the latent space of a pre-trained StyleGAN generator to benefit from its state-of-the-art performance and its rich, diverse and expressive latent space without any additional training. We further improve the StyleGAN output to find the optimal code in the latent space using a new optimization for GAN inversion technique. Our E2F-Net requires a minimum training process reducing the computational complexity as a secondary benefit. Through extensive experiments, we show that our method successfully reconstructs the whole face with high quality, surpassing current techniques, despite significantly less training and supervision efforts. We have generated seven eyes-to-face datasets based on well-known public face datasets for training and verifying our proposed methods. The code and datasets are publicly available.

Title: Large language models in 6G security: challenges and opportunities

• Authors: Tri Nguyen, Huong Nguyen, Ahmad Ijaz, Saeid Sheikhi, Athanasios V. Vasilakos, Panos Kostakos
• Subjects: cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2403.12239
• Pdf URL: https://arxiv.org/pdf/2403.12239
• Copy Paste: [[2403.12239]] Large language models in 6G security: challenges and opportunities(https://arxiv.org/abs/2403.12239)
• Keywords: generative
• Abstract: The rapid integration of Generative AI (GenAI) and Large Language Models (LLMs) in sectors such as education and healthcare have marked a significant advancement in technology. However, this growth has also led to a largely unexplored aspect: their security vulnerabilities. As the ecosystem that includes both offline and online models, various tools, browser plugins, and third-party applications continues to expand, it significantly widens the attack surface, thereby escalating the potential for security breaches. These expansions in the 6G and beyond landscape provide new avenues for adversaries to manipulate LLMs for malicious purposes. We focus on the security aspects of LLMs from the viewpoint of potential adversaries. We aim to dissect their objectives and methodologies, providing an in-depth analysis of known security weaknesses. This will include the development of a comprehensive threat taxonomy, categorizing various adversary behaviors. Also, our research will concentrate on how LLMs can be integrated into cybersecurity efforts by defense teams, also known as blue teams. We will explore the potential synergy between LLMs and blockchain technology, and how this combination could lead to the development of next-generation, fully autonomous security solutions. This approach aims to establish a unified cybersecurity strategy across the entire computing continuum, enhancing overall digital security infrastructure.

Title: FinLlama: Financial Sentiment Classification for Algorithmic Trading Applications

• Authors: Thanos Konstantinidis, Giorgos Iacovides, Mingxue Xu, Tony G. Constantinides, Danilo Mandic
• Subjects: cs.CL, cs.LG, q-fin.ST, q-fin.TR
• Abstract URL: https://arxiv.org/abs/2403.12285
• Pdf URL: https://arxiv.org/pdf/2403.12285
• Copy Paste: [[2403.12285]] FinLlama: Financial Sentiment Classification for Algorithmic Trading Applications(https://arxiv.org/abs/2403.12285)
• Keywords: generative
• Abstract: There are multiple sources of financial news online which influence market movements and trader's decisions. This highlights the need for accurate sentiment analysis, in addition to having appropriate algorithmic trading techniques, to arrive at better informed trading decisions. Standard lexicon based sentiment approaches have demonstrated their power in aiding financial decisions. However, they are known to suffer from issues related to context sensitivity and word ordering. Large Language Models (LLMs) can also be used in this context, but they are not finance-specific and tend to require significant computational resources. To facilitate a finance specific LLM framework, we introduce a novel approach based on the Llama 2 7B foundational model, in order to benefit from its generative nature and comprehensive language manipulation. This is achieved by fine-tuning the Llama2 7B model on a small portion of supervised financial sentiment analysis data, so as to jointly handle the complexities of financial lexicon and context, and further equipping it with a neural network based decision mechanism. Such a generator-classifier scheme, referred to as FinLlama, is trained not only to classify the sentiment valence but also quantify its strength, thus offering traders a nuanced insight into financial news articles. Complementing this, the implementation of parameter-efficient fine-tuning through LoRA optimises trainable parameters, thus minimising computational and memory requirements, without sacrificing accuracy. Simulation results demonstrate the ability of the proposed FinLlama to provide a framework for enhanced portfolio management decisions and increased market returns. These results underpin the ability of FinLlama to construct high-return portfolios which exhibit enhanced resilience, even during volatile periods and unpredictable market events.

Title: Removing Undesirable Concepts in Text-to-Image Generative Models with Learnable Prompts

• Authors: Anh Bui, Khanh Doan, Trung Le, Paul Montague, Tamas Abraham, Dinh Phung
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12326
• Pdf URL: https://arxiv.org/pdf/2403.12326
• Copy Paste: [[2403.12326]] Removing Undesirable Concepts in Text-to-Image Generative Models with Learnable Prompts(https://arxiv.org/abs/2403.12326)
• Keywords: diffusion, generative
• Abstract: Generative models have demonstrated remarkable potential in generating visually impressive content from textual descriptions. However, training these models on unfiltered internet data poses the risk of learning and subsequently propagating undesirable concepts, such as copyrighted or unethical content. In this paper, we propose a novel method to remove undesirable concepts from text-to-image generative models by incorporating a learnable prompt into the cross-attention module. This learnable prompt acts as additional memory to transfer the knowledge of undesirable concepts into it and reduce the dependency of these concepts on the model parameters and corresponding textual inputs. Because of this knowledge transfer into the prompt, erasing these undesirable concepts is more stable and has minimal negative impact on other concepts. We demonstrate the effectiveness of our method on the Stable Diffusion model, showcasing its superiority over state-of-the-art erasure methods in terms of removing undesirable content while preserving other unrelated elements.

Title: DMAD: Dual Memory Bank for Real-World Anomaly Detection

• Authors: Jianlong Hu, Xu Chen, Zhenye Gan, Jinlong Peng, Shengchuan Zhang, Jiangning Zhang, Yabiao Wang, Chengjie Wang, Liujuan Cao, Rongrong Ji
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12362
• Pdf URL: https://arxiv.org/pdf/2403.12362
• Copy Paste: [[2403.12362]] DMAD: Dual Memory Bank for Real-World Anomaly Detection(https://arxiv.org/abs/2403.12362)
• Keywords: anomaly
• Abstract: Training a unified model is considered to be more suitable for practical industrial anomaly detection scenarios due to its generalization ability and storage efficiency. However, this multi-class setting, which exclusively uses normal data, overlooks the few but important accessible annotated anomalies in the real world. To address the challenge of real-world anomaly detection, we propose a new framework named Dual Memory bank enhanced representation learning for Anomaly Detection (DMAD). This framework handles both unsupervised and semi-supervised scenarios in a unified (multi-class) setting. DMAD employs a dual memory bank to calculate feature distance and feature attention between normal and abnormal patterns, thereby encapsulating knowledge about normal and abnormal instances. This knowledge is then used to construct an enhanced representation for anomaly score learning. We evaluated DMAD on the MVTec-AD and VisA datasets. The results show that DMAD surpasses current state-of-the-art methods, highlighting DMAD's capability in handling the complexities of real-world anomaly detection scenarios.

Title: GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation

• Authors: Quankai Gao, Qiangeng Xu, Zhe Cao, Ben Mildenhall, Wenchao Ma, Le Chen, Danhang Tang, Ulrich Neumann
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12365
• Pdf URL: https://arxiv.org/pdf/2403.12365
• Copy Paste: [[2403.12365]] GaussianFlow: Splatting Gaussian Dynamics for 4D Content Creation(https://arxiv.org/abs/2403.12365)
• Keywords: generative
• Abstract: Creating 4D fields of Gaussian Splatting from images or videos is a challenging task due to its under-constrained nature. While the optimization can draw photometric reference from the input videos or be regulated by generative models, directly supervising Gaussian motions remains underexplored. In this paper, we introduce a novel concept, Gaussian flow, which connects the dynamics of 3D Gaussians and pixel velocities between consecutive frames. The Gaussian flow can be efficiently obtained by splatting Gaussian dynamics into the image space. This differentiable process enables direct dynamic supervision from optical flow. Our method significantly benefits 4D dynamic content generation and 4D novel view synthesis with Gaussian Splatting, especially for contents with rich motions that are hard to be handled by existing methods. The common color drifting issue that happens in 4D generation is also resolved with improved Guassian dynamics. Superior visual quality on extensive experiments demonstrates our method's effectiveness. Quantitative and qualitative evaluations show that our method achieves state-of-the-art results on both tasks of 4D generation and 4D novel view synthesis. Project page: https://zerg-overmind.github.io/GaussianFlow.github.io/

Title: Advancing Time Series Classification with Multimodal Language Modeling

• Authors: Mingyue Cheng, Yiheng Chen, Qi Liu, Zhiding Liu, Yucong Luo
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12371
• Pdf URL: https://arxiv.org/pdf/2403.12371
• Copy Paste: [[2403.12371]] Advancing Time Series Classification with Multimodal Language Modeling(https://arxiv.org/abs/2403.12371)
• Keywords: foundation model, generative
• Abstract: For the advancements of time series classification, scrutinizing previous studies, most existing methods adopt a common learning-to-classify paradigm - a time series classifier model tries to learn the relation between sequence inputs and target label encoded by one-hot distribution. Although effective, this paradigm conceals two inherent limitations: (1) encoding target categories with one-hot distribution fails to reflect the comparability and similarity between labels, and (2) it is very difficult to learn transferable model across domains, which greatly hinder the development of universal serving paradigm. In this work, we propose InstructTime, a novel attempt to reshape time series classification as a learning-to-generate paradigm. Relying on the powerful generative capacity of the pre-trained language model, the core idea is to formulate the classification of time series as a multimodal understanding task, in which both task-specific instructions and raw time series are treated as multimodal inputs while the label information is represented by texts. To accomplish this goal, three distinct designs are developed in the InstructTime. Firstly, a time series discretization module is designed to convert continuous time series into a sequence of hard tokens to solve the inconsistency issue across modal inputs. To solve the modality representation gap issue, for one thing, we introduce an alignment projected layer before feeding the transformed token of time series into language models. For another, we highlight the necessity of auto-regressive pre-training across domains, which can facilitate the transferability of the language model and boost the generalization performance. Extensive experiments are conducted over benchmark datasets, whose results uncover the superior performance of InstructTime and the potential for a universal foundation model in time series classification.

Title: Learning Transferable Time Series Classifier with Cross-Domain Pre-training from Language Model

• Authors: Mingyue Cheng, Xiaoyu Tao, Qi Liu, Hao Zhang, Yiheng Chen, Chenyi Lei
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12372
• Pdf URL: https://arxiv.org/pdf/2403.12372
• Copy Paste: [[2403.12372]] Learning Transferable Time Series Classifier with Cross-Domain Pre-training from Language Model(https://arxiv.org/abs/2403.12372)
• Keywords: self-supervised
• Abstract: Advancements in self-supervised pre-training (SSL) have significantly advanced the field of learning transferable time series representations, which can be very useful in enhancing the downstream task. Despite being effective, most existing works struggle to achieve cross-domain SSL pre-training, missing valuable opportunities to integrate patterns and features from different domains. The main challenge lies in the significant differences in the characteristics of time-series data across different domains, such as variations in the number of channels and temporal resolution scales. To address this challenge, we propose CrossTimeNet, a novel cross-domain SSL learning framework to learn transferable knowledge from various domains to largely benefit the target downstream task. One of the key characteristics of CrossTimeNet is the newly designed time series tokenization module, which could effectively convert the raw time series into a sequence of discrete tokens based on a reconstruction optimization process. Besides, we highlight that predicting a high proportion of corrupted tokens can be very helpful for extracting informative patterns across different domains during SSL pre-training, which has been largely overlooked in past years. Furthermore, unlike previous works, our work treats the pre-training language model (PLM) as the initialization of the encoder network, investigating the feasibility of transferring the knowledge learned by the PLM to the time series area. Through these efforts, the path to cross-domain pre-training of a generic time series model can be effectively paved. We conduct extensive experiments in a real-world scenario across various time series classification domains. The experimental results clearly confirm CrossTimeNet's superior performance.

Title: OV9D: Open-Vocabulary Category-Level 9D Object Pose and Size Estimation

• Authors: Junhao Cai, Yisheng He, Weihao Yuan, Siyu Zhu, Zilong Dong, Liefeng Bo, Qifeng Chen
• Subjects: cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.12396
• Pdf URL: https://arxiv.org/pdf/2403.12396
• Copy Paste: [[2403.12396]] OV9D: Open-Vocabulary Category-Level 9D Object Pose and Size Estimation(https://arxiv.org/abs/2403.12396)
• Keywords: diffusion, foundation model
• Abstract: This paper studies a new open-set problem, the open-vocabulary category-level object pose and size estimation. Given human text descriptions of arbitrary novel object categories, the robot agent seeks to predict the position, orientation, and size of the target object in the observed scene image. To enable such generalizability, we first introduce OO3D-9D, a large-scale photorealistic dataset for this task. Derived from OmniObject3D, OO3D-9D is the largest and most diverse dataset in the field of category-level object pose and size estimation. It includes additional annotations for the symmetry axis of each category, which help resolve symmetric ambiguity. Apart from the large-scale dataset, we find another key to enabling such generalizability is leveraging the strong prior knowledge in pre-trained visual-language foundation models. We then propose a framework built on pre-trained DinoV2 and text-to-image stable diffusion models to infer the normalized object coordinate space (NOCS) maps of the target instances. This framework fully leverages the visual semantic prior from DinoV2 and the aligned visual and language knowledge within the text-to-image diffusion model, which enables generalization to various text descriptions of novel categories. Comprehensive quantitative and qualitative experiments demonstrate that the proposed open-vocabulary method, trained on our large-scale synthesized data, significantly outperforms the baseline and can effectively generalize to real-world images of unseen categories. The project page is at https://ov9d.github.io.

Title: Electioneering the Network: Dynamic Multi-Step Adversarial Attacks for Community Canvassing

• Authors: Saurabh Sharma, Ambuj SIngh
• Subjects: cs.LG, cs.CR, cs.SI
• Abstract URL: https://arxiv.org/abs/2403.12399
• Pdf URL: https://arxiv.org/pdf/2403.12399
• Copy Paste: [[2403.12399]] Electioneering the Network: Dynamic Multi-Step Adversarial Attacks for Community Canvassing(https://arxiv.org/abs/2403.12399)
• Keywords: diffusion
• Abstract: The problem of online social network manipulation for community canvassing is of real concern in today's world. Motivated by the study of voter models, opinion and polarization dynamics on networks, we model community canvassing as a dynamic process over a network enabled via gradient-based attacks on GNNs. Existing attacks on GNNs are all single-step and do not account for the dynamic cascading nature of information diffusion in networks. We consider the realistic scenario where an adversary uses a GNN as a proxy to predict and manipulate voter preferences, especially uncertain voters. Gradient-based attacks on the GNN inform the adversary of strategic manipulations that can be made to proselytize targeted voters. In particular, we explore $\textit{minimum budget attacks for community canvassing}$ (MBACC). We show that the MBACC problem is NP-Hard and propose Dynamic Multi-Step Adversarial Community Canvassing (MAC) to address it. MAC makes dynamic local decisions based on the heuristic of low budget and high second-order influence to convert and perturb target voters. MAC is a dynamic multi-step attack that discovers low-budget and high-influence targets from which efficient cascading attacks can happen. We evaluate MAC against single-step baselines on the MBACC problem with multiple underlying networks and GNN models. Our experiments show the superiority of MAC which is able to discover efficient multi-hop attacks for adversarial community canvassing. Our code implementation and data is available at https://github.com/saurabhsharma1993/mac.

Title: Finding the Missing Data: A BERT-inspired Approach Against Package Loss in Wireless Sensing

• Authors: Zijian Zhao, Tingwei Chen, Fanyi Meng, Hang Li, Xiaoyang Li, Guangxu Zhu
• Subjects: cs.LG, cs.AI, eess.SP
• Abstract URL: https://arxiv.org/abs/2403.12400
• Pdf URL: https://arxiv.org/pdf/2403.12400
• Copy Paste: [[2403.12400]] Finding the Missing Data: A BERT-inspired Approach Against Package Loss in Wireless Sensing(https://arxiv.org/abs/2403.12400)
• Keywords: self-supervised
• Abstract: Despite the development of various deep learning methods for Wi-Fi sensing, package loss often results in noncontinuous estimation of the Channel State Information (CSI), which negatively impacts the performance of the learning models. To overcome this challenge, we propose a deep learning model based on Bidirectional Encoder Representations from Transformers (BERT) for CSI recovery, named CSI-BERT. CSI-BERT can be trained in an self-supervised manner on the target dataset without the need for additional data. Furthermore, unlike traditional interpolation methods that focus on one subcarrier at a time, CSI-BERT captures the sequential relationships across different subcarriers. Experimental results demonstrate that CSI-BERT achieves lower error rates and faster speed compared to traditional interpolation methods, even when facing with high loss rates. Moreover, by harnessing the recovered CSI obtained from CSI-BERT, other deep learning models like Residual Network and Recurrent Neural Network can achieve an average increase in accuracy of approximately 15\% in Wi-Fi sensing tasks. The collected dataset WiGesture and code for our model are publicly available at https://github.com/RS2002/CSI-BERT.

Title: An Empirical Study of Speech Language Models for Prompt-Conditioned Speech Synthesis

• Authors: Yifan Peng, Ilia Kulikov, Yilin Yang, Sravya Popuri, Hui Lu, Changhan Wang, Hongyu Gong
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2403.12402
• Pdf URL: https://arxiv.org/pdf/2403.12402
• Copy Paste: [[2403.12402]] An Empirical Study of Speech Language Models for Prompt-Conditioned Speech Synthesis(https://arxiv.org/abs/2403.12402)
• Keywords: in-context
• Abstract: Speech language models (LMs) are promising for high-quality speech synthesis through in-context learning. A typical speech LM takes discrete semantic units as content and a short utterance as prompt, and synthesizes speech which preserves the content's semantics but mimics the prompt's style. However, there is no systematic understanding on how the synthesized audio is controlled by the prompt and content. In this work, we conduct an empirical study of the widely used autoregressive (AR) and non-autoregressive (NAR) speech LMs and provide insights into the prompt design and content semantic units. Our analysis reveals that heterogeneous and nonstationary prompts hurt the audio quality in contrast to the previous finding that longer prompts always lead to better synthesis. Moreover, we find that the speaker style of the synthesized audio is also affected by the content in addition to the prompt. We further show that semantic units carry rich acoustic information such as pitch, tempo, volume and speech emphasis, which might be leaked from the content to the synthesized audio.

Title: Understanding Training-free Diffusion Guidance: Mechanisms and Limitations

• Authors: Yifei Shen, Xinyang Jiang, Yezhen Wang, Yifan Yang, Dongqi Han, Dongsheng Li
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12404
• Pdf URL: https://arxiv.org/pdf/2403.12404
• Copy Paste: [[2403.12404]] Understanding Training-free Diffusion Guidance: Mechanisms and Limitations(https://arxiv.org/abs/2403.12404)
• Keywords: diffusion
• Abstract: Adding additional control to pretrained diffusion models has become an increasingly popular research area, with extensive applications in computer vision, reinforcement learning, and AI for science. Recently, several studies have proposed training-free diffusion guidance by using off-the-shelf networks pretrained on clean images. This approach enables zero-shot conditional generation for universal control formats, which appears to offer a free lunch in diffusion guidance. In this paper, we aim to develop a deeper understanding of the operational mechanisms and fundamental limitations of training-free guidance. We offer a theoretical analysis that supports training-free guidance from the perspective of optimization, distinguishing it from classifier-based (or classifier-free) guidance. To elucidate their drawbacks, we theoretically demonstrate that training-free methods are more susceptible to adversarial gradients and exhibit slower convergence rates compared to classifier guidance. We then introduce a collection of techniques designed to overcome the limitations, accompanied by theoretical rationale and empirical evidence. Our experiments in image and motion generation confirm the efficacy of these techniques.

Title: ComboVerse: Compositional 3D Assets Creation Using Spatially-Aware Diffusion Guidance

• Authors: Yongwei Chen, Tengfei Wang, Tong Wu, Xingang Pan, Kui Jia, Ziwei Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12409
• Pdf URL: https://arxiv.org/pdf/2403.12409
• Copy Paste: [[2403.12409]] ComboVerse: Compositional 3D Assets Creation Using Spatially-Aware Diffusion Guidance(https://arxiv.org/abs/2403.12409)
• Keywords: diffusion
• Abstract: Generating high-quality 3D assets from a given image is highly desirable in various applications such as AR/VR. Recent advances in single-image 3D generation explore feed-forward models that learn to infer the 3D model of an object without optimization. Though promising results have been achieved in single object generation, these methods often struggle to model complex 3D assets that inherently contain multiple objects. In this work, we present ComboVerse, a 3D generation framework that produces high-quality 3D assets with complex compositions by learning to combine multiple models. 1) We first perform an in-depth analysis of this ``multi-object gap'' from both model and data perspectives. 2) Next, with reconstructed 3D models of different objects, we seek to adjust their sizes, rotation angles, and locations to create a 3D asset that matches the given image. 3) To automate this process, we apply spatially-aware score distillation sampling (SSDS) from pretrained diffusion models to guide the positioning of objects. Our proposed framework emphasizes spatial alignment of objects, compared with standard score distillation sampling, and thus achieves more accurate results. Extensive experiments validate ComboVerse achieves clear improvements over existing methods in generating compositional 3D assets.

Title: VisionGPT: LLM-Assisted Real-Time Anomaly Detection for Safe Visual Navigation

• Authors: Hao Wang, Jiayou Qin, Ashish Bastola, Xiwen Chen, John Suchanek, Zihao Gong, Abolfazl Razi
• Subjects: cs.CV, cs.HC
• Abstract URL: https://arxiv.org/abs/2403.12415
• Pdf URL: https://arxiv.org/pdf/2403.12415
• Copy Paste: [[2403.12415]] VisionGPT: LLM-Assisted Real-Time Anomaly Detection for Safe Visual Navigation(https://arxiv.org/abs/2403.12415)
• Keywords: anomaly
• Abstract: This paper explores the potential of Large Language Models(LLMs) in zero-shot anomaly detection for safe visual navigation. With the assistance of the state-of-the-art real-time open-world object detection model Yolo-World and specialized prompts, the proposed framework can identify anomalies within camera-captured frames that include any possible obstacles, then generate concise, audio-delivered descriptions emphasizing abnormalities, assist in safe visual navigation in complex circumstances. Moreover, our proposed framework leverages the advantages of LLMs and the open-vocabulary object detection model to achieve the dynamic scenario switch, which allows users to transition smoothly from scene to scene, which addresses the limitation of traditional visual navigation. Furthermore, this paper explored the performance contribution of different prompt components, provided the vision for future improvement in visual accessibility, and paved the way for LLMs in video anomaly detection and vision-language understanding.

Title: Precise-Physics Driven Text-to-3D Generation

• Authors: Qingshan Xu, Jiao Liu, Melvin Wong, Caishun Chen, Yew-Soon Ong
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12438
• Pdf URL: https://arxiv.org/pdf/2403.12438
• Copy Paste: [[2403.12438]] Precise-Physics Driven Text-to-3D Generation(https://arxiv.org/abs/2403.12438)
• Keywords: diffusion, generative
• Abstract: Text-to-3D generation has shown great promise in generating novel 3D content based on given text prompts. However, existing generative methods mostly focus on geometric or visual plausibility while ignoring precise physics perception for the generated 3D shapes. This greatly hinders the practicality of generated 3D shapes in real-world applications. In this work, we propose Phy3DGen, a precise-physics-driven text-to-3D generation method. By analyzing the solid mechanics of generated 3D shapes, we reveal that the 3D shapes generated by existing text-to-3D generation methods are impractical for real-world applications as the generated 3D shapes do not conform to the laws of physics. To this end, we leverage 3D diffusion models to provide 3D shape priors and design a data-driven differentiable physics layer to optimize 3D shape priors with solid mechanics. This allows us to optimize geometry efficiently and learn precise physics information about 3D shapes at the same time. Experimental results demonstrate that our method can consider both geometric plausibility and precise physics perception, further bridging 3D virtual modeling and precise physical worlds.

Title: Self-learning Canonical Space for Multi-view 3D Human Pose Estimation

• Authors: Xiaoben Li, Mancheng Meng, Ziyan Wu, Terrence Chen, Fan Yang, Dinggang Shen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12440
• Pdf URL: https://arxiv.org/pdf/2403.12440
• Copy Paste: [[2403.12440]] Self-learning Canonical Space for Multi-view 3D Human Pose Estimation(https://arxiv.org/abs/2403.12440)
• Keywords: self-supervised
• Abstract: Multi-view 3D human pose estimation is naturally superior to single view one, benefiting from more comprehensive information provided by images of multiple views. The information includes camera poses, 2D/3D human poses, and 3D geometry. However, the accurate annotation of these information is hard to obtain, making it challenging to predict accurate 3D human pose from multi-view images. To deal with this issue, we propose a fully self-supervised framework, named cascaded multi-view aggregating network (CMANet), to construct a canonical parameter space to holistically integrate and exploit multi-view information. In our framework, the multi-view information is grouped into two categories: 1) intra-view information , 2) inter-view information. Accordingly, CMANet consists of two components: intra-view module (IRV) and inter-view module (IEV). IRV is used for extracting initial camera pose and 3D human pose of each view; IEV is to fuse complementary pose information and cross-view 3D geometry for a final 3D human pose. To facilitate the aggregation of the intra- and inter-view, we define a canonical parameter space, depicted by per-view camera pose and human pose and shape parameters ($\theta$ and $\beta$) of SMPL model, and propose a two-stage learning procedure. At first stage, IRV learns to estimate camera pose and view-dependent 3D human pose supervised by confident output of an off-the-shelf 2D keypoint detector. At second stage, IRV is frozen and IEV further refines the camera pose and optimizes the 3D human pose by implicitly encoding the cross-view complement and 3D geometry constraint, achieved by jointly fitting predicted multi-view 2D keypoints. The proposed framework, modules, and learning strategy are demonstrated to be effective by comprehensive experiments and CMANet is superior to state-of-the-art methods in extensive quantitative and qualitative analysis.

Title: Do Generated Data Always Help Contrastive Learning?

• Authors: Yifei Wang, Jizhe Zhang, Yisen Wang
• Subjects: cs.LG, cs.AI, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.12448
• Pdf URL: https://arxiv.org/pdf/2403.12448
• Copy Paste: [[2403.12448]] Do Generated Data Always Help Contrastive Learning?(https://arxiv.org/abs/2403.12448)
• Keywords: diffusion, generative
• Abstract: Contrastive Learning (CL) has emerged as one of the most successful paradigms for unsupervised visual representation learning, yet it often depends on intensive manual data augmentations. With the rise of generative models, especially diffusion models, the ability to generate realistic images close to the real data distribution has been well recognized. These generated high-equality images have been successfully applied to enhance contrastive representation learning, a technique termed ``data inflation''. However, we find that the generated data (even from a good diffusion model like DDPM) may sometimes even harm contrastive learning. We investigate the causes behind this failure from the perspective of both data inflation and data augmentation. For the first time, we reveal the complementary roles that stronger data inflation should be accompanied by weaker augmentations, and vice versa. We also provide rigorous theoretical explanations for these phenomena via deriving its generalization bounds under data inflation. Drawing from these insights, we propose Adaptive Inflation (AdaInf), a purely data-centric strategy without introducing any extra computation cost. On benchmark datasets, AdaInf can bring significant improvements for various contrastive learning methods. Notably, without using external data, AdaInf obtains 94.70% linear accuracy on CIFAR-10 with SimCLR, setting a new record that surpasses many sophisticated methods. Code is available at https://github.com/PKU-ML/adainf.

Title: CrossTune: Black-Box Few-Shot Classification with Label Enhancement

• Authors: Danqing Luo, Chen Zhang, Yan Zhang, Haizhou Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.12468
• Pdf URL: https://arxiv.org/pdf/2403.12468
• Copy Paste: [[2403.12468]] CrossTune: Black-Box Few-Shot Classification with Label Enhancement(https://arxiv.org/abs/2403.12468)
• Keywords: in-context
• Abstract: Training or finetuning large-scale language models (LLMs) requires substantial computation resources, motivating recent efforts to explore parameter-efficient adaptation to downstream tasks. One approach is to treat these models as black boxes and use forward passes (Inference APIs) to interact with them. Current research focuses on adapting these black-box models to downstream tasks using gradient-free prompt optimization, but this often involves an expensive process of searching task-specific prompts. Therefore, we are motivated to study black-box language model adaptation without prompt search. Specifically, we introduce a label-enhanced cross-attention network called CrossTune, which models the semantic relatedness between the input text sequence and task-specific label descriptions. Its effectiveness is examined in the context of few-shot text classification. To improve the generalization of CrossTune, we utilize ChatGPT to generate additional training data through in-context learning. A switch mechanism is implemented to exclude low-quality ChatGPT-generated data. Through extensive experiments on seven benchmark text classification datasets, we demonstrate that our proposed approach outperforms the previous state-of-the-art gradient-free black-box tuning method by 5.7% on average. Even without using ChatGPT-augmented data, CrossTune performs better or comparably than previous black-box tuning methods, suggesting the effectiveness of our approach.

Title: SC-Diff: 3D Shape Completion with Latent Diffusion Models

• Authors: Juan D. Galvis, Xingxing Zuo, Simon Schaefer, Stefan Leutengger
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12470
• Pdf URL: https://arxiv.org/pdf/2403.12470
• Copy Paste: [[2403.12470]] SC-Diff: 3D Shape Completion with Latent Diffusion Models(https://arxiv.org/abs/2403.12470)
• Keywords: diffusion
• Abstract: This paper introduces a 3D shape completion approach using a 3D latent diffusion model optimized for completing shapes, represented as Truncated Signed Distance Functions (TSDFs), from partial 3D scans. Our method combines image-based conditioning through cross-attention and spatial conditioning through the integration of 3D features from captured partial scans. This dual guidance enables high-fidelity, realistic shape completions at superior resolutions. At the core of our approach is the compression of 3D data into a low-dimensional latent space using an auto-encoder inspired by 2D latent diffusion models. This compression facilitates the processing of higher-resolution shapes and allows us to apply our model across multiple object classes, a significant improvement over other existing diffusion-based shape completion methods, which often require a separate diffusion model for each class. We validated our approach against two common benchmarks in the field of shape completion, demonstrating competitive performance in terms of accuracy and realism and performing on par with state-of-the-art methods despite operating at a higher resolution with a single model for all object classes. We present a comprehensive evaluation of our model, showcasing its efficacy in handling diverse shape completion challenges, even on unseen object classes. The code will be released upon acceptance.

Title: NTK-Guided Few-Shot Class Incremental Learning

• Authors: Jingren Liu, Zhong Ji, Yanwei Pang, YunLong Yu
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12486
• Pdf URL: https://arxiv.org/pdf/2403.12486
• Copy Paste: [[2403.12486]] NTK-Guided Few-Shot Class Incremental Learning(https://arxiv.org/abs/2403.12486)
• Keywords: self-supervised
• Abstract: While anti-amnesia FSCIL learners often excel in incremental sessions, they tend to prioritize mitigating knowledge attrition over harnessing the model's potential for knowledge acquisition. In this paper, we delve into the foundations of model generalization in FSCIL through the lens of the Neural Tangent Kernel (NTK). Our primary design focus revolves around ensuring optimal NTK convergence and NTK-related generalization error, serving as the theoretical bedrock for exceptional generalization. To attain globally optimal NTK convergence, we employ a meta-learning mechanism grounded in mathematical principles to guide the optimization process within an expanded network. Furthermore, to reduce the NTK-related generalization error, we commence from the foundational level, optimizing the relevant factors constituting its generalization loss. Specifically, we initiate self-supervised pre-training on the base session to shape the initial network weights. Then they are carefully refined through curricular alignment, followed by the application of dual NTK regularization tailored specifically for both convolutional and linear layers. Through the combined effects of these measures, our network acquires robust NTK properties, significantly enhancing its foundational generalization. On popular FSCIL benchmark datasets, our NTK-FSCIL surpasses contemporary state-of-the-art approaches, elevating end-session accuracy by 2.9% to 8.7%.

Title: Task-Customized Mixture of Adapters for General Image Fusion

• Authors: Pengfei Zhu, Yang Sun, Bing Cao, Qinghua Hu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12494
• Pdf URL: https://arxiv.org/pdf/2403.12494
• Copy Paste: [[2403.12494]] Task-Customized Mixture of Adapters for General Image Fusion(https://arxiv.org/abs/2403.12494)
• Keywords: foundation model
• Abstract: General image fusion aims at integrating important information from multi-source images. However, due to the significant cross-task gap, the respective fusion mechanism varies considerably in practice, resulting in limited performance across subtasks. To handle this problem, we propose a novel task-customized mixture of adapters (TC-MoA) for general image fusion, adaptively prompting various fusion tasks in a unified model. We borrow the insight from the mixture of experts (MoE), taking the experts as efficient tuning adapters to prompt a pre-trained foundation model. These adapters are shared across different tasks and constrained by mutual information regularization, ensuring compatibility with different tasks while complementarity for multi-source images. The task-specific routing networks customize these adapters to extract task-specific information from different sources with dynamic dominant intensity, performing adaptive visual feature prompt fusion. Notably, our TC-MoA controls the dominant intensity bias for different fusion tasks, successfully unifying multiple fusion tasks in a single model. Extensive experiments show that TC-MoA outperforms the competing approaches in learning commonalities while retaining compatibility for general image fusion (multi-modal, multi-exposure, and multi-focus), and also demonstrating striking controllability on more generalization experiments. The code is available at https://github.com/YangSun22/TC-MoA .

Title: Generalized Consistency Trajectory Models for Image Manipulation

• Authors: Beomsu Kim, Jaemin Kim, Jeongsol Kim, Jong Chul Ye
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12510
• Pdf URL: https://arxiv.org/pdf/2403.12510
• Copy Paste: [[2403.12510]] Generalized Consistency Trajectory Models for Image Manipulation(https://arxiv.org/abs/2403.12510)
• Keywords: diffusion, generative
• Abstract: Diffusion-based generative models excel in unconditional generation, as well as on applied tasks such as image editing and restoration. The success of diffusion models lies in the iterative nature of diffusion: diffusion breaks down the complex process of mapping noise to data into a sequence of simple denoising tasks. Moreover, we are able to exert fine-grained control over the generation process by injecting guidance terms into each denoising step. However, the iterative process is also computationally intensive, often taking from tens up to thousands of function evaluations. Although consistency trajectory models (CTMs) enable traversal between any time points along the probability flow ODE (PFODE) and score inference with a single function evaluation, CTMs only allow translation from Gaussian noise to data. Thus, this work aims to unlock the full potential of CTMs by proposing generalized CTMs (GCTMs), which translate between arbitrary distributions via ODEs. We discuss the design space of GCTMs and demonstrate their efficacy in various image manipulation tasks such as image-to-image translation, restoration, and editing. Code: \url{https://github.com/1202kbs/GCTM}

Title: Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEs

• Authors: Md Ashiqur Rahman, Robert Joseph George, Mogab Elleithy, Daniel Leibovici, Zongyi Li, Boris Bonev, Colin White, Julius Berner, Raymond A. Yeh, Jean Kossaifi, Kamyar Azizzadenesheli, Anima Anandkumar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12553
• Pdf URL: https://arxiv.org/pdf/2403.12553
• Copy Paste: [[2403.12553]] Pretraining Codomain Attention Neural Operators for Solving Multiphysics PDEs(https://arxiv.org/abs/2403.12553)
• Keywords: self-supervised
• Abstract: Existing neural operator architectures face challenges when solving multiphysics problems with coupled partial differential equations (PDEs), due to complex geometries, interactions between physical variables, and the lack of large amounts of high-resolution training data. To address these issues, we propose Codomain Attention Neural Operator (CoDA-NO), which tokenizes functions along the codomain or channel space, enabling self-supervised learning or pretraining of multiple PDE systems. Specifically, we extend positional encoding, self-attention, and normalization layers to the function space. CoDA-NO can learn representations of different PDE systems with a single model. We evaluate CoDA-NO's potential as a backbone for learning multiphysics PDEs over multiple systems by considering few-shot learning settings. On complex downstream tasks with limited data, such as fluid flow simulations and fluid-structure interactions, we found CoDA-NO to outperform existing methods on the few-shot learning task by over $36\%$. The code is available at https://github.com/ashiq24/CoDA-NO.

Title: Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images

• Authors: Chaoqin Huang, Aofan Jiang, Jinghao Feng, Ya Zhang, Xinchao Wang, Yanfeng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12570
• Pdf URL: https://arxiv.org/pdf/2403.12570
• Copy Paste: [[2403.12570]] Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images(https://arxiv.org/abs/2403.12570)
• Keywords: anomaly
• Abstract: Recent advancements in large-scale visual-language pre-trained models have led to significant progress in zero-/few-shot anomaly detection within natural image domains. However, the substantial domain divergence between natural and medical images limits the effectiveness of these methodologies in medical anomaly detection. This paper introduces a novel lightweight multi-level adaptation and comparison framework to repurpose the CLIP model for medical anomaly detection. Our approach integrates multiple residual adapters into the pre-trained visual encoder, enabling a stepwise enhancement of visual features across different levels. This multi-level adaptation is guided by multi-level, pixel-wise visual-language feature alignment loss functions, which recalibrate the model's focus from object semantics in natural imagery to anomaly identification in medical images. The adapted features exhibit improved generalization across various medical data types, even in zero-shot scenarios where the model encounters unseen medical modalities and anatomical regions during training. Our experiments on medical anomaly detection benchmarks demonstrate that our method significantly surpasses current state-of-the-art models, with an average AUC improvement of 6.24% and 7.33% for anomaly classification, 2.03% and 2.37% for anomaly segmentation, under the zero-shot and few-shot settings, respectively. Source code is available at: https://github.com/MediaBrain-SJTU/MVFA-AD

Title: Real-IAD: A Real-World Multi-View Dataset for Benchmarking Versatile Industrial Anomaly Detection

• Authors: Chengjie Wang, Wenbing Zhu, Bin-Bin Gao, Zhenye Gan, Jianning Zhang, Zhihao Gu, Shuguang Qian, Mingang Chen, Lizhuang Ma
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12580
• Pdf URL: https://arxiv.org/pdf/2403.12580
• Copy Paste: [[2403.12580]] Real-IAD: A Real-World Multi-View Dataset for Benchmarking Versatile Industrial Anomaly Detection(https://arxiv.org/abs/2403.12580)
• Keywords: anomaly
• Abstract: Industrial anomaly detection (IAD) has garnered significant attention and experienced rapid development. However, the recent development of IAD approach has encountered certain difficulties due to dataset limitations. On the one hand, most of the state-of-the-art methods have achieved saturation (over 99% in AUROC) on mainstream datasets such as MVTec, and the differences of methods cannot be well distinguished, leading to a significant gap between public datasets and actual application scenarios. On the other hand, the research on various new practical anomaly detection settings is limited by the scale of the dataset, posing a risk of overfitting in evaluation results. Therefore, we propose a large-scale, Real-world, and multi-view Industrial Anomaly Detection dataset, named Real-IAD, which contains 150K high-resolution images of 30 different objects, an order of magnitude larger than existing datasets. It has a larger range of defect area and ratio proportions, making it more challenging than previous datasets. To make the dataset closer to real application scenarios, we adopted a multi-view shooting method and proposed sample-level evaluation metrics. In addition, beyond the general unsupervised anomaly detection setting, we propose a new setting for Fully Unsupervised Industrial Anomaly Detection (FUIAD) based on the observation that the yield rate in industrial production is usually greater than 60%, which has more practical application value. Finally, we report the results of popular IAD methods on the Real-IAD dataset, providing a highly challenging benchmark to promote the development of the IAD field.

Title: LASPA: Latent Spatial Alignment for Fast Training-free Single Image Editing

• Authors: Yazeed Alharbi, Peter Wonka
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12585
• Pdf URL: https://arxiv.org/pdf/2403.12585
• Copy Paste: [[2403.12585]] LASPA: Latent Spatial Alignment for Fast Training-free Single Image Editing(https://arxiv.org/abs/2403.12585)
• Keywords: diffusion
• Abstract: We present a novel, training-free approach for textual editing of real images using diffusion models. Unlike prior methods that rely on computationally expensive finetuning, our approach leverages LAtent SPatial Alignment (LASPA) to efficiently preserve image details. We demonstrate how the diffusion process is amenable to spatial guidance using a reference image, leading to semantically coherent edits. This eliminates the need for complex optimization and costly model finetuning, resulting in significantly faster editing compared to previous methods. Additionally, our method avoids the storage requirements associated with large finetuned models. These advantages make our approach particularly well-suited for editing on mobile devices and applications demanding rapid response times. While simple and fast, our method achieves 62-71\% preference in a user-study and significantly better model-based editing strength and image preservation scores.

Title: A Practical Guide to Statistical Distances for Evaluating Generative Models in Science

• Authors: Sebastian Bischoff, Alana Darcher, Michael Deistler, Richard Gao, Franziska Gerken, Manuel Gloeckler, Lisa Haxel, Jaivardhan Kapoor, Janne K Lappalainen, Jakob H Macke, Guy Moss, Matthijs Pals, Felix Pei, Rachel Rapp, A Erdem Sağtekin, Cornelius Schröder, Auguste Schulz, Zinovia Stefanidi, Shoji Toyota, Linda Ulmer, Julius Vetter
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.12636
• Pdf URL: https://arxiv.org/pdf/2403.12636
• Copy Paste: [[2403.12636]] A Practical Guide to Statistical Distances for Evaluating Generative Models in Science(https://arxiv.org/abs/2403.12636)
• Keywords: generative
• Abstract: Generative models are invaluable in many fields of science because of their ability to capture high-dimensional and complicated distributions, such as photo-realistic images, protein structures, and connectomes. How do we evaluate the samples these models generate? This work aims to provide an accessible entry point to understanding popular notions of statistical distances, requiring only foundational knowledge in mathematics and statistics. We focus on four commonly used notions of statistical distances representing different methodologies: Using low-dimensional projections (Sliced-Wasserstein; SW), obtaining a distance using classifiers (Classifier Two-Sample Tests; C2ST), using embeddings through kernels (Maximum Mean Discrepancy; MMD), or neural networks (Fr\'echet Inception Distance; FID). We highlight the intuition behind each distance and explain their merits, scalability, complexity, and pitfalls. To demonstrate how these distances are used in practice, we evaluate generative models from different scientific domains, namely a model of decision making and a model generating medical images. We showcase that distinct distances can give different results on similar data. Through this guide, we aim to help researchers to use, interpret, and evaluate statistical distances for generative models in science.

Title: Tuning-Free Image Customization with Image and Text Guidance

• Authors: Pengzhi Li, Qiang Nie, Ying Chen, Xi Jiang, Kai Wu, Yuhuan Lin, Yong Liu, Jinlong Peng, Chengjie Wang, Feng Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12658
• Pdf URL: https://arxiv.org/pdf/2403.12658
• Copy Paste: [[2403.12658]] Tuning-Free Image Customization with Image and Text Guidance(https://arxiv.org/abs/2403.12658)
• Keywords: diffusion
• Abstract: Despite significant advancements in image customization with diffusion models, current methods still have several limitations: 1) unintended changes in non-target areas when regenerating the entire image; 2) guidance solely by a reference image or text descriptions; and 3) time-consuming fine-tuning, which limits their practical application. In response, we introduce a tuning-free framework for simultaneous text-image-guided image customization, enabling precise editing of specific image regions within seconds. Our approach preserves the semantic features of the reference image subject while allowing modification of detailed attributes based on text descriptions. To achieve this, we propose an innovative attention blending strategy that blends self-attention features in the UNet decoder during the denoising process. To our knowledge, this is the first tuning-free method that concurrently utilizes text and image guidance for image customization in specific regions. Our approach outperforms previous methods in both human and quantitative evaluations, providing an efficient solution for various practical applications, such as image synthesis, design, and creative photography.

Title: Improving Interpretability of Scores in Anomaly Detection Based on Gaussian-Bernoulli Restricted Boltzmann Machine

• Authors: Kaiji Sekimoto, Muneki Yasuda
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.12672
• Pdf URL: https://arxiv.org/pdf/2403.12672
• Copy Paste: [[2403.12672]] Improving Interpretability of Scores in Anomaly Detection Based on Gaussian-Bernoulli Restricted Boltzmann Machine(https://arxiv.org/abs/2403.12672)
• Keywords: anomaly
• Abstract: Gaussian-Bernoulli restricted Boltzmann machines (GBRBMs) are often used for semi-supervised anomaly detection, where they are trained using only normal data points. In GBRBM-based anomaly detection, normal and anomalous data are classified based on a score that is identical to an energy function of the marginal GBRBM. However, the classification threshold is difficult to set to an appropriate value, as this score cannot be interpreted. In this study, we propose a measure that improves score's interpretability based on its cumulative distribution, and establish a guideline for setting the threshold using the interpretable measure. The results of numerical experiments show that the guideline is reasonable when setting the threshold solely using normal data points. Moreover, because identifying the measure involves computationally infeasible evaluation of the minimum score value, we also propose an evaluation method for the minimum score based on simulated annealing, which is widely used for optimization problems. The proposed evaluation method was also validated using numerical experiments.

Title: As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?

• Authors: Anjun Hu, Jindong Gu, Francesco Pinto, Konstantinos Kamnitsas, Philip Torr
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12693
• Pdf URL: https://arxiv.org/pdf/2403.12693
• Copy Paste: [[2403.12693]] As Firm As Their Foundations: Can open-sourced foundation models be used to create adversarial examples for downstream tasks?(https://arxiv.org/abs/2403.12693)
• Keywords: foundation model
• Abstract: Foundation models pre-trained on web-scale vision-language data, such as CLIP, are widely used as cornerstones of powerful machine learning systems. While pre-training offers clear advantages for downstream learning, it also endows downstream models with shared adversarial vulnerabilities that can be easily identified through the open-sourced foundation model. In this work, we expose such vulnerabilities in CLIP's downstream models and show that foundation models can serve as a basis for attacking their downstream systems. In particular, we propose a simple yet effective adversarial attack strategy termed Patch Representation Misalignment (PRM). Solely based on open-sourced CLIP vision encoders, this method produces adversaries that simultaneously fool more than 20 downstream models spanning 4 common vision-language tasks (semantic segmentation, object detection, image captioning and visual question-answering). Our findings highlight the concerning safety risks introduced by the extensive usage of public foundational models in the development of downstream systems, calling for extra caution in these scenarios.

Title: Learning Cross-view Visual Geo-localization without Ground Truth

• Authors: Haoyuan Li, Chang Xu, Wen Yang, Huai Yu, Gui-Song Xia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12702
• Pdf URL: https://arxiv.org/pdf/2403.12702
• Copy Paste: [[2403.12702]] Learning Cross-view Visual Geo-localization without Ground Truth(https://arxiv.org/abs/2403.12702)
• Keywords: self-supervised, foundation model
• Abstract: Cross-View Geo-Localization (CVGL) involves determining the geographical location of a query image by matching it with a corresponding GPS-tagged reference image. Current state-of-the-art methods predominantly rely on training models with labeled paired images, incurring substantial annotation costs and training burdens. In this study, we investigate the adaptation of frozen models for CVGL without requiring ground truth pair labels. We observe that training on unlabeled cross-view images presents significant challenges, including the need to establish relationships within unlabeled data and reconcile view discrepancies between uncertain queries and references. To address these challenges, we propose a self-supervised learning framework to train a learnable adapter for a frozen Foundation Model (FM). This adapter is designed to map feature distributions from diverse views into a uniform space using unlabeled data exclusively. To establish relationships within unlabeled data, we introduce an Expectation-Maximization-based Pseudo-labeling module, which iteratively estimates associations between cross-view features and optimizes the adapter. To maintain the robustness of the FM's representation, we incorporate an information consistency module with a reconstruction loss, ensuring that adapted features retain strong discriminative ability across views. Experimental results demonstrate that our proposed method achieves significant improvements over vanilla FMs and competitive accuracy compared to supervised methods, while necessitating fewer training parameters and relying solely on unlabeled data. Evaluation of our adaptation for task-specific models further highlights its broad applicability.

Title: AnimateDiff-Lightning: Cross-Model Diffusion Distillation

• Authors: Shanchuan Lin, Xiao Yang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12706
• Pdf URL: https://arxiv.org/pdf/2403.12706
• Copy Paste: [[2403.12706]] AnimateDiff-Lightning: Cross-Model Diffusion Distillation(https://arxiv.org/abs/2403.12706)
• Keywords: diffusion
• Abstract: We present AnimateDiff-Lightning for lightning-fast video generation. Our model uses progressive adversarial diffusion distillation to achieve new state-of-the-art in few-step video generation. We discuss our modifications to adapt it for the video modality. Furthermore, we propose to simultaneously distill the probability flow of multiple base diffusion models, resulting in a single distilled motion module with broader style compatibility. We are pleased to release our distilled AnimateDiff-Lightning model for the community's use.

Title: Diffusion-Driven Self-Supervised Learning for Shape Reconstruction and Pose Estimation

• Authors: Jingtao Sun, Yaonan Wang, Mingtao Feng, Chao Ding, Mike Zheng Shou, Ajmal Saeed Mian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12728
• Pdf URL: https://arxiv.org/pdf/2403.12728
• Copy Paste: [[2403.12728]] Diffusion-Driven Self-Supervised Learning for Shape Reconstruction and Pose Estimation(https://arxiv.org/abs/2403.12728)
• Keywords: diffusion, self-supervised
• Abstract: Fully-supervised category-level pose estimation aims to determine the 6-DoF poses of unseen instances from known categories, requiring expensive mannual labeling costs. Recently, various self-supervised category-level pose estimation methods have been proposed to reduce the requirement of the annotated datasets. However, most methods rely on synthetic data or 3D CAD model for self-supervised training, and they are typically limited to addressing single-object pose problems without considering multi-objective tasks or shape reconstruction. To overcome these challenges and limitations, we introduce a diffusion-driven self-supervised network for multi-object shape reconstruction and categorical pose estimation, only leveraging the shape priors. Specifically, to capture the SE(3)-equivariant pose features and 3D scale-invariant shape information, we present a Prior-Aware Pyramid 3D Point Transformer in our network. This module adopts a point convolutional layer with radial-kernels for pose-aware learning and a 3D scale-invariant graph convolution layer for object-level shape representation, respectively. Furthermore, we introduce a pretrain-to-refine self-supervised training paradigm to train our network. It enables proposed network to capture the associations between shape priors and observations, addressing the challenge of intra-class shape variations by utilising the diffusion mechanism. Extensive experiments conducted on four public datasets and a self-built dataset demonstrate that our method significantly outperforms state-of-the-art self-supervised category-level baselines and even surpasses some fully-supervised instance-level and category-level methods.

Title: Towards Multimodal In-Context Learning for Vision & Language Models

• Authors: Sivan Doveh, Shaked Perek, M. Jehanzeb Mirza, Amit Alfassy, Assaf Arbelle, Shimon Ullman, Leonid Karlinsky
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12736
• Pdf URL: https://arxiv.org/pdf/2403.12736
• Copy Paste: [[2403.12736]] Towards Multimodal In-Context Learning for Vision & Language Models(https://arxiv.org/abs/2403.12736)
• Keywords: in-context
• Abstract: Inspired by the emergence of Large Language Models (LLMs) that can truly understand human language, significant progress has been made in aligning other, non-language, modalities to be `understandable' by an LLM, primarily via converting their samples into a sequence of embedded language-like tokens directly fed into the LLM (decoder) input stream. However, so far limited attention has been given to transferring (and evaluating) one of the core LLM capabilities to the emerging VLMs, namely the In-Context Learning (ICL) ability, or in other words to guide VLMs to desired target downstream tasks or output structure using in-context image+text demonstrations. In this work, we dive deeper into analyzing the capabilities of some of the state-of-the-art VLMs to follow ICL instructions, discovering them to be somewhat lacking. We discover that even models that underwent large-scale mixed modality pre-training and were implicitly guided to make use of interleaved image and text information (intended to consume helpful context from multiple images) under-perform when prompted with few-shot (ICL) demonstrations, likely due to their lack of `direct' ICL instruction tuning. To test this conjecture, we propose a simple, yet surprisingly effective, strategy of extending a common VLM alignment framework with ICL support, methodology, and curriculum. We explore, analyze, and provide insights into effective data mixes, leading up to a significant 21.03% (and 11.3% on average) ICL performance boost over the strongest VLM baselines and a variety of ICL benchmarks. We also contribute new benchmarks for ICL evaluation in VLMs and discuss their advantages over the prior art.

Title: Towards Controllable Face Generation with Semantic Latent Diffusion Models

• Authors: Alex Ergasti, Claudio Ferrari, Tomaso Fontanini, Massimo Bertozzi, Andrea Prati
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12743
• Pdf URL: https://arxiv.org/pdf/2403.12743
• Copy Paste: [[2403.12743]] Towards Controllable Face Generation with Semantic Latent Diffusion Models(https://arxiv.org/abs/2403.12743)
• Keywords: diffusion
• Abstract: Semantic Image Synthesis (SIS) is among the most popular and effective techniques in the field of face generation and editing, thanks to its good generation quality and the versatility is brings along. Recent works attempted to go beyond the standard GAN-based framework, and started to explore Diffusion Models (DMs) for this task as these stand out with respect to GANs in terms of both quality and diversity. On the other hand, DMs lack in fine-grained controllability and reproducibility. To address that, in this paper we propose a SIS framework based on a novel Latent Diffusion Model architecture for human face generation and editing that is both able to reproduce and manipulate a real reference image and generate diversity-driven results. The proposed system utilizes both SPADE normalization and cross-attention layers to merge shape and style information and, by doing so, allows for a precise control over each of the semantic parts of the human face. This was not possible with previous methods in the state of the art. Finally, we performed an extensive set of experiments to prove that our model surpasses current state of the art, both qualitatively and quantitatively.

Title: WaveFace: Authentic Face Restoration with Efficient Frequency Recovery

• Authors: Yunqi Miao, Jiankang Deng, Jungong Han
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12760
• Pdf URL: https://arxiv.org/pdf/2403.12760
• Copy Paste: [[2403.12760]] WaveFace: Authentic Face Restoration with Efficient Frequency Recovery(https://arxiv.org/abs/2403.12760)
• Keywords: diffusion
• Abstract: Although diffusion models are rising as a powerful solution for blind face restoration, they are criticized for two problems: 1) slow training and inference speed, and 2) failure in preserving identity and recovering fine-grained facial details. In this work, we propose WaveFace to solve the problems in the frequency domain, where low- and high-frequency components decomposed by wavelet transformation are considered individually to maximize authenticity as well as efficiency. The diffusion model is applied to recover the low-frequency component only, which presents general information of the original image but 1/16 in size. To preserve the original identity, the generation is conditioned on the low-frequency component of low-quality images at each denoising step. Meanwhile, high-frequency components at multiple decomposition levels are handled by a unified network, which recovers complex facial details in a single step. Evaluations on four benchmark datasets show that: 1) WaveFace outperforms state-of-the-art methods in authenticity, especially in terms of identity preservation, and 2) authentic images are restored with the efficiency 10x faster than existing diffusion model-based BFR methods.

Title: Discover and Mitigate Multiple Biased Subgroups in Image Classifiers

• Authors: Zeliang Zhang, Mingqian Feng, Zhiheng Li, Chenliang Xu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.12777
• Pdf URL: https://arxiv.org/pdf/2403.12777
• Copy Paste: [[2403.12777]] Discover and Mitigate Multiple Biased Subgroups in Image Classifiers(https://arxiv.org/abs/2403.12777)
• Keywords: foundation model
• Abstract: Machine learning models can perform well on in-distribution data but often fail on biased subgroups that are underrepresented in the training data, hindering the robustness of models for reliable applications. Such subgroups are typically unknown due to the absence of subgroup labels. Discovering biased subgroups is the key to understanding models' failure modes and further improving models' robustness. Most previous works of subgroup discovery make an implicit assumption that models only underperform on a single biased subgroup, which does not hold on in-the-wild data where multiple biased subgroups exist. In this work, we propose Decomposition, Interpretation, and Mitigation (DIM), a novel method to address a more challenging but also more practical problem of discovering multiple biased subgroups in image classifiers. Our approach decomposes the image features into multiple components that represent multiple subgroups. This decomposition is achieved via a bilinear dimension reduction method, Partial Least Square (PLS), guided by useful supervision from the image classifier. We further interpret the semantic meaning of each subgroup component by generating natural language descriptions using vision-language foundation models. Finally, DIM mitigates multiple biased subgroups simultaneously via two strategies, including the data- and model-centric strategies. Extensive experiments on CIFAR-100 and Breeds datasets demonstrate the effectiveness of DIM in discovering and mitigating multiple biased subgroups. Furthermore, DIM uncovers the failure modes of the classifier on Hard ImageNet, showcasing its broader applicability to understanding model bias in image classifiers. The code is available at https://github.com/ZhangAIPI/DIM.

Title: ViTGaze: Gaze Following with Interaction Features in Vision Transformers

• Authors: Yuehao Song, Xinggang Wang, Jingfeng Yao, Wenyu Liu, Jinglin Zhang, Xiangmin Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12778
• Pdf URL: https://arxiv.org/pdf/2403.12778
• Copy Paste: [[2403.12778]] ViTGaze: Gaze Following with Interaction Features in Vision Transformers(https://arxiv.org/abs/2403.12778)
• Keywords: self-supervised
• Abstract: Gaze following aims to interpret human-scene interactions by predicting the person's focal point of gaze. Prevailing approaches often use multi-modality inputs, most of which adopt a two-stage framework. Hence their performance highly depends on the previous prediction accuracy. Others use a single-modality approach with complex decoders, increasing network computational load. Inspired by the remarkable success of pre-trained plain Vision Transformers (ViTs), we introduce a novel single-modality gaze following framework, ViTGaze. In contrast to previous methods, ViTGaze creates a brand new gaze following framework based mainly on powerful encoders (dec. param. less than 1%). Our principal insight lies in that the inter-token interactions within self-attention can be transferred to interactions between humans and scenes. Leveraging this presumption, we formulate a framework consisting of a 4D interaction encoder and a 2D spatial guidance module to extract human-scene interaction information from self-attention maps. Furthermore, our investigation reveals that ViT with self-supervised pre-training exhibits an enhanced ability to extract correlated information. A large number of experiments have been conducted to demonstrate the performance of the proposed method. Our method achieves state-of-the-art (SOTA) performance among all single-modality methods (3.4% improvement on AUC, 5.1% improvement on AP) and very comparable performance against multi-modality methods with 59% number of parameters less.

Title: Learning Neural Volumetric Pose Features for Camera Localization

• Authors: Jingyu Lin, Jiaqi Gu, Bojian Wu, Lubin Fan, Renjie Chen, Ligang Liu, Jieping Ye
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12800
• Pdf URL: https://arxiv.org/pdf/2403.12800
• Copy Paste: [[2403.12800]] Learning Neural Volumetric Pose Features for Camera Localization(https://arxiv.org/abs/2403.12800)
• Keywords: self-supervised
• Abstract: We introduce a novel neural volumetric pose feature, termed PoseMap, designed to enhance camera localization by encapsulating the information between images and the associated camera poses. Our framework leverages an Absolute Pose Regression (APR) architecture, together with an augmented NeRF module. This integration not only facilitates the generation of novel views to enrich the training dataset but also enables the learning of effective pose features. Additionally, we extend our architecture for self-supervised online alignment, allowing our method to be used and fine-tuned for unlabelled images within a unified framework. Experiments demonstrate that our method achieves 14.28% and 20.51% performance gain on average in indoor and outdoor benchmark scenes, outperforming existing APR methods with state-of-the-art accuracy.

Title: RelationVLM: Making Large Vision-Language Models Understand Visual Relations

• Authors: Zhipeng Huang, Zhizheng Zhang, Zheng-Jun Zha, Yan Lu, Baining Guo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12801
• Pdf URL: https://arxiv.org/pdf/2403.12801
• Copy Paste: [[2403.12801]] RelationVLM: Making Large Vision-Language Models Understand Visual Relations(https://arxiv.org/abs/2403.12801)
• Keywords: in-context
• Abstract: The development of Large Vision-Language Models (LVLMs) is striving to catch up with the success of Large Language Models (LLMs), yet it faces more challenges to be resolved. Very recent works enable LVLMs to localize object-level visual contents and ground text to them. Nonetheless, current LVLMs still struggle to precisely understand visual relations due to the lack of relevant data. In this work, we present RelationVLM, a large vision-language model capable of comprehending various levels and types of relations whether across multiple images or within a video. Specifically, we devise a multi-stage relation-aware training scheme and a series of corresponding data configuration strategies to bestow RelationVLM with the capabilities of understanding semantic relations, temporal associations and geometric transforms. Extensive case studies and quantitative evaluations show RelationVLM has strong capability in understanding such relations and emerges impressive in-context capability of reasoning from few-shot examples by comparison. This work fosters the advancements of LVLMs by enabling them to support a wider range of downstream applications toward artificial general intelligence.

Title: DreamDA: Generative Data Augmentation with Diffusion Models

• Authors: Yunxiang Fu, Chaoqi Chen, Yu Qiao, Yizhou Yu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12803
• Pdf URL: https://arxiv.org/pdf/2403.12803
• Copy Paste: [[2403.12803]] DreamDA: Generative Data Augmentation with Diffusion Models(https://arxiv.org/abs/2403.12803)
• Keywords: diffusion, generative
• Abstract: The acquisition of large-scale, high-quality data is a resource-intensive and time-consuming endeavor. Compared to conventional Data Augmentation (DA) techniques (e.g. cropping and rotation), exploiting prevailing diffusion models for data generation has received scant attention in classification tasks. Existing generative DA methods either inadequately bridge the domain gap between real-world and synthesized images, or inherently suffer from a lack of diversity. To solve these issues, this paper proposes a new classification-oriented framework DreamDA, which enables data synthesis and label generation by way of diffusion models. DreamDA generates diverse samples that adhere to the original data distribution by considering training images in the original data as seeds and perturbing their reverse diffusion process. In addition, since the labels of the generated data may not align with the labels of their corresponding seed images, we introduce a self-training paradigm for generating pseudo labels and training classifiers using the synthesized data. Extensive experiments across four tasks and five datasets demonstrate consistent improvements over strong baselines, revealing the efficacy of DreamDA in synthesizing high-quality and diverse images with accurate labels. Our code will be available at https://github.com/yunxiangfu2001/DreamDA.

Title: Compositional 3D Scene Synthesis with Scene Graph Guided Layout-Shape Generation

• Authors: Yao Wei, Martin Renqiang Min, George Vosselman, Li Erran Li, Michael Ying Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12848
• Pdf URL: https://arxiv.org/pdf/2403.12848
• Copy Paste: [[2403.12848]] Compositional 3D Scene Synthesis with Scene Graph Guided Layout-Shape Generation(https://arxiv.org/abs/2403.12848)
• Keywords: diffusion, generative
• Abstract: Compositional 3D scene synthesis has diverse applications across a spectrum of industries such as robotics, films, and video games, as it closely mirrors the complexity of real-world multi-object environments. Early works typically employ shape retrieval based frameworks which naturally suffer from limited shape diversity. Recent progresses have been made in shape generation with powerful generative models, such as diffusion models, which increases the shape fidelity. However, these approaches separately treat 3D shape generation and layout generation. The synthesized scenes are usually hampered by layout collision, which implies that the scene-level fidelity is still under-explored. In this paper, we aim at generating realistic and reasonable 3D scenes from scene graph. To enrich the representation capability of the given scene graph inputs, large language model is utilized to explicitly aggregate the global graph features with local relationship features. With a unified graph convolution network (GCN), graph features are extracted from scene graphs updated via joint layout-shape distribution. During scene generation, an IoU-based regularization loss is introduced to constrain the predicted 3D layouts. Benchmarked on the SG-FRONT dataset, our method achieves better 3D scene synthesis, especially in terms of scene-level fidelity. The source code will be released after publication.

Title: A Comparison of Deep Learning Architectures for Spacecraft Anomaly Detection

• Authors: Daniel Lakey, Tim Schlippe
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12864
• Pdf URL: https://arxiv.org/pdf/2403.12864
• Copy Paste: [[2403.12864]] A Comparison of Deep Learning Architectures for Spacecraft Anomaly Detection(https://arxiv.org/abs/2403.12864)
• Keywords: anomaly
• Abstract: Spacecraft operations are highly critical, demanding impeccable reliability and safety. Ensuring the optimal performance of a spacecraft requires the early detection and mitigation of anomalies, which could otherwise result in unit or mission failures. With the advent of deep learning, a surge of interest has been seen in leveraging these sophisticated algorithms for anomaly detection in space operations. This study aims to compare the efficacy of various deep learning architectures in detecting anomalies in spacecraft data. The deep learning models under investigation include Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM) networks, and Transformer-based architectures. Each of these models was trained and validated using a comprehensive dataset sourced from multiple spacecraft missions, encompassing diverse operational scenarios and anomaly types. Initial results indicate that while CNNs excel in identifying spatial patterns and may be effective for some classes of spacecraft data, LSTMs and RNNs show a marked proficiency in capturing temporal anomalies seen in time-series spacecraft telemetry. The Transformer-based architectures, given their ability to focus on both local and global contexts, have showcased promising results, especially in scenarios where anomalies are subtle and span over longer durations. Additionally, considerations such as computational efficiency, ease of deployment, and real-time processing capabilities were evaluated. While CNNs and LSTMs demonstrated a balance between accuracy and computational demands, Transformer architectures, though highly accurate, require significant computational resources. In conclusion, the choice of deep learning architecture for spacecraft anomaly detection is highly contingent on the nature of the data, the type of anomalies, and operational constraints.

Title: Wildfire danger prediction optimization with transfer learning

• Authors: Spiros Maggioros, Nikos Tsalkitzis
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12871
• Pdf URL: https://arxiv.org/pdf/2403.12871
• Copy Paste: [[2403.12871]] Wildfire danger prediction optimization with transfer learning(https://arxiv.org/abs/2403.12871)
• Keywords: anomaly
• Abstract: Convolutional Neural Networks (CNNs) have proven instrumental across various computer science domains, enabling advancements in object detection, classification, and anomaly detection. This paper explores the application of CNNs to analyze geospatial data specifically for identifying wildfire-affected areas. Leveraging transfer learning techniques, we fine-tuned CNN hyperparameters and integrated the Canadian Fire Weather Index (FWI) to assess moisture conditions. The study establishes a methodology for computing wildfire risk levels on a scale of 0 to 5, dynamically linked to weather patterns. Notably, through the integration of transfer learning, the CNN model achieved an impressive accuracy of 95\% in identifying burnt areas. This research sheds light on the inner workings of CNNs and their practical, real-time utility in predicting and mitigating wildfires. By combining transfer learning and CNNs, this study contributes a robust approach to assess burnt areas, facilitating timely interventions and preventative measures against conflagrations.

Title: Ultra-High-Resolution Image Synthesis with Pyramid Diffusion Model

• Authors: Jiajie Yang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12915
• Pdf URL: https://arxiv.org/pdf/2403.12915
• Copy Paste: [[2403.12915]] Ultra-High-Resolution Image Synthesis with Pyramid Diffusion Model(https://arxiv.org/abs/2403.12915)
• Keywords: diffusion, generative
• Abstract: We introduce the Pyramid Diffusion Model (PDM), a novel architecture designed for ultra-high-resolution image synthesis. PDM utilizes a pyramid latent representation, providing a broader design space that enables more flexible, structured, and efficient perceptual compression which enable AutoEncoder and Network of Diffusion to equip branches and deeper layers. To enhance PDM's capabilities for generative tasks, we propose the integration of Spatial-Channel Attention and Res-Skip Connection, along with the utilization of Spectral Norm and Decreasing Dropout Strategy for the Diffusion Network and AutoEncoder. In summary, PDM achieves the synthesis of images with a 2K resolution for the first time, demonstrated on two new datasets comprising images of sizes 2048x2048 pixels and 2048x1024 pixels respectively. We believe that this work offers an alternative approach to designing scalable image generative models, while also providing incremental reinforcement for existing frameworks.

Title: Contextual AD Narration with Interleaved Multimodal Sequence

• Authors: Hanlin Wang, Zhan Tong, Kecheng Zheng, Yujun Shen, Limin Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12922
• Pdf URL: https://arxiv.org/pdf/2403.12922
• Copy Paste: [[2403.12922]] Contextual AD Narration with Interleaved Multimodal Sequence(https://arxiv.org/abs/2403.12922)
• Keywords: foundation model
• Abstract: The Audio Description (AD) task aims to generate descriptions of visual elements for visually impaired individuals to help them access long-form video contents, like movie. With video feature, text, character bank and context information as inputs, the generated ADs are able to correspond to the characters by name and provide reasonable, contextual descriptions to help audience understand the storyline of movie. To achieve this goal, we propose to leverage pre-trained foundation models through a simple and unified framework to generate ADs with interleaved multimodal sequence as input, termed as Uni-AD. To enhance the alignment of features across various modalities with finer granularity, we introduce a simple and lightweight module that maps video features into the textual feature space. Moreover, we also propose a character-refinement module to provide more precise information by identifying the main characters who play more significant role in the video context. With these unique designs, we further incorporate contextual information and a contrastive loss into our architecture to generate more smooth and contextual ADs. Experiments on the MAD-eval dataset show that Uni-AD can achieve state-of-the-art performance on AD generation, which demonstrates the effectiveness of our approach. Code will be available at https://github.com/MCG-NJU/Uni-AD.

Title: You Only Sample Once: Taming One-Step Text-To-Image Synthesis by Self-Cooperative Diffusion GANs

• Authors: Yihong Luo, Xiaolong Chen, Jing Tang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12931
• Pdf URL: https://arxiv.org/pdf/2403.12931
• Copy Paste: [[2403.12931]] You Only Sample Once: Taming One-Step Text-To-Image Synthesis by Self-Cooperative Diffusion GANs(https://arxiv.org/abs/2403.12931)
• Keywords: diffusion, generative
• Abstract: We introduce YOSO, a novel generative model designed for rapid, scalable, and high-fidelity one-step image synthesis. This is achieved by integrating the diffusion process with GANs. Specifically, we smooth the distribution by the denoising generator itself, performing self-cooperative learning. We show that our method can serve as a one-step generation model training from scratch with competitive performance. Moreover, we show that our method can be extended to finetune pre-trained text-to-image diffusion for high-quality one-step text-to-image synthesis even with LoRA fine-tuning. In particular, we provide the first diffusion transformer that can generate images in one step trained on 512 resolution, with the capability of adapting to 1024 resolution without explicit training. Our code is provided at https://github.com/Luo-Yihong/YOSO.

Title: Zero-Reference Low-Light Enhancement via Physical Quadruple Priors

• Authors: Wenjing Wang, Huan Yang, Jianlong Fu, Jiaying Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12933
• Pdf URL: https://arxiv.org/pdf/2403.12933
• Copy Paste: [[2403.12933]] Zero-Reference Low-Light Enhancement via Physical Quadruple Priors(https://arxiv.org/abs/2403.12933)
• Keywords: diffusion, generative
• Abstract: Understanding illumination and reducing the need for supervision pose a significant challenge in low-light enhancement. Current approaches are highly sensitive to data usage during training and illumination-specific hyper-parameters, limiting their ability to handle unseen scenarios. In this paper, we propose a new zero-reference low-light enhancement framework trainable solely with normal light images. To accomplish this, we devise an illumination-invariant prior inspired by the theory of physical light transfer. This prior serves as the bridge between normal and low-light images. Then, we develop a prior-to-image framework trained without low-light data. During testing, this framework is able to restore our illumination-invariant prior back to images, automatically achieving low-light enhancement. Within this framework, we leverage a pretrained generative diffusion model for model ability, introduce a bypass decoder to handle detail distortion, as well as offer a lightweight version for practicality. Extensive experiments demonstrate our framework's superiority in various scenarios as well as good interpretability, robustness, and efficiency. Code is available on our project homepage: this http URL

Title: Segment Anything for comprehensive analysis of grapevine cluster architecture and berry properties

• Authors: Efrain Torres-Lomas, Jimena Lado-Jimena, Guillermo Garcia-Zamora, Luis Diaz-Garcia
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12935
• Pdf URL: https://arxiv.org/pdf/2403.12935
• Copy Paste: [[2403.12935]] Segment Anything for comprehensive analysis of grapevine cluster architecture and berry properties(https://arxiv.org/abs/2403.12935)
• Keywords: foundation model
• Abstract: Grape cluster architecture and compactness are complex traits influencing disease susceptibility, fruit quality, and yield. Evaluation methods for these traits include visual scoring, manual methodologies, and computer vision, with the latter being the most scalable approach. Most of the existing computer vision approaches for processing cluster images often rely on conventional segmentation or machine learning with extensive training and limited generalization. The Segment Anything Model (SAM), a novel foundation model trained on a massive image dataset, enables automated object segmentation without additional training. This study demonstrates out-of-the-box SAM's high accuracy in identifying individual berries in 2D cluster images. Using this model, we managed to segment approximately 3,500 cluster images, generating over 150,000 berry masks, each linked with spatial coordinates within their clusters. The correlation between human-identified berries and SAM predictions was very strong (Pearson r2=0.96). Although the visible berry count in images typically underestimates the actual cluster berry count due to visibility issues, we demonstrated that this discrepancy could be adjusted using a linear regression model (adjusted R2=0.87). We emphasized the critical importance of the angle at which the cluster is imaged, noting its substantial effect on berry counts and architecture. We proposed different approaches in which berry location information facilitated the calculation of complex features related to cluster architecture and compactness. Finally, we discussed SAM's potential integration into currently available pipelines for image generation and processing in vineyard conditions.

Title: GVGEN: Text-to-3D Generation with Volumetric Representation

• Authors: Xianglong He, Junyi Chen, Sida Peng, Di Huang, Yangguang Li, Xiaoshui Huang, Chun Yuan, Wanli Ouyang, Tong He
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12957
• Pdf URL: https://arxiv.org/pdf/2403.12957
• Copy Paste: [[2403.12957]] GVGEN: Text-to-3D Generation with Volumetric Representation(https://arxiv.org/abs/2403.12957)
• Keywords: diffusion
• Abstract: In recent years, 3D Gaussian splatting has emerged as a powerful technique for 3D reconstruction and generation, known for its fast and high-quality rendering capabilities. To address these shortcomings, this paper introduces a novel diffusion-based framework, GVGEN, designed to efficiently generate 3D Gaussian representations from text input. We propose two innovative techniques:(1) Structured Volumetric Representation. We first arrange disorganized 3D Gaussian points as a structured form GaussianVolume. This transformation allows the capture of intricate texture details within a volume composed of a fixed number of Gaussians. To better optimize the representation of these details, we propose a unique pruning and densifying method named the Candidate Pool Strategy, enhancing detail fidelity through selective optimization. (2) Coarse-to-fine Generation Pipeline. To simplify the generation of GaussianVolume and empower the model to generate instances with detailed 3D geometry, we propose a coarse-to-fine pipeline. It initially constructs a basic geometric structure, followed by the prediction of complete Gaussian attributes. Our framework, GVGEN, demonstrates superior performance in qualitative and quantitative assessments compared to existing 3D generation methods. Simultaneously, it maintains a fast generation speed ($\sim$7 seconds), effectively striking a balance between quality and efficiency.

Title: TexTile: A Differentiable Metric for Texture Tileability

• Authors: Carlos Rodriguez-Pardo, Dan Casas, Elena Garces, Jorge Lopez-Moreno
• Subjects: cs.CV, cs.AI, cs.GR, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.12961
• Pdf URL: https://arxiv.org/pdf/2403.12961
• Copy Paste: [[2403.12961]] TexTile: A Differentiable Metric for Texture Tileability(https://arxiv.org/abs/2403.12961)
• Keywords: diffusion
• Abstract: We introduce TexTile, a novel differentiable metric to quantify the degree upon which a texture image can be concatenated with itself without introducing repeating artifacts (i.e., the tileability). Existing methods for tileable texture synthesis focus on general texture quality, but lack explicit analysis of the intrinsic repeatability properties of a texture. In contrast, our TexTile metric effectively evaluates the tileable properties of a texture, opening the door to more informed synthesis and analysis of tileable textures. Under the hood, TexTile is formulated as a binary classifier carefully built from a large dataset of textures of different styles, semantics, regularities, and human annotations.Key to our method is a set of architectural modifications to baseline pre-train image classifiers to overcome their shortcomings at measuring tileability, along with a custom data augmentation and training regime aimed at increasing robustness and accuracy. We demonstrate that TexTile can be plugged into different state-of-the-art texture synthesis methods, including diffusion-based strategies, and generate tileable textures while keeping or even improving the overall texture quality. Furthermore, we show that TexTile can objectively evaluate any tileable texture synthesis method, whereas the current mix of existing metrics produces uncorrelated scores which heavily hinders progress in the field.

Title: FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation

• Authors: Shuai Yang, Yifan Zhou, Ziwei Liu, Chen Change Loy
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12962
• Pdf URL: https://arxiv.org/pdf/2403.12962
• Copy Paste: [[2403.12962]] FRESCO: Spatial-Temporal Correspondence for Zero-Shot Video Translation(https://arxiv.org/abs/2403.12962)
• Keywords: diffusion
• Abstract: The remarkable efficacy of text-to-image diffusion models has motivated extensive exploration of their potential application in video domains. Zero-shot methods seek to extend image diffusion models to videos without necessitating model training. Recent methods mainly focus on incorporating inter-frame correspondence into attention mechanisms. However, the soft constraint imposed on determining where to attend to valid features can sometimes be insufficient, resulting in temporal inconsistency. In this paper, we introduce FRESCO, intra-frame correspondence alongside inter-frame correspondence to establish a more robust spatial-temporal constraint. This enhancement ensures a more consistent transformation of semantically similar content across frames. Beyond mere attention guidance, our approach involves an explicit update of features to achieve high spatial-temporal consistency with the input video, significantly improving the visual coherence of the resulting translated videos. Extensive experiments demonstrate the effectiveness of our proposed framework in producing high-quality, coherent videos, marking a notable improvement over existing zero-shot methods.

Title: FouriScale: A Frequency Perspective on Training-Free High-Resolution Image Synthesis

• Authors: Linjiang Huang, Rongyao Fang, Aiping Zhang, Guanglu Song, Si Liu, Yu Liu, Hongsheng Li
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.12963
• Pdf URL: https://arxiv.org/pdf/2403.12963
• Copy Paste: [[2403.12963]] FouriScale: A Frequency Perspective on Training-Free High-Resolution Image Synthesis(https://arxiv.org/abs/2403.12963)
• Keywords: diffusion
• Abstract: In this study, we delve into the generation of high-resolution images from pre-trained diffusion models, addressing persistent challenges, such as repetitive patterns and structural distortions, that emerge when models are applied beyond their trained resolutions. To address this issue, we introduce an innovative, training-free approach FouriScale from the perspective of frequency domain analysis. We replace the original convolutional layers in pre-trained diffusion models by incorporating a dilation technique along with a low-pass operation, intending to achieve structural consistency and scale consistency across resolutions, respectively. Further enhanced by a padding-then-crop strategy, our method can flexibly handle text-to-image generation of various aspect ratios. By using the FouriScale as guidance, our method successfully balances the structural integrity and fidelity of generated images, achieving an astonishing capacity of arbitrary-size, high-resolution, and high-quality generation. With its simplicity and compatibility, our method can provide valuable insights for future explorations into the synthesis of ultra-high-resolution images. The code will be released at https://github.com/LeonHLJ/FouriScale.