2024-03-05

Title: IPED: An Implicit Perspective for Relational Triple Extraction based on Diffusion Model

• Authors: Jianli Zhao, Changhao Xu, Bin Jiang
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00808
• Pdf URL: https://arxiv.org/pdf/2403.00808
• Copy Paste: [[2403.00808]] IPED: An Implicit Perspective for Relational Triple Extraction based on Diffusion Model(https://arxiv.org/abs/2403.00808)
• Keywords: diffusion, generative
• Abstract: Relational triple extraction is a fundamental task in the field of information extraction, and a promising framework based on table filling has recently gained attention as a potential baseline for entity relation extraction. However, inherent shortcomings such as redundant information and incomplete triple recognition remain problematic. To address these challenges, we propose an Implicit Perspective for relational triple Extraction based on Diffusion model (IPED), an innovative approach for extracting relational triples. Our classifier-free solution adopts an implicit strategy using block coverage to complete the tables, avoiding the limitations of explicit tagging methods. Additionally, we introduce a generative model structure, the block-denoising diffusion model, to collaborate with our implicit perspective and effectively circumvent redundant information disruptions. Experimental results on two popular datasets demonstrate that IPED achieves state-of-the-art performance while gaining superior inference speed and low computational complexity. To support future research, we have made our source code publicly available online.

Title: Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes

• Authors: Xiaomeng Hu, Pin-Yu Chen, Tsung-Yi Ho
• Subjects: cs.CR, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.00867
• Pdf URL: https://arxiv.org/pdf/2403.00867
• Copy Paste: [[2403.00867]] Gradient Cuff: Detecting Jailbreak Attacks on Large Language Models by Exploring Refusal Loss Landscapes(https://arxiv.org/abs/2403.00867)
• Keywords: generative
• Abstract: Large Language Models (LLMs) are becoming a prominent generative AI tool, where the user enters a query and the LLM generates an answer. To reduce harm and misuse, efforts have been made to align these LLMs to human values using advanced training techniques such as Reinforcement Learning from Human Feedback (RLHF). However, recent studies have highlighted the vulnerability of LLMs to adversarial jailbreak attempts aiming at subverting the embedded safety guardrails. To address this challenge, this paper defines and investigates the Refusal Loss of LLMs and then proposes a method called Gradient Cuff to detect jailbreak attempts. Gradient Cuff exploits the unique properties observed in the refusal loss landscape, including functional values and its smoothness, to design an effective two-step detection strategy. Experimental results on two aligned LLMs (LLaMA-2-7B-Chat and Vicuna-7B-V1.5) and six types of jailbreak attacks (GCG, AutoDAN, PAIR, TAP, Base64, and LRL) show that Gradient Cuff can significantly improve the LLM's rejection capability for malicious jailbreak queries, while maintaining the model's performance for benign user queries by adjusting the detection threshold.

Title: SoftTiger: A Clinical Foundation Model for Healthcare Workflows

• Authors: Ye Chen, Igor Couto, Wei Cai, Cong Fu, Bruno Dorneles
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00868
• Pdf URL: https://arxiv.org/pdf/2403.00868
• Copy Paste: [[2403.00868]] SoftTiger: A Clinical Foundation Model for Healthcare Workflows(https://arxiv.org/abs/2403.00868)
• Keywords: foundation model
• Abstract: We release and introduce SoftTiger, a clinical large language model (CLaM) designed as a foundation model for healthcare workflows. The narrative and unstructured nature of clinical notes is a major obstacle for healthcare intelligentization. We address a critical problem of structuring clinical notes into clinical data, according to international interoperability standards. We collect and annotate data for three critical subtasks, namely, international patient summary, clinical impression and medical encounter. We then supervised fine-tuned a state-of-the-art LLM using public and credentialed clinical data. The training is orchestrated in a way that the target model can first support basic clinical tasks such as abbreviation expansion and temporal information extraction, and then learn to perform more complex downstream clinical tasks such as impression and encounter summary. Moreover, we address, several modeling challenges in the healthcare context, e.g., extra long context window. Our blind pairwise evaluation shows that SoftTiger outperforms other popular open-source models and GPT-3.5, comparable to Gemini-pro, and only has a mild gap from GPT-4. We believe that LLMs may become a step-stone towards healthcare digitalization and democratization. Therefore, we publicly release SoftTiger models at scales of 13 billion and 70 billion parameters, as well as datasets and code for our innovative scalable evaluation, hopefully, making a significant contribution to the healthcare industry.

Title: Improving Android Malware Detection Through Data Augmentation Using Wasserstein Generative Adversarial Networks

• Authors: Kawana Stalin, Mikias Berhanu Mekoya
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.00890
• Pdf URL: https://arxiv.org/pdf/2403.00890
• Copy Paste: [[2403.00890]] Improving Android Malware Detection Through Data Augmentation Using Wasserstein Generative Adversarial Networks(https://arxiv.org/abs/2403.00890)
• Keywords: generative
• Abstract: Generative Adversarial Networks (GANs) have demonstrated their versatility across various applications, including data augmentation and malware detection. This research explores the effectiveness of utilizing GAN-generated data to train a model for the detection of Android malware. Given the considerable storage requirements of Android applications, the study proposes a method to synthetically represent data using GANs, thereby reducing storage demands. The proposed methodology involves creating image representations of features extracted from an existing dataset. A GAN model is then employed to generate a more extensive dataset consisting of realistic synthetic grayscale images. Subsequently, this synthetic dataset is utilized to train a Convolutional Neural Network (CNN) designed to identify previously unseen Android malware applications. The study includes a comparative analysis of the CNN's performance when trained on real images versus synthetic images generated by the GAN. Furthermore, the research explores variations in performance between the Wasserstein Generative Adversarial Network (WGAN) and the Deep Convolutional Generative Adversarial Network (DCGAN). The investigation extends to studying the impact of image size and malware obfuscation on the classification model's effectiveness. The data augmentation approach implemented in this study resulted in a notable performance enhancement of the classification model, ranging from 1.5% to 7%, depending on the dataset. The achieved F1 score reached 97.5%. Keywords--Generative Adversarial Networks, Android Malware, Data Augmentation, Wasserstein Generative Adversarial Network

Title: G3DR: Generative 3D Reconstruction in ImageNet

• Authors: Pradyumna Reddy, Ismail Elezi, Jiankang Deng
• Subjects: cs.CV, cs.GR
• Abstract URL: https://arxiv.org/abs/2403.00939
• Pdf URL: https://arxiv.org/pdf/2403.00939
• Copy Paste: [[2403.00939]] G3DR: Generative 3D Reconstruction in ImageNet(https://arxiv.org/abs/2403.00939)
• Keywords: generative
• Abstract: We introduce a novel 3D generative method, Generative 3D Reconstruction (G3DR) in ImageNet, capable of generating diverse and high-quality 3D objects from single images, addressing the limitations of existing methods. At the heart of our framework is a novel depth regularization technique that enables the generation of scenes with high-geometric fidelity. G3DR also leverages a pretrained language-vision model, such as CLIP, to enable reconstruction in novel views and improve the visual realism of generations. Additionally, G3DR designs a simple but effective sampling procedure to further improve the quality of generations. G3DR offers diverse and efficient 3D asset generation based on class or text conditioning. Despite its simplicity, G3DR is able to beat state-of-theart methods, improving over them by up to 22% in perceptual metrics and 90% in geometry scores, while needing only half of the training time. Code is available at https://github.com/preddy5/G3DR

Title: BasedAI: A decentralized P2P network for Zero Knowledge Large Language Models (ZK-LLMs)

• Authors: Sean Wellington
• Subjects: cs.CR, cs.IR
• Abstract URL: https://arxiv.org/abs/2403.01008
• Pdf URL: https://arxiv.org/pdf/2403.01008
• Copy Paste: [[2403.01008]] BasedAI: A decentralized P2P network for Zero Knowledge Large Language Models (ZK-LLMs)(https://arxiv.org/abs/2403.01008)
• Keywords: generative
• Abstract: BasedAI is a distributed network of machines which introduces decentralized infrastructure capable of integrating Fully Homomorphic Encryption (FHE) with any large language model (LLM) connected to its network. The proposed framework embeds a default mechanism, called "Cerberus Squeezing", into the mining process which enables the transformation of a standard LLMs into encrypted zero-knowledge LLMs, or "ZK-LLMs", leveraging insights from generative adversarial networks for data privacy. This novel quantization mechanism empowers BasedAI miners to process and respond to prompts derived from User interaction with LLMs without the need for decrypting ei- ther the queries or their corresponding responses. The introduction of Cerberus Squeezing significantly improves performance degradation caused by quantized functions in current FHE-compliant computing environments by proactively optimizing calls between users, miners, and validators.

Title: FaiMA: Feature-aware In-context Learning for Multi-domain Aspect-based Sentiment Analysis

• Authors: Songhua Yang, Xinke Jiang, Hanjie Zhao, Wenxuan Zeng, Hongde Liu, Yuxiang Jia
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01063
• Pdf URL: https://arxiv.org/pdf/2403.01063
• Copy Paste: [[2403.01063]] FaiMA: Feature-aware In-context Learning for Multi-domain Aspect-based Sentiment Analysis(https://arxiv.org/abs/2403.01063)
• Keywords: in-context
• Abstract: Multi-domain aspect-based sentiment analysis (ABSA) seeks to capture fine-grained sentiment across diverse domains. While existing research narrowly focuses on single-domain applications constrained by methodological limitations and data scarcity, the reality is that sentiment naturally traverses multiple domains. Although large language models (LLMs) offer a promising solution for ABSA, it is difficult to integrate effectively with established techniques, including graph-based models and linguistics, because modifying their internal architecture is not easy. To alleviate this problem, we propose a novel framework, Feature-aware In-context Learning for Multi-domain ABSA (FaiMA). The core insight of FaiMA is to utilize in-context learning (ICL) as a feature-aware mechanism that facilitates adaptive learning in multi-domain ABSA tasks. Specifically, we employ a multi-head graph attention network as a text encoder optimized by heuristic rules for linguistic, domain, and sentiment features. Through contrastive learning, we optimize sentence representations by focusing on these diverse features. Additionally, we construct an efficient indexing mechanism, allowing FaiMA to stably retrieve highly relevant examples across multiple dimensions for any given input. To evaluate the efficacy of FaiMA, we build the first multi-domain ABSA benchmark dataset. Extensive experimental results demonstrate that FaiMA achieves significant performance improvements in multiple domains compared to baselines, increasing F1 by 2.07% on average. Source code and data sets are anonymously available at https://github.com/SupritYoung/FaiMA.

Title: LLMCRIT: Teaching Large Language Models to Use Criteria

• Authors: Weizhe Yuan, Pengfei Liu, Matthias Gallé
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01069
• Pdf URL: https://arxiv.org/pdf/2403.01069
• Copy Paste: [[2403.01069]] LLMCRIT: Teaching Large Language Models to Use Criteria(https://arxiv.org/abs/2403.01069)
• Keywords: in-context
• Abstract: Humans follow criteria when they execute tasks, and these criteria are directly used to assess the quality of task completion. Therefore, having models learn to use criteria to provide feedback can help humans or models to perform tasks better. However, existing research in this field tends to consider only a limited set of criteria or quality assessment aspects. To fill this gap, we propose a general framework that enables large language models (LLMs) to use comprehensive criteria for a task in delivering natural language feedback on task execution. In particular, we present a model-in-the-loop framework that semi-automatically derives criteria from collected guidelines for different writing tasks and constructs in-context demonstrations for each criterion. We choose three tasks from real-world scenarios to operationalize this idea: paper introduction writing, Python code writing, and Reddit post writing, and evaluate our feedback generation framework using different LLMs. The results reveal the fine-grained effects of incorporating criteria and demonstrations and provide valuable insights on how to teach LLMs to use criteria more effectively.

Title: $Γ$-VAE: Curvature regularized variational autoencoders for uncovering emergent low dimensional geometric structure in high dimensional data

• Authors: Jason Z. Kim, Nicolas Perrin-Gilbert, Erkan Narmanli, Paul Klein, Christopher R. Myers, Itai Cohen, Joshua J. Waterfall, James P. Sethna
• Subjects: cs.LG, cs.AI, physics.bio-ph, q-bio.GN
• Abstract URL: https://arxiv.org/abs/2403.01078
• Pdf URL: https://arxiv.org/pdf/2403.01078
• Copy Paste: [[2403.01078]] $Γ$-VAE: Curvature regularized variational autoencoders for uncovering emergent low dimensional geometric structure in high dimensional data(https://arxiv.org/abs/2403.01078)
• Keywords: generative
• Abstract: Natural systems with emergent behaviors often organize along low-dimensional subsets of high-dimensional spaces. For example, despite the tens of thousands of genes in the human genome, the principled study of genomics is fruitful because biological processes rely on coordinated organization that results in lower dimensional phenotypes. To uncover this organization, many nonlinear dimensionality reduction techniques have successfully embedded high-dimensional data into low-dimensional spaces by preserving local similarities between data points. However, the nonlinearities in these methods allow for too much curvature to preserve general trends across multiple non-neighboring data clusters, thereby limiting their interpretability and generalizability to out-of-distribution data. Here, we address both of these limitations by regularizing the curvature of manifolds generated by variational autoencoders, a process we coin ``$\Gamma$-VAE''. We demonstrate its utility using two example data sets: bulk RNA-seq from the The Cancer Genome Atlas (TCGA) and the Genotype Tissue Expression (GTEx); and single cell RNA-seq from a lineage tracing experiment in hematopoietic stem cell differentiation. We find that the resulting regularized manifolds identify mesoscale structure associated with different cancer cell types, and accurately re-embed tissues from completely unseen, out-of distribution cancers as if they were originally trained on them. Finally, we show that preserving long-range relationships to differentiated cells separates undifferentiated cells -- which have not yet specialized -- according to their eventual fate. Broadly, we anticipate that regularizing the curvature of generative models will enable more consistent, predictive, and generalizable models in any high-dimensional system with emergent low-dimensional behavior.

Title: Distilling Text Style Transfer With Self-Explanation From LLMs

• Authors: Chiyu Zhang, Honglong Cai, Yuezhang (Music)Li, Yuexin Wu, Le Hou, Muhammad Abdul-Mageed
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01106
• Pdf URL: https://arxiv.org/pdf/2403.01106
• Copy Paste: [[2403.01106]] Distilling Text Style Transfer With Self-Explanation From LLMs(https://arxiv.org/abs/2403.01106)
• Keywords: in-context
• Abstract: Text Style Transfer (TST) seeks to alter the style of text while retaining its core content. Given the constraints of limited parallel datasets for TST, we propose CoTeX, a framework that leverages large language models (LLMs) alongside chain-of-thought (CoT) prompting to facilitate TST. CoTeX distills the complex rewriting and reasoning capabilities of LLMs into more streamlined models capable of working with both non-parallel and parallel data. Through experimentation across four TST datasets, CoTeX is shown to surpass traditional supervised fine-tuning and knowledge distillation methods, particularly in low-resource settings. We conduct a comprehensive evaluation, comparing CoTeX against current unsupervised, supervised, in-context learning (ICL) techniques, and instruction-tuned LLMs. Furthermore, CoTeX distinguishes itself by offering transparent explanations for its style transfer process.

Title: Face Swap via Diffusion Model

• Authors: Feifei Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01108
• Pdf URL: https://arxiv.org/pdf/2403.01108
• Copy Paste: [[2403.01108]] Face Swap via Diffusion Model(https://arxiv.org/abs/2403.01108)
• Keywords: diffusion
• Abstract: This technical report presents a diffusion model based framework for face swapping between two portrait images. The basic framework consists of three components, i.e., IP-Adapter, ControlNet, and Stable Diffusion's inpainting pipeline, for face feature encoding, multi-conditional generation, and face inpainting respectively. Besides, I introduce facial guidance optimization and CodeFormer based blending to further improve the generation quality. Specifically, we engage a recent light-weighted customization method (i.e., DreamBooth-LoRA), to guarantee the identity consistency by 1) using a rare identifier "sks" to represent the source identity, and 2) injecting the image features of source portrait into each cross-attention layer like the text features. Then I resort to the strong inpainting ability of Stable Diffusion, and utilize canny image and face detection annotation of the target portrait as the conditions, to guide ContorlNet's generation and align source portrait with the target portrait. To further correct face alignment, we add the facial guidance loss to optimize the text embedding during the sample generation.

Title: OpenGraph: Towards Open Graph Foundation Models

• Authors: Lianghao Xia, Ben Kao, Chao Huang
• Subjects: cs.LG, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2403.01121
• Pdf URL: https://arxiv.org/pdf/2403.01121
• Copy Paste: [[2403.01121]] OpenGraph: Towards Open Graph Foundation Models(https://arxiv.org/abs/2403.01121)
• Keywords: foundation model
• Abstract: Graph learning has become indispensable for interpreting and harnessing relational data in diverse fields, ranging from recommendation systems to social network analysis. In this context, a variety of GNNs have emerged as promising methodologies for encoding the structural information of graphs. By effectively capturing the graph's underlying structure, these GNNs have shown great potential in enhancing performance in graph learning tasks, such as link prediction and node classification. However, despite their successes, a significant challenge persists: these advanced methods often face difficulties in generalizing to unseen graph data that significantly differs from the training instances. In this work, our aim is to advance the graph learning paradigm by developing a general graph foundation model. This model is designed to understand the complex topological patterns present in diverse graph data, enabling it to excel in zero-shot graph learning tasks across different downstream datasets. To achieve this goal, we address several key technical challenges in our OpenGraph model. Firstly, we propose a unified graph tokenizer to adapt our graph model to generalize well on unseen graph data, even when the underlying graph properties differ significantly from those encountered during training. Secondly, we develop a scalable graph transformer as the foundational encoder, which effectively captures node-wise dependencies within the global topological context. Thirdly, we introduce a data augmentation mechanism enhanced by a LLM to alleviate the limitations of data scarcity in real-world scenarios. Extensive experiments validate the effectiveness of our framework. By adapting our OpenGraph to new graph characteristics and comprehending the nuances of diverse graphs, our approach achieves remarkable zero-shot graph learning performance across various settings and domains.

Title: Text-guided Explorable Image Super-resolution

• Authors: Kanchana Vaishnavi Gandikota, Paramanand Chandramouli
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01124
• Pdf URL: https://arxiv.org/pdf/2403.01124
• Copy Paste: [[2403.01124]] Text-guided Explorable Image Super-resolution(https://arxiv.org/abs/2403.01124)
• Keywords: diffusion, generative
• Abstract: In this paper, we introduce the problem of zero-shot text-guided exploration of the solutions to open-domain image super-resolution. Our goal is to allow users to explore diverse, semantically accurate reconstructions that preserve data consistency with the low-resolution inputs for different large downsampling factors without explicitly training for these specific degradations. We propose two approaches for zero-shot text-guided super-resolution - i) modifying the generative process of text-to-image \textit{T2I} diffusion models to promote consistency with low-resolution inputs, and ii) incorporating language guidance into zero-shot diffusion-based restoration methods. We show that the proposed approaches result in diverse solutions that match the semantic meaning provided by the text prompt while preserving data consistency with the degraded inputs. We evaluate the proposed baselines for the task of extreme super-resolution and demonstrate advantages in terms of restoration quality, diversity, and explorability of solutions.

Title: Dynamic 3D Point Cloud Sequences as 2D Videos

• Authors: Yiming Zeng, Junhui Hou, Qijian Zhang, Siyu Ren, Wenping Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01129
• Pdf URL: https://arxiv.org/pdf/2403.01129
• Copy Paste: [[2403.01129]] Dynamic 3D Point Cloud Sequences as 2D Videos(https://arxiv.org/abs/2403.01129)
• Keywords: self-supervised
• Abstract: Dynamic 3D point cloud sequences serve as one of the most common and practical representation modalities of dynamic real-world environments. However, their unstructured nature in both spatial and temporal domains poses significant challenges to effective and efficient processing. Existing deep point cloud sequence modeling approaches imitate the mature 2D video learning mechanisms by developing complex spatio-temporal point neighbor grouping and feature aggregation schemes, often resulting in methods lacking effectiveness, efficiency, and expressive power. In this paper, we propose a novel generic representation called \textit{Structured Point Cloud Videos} (SPCVs). Intuitively, by leveraging the fact that 3D geometric shapes are essentially 2D manifolds, SPCV re-organizes a point cloud sequence as a 2D video with spatial smoothness and temporal consistency, where the pixel values correspond to the 3D coordinates of points. The structured nature of our SPCV representation allows for the seamless adaptation of well-established 2D image/video techniques, enabling efficient and effective processing and analysis of 3D point cloud sequences. To achieve such re-organization, we design a self-supervised learning pipeline that is geometrically regularized and driven by self-reconstructive and deformation field learning objectives. Additionally, we construct SPCV-based frameworks for both low-level and high-level 3D point cloud sequence processing and analysis tasks, including action recognition, temporal interpolation, and compression. Extensive experiments demonstrate the versatility and superiority of the proposed SPCV, which has the potential to offer new possibilities for deep learning on unstructured 3D point cloud sequences. Code will be released at https://github.com/ZENGYIMING-EAMON/SPCV.

Title: A Hybrid Model for Traffic Incident Detection based on Generative Adversarial Networks and Transformer Model

• Authors: Xinying Lu, Doudou Zhang, Jianli Xiao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01147
• Pdf URL: https://arxiv.org/pdf/2403.01147
• Copy Paste: [[2403.01147]] A Hybrid Model for Traffic Incident Detection based on Generative Adversarial Networks and Transformer Model(https://arxiv.org/abs/2403.01147)
• Keywords: generative
• Abstract: In addition to enhancing traffic safety and facilitating prompt emergency response, traffic incident detection plays an indispensable role in intelligent transportation systems by providing real-time traffic status information. This enables the realization of intelligent traffic control and management. Previous research has identified that apart from employing advanced algorithmic models, the effectiveness of detection is also significantly influenced by challenges related to acquiring large datasets and addressing dataset imbalances. A hybrid model combining transformer and generative adversarial networks (GANs) is proposed to address these challenges. Experiments are conducted on four real datasets to validate the superiority of the transformer in traffic incident detection. Additionally, GANs are utilized to expand the dataset and achieve a balanced ratio of 1:4, 2:3, and 1:1. The proposed model is evaluated against the baseline model. The results demonstrate that the proposed model enhances the dataset size, balances the dataset, and improves the performance of traffic incident detection in various aspects.

Title: Learn Suspected Anomalies from Event Prompts for Video Anomaly Detection

• Authors: Chenchen Tao, Chong Wang, Yuexian Zou, Xiaohao Peng, Jiafei Wu, Jiangbo Qian
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01169
• Pdf URL: https://arxiv.org/pdf/2403.01169
• Copy Paste: [[2403.01169]] Learn Suspected Anomalies from Event Prompts for Video Anomaly Detection(https://arxiv.org/abs/2403.01169)
• Keywords: anomaly
• Abstract: Most models for weakly supervised video anomaly detection (WS-VAD) rely on multiple instance learning, aiming to distinguish normal and abnormal snippets without specifying the type of anomaly. The ambiguous nature of anomaly definitions across contexts introduces bias in detecting abnormal and normal snippets within the abnormal bag. Taking the first step to show the model why it is anomalous, a novel framework is proposed to guide the learning of suspected anomalies from event prompts. Given a textual prompt dictionary of potential anomaly events and the captions generated from anomaly videos, the semantic anomaly similarity between them could be calculated to identify the suspected anomalous events for each video snippet. It enables a new multi-prompt learning process to constrain the visual-semantic features across all videos, as well as provides a new way to label pseudo anomalies for self-training. To demonstrate effectiveness, comprehensive experiments and detailed ablation studies are conducted on four datasets, namely XD-Violence, UCF-Crime, TAD, and ShanghaiTech. Our proposed model outperforms most state-of-the-art methods in terms of AP or AUC (82.6\%, 87.7\%, 93.1\%, and 97.4\%). Furthermore, it shows promising performance in open-set and cross-dataset cases.

Title: Leveraging Self-Supervised Learning for Scene Recognition in Child Sexual Abuse Imagery

• Authors: Pedro H. V. Valois, João Macedo, Leo S. F. Ribeiro, Jefersson A. dos Santos, Sandra Avila
• Subjects: cs.CV, cs.AI, cs.CY, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01183
• Pdf URL: https://arxiv.org/pdf/2403.01183
• Copy Paste: [[2403.01183]] Leveraging Self-Supervised Learning for Scene Recognition in Child Sexual Abuse Imagery(https://arxiv.org/abs/2403.01183)
• Keywords: self-supervised
• Abstract: Crime in the 21st century is split into a virtual and real world. However, the former has become a global menace to people's well-being and security in the latter. The challenges it presents must be faced with unified global cooperation, and we must rely more than ever on automated yet trustworthy tools to combat the ever-growing nature of online offenses. Over 10 million child sexual abuse reports are submitted to the US National Center for Missing & Exploited Children every year, and over 80% originated from online sources. Therefore, investigation centers and clearinghouses cannot manually process and correctly investigate all imagery. In light of that, reliable automated tools that can securely and efficiently deal with this data are paramount. In this sense, the scene recognition task looks for contextual cues in the environment, being able to group and classify child sexual abuse data without requiring to be trained on sensitive material. The scarcity and limitations of working with child sexual abuse images lead to self-supervised learning, a machine-learning methodology that leverages unlabeled data to produce powerful representations that can be more easily transferred to target tasks. This work shows that self-supervised deep learning models pre-trained on scene-centric data can reach 71.6% balanced accuracy on our indoor scene classification task and, on average, 2.2 percentage points better performance than a fully supervised version. We cooperate with Brazilian Federal Police experts to evaluate our indoor classification model on actual child abuse material. The results demonstrate a notable discrepancy between the features observed in widely used scene datasets and those depicted on sensitive materials.

Title: Training Unbiased Diffusion Models From Biased Dataset

• Authors: Yeongmin Kim, Byeonghu Na, Minsang Park, JoonHo Jang, Dongjun Kim, Wanmo Kang, Il-Chul Moon
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01189
• Pdf URL: https://arxiv.org/pdf/2403.01189
• Copy Paste: [[2403.01189]] Training Unbiased Diffusion Models From Biased Dataset(https://arxiv.org/abs/2403.01189)
• Keywords: diffusion, generative
• Abstract: With significant advancements in diffusion models, addressing the potential risks of dataset bias becomes increasingly important. Since generated outputs directly suffer from dataset bias, mitigating latent bias becomes a key factor in improving sample quality and proportion. This paper proposes time-dependent importance reweighting to mitigate the bias for the diffusion models. We demonstrate that the time-dependent density ratio becomes more precise than previous approaches, thereby minimizing error propagation in generative learning. While directly applying it to score-matching is intractable, we discover that using the time-dependent density ratio both for reweighting and score correction can lead to a tractable form of the objective function to regenerate the unbiased data density. Furthermore, we theoretically establish a connection with traditional score-matching, and we demonstrate its convergence to an unbiased distribution. The experimental evidence supports the usefulness of the proposed method, which outperforms baselines including time-independent importance reweighting on CIFAR-10, CIFAR-100, FFHQ, and CelebA with various bias settings. Our code is available at https://github.com/alsdudrla10/TIW-DSM.

Title: DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling

• Authors: Shanghaoran Quan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01197
• Pdf URL: https://arxiv.org/pdf/2403.01197
• Copy Paste: [[2403.01197]] DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling(https://arxiv.org/abs/2403.01197)
• Keywords: generative
• Abstract: The performance of the reward model (RM) is a critical factor in improving the effectiveness of the large language model (LLM) during alignment fine-tuning. There remain two challenges in RM training: 1) training the same RM using various categories of data may cause its generalization performance to suffer from multi-task disturbance, and 2) the human annotation consistency rate is generally only $60\%$ to $75\%$, causing training data to contain a lot of noise. To tackle these two challenges, we introduced the idea of Mixture-of-Experts (MoE) into the field of RM for the first time. We propose the Double-Layer MoE RM (DMoERM). The outer layer MoE is a sparse model. After classifying an input into task categories, we route it to the corresponding inner layer task-specific model. The inner layer MoE is a dense model. We decompose the specific task into multiple capability dimensions and individually fine-tune a LoRA expert on each one. Their outputs are then synthesized by an MLP to compute the final rewards. To minimize costs, we call a public LLM API to obtain the capability preference labels. The validation on manually labeled datasets confirms that our model attains superior consistency with human preference and outstrips advanced generative approaches. Meanwhile, through BoN sampling and RL experiments, we demonstrate that our model outperforms state-of-the-art ensemble methods of RM and mitigates the overoptimization problem. Our code and dataset are available at: https://github.com/quanshr/DMoERM-v1.

Title: TCIG: Two-Stage Controlled Image Generation with Quality Enhancement through Diffusion

• Authors: Salaheldin Mohamed
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01212
• Pdf URL: https://arxiv.org/pdf/2403.01212
• Copy Paste: [[2403.01212]] TCIG: Two-Stage Controlled Image Generation with Quality Enhancement through Diffusion(https://arxiv.org/abs/2403.01212)
• Keywords: diffusion
• Abstract: In recent years, significant progress has been made in the development of text- to-image generation models. However, these models still face limitations when it comes to achieving full controllability during the generation process. Often, spe- cific training or the use of limited models is required, and even then, they have certain restrictions. To address these challenges, A two-stage method that effec- tively combines controllability and high quality in the generation of images is proposed. This approach leverages the expertise of pre-trained models to achieve precise control over the generated images, while also harnessing the power of diffusion models to achieve state-of-the-art quality. By separating controllability from high quality, This method achieves outstanding results. It is compatible with both latent and image space diffusion models, ensuring versatility and flexibil- ity. Moreover, This approach consistently produces comparable outcomes to the current state-of-the-art methods in the field. Overall, This proposed method rep- resents a significant advancement in text-to-image generation, enabling improved controllability without compromising on the quality of the generated images.

Title: DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction

• Authors: Junwen Xiong, Peng Zhang, Tao You, Chuanyue Li, Wei Huang, Yufei Zha
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01226
• Pdf URL: https://arxiv.org/pdf/2403.01226
• Copy Paste: [[2403.01226]] DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction(https://arxiv.org/abs/2403.01226)
• Keywords: diffusion, generative
• Abstract: Audio-visual saliency prediction can draw support from diverse modality complements, but further performance enhancement is still challenged by customized architectures as well as task-specific loss functions. In recent studies, denoising diffusion models have shown more promising in unifying task frameworks owing to their inherent ability of generalization. Following this motivation, a novel Diffusion architecture for generalized audio-visual Saliency prediction (DiffSal) is proposed in this work, which formulates the prediction problem as a conditional generative task of the saliency map by utilizing input audio and video as the conditions. Based on the spatio-temporal audio-visual features, an extra network Saliency-UNet is designed to perform multi-modal attention modulation for progressive refinement of the ground-truth saliency map from the noisy map. Extensive experiments demonstrate that the proposed DiffSal can achieve excellent performance across six challenging audio-visual benchmarks, with an average relative improvement of 6.3\% over the previous state-of-the-art results by six metrics.

Title: Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal

• Authors: Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song, Junfeng Yao, Jinsong Su
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01244
• Pdf URL: https://arxiv.org/pdf/2403.01244
• Copy Paste: [[2403.01244]] Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal(https://arxiv.org/abs/2403.01244)
• Keywords: in-context
• Abstract: Large language models (LLMs) suffer from catastrophic forgetting during continual learning. Conventional rehearsal-based methods rely on previous training data to retain the model's ability, which may not be feasible in real-world applications. When conducting continual learning based on a publicly-released LLM checkpoint, the availability of the original training data may be non-existent. To address this challenge, we propose a framework called Self-Synthesized Rehearsal (SSR) that uses the LLM to generate synthetic instances for rehearsal. Concretely, we first employ the base LLM for in-context learning to generate synthetic instances. Subsequently, we utilize the latest LLM to refine the instance outputs based on the synthetic inputs, preserving its acquired ability. Finally, we select diverse high-quality synthetic instances for rehearsal in future stages. Experimental results demonstrate that SSR achieves superior or comparable performance compared to conventional rehearsal-based approaches while being more data-efficient. Besides, SSR effectively preserves the generalization capabilities of LLMs in general domains.

Title: AcME-AD: Accelerated Model Explanations for Anomaly Detection

• Authors: Valentina Zaccaria, David Dandolo, Chiara Masiero, Gian Antonio Susto
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01245
• Pdf URL: https://arxiv.org/pdf/2403.01245
• Copy Paste: [[2403.01245]] AcME-AD: Accelerated Model Explanations for Anomaly Detection(https://arxiv.org/abs/2403.01245)
• Keywords: anomaly
• Abstract: Pursuing fast and robust interpretability in Anomaly Detection is crucial, especially due to its significance in practical applications. Traditional Anomaly Detection methods excel in outlier identification but are often black-boxes, providing scant insights into their decision-making process. This lack of transparency compromises their reliability and hampers their adoption in scenarios where comprehending the reasons behind anomaly detection is vital. At the same time, getting explanations quickly is paramount in practical scenarios. To bridge this gap, we present AcME-AD, a novel approach rooted in Explainable Artificial Intelligence principles, designed to clarify Anomaly Detection models for tabular data. AcME-AD transcends the constraints of model-specific or resource-heavy explainability techniques by delivering a model-agnostic, efficient solution for interoperability. It offers local feature importance scores and a what-if analysis tool, shedding light on the factors contributing to each anomaly, thus aiding root cause analysis and decision-making. This paper elucidates AcME-AD's foundation, its benefits over existing methods, and validates its effectiveness with tests on both synthetic and real datasets. AcME-AD's implementation and experiment replication code is accessible in a public repository.

Title: SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code

• Authors: Ziniu Hu, Ahmet Iscen, Aashi Jain, Thomas Kipf, Yisong Yue, David A. Ross, Cordelia Schmid, Alireza Fathi
• Subjects: cs.CV, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01248
• Pdf URL: https://arxiv.org/pdf/2403.01248
• Copy Paste: [[2403.01248]] SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code(https://arxiv.org/abs/2403.01248)
• Keywords: foundation model, generative
• Abstract: This paper introduces SceneCraft, a Large Language Model (LLM) Agent converting text descriptions into Blender-executable Python scripts which render complex scenes with up to a hundred 3D assets. This process requires complex spatial planning and arrangement. We tackle these challenges through a combination of advanced abstraction, strategic planning, and library learning. SceneCraft first models a scene graph as a blueprint, detailing the spatial relationships among assets in the scene. SceneCraft then writes Python scripts based on this graph, translating relationships into numerical constraints for asset layout. Next, SceneCraft leverages the perceptual strengths of vision-language foundation models like GPT-V to analyze rendered images and iteratively refine the scene. On top of this process, SceneCraft features a library learning mechanism that compiles common script functions into a reusable library, facilitating continuous self-improvement without expensive LLM parameter tuning. Our evaluation demonstrates that SceneCraft surpasses existing LLM-based agents in rendering complex scenes, as shown by its adherence to constraints and favorable human assessments. We also showcase the broader application potential of SceneCraft by reconstructing detailed 3D scenes from the Sintel movie and guiding a video generative model with generated scenes as intermediary control signal.

Title: ICC: Quantifying Image Caption Concreteness for Multimodal Dataset Curation

• Authors: Moran Yanuka, Morris Alper, Hadar Averbuch-Elor, Raja Giryes
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01306
• Pdf URL: https://arxiv.org/pdf/2403.01306
• Copy Paste: [[2403.01306]] ICC: Quantifying Image Caption Concreteness for Multimodal Dataset Curation(https://arxiv.org/abs/2403.01306)
• Keywords: foundation model
• Abstract: Web-scale training on paired text-image data is becoming increasingly central to multimodal learning, but is challenged by the highly noisy nature of datasets in the wild. Standard data filtering approaches succeed in removing mismatched text-image pairs, but permit semantically related but highly abstract or subjective text. These approaches lack the fine-grained ability to isolate the most concrete samples that provide the strongest signal for learning in a noisy dataset. In this work, we propose a new metric, image caption concreteness, that evaluates caption text without an image reference to measure its concreteness and relevancy for use in multimodal learning. Our approach leverages strong foundation models for measuring visual-semantic information loss in multimodal representations. We demonstrate that this strongly correlates with human evaluation of concreteness in both single-word and sentence-level texts. Moreover, we show that curation using ICC complements existing approaches: It succeeds in selecting the highest quality samples from multimodal web-scale datasets to allow for efficient training in resource-constrained settings.

Title: Bespoke Non-Stationary Solvers for Fast Sampling of Diffusion and Flow Models

• Authors: Neta Shaul, Uriel Singer, Ricky T. Q. Chen, Matthew Le, Ali Thabet, Albert Pumarola, Yaron Lipman
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01329
• Pdf URL: https://arxiv.org/pdf/2403.01329
• Copy Paste: [[2403.01329]] Bespoke Non-Stationary Solvers for Fast Sampling of Diffusion and Flow Models(https://arxiv.org/abs/2403.01329)
• Keywords: diffusion
• Abstract: This paper introduces Bespoke Non-Stationary (BNS) Solvers, a solver distillation approach to improve sample efficiency of Diffusion and Flow models. BNS solvers are based on a family of non-stationary solvers that provably subsumes existing numerical ODE solvers and consequently demonstrate considerable improvement in sample approximation (PSNR) over these baselines. Compared to model distillation, BNS solvers benefit from a tiny parameter space ($<$200 parameters), fast optimization (two orders of magnitude faster), maintain diversity of samples, and in contrast to previous solver distillation approaches nearly close the gap from standard distillation methods such as Progressive Distillation in the low-medium NFE regime. For example, BNS solver achieves 45 PSNR / 1.76 FID using 16 NFE in class-conditional ImageNet-64. We experimented with BNS solvers for conditional image generation, text-to-image generation, and text-2-audio generation showing significant improvement in sample approximation (PSNR) in all.

Title: A Simple-but-effective Baseline for Training-free Class-Agnostic Counting

• Authors: Yuhao Lin, Haiming Xu, Lingqiao Liu, Javen Qinfeng Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01418
• Pdf URL: https://arxiv.org/pdf/2403.01418
• Copy Paste: [[2403.01418]] A Simple-but-effective Baseline for Training-free Class-Agnostic Counting(https://arxiv.org/abs/2403.01418)
• Keywords: foundation model
• Abstract: Class-Agnostic Counting (CAC) seeks to accurately count objects in a given image with only a few reference examples. While previous methods achieving this relied on additional training, recent efforts have shown that it's possible to accomplish this without training by utilizing pre-existing foundation models, particularly the Segment Anything Model (SAM), for counting via instance-level segmentation. Although promising, current training-free methods still lag behind their training-based counterparts in terms of performance. In this research, we present a straightforward training-free solution that effectively bridges this performance gap, serving as a strong baseline. The primary contribution of our work lies in the discovery of four key technologies that can enhance performance. Specifically, we suggest employing a superpixel algorithm to generate more precise initial point prompts, utilizing an image encoder with richer semantic knowledge to replace the SAM encoder for representing candidate objects, and adopting a multiscale mechanism and a transductive prototype scheme to update the representation of reference examples. By combining these four technologies, our approach achieves significant improvements over existing training-free methods and delivers performance on par with training-based ones.

Title: Introduction to Algogens

• Authors: Amir Shachar
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01426
• Pdf URL: https://arxiv.org/pdf/2403.01426
• Copy Paste: [[2403.01426]] Introduction to Algogens(https://arxiv.org/abs/2403.01426)
• Keywords: generative
• Abstract: This book introduces the concept of Algogens, a promising integration of generative AI with traditional algorithms aimed at improving problem-solving techniques across various fields. It provides an accessible overview of how Algogens combine AI's innovative potential with algorithms' reliability to tackle complex challenges more effectively than either could alone. The text explores the basics of Algogens, their development, applications, and advantages, such as better adaptability and efficiency. Through examples and case studies, readers will learn about Algogens' practical uses today and their potential for future cybersecurity, healthcare, and environmental science innovation. Acknowledging new technologies' challenges and ethical considerations, the book offers a balanced look at the prospects and obstacles facing Algogens. It invites a broad audience, including experts and newcomers, to engage with the topic and consider Algogens' role in advancing our problem-solving capabilities. This work is presented as a starting point for anyone interested in the intersection of AI and algorithms, encouraging further exploration and discussion on this emerging field. It aims to spark curiosity and contribute to the ongoing conversation about how technology can evolve to meet the complex demands of the AI era.

Title: On Diffusion Process in SE(3)-invariant Space

• Authors: Zihan Zhou, Ruiying Liu, Jiachen Zheng, Xiaoxue Wang, Tianshu Yu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01430
• Pdf URL: https://arxiv.org/pdf/2403.01430
• Copy Paste: [[2403.01430]] On Diffusion Process in SE(3)-invariant Space(https://arxiv.org/abs/2403.01430)
• Keywords: diffusion
• Abstract: Sampling viable 3D structures (e.g., molecules and point clouds) with SE(3)-invariance using diffusion-based models proved promising in a variety of real-world applications, wherein SE(3)-invariant properties can be naturally characterized by the inter-point distance manifold. However, due to the non-trivial geometry, we still lack a comprehensive understanding of the diffusion mechanism within such SE(3)-invariant space. This study addresses this gap by mathematically delineating the diffusion mechanism under SE(3)-invariance, via zooming into the interaction behavior between coordinates and the inter-point distance manifold through the lens of differential geometry. Upon this analysis, we propose accurate and projection-free diffusion SDE and ODE accordingly. Such formulations enable enhancing the performance and the speed of generation pathways; meanwhile offering valuable insights into other systems incorporating SE(3)-invariance.

Title: GuardT2I: Defending Text-to-Image Models from Adversarial Prompts

• Authors: Yijun Yang, Ruiyuan Gao, Xiao Yang, Jianyuan Zhong, Qiang Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01446
• Pdf URL: https://arxiv.org/pdf/2403.01446
• Copy Paste: [[2403.01446]] GuardT2I: Defending Text-to-Image Models from Adversarial Prompts(https://arxiv.org/abs/2403.01446)
• Keywords: generative
• Abstract: Recent advancements in Text-to-Image (T2I) models have raised significant safety concerns about their potential misuse for generating inappropriate or Not-Safe-For-Work (NSFW) contents, despite existing countermeasures such as NSFW classifiers or model fine-tuning for inappropriate concept removal. Addressing this challenge, our study unveils GuardT2I, a novel moderation framework that adopts a generative approach to enhance T2I models' robustness against adversarial prompts. Instead of making a binary classification, GuardT2I utilizes a Large Language Model (LLM) to conditionally transform text guidance embeddings within the T2I models into natural language for effective adversarial prompt detection, without compromising the models' inherent performance. Our extensive experiments reveal that GuardT2I outperforms leading commercial solutions like OpenAI-Moderation and Microsoft Azure Moderator by a significant margin across diverse adversarial scenarios.

Title: Representation Learning on Heterophilic Graph with Directional Neighborhood Attention

• Authors: Qincheng Lu, Jiaqi Zhu, Sitao Luan, Xiao-Wen Chang
• Subjects: cs.LG, cs.AI, cs.SI
• Abstract URL: https://arxiv.org/abs/2403.01475
• Pdf URL: https://arxiv.org/pdf/2403.01475
• Copy Paste: [[2403.01475]] Representation Learning on Heterophilic Graph with Directional Neighborhood Attention(https://arxiv.org/abs/2403.01475)
• Keywords: diffusion
• Abstract: Graph Attention Network (GAT) is one of the most popular Graph Neural Network (GNN) architecture, which employs the attention mechanism to learn edge weights and has demonstrated promising performance in various applications. However, since it only incorporates information from immediate neighborhood, it lacks the ability to capture long-range and global graph information, leading to unsatisfactory performance on some datasets, particularly on heterophilic graphs. To address this limitation, we propose the Directional Graph Attention Network (DGAT) in this paper. DGAT is able to combine the feature-based attention with the global directional information extracted from the graph topology. To this end, a new class of Laplacian matrices is proposed which can provably reduce the diffusion distance between nodes. Based on the new Laplacian, topology-guided neighbour pruning and edge adding mechanisms are proposed to remove the noisy and capture the helpful long-range neighborhood information. Besides, a global directional attention is designed to enable a topological-aware information propagation. The superiority of the proposed DGAT over the baseline GAT has also been verified through experiments on real-world benchmarks and synthetic data sets. It also outperforms the state-of-the-art (SOTA) models on 6 out of 7 real-world benchmark datasets.

Title: CCC: Color Classified Colorization

• Authors: Mrityunjoy Gain, Avi Deb Raha, Rameswar Debnath
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01476
• Pdf URL: https://arxiv.org/pdf/2403.01476
• Copy Paste: [[2403.01476]] CCC: Color Classified Colorization(https://arxiv.org/abs/2403.01476)
• Keywords: generative
• Abstract: Automatic colorization of gray images with objects of different colors and sizes is challenging due to inter- and intra-object color variation and the small area of the main objects due to extensive backgrounds. The learning process often favors dominant features, resulting in a biased model. In this paper, we formulate the colorization problem into a multinomial classification problem and then apply a weighted function to classes. We propose a set of formulas to transform color values into color classes and vice versa. Class optimization and balancing feature distribution are the keys for good performance. Observing class appearance on various extremely large-scale real-time images in practice, we propose 215 color classes for our colorization task. During training, we propose a class-weighted function based on true class appearance in each batch to ensure proper color saturation of individual objects. We establish a trade-off between major and minor classes to provide orthodox class prediction by eliminating major classes' dominance over minor classes. As we apply regularization to enhance the stability of the minor class, occasional minor noise may appear at the object's edges. We propose a novel object-selective color harmonization method empowered by the SAM to refine and enhance these edges. We propose a new color image evaluation metric, the Chromatic Number Ratio (CNR), to quantify the richness of color components. We compare our proposed model with state-of-the-art models using five different datasets: ADE, Celeba, COCO, Oxford 102 Flower, and ImageNet, in both qualitative and quantitative approaches. The experimental results show that our proposed model outstrips other models in visualization and CNR measurement criteria while maintaining satisfactory performance in regression (MSE, PSNR), similarity (SSIM, LPIPS, UIQI), and generative criteria (FID).

Title: EAGLE: Eigen Aggregation Learning for Object-Centric Unsupervised Semantic Segmentation

• Authors: Chanyoung Kim, Woojung Han, Dayun Ju, Seong Jae Hwang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01482
• Pdf URL: https://arxiv.org/pdf/2403.01482
• Copy Paste: [[2403.01482]] EAGLE: Eigen Aggregation Learning for Object-Centric Unsupervised Semantic Segmentation(https://arxiv.org/abs/2403.01482)
• Keywords: self-supervised
• Abstract: Semantic segmentation has innately relied on extensive pixel-level labeled annotated data, leading to the emergence of unsupervised methodologies. Among them, leveraging self-supervised Vision Transformers for unsupervised semantic segmentation (USS) has been making steady progress with expressive deep features. Yet, for semantically segmenting images with complex objects, a predominant challenge remains: the lack of explicit object-level semantic encoding in patch-level features. This technical limitation often leads to inadequate segmentation of complex objects with diverse structures. To address this gap, we present a novel approach, EAGLE, which emphasizes object-centric representation learning for unsupervised semantic segmentation. Specifically, we introduce EiCue, a spectral technique providing semantic and structural cues through an eigenbasis derived from the semantic similarity matrix of deep image features and color affinity from an image. Further, by incorporating our object-centric contrastive loss with EiCue, we guide our model to learn object-level representations with intra- and inter-image object-feature consistency, thereby enhancing semantic accuracy. Extensive experiments on COCO-Stuff, Cityscapes, and Potsdam-3 datasets demonstrate the state-of-the-art USS results of EAGLE with accurate and consistent semantic segmentation across complex scenes.

Title: Regeneration Based Training-free Attribution of Fake Images Generated by Text-to-Image Generative Models

• Authors: Meiling Li, Zhenxing Qian, Xinpeng Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01489
• Pdf URL: https://arxiv.org/pdf/2403.01489
• Copy Paste: [[2403.01489]] Regeneration Based Training-free Attribution of Fake Images Generated by Text-to-Image Generative Models(https://arxiv.org/abs/2403.01489)
• Keywords: generative
• Abstract: Text-to-image generative models have recently garnered significant attention due to their ability to generate images based on prompt descriptions. While these models have shown promising performance, concerns have been raised regarding the potential misuse of the generated fake images. In response to this, we have presented a simple yet effective training-free method to attribute fake images generated by text-to-image models to their source models. Given a test image to be attributed, we first inverse the textual prompt of the image, and then put the reconstructed prompt into different candidate models to regenerate candidate fake images. By calculating and ranking the similarity of the test image and the candidate images, we can determine the source of the image. This attribution allows model owners to be held accountable for any misuse of their models. Note that our approach does not limit the number of candidate text-to-image generative models. Comprehensive experiments reveal that (1) Our method can effectively attribute fake images to their source models, achieving comparable attribution performance with the state-of-the-art method; (2) Our method has high scalability ability, which is well adapted to real-world attribution scenarios. (3) The proposed method yields satisfactory robustness to common attacks, such as Gaussian blurring, JPEG compression, and Resizing. We also analyze the factors that influence the attribution performance, and explore the boost brought by the proposed method as a plug-in to improve the performance of existing SOTA. We hope our work can shed some light on the solutions to addressing the source of AI-generated images, as well as to prevent the misuse of text-to-image generative models.

Title: Learning A Physical-aware Diffusion Model Based on Transformer for Underwater Image Enhancement

• Authors: Chen Zhao, Chenyu Dong, Weiling Cai
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01497
• Pdf URL: https://arxiv.org/pdf/2403.01497
• Copy Paste: [[2403.01497]] Learning A Physical-aware Diffusion Model Based on Transformer for Underwater Image Enhancement(https://arxiv.org/abs/2403.01497)
• Keywords: diffusion
• Abstract: Underwater visuals undergo various complex degradations, inevitably influencing the efficiency of underwater vision tasks. Recently, diffusion models were employed to underwater image enhancement (UIE) tasks, and gained SOTA performance. However, these methods fail to consider the physical properties and underwater imaging mechanisms in the diffusion process, limiting information completion capacity of diffusion models. In this paper, we introduce a novel UIE framework, named PA-Diff, designed to exploiting the knowledge of physics to guide the diffusion process. PA-Diff consists of Physics Prior Generation (PPG) Branch and Physics-aware Diffusion Transformer (PDT) Branch. Our designed PPG branch is a plug-and-play network to produce the physics prior, which can be integrated into any deep framework. With utilizing the physics prior knowledge to guide the diffusion process, PDT branch can obtain underwater-aware ability and model the complex distribution in real-world underwater scenes. Extensive experiments prove that our method achieves best performance on UIE tasks.

Title: Applying Self-supervised Learning to Network Intrusion Detection for Network Flows with Graph Neural Network

• Authors: Renjie Xu, Guangwei Wu, Weiping Wang, Xing Gao, An He, Zhengpeng Zhang
• Subjects: cs.LG, cs.CR
• Abstract URL: https://arxiv.org/abs/2403.01501
• Pdf URL: https://arxiv.org/pdf/2403.01501
• Copy Paste: [[2403.01501]] Applying Self-supervised Learning to Network Intrusion Detection for Network Flows with Graph Neural Network(https://arxiv.org/abs/2403.01501)
• Keywords: self-supervised
• Abstract: Graph Neural Networks (GNNs) have garnered intensive attention for Network Intrusion Detection System (NIDS) due to their suitability for representing the network traffic flows. However, most present GNN-based methods for NIDS are supervised or semi-supervised. Network flows need to be manually annotated as supervisory labels, a process that is time-consuming or even impossible, making NIDS difficult to adapt to potentially complex attacks, especially in large-scale real-world scenarios. The existing GNN-based self-supervised methods focus on the binary classification of network flow as benign or not, and thus fail to reveal the types of attack in practice. This paper studies the application of GNNs to identify the specific types of network flows in an unsupervised manner. We first design an encoder to obtain graph embedding, that introduces the graph attention mechanism and considers the edge information as the only essential factor. Then, a self-supervised method based on graph contrastive learning is proposed. The method samples center nodes, and for each center node, generates subgraph by it and its direct neighbor nodes, and corresponding contrastive subgraph from the interpolated graph, and finally constructs positive and negative samples from subgraphs. Furthermore, a structured contrastive loss function based on edge features and graph local topology is introduced. To the best of our knowledge, it is the first GNN-based self-supervised method for the multiclass classification of network flows in NIDS. Detailed experiments conducted on four real-world databases (NF-Bot-IoT, NF-Bot-IoT-v2, NF-CSE-CIC-IDS2018, and NF-CSE-CIC-IDS2018-v2) systematically compare our model with the state-of-the-art supervised and self-supervised models, illustrating the considerable potential of our method. Our code is accessible through https://github.com/renj-xu/NEGSC.

Title: SCott: Accelerating Diffusion Models with Stochastic Consistency Distillation

• Authors: Hongjian Liu, Qingsong Xie, Zhijie Deng, Chen Chen, Shixiang Tang, Fueyang Fu, Zheng-jun Zha, Haonan Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01505
• Pdf URL: https://arxiv.org/pdf/2403.01505
• Copy Paste: [[2403.01505]] SCott: Accelerating Diffusion Models with Stochastic Consistency Distillation(https://arxiv.org/abs/2403.01505)
• Keywords: diffusion
• Abstract: The iterative sampling procedure employed by diffusion models (DMs) often leads to significant inference latency. To address this, we propose Stochastic Consistency Distillation (SCott) to enable accelerated text-to-image generation, where high-quality generations can be achieved with just 1-2 sampling steps, and further improvements can be obtained by adding additional steps. In contrast to vanilla consistency distillation (CD) which distills the ordinary differential equation solvers-based sampling process of a pretrained teacher model into a student, SCott explores the possibility and validates the efficacy of integrating stochastic differential equation (SDE) solvers into CD to fully unleash the potential of the teacher. SCott is augmented with elaborate strategies to control the noise strength and sampling process of the SDE solver. An adversarial loss is further incorporated to strengthen the sample quality with rare sampling steps. Empirically, on the MSCOCO-2017 5K dataset with a Stable Diffusion-V1.5 teacher, SCott achieves an FID (Frechet Inceptio Distance) of 22.1, surpassing that (23.4) of the 1-step InstaFlow (Liu et al., 2023) and matching that of 4-step UFOGen (Xue et al., 2023b). Moreover, SCott can yield more diverse samples than other consistency models for high-resolution image generation (Luo et al., 2023a), with up to 16% improvement in a qualified metric. The code and checkpoints are coming soon.

Title: Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semantics

• Authors: Zhu Liu, Cunliang Kong, Ying Liu, Maosong Sun
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01509
• Pdf URL: https://arxiv.org/pdf/2403.01509
• Copy Paste: [[2403.01509]] Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semantics(https://arxiv.org/abs/2403.01509)
• Keywords: generative
• Abstract: Large language models have achieved remarkable success in general language understanding tasks. However, as a family of generative methods with the objective of next token prediction, the semantic evolution with the depth of these models are not fully explored, unlike their predecessors, such as BERT-like architectures. In this paper, we specifically investigate the bottom-up evolution of lexical semantics for a popular LLM, namely Llama2, by probing its hidden states at the end of each layer using a contextualized word identification task. Our experiments show that the representations in lower layers encode lexical semantics, while the higher layers, with weaker semantic induction, are responsible for prediction. This is in contrast to models with discriminative objectives, such as mask language modeling, where the higher layers obtain better lexical semantics. The conclusion is further supported by the monotonic increase in performance via the hidden states for the last meaningless symbols, such as punctuation, in the prompting strategy.

Title: Revisiting Dynamic Evaluation: Online Adaptation for Large Language Models

• Authors: Amal Rannen-Triki, Jorg Bornschein, Razvan Pascanu, Marcus Hutter, Andras György, Alexandre Galashov, Yee Whye Teh, Michalis K. Titsias
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01518
• Pdf URL: https://arxiv.org/pdf/2403.01518
• Copy Paste: [[2403.01518]] Revisiting Dynamic Evaluation: Online Adaptation for Large Language Models(https://arxiv.org/abs/2403.01518)
• Keywords: in-context
• Abstract: We consider the problem of online fine tuning the parameters of a language model at test time, also known as dynamic evaluation. While it is generally known that this approach improves the overall predictive performance, especially when considering distributional shift between training and evaluation data, we here emphasize the perspective that online adaptation turns parameters into temporally changing states and provides a form of context-length extension with memory in weights, more in line with the concept of memory in neuroscience. We pay particular attention to the speed of adaptation (in terms of sample efficiency),sensitivity to the overall distributional drift, and the computational overhead for performing gradient computations and parameter updates. Our empirical study provides insights on when online adaptation is particularly interesting. We highlight that with online adaptation the conceptual distinction between in-context learning and fine tuning blurs: both are methods to condition the model on previously observed tokens.

Title: Neural Graph Generator: Feature-Conditioned Graph Generation using Latent Diffusion Models

• Authors: Iakovos Evdaimon, Giannis Nikolentzos, Michail Chatzianastasis, Hadi Abdine, Michalis Vazirgiannis
• Subjects: cs.LG, cs.SI
• Abstract URL: https://arxiv.org/abs/2403.01535
• Pdf URL: https://arxiv.org/pdf/2403.01535
• Copy Paste: [[2403.01535]] Neural Graph Generator: Feature-Conditioned Graph Generation using Latent Diffusion Models(https://arxiv.org/abs/2403.01535)
• Keywords: diffusion
• Abstract: Graph generation has emerged as a crucial task in machine learning, with significant challenges in generating graphs that accurately reflect specific properties. Existing methods often fall short in efficiently addressing this need as they struggle with the high-dimensional complexity and varied nature of graph properties. In this paper, we introduce the Neural Graph Generator (NGG), a novel approach which utilizes conditioned latent diffusion models for graph generation. NGG demonstrates a remarkable capacity to model complex graph patterns, offering control over the graph generation process. NGG employs a variational graph autoencoder for graph compression and a diffusion process in the latent vector space, guided by vectors summarizing graph statistics. We demonstrate NGG's versatility across various graph generation tasks, showing its capability to capture desired graph properties and generalize to unseen graphs. This work signifies a significant shift in graph generation methodologies, offering a more practical and efficient solution for generating diverse types of graphs with specific characteristics.

Title: Hyperspectral Image Analysis in Single-Modal and Multimodal setting using Deep Learning Techniques

• Authors: Shivam Pande
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01546
• Pdf URL: https://arxiv.org/pdf/2403.01546
• Copy Paste: [[2403.01546]] Hyperspectral Image Analysis in Single-Modal and Multimodal setting using Deep Learning Techniques(https://arxiv.org/abs/2403.01546)
• Keywords: self-supervised
• Abstract: Hyperspectral imaging provides precise classification for land use and cover due to its exceptional spectral resolution. However, the challenges of high dimensionality and limited spatial resolution hinder its effectiveness. This study addresses these challenges by employing deep learning techniques to efficiently process, extract features, and classify data in an integrated manner. To enhance spatial resolution, we integrate information from complementary modalities such as LiDAR and SAR data through multimodal learning. Moreover, adversarial learning and knowledge distillation are utilized to overcome issues stemming from domain disparities and missing modalities. We also tailor deep learning architectures to suit the unique characteristics of HSI data, utilizing 1D convolutional and recurrent neural networks to handle its continuous spectral dimension. Techniques like visual attention and feedback connections within the architecture bolster the robustness of feature extraction. Additionally, we tackle the issue of limited training samples through self-supervised learning methods, employing autoencoders for dimensionality reduction and exploring semi-supervised learning techniques that leverage unlabeled data. Our proposed approaches are evaluated across various HSI datasets, consistently outperforming existing state-of-the-art techniques.

Title: In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation

• Authors: Shiqi Chen, Miao Xiong, Junteng Liu, Zhengxuan Wu, Teng Xiao, Siyang Gao, Junxian He
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01548
• Pdf URL: https://arxiv.org/pdf/2403.01548
• Copy Paste: [[2403.01548]] In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation(https://arxiv.org/abs/2403.01548)
• Keywords: in-context
• Abstract: Large language models (LLMs) frequently hallucinate and produce factual errors, yet our understanding of why they make these errors remains limited. In this study, we delve into the underlying mechanisms of LLM hallucinations from the perspective of inner representations, and discover a salient pattern associated with hallucinations: correct generations tend to have sharper context activations in the hidden states of the in-context tokens, compared to the incorrect ones. Leveraging this insight, we propose an entropy-based metric to quantify the ``sharpness'' among the in-context hidden states and incorporate it into the decoding process to formulate a constrained decoding approach. Experiments on various knowledge-seeking and hallucination benchmarks demonstrate our approach's consistent effectiveness, for example, achieving up to an 8.6 point improvement on TruthfulQA. We believe this study can improve our understanding of hallucinations and serve as a practical solution for hallucination mitigation.

Title: Self-Supervised Representation Learning with Meta Comprehensive Regularization

• Authors: Huijie Guo, Ying Ba, Jie Hu, Lingyu Si, Wenwen Qiang, Lei Shi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01549
• Pdf URL: https://arxiv.org/pdf/2403.01549
• Copy Paste: [[2403.01549]] Self-Supervised Representation Learning with Meta Comprehensive Regularization(https://arxiv.org/abs/2403.01549)
• Keywords: self-supervised
• Abstract: Self-Supervised Learning (SSL) methods harness the concept of semantic invariance by utilizing data augmentation strategies to produce similar representations for different deformations of the same input. Essentially, the model captures the shared information among multiple augmented views of samples, while disregarding the non-shared information that may be beneficial for downstream tasks. To address this issue, we introduce a module called CompMod with Meta Comprehensive Regularization (MCR), embedded into existing self-supervised frameworks, to make the learned representations more comprehensive. Specifically, we update our proposed model through a bi-level optimization mechanism, enabling it to capture comprehensive features. Additionally, guided by the constrained extraction of features using maximum entropy coding, the self-supervised learning model learns more comprehensive features on top of learning consistent features. In addition, we provide theoretical support for our proposed method from information theory and causal counterfactual perspective. Experimental results show that our method achieves significant improvement in classification, object detection and instance segmentation tasks on multiple benchmark datasets.

Title: Transformers for Supervised Online Continual Learning

• Authors: Jorg Bornschein, Yazhe Li, Amal Rannen-Triki
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01554
• Pdf URL: https://arxiv.org/pdf/2403.01554
• Copy Paste: [[2403.01554]] Transformers for Supervised Online Continual Learning(https://arxiv.org/abs/2403.01554)
• Keywords: in-context
• Abstract: Transformers have become the dominant architecture for sequence modeling tasks such as natural language processing or audio processing, and they are now even considered for tasks that are not naturally sequential such as image classification. Their ability to attend to and to process a set of tokens as context enables them to develop in-context few-shot learning abilities. However, their potential for online continual learning remains relatively unexplored. In online continual learning, a model must adapt to a non-stationary stream of data, minimizing the cumulative nextstep prediction loss. We focus on the supervised online continual learning setting, where we learn a predictor $x_t \rightarrow y_t$ for a sequence of examples $(x_t, y_t)$. Inspired by the in-context learning capabilities of transformers and their connection to meta-learning, we propose a method that leverages these strengths for online continual learning. Our approach explicitly conditions a transformer on recent observations, while at the same time online training it with stochastic gradient descent, following the procedure introduced with Transformer-XL. We incorporate replay to maintain the benefits of multi-epoch training while adhering to the sequential protocol. We hypothesize that this combination enables fast adaptation through in-context learning and sustained longterm improvement via parametric learning. Our method demonstrates significant improvements over previous state-of-the-art results on CLOC, a challenging large-scale real-world benchmark for image geo-localization.

Title: Kick Back & Relax++: Scaling Beyond Ground-Truth Depth with SlowTV & CribsTV

• Authors: Jaime Spencer, Chris Russell, Simon Hadfield, Richard Bowden
• Subjects: cs.CV, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2403.01569
• Pdf URL: https://arxiv.org/pdf/2403.01569
• Copy Paste: [[2403.01569]] Kick Back & Relax++: Scaling Beyond Ground-Truth Depth with SlowTV & CribsTV(https://arxiv.org/abs/2403.01569)
• Keywords: self-supervised
• Abstract: Self-supervised learning is the key to unlocking generic computer vision systems. By eliminating the reliance on ground-truth annotations, it allows scaling to much larger data quantities. Unfortunately, self-supervised monocular depth estimation (SS-MDE) has been limited by the absence of diverse training data. Existing datasets have focused exclusively on urban driving in densely populated cities, resulting in models that fail to generalize beyond this domain. To address these limitations, this paper proposes two novel datasets: SlowTV and CribsTV. These are large-scale datasets curated from publicly available YouTube videos, containing a total of 2M training frames. They offer an incredibly diverse set of environments, ranging from snowy forests to coastal roads, luxury mansions and even underwater coral reefs. We leverage these datasets to tackle the challenging task of zero-shot generalization, outperforming every existing SS-MDE approach and even some state-of-the-art supervised methods. The generalization capabilities of our models are further enhanced by a range of components and contributions: 1) learning the camera intrinsics, 2) a stronger augmentation regime targeting aspect ratio changes, 3) support frame randomization, 4) flexible motion estimation, 5) a modern transformer-based architecture. We demonstrate the effectiveness of each component in extensive ablation experiments. To facilitate the development of future research, we make the datasets, code and pretrained models available to the public at https://github.com/jspenmar/slowtv_monodepth.

Title: Critical windows: non-asymptotic theory for feature emergence in diffusion models

• Authors: Marvin Li, Sitan Chen
• Subjects: cs.LG, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.01633
• Pdf URL: https://arxiv.org/pdf/2403.01633
• Copy Paste: [[2403.01633]] Critical windows: non-asymptotic theory for feature emergence in diffusion models(https://arxiv.org/abs/2403.01633)
• Keywords: diffusion
• Abstract: We develop theory to understand an intriguing property of diffusion models for image generation that we term critical windows. Empirically, it has been observed that there are narrow time intervals in sampling during which particular features of the final image emerge, e.g. the image class or background color (Ho et al., 2020b; Georgiev et al., 2023; Raya & Ambrogioni, 2023; Sclocchi et al., 2024; Biroli et al., 2024). While this is advantageous for interpretability as it implies one can localize properties of the generation to a small segment of the trajectory, it seems at odds with the continuous nature of the diffusion. We propose a formal framework for studying these windows and show that for data coming from a mixture of strongly log-concave densities, these windows can be provably bounded in terms of certain measures of inter- and intra-group separation. We also instantiate these bounds for concrete examples like well-conditioned Gaussian mixtures. Finally, we use our bounds to give a rigorous interpretation of diffusion models as hierarchical samplers that progressively "decide" output features over a discrete sequence of times. We validate our bounds with synthetic experiments. Additionally, preliminary experiments on Stable Diffusion suggest critical windows may serve as a useful tool for diagnosing fairness and privacy violations in real-world diffusion models.

Title: Theoretical Insights for Diffusion Guidance: A Case Study for Gaussian Mixture Models

• Authors: Yuchen Wu, Minshuo Chen, Zihao Li, Mengdi Wang, Yuting Wei
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.01639
• Pdf URL: https://arxiv.org/pdf/2403.01639
• Copy Paste: [[2403.01639]] Theoretical Insights for Diffusion Guidance: A Case Study for Gaussian Mixture Models(https://arxiv.org/abs/2403.01639)
• Keywords: diffusion
• Abstract: Diffusion models benefit from instillation of task-specific information into the score function to steer the sample generation towards desired properties. Such information is coined as guidance. For example, in text-to-image synthesis, text input is encoded as guidance to generate semantically aligned images. Proper guidance inputs are closely tied to the performance of diffusion models. A common observation is that strong guidance promotes a tight alignment to the task-specific information, while reducing the diversity of the generated samples. In this paper, we provide the first theoretical study towards understanding the influence of guidance on diffusion models in the context of Gaussian mixture models. Under mild conditions, we prove that incorporating diffusion guidance not only boosts classification confidence but also diminishes distribution diversity, leading to a reduction in the differential entropy of the output distribution. Our analysis covers the widely adopted sampling schemes including DDPM and DDIM, and leverages comparison inequalities for differential equations as well as the Fokker-Planck equation that characterizes the evolution of probability density function, which may be of independent theoretical interest.

Title: Improving Adversarial Energy-Based Model via Diffusion Process

• Authors: Cong Geng, Tian Han, Peng-Tao Jiang, Hao Zhang, Jinwei Chen, Søren Hauberg, Bo Li
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01666
• Pdf URL: https://arxiv.org/pdf/2403.01666
• Copy Paste: [[2403.01666]] Improving Adversarial Energy-Based Model via Diffusion Process(https://arxiv.org/abs/2403.01666)
• Keywords: diffusion, generative
• Abstract: Generative models have shown strong generation ability while efficient likelihood estimation is less explored. Energy-based models~(EBMs) define a flexible energy function to parameterize unnormalized densities efficiently but are notorious for being difficult to train. Adversarial EBMs introduce a generator to form a minimax training game to avoid expensive MCMC sampling used in traditional EBMs, but a noticeable gap between adversarial EBMs and other strong generative models still exists. Inspired by diffusion-based models, we embedded EBMs into each denoising step to split a long-generated process into several smaller steps. Besides, we employ a symmetric Jeffrey divergence and introduce a variational posterior distribution for the generator's training to address the main challenges that exist in adversarial EBMs. Our experiments show significant improvement in generation compared to existing adversarial EBMs, while also providing a useful energy function for efficient density estimation.

Title: HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances

• Authors: Supreeth Narasimhaswamy, Uttaran Bhattacharya, Xiang Chen, Ishita Dasgupta, Saayan Mitra, Minh Hoai
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01693
• Pdf URL: https://arxiv.org/pdf/2403.01693
• Copy Paste: [[2403.01693]] HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances(https://arxiv.org/abs/2403.01693)
• Keywords: diffusion, generative
• Abstract: Text-to-image generative models can generate high-quality humans, but realism is lost when generating hands. Common artifacts include irregular hand poses, shapes, incorrect numbers of fingers, and physically implausible finger orientations. To generate images with realistic hands, we propose a novel diffusion-based architecture called HanDiffuser that achieves realism by injecting hand embeddings in the generative process. HanDiffuser consists of two components: a Text-to-Hand-Params diffusion model to generate SMPL-Body and MANO-Hand parameters from input text prompts, and a Text-Guided Hand-Params-to-Image diffusion model to synthesize images by conditioning on the prompts and hand parameters generated by the previous component. We incorporate multiple aspects of hand representation, including 3D shapes and joint-level finger positions, orientations and articulations, for robust learning and reliable performance during inference. We conduct extensive quantitative and qualitative experiments and perform user studies to demonstrate the efficacy of our method in generating images with high-quality hands.

Title: Diffusion-TS: Interpretable Diffusion for General Time Series Generation

• Authors: Xinyu Yuan, Yan Qiao
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01742
• Pdf URL: https://arxiv.org/pdf/2403.01742
• Copy Paste: [[2403.01742]] Diffusion-TS: Interpretable Diffusion for General Time Series Generation(https://arxiv.org/abs/2403.01742)
• Keywords: diffusion, generative
• Abstract: Denoising diffusion probabilistic models (DDPMs) are becoming the leading paradigm for generative models. It has recently shown breakthroughs in audio synthesis, time series imputation and forecasting. In this paper, we propose Diffusion-TS, a novel diffusion-based framework that generates multivariate time series samples of high quality by using an encoder-decoder transformer with disentangled temporal representations, in which the decomposition technique guides Diffusion-TS to capture the semantic meaning of time series while transformers mine detailed sequential information from the noisy model input. Different from existing diffusion-based approaches, we train the model to directly reconstruct the sample instead of the noise in each diffusion step, combining a Fourier-based loss term. Diffusion-TS is expected to generate time series satisfying both interpretablity and realness. In addition, it is shown that the proposed Diffusion-TS can be easily extended to conditional generation tasks, such as forecasting and imputation, without any model changes. This also motivates us to further explore the performance of Diffusion-TS under irregular settings. Finally, through qualitative and quantitative experiments, results show that Diffusion-TS achieves the state-of-the-art results on various realistic analyses of time series.

Title: Decode Neural signal as Speech

• Authors: Yiqian Yang, Yiqun Duan, Qiang Zhang, Renjing Xu, Hui Xiong
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01748
• Pdf URL: https://arxiv.org/pdf/2403.01748
• Copy Paste: [[2403.01748]] Decode Neural signal as Speech(https://arxiv.org/abs/2403.01748)
• Keywords: generative
• Abstract: Decoding language from brain dynamics is an important open direction in the realm of brain-computer interface (BCI), especially considering the rapid growth of large language models. Compared to invasive-based signals which require electrode implantation surgery, non-invasive neural signals (e.g. EEG, MEG) have attracted increasing attention considering their safety and generality. However, the exploration is not adequate in three aspects: 1) previous methods mainly focus on EEG but none of the previous works address this problem on MEG with better signal quality; 2) prior works have predominantly used ``teacher-forcing" during generative decoding, which is impractical; 3) prior works are mostly ``BART-based" not fully auto-regressive, which performs better in other sequence tasks. In this paper, we explore the brain-to-text translation of MEG signals in a speech-decoding formation. Here we are the first to investigate a cross-attention-based ``whisper" model for generating text directly from MEG signals without teacher forcing. Our model achieves impressive BLEU-1 scores of 60.30 and 52.89 without pretraining \& teacher-forcing on two major datasets (\textit{GWilliams} and \textit{Schoffelen}). This paper conducts a comprehensive review to understand how speech decoding formation performs on the neural decoding tasks, including pretraining initialization, training \& evaluation set splitting, augmentation, and scaling law.

Title: Differentially Private Synthetic Data via Foundation Model APIs 2: Text

• Authors: Chulin Xie, Zinan Lin, Arturs Backurs, Sivakanth Gopi, Da Yu, Huseyin A Inan, Harsha Nori, Haotian Jiang, Huishuai Zhang, Yin Tat Lee, Bo Li, Sergey Yekhanin
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01749
• Pdf URL: https://arxiv.org/pdf/2403.01749
• Copy Paste: [[2403.01749]] Differentially Private Synthetic Data via Foundation Model APIs 2: Text(https://arxiv.org/abs/2403.01749)
• Keywords: diffusion, foundation model
• Abstract: Text data has become extremely valuable due to the emergence of machine learning algorithms that learn from it. A lot of high-quality text data generated in the real world is private and therefore cannot be shared or used freely due to privacy concerns. Generating synthetic replicas of private text data with a formal privacy guarantee, i.e., differential privacy (DP), offers a promising and scalable solution. However, existing methods necessitate DP finetuning of large language models (LLMs) on private data to generate DP synthetic data. This approach is not viable for proprietary LLMs (e.g., GPT-3.5) and also demands considerable computational resources for open-source LLMs. Lin et al. (2024) recently introduced the Private Evolution (PE) algorithm to generate DP synthetic images with only API access to diffusion models. In this work, we propose an augmented PE algorithm, named Aug-PE, that applies to the complex setting of text. We use API access to an LLM and generate DP synthetic text without any model training. We conduct comprehensive experiments on three benchmark datasets. Our results demonstrate that Aug-PE produces DP synthetic text that yields competitive utility with the SOTA DP finetuning baselines. This underscores the feasibility of relying solely on API access of LLMs to produce high-quality DP synthetic texts, thereby facilitating more accessible routes to privacy-preserving LLM applications. Our code and data are available at https://github.com/AI-secure/aug-pe.

Title: OOTDiffusion: Outfitting Fusion based Latent Diffusion for Controllable Virtual Try-on

• Authors: Yuhao Xu, Tao Gu, Weifeng Chen, Chengcai Chen
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01779
• Pdf URL: https://arxiv.org/pdf/2403.01779
• Copy Paste: [[2403.01779]] OOTDiffusion: Outfitting Fusion based Latent Diffusion for Controllable Virtual Try-on(https://arxiv.org/abs/2403.01779)
• Keywords: diffusion
• Abstract: Image-based virtual try-on (VTON), which aims to generate an outfitted image of a target human wearing an in-shop garment, is a challenging image-synthesis task calling for not only high fidelity of the outfitted human but also full preservation of garment details. To tackle this issue, we propose Outfitting over Try-on Diffusion (OOTDiffusion), leveraging the power of pretrained latent diffusion models and designing a novel network architecture for realistic and controllable virtual try-on. Without an explicit warping process, we propose an outfitting UNet to learn the garment detail features, and merge them with the target human body via our proposed outfitting fusion in the denoising process of diffusion models. In order to further enhance the controllability of our outfitting UNet, we introduce outfitting dropout to the training process, which enables us to adjust the strength of garment features through classifier-free guidance. Our comprehensive experiments on the VITON-HD and Dress Code datasets demonstrate that OOTDiffusion efficiently generates high-quality outfitted images for arbitrary human and garment images, which outperforms other VTON methods in both fidelity and controllability, indicating an impressive breakthrough in virtual try-on. Our source code is available at https://github.com/levihsu/OOTDiffusion.

Title: PointCore: Efficient Unsupervised Point Cloud Anomaly Detector Using Local-Global Features

• Authors: Baozhu Zhao, Qiwei Xiong, Xiaohan Zhang, Jingfeng Guo, Qi Liu, Xiaofen Xing, Xiangmin Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01804
• Pdf URL: https://arxiv.org/pdf/2403.01804
• Copy Paste: [[2403.01804]] PointCore: Efficient Unsupervised Point Cloud Anomaly Detector Using Local-Global Features(https://arxiv.org/abs/2403.01804)
• Keywords: anomaly
• Abstract: Three-dimensional point cloud anomaly detection that aims to detect anomaly data points from a training set serves as the foundation for a variety of applications, including industrial inspection and autonomous driving. However, existing point cloud anomaly detection methods often incorporate multiple feature memory banks to fully preserve local and global representations, which comes at the high cost of computational complexity and mismatches between features. To address that, we propose an unsupervised point cloud anomaly detection framework based on joint local-global features, termed PointCore. To be specific, PointCore only requires a single memory bank to store local (coordinate) and global (PointMAE) representations and different priorities are assigned to these local-global features, thereby reducing the computational cost and mismatching disturbance in inference. Furthermore, to robust against the outliers, a normalization ranking method is introduced to not only adjust values of different scales to a notionally common scale, but also transform densely-distributed data into a uniform distribution. Extensive experiments on Real3D-AD dataset demonstrate that PointCore achieves competitive inference time and the best performance in both detection and localization as compared to the state-of-the-art Reg3D-AD approach and several competitors.

Title: ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models

• Authors: Lukas Höllein, Aljaž Božič, Norman Müller, David Novotny, Hung-Yu Tseng, Christian Richardt, Michael Zollhöfer, Matthias Nießner
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01807
• Pdf URL: https://arxiv.org/pdf/2403.01807
• Copy Paste: [[2403.01807]] ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models(https://arxiv.org/abs/2403.01807)
• Keywords: diffusion
• Abstract: 3D asset generation is getting massive amounts of attention, inspired by the recent success of text-guided 2D content creation. Existing text-to-3D methods use pretrained text-to-image diffusion models in an optimization problem or fine-tune them on synthetic data, which often results in non-photorealistic 3D objects without backgrounds. In this paper, we present a method that leverages pretrained text-to-image models as a prior, and learn to generate multi-view images in a single denoising process from real-world data. Concretely, we propose to integrate 3D volume-rendering and cross-frame-attention layers into each block of the existing U-Net network of the text-to-image model. Moreover, we design an autoregressive generation that renders more 3D-consistent images at any viewpoint. We train our model on real-world datasets of objects and showcase its capabilities to generate instances with a variety of high-quality shapes and textures in authentic surroundings. Compared to the existing methods, the results generated by our method are consistent, and have favorable visual quality (-30% FID, -37% KID).

Title: Rethinking LLM Language Adaptation: A Case Study on Chinese Mixtral

• Authors: Yiming Cui, Xin Yao
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01851
• Pdf URL: https://arxiv.org/pdf/2403.01851
• Copy Paste: [[2403.01851]] Rethinking LLM Language Adaptation: A Case Study on Chinese Mixtral(https://arxiv.org/abs/2403.01851)
• Keywords: foundation model
• Abstract: Mixtral, a representative sparse mixture of experts (SMoE) language model, has received significant attention due to its unique model design and superior performance. Based on Mixtral-8x7B-v0.1, in this paper, we propose Chinese-Mixtral and Chinese-Mixtral-Instruct with improved Chinese language abilities by adopting further pre-training and instruction fine-tuning. Experimental results show that our Chinese-Mixtral and Chinese-Mixtral-Instruct successfully improve Chinese understanding and generation performance while retaining the original English abilities. Then, we discuss several key questions when performing language adaptation on large language models, including the necessity of extending the language-specific vocabulary and the choice of the initialization model (foundation model v.s. instruction model), by providing empirical results and analysis. We also present the visualizations of each expert to examine their importance on downstream tasks. Our resources are publicly available through \url{https://github.com/ymcui/Chinese-Mixtral}.

Title: An Improved Traditional Chinese Evaluation Suite for Foundation Model

• Authors: Zhi-Rui Tam, Ya-Ting Pai, Yen-Wei Lee, Sega Cheng, Hong-Han Shuai
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01858
• Pdf URL: https://arxiv.org/pdf/2403.01858
• Copy Paste: [[2403.01858]] An Improved Traditional Chinese Evaluation Suite for Foundation Model(https://arxiv.org/abs/2403.01858)
• Keywords: foundation model
• Abstract: We present TMMLU+, a comprehensive dataset designed for the Traditional Chinese massive multitask language understanding dataset. TMMLU+ is a multiple-choice question-answering dataset with 66 subjects from elementary to professional level. Compared to its predecessor, TMMLU, TMMLU+ is six times larger and boasts a more balanced subject distribution. We included benchmark results in TMMLU+ from closed-source models and 24 open-weight Chinese large language models of parameters ranging from 1.8B to 72B. Our findings reveal that Traditional Chinese models still trail behind their Simplified Chinese counterparts. Additionally, current large language models have yet to outperform human performance in average scores. We publicly release our dataset and the corresponding benchmark source code.

Title: CSE: Surface Anomaly Detection with Contrastively Selected Embedding

• Authors: Simon Thomine, Hichem Snoussi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01859
• Pdf URL: https://arxiv.org/pdf/2403.01859
• Copy Paste: [[2403.01859]] CSE: Surface Anomaly Detection with Contrastively Selected Embedding(https://arxiv.org/abs/2403.01859)
• Keywords: anomaly
• Abstract: Detecting surface anomalies of industrial materials poses a significant challenge within a myriad of industrial manufacturing processes. In recent times, various methodologies have emerged, capitalizing on the advantages of employing a network pre-trained on natural images for the extraction of representative features. Subsequently, these features are subjected to processing through a diverse range of techniques including memory banks, normalizing flow, and knowledge distillation, which have exhibited exceptional accuracy. This paper revisits approaches based on pre-trained features by introducing a novel method centered on target-specific embedding. To capture the most representative features of the texture under consideration, we employ a variant of a contrastive training procedure that incorporates both artificially generated defective samples and anomaly-free samples during training. Exploiting the intrinsic properties of surfaces, we derived a meaningful representation from the defect-free samples during training, facilitating a straightforward yet effective calculation of anomaly scores. The experiments conducted on the MVTEC AD and TILDA datasets demonstrate the competitiveness of our approach compared to state-of-the-art methods.

Title: Unsupervised Distance Metric Learning for Anomaly Detection Over Multivariate Time Series

• Authors: Hanyang Yuan, Qinglin Cai, Keting Yin
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.01895
• Pdf URL: https://arxiv.org/pdf/2403.01895
• Copy Paste: [[2403.01895]] Unsupervised Distance Metric Learning for Anomaly Detection Over Multivariate Time Series(https://arxiv.org/abs/2403.01895)
• Keywords: anomaly
• Abstract: Distance-based time series anomaly detection methods are prevalent due to their relative non-parametric nature and interpretability. However, the commonly used Euclidean distance is sensitive to noise. While existing works have explored dynamic time warping (DTW) for its robustness, they only support supervised tasks over multivariate time series (MTS), leaving a scarcity of unsupervised methods. In this work, we propose FCM-wDTW, an unsupervised distance metric learning method for anomaly detection over MTS, which encodes raw data into latent space and reveals normal dimension relationships through cluster centers. FCM-wDTW introduces locally weighted DTW into fuzzy C-means clustering and learns the optimal latent space efficiently, enabling anomaly identification via data reconstruction. Experiments with 11 different types of benchmarks demonstrate our method's competitive accuracy and efficiency.

Title: FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio

• Authors: Chao Xu, Yang Liu, Jiazheng Xing, Weida Wang, Mingze Sun, Jun Dan, Tianxin Huang, Siyuan Li, Zhi-Qi Cheng, Ying Tai, Baigui Sun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01901
• Pdf URL: https://arxiv.org/pdf/2403.01901
• Copy Paste: [[2403.01901]] FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio(https://arxiv.org/abs/2403.01901)
• Keywords: diffusion
• Abstract: In this paper, we abstract the process of people hearing speech, extracting meaningful cues, and creating various dynamically audio-consistent talking faces, termed Listening and Imagining, into the task of high-fidelity diverse talking faces generation from a single audio. Specifically, it involves two critical challenges: one is to effectively decouple identity, content, and emotion from entangled audio, and the other is to maintain intra-video diversity and inter-video consistency. To tackle the issues, we first dig out the intricate relationships among facial factors and simplify the decoupling process, tailoring a Progressive Audio Disentanglement for accurate facial geometry and semantics learning, where each stage incorporates a customized training module responsible for a specific factor. Secondly, to achieve visually diverse and audio-synchronized animation solely from input audio within a single model, we introduce the Controllable Coherent Frame generation, which involves the flexible integration of three trainable adapters with frozen Latent Diffusion Models (LDMs) to focus on maintaining facial geometry and semantics, as well as texture and temporal coherence between frames. In this way, we inherit high-quality diverse generation from LDMs while significantly improving their controllability at a low training cost. Extensive experiments demonstrate the flexibility and effectiveness of our method in handling this paradigm. The codes will be released at https://github.com/modelscope/facechain.

Title: Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?

• Authors: Evgeniia Razumovskaia, Ivan Vulić, Anna Korhonen
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.01929
• Pdf URL: https://arxiv.org/pdf/2403.01929
• Copy Paste: [[2403.01929]] Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?(https://arxiv.org/abs/2403.01929)
• Keywords: in-context
• Abstract: Supervised fine-tuning (SFT), supervised instruction tuning (SIT) and in-context learning (ICL) are three alternative, de facto standard approaches to few-shot learning. ICL has gained popularity recently with the advent of LLMs due to its simplicity and sample efficiency. Prior research has conducted only limited investigation into how these approaches work for multilingual few-shot learning, and the focus so far has been mostly on their performance. In this work, we present an extensive and systematic comparison of the three approaches, testing them on 6 high- and low-resource languages, three different NLU tasks, and a myriad of language and domain setups. Importantly, performance is only one aspect of the comparison, where we also analyse the approaches through the optics of their computational, inference and financial costs. Our observations show that supervised instruction tuning has the best trade-off between performance and resource requirements. As another contribution, we analyse the impact of target language adaptation of pretrained LLMs and find that the standard adaptation approaches can (superficially) improve target language generation capabilities, but language understanding elicited through ICL does not improve and remains limited, with low scores especially for low-resource languages.

Title: A Generative Model of Symmetry Transformations

• Authors: James Urquhart Allingham, Bruno Kacper Mlodozeniec, Shreyas Padhy, Javier Antorán, David Krueger, Richard E. Turner, Eric Nalisnick, José Miguel Hernández-Lobato
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2403.01946
• Pdf URL: https://arxiv.org/pdf/2403.01946
• Copy Paste: [[2403.01946]] A Generative Model of Symmetry Transformations(https://arxiv.org/abs/2403.01946)
• Keywords: generative
• Abstract: Correctly capturing the symmetry transformations of data can lead to efficient models with strong generalization capabilities, though methods incorporating symmetries often require prior knowledge. While recent advancements have been made in learning those symmetries directly from the dataset, most of this work has focused on the discriminative setting. In this paper, we construct a generative model that explicitly aims to capture symmetries in the data, resulting in a model that learns which symmetries are present in an interpretable way. We provide a simple algorithm for efficiently learning our generative model and demonstrate its ability to capture symmetries under affine and color transformations. Combining our symmetry model with existing generative models results in higher marginal test-log-likelihoods and robustness to data sparsification.

Title: Explicit Motion Handling and Interactive Prompting for Video Camouflaged Object Detection

• Authors: Xin Zhang, Tao Xiao, Gepeng Ji, Xuan Wu, Keren Fu, Qijun Zhao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.01968
• Pdf URL: https://arxiv.org/pdf/2403.01968
• Copy Paste: [[2403.01968]] Explicit Motion Handling and Interactive Prompting for Video Camouflaged Object Detection(https://arxiv.org/abs/2403.01968)
• Keywords: self-supervised
• Abstract: Camouflage poses challenges in distinguishing a static target, whereas any movement of the target can break this disguise. Existing video camouflaged object detection (VCOD) approaches take noisy motion estimation as input or model motion implicitly, restricting detection performance in complex dynamic scenes. In this paper, we propose a novel Explicit Motion handling and Interactive Prompting framework for VCOD, dubbed EMIP, which handles motion cues explicitly using a frozen pre-trained optical flow fundamental model. EMIP is characterized by a two-stream architecture for simultaneously conducting camouflaged segmentation and optical flow estimation. Interactions across the dual streams are realized in an interactive prompting way that is inspired by emerging visual prompt learning. Two learnable modules, i.e. the camouflaged feeder and motion collector, are designed to incorporate segmentation-to-motion and motion-to-segmentation prompts, respectively, and enhance outputs of the both streams. The prompt fed to the motion stream is learned by supervising optical flow in a self-supervised manner. Furthermore, we show that long-term historical information can also be incorporated as a prompt into EMIP and achieve more robust results with temporal consistency. Experimental results demonstrate that our EMIP achieves new state-of-the-art records on popular VCOD benchmarks. The code will be publicly available.

Title: Unveiling Hidden Links Between Unseen Security Entities

• Authors: Daniel Alfasi, Tal Shapira, Anat Bremler Barr
• Subjects: cs.CR, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02014
• Pdf URL: https://arxiv.org/pdf/2403.02014
• Copy Paste: [[2403.02014]] Unveiling Hidden Links Between Unseen Security Entities(https://arxiv.org/abs/2403.02014)
• Keywords: foundation model
• Abstract: The proliferation of software vulnerabilities poses a significant challenge for security databases and analysts tasked with their timely identification, classification, and remediation. With the National Vulnerability Database (NVD) reporting an ever-increasing number of vulnerabilities, the traditional manual analysis becomes untenably time-consuming and prone to errors. This paper introduces VulnScopper, an innovative approach that utilizes multi-modal representation learning, combining Knowledge Graphs (KG) and Natural Language Processing (NLP), to automate and enhance the analysis of software vulnerabilities. Leveraging ULTRA, a knowledge graph foundation model, combined with a Large Language Model (LLM), VulnScopper effectively handles unseen entities, overcoming the limitations of previous KG approaches. We evaluate VulnScopper on two major security datasets, the NVD and the Red Hat CVE database. Our method significantly improves the link prediction accuracy between Common Vulnerabilities and Exposures (CVEs), Common Weakness Enumeration (CWEs), and Common Platform Enumerations (CPEs). Our results show that VulnScopper outperforms existing methods, achieving up to 78% Hits@10 accuracy in linking CVEs to CPEs and CWEs and presenting an 11.7% improvement over large language models in predicting CWE labels based on the Red Hat database. Based on the NVD, only 6.37% of the linked CPEs are being published during the first 30 days; many of them are related to critical and high-risk vulnerabilities which, according to multiple compliance frameworks (such as CISA and PCI), should be remediated within 15-30 days. Our model can uncover new products linked to vulnerabilities, reducing remediation time and improving vulnerability management. We analyzed several CVEs from 2023 to showcase this ability.

Title: A Generative Approach for Wikipedia-Scale Visual Entity Recognition

• Authors: Mathilde Caron, Ahmet Iscen, Alireza Fathi, Cordelia Schmid
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02041
• Pdf URL: https://arxiv.org/pdf/2403.02041
• Copy Paste: [[2403.02041]] A Generative Approach for Wikipedia-Scale Visual Entity Recognition(https://arxiv.org/abs/2403.02041)
• Keywords: generative
• Abstract: In this paper, we address web-scale visual entity recognition, specifically the task of mapping a given query image to one of the 6 million existing entities in Wikipedia. One way of approaching a problem of such scale is using dual-encoder models (eg CLIP), where all the entity names and query images are embedded into a unified space, paving the way for an approximate k-NN search. Alternatively, it is also possible to re-purpose a captioning model to directly generate the entity names for a given image. In contrast, we introduce a novel Generative Entity Recognition (GER) framework, which given an input image learns to auto-regressively decode a semantic and discriminative ``code'' identifying the target entity. Our experiments demonstrate the efficacy of this GER paradigm, showcasing state-of-the-art performance on the challenging OVEN benchmark. GER surpasses strong captioning, dual-encoder, visual matching and hierarchical classification baselines, affirming its advantage in tackling the complexities of web-scale recognition.

Title: Multi-Spectral Remote Sensing Image Retrieval Using Geospatial Foundation Models

• Authors: Benedikt Blumenstiel, Viktoria Moor, Romeo Kienzler, Thomas Brunschwiler
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02059
• Pdf URL: https://arxiv.org/pdf/2403.02059
• Copy Paste: [[2403.02059]] Multi-Spectral Remote Sensing Image Retrieval Using Geospatial Foundation Models(https://arxiv.org/abs/2403.02059)
• Keywords: foundation model
• Abstract: Image retrieval enables an efficient search through vast amounts of satellite imagery and returns similar images to a query. Deep learning models can identify images across various semantic concepts without the need for annotations. This work proposes to use Geospatial Foundation Models, like Prithvi, for remote sensing image retrieval with multiple benefits: i) the models encode multi-spectral satellite data and ii) generalize without further fine-tuning. We introduce two datasets to the retrieval task and observe a strong performance: Prithvi processes six bands and achieves a mean Average Precision of 97.62\% on BigEarthNet-43 and 44.51\% on ForestNet-12, outperforming other RGB-based models. Further, we evaluate three compression methods with binarized embeddings balancing retrieval speed and accuracy. They match the retrieval speed of much shorter hash codes while maintaining the same accuracy as floating-point embeddings but with a 32-fold compression. The code is available at https://github.com/IBM/remote-sensing-image-retrieval.

Title: DiffMOT: A Real-time Diffusion-based Multiple Object Tracker with Non-linear Prediction

• Authors: Weiyi Lv, Yuhang Huang, Ning Zhang, Ruei-Sung Lin, Mei Han, Dan Zeng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02075
• Pdf URL: https://arxiv.org/pdf/2403.02075
• Copy Paste: [[2403.02075]] DiffMOT: A Real-time Diffusion-based Multiple Object Tracker with Non-linear Prediction(https://arxiv.org/abs/2403.02075)
• Keywords: diffusion
• Abstract: In Multiple Object Tracking, objects often exhibit non-linear motion of acceleration and deceleration, with irregular direction changes. Tacking-by-detection (TBD) with Kalman Filter motion prediction works well in pedestrian-dominant scenarios but falls short in complex situations when multiple objects perform non-linear and diverse motion simultaneously. To tackle the complex non-linear motion, we propose a real-time diffusion-based MOT approach named DiffMOT. Specifically, for the motion predictor component, we propose a novel Decoupled Diffusion-based Motion Predictor (D MP). It models the entire distribution of various motion presented by the data as a whole. It also predicts an individual object's motion conditioning on an individual's historical motion information. Furthermore, it optimizes the diffusion process with much less sampling steps. As a MOT tracker, the DiffMOT is real-time at 22.7FPS, and also outperforms the state-of-the-art on DanceTrack and SportsMOT datasets with 63.4 and 76.2 in HOTA metrics, respectively. To the best of our knowledge, DiffMOT is the first to introduce a diffusion probabilistic model into the MOT to tackle non-linear motion prediction.

Title: ResAdapter: Domain Consistent Resolution Adapter for Diffusion Models

• Authors: Jiaxiang Cheng, Pan Xie, Xin Xia, Jiashi Li, Jie Wu, Yuxi Ren, Huixia Li, Xuefeng Xiao, Min Zheng, Lean Fu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02084
• Pdf URL: https://arxiv.org/pdf/2403.02084
• Copy Paste: [[2403.02084]] ResAdapter: Domain Consistent Resolution Adapter for Diffusion Models(https://arxiv.org/abs/2403.02084)
• Keywords: diffusion
• Abstract: Recent advancement in text-to-image models (e.g., Stable Diffusion) and corresponding personalized technologies (e.g., DreamBooth and LoRA) enables individuals to generate high-quality and imaginative images. However, they often suffer from limitations when generating images with resolutions outside of their trained domain. To overcome this limitation, we present the Resolution Adapter (ResAdapter), a domain-consistent adapter designed for diffusion models to generate images with unrestricted resolutions and aspect ratios. Unlike other multi-resolution generation methods that process images of static resolution with complex post-process operations, ResAdapter directly generates images with the dynamical resolution. Especially, after learning a deep understanding of pure resolution priors, ResAdapter trained on the general dataset, generates resolution-free images with personalized diffusion models while preserving their original style domain. Comprehensive experiments demonstrate that ResAdapter with only 0.5M can process images with flexible resolutions for arbitrary diffusion models. More extended experiments demonstrate that ResAdapter is compatible with other modules (e.g., ControlNet, IP-Adapter and LCM-LoRA) for image generation across a broad range of resolutions, and can be integrated into other multi-resolution model (e.g., ElasticDiffusion) for efficiently generating higher-resolution images. Project link is https://res-adapter.github.io

Title: Leveraging Weakly Annotated Data for Hate Speech Detection in Code-Mixed Hinglish: A Feasibility-Driven Transfer Learning Approach with Large Language Models

• Authors: Sargam Yadav (1), Abhishek Kaushik (1), Kevin McDaid (1) ((1) Dundalk Institute of Technology, Dundalk)
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2403.02121
• Pdf URL: https://arxiv.org/pdf/2403.02121
• Copy Paste: [[2403.02121]] Leveraging Weakly Annotated Data for Hate Speech Detection in Code-Mixed Hinglish: A Feasibility-Driven Transfer Learning Approach with Large Language Models(https://arxiv.org/abs/2403.02121)
• Keywords: generative
• Abstract: The advent of Large Language Models (LLMs) has advanced the benchmark in various Natural Language Processing (NLP) tasks. However, large amounts of labelled training data are required to train LLMs. Furthermore, data annotation and training are computationally expensive and time-consuming. Zero and few-shot learning have recently emerged as viable options for labelling data using large pre-trained models. Hate speech detection in mix-code low-resource languages is an active problem area where the use of LLMs has proven beneficial. In this study, we have compiled a dataset of 100 YouTube comments, and weakly labelled them for coarse and fine-grained misogyny classification in mix-code Hinglish. Weak annotation was applied due to the labor-intensive annotation process. Zero-shot learning, one-shot learning, and few-shot learning and prompting approaches have then been applied to assign labels to the comments and compare them to human-assigned labels. Out of all the approaches, zero-shot classification using the Bidirectional Auto-Regressive Transformers (BART) large model and few-shot prompting using Generative Pre-trained Transformer- 3 (ChatGPT-3) achieve the best results

Title: UB-FineNet: Urban Building Fine-grained Classification Network for Open-access Satellite Images

• Authors: Zhiyi He, Wei Yao, Jie Shao, Puzuo Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02132
• Pdf URL: https://arxiv.org/pdf/2403.02132
• Copy Paste: [[2403.02132]] UB-FineNet: Urban Building Fine-grained Classification Network for Open-access Satellite Images(https://arxiv.org/abs/2403.02132)
• Keywords: diffusion
• Abstract: Fine classification of city-scale buildings from satellite remote sensing imagery is a crucial research area with significant implications for urban planning, infrastructure development, and population distribution analysis. However, the task faces big challenges due to low-resolution overhead images acquired from high altitude space-borne platforms and the long-tail sample distribution of fine-grained urban building categories, leading to severe class imbalance problem. To address these issues, we propose a deep network approach to fine-grained classification of urban buildings using open-access satellite images. A Denoising Diffusion Probabilistic Model (DDPM) based super-resolution method is first introduced to enhance the spatial resolution of satellite images, which benefits from domain-adaptive knowledge distillation. Then, a new fine-grained classification network with Category Information Balancing Module (CIBM) and Contrastive Supervision (CS) technique is proposed to mitigate the problem of class imbalance and improve the classification robustness and accuracy. Experiments on Hong Kong data set with 11 fine building types revealed promising classification results with a mean Top-1 accuracy of 60.45\%, which is on par with street-view image based approaches. Extensive ablation study shows that CIBM and CS improve Top-1 accuracy by 2.6\% and 3.5\% compared to the baseline method, respectively. And both modules can be easily inserted into other classification networks and similar enhancements have been achieved. Our research contributes to the field of urban analysis by providing a practical solution for fine classification of buildings in challenging mega city scenarios solely using open-access satellite images. The proposed method can serve as a valuable tool for urban planners, aiding in the understanding of economic, industrial, and population distribution.

Title: Point2Building: Reconstructing Buildings from Airborne LiDAR Point Clouds

• Authors: Yujia Liu, Anton Obukhov, Jan Dirk Wegner, Konrad Schindler
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02136
• Pdf URL: https://arxiv.org/pdf/2403.02136
• Copy Paste: [[2403.02136]] Point2Building: Reconstructing Buildings from Airborne LiDAR Point Clouds(https://arxiv.org/abs/2403.02136)
• Keywords: generative
• Abstract: We present a learning-based approach to reconstruct buildings as 3D polygonal meshes from airborne LiDAR point clouds. What makes 3D building reconstruction from airborne LiDAR hard is the large diversity of building designs and especially roof shapes, the low and varying point density across the scene, and the often incomplete coverage of building facades due to occlusions by vegetation or to the viewing angle of the sensor. To cope with the diversity of shapes and inhomogeneous and incomplete object coverage, we introduce a generative model that directly predicts 3D polygonal meshes from input point clouds. Our autoregressive model, called Point2Building, iteratively builds up the mesh by generating sequences of vertices and faces. This approach enables our model to adapt flexibly to diverse geometries and building structures. Unlike many existing methods that rely heavily on pre-processing steps like exhaustive plane detection, our model learns directly from the point cloud data, thereby reducing error propagation and increasing the fidelity of the reconstruction. We experimentally validate our method on a collection of airborne LiDAR data of Zurich, Berlin and Tallinn. Our method shows good generalization to diverse urban styles.

Title: Self-Supervised Facial Representation Learning with Facial Region Awareness

• Authors: Zheng Gao, Ioannis Patras
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02138
• Pdf URL: https://arxiv.org/pdf/2403.02138
• Copy Paste: [[2403.02138]] Self-Supervised Facial Representation Learning with Facial Region Awareness(https://arxiv.org/abs/2403.02138)
• Keywords: self-supervised
• Abstract: Self-supervised pre-training has been proved to be effective in learning transferable representations that benefit various visual tasks. This paper asks this question: can self-supervised pre-training learn general facial representations for various facial analysis tasks? Recent efforts toward this goal are limited to treating each face image as a whole, i.e., learning consistent facial representations at the image-level, which overlooks the consistency of local facial representations (i.e., facial regions like eyes, nose, etc). In this work, we make a first attempt to propose a novel self-supervised facial representation learning framework to learn consistent global and local facial representations, Facial Region Awareness (FRA). Specifically, we explicitly enforce the consistency of facial regions by matching the local facial representations across views, which are extracted with learned heatmaps highlighting the facial regions. Inspired by the mask prediction in supervised semantic segmentation, we obtain the heatmaps via cosine similarity between the per-pixel projection of feature maps and facial mask embeddings computed from learnable positional embeddings, which leverage the attention mechanism to globally look up the facial image for facial regions. To learn such heatmaps, we formulate the learning of facial mask embeddings as a deep clustering problem by assigning the pixel features from the feature maps to them. The transfer learning results on facial classification and regression tasks show that our FRA outperforms previous pre-trained models and more importantly, using ResNet as the unified backbone for various tasks, our FRA achieves comparable or even better performance compared with SOTA methods in facial analysis tasks.

Title: TripoSR: Fast 3D Object Reconstruction from a Single Image

• Authors: Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, Yan-Pei Cao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02151
• Pdf URL: https://arxiv.org/pdf/2403.02151
• Copy Paste: [[2403.02151]] TripoSR: Fast 3D Object Reconstruction from a Single Image(https://arxiv.org/abs/2403.02151)
• Keywords: generative
• Abstract: This technical report introduces TripoSR, a 3D reconstruction model leveraging transformer architecture for fast feed-forward 3D generation, producing 3D mesh from a single image in under 0.5 seconds. Building upon the LRM network architecture, TripoSR integrates substantial improvements in data processing, model design, and training techniques. Evaluations on public datasets show that TripoSR exhibits superior performance, both quantitatively and qualitatively, compared to other open-source alternatives. Released under the MIT license, TripoSR is intended to empower researchers, developers, and creatives with the latest advancements in 3D generative AI.

Title: Not all Layers of LLMs are Necessary during Inference

• Authors: Siqi Fan, Xin Jiang, Xiang Li, Xuying Meng, Peng Han, Shuo Shang, Aixin Sun, Yequan Wang, Zhongyuan Wang
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2403.02181
• Pdf URL: https://arxiv.org/pdf/2403.02181
• Copy Paste: [[2403.02181]] Not all Layers of LLMs are Necessary during Inference(https://arxiv.org/abs/2403.02181)
• Keywords: in-context
• Abstract: The inference phase of Large Language Models (LLMs) is very expensive. An ideal inference stage of LLMs could utilize fewer computational resources while still maintaining its capabilities (e.g., generalization and in-context learning ability). In this paper, we try to answer the question, "During LLM inference, can we use shallow layers for easy instances; and deep layers for hard ones?" To answer this question, we first indicate that Not all Layers are Necessary during Inference by statistically analyzing the activated layers across tasks. Then, we propose a simple algorithm named AdaInfer to determine the inference termination moment based on the input instance adaptively. More importantly, AdaInfer does not alter LLM parameters and maintains generalizability across tasks. Experiments on well-known LLMs (i.e., Llama2 series and OPT) show that AdaInfer saves an average of 14.8% of computational resources, even up to 50% on sentiment tasks, while maintaining comparable performance. Additionally, this method is orthogonal to other model acceleration techniques, potentially boosting inference efficiency further.

Title: Perceptive self-supervised learning network for noisy image watermark removal

• Authors: Chunwei Tian, Menghua Zheng, Bo Li, Yanning Zhang, Shichao Zhang, David Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02211
• Pdf URL: https://arxiv.org/pdf/2403.02211
• Copy Paste: [[2403.02211]] Perceptive self-supervised learning network for noisy image watermark removal(https://arxiv.org/abs/2403.02211)
• Keywords: self-supervised
• Abstract: Popular methods usually use a degradation model in a supervised way to learn a watermark removal model. However, it is true that reference images are difficult to obtain in the real world, as well as collected images by cameras suffer from noise. To overcome these drawbacks, we propose a perceptive self-supervised learning network for noisy image watermark removal (PSLNet) in this paper. PSLNet depends on a parallel network to remove noise and watermarks. The upper network uses task decomposition ideas to remove noise and watermarks in sequence. The lower network utilizes the degradation model idea to simultaneously remove noise and watermarks. Specifically, mentioned paired watermark images are obtained in a self supervised way, and paired noisy images (i.e., noisy and reference images) are obtained in a supervised way. To enhance the clarity of obtained images, interacting two sub-networks and fusing obtained clean images are used to improve the effects of image watermark removal in terms of structural information and pixel enhancement. Taking into texture information account, a mixed loss uses obtained images and features to achieve a robust model of noisy image watermark removal. Comprehensive experiments show that our proposed method is very effective in comparison with popular convolutional neural networks (CNNs) for noisy image watermark removal. Codes can be obtained at https://github.com/hellloxiaotian/PSLNet.

Title: DragTex: Generative Point-Based Texture Editing on 3D Mesh

• Authors: Yudi Zhang, Qi Xu, Lei Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02217
• Pdf URL: https://arxiv.org/pdf/2403.02217
• Copy Paste: [[2403.02217]] DragTex: Generative Point-Based Texture Editing on 3D Mesh(https://arxiv.org/abs/2403.02217)
• Keywords: diffusion, generative
• Abstract: Creating 3D textured meshes using generative artificial intelligence has garnered significant attention recently. While existing methods support text-based generative texture generation or editing on 3D meshes, they often struggle to precisely control pixels of texture images through more intuitive interaction. While 2D images can be edited generatively using drag interaction, applying this type of methods directly to 3D mesh textures still leads to issues such as the lack of local consistency among multiple views, error accumulation and long training times. To address these challenges, we propose a generative point-based 3D mesh texture editing method called DragTex. This method utilizes a diffusion model to blend locally inconsistent textures in the region near the deformed silhouette between different views, enabling locally consistent texture editing. Besides, we fine-tune a decoder to reduce reconstruction errors in the non-drag region, thereby mitigating overall error accumulation. Moreover, we train LoRA using multi-view images instead of training each view individually, which significantly shortens the training time. The experimental results show that our method effectively achieves dragging textures on 3D meshes and generates plausible textures that align with the desired intent of drag interaction.

Title: Transformers Provably Learn Feature-Position Correlations in Masked Image Modeling

• Authors: Yu Huang, Zixin Wen, Yuejie Chi, Yingbin Liang
• Subjects: cs.LG, math.OC, stat.ML
• Abstract URL: https://arxiv.org/abs/2403.02233
• Pdf URL: https://arxiv.org/pdf/2403.02233
• Copy Paste: [[2403.02233]] Transformers Provably Learn Feature-Position Correlations in Masked Image Modeling(https://arxiv.org/abs/2403.02233)
• Keywords: self-supervised
• Abstract: Masked image modeling (MIM), which predicts randomly masked patches from unmasked ones, has emerged as a promising approach in self-supervised vision pretraining. However, the theoretical understanding of MIM is rather limited, especially with the foundational architecture of transformers. In this paper, to the best of our knowledge, we provide the first end-to-end theory of learning one-layer transformers with softmax attention in MIM self-supervised pretraining. On the conceptual side, we posit a theoretical mechanism of how transformers, pretrained with MIM, produce empirically observed local and diverse attention patterns on data distributions with spatial structures that highlight feature-position correlations. On the technical side, our end-to-end analysis of the training dynamics of softmax-based transformers accommodates both input and position embeddings simultaneously, which is developed based on a novel approach to track the interplay between the attention of feature-position and position-wise correlations.

Title: 3DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priors

• Authors: Fangzhou Hong, Jiaxiang Tang, Ziang Cao, Min Shi, Tong Wu, Zhaoxi Chen, Tengfei Wang, Liang Pan, Dahua Lin, Ziwei Liu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02234
• Pdf URL: https://arxiv.org/pdf/2403.02234
• Copy Paste: [[2403.02234]] 3DTopia: Large Text-to-3D Generation Model with Hybrid Diffusion Priors(https://arxiv.org/abs/2403.02234)
• Keywords: diffusion
• Abstract: We present a two-stage text-to-3D generation system, namely 3DTopia, which generates high-quality general 3D assets within 5 minutes using hybrid diffusion priors. The first stage samples from a 3D diffusion prior directly learned from 3D data. Specifically, it is powered by a text-conditioned tri-plane latent diffusion model, which quickly generates coarse 3D samples for fast prototyping. The second stage utilizes 2D diffusion priors to further refine the texture of coarse 3D models from the first stage. The refinement consists of both latent and pixel space optimization for high-quality texture generation. To facilitate the training of the proposed system, we clean and caption the largest open-source 3D dataset, Objaverse, by combining the power of vision language models and large language models. Experiment results are reported qualitatively and quantitatively to show the performance of the proposed system. Our codes and models are available at https://github.com/3DTopia/3DTopia

Title: Birbal: An efficient 7B instruct-model fine-tuned with curated datasets

• Authors: Ashvini Kumar Jindal, Pawan Kumar Rajpoot, Ankur Parikh
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02247
• Pdf URL: https://arxiv.org/pdf/2403.02247
• Copy Paste: [[2403.02247]] Birbal: An efficient 7B instruct-model fine-tuned with curated datasets(https://arxiv.org/abs/2403.02247)
• Keywords: foundation model
• Abstract: LLMOps incur significant costs due to hardware requirements, hindering their widespread accessibility. Additionally, a lack of transparency in model training methods and data contributes to the majority of models being non-reproducible. To tackle these challenges, the LLM Efficiency Challenge was introduced at NeurIPS Workshop, aiming to adapt foundation models on a diverse set of tasks via fine-tuning on a single GPU (RTX 4090 or A100 with 40GB) within a 24-hour timeframe. In this system description paper, we introduce Birbal, our Mistral-7B based winning model, fine-tuned on a single RTX 4090 for 16 hours. Birbal's success lies in curating high-quality instructions covering diverse tasks, resulting in a 35% performance improvement over second-best Qwen-14B based submission.

Title: Detection of Non-recorded Word Senses in English and Swedish

• Authors: Jonathan Lautenschlager, Emma Sköldberg, Simon Hengchen, Dominik Schlechtweg
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2403.02285
• Pdf URL: https://arxiv.org/pdf/2403.02285
• Copy Paste: [[2403.02285]] Detection of Non-recorded Word Senses in English and Swedish(https://arxiv.org/abs/2403.02285)
• Keywords: in-context
• Abstract: This study addresses the task of Unknown Sense Detection in English and Swedish. The primary objective of this task is to determine whether the meaning of a particular word usage is documented in a dictionary or not. For this purpose, sense entries are compared with word usages from modern and historical corpora using a pre-trained Word-in-Context embedder that allows us to model this task in a few-shot scenario. Additionally, we use human annotations to adapt and evaluate our models. Compared to a random sample from a corpus, our model is able to considerably increase the detected number of word usages with non-recorded senses.

Title: UniCtrl: Improving the Spatiotemporal Consistency of Text-to-Video Diffusion Models via Training-Free Unified Attention Control

• Authors: Xuweiyi Chen, Tian Xia, Sihan Xu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2403.02332
• Pdf URL: https://arxiv.org/pdf/2403.02332
• Copy Paste: [[2403.02332]] UniCtrl: Improving the Spatiotemporal Consistency of Text-to-Video Diffusion Models via Training-Free Unified Attention Control(https://arxiv.org/abs/2403.02332)
• Keywords: diffusion
• Abstract: Video Diffusion Models have been developed for video generation, usually integrating text and image conditioning to enhance control over the generated content. Despite the progress, ensuring consistency across frames remains a challenge, particularly when using text prompts as control conditions. To address this problem, we introduce UniCtrl, a novel, plug-and-play method that is universally applicable to improve the spatiotemporal consistency and motion diversity of videos generated by text-to-video models without additional training. UniCtrl ensures semantic consistency across different frames through cross-frame self-attention control, and meanwhile, enhances the motion quality and spatiotemporal consistency through motion injection and spatiotemporal synchronization. Our experimental results demonstrate UniCtrl's efficacy in enhancing various text-to-video models, confirming its effectiveness and universality.