2024-04-29

Title: Anomaly Detection for Incident Response at Scale

• Authors: Hanzhang Wang, Gowtham Kumar Tangirala, Gilkara Pranav Naidu, Charles Mayville, Arighna Roy, Joanne Sun, Ramesh Babu Mandava
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.16887
• Pdf URL: https://arxiv.org/pdf/2404.16887
• Copy Paste: [[2404.16887]] Anomaly Detection for Incident Response at Scale(https://arxiv.org/abs/2404.16887)
• Keywords: anomaly
• Abstract: We present a machine learning-based anomaly detection product, AI Detect and Respond (AIDR), that monitors Walmart's business and system health in real-time. During the validation over 3 months, the product served predictions from over 3000 models to more than 25 application, platform, and operation teams, covering 63\% of major incidents and reducing the mean-time-to-detect (MTTD) by more than 7 minutes. Unlike previous anomaly detection methods, our solution leverages statistical, ML and deep learning models while continuing to incorporate rule-based static thresholds to incorporate domain-specific knowledge. Both univariate and multivariate ML models are deployed and maintained through distributed services for scalability and high availability. AIDR has a feedback loop that assesses model quality with a combination of drift detection algorithms and customer feedback. It also offers self-onboarding capabilities and customizability. AIDR has achieved success with various internal teams with lower time to detection and fewer false positives than previous methods. As we move forward, we aim to expand incident coverage and prevention, reduce noise, and integrate further with root cause recommendation (RCR) to enable an end-to-end AIDR experience.

Title: DE-CGAN: Boosting rTMS Treatment Prediction with Diversity Enhancing Conditional Generative Adversarial Networks

• Authors: Matthew Squires, Xiaohui Tao, Soman Elangovan, Raj Gururajan, Haoran Xie, Xujuan Zhou, Yuefeng Li, U Rajendra Acharya
• Subjects: cs.LG, cs.AI, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.16913
• Pdf URL: https://arxiv.org/pdf/2404.16913
• Copy Paste: [[2404.16913]] DE-CGAN: Boosting rTMS Treatment Prediction with Diversity Enhancing Conditional Generative Adversarial Networks(https://arxiv.org/abs/2404.16913)
• Keywords: generative
• Abstract: Repetitive Transcranial Magnetic Stimulation (rTMS) is a well-supported, evidence-based treatment for depression. However, patterns of response to this treatment are inconsistent. Emerging evidence suggests that artificial intelligence can predict rTMS treatment outcomes for most patients using fMRI connectivity features. While these models can reliably predict treatment outcomes for many patients for some underrepresented fMRI connectivity measures DNN models are unable to reliably predict treatment outcomes. As such we propose a novel method, Diversity Enhancing Conditional General Adversarial Network (DE-CGAN) for oversampling these underrepresented examples. DE-CGAN creates synthetic examples in difficult-to-classify regions by first identifying these data points and then creating conditioned synthetic examples to enhance data diversity. Through empirical experiments we show that a classification model trained using a diversity enhanced training set outperforms traditional data augmentation techniques and existing benchmark results. This work shows that increasing the diversity of a training dataset can improve classification model performance. Furthermore, this work provides evidence for the utility of synthetic patients providing larger more robust datasets for both AI researchers and psychiatrists to explore variable relationships.

Title: IDIL: Imitation Learning of Intent-Driven Expert Behavior

• Authors: Sangwon Seo, Vaibhav Unhelkar
• Subjects: cs.LG, cs.AI, cs.RO
• Abstract URL: https://arxiv.org/abs/2404.16989
• Pdf URL: https://arxiv.org/pdf/2404.16989
• Copy Paste: [[2404.16989]] IDIL: Imitation Learning of Intent-Driven Expert Behavior(https://arxiv.org/abs/2404.16989)
• Keywords: generative
• Abstract: When faced with accomplishing a task, human experts exhibit intentional behavior. Their unique intents shape their plans and decisions, resulting in experts demonstrating diverse behaviors to accomplish the same task. Due to the uncertainties encountered in the real world and their bounded rationality, experts sometimes adjust their intents, which in turn influences their behaviors during task execution. This paper introduces IDIL, a novel imitation learning algorithm to mimic these diverse intent-driven behaviors of experts. Iteratively, our approach estimates expert intent from heterogeneous demonstrations and then uses it to learn an intent-aware model of their behavior. Unlike contemporary approaches, IDIL is capable of addressing sequential tasks with high-dimensional state representations, while sidestepping the complexities and drawbacks associated with adversarial training (a mainstay of related techniques). Our empirical results suggest that the models generated by IDIL either match or surpass those produced by recent imitation learning benchmarks in metrics of task performance. Moreover, as it creates a generative model, IDIL demonstrates superior performance in intent inference metrics, crucial for human-agent interactions, and aptly captures a broad spectrum of expert behaviors.

Title: Türkçe Dil Modellerinin Performans Karşılaştırması Performance Comparison of Turkish Language Models

• Authors: Eren Dogan, M. Egemen Uzun, Atahan Uz, H. Emre Seyrek, Ahmed Zeer, Ezgi Sevi, H. Toprak Kesgin, M. Kaan Yuce, M. Fatih Amasyali
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2404.17010
• Pdf URL: https://arxiv.org/pdf/2404.17010
• Copy Paste: [[2404.17010]] Türkçe Dil Modellerinin Performans Karşılaştırması Performance Comparison of Turkish Language Models(https://arxiv.org/abs/2404.17010)
• Keywords: in-context
• Abstract: The developments that language models have provided in fulfilling almost all kinds of tasks have attracted the attention of not only researchers but also the society and have enabled them to become products. There are commercially successful language models available. However, users may prefer open-source language models due to cost, data privacy, or regulations. Yet, despite the increasing number of these models, there is no comprehensive comparison of their performance for Turkish. This study aims to fill this gap in the literature. A comparison is made among seven selected language models based on their contextual learning and question-answering abilities. Turkish datasets for contextual learning and question-answering were prepared, and both automatic and human evaluations were conducted. The results show that for question-answering, continuing pretraining before fine-tuning with instructional datasets is more successful in adapting multilingual models to Turkish and that in-context learning performances do not much related to question-answering performances.

Title: Dr-SAM: An End-to-End Framework for Vascular Segmentation, Diameter Estimation, and Anomaly Detection on Angiography Images

• Authors: Vazgen Zohranyan, Vagner Navasardyan, Hayk Navasardyan, Jan Borggrefe, Shant Navasardyan
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2404.17029
• Pdf URL: https://arxiv.org/pdf/2404.17029
• Copy Paste: [[2404.17029]] Dr-SAM: An End-to-End Framework for Vascular Segmentation, Diameter Estimation, and Anomaly Detection on Angiography Images(https://arxiv.org/abs/2404.17029)
• Keywords: anomaly
• Abstract: Recent advancements in AI have significantly transformed medical imaging, particularly in angiography, by enhancing diagnostic precision and patient care. However existing works are limited in analyzing the aorta and iliac arteries, above all for vascular anomaly detection and characterization. To close this gap, we propose Dr-SAM, a comprehensive multi-stage framework for vessel segmentation, diameter estimation, and anomaly analysis aiming to examine the peripheral vessels through angiography images. For segmentation we introduce a customized positive/negative point selection mechanism applied on top of the Segment Anything Model (SAM), specifically for medical (Angiography) images. Then we propose a morphological approach to determine the vessel diameters followed by our histogram-driven anomaly detection approach. Moreover, we introduce a new benchmark dataset for the comprehensive analysis of peripheral vessel angiography images which we hope can boost the upcoming research in this direction leading to enhanced diagnostic precision and ultimately better health outcomes for individuals facing vascular issues.

Title: Auto-Generating Weak Labels for Real & Synthetic Data to Improve Label-Scarce Medical Image Segmentation

• Authors: Tanvi Deshpande, Eva Prakash, Elsie Gyang Ross, Curtis Langlotz, Andrew Ng, Jeya Maria Jose Valanarasu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17033
• Pdf URL: https://arxiv.org/pdf/2404.17033
• Copy Paste: [[2404.17033]] Auto-Generating Weak Labels for Real & Synthetic Data to Improve Label-Scarce Medical Image Segmentation(https://arxiv.org/abs/2404.17033)
• Keywords: foundation model
• Abstract: The high cost of creating pixel-by-pixel gold-standard labels, limited expert availability, and presence of diverse tasks make it challenging to generate segmentation labels to train deep learning models for medical imaging tasks. In this work, we present a new approach to overcome the hurdle of costly medical image labeling by leveraging foundation models like Segment Anything Model (SAM) and its medical alternate MedSAM. Our pipeline has the ability to generate weak labels for any unlabeled medical image and subsequently use it to augment label-scarce datasets. We perform this by leveraging a model trained on a few gold-standard labels and using it to intelligently prompt MedSAM for weak label generation. This automation eliminates the manual prompting step in MedSAM, creating a streamlined process for generating labels for both real and synthetic images, regardless of quantity. We conduct experiments on label-scarce settings for multiple tasks pertaining to modalities ranging from ultrasound, dermatology, and X-rays to demonstrate the usefulness of our pipeline. The code is available at https://github.com/stanfordmlgroup/Auto-Generate-WLs/.

Title: Near to Mid-term Risks and Opportunities of Open Source Generative AI

• Authors: Francisco Eiras, Aleksandar Petrov, Bertie Vidgen, Christian Schroeder de Witt, Fabio Pizzati, Katherine Elkins, Supratik Mukhopadhyay, Adel Bibi, Botos Csaba, Fabro Steibel, Fazl Barez, Genevieve Smith, Gianluca Guadagni, Jon Chun, Jordi Cabot, Joseph Marvin Imperial, Juan A. Nolazco-Flores, Lori Landay, Matthew Jackson, Paul Röttger, Philip H.S. Torr, Trevor Darrell, Yong Suk Lee, Jakob Foerster
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.17047
• Pdf URL: https://arxiv.org/pdf/2404.17047
• Copy Paste: [[2404.17047]] Near to Mid-term Risks and Opportunities of Open Source Generative AI(https://arxiv.org/abs/2404.17047)
• Keywords: generative
• Abstract: In the next few years, applications of Generative AI are expected to revolutionize a number of different areas, ranging from science & medicine to education. The potential for these seismic changes has triggered a lively debate about potential risks and resulted in calls for tighter regulation, in particular from some of the major tech companies who are leading in AI development. This regulation is likely to put at risk the budding field of open source Generative AI. We argue for the responsible open sourcing of generative AI models in the near and medium term. To set the stage, we first introduce an AI openness taxonomy system and apply it to 40 current large language models. We then outline differential benefits and risks of open versus closed source AI and present potential risk mitigation, ranging from best practices to calls for technical and scientific contributions. We hope that this report will add a much needed missing voice to the current public discourse on near to mid-term AI safety and other societal impact.

Title: Unleashing the Potential of Fractional Calculus in Graph Neural Networks with FROND

• Authors: Qiyu Kang, Kai Zhao, Qinxu Ding, Feng Ji, Xuhao Li, Wenfei Liang, Yang Song, Wee Peng Tay
• Subjects: cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2404.17099
• Pdf URL: https://arxiv.org/pdf/2404.17099
• Copy Paste: [[2404.17099]] Unleashing the Potential of Fractional Calculus in Graph Neural Networks with FROND(https://arxiv.org/abs/2404.17099)
• Keywords: diffusion
• Abstract: We introduce the FRactional-Order graph Neural Dynamical network (FROND), a new continuous graph neural network (GNN) framework. Unlike traditional continuous GNNs that rely on integer-order differential equations, FROND employs the Caputo fractional derivative to leverage the non-local properties of fractional calculus. This approach enables the capture of long-term dependencies in feature updates, moving beyond the Markovian update mechanisms in conventional integer-order models and offering enhanced capabilities in graph representation learning. We offer an interpretation of the node feature updating process in FROND from a non-Markovian random walk perspective when the feature updating is particularly governed by a diffusion process. We demonstrate analytically that oversmoothing can be mitigated in this setting. Experimentally, we validate the FROND framework by comparing the fractional adaptations of various established integer-order continuous GNNs, demonstrating their consistently improved performance and underscoring the framework's potential as an effective extension to enhance traditional continuous GNNs. The code is available at \url{https://github.com/zknus/ICLR2024-FROND}.

Title: Synthesizing Iris Images using Generative Adversarial Networks: Survey and Comparative Analysis

• Authors: Shivangi Yadav, Arun Ross
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17105
• Pdf URL: https://arxiv.org/pdf/2404.17105
• Copy Paste: [[2404.17105]] Synthesizing Iris Images using Generative Adversarial Networks: Survey and Comparative Analysis(https://arxiv.org/abs/2404.17105)
• Keywords: generative
• Abstract: Biometric systems based on iris recognition are currently being used in border control applications and mobile devices. However, research in iris recognition is stymied by various factors such as limited datasets of bonafide irides and presentation attack instruments; restricted intra-class variations; and privacy concerns. Some of these issues can be mitigated by the use of synthetic iris data. In this paper, we present a comprehensive review of state-of-the-art GAN-based synthetic iris image generation techniques, evaluating their strengths and limitations in producing realistic and useful iris images that can be used for both training and testing iris recognition systems and presentation attack detectors. In this regard, we first survey the various methods that have been used for synthetic iris generation and specifically consider generators based on StyleGAN, RaSGAN, CIT-GAN, iWarpGAN, StarGAN, etc. We then analyze the images generated by these models for realism, uniqueness, and biometric utility. This comprehensive analysis highlights the pros and cons of various GANs in the context of developing robust iris matchers and presentation attack detectors.

Title: Sensor Response-Time Reduction using Long-Short Term Memory Network Forecasting

• Authors: Simon J. Ward, Muhamed Baljevic, Sharon M. Weiss
• Subjects: cs.LG, eess.SP
• Abstract URL: https://arxiv.org/abs/2404.17144
• Pdf URL: https://arxiv.org/pdf/2404.17144
• Copy Paste: [[2404.17144]] Sensor Response-Time Reduction using Long-Short Term Memory Network Forecasting(https://arxiv.org/abs/2404.17144)
• Keywords: diffusion
• Abstract: The response time of a biosensor is a crucial metric in safety-critical applications such as medical diagnostics where an earlier diagnosis can markedly improve patient outcomes. However, the speed at which a biosensor reaches a final equilibrium state can be limited by poor mass transport and long molecular diffusion times that increase the time it takes target molecules to reach the active sensing region of a biosensor. While optimization of system and sensor design can promote molecules reaching the sensing element faster, a simpler and complementary approach for response time reduction that is widely applicable across all sensor platforms is to use time-series forecasting to predict the ultimate steady-state sensor response. In this work, we show that ensembles of long short-term memory (LSTM) networks can accurately predict equilibrium biosensor response from a small quantity of initial time-dependent biosensor measurements, allowing for significant reduction in response time by a mean and median factor of improvement of 18.6 and 5.1, respectively. The ensemble of models also provides simultaneous estimation of uncertainty, which is vital to provide confidence in the predictions and subsequent safety-related decisions that are made. This approach is demonstrated on real-time experimental data collected by exposing porous silicon biosensors to buffered protein solutions using a multi-channel fluidic cell that enables the automated measurement of 100 porous silicon biosensors in parallel. The dramatic improvement in sensor response time achieved using LSTM network ensembles and associated uncertainty quantification opens the door to trustworthy and faster responding biosensors, enabling more rapid medical diagnostics for improved patient outcomes and healthcare access, as well as quicker identification of toxins in food and the environment.

Title: Neuro-Symbolic Embedding for Short and Effective Feature Selection via Autoregressive Generation

• Authors: Nanxu Gong, Wangyang Ying, Dongjie Wang, Yanjie Fu
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.17157
• Pdf URL: https://arxiv.org/pdf/2404.17157
• Copy Paste: [[2404.17157]] Neuro-Symbolic Embedding for Short and Effective Feature Selection via Autoregressive Generation(https://arxiv.org/abs/2404.17157)
• Keywords: generative
• Abstract: Feature selection aims to identify the optimal feature subset for enhancing downstream models. Effective feature selection can remove redundant features, save computational resources, accelerate the model learning process, and improve the model overall performance. However, existing works are often time-intensive to identify the effective feature subset within high-dimensional feature spaces. Meanwhile, these methods mainly utilize a single downstream task performance as the selection criterion, leading to the selected subsets that are not only redundant but also lack generalizability. To bridge these gaps, we reformulate feature selection through a neuro-symbolic lens and introduce a novel generative framework aimed at identifying short and effective feature subsets. More specifically, we found that feature ID tokens of the selected subset can be formulated as symbols to reflect the intricate correlations among features. Thus, in this framework, we first create a data collector to automatically collect numerous feature selection samples consisting of feature ID tokens, model performance, and the measurement of feature subset redundancy. Building on the collected data, an encoder-decoder-evaluator learning paradigm is developed to preserve the intelligence of feature selection into a continuous embedding space for efficient search. Within the learned embedding space, we leverage a multi-gradient search algorithm to find more robust and generalized embeddings with the objective of improving model performance and reducing feature subset redundancy. These embeddings are then utilized to reconstruct the feature ID tokens for executing the final feature selection. Ultimately, comprehensive experiments and case studies are conducted to validate the effectiveness of the proposed framework.

Title: DPGAN: A Dual-Path Generative Adversarial Network for Missing Data Imputation in Graphs

• Authors: Xindi Zheng, Yuwei Wu, Yu Pan, Wanyu Lin, Lei Ma, Jianjun Zhao
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2404.17164
• Pdf URL: https://arxiv.org/pdf/2404.17164
• Copy Paste: [[2404.17164]] DPGAN: A Dual-Path Generative Adversarial Network for Missing Data Imputation in Graphs(https://arxiv.org/abs/2404.17164)
• Keywords: generative
• Abstract: Missing data imputation poses a paramount challenge when dealing with graph data. Prior works typically are based on feature propagation or graph autoencoders to address this issue. However, these methods usually encounter the over-smoothing issue when dealing with missing data, as the graph neural network (GNN) modules are not explicitly designed for handling missing data. This paper proposes a novel framework, called Dual-Path Generative Adversarial Network (DPGAN), that can deal simultaneously with missing data and avoid over-smoothing problems. The crux of our work is that it admits both global and local representations of the input graph signal, which can capture the long-range dependencies. It is realized via our proposed generator, consisting of two key components, i.e., MLPUNet++ and GraphUNet++. Our generator is trained with a designated discriminator via an adversarial process. In particular, to avoid assessing the entire graph as did in the literature, our discriminator focuses on the local subgraph fidelity, thereby boosting the quality of the local imputation. The subgraph size is adjustable, allowing for control over the intensity of adversarial regularization. Comprehensive experiments across various benchmark datasets substantiate that DPGAN consistently rivals, if not outperforms, existing state-of-the-art imputation algorithms. The code is provided at \url{https://github.com/momoxia/DPGAN}.

Title: MovieChat+: Question-aware Sparse Memory for Long Video Question Answering

• Authors: Enxin Song, Wenhao Chai, Tian Ye, Jenq-Neng Hwang, Xi Li, Gaoang Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17176
• Pdf URL: https://arxiv.org/pdf/2404.17176
• Copy Paste: [[2404.17176]] MovieChat+: Question-aware Sparse Memory for Long Video Question Answering(https://arxiv.org/abs/2404.17176)
• Keywords: foundation model
• Abstract: Recently, integrating video foundation models and large language models to build a video understanding system can overcome the limitations of specific pre-defined vision tasks. Yet, existing methods either employ complex spatial-temporal modules or rely heavily on additional perception models to extract temporal features for video understanding, and they only perform well on short videos. For long videos, the computational complexity and memory costs associated with long-term temporal connections are significantly increased, posing additional challenges.Taking advantage of the Atkinson-Shiffrin memory model, with tokens in Transformers being employed as the carriers of memory in combination with our specially designed memory mechanism, we propose MovieChat to overcome these challenges. We lift pre-trained multi-modal large language models for understanding long videos without incorporating additional trainable temporal modules, employing a zero-shot approach. MovieChat achieves state-of-the-art performance in long video understanding, along with the released MovieChat-1K benchmark with 1K long video, 2K temporal grounding labels, and 14K manual annotations for validation of the effectiveness of our method. The code along with the dataset can be accessed via the following https://github.com/rese1f/MovieChat.

Title: Low-Rank Knowledge Decomposition for Medical Foundation Models

• Authors: Yuhang Zhou, Haolin Li, Siyuan Du, Jiangchao Yao, Ya Zhang, Yanfeng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17184
• Pdf URL: https://arxiv.org/pdf/2404.17184
• Copy Paste: [[2404.17184]] Low-Rank Knowledge Decomposition for Medical Foundation Models(https://arxiv.org/abs/2404.17184)
• Keywords: foundation model
• Abstract: The popularity of large-scale pre-training has promoted the development of medical foundation models. However, some studies have shown that although foundation models exhibit strong general feature extraction capabilities, their performance on specific tasks is still inferior to task-specific methods. In this paper, we explore a new perspective called ``Knowledge Decomposition'' to improve the performance on specific medical tasks, which deconstruct the foundation model into multiple lightweight expert models, each dedicated to a particular task, with the goal of improving specialization while concurrently mitigating resource expenditure. To accomplish the above objective, we design a novel framework named Low-Rank Knowledge Decomposition (LoRKD), which explicitly separates graidents by incorporating low-rank expert modules and the efficient knowledge separation convolution. Extensive experimental results demonstrate that the decomposed models perform well in terms of performance and transferability, even surpassing the original foundation models.

Title: Few-shot Calligraphy Style Learning

• Authors: Fangda Chen, Jiacheng Nie, Lichuan Jiang, Zhuoer Zeng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17199
• Pdf URL: https://arxiv.org/pdf/2404.17199
• Copy Paste: [[2404.17199]] Few-shot Calligraphy Style Learning(https://arxiv.org/abs/2404.17199)
• Keywords: diffusion, generative
• Abstract: We introduced "Presidifussion," a novel approach to learning and replicating the unique style of calligraphy of President Xu, using a pretrained diffusion model adapted through a two-stage training process. Initially, our model is pretrained on a diverse dataset containing works from various calligraphers. This is followed by fine-tuning on a smaller, specialized dataset of President Xu's calligraphy, comprising just under 200 images. Our method introduces innovative techniques of font image conditioning and stroke information conditioning, enabling the model to capture the intricate structural elements of Chinese characters. The effectiveness of our approach is demonstrated through a comparison with traditional methods like zi2zi and CalliGAN, with our model achieving comparable performance using significantly smaller datasets and reduced computational resources. This work not only presents a breakthrough in the digital preservation of calligraphic art but also sets a new standard for data-efficient generative modeling in the domain of cultural heritage digitization.

Title: Self-supervised visual learning in the low-data regime: a comparative evaluation

• Authors: Sotirios Konstantakos, Despina Ioanna Chalkiadaki, Ioannis Mademlis, Yuki M. Asano, Efstratios Gavves, Georgios Th. Papadopoulos
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17202
• Pdf URL: https://arxiv.org/pdf/2404.17202
• Copy Paste: [[2404.17202]] Self-supervised visual learning in the low-data regime: a comparative evaluation(https://arxiv.org/abs/2404.17202)
• Keywords: self-supervised
• Abstract: Self-Supervised Learning (SSL) is a valuable and robust training methodology for contemporary Deep Neural Networks (DNNs), enabling unsupervised pretraining on a `pretext task' that does not require ground-truth labels/annotation. This allows efficient representation learning from massive amounts of unlabeled training data, which in turn leads to increased accuracy in a `downstream task' by exploiting supervised transfer learning. Despite the relatively straightforward conceptualization and applicability of SSL, it is not always feasible to collect and/or to utilize very large pretraining datasets, especially when it comes to real-world application settings. In particular, in cases of specialized and domain-specific application scenarios, it may not be achievable or practical to assemble a relevant image pretraining dataset in the order of millions of instances or it could be computationally infeasible to pretrain at this scale. This motivates an investigation on the effectiveness of common SSL pretext tasks, when the pretraining dataset is of relatively limited/constrained size. In this context, this work introduces a taxonomy of modern visual SSL methods, accompanied by detailed explanations and insights regarding the main categories of approaches, and, subsequently, conducts a thorough comparative experimental evaluation in the low-data regime, targeting to identify: a) what is learnt via low-data SSL pretraining, and b) how do different SSL categories behave in such training scenarios. Interestingly, for domain-specific downstream tasks, in-domain low-data SSL pretraining outperforms the common approach of large-scale pretraining on general datasets. Grounded on the obtained results, valuable insights are highlighted regarding the performance of each category of SSL methods, which in turn suggest straightforward future research directions in the field.

Title: SAGHOG: Self-Supervised Autoencoder for Generating HOG Features for Writer Retrieval

• Authors: Marco Peer, Florian Kleber, Robert Sablatnig
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17221
• Pdf URL: https://arxiv.org/pdf/2404.17221
• Copy Paste: [[2404.17221]] SAGHOG: Self-Supervised Autoencoder for Generating HOG Features for Writer Retrieval(https://arxiv.org/abs/2404.17221)
• Keywords: self-supervised
• Abstract: This paper introduces SAGHOG, a self-supervised pretraining strategy for writer retrieval using HOG features of the binarized input image. Our preprocessing involves the application of the Segment Anything technique to extract handwriting from various datasets, ending up with about 24k documents, followed by training a vision transformer on reconstructing masked patches of the handwriting. SAGHOG is then finetuned by appending NetRVLAD as an encoding layer to the pretrained encoder. Evaluation of our approach on three historical datasets, Historical-WI, HisFrag20, and GRK-Papyri, demonstrates the effectiveness of SAGHOG for writer retrieval. Additionally, we provide ablation studies on our architecture and evaluate un- and supervised finetuning. Notably, on HisFrag20, SAGHOG outperforms related work with a mAP of 57.2 % - a margin of 11.6 % to the current state of the art, showcasing its robustness on challenging data, and is competitive on even small datasets, e.g. GRK-Papyri, where we achieve a Top-1 accuracy of 58.0%.

Title: ObjectAdd: Adding Objects into Image via a Training-Free Diffusion Modification Fashion

• Authors: Ziyue Zhang, Mingbao Lin, Rongrong Ji
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17230
• Pdf URL: https://arxiv.org/pdf/2404.17230
• Copy Paste: [[2404.17230]] ObjectAdd: Adding Objects into Image via a Training-Free Diffusion Modification Fashion(https://arxiv.org/abs/2404.17230)
• Keywords: diffusion
• Abstract: We introduce ObjectAdd, a training-free diffusion modification method to add user-expected objects into user-specified area. The motive of ObjectAdd stems from: first, describing everything in one prompt can be difficult, and second, users often need to add objects into the generated image. To accommodate with real world, our ObjectAdd maintains accurate image consistency after adding objects with technical innovations in: (1) embedding-level concatenation to ensure correct text embedding coalesce; (2) object-driven layout control with latent and attention injection to ensure objects accessing user-specified area; (3) prompted image inpainting in an attention refocusing & object expansion fashion to ensure rest of the image stays the same. With a text-prompted image, our ObjectAdd allows users to specify a box and an object, and achieves: (1) adding object inside the box area; (2) exact content outside the box area; (3) flawless fusion between the two areas

Title: Parameter Efficient Fine-tuning of Self-supervised ViTs without Catastrophic Forgetting

• Authors: Reza Akbarian Bafghi, Nidhin Harilal, Claire Monteleoni, Maziar Raissi
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17245
• Pdf URL: https://arxiv.org/pdf/2404.17245
• Copy Paste: [[2404.17245]] Parameter Efficient Fine-tuning of Self-supervised ViTs without Catastrophic Forgetting(https://arxiv.org/abs/2404.17245)
• Keywords: self-supervised
• Abstract: Artificial neural networks often suffer from catastrophic forgetting, where learning new concepts leads to a complete loss of previously acquired knowledge. We observe that this issue is particularly magnified in vision transformers (ViTs), where post-pre-training and fine-tuning on new tasks can significantly degrade the model's original general abilities. For instance, a DINO ViT-Base/16 pre-trained on ImageNet-1k loses over 70% accuracy on ImageNet-1k after just 10 iterations of fine-tuning on CIFAR-100. Overcoming this stability-plasticity dilemma is crucial for enabling ViTs to continuously learn and adapt to new domains while preserving their initial knowledge. In this work, we study two new parameter-efficient fine-tuning strategies: (1)~Block Expansion, and (2) Low-rank adaptation (LoRA). Our experiments reveal that using either Block Expansion or LoRA on self-supervised pre-trained ViTs surpass fully fine-tuned ViTs in new domains while offering significantly greater parameter efficiency. Notably, we find that Block Expansion experiences only a minimal performance drop in the pre-training domain, thereby effectively mitigating catastrophic forgetting in pre-trained ViTs.

Title: Comparison of self-supervised in-domain and supervised out-domain transfer learning for bird species recognition

• Authors: Houtan Ghaffari, Paul Devos
• Subjects: cs.LG, cs.CV, cs.SD
• Abstract URL: https://arxiv.org/abs/2404.17252
• Pdf URL: https://arxiv.org/pdf/2404.17252
• Copy Paste: [[2404.17252]] Comparison of self-supervised in-domain and supervised out-domain transfer learning for bird species recognition(https://arxiv.org/abs/2404.17252)
• Keywords: self-supervised
• Abstract: Transferring the weights of a pre-trained model to assist another task has become a crucial part of modern deep learning, particularly in data-scarce scenarios. Pre-training refers to the initial step of training models outside the current task of interest, typically on another dataset. It can be done via supervised models using human-annotated datasets or self-supervised models trained on unlabeled datasets. In both cases, many pre-trained models are available to fine-tune for the task of interest. Interestingly, research has shown that pre-trained models from ImageNet can be helpful for audio tasks despite being trained on image datasets. Hence, it's unclear whether in-domain models would be advantageous compared to competent out-domain models, such as convolutional neural networks from ImageNet. Our experiments will demonstrate the usefulness of in-domain models and datasets for bird species recognition by leveraging VICReg, a recent and powerful self-supervised method.

Title: Trinity Detector:text-assisted and attention mechanisms based spectral fusion for diffusion generation image detection

• Authors: Jiawei Song, Dengpan Ye, Yunming Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17254
• Pdf URL: https://arxiv.org/pdf/2404.17254
• Copy Paste: [[2404.17254]] Trinity Detector:text-assisted and attention mechanisms based spectral fusion for diffusion generation image detection(https://arxiv.org/abs/2404.17254)
• Keywords: diffusion
• Abstract: Artificial Intelligence Generated Content (AIGC) techniques, represented by text-to-image generation, have led to a malicious use of deep forgeries, raising concerns about the trustworthiness of multimedia content. Adapting traditional forgery detection methods to diffusion models proves challenging. Thus, this paper proposes a forgery detection method explicitly designed for diffusion models called Trinity Detector. Trinity Detector incorporates coarse-grained text features through a CLIP encoder, coherently integrating them with fine-grained artifacts in the pixel domain for comprehensive multimodal detection. To heighten sensitivity to diffusion-generated image features, a Multi-spectral Channel Attention Fusion Unit (MCAF) is designed, extracting spectral inconsistencies through adaptive fusion of diverse frequency bands and further integrating spatial co-occurrence of the two modalities. Extensive experimentation validates that our Trinity Detector method outperforms several state-of-the-art methods, our performance is competitive across all datasets and up to 17.6\% improvement in transferability in the diffusion datasets.

Title: Metronome: tracing variation in poetic meters via local sequence alignment

• Authors: Ben Nagy, Artjoms Šeļa, Mirella De Sisto, Petr Plecháč
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2404.17337
• Pdf URL: https://arxiv.org/pdf/2404.17337
• Copy Paste: [[2404.17337]] Metronome: tracing variation in poetic meters via local sequence alignment(https://arxiv.org/abs/2404.17337)
• Keywords: diffusion
• Abstract: All poetic forms come from somewhere. Prosodic templates can be copied for generations, altered by individuals, imported from foreign traditions, or fundamentally changed under the pressures of language evolution. Yet these relationships are notoriously difficult to trace across languages and times. This paper introduces an unsupervised method for detecting structural similarities in poems using local sequence alignment. The method relies on encoding poetic texts as strings of prosodic features using a four-letter alphabet; these sequences are then aligned to derive a distance measure based on weighted symbol (mis)matches. Local alignment allows poems to be clustered according to emergent properties of their underlying prosodic patterns. We evaluate method performance on a meter recognition tasks against strong baselines and show its potential for cross-lingual and historical research using three short case studies: 1) mutations in quantitative meter in classical Latin, 2) European diffusion of the Renaissance hendecasyllable, and 3) comparative alignment of modern meters in 18--19th century Czech, German and Russian. We release an implementation of the algorithm as a Python package with an open license.

Title: UniRGB-IR: A Unified Framework for Visible-Infrared Downstream Tasks via Adapter Tuning

• Authors: Maoxun Yuan, Bo Cui, Tianyi Zhao, Xingxing Wei
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17360
• Pdf URL: https://arxiv.org/pdf/2404.17360
• Copy Paste: [[2404.17360]] UniRGB-IR: A Unified Framework for Visible-Infrared Downstream Tasks via Adapter Tuning(https://arxiv.org/abs/2404.17360)
• Keywords: foundation model
• Abstract: Semantic analysis on visible (RGB) and infrared (IR) images has gained attention for its ability to be more accurate and robust under low-illumination and complex weather conditions. Due to the lack of pre-trained foundation models on the large-scale infrared image datasets, existing methods prefer to design task-specific frameworks and directly fine-tune them with pre-trained foundation models on their RGB-IR semantic relevance datasets, which results in poor scalability and limited generalization. In this work, we propose a scalable and efficient framework called UniRGB-IR to unify RGB-IR downstream tasks, in which a novel adapter is developed to efficiently introduce richer RGB-IR features into the pre-trained RGB-based foundation model. Specifically, our framework consists of a vision transformer (ViT) foundation model, a Multi-modal Feature Pool (MFP) module and a Supplementary Feature Injector (SFI) module. The MFP and SFI modules cooperate with each other as an adpater to effectively complement the ViT features with the contextual multi-scale features. During training process, we freeze the entire foundation model to inherit prior knowledge and only optimize the MFP and SFI modules. Furthermore, to verify the effectiveness of our framework, we utilize the ViT-Base as the pre-trained foundation model to perform extensive experiments. Experimental results on various RGB-IR downstream tasks demonstrate that our method can achieve state-of-the-art performance. The source code and results are available at https://github.com/PoTsui99/UniRGB-IR.git.

Title: MV-VTON: Multi-View Virtual Try-On with Diffusion Models

• Authors: Haoyu Wang, Zhilu Zhang, Donglin Di, Shiliang Zhang, Wangmeng Zuo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17364
• Pdf URL: https://arxiv.org/pdf/2404.17364
• Copy Paste: [[2404.17364]] MV-VTON: Multi-View Virtual Try-On with Diffusion Models(https://arxiv.org/abs/2404.17364)
• Keywords: diffusion
• Abstract: The goal of image-based virtual try-on is to generate an image of the target person naturally wearing the given clothing. However, most existing methods solely focus on the frontal try-on using the frontal clothing. When the views of the clothing and person are significantly inconsistent, particularly when the person's view is non-frontal, the results are unsatisfactory. To address this challenge, we introduce Multi-View Virtual Try-ON (MV-VTON), which aims to reconstruct the dressing results of a person from multiple views using the given clothes. On the one hand, given that single-view clothes provide insufficient information for MV-VTON, we instead employ two images, i.e., the frontal and back views of the clothing, to encompass the complete view as much as possible. On the other hand, the diffusion models that have demonstrated superior abilities are adopted to perform our MV-VTON. In particular, we propose a view-adaptive selection method where hard-selection and soft-selection are applied to the global and local clothing feature extraction, respectively. This ensures that the clothing features are roughly fit to the person's view. Subsequently, we suggest a joint attention block to align and fuse clothing features with person features. Additionally, we collect a MV-VTON dataset, i.e., Multi-View Garment (MVG), in which each person has multiple photos with diverse views and poses. Experiments show that the proposed method not only achieves state-of-the-art results on MV-VTON task using our MVG dataset, but also has superiority on frontal-view virtual try-on task using VITON-HD and DressCode datasets. Codes and datasets will be publicly released at https://github.com/hywang2002/MV-VTON .

Title: Frequency-Guided Multi-Level Human Action Anomaly Detection with Normalizing Flows

• Authors: Shun Maeda, Chunzhi Gu, Jun Yu, Shogo Tokai, Shangce Gao, Chao Zhang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17381
• Pdf URL: https://arxiv.org/pdf/2404.17381
• Copy Paste: [[2404.17381]] Frequency-Guided Multi-Level Human Action Anomaly Detection with Normalizing Flows(https://arxiv.org/abs/2404.17381)
• Keywords: anomaly
• Abstract: We introduce the task of human action anomaly detection (HAAD), which aims to identify anomalous motions in an unsupervised manner given only the pre-determined normal category of training action samples. Compared to prior human-related anomaly detection tasks which primarily focus on unusual events from videos, HAAD involves the learning of specific action labels to recognize semantically anomalous human behaviors. To address this task, we propose a normalizing flow (NF)-based detection framework where the sample likelihood is effectively leveraged to indicate anomalies. As action anomalies often occur in some specific body parts, in addition to the full-body action feature learning, we incorporate extra encoding streams into our framework for a finer modeling of body subsets. Our framework is thus multi-level to jointly discover global and local motion anomalies. Furthermore, to show awareness of the potentially jittery data during recording, we resort to discrete cosine transformation by converting the action samples from the temporal to the frequency domain to mitigate the issue of data instability. Extensive experimental results on two human action datasets demonstrate that our method outperforms the baselines formed by adapting state-of-the-art human activity AD approaches to our task of HAAD.

Title: Multi-view Image Prompted Multi-view Diffusion for Improved 3D Generation

• Authors: Seungwook Kim, Yichun Shi, Kejie Li, Minsu Cho, Peng Wang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17419
• Pdf URL: https://arxiv.org/pdf/2404.17419
• Copy Paste: [[2404.17419]] Multi-view Image Prompted Multi-view Diffusion for Improved 3D Generation(https://arxiv.org/abs/2404.17419)
• Keywords: diffusion
• Abstract: Using image as prompts for 3D generation demonstrate particularly strong performances compared to using text prompts alone, for images provide a more intuitive guidance for the 3D generation process. In this work, we delve into the potential of using multiple image prompts, instead of a single image prompt, for 3D generation. Specifically, we build on ImageDream, a novel image-prompt multi-view diffusion model, to support multi-view images as the input prompt. Our method, dubbed MultiImageDream, reveals that transitioning from a single-image prompt to multiple-image prompts enhances the performance of multi-view and 3D object generation according to various quantitative evaluation metrics and qualitative assessments. This advancement is achieved without the necessity of fine-tuning the pre-trained ImageDream multi-view diffusion model.

Title: Domain Adaptive and Fine-grained Anomaly Detection for Single-cell Sequencing Data and Beyond

• Authors: Kaichen Xu, Yueyang Ding, Suyang Hou, Weiqiang Zhan, Nisang Chen, Jun Wang, Xiaobo Sun
• Subjects: cs.LG, cs.AI, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2404.17454
• Pdf URL: https://arxiv.org/pdf/2404.17454
• Copy Paste: [[2404.17454]] Domain Adaptive and Fine-grained Anomaly Detection for Single-cell Sequencing Data and Beyond(https://arxiv.org/abs/2404.17454)
• Keywords: generative, anomaly
• Abstract: Fined-grained anomalous cell detection from affected tissues is critical for clinical diagnosis and pathological research. Single-cell sequencing data provide unprecedented opportunities for this task. However, current anomaly detection methods struggle to handle domain shifts prevalent in multi-sample and multi-domain single-cell sequencing data, leading to suboptimal performance. Moreover, these methods fall short of distinguishing anomalous cells into pathologically distinct subtypes. In response, we propose ACSleuth, a novel, reconstruction deviation-guided generative framework that integrates the detection, domain adaptation, and fine-grained annotating of anomalous cells into a methodologically cohesive workflow. Notably, we present the first theoretical analysis of using reconstruction deviations output by generative models for anomaly detection in lieu of domain shifts. This analysis informs us to develop a novel and superior maximum mean discrepancy-based anomaly scorer in ACSleuth. Extensive benchmarks over various single-cell data and other types of tabular data demonstrate ACSleuth's superiority over the state-of-the-art methods in identifying and subtyping anomalies in multi-sample and multi-domain contexts. Our code is available at https://github.com/Catchxu/ACsleuth.

Title: TextGaze: Gaze-Controllable Face Generation with Natural Language

• Authors: Hengfei Wang, Zhongqun Zhang, Yihua Cheng, Hyung Jin Chang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17486
• Pdf URL: https://arxiv.org/pdf/2404.17486
• Copy Paste: [[2404.17486]] TextGaze: Gaze-Controllable Face Generation with Natural Language(https://arxiv.org/abs/2404.17486)
• Keywords: diffusion
• Abstract: Generating face image with specific gaze information has attracted considerable attention. Existing approaches typically input gaze values directly for face generation, which is unnatural and requires annotated gaze datasets for training, thereby limiting its application. In this paper, we present a novel gaze-controllable face generation task. Our approach inputs textual descriptions that describe human gaze and head behavior and generates corresponding face images. Our work first introduces a text-of-gaze dataset containing over 90k text descriptions spanning a dense distribution of gaze and head poses. We further propose a gaze-controllable text-to-face method. Our method contains a sketch-conditioned face diffusion module and a model-based sketch diffusion module. We define a face sketch based on facial landmarks and eye segmentation map. The face diffusion module generates face images from the face sketch, and the sketch diffusion module employs a 3D face model to generate face sketch from text description. Experiments on the FFHQ dataset show the effectiveness of our method. We will release our dataset and code for future research.

Title: HYPE: Hyperbolic Entailment Filtering for Underspecified Images and Texts

• Authors: Wonjae Kim, Sanghyuk Chun, Taekyung Kim, Dongyoon Han, Sangdoo Yun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17507
• Pdf URL: https://arxiv.org/pdf/2404.17507
• Copy Paste: [[2404.17507]] HYPE: Hyperbolic Entailment Filtering for Underspecified Images and Texts(https://arxiv.org/abs/2404.17507)
• Keywords: self-supervised
• Abstract: In an era where the volume of data drives the effectiveness of self-supervised learning, the specificity and clarity of data semantics play a crucial role in model training. Addressing this, we introduce HYPerbolic Entailment filtering (HYPE), a novel methodology designed to meticulously extract modality-wise meaningful and well-aligned data from extensive, noisy image-text pair datasets. Our approach leverages hyperbolic embeddings and the concept of entailment cones to evaluate and filter out samples with meaningless or underspecified semantics, focusing on enhancing the specificity of each data sample. HYPE not only demonstrates a significant improvement in filtering efficiency but also sets a new state-of-the-art in the DataComp benchmark when combined with existing filtering techniques. This breakthrough showcases the potential of HYPE to refine the data selection process, thereby contributing to the development of more accurate and efficient self-supervised learning models. Additionally, the image specificity $\epsilon_{i}$ can be independently applied to induce an image-only dataset from an image-text or image-only data pool for training image-only self-supervised models and showed superior performance when compared to the dataset induced by CLIP score.

Title: Exploring the Distinctiveness and Fidelity of the Descriptions Generated by Large Vision-Language Models

• Authors: Yuhang Huang, Zihan Wu, Chongyang Gao, Jiawei Peng, Xu Yang
• Subjects: cs.CV, cs.MM
• Abstract URL: https://arxiv.org/abs/2404.17534
• Pdf URL: https://arxiv.org/pdf/2404.17534
• Copy Paste: [[2404.17534]] Exploring the Distinctiveness and Fidelity of the Descriptions Generated by Large Vision-Language Models(https://arxiv.org/abs/2404.17534)
• Keywords: generative
• Abstract: Large Vision-Language Models (LVLMs) are gaining traction for their remarkable ability to process and integrate visual and textual data. Despite their popularity, the capacity of LVLMs to generate precise, fine-grained textual descriptions has not been fully explored. This study addresses this gap by focusing on \textit{distinctiveness} and \textit{fidelity}, assessing how models like Open-Flamingo, IDEFICS, and MiniGPT-4 can distinguish between similar objects and accurately describe visual features. We proposed the Textual Retrieval-Augmented Classification (TRAC) framework, which, by leveraging its generative capabilities, allows us to delve deeper into analyzing fine-grained visual description generation. This research provides valuable insights into the generation quality of LVLMs, enhancing the understanding of multimodal language models. Notably, MiniGPT-4 stands out for its better ability to generate fine-grained descriptions, outperforming the other two models in this aspect. The code is provided at \url{https://anonymous.4open.science/r/Explore_FGVDs-E277}.

Title: MaPa: Text-driven Photorealistic Material Painting for 3D Shapes

• Authors: Shangzhan Zhang, Sida Peng, Tao Xu, Yuanbo Yang, Tianrun Chen, Nan Xue, Yujun Shen, Hujun Bao, Ruizhen Hu, Xiaowei Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17569
• Pdf URL: https://arxiv.org/pdf/2404.17569
• Copy Paste: [[2404.17569]] MaPa: Text-driven Photorealistic Material Painting for 3D Shapes(https://arxiv.org/abs/2404.17569)
• Keywords: diffusion, generative
• Abstract: This paper aims to generate materials for 3D meshes from text descriptions. Unlike existing methods that synthesize texture maps, we propose to generate segment-wise procedural material graphs as the appearance representation, which supports high-quality rendering and provides substantial flexibility in editing. Instead of relying on extensive paired data, i.e., 3D meshes with material graphs and corresponding text descriptions, to train a material graph generative model, we propose to leverage the pre-trained 2D diffusion model as a bridge to connect the text and material graphs. Specifically, our approach decomposes a shape into a set of segments and designs a segment-controlled diffusion model to synthesize 2D images that are aligned with mesh parts. Based on generated images, we initialize parameters of material graphs and fine-tune them through the differentiable rendering module to produce materials in accordance with the textual description. Extensive experiments demonstrate the superior performance of our framework in photorealism, resolution, and editability over existing methods. Project page: https://zhanghe3z.github.io/MaPa/

Title: Tunnel Try-on: Excavating Spatial-temporal Tunnels for High-quality Virtual Try-on in Videos

• Authors: Zhengze Xu, Mengting Chen, Zhao Wang, Linyu Xing, Zhonghua Zhai, Nong Sang, Jinsong Lan, Shuai Xiao, Changxin Gao
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2404.17571
• Pdf URL: https://arxiv.org/pdf/2404.17571
• Copy Paste: [[2404.17571]] Tunnel Try-on: Excavating Spatial-temporal Tunnels for High-quality Virtual Try-on in Videos(https://arxiv.org/abs/2404.17571)
• Keywords: diffusion
• Abstract: Video try-on is a challenging task and has not been well tackled in previous works. The main obstacle lies in preserving the details of the clothing and modeling the coherent motions simultaneously. Faced with those difficulties, we address video try-on by proposing a diffusion-based framework named "Tunnel Try-on." The core idea is excavating a "focus tunnel" in the input video that gives close-up shots around the clothing regions. We zoom in on the region in the tunnel to better preserve the fine details of the clothing. To generate coherent motions, we first leverage the Kalman filter to construct smooth crops in the focus tunnel and inject the position embedding of the tunnel into attention layers to improve the continuity of the generated videos. In addition, we develop an environment encoder to extract the context information outside the tunnels as supplementary cues. Equipped with these techniques, Tunnel Try-on keeps the fine details of the clothing and synthesizes stable and smooth videos. Demonstrating significant advancements, Tunnel Try-on could be regarded as the first attempt toward the commercial-level application of virtual try-on in videos.