diffusion

Title: Learning A Coarse-to-Fine Diffusion Transformer for Image Restoration. (arXiv:2308.08730v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08730
• Code URL: https://github.com/wlydlut/c2f-dft
• Copy Paste: [[2308.08730]] Learning A Coarse-to-Fine Diffusion Transformer for Image Restoration(http://arxiv.org/abs/2308.08730)
• Summary:

  Recent years have witnessed the remarkable performance of diffusion models in various vision tasks. However, for image restoration that aims to recover clear images with sharper details from given degraded observations, diffusion-based methods may fail to recover promising results due to inaccurate noise estimation. Moreover, simple constraining noises cannot effectively learn complex degradation information, which subsequently hinders the model capacity. To solve the above problems, we propose a coarse-to-fine diffusion Transformer (C2F-DFT) for image restoration. Specifically, our C2F-DFT contains diffusion self-attention (DFSA) and diffusion feed-forward network (DFN) within a new coarse-to-fine training scheme. The DFSA and DFN respectively capture the long-range diffusion dependencies and learn hierarchy diffusion representation to facilitate better restoration. In the coarse training stage, our C2F-DFT estimates noises and then generates the final clean image by a sampling algorithm. To further improve the restoration quality, we propose a simple yet effective fine training scheme. It first exploits the coarse-trained diffusion model with fixed steps to generate restoration results, which then would be constrained with corresponding ground-truth ones to optimize the models to remedy the unsatisfactory results affected by inaccurate noise estimation. Extensive experiments show that C2F-DFT significantly outperforms diffusion-based restoration method IR-SDE and achieves competitive performance compared with Transformer-based state-of-the-art methods on $3$ tasks, including deraining, deblurring, and real denoising.

Title: Watch Your Steps: Local Image and Scene Editing by Text Instructions. (arXiv:2308.08947v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08947
• Code URL: null
• Copy Paste: [[2308.08947]] Watch Your Steps: Local Image and Scene Editing by Text Instructions(http://arxiv.org/abs/2308.08947)
• Summary:

  Denoising diffusion models have enabled high-quality image generation and editing. We present a method to localize the desired edit region implicit in a text instruction. We leverage InstructPix2Pix (IP2P) and identify the discrepancy between IP2P predictions with and without the instruction. This discrepancy is referred to as the relevance map. The relevance map conveys the importance of changing each pixel to achieve the edits, and is used to to guide the modifications. This guidance ensures that the irrelevant pixels remain unchanged. Relevance maps are further used to enhance the quality of text-guided editing of 3D scenes in the form of neural radiance fields. A field is trained on relevance maps of training views, denoted as the relevance field, defining the 3D region within which modifications should be made. We perform iterative updates on the training views guided by rendered relevance maps from the relevance field. Our method achieves state-of-the-art performance on both image and NeRF editing tasks. Project page: https://ashmrz.github.io/WatchYourSteps/

Title: Enhancing Phrase Representation by Information Bottleneck Guided Text Diffusion Process for Keyphrase Extraction. (arXiv:2308.08739v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.08739
• Code URL: null
• Copy Paste: [[2308.08739]] Enhancing Phrase Representation by Information Bottleneck Guided Text Diffusion Process for Keyphrase Extraction(http://arxiv.org/abs/2308.08739)
• Summary:

  Keyphrase extraction (KPE) is an important task in Natural Language Processing for many scenarios, which aims to extract keyphrases that are present in a given document. Many existing supervised methods treat KPE as sequential labeling, span-level classification, or generative tasks. However, these methods lack the ability to utilize keyphrase information, which may result in biased results. In this study, we propose Diff-KPE, which leverages the supervised Variational Information Bottleneck (VIB) to guide the text diffusion process for generating enhanced keyphrase representations. Diff-KPE first generates the desired keyphrase embeddings conditioned on the entire document and then injects the generated keyphrase embeddings into each phrase representation. A ranking network and VIB are then optimized together with rank loss and classification loss, respectively. This design of Diff-KPE allows us to rank each candidate phrase by utilizing both the information of keyphrases and the document. Experiments show that Diff-KPE outperforms existing KPE methods on a large open domain keyphrase extraction benchmark, OpenKP, and a scientific domain dataset, KP20K.

self-supervised

Title: A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation. (arXiv:2308.08849v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08849
• Code URL: https://github.com/wentaol86/awesome-body-language
• Copy Paste: [[2308.08849]] A Survey on Deep Multi-modal Learning for Body Language Recognition and Generation(http://arxiv.org/abs/2308.08849)
• Summary:

  Body language (BL) refers to the non-verbal communication expressed through physical movements, gestures, facial expressions, and postures. It is a form of communication that conveys information, emotions, attitudes, and intentions without the use of spoken or written words. It plays a crucial role in interpersonal interactions and can complement or even override verbal communication. Deep multi-modal learning techniques have shown promise in understanding and analyzing these diverse aspects of BL. The survey emphasizes their applications to BL generation and recognition. Several common BLs are considered i.e., Sign Language (SL), Cued Speech (CS), Co-speech (CoS), and Talking Head (TH), and we have conducted an analysis and established the connections among these four BL for the first time. Their generation and recognition often involve multi-modal approaches. Benchmark datasets for BL research are well collected and organized, along with the evaluation of SOTA methods on these datasets. The survey highlights challenges such as limited labeled data, multi-modal learning, and the need for domain adaptation to generalize models to unseen speakers or languages. Future research directions are presented, including exploring self-supervised learning techniques, integrating contextual information from other modalities, and exploiting large-scale pre-trained multi-modal models. In summary, this survey paper provides a comprehensive understanding of deep multi-modal learning for various BL generations and recognitions for the first time. By analyzing advancements, challenges, and future directions, it serves as a valuable resource for researchers and practitioners in advancing this field. n addition, we maintain a continuously updated paper list for deep multi-modal learning for BL recognition and generation: https://github.com/wentaoL86/awesome-body-language.

Title: SRMAE: Masked Image Modeling for Scale-Invariant Deep Representations. (arXiv:2308.08884v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08884
• Code URL: null
• Copy Paste: [[2308.08884]] SRMAE: Masked Image Modeling for Scale-Invariant Deep Representations(http://arxiv.org/abs/2308.08884)
• Summary:

  Due to the prevalence of scale variance in nature images, we propose to use image scale as a self-supervised signal for Masked Image Modeling (MIM). Our method involves selecting random patches from the input image and downsampling them to a low-resolution format. Our framework utilizes the latest advances in super-resolution (SR) to design the prediction head, which reconstructs the input from low-resolution clues and other patches. After 400 epochs of pre-training, our Super Resolution Masked Autoencoders (SRMAE) get an accuracy of 82.1% on the ImageNet-1K task. Image scale signal also allows our SRMAE to capture scale invariance representation. For the very low resolution (VLR) recognition task, our model achieves the best performance, surpassing DeriveNet by 1.3%. Our method also achieves an accuracy of 74.84% on the task of recognizing low-resolution facial expressions, surpassing the current state-of-the-art FMD by 9.48%.

Title: Identity-Seeking Self-Supervised Representation Learning for Generalizable Person Re-identification. (arXiv:2308.08887v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08887
• Code URL: https://github.com/dcp15/isr_iccv2023_oral
• Copy Paste: [[2308.08887]] Identity-Seeking Self-Supervised Representation Learning for Generalizable Person Re-identification(http://arxiv.org/abs/2308.08887)
• Summary:

  This paper aims to learn a domain-generalizable (DG) person re-identification (ReID) representation from large-scale videos \textbf{without any annotation}. Prior DG ReID methods employ limited labeled data for training due to the high cost of annotation, which restricts further advances. To overcome the barriers of data and annotation, we propose to utilize large-scale unsupervised data for training. The key issue lies in how to mine identity information. To this end, we propose an Identity-seeking Self-supervised Representation learning (ISR) method. ISR constructs positive pairs from inter-frame images by modeling the instance association as a maximum-weight bipartite matching problem. A reliability-guided contrastive loss is further presented to suppress the adverse impact of noisy positive pairs, ensuring that reliable positive pairs dominate the learning process. The training cost of ISR scales approximately linearly with the data size, making it feasible to utilize large-scale data for training. The learned representation exhibits superior generalization ability. \textbf{Without human annotation and fine-tuning, ISR achieves 87.0\% Rank-1 on Market-1501 and 56.4\% Rank-1 on MSMT17}, outperforming the best supervised domain-generalizable method by 5.0\% and 19.5\%, respectively. In the pre-training$\rightarrow$fine-tuning scenario, ISR achieves state-of-the-art performance, with 88.4\% Rank-1 on MSMT17. The code is at \url{https://github.com/dcp15/ISR_ICCV2023_Oral}.

Title: BIOptimus: Pre-training an Optimal Biomedical Language Model with Curriculum Learning for Named Entity Recognition. (arXiv:2308.08625v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.08625
• Code URL: https://github.com/rttl-ai/bioptimus
• Copy Paste: [[2308.08625]] BIOptimus: Pre-training an Optimal Biomedical Language Model with Curriculum Learning for Named Entity Recognition(http://arxiv.org/abs/2308.08625)
• Summary:

  Using language models (LMs) pre-trained in a self-supervised setting on large corpora and then fine-tuning for a downstream task has helped to deal with the problem of limited label data for supervised learning tasks such as Named Entity Recognition (NER). Recent research in biomedical language processing has offered a number of biomedical LMs pre-trained using different methods and techniques that advance results on many BioNLP tasks, including NER. However, there is still a lack of a comprehensive comparison of pre-training approaches that would work more optimally in the biomedical domain. This paper aims to investigate different pre-training methods, such as pre-training the biomedical LM from scratch and pre-training it in a continued fashion. We compare existing methods with our proposed pre-training method of initializing weights for new tokens by distilling existing weights from the BERT model inside the context where the tokens were found. The method helps to speed up the pre-training stage and improve performance on NER. In addition, we compare how masking rate, corruption strategy, and masking strategies impact the performance of the biomedical LM. Finally, using the insights from our experiments, we introduce a new biomedical LM (BIOptimus), which is pre-trained using Curriculum Learning (CL) and contextualized weight distillation method. Our model sets new states of the art on several biomedical Named Entity Recognition (NER) tasks. We release our code and all pre-trained models

foundation model

Title: SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation. (arXiv:2308.08746v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08746
• Code URL: https://github.com/wenxi-yue/surgicalsam
• Copy Paste: [[2308.08746]] SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation(http://arxiv.org/abs/2308.08746)
• Summary:

  The Segment Anything Model (SAM) is a powerful foundation model that has revolutionised image segmentation. To apply SAM to surgical instrument segmentation, a common approach is to locate precise points or boxes of instruments and then use them as prompts for SAM in a zero-shot manner. However, we observe two problems with this naive pipeline: (1) the domain gap between natural objects and surgical instruments leads to poor generalisation of SAM; and (2) SAM relies on precise point or box locations for accurate segmentation, requiring either extensive manual guidance or a well-performing specialist detector for prompt preparation, which leads to a complex multi-stage pipeline. To address these problems, we introduce SurgicalSAM, a novel end-to-end efficient-tuning approach for SAM to effectively integrate surgical-specific information with SAM's pre-trained knowledge for improved generalisation. Specifically, we propose a lightweight prototype-based class prompt encoder for tuning, which directly generates prompt embeddings from class prototypes and eliminates the use of explicit prompts for improved robustness and a simpler pipeline. In addition, to address the low inter-class variance among surgical instrument categories, we propose contrastive prototype learning, further enhancing the discrimination of the class prototypes for more accurate class prompting. The results of extensive experiments on both EndoVis2018 and EndoVis2017 datasets demonstrate that SurgicalSAM achieves state-of-the-art performance while only requiring a small number of tunable parameters. The source code will be released at https://github.com/wenxi-yue/SurgicalSAM.

generative

Title: Fair GANs through model rebalancing with synthetic data. (arXiv:2308.08638v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08638
• Code URL: null
• Copy Paste: [[2308.08638]] Fair GANs through model rebalancing with synthetic data(http://arxiv.org/abs/2308.08638)
• Summary:

  Deep generative models require large amounts of training data. This often poses a problem as the collection of datasets can be expensive and difficult, in particular datasets that are representative of the appropriate underlying distribution (e.g. demographic). This introduces biases in datasets which are further propagated in the models. We present an approach to mitigate biases in an existing generative adversarial network by rebalancing the model distribution. We do so by generating balanced data from an existing unbalanced deep generative model using latent space exploration and using this data to train a balanced generative model. Further, we propose a bias mitigation loss function that shows improvements in the fairness metric even when trained with unbalanced datasets. We show results for the Stylegan2 models while training on the FFHQ dataset for racial fairness and see that the proposed approach improves on the fairness metric by almost 5 times, whilst maintaining image quality. We further validate our approach by applying it to an imbalanced Cifar-10 dataset. Lastly, we argue that the traditionally used image quality metrics such as Frechet inception distance (FID) are unsuitable for bias mitigation problems.

Title: Fast Inference and Update of Probabilistic Density Estimation on Trajectory Prediction. (arXiv:2308.08824v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08824
• Code URL: https://github.com/meaten/flowchain-iccv2023
• Copy Paste: [[2308.08824]] Fast Inference and Update of Probabilistic Density Estimation on Trajectory Prediction(http://arxiv.org/abs/2308.08824)
• Summary:

  Safety-critical applications such as autonomous vehicles and social robots require fast computation and accurate probability density estimation on trajectory prediction. To address both requirements, this paper presents a new normalizing flow-based trajectory prediction model named FlowChain. FlowChain is a stack of conditional continuously-indexed flows (CIFs) that are expressive and allow analytical probability density computation. This analytical computation is faster than the generative models that need additional approximations such as kernel density estimation. Moreover, FlowChain is more accurate than the Gaussian mixture-based models due to fewer assumptions on the estimated density. FlowChain also allows a rapid update of estimated probability densities. This update is achieved by adopting the \textit{newest observed position} and reusing the flow transformations and its log-det-jacobians that represent the \textit{motion trend}. This update is completed in less than one millisecond because this reuse greatly omits the computational cost. Experimental results showed our FlowChain achieved state-of-the-art trajectory prediction accuracy compared to previous methods. Furthermore, our FlowChain demonstrated superiority in the accuracy and speed of density estimation. Our code is available at \url{https://github.com/meaten/FlowChain-ICCV2023}

Title: Reinforced Self-Training (ReST) for Language Modeling. (arXiv:2308.08998v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.08998
• Code URL: null
• Copy Paste: [[2308.08998]] Reinforced Self-Training (ReST) for Language Modeling(http://arxiv.org/abs/2308.08998)
• Summary:

  Reinforcement learning from human feedback (RLHF) can improve the quality of large language model's (LLM) outputs by aligning them with human preferences. We propose a simple algorithm for aligning LLMs with human preferences inspired by growing batch reinforcement learning (RL), which we call Reinforced Self-Training (ReST). Given an initial LLM policy, ReST produces a dataset by generating samples from the policy, which are then used to improve the LLM policy using offline RL algorithms. ReST is more efficient than typical online RLHF methods because the training dataset is produced offline, which allows data reuse. While ReST is a general approach applicable to all generative learning settings, we focus on its application to machine translation. Our results show that ReST can substantially improve translation quality, as measured by automated metrics and human evaluation on machine translation benchmarks in a compute and sample-efficient manner.

Title: An Effective Deep Learning Based Multi-Class Classification of DoS and DDoS Attack Detection. (arXiv:2308.08803v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.08803
• Code URL: null
• Copy Paste: [[2308.08803]] An Effective Deep Learning Based Multi-Class Classification of DoS and DDoS Attack Detection(http://arxiv.org/abs/2308.08803)
• Summary:

  In the past few years, cybersecurity is becoming very important due to the rise in internet users. The internet attacks such as Denial of service (DoS) and Distributed Denial of Service (DDoS) attacks severely harm a website or server and make them unavailable to other users. Network Monitoring and control systems have found it challenging to identify the many classes of DoS and DDoS attacks since each operates uniquely. Hence a powerful technique is required for attack detection. Traditional machine learning techniques are inefficient in handling extensive network data and cannot extract high-level features for attack detection. Therefore, an effective deep learning-based intrusion detection system is developed in this paper for DoS and DDoS attack classification. This model includes various phases and starts with the Deep Convolutional Generative Adversarial Networks (DCGAN) based technique to address the class imbalance issue in the dataset. Then a deep learning algorithm based on ResNet-50 extracts the critical features for each class in the dataset. After that, an optimized AlexNet-based classifier is implemented for detecting the attacks separately, and the essential parameters of the classifier are optimized using the Atom search optimization algorithm. The proposed approach was evaluated on benchmark datasets, CCIDS2019 and UNSW-NB15, using key classification metrics and achieved 99.37% accuracy for the UNSW-NB15 dataset and 99.33% for the CICIDS2019 dataset. The investigational results demonstrate that the suggested approach performs superior to other competitive techniques in identifying DoS and DDoS attacks.

anomaly

Title: Improving Anomaly Segmentation with Multi-Granularity Cross-Domain Alignment. (arXiv:2308.08696v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08696
• Code URL: null
• Copy Paste: [[2308.08696]] Improving Anomaly Segmentation with Multi-Granularity Cross-Domain Alignment(http://arxiv.org/abs/2308.08696)
• Summary:

  Anomaly segmentation plays a crucial role in identifying anomalous objects within images, which facilitates the detection of road anomalies for autonomous driving. Although existing methods have shown impressive results in anomaly segmentation using synthetic training data, the domain discrepancies between synthetic training data and real test data are often neglected. To address this issue, the Multi-Granularity Cross-Domain Alignment (MGCDA) framework is proposed for anomaly segmentation in complex driving environments. It uniquely combines a new Multi-source Domain Adversarial Training (MDAT) module and a novel Cross-domain Anomaly-aware Contrastive Learning (CACL) method to boost the generality of the model, seamlessly integrating multi-domain data at both scene and sample levels. Multi-source domain adversarial loss and a dynamic label smoothing strategy are integrated into the MDAT module to facilitate the acquisition of domain-invariant features at the scene level, through adversarial training across multiple stages. CACL aligns sample-level representations with contrastive loss on cross-domain data, which utilizes an anomaly-aware sampling strategy to efficiently sample hard samples and anchors. The proposed framework has decent properties of parameter-free during the inference stage and is compatible with other anomaly segmentation networks. Experimental conducted on Fishyscapes and RoadAnomaly datasets demonstrate that the proposed framework achieves state-of-the-art performance.

Title: Beyond Sharing: Conflict-Aware Multivariate Time Series Anomaly Detection. (arXiv:2308.08915v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.08915
• Code URL: https://github.com/dawnvince/mts_cad
• Copy Paste: [[2308.08915]] Beyond Sharing: Conflict-Aware Multivariate Time Series Anomaly Detection(http://arxiv.org/abs/2308.08915)
• Summary:

  Massive key performance indicators (KPIs) are monitored as multivariate time series data (MTS) to ensure the reliability of the software applications and service system. Accurately detecting the abnormality of MTS is very critical for subsequent fault elimination. The scarcity of anomalies and manual labeling has led to the development of various self-supervised MTS anomaly detection (AD) methods, which optimize an overall objective/loss encompassing all metrics' regression objectives/losses. However, our empirical study uncovers the prevalence of conflicts among metrics' regression objectives, causing MTS models to grapple with different losses. This critical aspect significantly impacts detection performance but has been overlooked in existing approaches. To address this problem, by mimicking the design of multi-gate mixture-of-experts (MMoE), we introduce CAD, a Conflict-aware multivariate KPI Anomaly Detection algorithm. CAD offers an exclusive structure for each metric to mitigate potential conflicts while fostering inter-metric promotions. Upon thorough investigation, we find that the poor performance of vanilla MMoE mainly comes from the input-output misalignment settings of MTS formulation and convergence issues arising from expansive tasks. To address these challenges, we propose a straightforward yet effective task-oriented metric selection and p&s (personalized and shared) gating mechanism, which establishes CAD as the first practicable multi-task learning (MTL) based MTS AD model. Evaluations on multiple public datasets reveal that CAD obtains an average F1-score of 0.943 across three public datasets, notably outperforming state-of-the-art methods. Our code is accessible at https://github.com/dawnvince/MTS_CAD.

in-context

Title: Exploring Demonstration Ensembling for In-context Learning. (arXiv:2308.08780v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.08780
• Code URL: https://github.com/mukhal/icl-ensembling
• Copy Paste: [[2308.08780]] Exploring Demonstration Ensembling for In-context Learning(http://arxiv.org/abs/2308.08780)
• Summary:

  In-context learning (ICL) operates by showing language models (LMs) examples of input-output pairs for a given task, i.e., demonstrations. The standard approach for ICL is to prompt the LM with concatenated demonstrations followed by the test input. This approach suffers from some issues. First, concatenation offers almost no control over the contribution of each demo to the model prediction. This can be sub-optimal when some demonstrations are irrelevant to the test example. Second, due to the input length limit of some transformer models, it might be infeasible to fit many examples into the context, especially when dealing with long-input tasks. In this work, we explore Demonstration Ensembling (DENSE) as an alternative to simple concatenation. \model predicts outputs using subsets (i.e., buckets) of the demonstrations and then combines the output probabilities resulting from each subset to produce the final prediction. We study different ensembling methods using GPT-j and experiment on 12 language tasks. Our experiments show weighted max ensembling to outperform vanilla concatenation by as large as 2.4 average points. Code available at \url{https://github.com/mukhal/icl-ensembling}.