diffusion

Title: Synthetic Image Detection: Highlights from the IEEE Video and Image Processing Cup 2022 Student Competition. (arXiv:2309.12428v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.12428
• Code URL: null
• Copy Paste: [[2309.12428]] Synthetic Image Detection: Highlights from the IEEE Video and Image Processing Cup 2022 Student Competition(http://arxiv.org/abs/2309.12428)
• Summary:

  The Video and Image Processing (VIP) Cup is a student competition that takes place each year at the IEEE International Conference on Image Processing. The 2022 IEEE VIP Cup asked undergraduate students to develop a system capable of distinguishing pristine images from generated ones. The interest in this topic stems from the incredible advances in the AI-based generation of visual data, with tools that allows the synthesis of highly realistic images and videos. While this opens up a large number of new opportunities, it also undermines the trustworthiness of media content and fosters the spread of disinformation on the internet. Recently there was strong concern about the generation of extremely realistic images by means of editing software that includes the recent technology on diffusion models. In this context, there is a need to develop robust and automatic tools for synthetic image detection.

Title: License Plate Super-Resolution Using Diffusion Models. (arXiv:2309.12506v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.12506
• Code URL: null
• Copy Paste: [[2309.12506]] License Plate Super-Resolution Using Diffusion Models(http://arxiv.org/abs/2309.12506)
• Summary:

  In surveillance, accurately recognizing license plates is hindered by their often low quality and small dimensions, compromising recognition precision. Despite advancements in AI-based image super-resolution, methods like Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs) still fall short in enhancing license plate images. This study leverages the cutting-edge diffusion model, which has consistently outperformed other deep learning techniques in image restoration. By training this model using a curated dataset of Saudi license plates, both in low and high resolutions, we discovered the diffusion model's superior efficacy. The method achieves a 12.55\% and 37.32% improvement in Peak Signal-to-Noise Ratio (PSNR) over SwinIR and ESRGAN, respectively. Moreover, our method surpasses these techniques in terms of Structural Similarity Index (SSIM), registering a 4.89% and 17.66% improvement over SwinIR and ESRGAN, respectively. Furthermore, 92% of human evaluators preferred our images over those from other algorithms. In essence, this research presents a pioneering solution for license plate super-resolution, with tangible potential for surveillance systems.

Title: Synthetic Boost: Leveraging Synthetic Data for Enhanced Vision-Language Segmentation in Echocardiography. (arXiv:2309.12829v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.12829
• Code URL: https://github.com/naamiinepal/synthetic-boost
• Copy Paste: [[2309.12829]] Synthetic Boost: Leveraging Synthetic Data for Enhanced Vision-Language Segmentation in Echocardiography(http://arxiv.org/abs/2309.12829)
• Summary:

  Accurate segmentation is essential for echocardiography-based assessment of cardiovascular diseases (CVDs). However, the variability among sonographers and the inherent challenges of ultrasound images hinder precise segmentation. By leveraging the joint representation of image and text modalities, Vision-Language Segmentation Models (VLSMs) can incorporate rich contextual information, potentially aiding in accurate and explainable segmentation. However, the lack of readily available data in echocardiography hampers the training of VLSMs. In this study, we explore using synthetic datasets from Semantic Diffusion Models (SDMs) to enhance VLSMs for echocardiography segmentation. We evaluate results for two popular VLSMs (CLIPSeg and CRIS) using seven different kinds of language prompts derived from several attributes, automatically extracted from echocardiography images, segmentation masks, and their metadata. Our results show improved metrics and faster convergence when pretraining VLSMs on SDM-generated synthetic images before finetuning on real images. The code, configs, and prompts are available at https://github.com/naamiinepal/synthetic-boost.

Title: MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary Instance Segmentation. (arXiv:2309.13042v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.13042
• Code URL: https://github.com/jiahao000/mosaicfusion
• Copy Paste: [[2309.13042]] MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary Instance Segmentation(http://arxiv.org/abs/2309.13042)
• Summary:

  We present MosaicFusion, a simple yet effective diffusion-based data augmentation approach for large vocabulary instance segmentation. Our method is training-free and does not rely on any label supervision. Two key designs enable us to employ an off-the-shelf text-to-image diffusion model as a useful dataset generator for object instances and mask annotations. First, we divide an image canvas into several regions and perform a single round of diffusion process to generate multiple instances simultaneously, conditioning on different text prompts. Second, we obtain corresponding instance masks by aggregating cross-attention maps associated with object prompts across layers and diffusion time steps, followed by simple thresholding and edge-aware refinement processing. Without bells and whistles, our MosaicFusion can produce a significant amount of synthetic labeled data for both rare and novel categories. Experimental results on the challenging LVIS long-tailed and open-vocabulary benchmarks demonstrate that MosaicFusion can significantly improve the performance of existing instance segmentation models, especially for rare and novel categories. Code will be released at https://github.com/Jiahao000/MosaicFusion.

Title: A Diffusion-Model of Joint Interactive Navigation. (arXiv:2309.12508v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.12508
• Code URL: null
• Copy Paste: [[2309.12508]] A Diffusion-Model of Joint Interactive Navigation(http://arxiv.org/abs/2309.12508)
• Summary:

  Simulation of autonomous vehicle systems requires that simulated traffic participants exhibit diverse and realistic behaviors. The use of prerecorded real-world traffic scenarios in simulation ensures realism but the rarity of safety critical events makes large scale collection of driving scenarios expensive. In this paper, we present DJINN - a diffusion based method of generating traffic scenarios. Our approach jointly diffuses the trajectories of all agents, conditioned on a flexible set of state observations from the past, present, or future. On popular trajectory forecasting datasets, we report state of the art performance on joint trajectory metrics. In addition, we demonstrate how DJINN flexibly enables direct test-time sampling from a variety of valuable conditional distributions including goal-based sampling, behavior-class sampling, and scenario editing.

self-supervised

Title: Masking Improves Contrastive Self-Supervised Learning for ConvNets, and Saliency Tells You Where. (arXiv:2309.12757v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.12757
• Code URL: null
• Copy Paste: [[2309.12757]] Masking Improves Contrastive Self-Supervised Learning for ConvNets, and Saliency Tells You Where(http://arxiv.org/abs/2309.12757)
• Summary:

  While image data starts to enjoy the simple-but-effective self-supervised learning scheme built upon masking and self-reconstruction objective thanks to the introduction of tokenization procedure and vision transformer backbone, convolutional neural networks as another important and widely-adopted architecture for image data, though having contrastive-learning techniques to drive the self-supervised learning, still face the difficulty of leveraging such straightforward and general masking operation to benefit their learning process significantly. In this work, we aim to alleviate the burden of including masking operation into the contrastive-learning framework for convolutional neural networks as an extra augmentation method. In addition to the additive but unwanted edges (between masked and unmasked regions) as well as other adverse effects caused by the masking operations for ConvNets, which have been discussed by prior works, we particularly identify the potential problem where for one view in a contrastive sample-pair the randomly-sampled masking regions could be overly concentrated on important/salient objects thus resulting in misleading contrastiveness to the other view. To this end, we propose to explicitly take the saliency constraint into consideration in which the masked regions are more evenly distributed among the foreground and background for realizing the masking-based augmentation. Moreover, we introduce hard negative samples by masking larger regions of salient patches in an input image. Extensive experiments conducted on various datasets, contrastive learning mechanisms, and downstream tasks well verify the efficacy as well as the superior performance of our proposed method with respect to several state-of-the-art baselines.

Title: On Separate Normalization in Self-supervised Transformers. (arXiv:2309.12931v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.12931
• Code URL: null
• Copy Paste: [[2309.12931]] On Separate Normalization in Self-supervised Transformers(http://arxiv.org/abs/2309.12931)
• Summary:

  Self-supervised training methods for transformers have demonstrated remarkable performance across various domains. Previous transformer-based models, such as masked autoencoders (MAE), typically utilize a single normalization layer for both the [CLS] symbol and the tokens. We propose in this paper a simple modification that employs separate normalization layers for the tokens and the [CLS] symbol to better capture their distinct characteristics and enhance downstream task performance. Our method aims to alleviate the potential negative effects of using the same normalization statistics for both token types, which may not be optimally aligned with their individual roles. We empirically show that by utilizing a separate normalization layer, the [CLS] embeddings can better encode the global contextual information and are distributed more uniformly in its anisotropic space. When replacing the conventional normalization layer with the two separate layers, we observe an average 2.7% performance improvement over the image, natural language, and graph domains.

foundation model

Title: BayesDLL: Bayesian Deep Learning Library. (arXiv:2309.12928v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.12928
• Code URL: https://github.com/minyoungkim21/bayesdll
• Copy Paste: [[2309.12928]] BayesDLL: Bayesian Deep Learning Library(http://arxiv.org/abs/2309.12928)
• Summary:

  We release a new Bayesian neural network library for PyTorch for large-scale deep networks. Our library implements mainstream approximate Bayesian inference algorithms: variational inference, MC-dropout, stochastic-gradient MCMC, and Laplace approximation. The main differences from other existing Bayesian neural network libraries are as follows: 1) Our library can deal with very large-scale deep networks including Vision Transformers (ViTs). 2) We need virtually zero code modifications for users (e.g., the backbone network definition codes do not neet to be modified at all). 3) Our library also allows the pre-trained model weights to serve as a prior mean, which is very useful for performing Bayesian inference with the large-scale foundation models like ViTs that are hard to optimise from scratch with the downstream data alone. Our code is publicly available at: \url{https://github.com/SamsungLabs/BayesDLL}\footnote{A mirror repository is also available at: \url{https://github.com/minyoungkim21/BayesDLL}.}.

generative

Title: Foundation Metrics: Quantifying Effectiveness of Healthcare Conversations powered by Generative AI. (arXiv:2309.12444v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.12444
• Code URL: null
• Copy Paste: [[2309.12444]] Foundation Metrics: Quantifying Effectiveness of Healthcare Conversations powered by Generative AI(http://arxiv.org/abs/2309.12444)
• Summary:

  Generative Artificial Intelligence is set to revolutionize healthcare delivery by transforming traditional patient care into a more personalized, efficient, and proactive process. Chatbots, serving as interactive conversational models, will probably drive this patient-centered transformation in healthcare. Through the provision of various services, including diagnosis, personalized lifestyle recommendations, and mental health support, the objective is to substantially augment patient health outcomes, all the while mitigating the workload burden on healthcare providers. The life-critical nature of healthcare applications necessitates establishing a unified and comprehensive set of evaluation metrics for conversational models. Existing evaluation metrics proposed for various generic large language models (LLMs) demonstrate a lack of comprehension regarding medical and health concepts and their significance in promoting patients' well-being. Moreover, these metrics neglect pivotal user-centered aspects, including trust-building, ethics, personalization, empathy, user comprehension, and emotional support. The purpose of this paper is to explore state-of-the-art LLM-based evaluation metrics that are specifically applicable to the assessment of interactive conversational models in healthcare. Subsequently, we present an comprehensive set of evaluation metrics designed to thoroughly assess the performance of healthcare chatbots from an end-user perspective. These metrics encompass an evaluation of language processing abilities, impact on real-world clinical tasks, and effectiveness in user-interactive conversations. Finally, we engage in a discussion concerning the challenges associated with defining and implementing these metrics, with particular emphasis on confounding factors such as the target audience, evaluation methods, and prompt techniques involved in the evaluation process.

Title: Learning to Diversify Neural Text Generation via Degenerative Model. (arXiv:2309.12619v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.12619
• Code URL: null
• Copy Paste: [[2309.12619]] Learning to Diversify Neural Text Generation via Degenerative Model(http://arxiv.org/abs/2309.12619)
• Summary:

  Neural language models often fail to generate diverse and informative texts, limiting their applicability in real-world problems. While previous approaches have proposed to address these issues by identifying and penalizing undesirable behaviors (e.g., repetition, overuse of frequent words) from language models, we propose an alternative approach based on an observation: models primarily learn attributes within examples that are likely to cause degeneration problems. Based on this observation, we propose a new approach to prevent degeneration problems by training two models. Specifically, we first train a model that is designed to amplify undesirable patterns. We then enhance the diversity of the second model by focusing on patterns that the first model fails to learn. Extensive experiments on two tasks, namely language modeling and dialogue generation, demonstrate the effectiveness of our approach.

Title: Change Management using Generative Modeling on Digital Twins. (arXiv:2309.12421v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.12421
• Code URL: null
• Copy Paste: [[2309.12421]] Change Management using Generative Modeling on Digital Twins(http://arxiv.org/abs/2309.12421)
• Summary:

  A key challenge faced by small and medium-sized business entities is securely managing software updates and changes. Specifically, with rapidly evolving cybersecurity threats, changes/updates/patches to software systems are necessary to stay ahead of emerging threats and are often mandated by regulators or statutory authorities to counter these. However, security patches/updates require stress testing before they can be released in the production system. Stress testing in production environments is risky and poses security threats. Large businesses usually have a non-production environment where such changes can be made and tested before being released into production. Smaller businesses do not have such facilities. In this work, we show how "digital twins", especially for a mix of IT and IoT environments, can be created on the cloud. These digital twins act as a non-production environment where changes can be applied, and the system can be securely tested before patch release. Additionally, the non-production digital twin can be used to collect system data and run stress tests on the environment, both manually and automatically. In this paper, we show how using a small sample of real data/interactions, Generative Artificial Intelligence (AI) models can be used to generate testing scenarios to check for points of failure.

anomaly

Title: mixed attention auto encoder for multi-class industrial anomaly detection. (arXiv:2309.12700v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.12700
• Code URL: null
• Copy Paste: [[2309.12700]] mixed attention auto encoder for multi-class industrial anomaly detection(http://arxiv.org/abs/2309.12700)
• Summary:

  Most existing methods for unsupervised industrial anomaly detection train a separate model for each object category. This kind of approach can easily capture the category-specific feature distributions, but results in high storage cost and low training efficiency. In this paper, we propose a unified mixed-attention auto encoder (MAAE) to implement multi-class anomaly detection with a single model. To alleviate the performance degradation due to the diverse distribution patterns of different categories, we employ spatial attentions and channel attentions to effectively capture the global category information and model the feature distributions of multiple classes. Furthermore, to simulate the realistic noises on features and preserve the surface semantics of objects from different categories which are essential for detecting the subtle anomalies, we propose an adaptive noise generator and a multi-scale fusion module for the pre-trained features. MAAE delivers remarkable performances on the benchmark dataset compared with the state-of-the-art methods.

Title: On Data Fabrication in Collaborative Vehicular Perception: Attacks and Countermeasures. (arXiv:2309.12955v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.12955
• Code URL: null
• Copy Paste: [[2309.12955]] On Data Fabrication in Collaborative Vehicular Perception: Attacks and Countermeasures(http://arxiv.org/abs/2309.12955)
• Summary:

  Collaborative perception, which greatly enhances the sensing capability of connected and autonomous vehicles (CAVs) by incorporating data from external resources, also brings forth potential security risks. CAVs' driving decisions rely on remote untrusted data, making them susceptible to attacks carried out by malicious participants in the collaborative perception system. However, security analysis and countermeasures for such threats are absent. To understand the impact of the vulnerability, we break the ground by proposing various real-time data fabrication attacks in which the attacker delivers crafted malicious data to victims in order to perturb their perception results, leading to hard brakes or increased collision risks. Our attacks demonstrate a high success rate of over 86\% on high-fidelity simulated scenarios and are realizable in real-world experiments. To mitigate the vulnerability, we present a systematic anomaly detection approach that enables benign vehicles to jointly reveal malicious fabrication. It detects 91.5% of attacks with a false positive rate of 3% in simulated scenarios and significantly mitigates attack impacts in real-world scenarios.

in-context

Title: HRoT: Hybrid prompt strategy and Retrieval of Thought for Table-Text Hybrid Question Answering. (arXiv:2309.12669v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.12669
• Code URL: null
• Copy Paste: [[2309.12669]] HRoT: Hybrid prompt strategy and Retrieval of Thought for Table-Text Hybrid Question Answering(http://arxiv.org/abs/2309.12669)
• Summary:

  Answering numerical questions over hybrid contents from the given tables and text(TextTableQA) is a challenging task. Recently, Large Language Models (LLMs) have gained significant attention in the NLP community. With the emergence of large language models, In-Context Learning and Chain-of-Thought prompting have become two particularly popular research topics in this field. In this paper, we introduce a new prompting strategy called Hybrid prompt strategy and Retrieval of Thought for TextTableQA. Through In-Context Learning, we prompt the model to develop the ability of retrieval thinking when dealing with hybrid data. Our method achieves superior performance compared to the fully-supervised SOTA on the MultiHiertt dataset in the few-shot setting.

Title: Affect Recognition in Conversations Using Large Language Models. (arXiv:2309.12881v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.12881
• Code URL: null
• Copy Paste: [[2309.12881]] Affect Recognition in Conversations Using Large Language Models(http://arxiv.org/abs/2309.12881)
• Summary:

  Affect recognition, encompassing emotions, moods, and feelings, plays a pivotal role in human communication. In the realm of conversational artificial intelligence (AI), the ability to discern and respond to human affective cues is a critical factor for creating engaging and empathetic interactions. This study delves into the capacity of large language models (LLMs) to recognise human affect in conversations, with a focus on both open-domain chit-chat dialogues and task-oriented dialogues. Leveraging three diverse datasets, namely IEMOCAP, EmoWOZ, and DAIC-WOZ, covering a spectrum of dialogues from casual conversations to clinical interviews, we evaluated and compared LLMs' performance in affect recognition. Our investigation explores the zero-shot and few-shot capabilities of LLMs through in-context learning (ICL) as well as their model capacities through task-specific fine-tuning. Additionally, this study takes into account the potential impact of automatic speech recognition (ASR) errors on LLM predictions. With this work, we aim to shed light on the extent to which LLMs can replicate human-like affect recognition capabilities in conversations.