secure

security

Title: DecentRAN: Decentralized Radio Access Network for 5.5G and beyond. (arXiv:2303.17210v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2303.17210
• Code URL: null
• Copy Paste: [[2303.17210] DecentRAN: Decentralized Radio Access Network for 5](http://arxiv.org/abs/2303.17210) #security
• Summary:

  Radio Access Network faces challenges from privacy and flexible wide area and local area network access. RAN is limited from providing local service directly due to centralized design of cellular network and concerns of user privacy and data security. DecentRAN or Decentralized Radio Access Network offers an alternative perspective to cope with the emerging demands of 5G Non-public Network and the hybrid deployment of 5GS and Wi-Fi in the campus network. Starting from Public key as an Identity, independent mutual authentication between UE and RAN are made possible in a privacy-preserving manner. With the introduction of decentralized architecture and network functions using blockchain and smart contracts, DecentRAN has ability to provide users with locally managed, end-to-end encrypted 5G NPN and the potential connectivity to Local Area Network via campus routers. Furthermore, the performance regarding throughput and latency are discussed, offering the deployment guidance for DecentRAN.

privacy

Title: C-SFDA: A Curriculum Learning Aided Self-Training Framework for Efficient Source Free Domain Adaptation. (arXiv:2303.17132v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17132
• Code URL: null
• Copy Paste: [[2303.17132] C-SFDA: A Curriculum Learning Aided Self-Training Framework for Efficient Source Free Domain Adaptation](http://arxiv.org/abs/2303.17132) #privacy
• Summary:

  Unsupervised domain adaptation (UDA) approaches focus on adapting models trained on a labeled source domain to an unlabeled target domain. UDA methods have a strong assumption that the source data is accessible during adaptation, which may not be feasible in many real-world scenarios due to privacy concerns and resource constraints of devices. In this regard, source-free domain adaptation (SFDA) excels as access to source data is no longer required during adaptation. Recent state-of-the-art (SOTA) methods on SFDA mostly focus on pseudo-label refinement based self-training which generally suffers from two issues: i) inevitable occurrence of noisy pseudo-labels that could lead to early training time memorization, ii) refinement process requires maintaining a memory bank which creates a significant burden in resource constraint scenarios. To address these concerns, we propose C-SFDA, a curriculum learning aided self-training framework for SFDA that adapts efficiently and reliably to changes across domains based on selective pseudo-labeling. Specifically, we employ a curriculum learning scheme to promote learning from a restricted amount of pseudo labels selected based on their reliabilities. This simple yet effective step successfully prevents label noise propagation during different stages of adaptation and eliminates the need for costly memory-bank based label refinement. Our extensive experimental evaluations on both image recognition and semantic segmentation tasks confirm the effectiveness of our method. C-SFDA is readily applicable to online test-time domain adaptation and also outperforms previous SOTA methods in this task.

Title: Have it your way: Individualized Privacy Assignment for DP-SGD. (arXiv:2303.17046v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2303.17046
• Code URL: null
• Copy Paste: [[2303.17046] Have it your way: Individualized Privacy Assignment for DP-SGD](http://arxiv.org/abs/2303.17046) #privacy
• Summary:

  When training a machine learning model with differential privacy, one sets a privacy budget. This budget represents a maximal privacy violation that any user is willing to face by contributing their data to the training set. We argue that this approach is limited because different users may have different privacy expectations. Thus, setting a uniform privacy budget across all points may be overly conservative for some users or, conversely, not sufficiently protective for others. In this paper, we capture these preferences through individualized privacy budgets. To demonstrate their practicality, we introduce a variant of Differentially Private Stochastic Gradient Descent (DP-SGD) which supports such individualized budgets. DP-SGD is the canonical approach to training models with differential privacy. We modify its data sampling and gradient noising mechanisms to arrive at our approach, which we call Individualized DP-SGD (IDP-SGD). Because IDP-SGD provides privacy guarantees tailored to the preferences of individual users and their data points, we find it empirically improves privacy-utility trade-offs.

Title: Not Yet Another Digital ID: Privacy-preserving Humanitarian Aid Distribution. (arXiv:2303.17343v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2303.17343
• Code URL: null
• Copy Paste: [[2303.17343] Not Yet Another Digital ID: Privacy-preserving Humanitarian Aid Distribution](http://arxiv.org/abs/2303.17343) #privacy
• Summary:

  Humanitarian aid-distribution programs help bring physical goods (e.g., food, blankets) to people in need. Traditional paper-based solutions to support aid distribution do not scale to large populations and are hard to secure. Existing digital solutions solve these issues, at the cost of collecting large amount of personal information. Failing to protect aid recipients' privacy can result on harms for them and enables surveillance in the long run. In collaboration with the International Committee of the Red Cross, we build a safe aid-distribution system in this paper. We first systematize the requirements such a system should satisfy and then propose a decentralized solution based on the use of tokens. Our design provides strong scalability and accountability, at the same time, ensures privacy by design. We provide two instantiations of our design, on a smart card and on a smartphone. We formally prove the security and privacy properties of our design, and empirically show that the two instantiations can scale to hundreds of thousands of recipients.

Title: Infinite Horizon Privacy in Networked Control Systems: Utility/Privacy Tradeoffs and Design Tools. (arXiv:2303.17519v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2303.17519
• Code URL: null
• Copy Paste: [[2303.17519] Infinite Horizon Privacy in Networked Control Systems: Utility/Privacy Tradeoffs and Design Tools](http://arxiv.org/abs/2303.17519) #privacy
• Summary:

  We address the problem of synthesizing distorting mechanisms that maximize infinite horizon privacy for Networked Control Systems (NCSs). We consider stochastic LTI systems where information about the system state is obtained through noisy sensor measurements and transmitted to a (possibly adversarial) remote station via unsecured/public communication networks to compute control actions (a remote LQR controller). Because the network/station is untrustworthy, adversaries might access sensor and control data and estimate the system state. To mitigate this risk, we pass sensor and control data through distorting (privacy-preserving) mechanisms before transmission and send the distorted data through the communication network. These mechanisms consist of a linear coordinate transformation and additive-dependent Gaussian vectors. We formulate the synthesis of the distorting mechanisms as a convex program. In this convex program, we minimize the infinite horizon mutual information (our privacy metric) between the system state and its optimal estimate at the remote station for a desired upper bound on the control performance degradation (LQR cost) induced by the distortion mechanism.

Title: TorKameleon: Improving Tor's Censorship Resistance With K-anonimization and Media-based Covert Channels. (arXiv:2303.17544v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2303.17544
• Code URL: null
• Copy Paste: [[2303.17544] TorKameleon: Improving Tor's Censorship Resistance With K-anonimization and Media-based Covert Channels](http://arxiv.org/abs/2303.17544) #privacy
• Summary:

  The use of anonymity networks such as Tor and similar tools can greatly enhance the privacy and anonymity of online communications. Tor, in particular, is currently the most widely used system for ensuring anonymity on the Internet. However, recent research has shown that Tor is vulnerable to correlation attacks carried out by state-level adversaries or colluding Internet censors. Therefore, new and more effective solutions emerged to protect online anonymity. Promising results have been achieved by implementing covert channels based on media traffic in modern anonymization systems, which have proven to be a reliable and practical approach to defend against powerful traffic correlation attacks. In this paper, we present TorKameleon, a censorship evasion solution that better protects Tor users from powerful traffic correlation attacks carried out by state-level adversaries. TorKameleon can be used either as a fully integrated Tor pluggable transport or as a standalone anonymization system that uses K-anonymization and encapsulation of user traffic in covert media channels. Our main goal is to protect users from machine and deep learning correlation attacks on anonymization networks like Tor. We have developed the TorKameleon prototype and performed extensive validations to verify the accuracy and experimental performance of the proposed solution in the Tor environment, including state-of-the-art active correlation attacks. As far as we know, we are the first to develop and study a system that uses both anonymization mechanisms described above against active correlation attacks.

protect

Title: MaLP: Manipulation Localization Using a Proactive Scheme. (arXiv:2303.16976v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.16976
• Code URL: https://github.com/vishal3477/pro_loc
• Copy Paste: [[2303.16976] MaLP: Manipulation Localization Using a Proactive Scheme](http://arxiv.org/abs/2303.16976) #protect
• Summary:

  Advancements in the generation quality of various Generative Models (GMs) has made it necessary to not only perform binary manipulation detection but also localize the modified pixels in an image. However, prior works termed as passive for manipulation localization exhibit poor generalization performance over unseen GMs and attribute modifications. To combat this issue, we propose a proactive scheme for manipulation localization, termed MaLP. We encrypt the real images by adding a learned template. If the image is manipulated by any GM, this added protection from the template not only aids binary detection but also helps in identifying the pixels modified by the GM. The template is learned by leveraging local and global-level features estimated by a two-branch architecture. We show that MaLP performs better than prior passive works. We also show the generalizability of MaLP by testing on 22 different GMs, providing a benchmark for future research on manipulation localization. Finally, we show that MaLP can be used as a discriminator for improving the generation quality of GMs. Our models/codes are available at www.github.com/vishal3477/pro_loc.

defense

Title: Adversarial Attack and Defense for Dehazing Networks. (arXiv:2303.17255v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17255
• Code URL: https://github.com/guijiejie/aadn
• Copy Paste: [[2303.17255] Adversarial Attack and Defense for Dehazing Networks](http://arxiv.org/abs/2303.17255) #defense
• Summary:

  The research on single image dehazing task has been widely explored. However, as far as we know, no comprehensive study has been conducted on the robustness of the well-trained dehazing models. Therefore, there is no evidence that the dehazing networks can resist malicious attacks. In this paper, we focus on designing a group of attack methods based on first order gradient to verify the robustness of the existing dehazing algorithms. By analyzing the general goal of image dehazing task, five attack methods are proposed, which are prediction, noise, mask, ground-truth and input attack. The corresponding experiments are conducted on six datasets with different scales. Further, the defense strategy based on adversarial training is adopted for reducing the negative effects caused by malicious attacks. In summary, this paper defines a new challenging problem for image dehazing area, which can be called as adversarial attack on dehazing networks (AADN). Code is available at https://github.com/guijiejie/AADN.

attack

Title: A Tensor-based Convolutional Neural Network for Small Dataset Classification. (arXiv:2303.17061v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17061
• Code URL: null
• Copy Paste: [[2303.17061] A Tensor-based Convolutional Neural Network for Small Dataset Classification](http://arxiv.org/abs/2303.17061) #attack
• Summary:

  Inspired by the ConvNets with structured hidden representations, we propose a Tensor-based Neural Network, TCNN. Different from ConvNets, TCNNs are composed of structured neurons rather than scalar neurons, and the basic operation is neuron tensor transformation. Unlike other structured ConvNets, where the part-whole relationships are modeled explicitly, the relationships are learned implicitly in TCNNs. Also, the structured neurons in TCNNs are high-rank tensors rather than vectors or matrices. We compare TCNNs with current popular ConvNets, including ResNets, MobileNets, EfficientNets, RegNets, etc., on CIFAR10, CIFAR100, and Tiny ImageNet. The experiment shows that TCNNs have higher efficiency in terms of parameters. TCNNs also show higher robustness against white-box adversarial attacks on MNIST compared to ConvNets.

Title: FeDiSa: A Semi-asynchronous Federated Learning Framework for Power System Fault and Cyberattack Discrimination. (arXiv:2303.16956v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2303.16956
• Code URL: null
• Copy Paste: [[2303.16956] FeDiSa: A Semi-asynchronous Federated Learning Framework for Power System Fault and Cyberattack Discrimination](http://arxiv.org/abs/2303.16956) #attack
• Summary:

  With growing security and privacy concerns in the Smart Grid domain, intrusion detection on critical energy infrastructure has become a high priority in recent years. To remedy the challenges of privacy preservation and decentralized power zones with strategic data owners, Federated Learning (FL) has contemporarily surfaced as a viable privacy-preserving alternative which enables collaborative training of attack detection models without requiring the sharing of raw data. To address some of the technical challenges associated with conventional synchronous FL, this paper proposes FeDiSa, a novel Semi-asynchronous Federated learning framework for power system faults and cyberattack Discrimination which takes into account communication latency and stragglers. Specifically, we propose a collaborative training of deep auto-encoder by Supervisory Control and Data Acquisition sub-systems which upload their local model updates to a control centre, which then perform a semi-asynchronous model aggregation for a new global model parameters based on a buffer system and a preset cut-off time. Experiments on the proposed framework using publicly available industrial control systems datasets reveal superior attack detection accuracy whilst preserving data confidentiality and minimizing the adverse effects of communication latency and stragglers. Furthermore, we see a 35% improvement in training time, thus validating the robustness of our proposed method.

Title: Innovative Countermeasures to Defeat Cyber Attacks Against Blockchain Wallets: A Crypto Terminal Use Case. (arXiv:2303.17206v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2303.17206
• Code URL: null
• Copy Paste: [[2303.17206] Innovative Countermeasures to Defeat Cyber Attacks Against Blockchain Wallets: A Crypto Terminal Use Case](http://arxiv.org/abs/2303.17206) #attack
• Summary:

  Blockchain transactions are signed by private keys. Secure key storage and tamper-proof computers are essential requirements for deploying a trusted infrastructure. In this paper, we identify some threats against blockchain wallets and propose a set of physical and logical countermeasures to thwart them. We present the crypto terminal device, operating with a removable secure element, built on open software and hardware architectures, capable of detecting a cloned device or corrupted software. These technologies are based on tamper-resistant computing (javacard), smart card anti-cloning, smart card content attestation, application firewall, bare-metal architecture, remote attestation, dynamic Physical Unclonable Function (dPUF), and programming tokens as a root of trust.This paper is an extended version of the paper ''Innovative Countermeasures to Defeat Cyber Attacks Against Blockchain Wallets,'' 2021 5th Cyber Security in Networking Conference (CSNet), 2021, pp. 49-54, doi: 10.1109/CSNet52717.2021.9614649

Title: Differential Area Analysis for Ransomware: Attacks, Countermeasures, and Limitations. (arXiv:2303.17351v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2303.17351
• Code URL: null
• Copy Paste: [[2303.17351] Differential Area Analysis for Ransomware: Attacks, Countermeasures, and Limitations](http://arxiv.org/abs/2303.17351) #attack
• Summary:

  Crypto-ransomware attacks have been a growing threat over the last few years. The goal of every ransomware strain is encrypting user data, such that attackers can later demand users a ransom for unlocking their data. To maximise their earning chances, attackers equip their ransomware with strong encryption which produce files with high entropy values. Davies et al. proposed Differential Area Analysis (DAA), a technique that analyses files headers to differentiate compressed, regularly encrypted, and ransomware-encrypted files. In this paper, first we propose three different attacks to perform malicious header manipulation and bypass DAA detection. Then, we propose three countermeasures, namely 2-Fragments (2F), 3-Fragments (3F), and 4-Fragments (4F), which can be applied equally against each of the three attacks we propose. We conduct a number of experiments to analyse the ability of our countermeasures to detect ransomware-encrypted files, whether implementing our proposed attacks or not. Last, we test the robustness of our own countermeasures by analysing the performance, in terms of files per second analysed and resilience to extensive injection of low-entropy data. Our results show that our detection countermeasures are viable and deployable alternatives to DAA.

Title: URSID: Using formalism to Refine attack Scenarios for vulnerable Infrastructure Deployment. (arXiv:2303.17373v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2303.17373
• Code URL: null
• Copy Paste: [[2303.17373] URSID: Using formalism to Refine attack Scenarios for vulnerable Infrastructure Deployment](http://arxiv.org/abs/2303.17373) #attack
• Summary:

  In this paper we propose a novel way of deploying vulnerable architectures for defense and research purposes, which aims to generate deception platforms based on the formal description of a scenario. An attack scenario is described by an attack graph in which transitions are labeled by ATT&CK techniques or procedures. The state of the attacker is modeled as a set of secrets he acquires and a set of nodes he controls. Descriptions of a single scenario on a technical level can then be declined into several different scenarios on a procedural level, and each of these scenarios can be deployed into its own vulnerable architecture. To achieve this goal we introduce the notion of architecture constraints, as some procedures may only be exploited on system presenting special properties, such as having a specific operating system version. Finally, we present our deployment process for converting one of these scenarios into a vulnerable infrastructure, and offer an online proof of concept demonstration of our tool, where readers may deploy locally deploy a complete scenario inspired by the threat actor APT-29.

Title: Mole Recruitment: Poisoning of Image Classifiers via Selective Batch Sampling. (arXiv:2303.17080v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2303.17080
• Code URL: null
• Copy Paste: [[2303.17080] Mole Recruitment: Poisoning of Image Classifiers via Selective Batch Sampling](http://arxiv.org/abs/2303.17080) #attack
• Summary:

  In this work, we present a data poisoning attack that confounds machine learning models without any manipulation of the image or label. This is achieved by simply leveraging the most confounding natural samples found within the training data itself, in a new form of a targeted attack coined "Mole Recruitment." We define moles as the training samples of a class that appear most similar to samples of another class, and show that simply restructuring training batches with an optimal number of moles can lead to significant degradation in the performance of the targeted class. We show the efficacy of this novel attack in an offline setting across several standard image classification datasets, and demonstrate the real-world viability of this attack in a continual learning (CL) setting. Our analysis reveals that state-of-the-art models are susceptible to Mole Recruitment, thereby exposing a previously undetected vulnerability of image classifiers.

robust

Title: De-coupling and De-positioning Dense Self-supervised Learning. (arXiv:2303.16947v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.16947
• Code URL: null
• Copy Paste: [[2303.16947] De-coupling and De-positioning Dense Self-supervised Learning](http://arxiv.org/abs/2303.16947) #robust
• Summary:

  Dense Self-Supervised Learning (SSL) methods address the limitations of using image-level feature representations when handling images with multiple objects. Although the dense features extracted by employing segmentation maps and bounding boxes allow networks to perform SSL for each object, we show that they suffer from coupling and positional bias, which arise from the receptive field increasing with layer depth and zero-padding. We address this by introducing three data augmentation strategies, and leveraging them in (i) a decoupling module that aims to robustify the network to variations in the object's surroundings, and (ii) a de-positioning module that encourages the network to discard positional object information. We demonstrate the benefits of our method on COCO and on a new challenging benchmark, OpenImage-MINI, for object classification, semantic segmentation, and object detection. Our extensive experiments evidence the better generalization of our method compared to the SOTA dense SSL methods

Title: ImageNet-E: Benchmarking Neural Network Robustness via Attribute Editing. (arXiv:2303.17096v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17096
• Code URL: https://github.com/alibaba/easyrobust
• Copy Paste: [[2303.17096] ImageNet-E: Benchmarking Neural Network Robustness via Attribute Editing](http://arxiv.org/abs/2303.17096) #robust
• Summary:

  Recent studies have shown that higher accuracy on ImageNet usually leads to better robustness against different corruptions. Therefore, in this paper, instead of following the traditional research paradigm that investigates new out-of-distribution corruptions or perturbations deep models may encounter, we conduct model debugging in in-distribution data to explore which object attributes a model may be sensitive to. To achieve this goal, we create a toolkit for object editing with controls of backgrounds, sizes, positions, and directions, and create a rigorous benchmark named ImageNet-E(diting) for evaluating the image classifier robustness in terms of object attributes. With our ImageNet-E, we evaluate the performance of current deep learning models, including both convolutional neural networks and vision transformers. We find that most models are quite sensitive to attribute changes. A small change in the background can lead to an average of 9.23\% drop on top-1 accuracy. We also evaluate some robust models including both adversarially trained models and other robust trained models and find that some models show worse robustness against attribute changes than vanilla models. Based on these findings, we discover ways to enhance attribute robustness with preprocessing, architecture designs, and training strategies. We hope this work can provide some insights to the community and open up a new avenue for research in robust computer vision. The code and dataset are available at https://github.com/alibaba/easyrobust.

Title: DAMO-StreamNet: Optimizing Streaming Perception in Autonomous Driving. (arXiv:2303.17144v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17144
• Code URL: null
• Copy Paste: [[2303.17144] DAMO-StreamNet: Optimizing Streaming Perception in Autonomous Driving](http://arxiv.org/abs/2303.17144) #robust
• Summary:

  Real-time perception, or streaming perception, is a crucial aspect of autonomous driving that has yet to be thoroughly explored in existing research. To address this gap, we present DAMO-StreamNet, an optimized framework that combines recent advances from the YOLO series with a comprehensive analysis of spatial and temporal perception mechanisms, delivering a cutting-edge solution. The key innovations of DAMO-StreamNet are: (1) A robust neck structure incorporating deformable convolution, enhancing the receptive field and feature alignment capabilities. (2) A dual-branch structure that integrates short-path semantic features and long-path temporal features, improving motion state prediction accuracy. (3) Logits-level distillation for efficient optimization, aligning the logits of teacher and student networks in semantic space. (4) A real-time forecasting mechanism that updates support frame features with the current frame, ensuring seamless streaming perception during inference. Our experiments demonstrate that DAMO-StreamNet surpasses existing state-of-the-art methods, achieving 37.8% (normal size (600, 960)) and 43.3% (large size (1200, 1920)) sAP without using extra data. This work not only sets a new benchmark for real-time perception but also provides valuable insights for future research. Additionally, DAMO-StreamNet can be applied to various autonomous systems, such as drones and robots, paving the way for real-time perception.

Title: LatentForensics: Towards lighter deepfake detection in the StyleGAN latent space. (arXiv:2303.17222v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17222
• Code URL: null
• Copy Paste: [[2303.17222] LatentForensics: Towards lighter deepfake detection in the StyleGAN latent space](http://arxiv.org/abs/2303.17222) #robust
• Summary:

  The classification of forged videos has been a challenge for the past few years. Deepfake classifiers can now reliably predict whether or not video frames have been tampered with. However, their performance is tied to both the dataset used for training and the analyst's computational power. We propose a deepfake classification method that operates in the latent space of a state-of-the-art generative adversarial network (GAN) trained on high-quality face images. The proposed method leverages the structure of the latent space of StyleGAN to learn a lightweight classification model. Experimental results on a standard dataset reveal that the proposed approach outperforms other state-of-the-art deepfake classification methods. To the best of our knowledge, this is the first study showing the interest of the latent space of StyleGAN for deepfake classification. Combined with other recent studies on the interpretation and manipulation of this latent space, we believe that the proposed approach can help in developing robust deepfake classification methods based on interpretable high-level properties of face images.

Title: FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation. (arXiv:2303.17225v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17225
• Code URL: null
• Copy Paste: [[2303.17225] FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation](http://arxiv.org/abs/2303.17225) #robust
• Summary:

  Recently, open-vocabulary learning has emerged to accomplish segmentation for arbitrary categories of text-based descriptions, which popularizes the segmentation system to more general-purpose application scenarios. However, existing methods devote to designing specialized architectures or parameters for specific segmentation tasks. These customized design paradigms lead to fragmentation between various segmentation tasks, thus hindering the uniformity of segmentation models. Hence in this paper, we propose FreeSeg, a generic framework to accomplish Unified, Universal and Open-Vocabulary Image Segmentation. FreeSeg optimizes an all-in-one network via one-shot training and employs the same architecture and parameters to handle diverse segmentation tasks seamlessly in the inference procedure. Additionally, adaptive prompt learning facilitates the unified model to capture task-aware and category-sensitive concepts, improving model robustness in multi-task and varied scenarios. Extensive experimental results demonstrate that FreeSeg establishes new state-of-the-art results in performance and generalization on three segmentation tasks, which outperforms the best task-specific architectures by a large margin: 5.5% mIoU on semantic segmentation, 17.6% mAP on instance segmentation, 20.1% PQ on panoptic segmentation for the unseen class on COCO.

Title: Investigating and Mitigating the Side Effects of Noisy Views in Multi-view Clustering in Practical Scenarios. (arXiv:2303.17245v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2303.17245
• Code URL: null
• Copy Paste: [[2303.17245] Investigating and Mitigating the Side Effects of Noisy Views in Multi-view Clustering in Practical Scenarios](http://arxiv.org/abs/2303.17245) #robust
• Summary:

  Multi-view clustering (MvC) aims at exploring the category structure among multi-view data without label supervision. Multiple views provide more information than single views and thus existing MvC methods can achieve satisfactory performance. However, their performance might seriously degenerate when the views are noisy in practical scenarios. In this paper, we first formally investigate the drawback of noisy views and then propose a theoretically grounded deep MvC method (namely MvCAN) to address this issue. Specifically, we propose a novel MvC objective that enables un-shared parameters and inconsistent clustering predictions across multiple views to reduce the side effects of noisy views. Furthermore, a non-parametric iterative process is designed to generate a robust learning target for mining multiple views' useful information. Theoretical analysis reveals that MvCAN works by achieving the multi-view consistency, complementarity, and noise robustness. Finally, experiments on public datasets demonstrate that MvCAN outperforms state-of-the-art methods and is robust against the existence of noisy views.

Title: Impact of Video Processing Operations in Deepfake Detection. (arXiv:2303.17247v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17247
• Code URL: null
• Copy Paste: [[2303.17247] Impact of Video Processing Operations in Deepfake Detection](http://arxiv.org/abs/2303.17247) #robust
• Summary:

  The detection of digital face manipulation in video has attracted extensive attention due to the increased risk to public trust. To counteract the malicious usage of such techniques, deep learning-based deepfake detection methods have been developed and have shown impressive results. However, the performance of these detectors is often evaluated using benchmarks that hardly reflect real-world situations. For example, the impact of various video processing operations on detection accuracy has not been systematically assessed. To address this gap, this paper first analyzes numerous real-world influencing factors and typical video processing operations. Then, a more systematic assessment methodology is proposed, which allows for a quantitative evaluation of a detector's robustness under the influence of different processing operations. Moreover, substantial experiments have been carried out on three popular deepfake detectors, which give detailed analyses on the impact of each operation and bring insights to foster future research.

Title: Model-agnostic explainable artificial intelligence for object detection in image data. (arXiv:2303.17249v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17249
• Code URL: null
• Copy Paste: [[2303.17249] Model-agnostic explainable artificial intelligence for object detection in image data](http://arxiv.org/abs/2303.17249) #robust
• Summary:

  Object detection is a fundamental task in computer vision, which has been greatly progressed through developing large and intricate deep learning models. However, the lack of transparency is a big challenge that may not allow the widespread adoption of these models. Explainable artificial intelligence is a field of research where methods are developed to help users understand the behavior, decision logics, and vulnerabilities of AI-based systems. Black-box explanation refers to explaining decisions of an AI system without having access to its internals. In this paper, we design and implement a black-box explanation method named Black-box Object Detection Explanation by Masking (BODEM) through adopting a new masking approach for AI-based object detection systems. We propose local and distant masking to generate multiple versions of an input image. Local masks are used to disturb pixels within a target object to figure out how the object detector reacts to these changes, while distant masks are used to assess how the detection model's decisions are affected by disturbing pixels outside the object. A saliency map is then created by estimating the importance of pixels through measuring the difference between the detection output before and after masking. Finally, a heatmap is created that visualizes how important pixels within the input image are to the detected objects. The experimentations on various object detection datasets and models showed that BODEM can be effectively used to explain the behavior of object detectors and reveal their vulnerabilities. This makes BODEM suitable for explaining and validating AI based object detection systems in black-box software testing scenarios. Furthermore, we conducted data augmentation experiments that showed local masks produced by BODEM can be used for further training the object detectors and improve their detection accuracy and robustness.

Title: JCDNet: Joint of Common and Definite phases Network for Weakly Supervised Temporal Action Localization. (arXiv:2303.17294v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17294
• Code URL: null
• Copy Paste: [[2303.17294] JCDNet: Joint of Common and Definite phases Network for Weakly Supervised Temporal Action Localization](http://arxiv.org/abs/2303.17294) #robust
• Summary:

  Weakly-supervised temporal action localization aims to localize action instances in untrimmed videos with only video-level supervision. We witness that different actions record common phases, e.g., the run-up in the HighJump and LongJump. These different actions are defined as conjoint actions, whose rest parts are definite phases, e.g., leaping over the bar in a HighJump. Compared with the common phases, the definite phases are more easily localized in existing researches. Most of them formulate this task as a Multiple Instance Learning paradigm, in which the common phases are tended to be confused with the background, and affect the localization completeness of the conjoint actions. To tackle this challenge, we propose a Joint of Common and Definite phases Network (JCDNet) by improving feature discriminability of the conjoint actions. Specifically, we design a Class-Aware Discriminative module to enhance the contribution of the common phases in classification by the guidance of the coarse definite-phase features. Besides, we introduce a temporal attention module to learn robust action-ness scores via modeling temporal dependencies, distinguishing the common phases from the background. Extensive experiments on three datasets (THUMOS14, ActivityNetv1.2, and a conjoint-action subset) demonstrate that JCDNet achieves competitive performance against the state-of-the-art methods. Keywords: weakly-supervised learning, temporal action localization, conjoint action

Title: Understanding the Robustness of 3D Object Detection with Bird's-Eye-View Representations in Autonomous Driving. (arXiv:2303.17297v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17297
• Code URL: null
• Copy Paste: [[2303.17297] Understanding the Robustness of 3D Object Detection with Bird's-Eye-View Representations in Autonomous Driving](http://arxiv.org/abs/2303.17297) #robust
• Summary:

  3D object detection is an essential perception task in autonomous driving to understand the environments. The Bird's-Eye-View (BEV) representations have significantly improved the performance of 3D detectors with camera inputs on popular benchmarks. However, there still lacks a systematic understanding of the robustness of these vision-dependent BEV models, which is closely related to the safety of autonomous driving systems. In this paper, we evaluate the natural and adversarial robustness of various representative models under extensive settings, to fully understand their behaviors influenced by explicit BEV features compared with those without BEV. In addition to the classic settings, we propose a 3D consistent patch attack by applying adversarial patches in the 3D space to guarantee the spatiotemporal consistency, which is more realistic for the scenario of autonomous driving. With substantial experiments, we draw several findings: 1) BEV models tend to be more stable than previous methods under different natural conditions and common corruptions due to the expressive spatial representations; 2) BEV models are more vulnerable to adversarial noises, mainly caused by the redundant BEV features; 3) Camera-LiDAR fusion models have superior performance under different settings with multi-modal inputs, but BEV fusion model is still vulnerable to adversarial noises of both point cloud and image. These findings alert the safety issue in the applications of BEV detectors and could facilitate the development of more robust models.

Title: The impact of training dataset size and ensemble inference strategies on head and neck auto-segmentation. (arXiv:2303.17318v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17318
• Code URL: null
• Copy Paste: [[2303.17318] The impact of training dataset size and ensemble inference strategies on head and neck auto-segmentation](http://arxiv.org/abs/2303.17318) #robust
• Summary:

  Convolutional neural networks (CNNs) are increasingly being used to automate segmentation of organs-at-risk in radiotherapy. Since large sets of highly curated data are scarce, we investigated how much data is required to train accurate and robust head and neck auto-segmentation models. For this, an established 3D CNN was trained from scratch with different sized datasets (25-1000 scans) to segment the brainstem, parotid glands and spinal cord in CTs. Additionally, we evaluated multiple ensemble techniques to improve the performance of these models. The segmentations improved with training set size up to 250 scans and the ensemble methods significantly improved performance for all organs. The impact of the ensemble methods was most notable in the smallest datasets, demonstrating their potential for use in cases where large training datasets are difficult to obtain.

Title: PMatch: Paired Masked Image Modeling for Dense Geometric Matching. (arXiv:2303.17342v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17342
• Code URL: null
• Copy Paste: [[2303.17342] PMatch: Paired Masked Image Modeling for Dense Geometric Matching](http://arxiv.org/abs/2303.17342) #robust
• Summary:

  Dense geometric matching determines the dense pixel-wise correspondence between a source and support image corresponding to the same 3D structure. Prior works employ an encoder of transformer blocks to correlate the two-frame features. However, existing monocular pretraining tasks, e.g., image classification, and masked image modeling (MIM), can not pretrain the cross-frame module, yielding less optimal performance. To resolve this, we reformulate the MIM from reconstructing a single masked image to reconstructing a pair of masked images, enabling the pretraining of transformer module. Additionally, we incorporate a decoder into pretraining for improved upsampling results. Further, to be robust to the textureless area, we propose a novel cross-frame global matching module (CFGM). Since the most textureless area is planar surfaces, we propose a homography loss to further regularize its learning. Combined together, we achieve the State-of-The-Art (SoTA) performance on geometric matching. Codes and models are available at https://github.com/ShngJZ/PMatch.

Title: Complementary Random Masking for RGB-Thermal Semantic Segmentation. (arXiv:2303.17386v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17386
• Code URL: https://github.com/UkcheolShin/CRM_RGBTSeg
• Copy Paste: [[2303.17386] Complementary Random Masking for RGB-Thermal Semantic Segmentation](http://arxiv.org/abs/2303.17386) #robust
• Summary:

  RGB-thermal semantic segmentation is one potential solution to achieve reliable semantic scene understanding in adverse weather and lighting conditions. However, the previous studies mostly focus on designing a multi-modal fusion module without consideration of the nature of multi-modality inputs. Therefore, the networks easily become over-reliant on a single modality, making it difficult to learn complementary and meaningful representations for each modality. This paper proposes 1) a complementary random masking strategy of RGB-T images and 2) self-distillation loss between clean and masked input modalities. The proposed masking strategy prevents over-reliance on a single modality. It also improves the accuracy and robustness of the neural network by forcing the network to segment and classify objects even when one modality is partially available. Also, the proposed self-distillation loss encourages the network to extract complementary and meaningful representations from a single modality or complementary masked modalities. Based on the proposed method, we achieve state-of-the-art performance over three RGB-T semantic segmentation benchmarks. Our source code is available at https://github.com/UkcheolShin/CRM_RGBTSeg.

Title: NN-Copula-CD: A Copula-Guided Interpretable Neural Network for Change Detection in Heterogeneous Remote Sensing Images. (arXiv:2303.17448v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17448
• Code URL: null
• Copy Paste: [[2303.17448] NN-Copula-CD: A Copula-Guided Interpretable Neural Network for Change Detection in Heterogeneous Remote Sensing Images](http://arxiv.org/abs/2303.17448) #robust
• Summary:

  Change detection (CD) in heterogeneous remote sensing images is a practical and challenging issue for real-life emergencies. In the past decade, the heterogeneous CD problem has significantly benefited from the development of deep neural networks (DNN). However, the data-driven DNNs always perform like a black box where the lack of interpretability limits the trustworthiness and controllability of DNNs in most practical CD applications. As a strong knowledge-driven tool to measure correlation between random variables, Copula theory has been introduced into CD, yet it suffers from non-robust CD performance without manual prior selection for Copula functions. To address the above issues, we propose a knowledge-data-driven heterogeneous CD method (NN-Copula-CD) based on the Copula-guided interpretable neural network. In our NN-Copula-CD, the mathematical characteristics of Copula are designed as the losses to supervise a simple fully connected neural network to learn the correlation between bi-temporal image patches, and then the changed regions are identified via binary classification for the correlation coefficients of all image patch pairs of the bi-temporal images. We conduct in-depth experiments on three datasets with multimodal images (e.g., Optical, SAR, and NIR), where the quantitative results and visualized analysis demonstrate both the effectiveness and interpretability of the proposed NN-Copula-CD.

Title: PoseFormerV2: Exploring Frequency Domain for Efficient and Robust 3D Human Pose Estimation. (arXiv:2303.17472v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17472
• Code URL: https://github.com/qitaozhao/poseformerv2
• Copy Paste: [[2303.17472] PoseFormerV2: Exploring Frequency Domain for Efficient and Robust 3D Human Pose Estimation](http://arxiv.org/abs/2303.17472) #robust
• Summary:

  Recently, transformer-based methods have gained significant success in sequential 2D-to-3D lifting human pose estimation. As a pioneering work, PoseFormer captures spatial relations of human joints in each video frame and human dynamics across frames with cascaded transformer layers and has achieved impressive performance. However, in real scenarios, the performance of PoseFormer and its follow-ups is limited by two factors: (a) The length of the input joint sequence; (b) The quality of 2D joint detection. Existing methods typically apply self-attention to all frames of the input sequence, causing a huge computational burden when the frame number is increased to obtain advanced estimation accuracy, and they are not robust to noise naturally brought by the limited capability of 2D joint detectors. In this paper, we propose PoseFormerV2, which exploits a compact representation of lengthy skeleton sequences in the frequency domain to efficiently scale up the receptive field and boost robustness to noisy 2D joint detection. With minimum modifications to PoseFormer, the proposed method effectively fuses features both in the time domain and frequency domain, enjoying a better speed-accuracy trade-off than its precursor. Extensive experiments on two benchmark datasets (i.e., Human3.6M and MPI-INF-3DHP) demonstrate that the proposed approach significantly outperforms the original PoseFormer and other transformer-based variants. Code is released at \url{https://github.com/QitaoZhao/PoseFormerV2}.

Title: 3D Line Mapping Revisited. (arXiv:2303.17504v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17504
• Code URL: https://github.com/cvg/limap
• Copy Paste: [[2303.17504] 3D Line Mapping Revisited](http://arxiv.org/abs/2303.17504) #robust
• Summary:

  In contrast to sparse keypoints, a handful of line segments can concisely encode the high-level scene layout, as they often delineate the main structural elements. In addition to offering strong geometric cues, they are also omnipresent in urban landscapes and indoor scenes. Despite their apparent advantages, current line-based reconstruction methods are far behind their point-based counterparts. In this paper we aim to close the gap by introducing LIMAP, a library for 3D line mapping that robustly and efficiently creates 3D line maps from multi-view imagery. This is achieved through revisiting the degeneracy problem of line triangulation, carefully crafted scoring and track building, and exploiting structural priors such as line coincidence, parallelism, and orthogonality. Our code integrates seamlessly with existing point-based Structure-from-Motion methods and can leverage their 3D points to further improve the line reconstruction. Furthermore, as a byproduct, the method is able to recover 3D association graphs between lines and points / vanishing points (VPs). In thorough experiments, we show that LIMAP significantly outperforms existing approaches for 3D line mapping. Our robust 3D line maps also open up new research directions. We show two example applications: visual localization and bundle adjustment, where integrating lines alongside points yields the best results. Code is available at https://github.com/cvg/limap.

Title: CAusal and collaborative proxy-tasKs lEarning for Semi-Supervised Domain Adaptation. (arXiv:2303.17526v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17526
• Code URL: null
• Copy Paste: [[2303.17526] CAusal and collaborative proxy-tasKs lEarning for Semi-Supervised Domain Adaptation](http://arxiv.org/abs/2303.17526) #robust
• Summary:

  Semi-supervised domain adaptation (SSDA) adapts a learner to a new domain by effectively utilizing source domain data and a few labeled target samples. It is a practical yet under-investigated research topic. In this paper, we analyze the SSDA problem from two perspectives that have previously been overlooked, and correspondingly decompose it into two \emph{key subproblems}: \emph{robust domain adaptation (DA) learning} and \emph{maximal cross-domain data utilization}. \textbf{(i)} From a causal theoretical view, a robust DA model should distinguish the invariant ``concept'' (key clue to image label) from the nuisance of confounding factors across domains. To achieve this goal, we propose to generate \emph{concept-invariant samples} to enable the model to classify the samples through causal intervention, yielding improved generalization guarantees; \textbf{(ii)} Based on the robust DA theory, we aim to exploit the maximal utilization of rich source domain data and a few labeled target samples to boost SSDA further. Consequently, we propose a collaboratively debiasing learning framework that utilizes two complementary semi-supervised learning (SSL) classifiers to mutually exchange their unbiased knowledge, which helps unleash the potential of source and target domain training data, thereby producing more convincing pseudo-labels. Such obtained labels facilitate cross-domain feature alignment and duly improve the invariant concept learning. In our experimental study, we show that the proposed model significantly outperforms SOTA methods in terms of effectiveness and generalisability on SSDA datasets.

Title: Neglected Free Lunch -- Learning Image Classifiers Using Annotation Byproducts. (arXiv:2303.17595v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17595
• Code URL: https://github.com/naver-ai/neglectedfreelunch
• Copy Paste: [[2303.17595] Neglected Free Lunch -- Learning Image Classifiers Using Annotation Byproducts](http://arxiv.org/abs/2303.17595) #robust
• Summary:

  Supervised learning of image classifiers distills human knowledge into a parametric model through pairs of images and corresponding labels (X,Y). We argue that this simple and widely used representation of human knowledge neglects rich auxiliary information from the annotation procedure, such as the time-series of mouse traces and clicks left after image selection. Our insight is that such annotation byproducts Z provide approximate human attention that weakly guides the model to focus on the foreground cues, reducing spurious correlations and discouraging shortcut learning. To verify this, we create ImageNet-AB and COCO-AB. They are ImageNet and COCO training sets enriched with sample-wise annotation byproducts, collected by replicating the respective original annotation tasks. We refer to the new paradigm of training models with annotation byproducts as learning using annotation byproducts (LUAB). We show that a simple multitask loss for regressing Z together with Y already improves the generalisability and robustness of the learned models. Compared to the original supervised learning, LUAB does not require extra annotation costs. ImageNet-AB and COCO-AB are at https://github.com/naver-ai/NeglectedFreeLunch.

Title: Robo3D: Towards Robust and Reliable 3D Perception against Corruptions. (arXiv:2303.17597v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17597
• Code URL: https://github.com/ldkong1205/Robo3D
• Copy Paste: [[2303.17597] Robo3D: Towards Robust and Reliable 3D Perception against Corruptions](http://arxiv.org/abs/2303.17597) #robust
• Summary:

  The robustness of 3D perception systems under natural corruptions from environments and sensors is pivotal for safety-critical applications. Existing large-scale 3D perception datasets often contain data that are meticulously cleaned. Such configurations, however, cannot reflect the reliability of perception models during the deployment stage. In this work, we present Robo3D, the first comprehensive benchmark heading toward probing the robustness of 3D detectors and segmentors under out-of-distribution scenarios against natural corruptions that occur in real-world environments. Specifically, we consider eight corruption types stemming from adversarial weather conditions, external disturbances, and internal sensor failure. We uncover that, although promising results have been progressively achieved on standard benchmarks, state-of-the-art 3D perception models are at risk of being vulnerable to corruptions. We draw key observations on the use of data representations, augmentation schemes, and training strategies, that could severely affect the model's performance. To pursue better robustness, we propose a density-insensitive training framework along with a simple flexible voxelization strategy to enhance the model resiliency. We hope our benchmark and approach could inspire future research in designing more robust and reliable 3D perception models. Our robustness benchmark suite is publicly available.

Title: AvatarCraft: Transforming Text into Neural Human Avatars with Parameterized Shape and Pose Control. (arXiv:2303.17606v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17606
• Code URL: null
• Copy Paste: [[2303.17606] AvatarCraft: Transforming Text into Neural Human Avatars with Parameterized Shape and Pose Control](http://arxiv.org/abs/2303.17606) #robust
• Summary:

  Neural implicit fields are powerful for representing 3D scenes and generating high-quality novel views, but it remains challenging to use such implicit representations for creating a 3D human avatar with a specific identity and artistic style that can be easily animated. Our proposed method, AvatarCraft, addresses this challenge by using diffusion models to guide the learning of geometry and texture for a neural avatar based on a single text prompt. We carefully design the optimization framework of neural implicit fields, including a coarse-to-fine multi-bounding box training strategy, shape regularization, and diffusion-based constraints, to produce high-quality geometry and texture. Additionally, we make the human avatar animatable by deforming the neural implicit field with an explicit warping field that maps the target human mesh to a template human mesh, both represented using parametric human models. This simplifies animation and reshaping of the generated avatar by controlling pose and shape parameters. Extensive experiments on various text descriptions show that AvatarCraft is effective and robust in creating human avatars and rendering novel views, poses, and shapes. Our project page is: \url{https://avatar-craft.github.io/}.

Title: Medical Intervention Duration Estimation Using Language-enhanced Transformer Encoder with Medical Prompts. (arXiv:2303.17408v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2303.17408
• Code URL: null
• Copy Paste: [[2303.17408] Medical Intervention Duration Estimation Using Language-enhanced Transformer Encoder with Medical Prompts](http://arxiv.org/abs/2303.17408) #robust
• Summary:

  In recent years, estimating the duration of medical intervention based on electronic health records (EHRs) has gained significant attention in the filed of clinical decision support. However, current models largely focus on structured data, leaving out information from the unstructured clinical free-text data. To address this, we present a novel language-enhanced transformer-based framework, which projects all relevant clinical data modalities (continuous, categorical, binary, and free-text features) into a harmonized language latent space using a pre-trained sentence encoder with the help of medical prompts. The proposed method enables the integration of information from different modalities within the cell transformer encoder and leads to more accurate duration estimation for medical intervention. Our experimental results on both US-based (length of stay in ICU estimation) and Asian (surgical duration prediction) medical datasets demonstrate the effectiveness of our proposed framework, which outperforms tailored baseline approaches and exhibits robustness to data corruption in EHRs.

Title: Recognition, recall, and retention of few-shot memories in large language models. (arXiv:2303.17557v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2303.17557
• Code URL: https://github.com/eminorhan/llm-memory
• Copy Paste: [[2303.17557] Recognition, recall, and retention of few-shot memories in large language models](http://arxiv.org/abs/2303.17557) #robust
• Summary:

  The training of modern large language models (LLMs) takes place in a regime where most training examples are seen only a few times by the model during the course of training. What does a model remember about such examples seen only a few times during training and how long does that memory persist in the face of continuous training with new examples? Here, we investigate these questions through simple recognition, recall, and retention experiments with LLMs. In recognition experiments, we ask if the model can distinguish the seen example from a novel example; in recall experiments, we ask if the model can correctly recall the seen example when cued by a part of it; and in retention experiments, we periodically probe the model's memory for the original examples as the model is trained continuously with new examples. We find that a single exposure is generally sufficient for a model to achieve near perfect accuracy even in very challenging recognition experiments. We estimate that the recognition performance of even small language models easily exceeds human recognition performance reported in similar experiments with humans (Shepard, 1967). Achieving near perfect recall takes more exposures, but most models can do it in just 3 exposures. The flip side of this remarkable capacity for fast learning is that precise memories are quickly overwritten: recall performance for the original examples drops steeply over the first 10 training updates with new examples, followed by a more gradual decline. Even after 100K updates, however, some of the original examples are still recalled near perfectly. A qualitatively similar retention pattern has been observed in human long-term memory retention studies before (Bahrick, 1984). Finally, recognition is much more robust to interference than recall and memory for natural language sentences is generally superior to memory for stimuli without structure.

Title: Fuzzified advanced robust hashes for identification of digital and physical objects. (arXiv:2303.17499v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2303.17499
• Code URL: null
• Copy Paste: [[2303.17499] Fuzzified advanced robust hashes for identification of digital and physical objects](http://arxiv.org/abs/2303.17499) #robust
• Summary:

  With the rising numbers for IoT objects, it is becoming easier to penetrate counterfeit objects into the mainstream market by adversaries. Such infiltration of bogus products can be addressed with third-party-verifiable identification. Generally, state-of-the-art identification schemes do not guarantee that an identifier e.g. barcodes or RFID itself cannot be forged. This paper introduces identification patterns representing the objects intrinsic identity by robust hashes and not only by generated identification patterns. Inspired by these two notions, a collection of uniquely identifiable attributes called quasi-identifiers (QI) can be used to identify an object. Since all attributes do not contribute equally towards an object's identity, each QI has a different contribution towards the identifier. A robust hash developed utilising the QI has been named fuzzified robust hashes (FaR hashes), which can be used as an object identifier. Although the FaR hash is a single hash string, selected bits change in response to the modification of QI. On the other hand, other QIs in the object are more important for the object's identity. If these QIs change, the complete FaR hash is going to change. The calculation of FaR hash using attributes should allow third parties to generate the identifier and compare it with the current one to verify the genuineness of the object.

Title: MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations. (arXiv:2303.17156v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2303.17156
• Code URL: null
• Copy Paste: [[2303.17156] MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations](http://arxiv.org/abs/2303.17156) #robust
• Summary:

  We study a new paradigm for sequential decision making, called offline Policy Learning from Observation (PLfO). Offline PLfO aims to learn policies using datasets with substandard qualities: 1) only a subset of trajectories is labeled with rewards, 2) labeled trajectories may not contain actions, 3) labeled trajectories may not be of high quality, and 4) the overall data may not have full coverage. Such imperfection is common in real-world learning scenarios, so offline PLfO encompasses many existing offline learning setups, including offline imitation learning (IL), ILfO, and reinforcement learning (RL). In this work, we present a generic approach, called Modality-agnostic Adversarial Hypothesis Adaptation for Learning from Observations (MAHALO), for offline PLfO. Built upon the pessimism concept in offline RL, MAHALO optimizes the policy using a performance lower bound that accounts for uncertainty due to the dataset's insufficient converge. We implement this idea by adversarially training data-consistent critic and reward functions in policy optimization, which forces the learned policy to be robust to the data deficiency. We show that MAHALO consistently outperforms or matches specialized algorithms across a variety of offline PLfO tasks in theory and experiments.

biometric

steal

extraction

Title: T-FFTRadNet: Object Detection with Swin Vision Transformers from Raw ADC Radar Signals. (arXiv:2303.16940v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.16940
• Code URL: null
• Copy Paste: [[2303.16940] T-FFTRadNet: Object Detection with Swin Vision Transformers from Raw ADC Radar Signals](http://arxiv.org/abs/2303.16940) #extraction
• Summary:

  Object detection utilizing Frequency Modulated Continous Wave radar is becoming increasingly popular in the field of autonomous systems. Radar does not possess the same drawbacks seen by other emission-based sensors such as LiDAR, primarily the degradation or loss of return signals due to weather conditions such as rain or snow. However, radar does possess traits that make it unsuitable for standard emission-based deep learning representations such as point clouds. Radar point clouds tend to be sparse and therefore information extraction is not efficient. To overcome this, more traditional digital signal processing pipelines were adapted to form inputs residing directly in the frequency domain via Fast Fourier Transforms. Commonly, three transformations were used to form Range-Azimuth-Doppler cubes in which deep learning algorithms could perform object detection. This too has drawbacks, namely the pre-processing costs associated with performing multiple Fourier Transforms and normalization. We explore the possibility of operating on raw radar inputs from analog to digital converters via the utilization of complex transformation layers. Moreover, we introduce hierarchical Swin Vision transformers to the field of radar object detection and show their capability to operate on inputs varying in pre-processing, along with different radar configurations, i.e. relatively low and high numbers of transmitters and receivers, while obtaining on par or better results than the state-of-the-art.

Title: PartManip: Learning Cross-Category Generalizable Part Manipulation Policy from Point Cloud Observations. (arXiv:2303.16958v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.16958
• Code URL: null
• Copy Paste: [[2303.16958] PartManip: Learning Cross-Category Generalizable Part Manipulation Policy from Point Cloud Observations](http://arxiv.org/abs/2303.16958) #extraction
• Summary:

  Learning a generalizable object manipulation policy is vital for an embodied agent to work in complex real-world scenes. Parts, as the shared components in different object categories, have the potential to increase the generalization ability of the manipulation policy and achieve cross-category object manipulation. In this work, we build the first large-scale, part-based cross-category object manipulation benchmark, PartManip, which is composed of 11 object categories, 494 objects, and 1432 tasks in 6 task classes. Compared to previous work, our benchmark is also more diverse and realistic, i.e., having more objects and using sparse-view point cloud as input without oracle information like part segmentation. To tackle the difficulties of vision-based policy learning, we first train a state-based expert with our proposed part-based canonicalization and part-aware rewards, and then distill the knowledge to a vision-based student. We also find an expressive backbone is essential to overcome the large diversity of different objects. For cross-category generalization, we introduce domain adversarial learning for domain-invariant feature extraction. Extensive experiments in simulation show that our learned policy can outperform other methods by a large margin, especially on unseen object categories. We also demonstrate our method can successfully manipulate novel objects in the real world.

Title: Online Camera-to-ground Calibration for Autonomous Driving. (arXiv:2303.17137v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17137
• Code URL: null
• Copy Paste: [[2303.17137] Online Camera-to-ground Calibration for Autonomous Driving](http://arxiv.org/abs/2303.17137) #extraction
• Summary:

  Online camera-to-ground calibration is to generate a non-rigid body transformation between the camera and the road surface in a real-time manner. Existing solutions utilize static calibration, suffering from environmental variations such as tire pressure changes, vehicle loading volume variations, and road surface diversity. Other online solutions exploit the usage of road elements or photometric consistency between overlapping views across images, which require continuous detection of specific targets on the road or assistance with multiple cameras to facilitate calibration. In our work, we propose an online monocular camera-to-ground calibration solution that does not utilize any specific targets while driving. We perform a coarse-to-fine approach for ground feature extraction through wheel odometry and estimate the camera-to-ground calibration parameters through a sliding-window-based factor graph optimization. Considering the non-rigid transformation of camera-to-ground while driving, we provide metrics to quantify calibration performance and stopping criteria to report/broadcast our satisfying calibration results. Extensive experiments using real-world data demonstrate that our algorithm is effective and outperforms state-of-the-art techniques.

Title: Unsupervised Anomaly Detection with Local-Sensitive VQVAE and Global-Sensitive Transformers. (arXiv:2303.17505v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17505
• Code URL: null
• Copy Paste: [[2303.17505] Unsupervised Anomaly Detection with Local-Sensitive VQVAE and Global-Sensitive Transformers](http://arxiv.org/abs/2303.17505) #extraction
• Summary:

  Unsupervised anomaly detection (UAD) has been widely implemented in industrial and medical applications, which reduces the cost of manual annotation and improves efficiency in disease diagnosis. Recently, deep auto-encoder with its variants has demonstrated its advantages in many UAD scenarios. Training on the normal data, these models are expected to locate anomalies by producing higher reconstruction error for the abnormal areas than the normal ones. However, this assumption does not always hold because of the uncontrollable generalization capability. To solve this problem, we present LSGS, a method that builds on Vector Quantised-Variational Autoencoder (VQVAE) with a novel aggregated codebook and transformers with global attention. In this work, the VQVAE focus on feature extraction and reconstruction of images, and the transformers fit the manifold and locate anomalies in the latent space. Then, leveraging the generated encoding sequences that conform to a normal distribution, we can reconstruct a more accurate image for locating the anomalies. Experiments on various datasets demonstrate the effectiveness of the proposed method.

Title: BEVERS: A General, Simple, and Performant Framework for Automatic Fact Verification. (arXiv:2303.16974v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2303.16974
• Code URL: null
• Copy Paste: [[2303.16974] BEVERS: A General, Simple, and Performant Framework for Automatic Fact Verification](http://arxiv.org/abs/2303.16974) #extraction
• Summary:

  Automatic fact verification has become an increasingly popular topic in recent years and among datasets the Fact Extraction and VERification (FEVER) dataset is one of the most popular. In this work we present BEVERS, a tuned baseline system for the FEVER dataset. Our pipeline uses standard approaches for document retrieval, sentence selection, and final claim classification, however, we spend considerable effort ensuring optimal performance for each component. The results are that BEVERS achieves the highest FEVER score and label accuracy among all systems, published or unpublished. We also apply this pipeline to another fact verification dataset, Scifact, and achieve the highest label accuracy among all systems on that dataset as well. We also make our full code available.

Title: TLAG: An Informative Trigger and Label-Aware Knowledge Guided Model for Dialogue-based Relation Extraction. (arXiv:2303.17119v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2303.17119
• Code URL: null
• Copy Paste: [[2303.17119] TLAG: An Informative Trigger and Label-Aware Knowledge Guided Model for Dialogue-based Relation Extraction](http://arxiv.org/abs/2303.17119) #extraction
• Summary:

  Dialogue-based Relation Extraction (DRE) aims to predict the relation type of argument pairs that are mentioned in dialogue. The latest trigger-enhanced methods propose trigger prediction tasks to promote DRE. However, these methods are not able to fully leverage the trigger information and even bring noise to relation extraction. To solve these problems, we propose TLAG, which fully leverages the trigger and label-aware knowledge to guide the relation extraction. First, we design an adaptive trigger fusion module to fully leverage the trigger information. Then, we introduce label-aware knowledge to further promote our model's performance. Experimental results on the DialogRE dataset show that our TLAG outperforms the baseline models, and detailed analyses demonstrate the effectiveness of our approach.

Title: Topics in the Haystack: Extracting and Evaluating Topics beyond Coherence. (arXiv:2303.17324v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2303.17324
• Code URL: null
• Copy Paste: [[2303.17324] Topics in the Haystack: Extracting and Evaluating Topics beyond Coherence](http://arxiv.org/abs/2303.17324) #extraction
• Summary:

  Extracting and identifying latent topics in large text corpora has gained increasing importance in Natural Language Processing (NLP). Most models, whether probabilistic models similar to Latent Dirichlet Allocation (LDA) or neural topic models, follow the same underlying approach of topic interpretability and topic extraction. We propose a method that incorporates a deeper understanding of both sentence and document themes, and goes beyond simply analyzing word frequencies in the data. This allows our model to detect latent topics that may include uncommon words or neologisms, as well as words not present in the documents themselves. Additionally, we propose several new evaluation metrics based on intruder words and similarity measures in the semantic space. We present correlation coefficients with human identification of intruder words and achieve near-human level results at the word-intrusion task. We demonstrate the competitive performance of our method with a large benchmark study, and achieve superior results compared to state-of-the-art topic modeling and document clustering models.

membership infer

federate

Title: Federated Stochastic Bandit Learning with Unobserved Context. (arXiv:2303.17043v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2303.17043
• Code URL: null
• Copy Paste: [[2303.17043] Federated Stochastic Bandit Learning with Unobserved Context](http://arxiv.org/abs/2303.17043) #federate
• Summary:

  We study the problem of federated stochastic multi-arm contextual bandits with unknown contexts, in which M agents are faced with different bandits and collaborate to learn. The communication model consists of a central server and the agents share their estimates with the central server periodically to learn to choose optimal actions in order to minimize the total regret. We assume that the exact contexts are not observable and the agents observe only a distribution of the contexts. Such a situation arises, for instance, when the context itself is a noisy measurement or based on a prediction mechanism. Our goal is to develop a distributed and federated algorithm that facilitates collaborative learning among the agents to select a sequence of optimal actions so as to maximize the cumulative reward. By performing a feature vector transformation, we propose an elimination-based algorithm and prove the regret bound for linearly parametrized reward functions. Finally, we validated the performance of our algorithm and compared it with another baseline approach using numerical simulations on synthetic data and on the real-world movielens dataset.

Title: DPP-based Client Selection for Federated Learning with Non-IID Data. (arXiv:2303.17358v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2303.17358
• Code URL: null
• Copy Paste: [[2303.17358] DPP-based Client Selection for Federated Learning with Non-IID Data](http://arxiv.org/abs/2303.17358) #federate
• Summary:

  This paper proposes a client selection (CS) method to tackle the communication bottleneck of federated learning (FL) while concurrently coping with FL's data heterogeneity issue. Specifically, we first analyze the effect of CS in FL and show that FL training can be accelerated by adequately choosing participants to diversify the training dataset in each round of training. Based on this, we leverage data profiling and determinantal point process (DPP) sampling techniques to develop an algorithm termed Federated Learning with DPP-based Participant Selection (FL-DP$^3$S). This algorithm effectively diversifies the participants' datasets in each round of training while preserving their data privacy. We conduct extensive experiments to examine the efficacy of our proposed method. The results show that our scheme attains a faster convergence rate, as well as a smaller communication overhead than several baselines.

fair

Title: Are Neural Architecture Search Benchmarks Well Designed? A Deeper Look Into Operation Importance. (arXiv:2303.16938v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2303.16938
• Code URL: https://github.com/vascolopes/nas-benchmark-evaluation
• Copy Paste: [[2303.16938] Are Neural Architecture Search Benchmarks Well Designed? A Deeper Look Into Operation Importance](http://arxiv.org/abs/2303.16938) #fair
• Summary:

  Neural Architecture Search (NAS) benchmarks significantly improved the capability of developing and comparing NAS methods while at the same time drastically reduced the computational overhead by providing meta-information about thousands of trained neural networks. However, tabular benchmarks have several drawbacks that can hinder fair comparisons and provide unreliable results. These usually focus on providing a small pool of operations in heavily constrained search spaces -- usually cell-based neural networks with pre-defined outer-skeletons. In this work, we conducted an empirical analysis of the widely used NAS-Bench-101, NAS-Bench-201 and TransNAS-Bench-101 benchmarks in terms of their generability and how different operations influence the performance of the generated architectures. We found that only a subset of the operation pool is required to generate architectures close to the upper-bound of the performance range. Also, the performance distribution is negatively skewed, having a higher density of architectures in the upper-bound range. We consistently found convolution layers to have the highest impact on the architecture's performance, and that specific combination of operations favors top-scoring architectures. These findings shed insights on the correct evaluation and comparison of NAS methods using NAS benchmarks, showing that directly searching on NAS-Bench-201, ImageNet16-120 and TransNAS-Bench-101 produces more reliable results than searching only on CIFAR-10. Furthermore, with this work we provide suggestions for future benchmark evaluations and design. The code used to conduct the evaluations is available at https://github.com/VascoLopes/NAS-Benchmark-Evaluation.

Title: Fairness-Aware Data Valuation for Supervised Learning. (arXiv:2303.16963v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2303.16963
• Code URL: null
• Copy Paste: [[2303.16963] Fairness-Aware Data Valuation for Supervised Learning](http://arxiv.org/abs/2303.16963) #fair
• Summary:

  Data valuation is a ML field that studies the value of training instances towards a given predictive task. Although data bias is one of the main sources of downstream model unfairness, previous work in data valuation does not consider how training instances may influence both performance and fairness of ML models. Thus, we propose Fairness-Aware Data vauatiOn (FADO), a data valuation framework that can be used to incorporate fairness concerns into a series of ML-related tasks (e.g., data pre-processing, exploratory data analysis, active learning). We propose an entropy-based data valuation metric suited to address our two-pronged goal of maximizing both performance and fairness, which is more computationally efficient than existing metrics. We then show how FADO can be applied as the basis for unfairness mitigation pre-processing techniques. Our methods achieve promising results -- up to a 40 p.p. improvement in fairness at a less than 1 p.p. loss in performance compared to a baseline -- and promote fairness in a data-centric way, where a deeper understanding of data quality takes center stage.

Title: Non-Invasive Fairness in Learning through the Lens of Data Drift. (arXiv:2303.17566v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2303.17566
• Code URL: null
• Copy Paste: [[2303.17566] Non-Invasive Fairness in Learning through the Lens of Data Drift](http://arxiv.org/abs/2303.17566) #fair
• Summary:

  Machine Learning (ML) models are widely employed to drive many modern data systems. While they are undeniably powerful tools, ML models often demonstrate imbalanced performance and unfair behaviors. The root of this problem often lies in the fact that different subpopulations commonly display divergent trends: as a learning algorithm tries to identify trends in the data, it naturally favors the trends of the majority groups, leading to a model that performs poorly and unfairly for minority populations. Our goal is to improve the fairness and trustworthiness of ML models by applying only non-invasive interventions, i.e., without altering the data or the learning algorithm. We use a simple but key insight: the divergence of trends between different populations, and, consecutively, between a learned model and minority populations, is analogous to data drift, which indicates the poor conformance between parts of the data and the trained model. We explore two strategies (model-splitting and reweighing) to resolve this drift, aiming to improve the overall conformance of models to the underlying data. Both our methods introduce novel ways to employ the recently-proposed data profiling primitive of Conformance Constraints. Our experimental evaluation over 7 real-world datasets shows that both DifFair and ConFair improve the fairness of ML models.

interpretability

explainability

watermark

diffusion

Title: HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion. (arXiv:2303.17015v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17015
• Code URL: null
• Copy Paste: [[2303.17015] HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion](http://arxiv.org/abs/2303.17015) #diffusion
• Summary:

  Implicit neural fields, typically encoded by a multilayer perceptron (MLP) that maps from coordinates (e.g., xyz) to signals (e.g., signed distances), have shown remarkable promise as a high-fidelity and compact representation. However, the lack of a regular and explicit grid structure also makes it challenging to apply generative modeling directly on implicit neural fields in order to synthesize new data. To this end, we propose HyperDiffusion, a novel approach for unconditional generative modeling of implicit neural fields. HyperDiffusion operates directly on MLP weights and generates new neural implicit fields encoded by synthesized MLP parameters. Specifically, a collection of MLPs is first optimized to faithfully represent individual data samples. Subsequently, a diffusion process is trained in this MLP weight space to model the underlying distribution of neural implicit fields. HyperDiffusion enables diffusion modeling over a implicit, compact, and yet high-fidelity representation of complex signals across 3D shapes and 4D mesh animations within one single unified framework.

Title: DiffCollage: Parallel Generation of Large Content with Diffusion Models. (arXiv:2303.17076v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17076
• Code URL: null
• Copy Paste: [[2303.17076] DiffCollage: Parallel Generation of Large Content with Diffusion Models](http://arxiv.org/abs/2303.17076) #diffusion
• Summary:

  We present DiffCollage, a compositional diffusion model that can generate large content by leveraging diffusion models trained on generating pieces of the large content. Our approach is based on a factor graph representation where each factor node represents a portion of the content and a variable node represents their overlap. This representation allows us to aggregate intermediate outputs from diffusion models defined on individual nodes to generate content of arbitrary size and shape in parallel without resorting to an autoregressive generation procedure. We apply DiffCollage to various tasks, including infinite image generation, panorama image generation, and long-duration text-guided motion generation. Extensive experimental results with a comparison to strong autoregressive baselines verify the effectiveness of our approach.

Title: Discriminative Class Tokens for Text-to-Image Diffusion Models. (arXiv:2303.17155v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17155
• Code URL: https://github.com/idansc/discriminative_class_tokens
• Copy Paste: [[2303.17155] Discriminative Class Tokens for Text-to-Image Diffusion Models](http://arxiv.org/abs/2303.17155) #diffusion
• Summary:

  Recent advances in text-to-image diffusion models have enabled the generation of diverse and high-quality images. However, generated images often fall short of depicting subtle details and are susceptible to errors due to ambiguity in the input text. One way of alleviating these issues is to train diffusion models on class-labeled datasets. This comes with a downside, doing so limits their expressive power: (i) supervised datasets are generally small compared to large-scale scraped text-image datasets on which text-to-image models are trained, and so the quality and diversity of generated images are severely affected, or (ii) the input is a hard-coded label, as opposed to free-form text, which limits the control over the generated images.

In this work, we propose a non-invasive fine-tuning technique that capitalizes on the expressive potential of free-form text while achieving high accuracy through discriminative signals from a pretrained classifier, which guides the generation. This is done by iteratively modifying the embedding of a single input token of a text-to-image diffusion model, using the classifier, by steering generated images toward a given target class. Our method is fast compared to prior fine-tuning methods and does not require a collection of in-class images or retraining of a noise-tolerant classifier. We evaluate our method extensively, showing that the generated images are: (i) more accurate and of higher quality than standard diffusion models, (ii) can be used to augment training data in a low-resource setting, and (iii) reveal information about the data used to train the guiding classifier. The code is available at \url{https://github.com/idansc/discriminative_class_tokens}

Title: LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation. (arXiv:2303.17189v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17189
• Code URL: https://github.com/zgctroy/layoutdiffusion
• Copy Paste: [[2303.17189] LayoutDiffusion: Controllable Diffusion Model for Layout-to-image Generation](http://arxiv.org/abs/2303.17189) #diffusion
• Summary:

  Recently, diffusion models have achieved great success in image synthesis. However, when it comes to the layout-to-image generation where an image often has a complex scene of multiple objects, how to make strong control over both the global layout map and each detailed object remains a challenging task. In this paper, we propose a diffusion model named LayoutDiffusion that can obtain higher generation quality and greater controllability than the previous works. To overcome the difficult multimodal fusion of image and layout, we propose to construct a structural image patch with region information and transform the patched image into a special layout to fuse with the normal layout in a unified form. Moreover, Layout Fusion Module (LFM) and Object-aware Cross Attention (OaCA) are proposed to model the relationship among multiple objects and designed to be object-aware and position-sensitive, allowing for precisely controlling the spatial related information. Extensive experiments show that our LayoutDiffusion outperforms the previous SOTA methods on FID, CAS by relatively 46.35%, 26.70% on COCO-stuff and 44.29%, 41.82% on VG. Code is available at https://github.com/ZGCTroy/LayoutDiffusion.

Title: PAIR-Diffusion: Object-Level Image Editing with Structure-and-Appearance Paired Diffusion Models. (arXiv:2303.17546v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17546
• Code URL: https://github.com/picsart-ai-research/pair-diffusion
• Copy Paste: [[2303.17546] PAIR-Diffusion: Object-Level Image Editing with Structure-and-Appearance Paired Diffusion Models](http://arxiv.org/abs/2303.17546) #diffusion
• Summary:

  Image editing using diffusion models has witnessed extremely fast-paced growth recently. There are various ways in which previous works enable controlling and editing images. Some works use high-level conditioning such as text, while others use low-level conditioning. Nevertheless, most of them lack fine-grained control over the properties of the different objects present in the image, i.e. object-level image editing. In this work, we consider an image as a composition of multiple objects, each defined by various properties. Out of these properties, we identify structure and appearance as the most intuitive to understand and useful for editing purposes. We propose Structure-and-Appearance Paired Diffusion model (PAIR-Diffusion), which is trained using structure and appearance information explicitly extracted from the images. The proposed model enables users to inject a reference image's appearance into the input image at both the object and global levels. Additionally, PAIR-Diffusion allows editing the structure while maintaining the style of individual components of the image unchanged. We extensively evaluate our method on LSUN datasets and the CelebA-HQ face dataset, and we demonstrate fine-grained control over both structure and appearance at the object level. We also applied the method to Stable Diffusion to edit any real image at the object level.

Title: DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder. (arXiv:2303.17550v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17550
• Code URL: null
• Copy Paste: [[2303.17550] DAE-Talker: High Fidelity Speech-Driven Talking Face Generation with Diffusion Autoencoder](http://arxiv.org/abs/2303.17550) #diffusion
• Summary:

  While recent research has made significant progress in speech-driven talking face generation, the quality of the generated video still lags behind that of real recordings. One reason for this is the use of handcrafted intermediate representations like facial landmarks and 3DMM coefficients, which are designed based on human knowledge and are insufficient to precisely describe facial movements. Additionally, these methods require an external pretrained model for extracting these representations, whose performance sets an upper bound on talking face generation. To address these limitations, we propose a novel method called DAE-Talker that leverages data-driven latent representations obtained from a diffusion autoencoder (DAE). DAE contains an image encoder that encodes an image into a latent vector and a DDIM image decoder that reconstructs the image from it. We train our DAE on talking face video frames and then extract their latent representations as the training target for a Conformer-based speech2latent model. This allows DAE-Talker to synthesize full video frames and produce natural head movements that align with the content of speech, rather than relying on a predetermined head pose from a template video. We also introduce pose modelling in speech2latent for pose controllability. Additionally, we propose a novel method for generating continuous video frames with the DDIM image decoder trained on individual frames, eliminating the need for modelling the joint distribution of consecutive frames directly. Our experiments show that DAE-Talker outperforms existing popular methods in lip-sync, video fidelity, and pose naturalness. We also conduct ablation studies to analyze the effectiveness of the proposed techniques and demonstrate the pose controllability of DAE-Talker.

Title: DDP: Diffusion Model for Dense Visual Prediction. (arXiv:2303.17559v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17559
• Code URL: null
• Copy Paste: [[2303.17559] DDP: Diffusion Model for Dense Visual Prediction](http://arxiv.org/abs/2303.17559) #diffusion
• Summary:

  We propose a simple, efficient, yet powerful framework for dense visual predictions based on the conditional diffusion pipeline. Our approach follows a "noise-to-map" generative paradigm for prediction by progressively removing noise from a random Gaussian distribution, guided by the image. The method, called DDP, efficiently extends the denoising diffusion process into the modern perception pipeline. Without task-specific design and architecture customization, DDP is easy to generalize to most dense prediction tasks, e.g., semantic segmentation and depth estimation. In addition, DDP shows attractive properties such as dynamic inference and uncertainty awareness, in contrast to previous single-step discriminative methods. We show top results on three representative tasks with six diverse benchmarks, without tricks, DDP achieves state-of-the-art or competitive performance on each task compared to the specialist counterparts. For example, semantic segmentation (83.9 mIoU on Cityscapes), BEV map segmentation (70.6 mIoU on nuScenes), and depth estimation (0.05 REL on KITTI). We hope that our approach will serve as a solid baseline and facilitate future research

Title: Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models. (arXiv:2303.17591v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17591
• Code URL: https://github.com/shi-labs/forget-me-not
• Copy Paste: [[2303.17591] Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models](http://arxiv.org/abs/2303.17591) #diffusion
• Summary:

  The unlearning problem of deep learning models, once primarily an academic concern, has become a prevalent issue in the industry. The significant advances in text-to-image generation techniques have prompted global discussions on privacy, copyright, and safety, as numerous unauthorized personal IDs, content, artistic creations, and potentially harmful materials have been learned by these models and later utilized to generate and distribute uncontrolled content. To address this challenge, we propose \textbf{Forget-Me-Not}, an efficient and low-cost solution designed to safely remove specified IDs, objects, or styles from a well-configured text-to-image model in as little as 30 seconds, without impairing its ability to generate other content. Alongside our method, we introduce the \textbf{Memorization Score (M-Score)} and \textbf{ConceptBench} to measure the models' capacity to generate general concepts, grouped into three primary categories: ID, object, and style. Using M-Score and ConceptBench, we demonstrate that Forget-Me-Not can effectively eliminate targeted concepts while maintaining the model's performance on other concepts. Furthermore, Forget-Me-Not offers two practical extensions: a) removal of potentially harmful or NSFW content, and b) enhancement of model accuracy, inclusion and diversity through \textbf{concept correction and disentanglement}. It can also be adapted as a lightweight model patch for Stable Diffusion, allowing for concept manipulation and convenient distribution. To encourage future research in this critical area and promote the development of safe and inclusive generative models, we will open-source our code and ConceptBench at \href{https://github.com/SHI-Labs/Forget-Me-Not}{https://github.com/SHI-Labs/Forget-Me-Not}.

Title: Consistent View Synthesis with Pose-Guided Diffusion Models. (arXiv:2303.17598v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17598
• Code URL: null
• Copy Paste: [[2303.17598] Consistent View Synthesis with Pose-Guided Diffusion Models](http://arxiv.org/abs/2303.17598) #diffusion
• Summary:

  Novel view synthesis from a single image has been a cornerstone problem for many Virtual Reality applications that provide immersive experiences. However, most existing techniques can only synthesize novel views within a limited range of camera motion or fail to generate consistent and high-quality novel views under significant camera movement. In this work, we propose a pose-guided diffusion model to generate a consistent long-term video of novel views from a single image. We design an attention layer that uses epipolar lines as constraints to facilitate the association between different viewpoints. Experimental results on synthetic and real-world datasets demonstrate the effectiveness of the proposed diffusion model against state-of-the-art transformer-based and GAN-based approaches.

Title: Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models. (arXiv:2303.17599v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17599
• Code URL: https://github.com/baaivision/vid2vid-zero
• Copy Paste: [[2303.17599] Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models](http://arxiv.org/abs/2303.17599) #diffusion
• Summary:

  Large-scale text-to-image diffusion models achieve unprecedented success in image generation and editing. However, how to extend such success to video editing is unclear. Recent initial attempts at video editing require significant text-to-video data and computation resources for training, which is often not accessible. In this work, we propose vid2vid-zero, a simple yet effective method for zero-shot video editing. Our vid2vid-zero leverages off-the-shelf image diffusion models, and doesn't require training on any video. At the core of our method is a null-text inversion module for text-to-video alignment, a cross-frame modeling module for temporal consistency, and a spatial regularization module for fidelity to the original video. Without any training, we leverage the dynamic nature of the attention mechanism to enable bi-directional temporal modeling at test time. Experiments and analyses show promising results in editing attributes, subjects, places, etc., in real-world videos. Code will be made available at \url{https://github.com/baaivision/vid2vid-zero}.

Title: Token Merging for Fast Stable Diffusion. (arXiv:2303.17604v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2303.17604
• Code URL: https://github.com/dbolya/tomesd
• Copy Paste: [[2303.17604] Token Merging for Fast Stable Diffusion](http://arxiv.org/abs/2303.17604) #diffusion
• Summary:

  The landscape of image generation has been forever changed by open vocabulary diffusion models. However, at their core these models use transformers, which makes generation slow. Better implementations to increase the throughput of these transformers have emerged, but they still evaluate the entire model. In this paper, we instead speed up diffusion models by exploiting natural redundancy in generated images by merging redundant tokens. After making some diffusion-specific improvements to Token Merging (ToMe), our ToMe for Stable Diffusion can reduce the number of tokens in an existing Stable Diffusion model by up to 60% while still producing high quality images without any extra training. In the process, we speed up image generation by up to 2x and reduce memory consumption by up to 5.6x. Furthermore, this speed-up stacks with efficient implementations such as xFormers, minimally impacting quality while being up to 5.4x faster for large images. Code is available at https://github.com/dbolya/tomesd.