secure

Title: LISA: LIghtweight single-server Secure Aggregation with a public source of randomness. (arXiv:2308.02208v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.02208
• Code URL: null
• Copy Paste: [[2308.02208]] LISA: LIghtweight single-server Secure Aggregation with a public source of randomness(http://arxiv.org/abs/2308.02208)
• Summary:

  Secure Aggregation (SA) is a key component of privacy-friendly federated learning applications, where the server learns the sum of many user-supplied gradients, while individual gradients are kept private. State-of-the-art SA protocols protect individual inputs with zero-sum random shares that are distributed across users, have a per-user overhead that is logarithmic in the number of users, and take more than 5 rounds of interaction. In this paper, we introduce LISA, an SA protocol that leverages a source of public randomness to minimize per-user overhead and the number of rounds. In particular, LISA requires only two rounds and has a communication overhead that is asymptotically equal to that of a non-private protocol -- one where inputs are provided to the server in the clear -- for most of the users. In a nutshell, LISA uses public randomness to select a subset of the users -- a committee -- that aid the server to recover the aggregated input. Users blind their individual contributions with randomness shared with each of the committee members; each committee member provides the server with an aggregate of the randomness shared with each user. Hence, as long as one committee member is honest, the server cannot learn individual inputs but only the sum of threshold-many inputs. We compare LISA with state-of-the-art SA protocols both theoretically and by means of simulations and present results of our experiments. We also integrate LISA in a Federated Learning pipeline and compare its performance with a non-private protocol.

security

Title: Explaining Relation Classification Models with Semantic Extents. (arXiv:2308.02193v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.02193
• Code URL: https://github.com/mslars/semantic_extents
• Copy Paste: [[2308.02193]] Explaining Relation Classification Models with Semantic Extents(http://arxiv.org/abs/2308.02193)
• Summary:

  In recent years, the development of large pretrained language models, such as BERT and GPT, significantly improved information extraction systems on various tasks, including relation classification. State-of-the-art systems are highly accurate on scientific benchmarks. A lack of explainability is currently a complicating factor in many real-world applications. Comprehensible systems are necessary to prevent biased, counterintuitive, or harmful decisions.

  We introduce semantic extents, a concept to analyze decision patterns for the relation classification task. Semantic extents are the most influential parts of texts concerning classification decisions. Our definition allows similar procedures to determine semantic extents for humans and models. We provide an annotation tool and a software framework to determine semantic extents for humans and models conveniently and reproducibly. Comparing both reveals that models tend to learn shortcut patterns from data. These patterns are hard to detect with current interpretability methods, such as input reductions. Our approach can help detect and eliminate spurious decision patterns during model development. Semantic extents can increase the reliability and security of natural language processing systems. Semantic extents are an essential step in enabling applications in critical areas like healthcare or finance. Moreover, our work opens new research directions for developing methods to explain deep learning models.

Title: SoK: The Ghost Trilemma. (arXiv:2308.02202v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.02202
• Code URL: null
• Copy Paste: [[2308.02202]] SoK: The Ghost Trilemma(http://arxiv.org/abs/2308.02202)
• Summary:

  Trolls, bots, and sybils distort online discourse and compromise the security of networked platforms. User identity is central to the vectors of attack and manipulation employed in these contexts. However it has long seemed that, try as it might, the security community has been unable to stem the rising tide of such problems. We posit the Ghost Trilemma, that there are three key properties of identity -- sentience, location, and uniqueness -- that cannot be simultaneously verified in a fully-decentralized setting. Many fully-decentralized systems -- whether for communication or social coordination -- grapple with this trilemma in some way, perhaps unknowingly. We examine the design space, use cases, problems with prior approaches, and possible paths forward. We sketch a proof of this trilemma and outline options for practical, incrementally deployable schemes to achieve an acceptable tradeoff of trust in centralized trust anchors, decentralized operation, and an ability to withstand a range of attacks, while protecting user privacy.

Title: Security Evaluation of Compressible and Learnable Image Encryption Against Jigsaw Puzzle Solver Attacks. (arXiv:2308.02227v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.02227
• Code URL: null
• Copy Paste: [[2308.02227]] Security Evaluation of Compressible and Learnable Image Encryption Against Jigsaw Puzzle Solver Attacks(http://arxiv.org/abs/2308.02227)
• Summary:

  Several learnable image encryption schemes have been developed for privacy-preserving image classification. This paper focuses on the security block-based image encryption methods that are learnable and JPEG-friendly. Permuting divided blocks in an image is known to enhance robustness against ciphertext-only attacks (COAs), but recently jigsaw puzzle solver attacks have been demonstrated to be able to restore visual information on the encrypted images. In contrast, it has never been confirmed whether encrypted images including noise caused by JPEG-compression are robust. Accordingly, the aim of this paper is to evaluate the security of compressible and learnable encrypted images against jigsaw puzzle solver attacks. In experiments, the security evaluation was carried out on the CIFAR-10 and STL-10 datasets under JPEG-compression.

Title: Improving the Security of United States Elections with Robust Optimization. (arXiv:2308.02306v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.02306
• Code URL: null
• Copy Paste: [[2308.02306]] Improving the Security of United States Elections with Robust Optimization(http://arxiv.org/abs/2308.02306)
• Summary:

  For more than a century, election officials across the United States have inspected voting machines before elections using a procedure called Logic and Accuracy Testing (LAT). This procedure consists of election officials casting a test deck of ballots into each voting machine and confirming the machine produces the expected vote total for each candidate. We bring a scientific perspective to LAT by introducing the first formal approach to designing test decks with rigorous security guarantees. Specifically, our approach employs robust optimization to find test decks that are guaranteed to detect any voting machine misconfiguration that would cause votes to be swapped across candidates. Out of all the test decks with this security guarantee, our robust optimization problem yields the test deck with the minimum number of ballots, thereby minimizing implementation costs for election officials. To facilitate deployment at scale, we develop a practically efficient exact algorithm for solving our robust optimization problems based on the cutting plane method. In partnership with the Michigan Bureau of Elections, we retrospectively applied our approach to all 6928 ballot styles from Michigan's November 2022 general election; this retrospective study reveals that the test decks with rigorous security guarantees obtained by our approach require, on average, only 1.2% more ballots than current practice. Our approach has since been piloted in real-world elections by the Michigan Bureau of Elections as a low-cost way to improve election security and increase public trust in democratic institutions.

Title: Model Provenance via Model DNA. (arXiv:2308.02121v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02121
• Code URL: null
• Copy Paste: [[2308.02121]] Model Provenance via Model DNA(http://arxiv.org/abs/2308.02121)
• Summary:

  Understanding the life cycle of the machine learning (ML) model is an intriguing area of research (e.g., understanding where the model comes from, how it is trained, and how it is used). This paper focuses on a novel problem within this field, namely Model Provenance (MP), which concerns the relationship between a target model and its pre-training model and aims to determine whether a source model serves as the provenance for a target model. This is an important problem that has significant implications for ensuring the security and intellectual property of machine learning models but has not received much attention in the literature. To fill in this gap, we introduce a novel concept of Model DNA which represents the unique characteristics of a machine learning model. We utilize a data-driven and model-driven representation learning method to encode the model's training data and input-output information as a compact and comprehensive representation (i.e., DNA) of the model. Using this model DNA, we develop an efficient framework for model provenance identification, which enables us to identify whether a source model is a pre-training model of a target model. We conduct evaluations on both computer vision and natural language processing tasks using various models, datasets, and scenarios to demonstrate the effectiveness of our approach in accurately identifying model provenance.

privacy

Title: BlockChain I/O: Enabling Cross-Chain Commerce. (arXiv:2308.02163v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.02163
• Code URL: null
• Copy Paste: [[2308.02163]] BlockChain I/O: Enabling Cross-Chain Commerce(http://arxiv.org/abs/2308.02163)
• Summary:

  By enabling users to safely transfer digital tokens without trusted intermediaries, blockchains have fueled the rise of Decentralized Finance (DeFi). However, the current DeFi ecosystem consists of multiple independent blockchains, and cross-chain token trading is a challenge because the desirable properties of individual blockchains do not always generalize to a multi-chain setting. Recently, advances have been made in the generalization of these properties, but there is still a lack of an overarching framework that provides the full set of properties required for practical cross-chain commerce: transaction atomicity, stablecoin support, privacy-preserving digital identities, and general applicability.

  In this paper, we present BlockChain I/O to provide such a framework. BlockChain I/O uses entities called cross-chain services to relay information between different chains. Cross-chain services cannot violate transaction atomicity, and are disincentivized from other types of misbehavior -- i.e., causing delays or misrepresenting information -- through an audit system. BlockChain I/O uses stablecoins to mitigate price fluctuations, and a Digital ID system to allow users to prove aspects of their identity without violating privacy. After presenting the core architecture of BlockChain I/O, we demonstrate how to use it to implement a cross-chain marketplace. Finally, we use an experimental evaluation to demonstrate BlockChain I/O's practical performance.

Title: Poster: Patient Community -- A Test Bed For Privacy Threat Analysis. (arXiv:2308.02272v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.02272
• Code URL: null
• Copy Paste: [[2308.02272]] Poster: Patient Community -- A Test Bed For Privacy Threat Analysis(http://arxiv.org/abs/2308.02272)
• Summary:

  Research and development of privacy analysis tools currently suffers from a lack of test beds for evaluation and comparison of such tools. In this work, we propose a benchmark application that implements an extensive list of privacy weaknesses based on the LINDDUN methodology. It represents a social network for patients whose architecture has first been described in an example analysis conducted by one of the LINDDUN authors. We have implemented this architecture and extended it with more privacy threats to build a test bed that enables comprehensive and independent testing of analysis tools.

protect

Title: Training Data Protection with Compositional Diffusion Models. (arXiv:2308.01937v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.01937
• Code URL: null
• Copy Paste: [[2308.01937]] Training Data Protection with Compositional Diffusion Models(http://arxiv.org/abs/2308.01937)
• Summary:

  We introduce Compartmentalized Diffusion Models (CDM), a method to train different diffusion models (or prompts) on distinct data sources and arbitrarily compose them at inference time. The individual models can be trained in isolation, at different times, and on different distributions and domains and can be later composed to achieve performance comparable to a paragon model trained on all data simultaneously. Furthermore, each model only contains information about the subset of the data it was exposed to during training, enabling several forms of training data protection. In particular, CDMs are the first method to enable both selective forgetting and continual learning for large-scale diffusion models, as well as allowing serving customized models based on the user's access rights. CDMs also allow determining the importance of a subset of the data in generating particular samples.

Title: From Prompt Injections to SQL Injection Attacks: How Protected is Your LLM-Integrated Web Application?. (arXiv:2308.01990v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.01990
• Code URL: null
• Copy Paste: [[2308.01990]] From Prompt Injections to SQL Injection Attacks: How Protected is Your LLM-Integrated Web Application?(http://arxiv.org/abs/2308.01990)
• Summary:

  Large Language Models (LLMs) have found widespread applications in various domains, including web applications, where they facilitate human interaction via chatbots with natural language interfaces. Internally, aided by an LLM-integration middleware such as Langchain, user prompts are translated into SQL queries used by the LLM to provide meaningful responses to users. However, unsanitized user prompts can lead to SQL injection attacks, potentially compromising the security of the database. Despite the growing interest in prompt injection vulnerabilities targeting LLMs, the specific risks of generating SQL injection attacks through prompt injections have not been extensively studied. In this paper, we present a comprehensive examination of prompt-to-SQL (P$_2$SQL) injections targeting web applications based on the Langchain framework. Using Langchain as our case study, we characterize P$_2$SQL injections, exploring their variants and impact on application security through multiple concrete examples. Furthermore, we evaluate 7 state-of-the-art LLMs, demonstrating the pervasiveness of P$_2$SQL attacks across language models. Our findings indicate that LLM-integrated applications based on Langchain are highly susceptible to P$_2$SQL injection attacks, warranting the adoption of robust defenses. To counter these attacks, we propose four effective defense techniques that can be integrated as extensions to the Langchain framework. We validate the defenses through an experimental evaluation with a real-world use case application.

Title: An Empirical Study on Fairness Improvement with Multiple Protected Attributes. (arXiv:2308.01923v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.01923
• Code URL: null
• Copy Paste: [[2308.01923]] An Empirical Study on Fairness Improvement with Multiple Protected Attributes(http://arxiv.org/abs/2308.01923)
• Summary:

  Existing research mostly improves the fairness of Machine Learning (ML) software regarding a single protected attribute at a time, but this is unrealistic given that many users have multiple protected attributes. This paper conducts an extensive study of fairness improvement regarding multiple protected attributes, covering 11 state-of-the-art fairness improvement methods. We analyze the effectiveness of these methods with different datasets, metrics, and ML models when considering multiple protected attributes. The results reveal that improving fairness for a single protected attribute can largely decrease fairness regarding unconsidered protected attributes. This decrease is observed in up to 88.3% of scenarios (57.5% on average). More surprisingly, we find little difference in accuracy loss when considering single and multiple protected attributes, indicating that accuracy can be maintained in the multiple-attribute paradigm. However, the effect on precision and recall when handling multiple protected attributes is about 5 times and 8 times that of a single attribute. This has important implications for future fairness research: reporting only accuracy as the ML performance metric, which is currently common in the literature, is inadequate.

defense

Title: Universal Defensive Underpainting Patch: Making Your Text Invisible to Optical Character Recognition. (arXiv:2308.02369v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02369
• Code URL: https://github.com/qrickdd/udup
• Copy Paste: [[2308.02369]] Universal Defensive Underpainting Patch: Making Your Text Invisible to Optical Character Recognition(http://arxiv.org/abs/2308.02369)
• Summary:

  Optical Character Recognition (OCR) enables automatic text extraction from scanned or digitized text images, but it also makes it easy to pirate valuable or sensitive text from these images. Previous methods to prevent OCR piracy by distorting characters in text images are impractical in real-world scenarios, as pirates can capture arbitrary portions of the text images, rendering the defenses ineffective. In this work, we propose a novel and effective defense mechanism termed the Universal Defensive Underpainting Patch (UDUP) that modifies the underpainting of text images instead of the characters. UDUP is created through an iterative optimization process to craft a small, fixed-size defensive patch that can generate non-overlapping underpainting for text images of any size. Experimental results show that UDUP effectively defends against unauthorized OCR under the setting of any screenshot range or complex image background. It is agnostic to the content, size, colors, and languages of characters, and is robust to typical image operations such as scaling and compressing. In addition, the transferability of UDUP is demonstrated by evading several off-the-shelf OCRs. The code is available at https://github.com/QRICKDD/UDUP.

attack

Title: BlindSage: Label Inference Attacks against Node-level Vertical Federated Graph Neural Networks. (arXiv:2308.02465v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02465
• Code URL: null
• Copy Paste: [[2308.02465]] BlindSage: Label Inference Attacks against Node-level Vertical Federated Graph Neural Networks(http://arxiv.org/abs/2308.02465)
• Summary:

  Federated learning enables collaborative training of machine learning models by keeping the raw data of the involved workers private. One of its main objectives is to improve the models' privacy, security, and scalability. Vertical Federated Learning (VFL) offers an efficient cross-silo setting where a few parties collaboratively train a model without sharing the same features. In such a scenario, classification labels are commonly considered sensitive information held exclusively by one (active) party, while other (passive) parties use only their local information. Recent works have uncovered important flaws of VFL, leading to possible label inference attacks under the assumption that the attacker has some, even limited, background knowledge on the relation between labels and data. In this work, we are the first (to the best of our knowledge) to investigate label inference attacks on VFL using a zero-background knowledge strategy. To concretely formulate our proposal, we focus on Graph Neural Networks (GNNs) as a target model for the underlying VFL. In particular, we refer to node classification tasks, which are widely studied, and GNNs have shown promising results. Our proposed attack, BlindSage, provides impressive results in the experiments, achieving nearly 100% accuracy in most cases. Even when the attacker has no information about the used architecture or the number of classes, the accuracy remained above 85% in most instances. Finally, we observe that well-known defenses cannot mitigate our attack without affecting the model's performance on the main classification task.

Title: IoT and Man-in-the-Middle Attacks. (arXiv:2308.02479v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.02479
• Code URL: null
• Copy Paste: [[2308.02479]] IoT and Man-in-the-Middle Attacks(http://arxiv.org/abs/2308.02479)
• Summary:

  This paper provides an overview of the Internet of Things (IoT) and its significance. It discusses the concept of Man-in-the-Middle (MitM) attacks in detail, including their causes, potential solutions, and challenges in detecting and preventing such attacks. The paper also addresses the current issues related to IoT security and explores future methods and facilities for improving detection and prevention mechanisms against MitM.

robust

Title: TSMD: A Database for Static Color Mesh Quality Assessment Study. (arXiv:2308.01940v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01940
• Code URL: null
• Copy Paste: [[2308.01940]] TSMD: A Database for Static Color Mesh Quality Assessment Study(http://arxiv.org/abs/2308.01940)
• Summary:

  Static meshes with texture map are widely used in modern industrial and manufacturing sectors, attracting considerable attention in the mesh compression community due to its huge amount of data. To facilitate the study of static mesh compression algorithm and objective quality metric, we create the Tencent - Static Mesh Dataset (TSMD) containing 42 reference meshes with rich visual characteristics. 210 distorted samples are generated by the lossy compression scheme developed for the Call for Proposals on polygonal static mesh coding, released on June 23 by the Alliance for Open Media Volumetric Visual Media group. Using processed video sequences, a large-scale, crowdsourcing-based, subjective experiment was conducted to collect subjective scores from 74 viewers. The dataset undergoes analysis to validate its sample diversity and Mean Opinion Scores (MOS) accuracy, establishing its heterogeneous nature and reliability. State-of-the-art objective metrics are evaluated on the new dataset. Pearson and Spearman correlations around 0.75 are reported, deviating from results typically observed on less heterogeneous datasets, demonstrating the need for further development of more robust metrics. The TSMD, including meshes, PVSs, bitstreams, and MOS, is made publicly available at the following location: https://multimedia.tencent.com/resources/tsmd.

Title: RealCQA: Scientific Chart Question Answering as a Test-bed for First-Order Logic. (arXiv:2308.01979v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01979
• Code URL: https://github.com/cse-ai-lab/RealCQA
• Copy Paste: [[2308.01979]] RealCQA: Scientific Chart Question Answering as a Test-bed for First-Order Logic(http://arxiv.org/abs/2308.01979)
• Summary:

  We present a comprehensive study of chart visual question-answering(QA) task, to address the challenges faced in comprehending and extracting data from chart visualizations within documents. Despite efforts to tackle this problem using synthetic charts, solutions are limited by the shortage of annotated real-world data. To fill this gap, we introduce a benchmark and dataset for chart visual QA on real-world charts, offering a systematic analysis of the task and a novel taxonomy for template-based chart question creation. Our contribution includes the introduction of a new answer type, 'list', with both ranked and unranked variations. Our study is conducted on a real-world chart dataset from scientific literature, showcasing higher visual complexity compared to other works. Our focus is on template-based QA and how it can serve as a standard for evaluating the first-order logic capabilities of models. The results of our experiments, conducted on a real-world out-of-distribution dataset, provide a robust evaluation of large-scale pre-trained models and advance the field of chart visual QA and formal logic verification for neural networks in general.

Title: AdvFAS: A robust face anti-spoofing framework against adversarial examples. (arXiv:2308.02116v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02116
• Code URL: null
• Copy Paste: [[2308.02116]] AdvFAS: A robust face anti-spoofing framework against adversarial examples(http://arxiv.org/abs/2308.02116)
• Summary:

  Ensuring the reliability of face recognition systems against presentation attacks necessitates the deployment of face anti-spoofing techniques. Despite considerable advancements in this domain, the ability of even the most state-of-the-art methods to defend against adversarial examples remains elusive. While several adversarial defense strategies have been proposed, they typically suffer from constrained practicability due to inevitable trade-offs between universality, effectiveness, and efficiency. To overcome these challenges, we thoroughly delve into the coupled relationship between adversarial detection and face anti-spoofing. Based on this, we propose a robust face anti-spoofing framework, namely AdvFAS, that leverages two coupled scores to accurately distinguish between correctly detected and wrongly detected face images. Extensive experiments demonstrate the effectiveness of our framework in a variety of settings, including different attacks, datasets, and backbones, meanwhile enjoying high accuracy on clean examples. Moreover, we successfully apply the proposed method to detect real-world adversarial examples.

Title: Robust Self-Supervised Extrinsic Self-Calibration. (arXiv:2308.02153v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02153
• Code URL: null
• Copy Paste: [[2308.02153]] Robust Self-Supervised Extrinsic Self-Calibration(http://arxiv.org/abs/2308.02153)
• Summary:

  Autonomous vehicles and robots need to operate over a wide variety of scenarios in order to complete tasks efficiently and safely. Multi-camera self-supervised monocular depth estimation from videos is a promising way to reason about the environment, as it generates metrically scaled geometric predictions from visual data without requiring additional sensors. However, most works assume well-calibrated extrinsics to fully leverage this multi-camera setup, even though accurate and efficient calibration is still a challenging problem. In this work, we introduce a novel method for extrinsic calibration that builds upon the principles of self-supervised monocular depth and ego-motion learning. Our proposed curriculum learning strategy uses monocular depth and pose estimators with velocity supervision to estimate extrinsics, and then jointly learns extrinsic calibration along with depth and pose for a set of overlapping cameras rigidly attached to a moving vehicle. Experiments on a benchmark multi-camera dataset (DDAD) demonstrate that our method enables self-calibration in various scenes robustly and efficiently compared to a traditional vision-based pose estimation pipeline. Furthermore, we demonstrate the benefits of extrinsics self-calibration as a way to improve depth prediction via joint optimization.

Title: Paired Competing Neurons Improving STDP Supervised Local Learning In Spiking Neural Networks. (arXiv:2308.02194v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02194
• Code URL: null
• Copy Paste: [[2308.02194]] Paired Competing Neurons Improving STDP Supervised Local Learning In Spiking Neural Networks(http://arxiv.org/abs/2308.02194)
• Summary:

  Direct training of Spiking Neural Networks (SNNs) on neuromorphic hardware has the potential to significantly reduce the high energy consumption of Artificial Neural Networks (ANNs) training on modern computers. The biological plausibility of SNNs allows them to benefit from bio-inspired plasticity rules, such as Spike Timing-Dependent Plasticity (STDP). STDP offers gradient-free and unsupervised local learning, which can be easily implemented on neuromorphic hardware. However, relying solely on unsupervised STDP to perform classification tasks is not enough. In this paper, we propose Stabilized Supervised STDP (S2-STDP), a supervised STDP learning rule to train the classification layer of an SNN equipped with unsupervised STDP. S2-STDP integrates error-modulated weight updates that align neuron spikes with desired timestamps derived from the average firing time within the layer. Then, we introduce a training architecture called Paired Competing Neurons (PCN) to further enhance the learning capabilities of our classification layer trained with S2-STDP. PCN associates each class with paired neurons and encourages neuron specialization through intra-class competition. We evaluated our proposed methods on image recognition datasets, including MNIST, Fashion-MNIST, and CIFAR-10. Results showed that our methods outperform current supervised STDP-based state of the art, for comparable architectures and numbers of neurons. Also, the use of PCN enhances the performance of S2-STDP, regardless of the configuration, and without introducing any hyperparameters.Further analysis demonstrated that our methods exhibited improved hyperparameter robustness, which reduces the need for tuning.

Title: MSECNet: Accurate and Robust Normal Estimation for 3D Point Clouds by Multi-Scale Edge Conditioning. (arXiv:2308.02237v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02237
• Code URL: null
• Copy Paste: [[2308.02237]] MSECNet: Accurate and Robust Normal Estimation for 3D Point Clouds by Multi-Scale Edge Conditioning(http://arxiv.org/abs/2308.02237)
• Summary:

  Estimating surface normals from 3D point clouds is critical for various applications, including surface reconstruction and rendering. While existing methods for normal estimation perform well in regions where normals change slowly, they tend to fail where normals vary rapidly. To address this issue, we propose a novel approach called MSECNet, which improves estimation in normal varying regions by treating normal variation modeling as an edge detection problem. MSECNet consists of a backbone network and a multi-scale edge conditioning (MSEC) stream. The MSEC stream achieves robust edge detection through multi-scale feature fusion and adaptive edge detection. The detected edges are then combined with the output of the backbone network using the edge conditioning module to produce edge-aware representations. Extensive experiments show that MSECNet outperforms existing methods on both synthetic (PCPNet) and real-world (SceneNN) datasets while running significantly faster. We also conduct various analyses to investigate the contribution of each component in the MSEC stream. Finally, we demonstrate the effectiveness of our approach in surface reconstruction.

Title: RAHNet: Retrieval Augmented Hybrid Network for Long-tailed Graph Classification. (arXiv:2308.02335v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02335
• Code URL: null
• Copy Paste: [[2308.02335]] RAHNet: Retrieval Augmented Hybrid Network for Long-tailed Graph Classification(http://arxiv.org/abs/2308.02335)
• Summary:

  Graph classification is a crucial task in many real-world multimedia applications, where graphs can represent various multimedia data types such as images, videos, and social networks. Previous efforts have applied graph neural networks (GNNs) in balanced situations where the class distribution is balanced. However, real-world data typically exhibit long-tailed class distributions, resulting in a bias towards the head classes when using GNNs and limited generalization ability over the tail classes. Recent approaches mainly focus on re-balancing different classes during model training, which fails to explicitly introduce new knowledge and sacrifices the performance of the head classes. To address these drawbacks, we propose a novel framework called Retrieval Augmented Hybrid Network (RAHNet) to jointly learn a robust feature extractor and an unbiased classifier in a decoupled manner. In the feature extractor training stage, we develop a graph retrieval module to search for relevant graphs that directly enrich the intra-class diversity for the tail classes. Moreover, we innovatively optimize a category-centered supervised contrastive loss to obtain discriminative representations, which is more suitable for long-tailed scenarios. In the classifier fine-tuning stage, we balance the classifier weights with two weight regularization techniques, i.e., Max-norm and weight decay. Experiments on various popular benchmarks verify the superiority of the proposed method against state-of-the-art approaches.

Title: RobustMQ: Benchmarking Robustness of Quantized Models. (arXiv:2308.02350v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02350
• Code URL: null
• Copy Paste: [[2308.02350]] RobustMQ: Benchmarking Robustness of Quantized Models(http://arxiv.org/abs/2308.02350)
• Summary:

  Quantization has emerged as an essential technique for deploying deep neural networks (DNNs) on devices with limited resources. However, quantized models exhibit vulnerabilities when exposed to various noises in real-world applications. Despite the importance of evaluating the impact of quantization on robustness, existing research on this topic is limited and often disregards established principles of robustness evaluation, resulting in incomplete and inconclusive findings. To address this gap, we thoroughly evaluated the robustness of quantized models against various noises (adversarial attacks, natural corruptions, and systematic noises) on ImageNet. The comprehensive evaluation results empirically provide valuable insights into the robustness of quantized models in various scenarios, for example: (1) quantized models exhibit higher adversarial robustness than their floating-point counterparts, but are more vulnerable to natural corruptions and systematic noises; (2) in general, increasing the quantization bit-width results in a decrease in adversarial robustness, an increase in natural robustness, and an increase in systematic robustness; (3) among corruption methods, \textit{impulse noise} and \textit{glass blur} are the most harmful to quantized models, while \textit{brightness} has the least impact; (4) among systematic noises, the \textit{nearest neighbor interpolation} has the highest impact, while bilinear interpolation, cubic interpolation, and area interpolation are the three least harmful. Our research contributes to advancing the robust quantization of models and their deployment in real-world scenarios.

Title: Learning Regionalization within a Differentiable High-Resolution Hydrological Model using Accurate Spatial Cost Gradients. (arXiv:2308.02040v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02040
• Code URL: null
• Copy Paste: [[2308.02040]] Learning Regionalization within a Differentiable High-Resolution Hydrological Model using Accurate Spatial Cost Gradients(http://arxiv.org/abs/2308.02040)
• Summary:

  Estimating spatially distributed hydrological parameters in ungauged catchments poses a challenging regionalization problem and requires imposing spatial constraints given the sparsity of discharge data. A possible approach is to search for a transfer function that quantitatively relates physical descriptors to conceptual model parameters. This paper introduces a Hybrid Data Assimilation and Parameter Regionalization (HDA-PR) approach incorporating learnable regionalization mappings, based on either multivariate regressions or neural networks, into a differentiable hydrological model. It enables the exploitation of heterogeneous datasets across extensive spatio-temporal computational domains within a high-dimensional regionalization context, using accurate adjoint-based gradients. The inverse problem is tackled with a multi-gauge calibration cost function accounting for information from multiple observation sites. HDA-PR was tested on high-resolution, hourly and kilometric regional modeling of two flash-flood-prone areas located in the South of France. In both study areas, the median Nash-Sutcliffe efficiency (NSE) scores ranged from 0.52 to 0.78 at pseudo-ungauged sites over calibration and validation periods. These results highlight a strong regionalization performance of HDA-PR, improving NSE by up to 0.57 compared to the baseline model calibrated with lumped parameters, and achieving a performance comparable to the reference solution obtained with local uniform calibration (median NSE from 0.59 to 0.79). Multiple evaluation metrics based on flood-oriented hydrological signatures are also employed to assess the accuracy and robustness of the approach. The regionalization method is amenable to state-parameter correction from multi-source data over a range of time scales needed for operational data assimilation, and it is adaptable to other differentiable geophysical models.

Title: Adapting to Change: Robust Counterfactual Explanations in Dynamic Data Landscapes. (arXiv:2308.02353v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02353
• Code URL: https://github.com/bardhprenkaj/hansel
• Copy Paste: [[2308.02353]] Adapting to Change: Robust Counterfactual Explanations in Dynamic Data Landscapes(http://arxiv.org/abs/2308.02353)
• Summary:

  We introduce a novel semi-supervised Graph Counterfactual Explainer (GCE) methodology, Dynamic GRAph Counterfactual Explainer (DyGRACE). It leverages initial knowledge about the data distribution to search for valid counterfactuals while avoiding using information from potentially outdated decision functions in subsequent time steps. Employing two graph autoencoders (GAEs), DyGRACE learns the representation of each class in a binary classification scenario. The GAEs minimise the reconstruction error between the original graph and its learned representation during training. The method involves (i) optimising a parametric density function (implemented as a logistic regression function) to identify counterfactuals by maximising the factual autoencoder's reconstruction error, (ii) minimising the counterfactual autoencoder's error, and (iii) maximising the similarity between the factual and counterfactual graphs. This semi-supervised approach is independent of an underlying black-box oracle. A logistic regression model is trained on a set of graph pairs to learn weights that aid in finding counterfactuals. At inference, for each unseen graph, the logistic regressor identifies the best counterfactual candidate using these learned weights, while the GAEs can be iteratively updated to represent the continual adaptation of the learned graph representation over iterations. DyGRACE is quite effective and can act as a drift detector, identifying distributional drift based on differences in reconstruction errors between iterations. It avoids reliance on the oracle's predictions in successive iterations, thereby increasing the efficiency of counterfactual discovery. DyGRACE, with its capacity for contrastive learning and drift detection, will offer new avenues for semi-supervised learning and explanation generation.

biometric

Title: On the Biometric Capacity of Generative Face Models. (arXiv:2308.02065v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02065
• Code URL: https://github.com/human-analysis/capacity-generative-face-models
• Copy Paste: [[2308.02065]] On the Biometric Capacity of Generative Face Models(http://arxiv.org/abs/2308.02065)
• Summary:

  There has been tremendous progress in generating realistic faces with high fidelity over the past few years. Despite this progress, a crucial question remains unanswered: "Given a generative face model, how many unique identities can it generate?" In other words, what is the biometric capacity of the generative face model? A scientific basis for answering this question will benefit evaluating and comparing different generative face models and establish an upper bound on their scalability. This paper proposes a statistical approach to estimate the biometric capacity of generated face images in a hyperspherical feature space. We employ our approach on multiple generative models, including unconditional generators like StyleGAN, Latent Diffusion Model, and "Generated Photos," as well as DCFace, a class-conditional generator. We also estimate capacity w.r.t. demographic attributes such as gender and age. Our capacity estimates indicate that (a) under ArcFace representation at a false acceptance rate (FAR) of 0.1%, StyleGAN3 and DCFace have a capacity upper bound of $1.43\times10^6$ and $1.190\times10^4$, respectively; (b) the capacity reduces drastically as we lower the desired FAR with an estimate of $1.796\times10^4$ and $562$ at FAR of 1% and 10%, respectively, for StyleGAN3; (c) there is no discernible disparity in the capacity w.r.t gender; and (d) for some generative models, there is an appreciable disparity in the capacity w.r.t age. Code is available at https://github.com/human-analysis/capacity-generative-face-models.

steal

extraction

Title: SpaDen : Sparse and Dense Keypoint Estimation for Real-World Chart Understanding. (arXiv:2308.01971v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01971
• Code URL: null
• Copy Paste: [[2308.01971]] SpaDen : Sparse and Dense Keypoint Estimation for Real-World Chart Understanding(http://arxiv.org/abs/2308.01971)
• Summary:

  We introduce a novel bottom-up approach for the extraction of chart data. Our model utilizes images of charts as inputs and learns to detect keypoints (KP), which are used to reconstruct the components within the plot area. Our novelty lies in detecting a fusion of continuous and discrete KP as predicted heatmaps. A combination of sparse and dense per-pixel objectives coupled with a uni-modal self-attention-based feature-fusion layer is applied to learn KP embeddings. Further leveraging deep metric learning for unsupervised clustering, allows us to segment the chart plot area into various objects. By further matching the chart components to the legend, we are able to obtain the data series names. A post-processing threshold is applied to the KP embeddings to refine the object reconstructions and improve accuracy. Our extensive experiments include an evaluation of different modules for KP estimation and the combination of deep layer aggregation and corner pooling approaches. The results of our experiments provide extensive evaluation for the task of real-world chart data extraction.

Title: DTF-Net: Category-Level Pose Estimation and Shape Reconstruction via Deformable Template Field. (arXiv:2308.02239v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02239
• Code URL: null
• Copy Paste: [[2308.02239]] DTF-Net: Category-Level Pose Estimation and Shape Reconstruction via Deformable Template Field(http://arxiv.org/abs/2308.02239)
• Summary:

  Estimating 6D poses and reconstructing 3D shapes of objects in open-world scenes from RGB-depth image pairs is challenging. Many existing methods rely on learning geometric features that correspond to specific templates while disregarding shape variations and pose differences among objects in the same category. As a result, these methods underperform when handling unseen object instances in complex environments. In contrast, other approaches aim to achieve category-level estimation and reconstruction by leveraging normalized geometric structure priors, but the static prior-based reconstruction struggles with substantial intra-class variations. To solve these problems, we propose the DTF-Net, a novel framework for pose estimation and shape reconstruction based on implicit neural fields of object categories. In DTF-Net, we design a deformable template field to represent the general category-wise shape latent features and intra-category geometric deformation features. The field establishes continuous shape correspondences, deforming the category template into arbitrary observed instances to accomplish shape reconstruction. We introduce a pose regression module that shares the deformation features and template codes from the fields to estimate the accurate 6D pose of each object in the scene. We integrate a multi-modal representation extraction module to extract object features and semantic masks, enabling end-to-end inference. Moreover, during training, we implement a shape-invariant training strategy and a viewpoint sampling method to further enhance the model's capability to extract object pose features. Extensive experiments on the REAL275 and CAMERA25 datasets demonstrate the superiority of DTF-Net in both synthetic and real scenes. Furthermore, we show that DTF-Net effectively supports grasping tasks with a real robot arm.

Title: RegionBLIP: A Unified Multi-modal Pre-training Framework for Holistic and Regional Comprehension. (arXiv:2308.02299v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02299
• Code URL: https://github.com/mightyzau/regionblip
• Copy Paste: [[2308.02299]] RegionBLIP: A Unified Multi-modal Pre-training Framework for Holistic and Regional Comprehension(http://arxiv.org/abs/2308.02299)
• Summary:

  In this work, we investigate extending the comprehension of Multi-modal Large Language Models (MLLMs) to regional objects. To this end, we propose to extract features corresponding to regional objects as soft prompts for LLM, which provides a straightforward and scalable approach and eliminates the need for LLM fine-tuning. To effectively extract regional features from regular image features and irregular point cloud features, we present a novel and unified position-assisted feature extraction module. Furthermore, training an MLLM from scratch is highly time-consuming. Thus, we propose incrementally extending existing pre-trained MLLMs to comprehend more modalities and the regional objects of those modalities. Specifically, we freeze the Q-Former from BLIP-2, an impressive MLLM, and optimize the modality-specific Lora parameters in Q-Former and LLM for each newly introduced modality. The freezing of the Q-Former eliminates the need for extensive pre-training on massive image-text data. The freezed Q-Former pre-trained from massive image-text data is also beneficial for the pre-training on image-region-text data. We name our framework RegionBLIP. We pre-train RegionBLIP on image-region-text, point-cloud-text, and point-cloud-region-text data. Experimental results verify that \Ours{} can preserve the image comprehension capability of BILP-2 and further gain a comprehension of the newly introduced point cloud modality and regional objects. The Data, Code, and Pre-trained models will be available at https://github.com/mightyzau/RegionBLIP.

Title: Efficient Sentiment Analysis: A Resource-Aware Evaluation of Feature Extraction Techniques, Ensembling, and Deep Learning Models. (arXiv:2308.02022v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.02022
• Code URL: null
• Copy Paste: [[2308.02022]] Efficient Sentiment Analysis: A Resource-Aware Evaluation of Feature Extraction Techniques, Ensembling, and Deep Learning Models(http://arxiv.org/abs/2308.02022)
• Summary:

  While reaching for NLP systems that maximize accuracy, other important metrics of system performance are often overlooked. Prior models are easily forgotten despite their possible suitability in settings where large computing resources are unavailable or relatively more costly. In this paper, we perform a broad comparative evaluation of document-level sentiment analysis models with a focus on resource costs that are important for the feasibility of model deployment and general climate consciousness. Our experiments consider different feature extraction techniques, the effect of ensembling, task-specific deep learning modeling, and domain-independent large language models (LLMs). We find that while a fine-tuned LLM achieves the best accuracy, some alternate configurations provide huge (up to 24, 283 *) resource savings for a marginal (<1%) loss in accuracy. Furthermore, we find that for smaller datasets, the differences in accuracy shrink while the difference in resource consumption grows further.

Title: Chinese Financial Text Emotion Mining: GCGTS -- A Character Relationship-based Approach for Simultaneous Aspect-Opinion Pair Extraction. (arXiv:2308.02113v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.02113
• Code URL: null
• Copy Paste: [[2308.02113]] Chinese Financial Text Emotion Mining: GCGTS -- A Character Relationship-based Approach for Simultaneous Aspect-Opinion Pair Extraction(http://arxiv.org/abs/2308.02113)
• Summary:

  Aspect-Opinion Pair Extraction (AOPE) from Chinese financial texts is a specialized task in fine-grained text sentiment analysis. The main objective is to extract aspect terms and opinion terms simultaneously from a diverse range of financial texts. Previous studies have mainly focused on developing grid annotation schemes within grid-based models to facilitate this extraction process. However, these methods often rely on character-level (token-level) feature encoding, which may overlook the logical relationships between Chinese characters within words. To address this limitation, we propose a novel method called Graph-based Character-level Grid Tagging Scheme (GCGTS). The GCGTS method explicitly incorporates syntactic structure using Graph Convolutional Networks (GCN) and unifies the encoding of characters within the same syntactic semantic unit (Chinese word level). Additionally, we introduce an image convolutional structure into the grid model to better capture the local relationships between characters within evaluation units. This innovative structure reduces the excessive reliance on pre-trained language models and emphasizes the modeling of structure and local relationships, thereby improving the performance of the model on Chinese financial texts. Through comparative experiments with advanced models such as Synchronous Double-channel Recurrent Network (SDRN) and Grid Tagging Scheme (GTS), the proposed GCGTS model demonstrates significant improvements in performance.

Title: Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Text. (arXiv:2308.02357v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.02357
• Code URL: null
• Copy Paste: [[2308.02357]] Text2KGBench: A Benchmark for Ontology-Driven Knowledge Graph Generation from Text(http://arxiv.org/abs/2308.02357)
• Summary:

  The recent advances in large language models (LLM) and foundation models with emergent capabilities have been shown to improve the performance of many NLP tasks. LLMs and Knowledge Graphs (KG) can complement each other such that LLMs can be used for KG construction or completion while existing KGs can be used for different tasks such as making LLM outputs explainable or fact-checking in Neuro-Symbolic manner. In this paper, we present Text2KGBench, a benchmark to evaluate the capabilities of language models to generate KGs from natural language text guided by an ontology. Given an input ontology and a set of sentences, the task is to extract facts from the text while complying with the given ontology (concepts, relations, domain/range constraints) and being faithful to the input sentences. We provide two datasets (i) Wikidata-TekGen with 10 ontologies and 13,474 sentences and (ii) DBpedia-WebNLG with 19 ontologies and 4,860 sentences. We define seven evaluation metrics to measure fact extraction performance, ontology conformance, and hallucinations by LLMs. Furthermore, we provide results for two baseline models, Vicuna-13B and Alpaca-LoRA-13B using automatic prompt generation from test cases. The baseline results show that there is room for improvement using both Semantic Web and Natural Language Processing techniques.

membership infer

federate

Title: Flexible Differentially Private Vertical Federated Learning with Adaptive Feature Embeddings. (arXiv:2308.02362v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.02362
• Code URL: null
• Copy Paste: [[2308.02362]] Flexible Differentially Private Vertical Federated Learning with Adaptive Feature Embeddings(http://arxiv.org/abs/2308.02362)
• Summary:

  The emergence of vertical federated learning (VFL) has stimulated concerns about the imperfection in privacy protection, as shared feature embeddings may reveal sensitive information under privacy attacks. This paper studies the delicate equilibrium between data privacy and task utility goals of VFL under differential privacy (DP). To address the generality issue of prior arts, this paper advocates a flexible and generic approach that decouples the two goals and addresses them successively. Specifically, we initially derive a rigorous privacy guarantee by applying norm clipping on shared feature embeddings, which is applicable across various datasets and models. Subsequently, we demonstrate that task utility can be optimized via adaptive adjustments on the scale and distribution of feature embeddings in an accuracy-appreciative way, without compromising established DP mechanisms. We concretize our observation into the proposed VFL-AFE framework, which exhibits effectiveness against privacy attacks and the capacity to retain favorable task utility, as substantiated by extensive experiments.

Title: SoK: Assessing the State of Applied Federated Machine Learning. (arXiv:2308.02454v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02454
• Code URL: null
• Copy Paste: [[2308.02454]] SoK: Assessing the State of Applied Federated Machine Learning(http://arxiv.org/abs/2308.02454)
• Summary:

  Machine Learning (ML) has shown significant potential in various applications; however, its adoption in privacy-critical domains has been limited due to concerns about data privacy. A promising solution to this issue is Federated Machine Learning (FedML), a model-to-data approach that prioritizes data privacy. By enabling ML algorithms to be applied directly to distributed data sources without sharing raw data, FedML offers enhanced privacy protections, making it suitable for privacy-critical environments. Despite its theoretical benefits, FedML has not seen widespread practical implementation. This study aims to explore the current state of applied FedML and identify the challenges hindering its practical adoption. Through a comprehensive systematic literature review, we assess 74 relevant papers to analyze the real-world applicability of FedML. Our analysis focuses on the characteristics and emerging trends of FedML implementations, as well as the motivational drivers and application domains. We also discuss the encountered challenges in integrating FedML into real-life settings. By shedding light on the existing landscape and potential obstacles, this research contributes to the further development and implementation of FedML in privacy-critical scenarios.

Title: Scaling Survival Analysis in Healthcare with Federated Survival Forests: A Comparative Study on Heart Failure and Breast Cancer Genomics. (arXiv:2308.02382v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02382
• Code URL: null
• Copy Paste: [[2308.02382]] Scaling Survival Analysis in Healthcare with Federated Survival Forests: A Comparative Study on Heart Failure and Breast Cancer Genomics(http://arxiv.org/abs/2308.02382)
• Summary:

  Survival analysis is a fundamental tool in medicine, modeling the time until an event of interest occurs in a population. However, in real-world applications, survival data are often incomplete, censored, distributed, and confidential, especially in healthcare settings where privacy is critical. The scarcity of data can severely limit the scalability of survival models to distributed applications that rely on large data pools. Federated learning is a promising technique that enables machine learning models to be trained on multiple datasets without compromising user privacy, making it particularly well-suited for addressing the challenges of survival data and large-scale survival applications. Despite significant developments in federated learning for classification and regression, many directions remain unexplored in the context of survival analysis. In this work, we propose an extension of the Federated Survival Forest algorithm, called FedSurF++. This federated ensemble method constructs random survival forests in heterogeneous federations. Specifically, we investigate several new tree sampling methods from client forests and compare the results with state-of-the-art survival models based on neural networks. The key advantage of FedSurF++ is its ability to achieve comparable performance to existing methods while requiring only a single communication round to complete. The extensive empirical investigation results in a significant improvement from the algorithmic and privacy preservation perspectives, making the original FedSurF algorithm more efficient, robust, and private. We also present results on two real-world datasets demonstrating the success of FedSurF++ in real-world healthcare studies. Our results underscore the potential of FedSurF++ to improve the scalability and effectiveness of survival analysis in distributed settings while preserving user privacy.

fair

Title: Target specification bias, counterfactual prediction, and algorithmic fairness in healthcare. (arXiv:2308.02081v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02081
• Code URL: null
• Copy Paste: [[2308.02081]] Target specification bias, counterfactual prediction, and algorithmic fairness in healthcare(http://arxiv.org/abs/2308.02081)
• Summary:

  Bias in applications of machine learning (ML) to healthcare is usually attributed to unrepresentative or incomplete data, or to underlying health disparities. This article identifies a more pervasive source of bias that affects the clinical utility of ML-enabled prediction tools: target specification bias. Target specification bias arises when the operationalization of the target variable does not match its definition by decision makers. The mismatch is often subtle, and stems from the fact that decision makers are typically interested in predicting the outcomes of counterfactual, rather than actual, healthcare scenarios. Target specification bias persists independently of data limitations and health disparities. When left uncorrected, it gives rise to an overestimation of predictive accuracy, to inefficient utilization of medical resources, and to suboptimal decisions that can harm patients. Recent work in metrology - the science of measurement - suggests ways of counteracting target specification bias and avoiding its harmful consequences.

interpretability

Title: ParaFuzz: An Interpretability-Driven Technique for Detecting Poisoned Samples in NLP. (arXiv:2308.02122v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.02122
• Code URL: null
• Copy Paste: [[2308.02122]] ParaFuzz: An Interpretability-Driven Technique for Detecting Poisoned Samples in NLP(http://arxiv.org/abs/2308.02122)
• Summary:

  Backdoor attacks have emerged as a prominent threat to natural language processing (NLP) models, where the presence of specific triggers in the input can lead poisoned models to misclassify these inputs to predetermined target classes. Current detection mechanisms are limited by their inability to address more covert backdoor strategies, such as style-based attacks. In this work, we propose an innovative test-time poisoned sample detection framework that hinges on the interpretability of model predictions, grounded in the semantic meaning of inputs. We contend that triggers (e.g., infrequent words) are not supposed to fundamentally alter the underlying semantic meanings of poisoned samples as they want to stay stealthy. Based on this observation, we hypothesize that while the model's predictions for paraphrased clean samples should remain stable, predictions for poisoned samples should revert to their true labels upon the mutations applied to triggers during the paraphrasing process. We employ ChatGPT, a state-of-the-art large language model, as our paraphraser and formulate the trigger-removal task as a prompt engineering problem. We adopt fuzzing, a technique commonly used for unearthing software vulnerabilities, to discover optimal paraphrase prompts that can effectively eliminate triggers while concurrently maintaining input semantics. Experiments on 4 types of backdoor attacks, including the subtle style backdoors, and 4 distinct datasets demonstrate that our approach surpasses baseline methods, including STRIP, RAP, and ONION, in precision and recall.

explainability

watermark

diffusion

Title: SDDM: Score-Decomposed Diffusion Models on Manifolds for Unpaired Image-to-Image Translation. (arXiv:2308.02154v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02154
• Code URL: null
• Copy Paste: [[2308.02154]] SDDM: Score-Decomposed Diffusion Models on Manifolds for Unpaired Image-to-Image Translation(http://arxiv.org/abs/2308.02154)
• Summary:

  Recent score-based diffusion models (SBDMs) show promising results in unpaired image-to-image translation (I2I). However, existing methods, either energy-based or statistically-based, provide no explicit form of the interfered intermediate generative distributions. This work presents a new score-decomposed diffusion model (SDDM) on manifolds to explicitly optimize the tangled distributions during image generation. SDDM derives manifolds to make the distributions of adjacent time steps separable and decompose the score function or energy guidance into an image ``denoising" part and a content ``refinement" part. To refine the image in the same noise level, we equalize the refinement parts of the score function and energy guidance, which permits multi-objective optimization on the manifold. We also leverage the block adaptive instance normalization module to construct manifolds with lower dimensions but still concentrated with the perturbed reference image. SDDM outperforms existing SBDM-based methods with much fewer diffusion steps on several I2I benchmarks.

Title: Painterly Image Harmonization using Diffusion Model. (arXiv:2308.02228v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02228
• Code URL: https://github.com/bcmi/phdiffusion-painterly-image-harmonization
• Copy Paste: [[2308.02228]] Painterly Image Harmonization using Diffusion Model(http://arxiv.org/abs/2308.02228)
• Summary:

  Painterly image harmonization aims to insert photographic objects into paintings and obtain artistically coherent composite images. Previous methods for this task mainly rely on inference optimization or generative adversarial network, but they are either very time-consuming or struggling at fine control of the foreground objects (e.g., texture and content details). To address these issues, we propose a novel Painterly Harmonization stable Diffusion model (PHDiffusion), which includes a lightweight adaptive encoder and a Dual Encoder Fusion (DEF) module. Specifically, the adaptive encoder and the DEF module first stylize foreground features within each encoder. Then, the stylized foreground features from both encoders are combined to guide the harmonization process. During training, besides the noise loss in diffusion model, we additionally employ content loss and two style losses, i.e., AdaIN style loss and contrastive style loss, aiming to balance the trade-off between style migration and content preservation. Compared with the state-of-the-art models from related fields, our PHDiffusion can stylize the foreground more sufficiently and simultaneously retain finer content. Our code and model are available at https://github.com/bcmi/PHDiffusion-Painterly-Image-Harmonization.

Title: Diffusion-Augmented Depth Prediction with Sparse Annotations. (arXiv:2308.02283v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02283
• Code URL: null
• Copy Paste: [[2308.02283]] Diffusion-Augmented Depth Prediction with Sparse Annotations(http://arxiv.org/abs/2308.02283)
• Summary:

  Depth estimation aims to predict dense depth maps. In autonomous driving scenes, sparsity of annotations makes the task challenging. Supervised models produce concave objects due to insufficient structural information. They overfit to valid pixels and fail to restore spatial structures. Self-supervised methods are proposed for the problem. Their robustness is limited by pose estimation, leading to erroneous results in natural scenes. In this paper, we propose a supervised framework termed Diffusion-Augmented Depth Prediction (DADP). We leverage the structural characteristics of diffusion model to enforce depth structures of depth models in a plug-and-play manner. An object-guided integrality loss is also proposed to further enhance regional structure integrality by fetching objective information. We evaluate DADP on three driving benchmarks and achieve significant improvements in depth structures and robustness. Our work provides a new perspective on depth estimation with sparse annotations in autonomous driving scenes.

Title: Improved Order Analysis and Design of Exponential Integrator for Diffusion Models Sampling. (arXiv:2308.02157v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02157
• Code URL: null
• Copy Paste: [[2308.02157]] Improved Order Analysis and Design of Exponential Integrator for Diffusion Models Sampling(http://arxiv.org/abs/2308.02157)
• Summary:

  Efficient differential equation solvers have significantly reduced the sampling time of diffusion models (DMs) while retaining high sampling quality. Among these solvers, exponential integrators (EI) have gained prominence by demonstrating state-of-the-art performance. However, existing high-order EI-based sampling algorithms rely on degenerate EI solvers, resulting in inferior error bounds and reduced accuracy in contrast to the theoretically anticipated results under optimal settings. This situation makes the sampling quality extremely vulnerable to seemingly innocuous design choices such as timestep schedules. For example, an inefficient timestep scheduler might necessitate twice the number of steps to achieve a quality comparable to that obtained through carefully optimized timesteps. To address this issue, we reevaluate the design of high-order differential solvers for DMs. Through a thorough order analysis, we reveal that the degeneration of existing high-order EI solvers can be attributed to the absence of essential order conditions. By reformulating the differential equations in DMs and capitalizing on the theory of exponential integrators, we propose refined EI solvers that fulfill all the order conditions, which we designate as Refined Exponential Solver (RES). Utilizing these improved solvers, RES exhibits more favorable error bounds theoretically and achieves superior sampling efficiency and stability in practical applications. For instance, a simple switch from the single-step DPM-Solver++ to our order-satisfied RES solver when Number of Function Evaluations (NFE) $=9$, results in a reduction of numerical defects by $25.2\%$ and FID improvement of $25.4\%$ (16.77 vs 12.51) on a pre-trained ImageNet diffusion model.

Title: Diffusion probabilistic models enhance variational autoencoder for crystal structure generative modeling. (arXiv:2308.02165v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02165
• Code URL: null
• Copy Paste: [[2308.02165]] Diffusion probabilistic models enhance variational autoencoder for crystal structure generative modeling(http://arxiv.org/abs/2308.02165)
• Summary:

  The crystal diffusion variational autoencoder (CDVAE) is a machine learning model that leverages score matching to generate realistic crystal structures that preserve crystal symmetry. In this study, we leverage novel diffusion probabilistic (DP) models to denoise atomic coordinates rather than adopting the standard score matching approach in CDVAE. Our proposed DP-CDVAE model can reconstruct and generate crystal structures whose qualities are statistically comparable to those of the original CDVAE. Furthermore, notably, when comparing the carbon structures generated by the DP-CDVAE model with relaxed structures obtained from density functional theory calculations, we find that the DP-CDVAE generated structures are remarkably closer to their respective ground states. The energy differences between these structures and the true ground states are, on average, 68.1 meV/atom lower than those generated by the original CDVAE. This significant improvement in the energy accuracy highlights the effectiveness of the DP-CDVAE model in generating crystal structures that better represent their ground-state configurations.

noise learning

data-free

transformer

Title: Dynamic Token-Pass Transformers for Semantic Segmentation. (arXiv:2308.01944v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01944
• Code URL: null
• Copy Paste: [[2308.01944]] Dynamic Token-Pass Transformers for Semantic Segmentation(http://arxiv.org/abs/2308.01944)
• Summary:

  Vision transformers (ViT) usually extract features via forwarding all the tokens in the self-attention layers from top to toe. In this paper, we introduce dynamic token-pass vision transformers (DoViT) for semantic segmentation, which can adaptively reduce the inference cost for images with different complexity. DoViT gradually stops partial easy tokens from self-attention calculation and keeps the hard tokens forwarding until meeting the stopping criteria. We employ lightweight auxiliary heads to make the token-pass decision and divide the tokens into keeping/stopping parts. With a token separate calculation, the self-attention layers are speeded up with sparse tokens and still work friendly with hardware. A token reconstruction module is built to collect and reset the grouped tokens to their original position in the sequence, which is necessary to predict correct semantic masks. We conduct extensive experiments on two common semantic segmentation tasks, and demonstrate that our method greatly reduces about 40% $\sim$ 60% FLOPs and the drop of mIoU is within 0.8% for various segmentation transformers. The throughput and inference speed of ViT-L/B are increased to more than 2$\times$ on Cityscapes.

Title: A Multidimensional Analysis of Social Biases in Vision Transformers. (arXiv:2308.01948v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.01948
• Code URL: null
• Copy Paste: [[2308.01948]] A Multidimensional Analysis of Social Biases in Vision Transformers(http://arxiv.org/abs/2308.01948)
• Summary:

  The embedding spaces of image models have been shown to encode a range of social biases such as racism and sexism. Here, we investigate specific factors that contribute to the emergence of these biases in Vision Transformers (ViT). Therefore, we measure the impact of training data, model architecture, and training objectives on social biases in the learned representations of ViTs. Our findings indicate that counterfactual augmentation training using diffusion-based image editing can mitigate biases, but does not eliminate them. Moreover, we find that larger models are less biased than smaller models, and that models trained using discriminative objectives are less biased than those trained using generative objectives. In addition, we observe inconsistencies in the learned social biases. To our surprise, ViTs can exhibit opposite biases when trained on the same data set using different self-supervised objectives. Our findings give insights into the factors that contribute to the emergence of social biases and suggests that we could achieve substantial fairness improvements based on model design choices.

Title: M2Former: Multi-Scale Patch Selection for Fine-Grained Visual Recognition. (arXiv:2308.02161v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02161
• Code URL: null
• Copy Paste: [[2308.02161]] M2Former: Multi-Scale Patch Selection for Fine-Grained Visual Recognition(http://arxiv.org/abs/2308.02161)
• Summary:

  Recently, vision Transformers (ViTs) have been actively applied to fine-grained visual recognition (FGVR). ViT can effectively model the interdependencies between patch-divided object regions through an inherent self-attention mechanism. In addition, patch selection is used with ViT to remove redundant patch information and highlight the most discriminative object patches. However, existing ViT-based FGVR models are limited to single-scale processing, and their fixed receptive fields hinder representational richness and exacerbate vulnerability to scale variability. Therefore, we propose multi-scale patch selection (MSPS) to improve the multi-scale capabilities of existing ViT-based models. Specifically, MSPS selects salient patches of different scales at different stages of a multi-scale vision Transformer (MS-ViT). In addition, we introduce class token transfer (CTT) and multi-scale cross-attention (MSCA) to model cross-scale interactions between selected multi-scale patches and fully reflect them in model decisions. Compared to previous single-scale patch selection (SSPS), our proposed MSPS encourages richer object representations based on feature hierarchy and consistently improves performance from small-sized to large-sized objects. As a result, we propose M2Former, which outperforms CNN-/ViT-based models on several widely used FGVR benchmarks.

Title: Scene-aware Human Pose Generation using Transformer. (arXiv:2308.02177v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02177
• Code URL: null
• Copy Paste: [[2308.02177]] Scene-aware Human Pose Generation using Transformer(http://arxiv.org/abs/2308.02177)
• Summary:

  Affordance learning considers the interaction opportunities for an actor in the scene and thus has wide application in scene understanding and intelligent robotics. In this paper, we focus on contextual affordance learning, i.e., using affordance as context to generate a reasonable human pose in a scene. Existing scene-aware human pose generation methods could be divided into two categories depending on whether using pose templates. Our proposed method belongs to the template-based category, which benefits from the representative pose templates. Moreover, inspired by recent transformer-based methods, we associate each query embedding with a pose template, and use the interaction between query embeddings and scene feature map to effectively predict the scale and offsets for each pose template. In addition, we employ knowledge distillation to facilitate the offset learning given the predicted scale. Comprehensive experiments on Sitcom dataset demonstrate the effectiveness of our method.

Title: Bengali Fake Reviews: A Benchmark Dataset and Detection System. (arXiv:2308.01987v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.01987
• Code URL: null
• Copy Paste: [[2308.01987]] Bengali Fake Reviews: A Benchmark Dataset and Detection System(http://arxiv.org/abs/2308.01987)
• Summary:

  The proliferation of fake reviews on various online platforms has created a major concern for both consumers and businesses. Such reviews can deceive customers and cause damage to the reputation of products or services, making it crucial to identify them. Although the detection of fake reviews has been extensively studied in English language, detecting fake reviews in non-English languages such as Bengali is still a relatively unexplored research area. This paper introduces the Bengali Fake Review Detection (BFRD) dataset, the first publicly available dataset for identifying fake reviews in Bengali. The dataset consists of 7710 non-fake and 1339 fake food-related reviews collected from social media posts. To convert non-Bengali words in a review, a unique pipeline has been proposed that translates English words to their corresponding Bengali meaning and also back transliterates Romanized Bengali to Bengali. We have conducted rigorous experimentation using multiple deep learning and pre-trained transformer language models to develop a reliable detection system. Finally, we propose a weighted ensemble model that combines four pre-trained transformers: BanglaBERT, BanglaBERT Base, BanglaBERT Large, and BanglaBERT Generator . According to the experiment results, the proposed ensemble model obtained a weighted F1-score of 0.9843 on 13390 reviews, including 1339 actual fake reviews and 5356 augmented fake reviews generated with the nlpaug library. The remaining 6695 reviews were randomly selected from the 7710 non-fake instances. The model achieved a 0.9558 weighted F1-score when the fake reviews were augmented using the bnaug library.

Title: A Transformer-based Prediction Method for Depth of Anesthesia During Target-controlled Infusion of Propofol and Remifentanil. (arXiv:2308.01929v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.01929
• Code URL: https://github.com/heeeyk/transformer-doa-prediction
• Copy Paste: [[2308.01929]] A Transformer-based Prediction Method for Depth of Anesthesia During Target-controlled Infusion of Propofol and Remifentanil(http://arxiv.org/abs/2308.01929)
• Summary:

  Accurately predicting anesthetic effects is essential for target-controlled infusion systems. The traditional (PK-PD) models for Bispectral index (BIS) prediction require manual selection of model parameters, which can be challenging in clinical settings. Recently proposed deep learning methods can only capture general trends and may not predict abrupt changes in BIS. To address these issues, we propose a transformer-based method for predicting the depth of anesthesia (DOA) using drug infusions of propofol and remifentanil. Our method employs long short-term memory (LSTM) and gate residual network (GRN) networks to improve the efficiency of feature fusion and applies an attention mechanism to discover the interactions between the drugs. We also use label distribution smoothing and reweighting losses to address data imbalance. Experimental results show that our proposed method outperforms traditional PK-PD models and previous deep learning methods, effectively predicting anesthetic depth under sudden and deep anesthesia conditions.

generative

Title: Towards Generalist Foundation Model for Radiology. (arXiv:2308.02463v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02463
• Code URL: https://github.com/chaoyi-wu/radfm
• Copy Paste: [[2308.02463]] Towards Generalist Foundation Model for Radiology(http://arxiv.org/abs/2308.02463)
• Summary:

  In this study, we aim to initiate the development of Radiology Foundation Model, termed as RadFM.We consider the construction of foundational models from the perspectives of data, model design, and evaluation thoroughly. Our contribution can be concluded as follows: (i), we construct a large-scale Medical Multi-modal Dataset, MedMD, consisting of 16M 2D and 3D medical scans. To the best of our knowledge, this is the first multi-modal dataset containing 3D medical scans. (ii), We propose an architecture that enables visually conditioned generative pre-training, allowing for the integration of text input interleaved with 2D or 3D medical scans to generate response for diverse radiologic tasks. The model was initially pre-trained on MedMD and subsequently domain-specific fine-tuned on RadMD, a radiologic cleaned version of MedMD, containing 3M radiologic visual-language pairs. (iii), we propose a new evaluation benchmark that comprises five tasks, aiming to comprehensively assess the capability of foundation models in handling practical clinical problems. Our experimental results confirm that RadFM significantly outperforms existing multi-modal foundation models. The codes, data, and model checkpoint will all be made publicly available to promote further research and development in the field.

Title: You talk what you read: Understanding News Comment Behavior by Dispositional and Situational Attribution. (arXiv:2308.02168v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.02168
• Code URL: null
• Copy Paste: [[2308.02168]] You talk what you read: Understanding News Comment Behavior by Dispositional and Situational Attribution(http://arxiv.org/abs/2308.02168)
• Summary:

  Many news comment mining studies are based on the assumption that comment is explicitly linked to the corresponding news. In this paper, we observed that users' comments are also heavily influenced by their individual characteristics embodied by the interaction history. Therefore, we position to understand news comment behavior by considering both the dispositional factors from news interaction history, and the situational factors from corresponding news. A three-part encoder-decoder framework is proposed to model the generative process of news comment. The resultant dispositional and situational attribution contributes to understanding user focus and opinions, which are validated in applications of reader-aware news summarization and news aspect-opinion forecasting.

large language model

Title: Domain specificity and data efficiency in typo tolerant spell checkers: the case of search in online marketplaces. (arXiv:2308.01976v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.01976
• Code URL: null
• Copy Paste: [[2308.01976]] Domain specificity and data efficiency in typo tolerant spell checkers: the case of search in online marketplaces(http://arxiv.org/abs/2308.01976)
• Summary:

  Typographical errors are a major source of frustration for visitors of online marketplaces. Because of the domain-specific nature of these marketplaces and the very short queries users tend to search for, traditional spell cheking solutions do not perform well in correcting typos. We present a data augmentation method to address the lack of annotated typo data and train a recurrent neural network to learn context-limited domain-specific embeddings. Those embeddings are deployed in a real-time inferencing API for the Microsoft AppSource marketplace to find the closest match between a misspelled user query and the available product names. Our data efficient solution shows that controlled high quality synthetic data may be a powerful tool especially considering the current climate of large language models which rely on prohibitively huge and often uncontrolled datasets.

Title: The Unequal Opportunities of Large Language Models: Revealing Demographic Bias through Job Recommendations. (arXiv:2308.02053v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.02053
• Code URL: null
• Copy Paste: [[2308.02053]] The Unequal Opportunities of Large Language Models: Revealing Demographic Bias through Job Recommendations(http://arxiv.org/abs/2308.02053)
• Summary:

  Large Language Models (LLMs) have seen widespread deployment in various real-world applications. Understanding these biases is crucial to comprehend the potential downstream consequences when using LLMs to make decisions, particularly for historically disadvantaged groups. In this work, we propose a simple method for analyzing and comparing demographic bias in LLMs, through the lens of job recommendations. We demonstrate the effectiveness of our method by measuring intersectional biases within ChatGPT and LLaMA, two cutting-edge LLMs. Our experiments primarily focus on uncovering gender identity and nationality bias; however, our method can be extended to examine biases associated with any intersection of demographic identities. We identify distinct biases in both models toward various demographic identities, such as both models consistently suggesting low-paying jobs for Mexican workers or preferring to recommend secretarial roles to women. Our study highlights the importance of measuring the bias of LLMs in downstream applications to understand the potential for harm and inequitable outcomes.

Title: Scaling Clinical Trial Matching Using Large Language Models: A Case Study in Oncology. (arXiv:2308.02180v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.02180
• Code URL: null
• Copy Paste: [[2308.02180]] Scaling Clinical Trial Matching Using Large Language Models: A Case Study in Oncology(http://arxiv.org/abs/2308.02180)
• Summary:

  Clinical trial matching is a key process in health delivery and discovery. In practice, it is plagued by overwhelming unstructured data and unscalable manual processing. In this paper, we conduct a systematic study on scaling clinical trial matching using large language models (LLMs), with oncology as the focus area. Our study is grounded in a clinical trial matching system currently in test deployment at a large U.S. health network. Initial findings are promising: out of box, cutting-edge LLMs, such as GPT-4, can already structure elaborate eligibility criteria of clinical trials and extract complex matching logic (e.g., nested AND/OR/NOT). While still far from perfect, LLMs substantially outperform prior strong baselines and may serve as a preliminary solution to help triage patient-trial candidates with humans in the loop. Our study also reveals a few significant growth areas for applying LLMs to end-to-end clinical trial matching, such as context limitation and accuracy, especially in structuring patient information from longitudinal medical records.

Title: Learning to Paraphrase Sentences to Different Complexity Levels. (arXiv:2308.02226v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.02226
• Code URL: https://github.com/alisonhc/change-complexity
• Copy Paste: [[2308.02226]] Learning to Paraphrase Sentences to Different Complexity Levels(http://arxiv.org/abs/2308.02226)
• Summary:

  While sentence simplification is an active research topic in NLP, its adjacent tasks of sentence complexification and same-level paraphrasing are not. To train models on all three tasks, we present two new unsupervised datasets. We compare these datasets, one labeled by a weak classifier and the other by a rule-based approach, with a single supervised dataset. Using these three datasets for training, we perform extensive experiments on both multitasking and prompting strategies. Compared to other systems trained on unsupervised parallel data, models trained on our weak classifier labeled dataset achieve state-of-the-art performance on the ASSET simplification benchmark. Our models also outperform previous work on sentence level targeting. Finally, we establish how a handful of Large Language Models perform on these tasks under a zero-shot setting.

segmentation

Title: UGainS: Uncertainty Guided Anomaly Instance Segmentation. (arXiv:2308.02046v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02046
• Code URL: null
• Copy Paste: [[2308.02046]] UGainS: Uncertainty Guided Anomaly Instance Segmentation(http://arxiv.org/abs/2308.02046)
• Summary:

  A single unexpected object on the road can cause an accident or may lead to injuries. To prevent this, we need a reliable mechanism for finding anomalous objects on the road. This task, called anomaly segmentation, can be a stepping stone to safe and reliable autonomous driving. Current approaches tackle anomaly segmentation by assigning an anomaly score to each pixel and by grouping anomalous regions using simple heuristics. However, pixel grouping is a limiting factor when it comes to evaluating the segmentation performance of individual anomalous objects. To address the issue of grouping multiple anomaly instances into one, we propose an approach that produces accurate anomaly instance masks. Our approach centers on an out-of-distribution segmentation model for identifying uncertain regions and a strong generalist segmentation model for anomaly instances segmentation. We investigate ways to use uncertain regions to guide such a segmentation model to perform segmentation of anomalous instances. By incorporating strong object priors from a generalist model we additionally improve the per-pixel anomaly segmentation performance. Our approach outperforms current pixel-level anomaly segmentation methods, achieving an AP of 80.08% and 88.98% on the Fishyscapes Lost and Found and the RoadAnomaly validation sets respectively. Project page: https://vision.rwth-aachen.de/ugains

Title: Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation. (arXiv:2308.02097v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02097
• Code URL: https://github.com/jinyuanliu-cv/segmif
• Copy Paste: [[2308.02097]] Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation(http://arxiv.org/abs/2308.02097)
• Summary:

  Multi-modality image fusion and segmentation play a vital role in autonomous driving and robotic operation. Early efforts focus on boosting the performance for only one task, \emph{e.g.,} fusion or segmentation, making it hard to reach~`Best of Both Worlds'. To overcome this issue, in this paper, we propose a \textbf{M}ulti-\textbf{i}nteractive \textbf{F}eature learning architecture for image fusion and \textbf{Seg}mentation, namely SegMiF, and exploit dual-task correlation to promote the performance of both tasks. The SegMiF is of a cascade structure, containing a fusion sub-network and a commonly used segmentation sub-network. By slickly bridging intermediate features between two components, the knowledge learned from the segmentation task can effectively assist the fusion task. Also, the benefited fusion network supports the segmentation one to perform more pretentiously. Besides, a hierarchical interactive attention block is established to ensure fine-grained mapping of all the vital information between two tasks, so that the modality/semantic features can be fully mutual-interactive. In addition, a dynamic weight factor is introduced to automatically adjust the corresponding weights of each task, which can balance the interactive feature correspondence and break through the limitation of laborious tuning. Furthermore, we construct a smart multi-wave binocular imaging system and collect a full-time multi-modality benchmark with 15 annotated pixel-level categories for image fusion and segmentation. Extensive experiments on several public datasets and our benchmark demonstrate that the proposed method outputs visually appealing fused images and perform averagely $7.66\%$ higher segmentation mIoU in the real-world scene than the state-of-the-art approaches. The source code and benchmark are available at \url{https://github.com/JinyuanLiu-CV/SegMiF}.

Title: Rethinking Class Activation Maps for Segmentation: Revealing Semantic Information in Shallow Layers by Reducing Noise. (arXiv:2308.02118v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02118
• Code URL: null
• Copy Paste: [[2308.02118]] Rethinking Class Activation Maps for Segmentation: Revealing Semantic Information in Shallow Layers by Reducing Noise(http://arxiv.org/abs/2308.02118)
• Summary:

  Class activation maps are widely used for explaining deep neural networks. Due to its ability to highlight regions of interest, it has evolved in recent years as a key step in weakly supervised learning. A major limitation to the performance of the class activation maps is the small spatial resolution of the feature maps in the last layer of the convolutional neural network. Therefore, we expect to generate high-resolution feature maps that result in high-quality semantic information. In this paper, we rethink the properties of semantic information in shallow feature maps. We find that the shallow feature maps still have fine-grained non-discriminative features while mixing considerable non-target noise. Furthermore, we propose a simple gradient-based denoising method to filter the noise by truncating the positive gradient. Our proposed scheme can be easily deployed in other CAM-related methods, facilitating these methods to obtain higher-quality class activation maps. We evaluate the proposed approach through a weakly-supervised semantic segmentation task, and a large number of experiments demonstrate the effectiveness of our approach.

Title: Learning Referring Video Object Segmentation from Weak Annotation. (arXiv:2308.02162v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02162
• Code URL: https://github.com/wangbo-zhao/wrvos
• Copy Paste: [[2308.02162]] Learning Referring Video Object Segmentation from Weak Annotation(http://arxiv.org/abs/2308.02162)
• Summary:

  Referring video object segmentation (RVOS) is a task that aims to segment the target object in all video frames based on a sentence describing the object. Previous RVOS methods have achieved significant performance with densely-annotated datasets, whose construction is expensive and time-consuming. To relieve the burden of data annotation while maintaining sufficient supervision for segmentation, we propose a new annotation scheme, in which we label the frame where the object first appears with a mask and use bounding boxes for the subsequent frames. Based on this scheme, we propose a method to learn from this weak annotation. Specifically, we design a cross frame segmentation method, which uses the language-guided dynamic filters to thoroughly leverage the valuable mask annotation and bounding boxes. We further develop a bi-level contrastive learning method to encourage the model to learn discriminative representation at the pixel level. Extensive experiments and ablative analyses show that our method is able to achieve competitive performance without the demand of dense mask annotation. The code will be available at https://github.com/wangbo-zhao/WRVOS/.

Title: Synthetic outlier generation for anomaly detection in autonomous driving. (arXiv:2308.02184v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02184
• Code URL: null
• Copy Paste: [[2308.02184]] Synthetic outlier generation for anomaly detection in autonomous driving(http://arxiv.org/abs/2308.02184)
• Summary:

  Anomaly detection, or outlier detection, is a crucial task in various domains to identify instances that significantly deviate from established patterns or the majority of data. In the context of autonomous driving, the identification of anomalies is particularly important to prevent safety-critical incidents, as deep learning models often exhibit overconfidence in anomalous or outlier samples. In this study, we explore different strategies for training an image semantic segmentation model with an anomaly detection module. By introducing modifications to the training stage of the state-of-the-art DenseHybrid model, we achieve significant performance improvements in anomaly detection. Moreover, we propose a simplified detector that achieves comparable results to our modified DenseHybrid approach, while also surpassing the performance of the original DenseHybrid model. These findings demonstrate the efficacy of our proposed strategies for enhancing anomaly detection in the context of autonomous driving.

Title: ES-MVSNet: Efficient Framework for End-to-end Self-supervised Multi-View Stereo. (arXiv:2308.02191v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02191
• Code URL: null
• Copy Paste: [[2308.02191]] ES-MVSNet: Efficient Framework for End-to-end Self-supervised Multi-View Stereo(http://arxiv.org/abs/2308.02191)
• Summary:

  Compared to the multi-stage self-supervised multi-view stereo (MVS) method, the end-to-end (E2E) approach has received more attention due to its concise and efficient training pipeline. Recent E2E self-supervised MVS approaches have integrated third-party models (such as optical flow models, semantic segmentation models, NeRF models, etc.) to provide additional consistency constraints, which grows GPU memory consumption and complicates the model's structure and training pipeline. In this work, we propose an efficient framework for end-to-end self-supervised MVS, dubbed ES-MVSNet. To alleviate the high memory consumption of current E2E self-supervised MVS frameworks, we present a memory-efficient architecture that reduces memory usage by 43% without compromising model performance. Furthermore, with the novel design of asymmetric view selection policy and region-aware depth consistency, we achieve state-of-the-art performance among E2E self-supervised MVS methods, without relying on third-party models for additional consistency signals. Extensive experiments on DTU and Tanks&Temples benchmarks demonstrate that the proposed ES-MVSNet approach achieves state-of-the-art performance among E2E self-supervised MVS methods and competitive performance to many supervised and multi-stage self-supervised methods.

Title: Deep Semantic Model Fusion for Ancient Agricultural Terrace Detection. (arXiv:2308.02225v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02225
• Code URL: https://github.com/wangyi111/international-archaeology-ai-challenge
• Copy Paste: [[2308.02225]] Deep Semantic Model Fusion for Ancient Agricultural Terrace Detection(http://arxiv.org/abs/2308.02225)
• Summary:

  Discovering ancient agricultural terraces in desert regions is important for the monitoring of long-term climate changes on the Earth's surface. However, traditional ground surveys are both costly and limited in scale. With the increasing accessibility of aerial and satellite data, machine learning techniques bear large potential for the automatic detection and recognition of archaeological landscapes. In this paper, we propose a deep semantic model fusion method for ancient agricultural terrace detection. The input data includes aerial images and LiDAR generated terrain features in the Negev desert. Two deep semantic segmentation models, namely DeepLabv3+ and UNet, with EfficientNet backbone, are trained and fused to provide segmentation maps of ancient terraces and walls. The proposed method won the first prize in the International AI Archaeology Challenge. Codes are available at https://github.com/wangyi111/international-archaeology-ai-challenge.

Title: On the Calibration of Uncertainty Estimation in LiDAR-based Semantic Segmentation. (arXiv:2308.02248v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02248
• Code URL: null
• Copy Paste: [[2308.02248]] On the Calibration of Uncertainty Estimation in LiDAR-based Semantic Segmentation(http://arxiv.org/abs/2308.02248)
• Summary:

  The confidence calibration of deep learning-based perception models plays a crucial role in their reliability. Especially in the context of autonomous driving, downstream tasks like prediction and planning depend on accurate confidence estimates. In point-wise multiclass classification tasks like sematic segmentation the model has to deal with heavy class imbalances. Due to their underrepresentation, the confidence calibration of classes with smaller instances is challenging but essential, not only for safety reasons. We propose a metric to measure the confidence calibration quality of a semantic segmentation model with respect to individual classes. It is calculated by computing sparsification curves for each class based on the uncertainty estimates. We use the classification calibration metric to evaluate uncertainty estimation methods with respect to their confidence calibration of underrepresented classes. We furthermore suggest a double use for the method to automatically find label problems to improve the quality of hand- or auto-annotated datasets.

Title: Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP. (arXiv:2308.02487v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.02487
• Code URL: https://github.com/bytedance/fc-clip
• Copy Paste: [[2308.02487]] Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP(http://arxiv.org/abs/2308.02487)
• Summary:

  Open-vocabulary segmentation is a challenging task requiring segmenting and recognizing objects from an open set of categories. One way to address this challenge is to leverage multi-modal models, such as CLIP, to provide image and text features in a shared embedding space, which bridges the gap between closed-vocabulary and open-vocabulary recognition. Hence, existing methods often adopt a two-stage framework to tackle the problem, where the inputs first go through a mask generator and then through the CLIP model along with the predicted masks. This process involves extracting features from images multiple times, which can be ineffective and inefficient. By contrast, we propose to build everything into a single-stage framework using a shared Frozen Convolutional CLIP backbone, which not only significantly simplifies the current two-stage pipeline, but also remarkably yields a better accuracy-cost trade-off. The proposed FC-CLIP, benefits from the following observations: the frozen CLIP backbone maintains the ability of open-vocabulary classification and can also serve as a strong mask generator, and the convolutional CLIP generalizes well to a larger input resolution than the one used during contrastive image-text pretraining. When training on COCO panoptic data only and testing in a zero-shot manner, FC-CLIP achieve 26.8 PQ, 16.8 AP, and 34.1 mIoU on ADE20K, 18.2 PQ, 27.9 mIoU on Mapillary Vistas, 44.0 PQ, 26.8 AP, 56.2 mIoU on Cityscapes, outperforming the prior art by +4.2 PQ, +2.4 AP, +4.2 mIoU on ADE20K, +4.0 PQ on Mapillary Vistas and +20.1 PQ on Cityscapes, respectively. Additionally, the training and testing time of FC-CLIP is 7.5x and 6.6x significantly faster than the same prior art, while using 5.9x fewer parameters. FC-CLIP also sets a new state-of-the-art performance across various open-vocabulary semantic segmentation datasets. Code at https://github.com/bytedance/fc-clip

Title: A Graphical Approach to Document Layout Analysis. (arXiv:2308.02051v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02051
• Code URL: null
• Copy Paste: [[2308.02051]] A Graphical Approach to Document Layout Analysis(http://arxiv.org/abs/2308.02051)
• Summary:

  Document layout analysis (DLA) is the task of detecting the distinct, semantic content within a document and correctly classifying these items into an appropriate category (e.g., text, title, figure). DLA pipelines enable users to convert documents into structured machine-readable formats that can then be used for many useful downstream tasks. Most existing state-of-the-art (SOTA) DLA models represent documents as images, discarding the rich metadata available in electronically generated PDFs. Directly leveraging this metadata, we represent each PDF page as a structured graph and frame the DLA problem as a graph segmentation and classification problem. We introduce the Graph-based Layout Analysis Model (GLAM), a lightweight graph neural network competitive with SOTA models on two challenging DLA datasets - while being an order of magnitude smaller than existing models. In particular, the 4-million parameter GLAM model outperforms the leading 140M+ parameter computer vision-based model on 5 of the 11 classes on the DocLayNet dataset. A simple ensemble of these two models achieves a new state-of-the-art on DocLayNet, increasing mAP from 76.8 to 80.8. Overall, GLAM is over 5 times more efficient than SOTA models, making GLAM a favorable engineering choice for DLA tasks.

Title: Frustratingly Easy Model Generalization by Dummy Risk Minimization. (arXiv:2308.02287v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.02287
• Code URL: null
• Copy Paste: [[2308.02287]] Frustratingly Easy Model Generalization by Dummy Risk Minimization(http://arxiv.org/abs/2308.02287)
• Summary:

  Empirical risk minimization (ERM) is a fundamental machine learning paradigm. However, its generalization ability is limited in various tasks. In this paper, we devise Dummy Risk Minimization (DuRM), a frustratingly easy and general technique to improve the generalization of ERM. DuRM is extremely simple to implement: just enlarging the dimension of the output logits and then optimizing using standard gradient descent. Moreover, we validate the efficacy of DuRM on both theoretical and empirical analysis. Theoretically, we show that DuRM derives greater variance of the gradient, which facilitates model generalization by observing better flat local minima. Empirically, we conduct evaluations of DuRM across different datasets, modalities, and network architectures on diverse tasks, including conventional classification, semantic segmentation, out-of-distribution generalization, adverserial training, and long-tailed recognition. Results demonstrate that DuRM could consistently improve the performance under all tasks with an almost free lunch manner. Furthermore, we show that DuRM is compatible with existing generalization techniques and we discuss possible limitations. We hope that DuRM could trigger new interest in the fundamental research on risk minimization.