secure

Title: Split-State Non-Malleable Codes and Secret Sharing Schemes for Quantum Messages. (arXiv:2308.06466v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.06466
• Code URL: null
• Copy Paste: [[2308.06466]] Split-State Non-Malleable Codes and Secret Sharing Schemes for Quantum Messages(http://arxiv.org/abs/2308.06466)
• Summary:

  Non-malleable codes are fundamental objects at the intersection of cryptography and coding theory. These codes provide security guarantees even in settings where error correction and detection are impossible, and have found applications to several other cryptographic tasks. Roughly speaking, a non-malleable code for a family of tampering functions guarantees that no adversary can tamper (using functions from this family) the encoding of a given message into the encoding of a related distinct message. Non-malleable secret sharing schemes are a strengthening of non-malleable codes which satisfy additional privacy and reconstruction properties.

  We first focus on the $2$-split-state tampering model, one of the strongest and most well-studied adversarial tampering models. Here, a codeword is split into two parts which are stored in physically distant servers, and the adversary can then independently tamper with each part using arbitrary functions. This model can be naturally extended to the secret sharing setting with several parties by having the adversary independently tamper with each share.

  Previous works on non-malleable coding and secret sharing in the split-state tampering model only considered the encoding of \emph{classical} messages. Furthermore, until the recent work by Aggarwal, Boddu, and Jain (arXiv 2022), adversaries with quantum capabilities and \emph{shared entanglement} had not been considered, and it is a priori not clear whether previous schemes remain secure in this model.

  In this work, we introduce the notions of split-state non-malleable codes and secret sharing schemes for quantum messages secure against quantum adversaries with shared entanglement. We also present explicit constructions of such schemes that achieve low-error non-malleability.

security

Title: Improved YOLOv8 Detection Algorithm in Security Inspection Image. (arXiv:2308.06452v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06452
• Code URL: null
• Copy Paste: [[2308.06452]] Improved YOLOv8 Detection Algorithm in Security Inspection Image(http://arxiv.org/abs/2308.06452)
• Summary:

  Security inspection is the first line of defense to ensure the safety of people's lives and property, and intelligent security inspection is an inevitable trend in the future development of the security inspection industry. Aiming at the problems of overlapping detection objects, false detection of contraband, and missed detection in the process of X-ray image detection, an improved X-ray contraband detection algorithm CSS-YOLO based on YOLOv8s is proposed.

Title: On the Security Bootstrapping in Named Data Networking. (arXiv:2308.06490v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.06490
• Code URL: null
• Copy Paste: [[2308.06490]] On the Security Bootstrapping in Named Data Networking(http://arxiv.org/abs/2308.06490)
• Summary:

  By requiring all data packets been cryptographically authenticatable, the Named Data Networking (NDN) architecture design provides a basic building block for secured networking. This basic NDN function requires that all entities in an NDN network go through a security bootstrapping process to obtain the initial security credentials. Recent years have witnessed a number of proposed solutions for NDN security bootstrapping protocols. Built upon the existing results, in this paper we take the next step to develop a systematic model of security bootstrapping: Trust-domain Entity Bootstrapping (TEB). This model is based on the emerging concept of trust domain and describes the steps and their dependencies in the bootstrapping process. We evaluate the expressiveness and sufficiency of this model by using it to describe several current bootstrapping protocols.

Title: PQC-HA: A Framework for Prototyping and In-Hardware Evaluation of Post-Quantum Cryptography Hardware Accelerators. (arXiv:2308.06621v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.06621
• Code URL: null
• Copy Paste: [[2308.06621]] PQC-HA: A Framework for Prototyping and In-Hardware Evaluation of Post-Quantum Cryptography Hardware Accelerators(http://arxiv.org/abs/2308.06621)
• Summary:

  In the third round of the NIST Post-Quantum Cryptography standardization project, the focus is on optimizing software and hardware implementations of candidate schemes. The winning schemes are CRYSTALS Kyber and CRYSTALS Dilithium, which serve as a Key Encapsulation Mechanism (KEM) and Digital Signature Algorithm (DSA), respectively. This study utilizes the TaPaSCo open-source framework to create hardware building blocks for both schemes using High-level Synthesis (HLS) from minimally modified ANSI C software reference implementations across all security levels. Additionally, a generic TaPaSCo host runtime application is developed in Rust to verify their functionality through the standard NIST interface, utilizing the corresponding Known Answer Test mechanism on actual hardware. Building on this foundation, the communication overhead for TaPaSCo hardware accelerators on PCIe-connected FPGA devices is evaluated and compared with previous work and optimized AVX2 software reference implementations. The results demonstrate the feasibility of verifying and evaluating the performance of Post-Quantum Cryptography accelerators on real hardware using TaPaSCo. Furthermore, the off-chip accelerator communication overhead of the NIST standard interface is measured, which, on its own, outweighs the execution wall clock time of the optimized software reference implementation of Kyber at Security Level 1.

Title: Helion: Enabling Natural Testing of Smart Homes. (arXiv:2308.06695v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.06695
• Code URL: null
• Copy Paste: [[2308.06695]] Helion: Enabling Natural Testing of Smart Homes(http://arxiv.org/abs/2308.06695)
• Summary:

  Prior work has developed numerous systems that test the security and safety of smart homes. For these systems to be applicable in practice, it is necessary to test them with realistic scenarios that represent the use of the smart home, i.e., home automation, in the wild. This demo paper presents the technical details and usage of Helion, a system that uses n-gram language modeling to learn the regularities in user-driven programs, i.e., routines developed for the smart home, and predicts natural scenarios of home automation, i.e., event sequences that reflect realistic home automation usage. We demonstrate the HelionHA platform, developed by integrating Helion with the popular Home Assistant smart home platform. HelionHA allows an end-to-end exploration of Helion's scenarios by executing them as test cases with real and virtual smart home devices.

Title: ALGAN: Time Series Anomaly Detection with Adjusted-LSTM GAN. (arXiv:2308.06663v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.06663
• Code URL: null
• Copy Paste: [[2308.06663]] ALGAN: Time Series Anomaly Detection with Adjusted-LSTM GAN(http://arxiv.org/abs/2308.06663)
• Summary:

  Anomaly detection in time series data, to identify points that deviate from normal behaviour, is a common problem in various domains such as manufacturing, medical imaging, and cybersecurity. Recently, Generative Adversarial Networks (GANs) are shown to be effective in detecting anomalies in time series data. The neural network architecture of GANs (i.e. Generator and Discriminator) can significantly improve anomaly detection accuracy. In this paper, we propose a new GAN model, named Adjusted-LSTM GAN (ALGAN), which adjusts the output of an LSTM network for improved anomaly detection in both univariate and multivariate time series data in an unsupervised setting. We evaluate the performance of ALGAN on 46 real-world univariate time series datasets and a large multivariate dataset that spans multiple domains. Our experiments demonstrate that ALGAN outperforms traditional, neural network-based, and other GAN-based methods for anomaly detection in time series data.

privacy

Title: Sparsity and Privacy in Secret Sharing: A Fundamental Trade-Off. (arXiv:2308.06413v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.06413
• Code URL: null
• Copy Paste: [[2308.06413]] Sparsity and Privacy in Secret Sharing: A Fundamental Trade-Off(http://arxiv.org/abs/2308.06413)
• Summary:

  This work investigates the design of sparse secret sharing schemes that encode a sparse private matrix into sparse shares. This investigation is motivated by distributed computing, where the multiplication of sparse and private matrices is moved from a computationally weak main node to untrusted worker machines. Classical secret-sharing schemes produce dense shares. However, sparsity can help speed up the computation. We show that, for matrices with i.i.d. entries, sparsity in the shares comes at a fundamental cost of weaker privacy. We derive a fundamental tradeoff between sparsity and privacy and construct optimal sparse secret sharing schemes that produce shares that leak the minimum amount of information for a desired sparsity of the shares. We apply our schemes to distributed sparse and private matrix multiplication schemes with no colluding workers while tolerating stragglers. For the setting of two non-communicating clusters of workers, we design a sparse one-time pad so that no private information is leaked to a cluster of untrusted and colluding workers, and the shares with bounded but non-zero leakage are assigned to a cluster of partially trusted workers. We conclude by discussing the necessity of using permutations for matrices with correlated entries.

protect

Title: SGX-MR-Prot: Efficient and Developer-Friendly Access-Pattern Protection in Trusted Execution Environments. (arXiv:2308.06445v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.06445
• Code URL: null
• Copy Paste: [[2308.06445]] SGX-MR-Prot: Efficient and Developer-Friendly Access-Pattern Protection in Trusted Execution Environments(http://arxiv.org/abs/2308.06445)
• Summary:

  Trusted Execution Environments, such as Intel SGX, use hardware supports to ensure the confidentiality and integrity of applications against a compromised cloud system. However, side channels like access patterns remain for adversaries to exploit and obtain sensitive information. Common approaches use oblivious programs or primitives, such as ORAM, to make access patterns oblivious to input data, which are challenging to develop. This demonstration shows a prototype SGX-MR-Prot for efficiently protecting access patterns of SGX-based data-intensive applications and minimizing developers' efforts. SGX-MR-Prot uses the MapReduce framework to regulate application dataflows to reduce the cost of access-pattern protection and hide the data oblivious details from SGX developers. This demonstration will allow users to intuitively understand the unique contributions of the framework-based protection approach via interactive exploration and visualization.

defense

attack

Title: DFM-X: Augmentation by Leveraging Prior Knowledge of Shortcut Learning. (arXiv:2308.06622v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06622
• Code URL: https://github.com/nis-research/dfmx-augmentation
• Copy Paste: [[2308.06622]] DFM-X: Augmentation by Leveraging Prior Knowledge of Shortcut Learning(http://arxiv.org/abs/2308.06622)
• Summary:

  Neural networks are prone to learn easy solutions from superficial statistics in the data, namely shortcut learning, which impairs generalization and robustness of models. We propose a data augmentation strategy, named DFM-X, that leverages knowledge about frequency shortcuts, encoded in Dominant Frequencies Maps computed for image classification models. We randomly select X% training images of certain classes for augmentation, and process them by retaining the frequencies included in the DFMs of other classes. This strategy compels the models to leverage a broader range of frequencies for classification, rather than relying on specific frequency sets. Thus, the models learn more deep and task-related semantics compared to their counterpart trained with standard setups. Unlike other commonly used augmentation techniques which focus on increasing the visual variations of training data, our method targets exploiting the original data efficiently, by distilling prior knowledge about destructive learning behavior of models from data. Our experimental results demonstrate that DFM-X improves robustness against common corruptions and adversarial attacks. It can be seamlessly integrated with other augmentation techniques to further enhance the robustness of models.

Title: White-box Membership Inference Attacks against Diffusion Models. (arXiv:2308.06405v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.06405
• Code URL: null
• Copy Paste: [[2308.06405]] White-box Membership Inference Attacks against Diffusion Models(http://arxiv.org/abs/2308.06405)
• Summary:

  Diffusion models have begun to overshadow GANs and other generative models in industrial applications due to their superior image generation performance. The complex architecture of these models furnishes an extensive array of attack features. In light of this, we aim to design membership inference attacks (MIAs) catered to diffusion models. We first conduct an exhaustive analysis of existing MIAs on diffusion models, taking into account factors such as black-box/white-box models and the selection of attack features. We found that white-box attacks are highly applicable in real-world scenarios, and the most effective attacks presently are white-box. Departing from earlier research, which employs model loss as the attack feature for white-box MIAs, we employ model gradients in our attack, leveraging the fact that these gradients provide a more profound understanding of model responses to various samples. We subject these models to rigorous testing across a range of parameters, including training steps, sampling frequency, diffusion steps, and data variance. Across all experimental settings, our method consistently demonstrated near-flawless attack performance, with attack success rate approaching $100\%$ and attack AUCROC near $1.0$. We also evaluate our attack against common defense mechanisms, and observe our attacks continue to exhibit commendable performance.

Title: Making Your Program Oblivious: a Comparative Study for Side-channel-safe Confidential Computing. (arXiv:2308.06442v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.06442
• Code URL: null
• Copy Paste: [[2308.06442]] Making Your Program Oblivious: a Comparative Study for Side-channel-safe Confidential Computing(http://arxiv.org/abs/2308.06442)
• Summary:

  Trusted Execution Environments (TEEs) are gradually adopted by major cloud providers, offering a practical option of \emph{confidential computing} for users who don't fully trust public clouds. TEEs use CPU-enabled hardware features to eliminate direct breaches from compromised operating systems or hypervisors. However, recent studies have shown that side-channel attacks are still effective on TEEs. An appealing solution is to convert applications to be \emph{data oblivious} to deter many side-channel attacks. While a few research prototypes on TEEs have adopted specific data oblivious operations, the general conversion approaches have never been thoroughly compared against and tested on benchmark TEE applications. These limitations make it difficult for researchers and practitioners to choose and adopt a suitable data oblivious approach for their applications. To address these issues, we conduct a comprehensive analysis of several representative conversion approaches and implement benchmark TEE applications with them. We also perform an extensive empirical study to provide insights into their performance and ease of use.

Title: "Zero Cost'' Majority Attacks on Permissionless Blockchains. (arXiv:2308.06568v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.06568
• Code URL: null
• Copy Paste: [[2308.06568]] "Zero Cost'' Majority Attacks on Permissionless Blockchains(http://arxiv.org/abs/2308.06568)
• Summary:

  The core premise of permissionless blockchains is their reliable and secure operation without the need to trust any individual agent. At the heart of blockchain consensus mechanisms is an explicit cost (whether work or stake) for participation in the network and the opportunity to add blocks to the blockchain. A key rationale for that cost is to make attacks on the network, which could be theoretically carried out if a majority of nodes were controlled by a single entity, too expensive to be worthwhile. We demonstrate that a majority attacker can successfully attack with a {\em negative cost}, which shows that the protocol mechanisms are insufficient to create a secure network, and emphasizes the importance of socially driven mechanisms external to the protocol. At the same time, negative cost enables a new type of majority attack that is more likely to elude external scrutiny.

Title: Not So Robust After All: Evaluating the Robustness of Deep Neural Networks to Unseen Adversarial Attacks. (arXiv:2308.06467v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.06467
• Code URL: null
• Copy Paste: [[2308.06467]] Not So Robust After All: Evaluating the Robustness of Deep Neural Networks to Unseen Adversarial Attacks(http://arxiv.org/abs/2308.06467)
• Summary:

  Deep neural networks (DNNs) have gained prominence in various applications, such as classification, recognition, and prediction, prompting increased scrutiny of their properties. A fundamental attribute of traditional DNNs is their vulnerability to modifications in input data, which has resulted in the investigation of adversarial attacks. These attacks manipulate the data in order to mislead a DNN. This study aims to challenge the efficacy and generalization of contemporary defense mechanisms against adversarial attacks. Specifically, we explore the hypothesis proposed by Ilyas et. al, which posits that DNN image features can be either robust or non-robust, with adversarial attacks targeting the latter. This hypothesis suggests that training a DNN on a dataset consisting solely of robust features should produce a model resistant to adversarial attacks. However, our experiments demonstrate that this is not universally true. To gain further insights into our findings, we analyze the impact of adversarial attack norms on DNN representations, focusing on samples subjected to $L_2$ and $L_{\infty}$ norm attacks. Further, we employ canonical correlation analysis, visualize the representations, and calculate the mean distance between these representations and various DNN decision boundaries. Our results reveal a significant difference between $L_2$ and $L_{\infty}$ norms, which could provide insights into the potential dangers posed by $L_{\infty}$ norm attacks, previously underestimated by the research community.

robust

Title: Towards Packaging Unit Detection for Automated Palletizing Tasks. (arXiv:2308.06306v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06306
• Code URL: null
• Copy Paste: [[2308.06306]] Towards Packaging Unit Detection for Automated Palletizing Tasks(http://arxiv.org/abs/2308.06306)
• Summary:

  For various automated palletizing tasks, the detection of packaging units is a crucial step preceding the actual handling of the packaging units by an industrial robot. We propose an approach to this challenging problem that is fully trained on synthetically generated data and can be robustly applied to arbitrary real world packaging units without further training or setup effort. The proposed approach is able to handle sparse and low quality sensor data, can exploit prior knowledge if available and generalizes well to a wide range of products and application scenarios. To demonstrate the practical use of our approach, we conduct an extensive evaluation on real-world data with a wide range of different retail products. Further, we integrated our approach in a lab demonstrator and a commercial solution will be marketed through an industrial partner.

Title: U-RED: Unsupervised 3D Shape Retrieval and Deformation for Partial Point Clouds. (arXiv:2308.06383v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06383
• Code URL: https://github.com/zhangcyg/u-red
• Copy Paste: [[2308.06383]] U-RED: Unsupervised 3D Shape Retrieval and Deformation for Partial Point Clouds(http://arxiv.org/abs/2308.06383)
• Summary:

  In this paper, we propose U-RED, an Unsupervised shape REtrieval and Deformation pipeline that takes an arbitrary object observation as input, typically captured by RGB images or scans, and jointly retrieves and deforms the geometrically similar CAD models from a pre-established database to tightly match the target. Considering existing methods typically fail to handle noisy partial observations, U-RED is designed to address this issue from two aspects. First, since one partial shape may correspond to multiple potential full shapes, the retrieval method must allow such an ambiguous one-to-many relationship. Thereby U-RED learns to project all possible full shapes of a partial target onto the surface of a unit sphere. Then during inference, each sampling on the sphere will yield a feasible retrieval. Second, since real-world partial observations usually contain noticeable noise, a reliable learned metric that measures the similarity between shapes is necessary for stable retrieval. In U-RED, we design a novel point-wise residual-guided metric that allows noise-robust comparison. Extensive experiments on the synthetic datasets PartNet, ComplementMe and the real-world dataset Scan2CAD demonstrate that U-RED surpasses existing state-of-the-art approaches by 47.3%, 16.7% and 31.6% respectively under Chamfer Distance.

Title: Distributionally Robust Optimization and Invariant Representation Learning for Addressing Subgroup Underrepresentation: Mechanisms and Limitations. (arXiv:2308.06434v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06434
• Code URL: null
• Copy Paste: [[2308.06434]] Distributionally Robust Optimization and Invariant Representation Learning for Addressing Subgroup Underrepresentation: Mechanisms and Limitations(http://arxiv.org/abs/2308.06434)
• Summary:

  Spurious correlation caused by subgroup underrepresentation has received increasing attention as a source of bias that can be perpetuated by deep neural networks (DNNs). Distributionally robust optimization has shown success in addressing this bias, although the underlying working mechanism mostly relies on upweighting under-performing samples as surrogates for those underrepresented in data. At the same time, while invariant representation learning has been a powerful choice for removing nuisance-sensitive features, it has been little considered in settings where spurious correlations are caused by significant underrepresentation of subgroups. In this paper, we take the first step to better understand and improve the mechanisms for debiasing spurious correlation due to subgroup underrepresentation in medical image classification. Through a comprehensive evaluation study, we first show that 1) generalized reweighting of under-performing samples can be problematic when bias is not the only cause for poor performance, while 2) naive invariant representation learning suffers from spurious correlations itself. We then present a novel approach that leverages robust optimization to facilitate the learning of invariant representations at the presence of spurious correlations. Finetuned classifiers utilizing such representation demonstrated improved abilities to reduce subgroup performance disparity, while maintaining high average and worst-group performance.

Title: Semantic Equivariant Mixup. (arXiv:2308.06451v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06451
• Code URL: null
• Copy Paste: [[2308.06451]] Semantic Equivariant Mixup(http://arxiv.org/abs/2308.06451)
• Summary:

  Mixup is a well-established data augmentation technique, which can extend the training distribution and regularize the neural networks by creating ''mixed'' samples based on the label-equivariance assumption, i.e., a proportional mixup of the input data results in the corresponding labels being mixed in the same proportion. However, previous mixup variants may fail to exploit the label-independent information in mixed samples during training, which usually contains richer semantic information. To further release the power of mixup, we first improve the previous label-equivariance assumption by the semantic-equivariance assumption, which states that the proportional mixup of the input data should lead to the corresponding representation being mixed in the same proportion. Then a generic mixup regularization at the representation level is proposed, which can further regularize the model with the semantic information in mixed samples. At a high level, the proposed semantic equivariant mixup (sem) encourages the structure of the input data to be preserved in the representation space, i.e., the change of input will result in the obtained representation information changing in the same way. Different from previous mixup variants, which tend to over-focus on the label-related information, the proposed method aims to preserve richer semantic information in the input with semantic-equivariance assumption, thereby improving the robustness of the model against distribution shifts. We conduct extensive empirical studies and qualitative analyzes to demonstrate the effectiveness of our proposed method. The code of the manuscript is in the supplement.

Title: EgoPoser: Robust Real-Time Ego-Body Pose Estimation in Large Scenes. (arXiv:2308.06493v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06493
• Code URL: null
• Copy Paste: [[2308.06493]] EgoPoser: Robust Real-Time Ego-Body Pose Estimation in Large Scenes(http://arxiv.org/abs/2308.06493)
• Summary:

  Full-body ego-pose estimation from head and hand poses alone has become an active area of research to power articulate avatar representation on headset-based platforms. However, existing methods over-rely on the confines of the motion-capture spaces in which datasets were recorded, while simultaneously assuming continuous capture of joint motions and uniform body dimensions. In this paper, we propose EgoPoser, which overcomes these limitations by 1) rethinking the input representation for headset-based ego-pose estimation and introducing a novel motion decomposition method that predicts full-body pose independent of global positions, 2) robustly modeling body pose from intermittent hand position and orientation tracking only when inside a headset's field of view, and 3) generalizing across various body sizes for different users. Our experiments show that EgoPoser outperforms state-of-the-art methods both qualitatively and quantitatively, while maintaining a high inference speed of over 600 fps. EgoPoser establishes a robust baseline for future work, where full-body pose estimation needs no longer rely on outside-in capture and can scale to large-scene environments.

Title: Dealing with Small Annotated Datasets for Deep Learning in Medical Imaging: An Evaluation of Self-Supervised Pre-Training on CT Scans Comparing Contrastive and Masked Autoencoder Methods for Convolutional Models. (arXiv:2308.06534v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06534
• Code URL: https://github.com/wolfda95/ssl-medicalimagining-cl-mae
• Copy Paste: [[2308.06534]] Dealing with Small Annotated Datasets for Deep Learning in Medical Imaging: An Evaluation of Self-Supervised Pre-Training on CT Scans Comparing Contrastive and Masked Autoencoder Methods for Convolutional Models(http://arxiv.org/abs/2308.06534)
• Summary:

  Deep learning in medical imaging has the potential to minimize the risk of diagnostic errors, reduce radiologist workload, and accelerate diagnosis. Training such deep learning models requires large and accurate datasets, with annotations for all training samples. However, in the medical imaging domain, annotated datasets for specific tasks are often small due to the high complexity of annotations, limited access, or the rarity of diseases. To address this challenge, deep learning models can be pre-trained on large image datasets without annotations using methods from the field of self-supervised learning. After pre-training, small annotated datasets are sufficient to fine-tune the models for a specific task, the so-called ``downstream task". The most popular self-supervised pre-training approaches in medical imaging are based on contrastive learning. However, recent studies in natural image processing indicate a strong potential for masked autoencoder approaches. Our work compares state-of-the-art contrastive learning methods with the recently introduced masked autoencoder approach "SparK" for convolutional neural networks (CNNs) on medical images. Therefore we pre-train on a large unannotated CT image dataset and fine-tune on several downstream CT classification tasks. Due to the challenge of obtaining sufficient annotated training data in the medical imaging domain, it is of particular interest to evaluate how the self-supervised pre-training methods perform on small downstream datasets. By experimenting with gradually reducing the training dataset size of our downstream tasks, we find that the reduction has different effects depending on the type of pre-training chosen. The SparK pre-training method is more robust to the training dataset size than the contrastive methods. Based on our results, we propose the SparK pre-training for medical downstream tasks with small datasets.

Title: 4DRVO-Net: Deep 4D Radar-Visual Odometry Using Multi-Modal and Multi-Scale Adaptive Fusion. (arXiv:2308.06573v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06573
• Code URL: null
• Copy Paste: [[2308.06573]] 4DRVO-Net: Deep 4D Radar-Visual Odometry Using Multi-Modal and Multi-Scale Adaptive Fusion(http://arxiv.org/abs/2308.06573)
• Summary:

  Four-dimensional (4D) radar--visual odometry (4DRVO) integrates complementary information from 4D radar and cameras, making it an attractive solution for achieving accurate and robust pose estimation. However, 4DRVO may exhibit significant tracking errors owing to three main factors: 1) sparsity of 4D radar point clouds; 2) inaccurate data association and insufficient feature interaction between the 4D radar and camera; and 3) disturbances caused by dynamic objects in the environment, affecting odometry estimation. In this paper, we present 4DRVO-Net, which is a method for 4D radar--visual odometry. This method leverages the feature pyramid, pose warping, and cost volume (PWC) network architecture to progressively estimate and refine poses. Specifically, we propose a multi-scale feature extraction network called Radar-PointNet++ that fully considers rich 4D radar point information, enabling fine-grained learning for sparse 4D radar point clouds. To effectively integrate the two modalities, we design an adaptive 4D radar--camera fusion module (A-RCFM) that automatically selects image features based on 4D radar point features, facilitating multi-scale cross-modal feature interaction and adaptive multi-modal feature fusion. In addition, we introduce a velocity-guided point-confidence estimation module to measure local motion patterns, reduce the influence of dynamic objects and outliers, and provide continuous updates during pose refinement. We demonstrate the excellent performance of our method and the effectiveness of each module design on both the VoD and in-house datasets. Our method outperforms all learning-based and geometry-based methods for most sequences in the VoD dataset. Furthermore, it has exhibited promising performance that closely approaches that of the 64-line LiDAR odometry results of A-LOAM without mapping optimization.

Title: On the Interplay of Convolutional Padding and Adversarial Robustness. (arXiv:2308.06612v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06612
• Code URL: null
• Copy Paste: [[2308.06612]] On the Interplay of Convolutional Padding and Adversarial Robustness(http://arxiv.org/abs/2308.06612)
• Summary:

  It is common practice to apply padding prior to convolution operations to preserve the resolution of feature-maps in Convolutional Neural Networks (CNN). While many alternatives exist, this is often achieved by adding a border of zeros around the inputs. In this work, we show that adversarial attacks often result in perturbation anomalies at the image boundaries, which are the areas where padding is used. Consequently, we aim to provide an analysis of the interplay between padding and adversarial attacks and seek an answer to the question of how different padding modes (or their absence) affect adversarial robustness in various scenarios.

Title: ADRMX: Additive Disentanglement of Domain Features with Remix Loss. (arXiv:2308.06624v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.06624
• Code URL: null
• Copy Paste: [[2308.06624]] ADRMX: Additive Disentanglement of Domain Features with Remix Loss(http://arxiv.org/abs/2308.06624)
• Summary:

  The common assumption that train and test sets follow similar distributions is often violated in deployment settings. Given multiple source domains, domain generalization aims to create robust models capable of generalizing to new unseen domains. To this end, most of existing studies focus on extracting domain invariant features across the available source domains in order to mitigate the effects of inter-domain distributional changes. However, this approach may limit the model's generalization capacity by relying solely on finding common features among the source domains. It overlooks the potential presence of domain-specific characteristics that could be prevalent in a subset of domains, potentially containing valuable information. In this work, a novel architecture named Additive Disentanglement of Domain Features with Remix Loss (ADRMX) is presented, which addresses this limitation by incorporating domain variant features together with the domain invariant ones using an original additive disentanglement strategy. Moreover, a new data augmentation technique is introduced to further support the generalization capacity of ADRMX, where samples from different domains are mixed within the latent space. Through extensive experiments conducted on DomainBed under fair conditions, ADRMX is shown to achieve state-of-the-art performance. Code will be made available at GitHub after the revision process.

Title: Compositional Feature Augmentation for Unbiased Scene Graph Generation. (arXiv:2308.06712v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06712
• Code URL: null
• Copy Paste: [[2308.06712]] Compositional Feature Augmentation for Unbiased Scene Graph Generation(http://arxiv.org/abs/2308.06712)
• Summary:

  Scene Graph Generation (SGG) aims to detect all the visual relation triplets <sub, pred, obj> in a given image. With the emergence of various advanced techniques for better utilizing both the intrinsic and extrinsic information in each relation triplet, SGG has achieved great progress over the recent years. However, due to the ubiquitous long-tailed predicate distributions, today's SGG models are still easily biased to the head predicates. Currently, the most prevalent debiasing solutions for SGG are re-balancing methods, e.g., changing the distributions of original training samples. In this paper, we argue that all existing re-balancing strategies fail to increase the diversity of the relation triplet features of each predicate, which is critical for robust SGG. To this end, we propose a novel Compositional Feature Augmentation (CFA) strategy, which is the first unbiased SGG work to mitigate the bias issue from the perspective of increasing the diversity of triplet features. Specifically, we first decompose each relation triplet feature into two components: intrinsic feature and extrinsic feature, which correspond to the intrinsic characteristics and extrinsic contexts of a relation triplet, respectively. Then, we design two different feature augmentation modules to enrich the feature diversity of original relation triplets by replacing or mixing up either their intrinsic or extrinsic features from other samples. Due to its model-agnostic nature, CFA can be seamlessly incorporated into various SGG frameworks. Extensive ablations have shown that CFA achieves a new state-of-the-art performance on the trade-off between different metrics.

Title: Understanding the robustness difference between stochastic gradient descent and adaptive gradient methods. (arXiv:2308.06703v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.06703
• Code URL: null
• Copy Paste: [[2308.06703]] Understanding the robustness difference between stochastic gradient descent and adaptive gradient methods(http://arxiv.org/abs/2308.06703)
• Summary:

  Stochastic gradient descent (SGD) and adaptive gradient methods, such as Adam and RMSProp, have been widely used in training deep neural networks. We empirically show that while the difference between the standard generalization performance of models trained using these methods is small, those trained using SGD exhibit far greater robustness under input perturbations. Notably, our investigation demonstrates the presence of irrelevant frequencies in natural datasets, where alterations do not affect models' generalization performance. However, models trained with adaptive methods show sensitivity to these changes, suggesting that their use of irrelevant frequencies can lead to solutions sensitive to perturbations. To better understand this difference, we study the learning dynamics of gradient descent (GD) and sign gradient descent (signGD) on a synthetic dataset that mirrors natural signals. With a three-dimensional input space, the models optimized with GD and signGD have standard risks close to zero but vary in their adversarial risks. Our result shows that linear models' robustness to $\ell_2$-norm bounded changes is inversely proportional to the model parameters' weight norm: a smaller weight norm implies better robustness. In the context of deep learning, our experiments show that SGD-trained neural networks show smaller Lipschitz constants, explaining the better robustness to input perturbations than those trained with adaptive gradient methods.

biometric

steal

Title: GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher. (arXiv:2308.06463v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.06463
• Code URL: https://github.com/robustnlp/cipherchat
• Copy Paste: [[2308.06463]] GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher(http://arxiv.org/abs/2308.06463)
• Summary:

  Safety lies at the core of the development of Large Language Models (LLMs). There is ample work on aligning LLMs with human ethics and preferences, including data filtering in pretraining, supervised fine-tuning, reinforcement learning from human feedback, and red teaming, etc. In this study, we discover that chat in cipher can bypass the safety alignment techniques of LLMs, which are mainly conducted in natural languages. We propose a novel framework CipherChat to systematically examine the generalizability of safety alignment to non-natural languages -- ciphers. CipherChat enables humans to chat with LLMs through cipher prompts topped with system role descriptions and few-shot enciphered demonstrations. We use CipherChat to assess state-of-the-art LLMs, including ChatGPT and GPT-4 for different representative human ciphers across 11 safety domains in both English and Chinese. Experimental results show that certain ciphers succeed almost 100% of the time to bypass the safety alignment of GPT-4 in several safety domains, demonstrating the necessity of developing safety alignment for non-natural languages. Notably, we identify that LLMs seem to have a ''secret cipher'', and propose a novel SelfCipher that uses only role play and several demonstrations in natural language to evoke this capability. SelfCipher surprisingly outperforms existing human ciphers in almost all cases. Our code and data will be released at https://github.com/RobustNLP/CipherChat.

extraction

Title: Condition-Adaptive Graph Convolution Learning for Skeleton-Based Gait Recognition. (arXiv:2308.06707v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06707
• Code URL: https://github.com/oliverhxh/cag
• Copy Paste: [[2308.06707]] Condition-Adaptive Graph Convolution Learning for Skeleton-Based Gait Recognition(http://arxiv.org/abs/2308.06707)
• Summary:

  Graph convolutional networks have been widely applied in skeleton-based gait recognition. A key challenge in this task is to distinguish the individual walking styles of different subjects across various views. Existing state-of-the-art methods employ uniform convolutions to extract features from diverse sequences and ignore the effects of viewpoint changes. To overcome these limitations, we propose a condition-adaptive graph (CAG) convolution network that can dynamically adapt to the specific attributes of each skeleton sequence and the corresponding view angle. In contrast to using fixed weights for all joints and sequences, we introduce a joint-specific filter learning (JSFL) module in the CAG method, which produces sequence-adaptive filters at the joint level. The adaptive filters capture fine-grained patterns that are unique to each joint, enabling the extraction of diverse spatial-temporal information about body parts. Additionally, we design a view-adaptive topology learning (VATL) module that generates adaptive graph topologies. These graph topologies are used to correlate the joints adaptively according to the specific view conditions. Thus, CAG can simultaneously adjust to various walking styles and viewpoints. Experiments on the two most widely used datasets (i.e., CASIA-B and OU-MVLP) show that CAG surpasses all previous skeleton-based methods. Moreover, the recognition performance can be enhanced by simply combining CAG with appearance-based methods, demonstrating the ability of CAG to provide useful complementary information.The source code will be available at https://github.com/OliverHxh/CAG.

Title: StairNetV3: Depth-aware Stair Modeling using Deep Learning. (arXiv:2308.06715v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06715
• Code URL: null
• Copy Paste: [[2308.06715]] StairNetV3: Depth-aware Stair Modeling using Deep Learning(http://arxiv.org/abs/2308.06715)
• Summary:

  Vision-based stair perception can help autonomous mobile robots deal with the challenge of climbing stairs, especially in unfamiliar environments. To address the problem that current monocular vision methods are difficult to model stairs accurately without depth information, this paper proposes a depth-aware stair modeling method for monocular vision. Specifically, we take the extraction of stair geometric features and the prediction of depth images as joint tasks in a convolutional neural network (CNN), with the designed information propagation architecture, we can achieve effective supervision for stair geometric feature learning by depth information. In addition, to complete the stair modeling, we take the convex lines, concave lines, tread surfaces and riser surfaces as stair geometric features and apply Gaussian kernels to enable the network to predict contextual information within the stair lines. Combined with the depth information obtained by depth sensors, we propose a stair point cloud reconstruction method that can quickly get point clouds belonging to the stair step surfaces. Experiments on our dataset show that our method has a significant improvement over the previous best monocular vision method, with an intersection over union (IOU) increase of 3.4 %, and the lightweight version has a fast detection speed and can meet the requirements of most real-time applications. Our dataset is available at https://data.mendeley.com/datasets/6kffmjt7g2/1.

Title: MC-DRE: Multi-Aspect Cross Integration for Drug Event/Entity Extraction. (arXiv:2308.06546v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.06546
• Code URL: null
• Copy Paste: [[2308.06546]] MC-DRE: Multi-Aspect Cross Integration for Drug Event/Entity Extraction(http://arxiv.org/abs/2308.06546)
• Summary:

  Extracting meaningful drug-related information chunks, such as adverse drug events (ADE), is crucial for preventing morbidity and saving many lives. Most ADE are reported via an unstructured conversation with the medical context. Hence, applying a general entity recognition approach is not sufficient enough. The key is how to integrate and align multiple crucial aspects to detect drug event information, including drug event semantics, syntactic structures, and medical domain terminology. In this paper, we propose a new multi-aspect cross-integration framework for drug entity/event detection by capturing and aligning different context/language/knowledge properties from drug-related documents. We first construct multi-aspect encoders to describe semantic, syntactic, and medical document contextual information by conducting those slot tagging tasks, main drug entity/event detection, part-of-speech tagging, and general medical named entity recognition. Then, each encoder conducts cross integration and alignment with other contextual information in three ways, including the key-value cross, attention cross, and feedforward cross, so the multi-encoders are integrated in depth. Then, we perform extensive experiments on two widely used drug-related entity recognition downstream tasks, flat entity detection and discontinuous event extraction. Our model significantly outperforms all recent twelve state-of-the-art models. The implementation code will be released at~\url{https://github.com/adlnlp/mc-dre}.

Title: MT4CrossOIE: Multi-stage Tuning for Cross-lingual Open Information Extraction. (arXiv:2308.06552v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.06552
• Code URL: https://github.com/CSJianYang/Multilingual-Multimodal-NLP/tree/main/MT4CrossOIE
• Copy Paste: [[2308.06552]] MT4CrossOIE: Multi-stage Tuning for Cross-lingual Open Information Extraction(http://arxiv.org/abs/2308.06552)
• Summary:

  Cross-lingual open information extraction aims to extract structured information from raw text across multiple languages. Previous work uses a shared cross-lingual pre-trained model to handle the different languages but underuses the potential of the language-specific representation. In this paper, we propose an effective multi-stage tuning framework called MT4CrossIE, designed for enhancing cross-lingual open information extraction by injecting language-specific knowledge into the shared model. Specifically, the cross-lingual pre-trained model is first tuned in a shared semantic space (e.g., embedding matrix) in the fixed encoder and then other components are optimized in the second stage. After enough training, we freeze the pre-trained model and tune the multiple extra low-rank language-specific modules using mixture-of-LoRAs for model-based cross-lingual transfer. In addition, we leverage two-stage prompting to encourage the large language model (LLM) to annotate the multi-lingual raw data for data-based cross-lingual transfer. The model is trained with multi-lingual objectives on our proposed dataset OpenIE4++ by combing the model-based and data-based transfer techniques. Experimental results on various benchmarks emphasize the importance of aggregating multiple plug-in-and-play language-specific modules and demonstrate the effectiveness of MT4CrossIE in cross-lingual OIE\footnote{\url{https://github.com/CSJianYang/Multilingual-Multimodal-NLP}}.

membership infer

federate

Title: Performance Analysis for Resource Constrained Decentralized Federated Learning Over Wireless Networks. (arXiv:2308.06496v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.06496
• Code URL: null
• Copy Paste: [[2308.06496]] Performance Analysis for Resource Constrained Decentralized Federated Learning Over Wireless Networks(http://arxiv.org/abs/2308.06496)
• Summary:

  Federated learning (FL) can lead to significant communication overhead and reliance on a central server. To address these challenges, decentralized federated learning (DFL) has been proposed as a more resilient framework. DFL involves parameter exchange between devices through a wireless network. This study analyzes the performance of resource-constrained DFL using different communication schemes (digital and analog) over wireless networks to optimize communication efficiency. Specifically, we provide convergence bounds for both digital and analog transmission approaches, enabling analysis of the model performance trained on DFL. Furthermore, for digital transmission, we investigate and analyze resource allocation between computation and communication and convergence rates, obtaining its communication complexity and the minimum probability of correction communication required for convergence guarantee. For analog transmission, we discuss the impact of channel fading and noise on the model performance and the maximum errors accumulation with convergence guarantee over fading channels. Finally, we conduct numerical simulations to evaluate the performance and convergence rate of convolutional neural networks (CNNs) and Vision Transformer (ViT) trained in the DFL framework on fashion-MNIST and CIFAR-10 datasets. Our simulation results validate our analysis and discussion, revealing how to improve performance by optimizing system parameters under different communication conditions.

Title: SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models. (arXiv:2308.06522v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.06522
• Code URL: null
• Copy Paste: [[2308.06522]] SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models(http://arxiv.org/abs/2308.06522)
• Summary:

  Transfer learning via fine-tuning pre-trained transformer models has gained significant success in delivering state-of-the-art results across various NLP tasks. In the absence of centralized data, Federated Learning (FL) can benefit from distributed and private data of the FL edge clients for fine-tuning. However, due to the limited communication, computation, and storage capabilities of edge devices and the huge sizes of popular transformer models, efficient fine-tuning is crucial to make federated training feasible. This work explores the opportunities and challenges associated with applying parameter efficient fine-tuning (PEFT) methods in different FL settings for language tasks. Specifically, our investigation reveals that as the data across users becomes more diverse, the gap between fully fine-tuning the model and employing PEFT methods widens. To bridge this performance gap, we propose a method called SLoRA, which overcomes the key limitations of LoRA in high heterogeneous data scenarios through a novel data-driven initialization technique. Our experimental results demonstrate that SLoRA achieves performance comparable to full fine-tuning, with significant sparse updates with approximately $\sim 1\%$ density while reducing training time by up to $90\%$.

fair

interpretability

explainability

watermark

diffusion

Title: ModelScope Text-to-Video Technical Report. (arXiv:2308.06571v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06571
• Code URL: null
• Copy Paste: [[2308.06571]] ModelScope Text-to-Video Technical Report(http://arxiv.org/abs/2308.06571)
• Summary:

  This paper introduces ModelScopeT2V, a text-to-video synthesis model that evolves from a text-to-image synthesis model (i.e., Stable Diffusion). ModelScopeT2V incorporates spatio-temporal blocks to ensure consistent frame generation and smooth movement transitions. The model could adapt to varying frame numbers during training and inference, rendering it suitable for both image-text and video-text datasets. ModelScopeT2V brings together three components (i.e., VQGAN, a text encoder, and a denoising UNet), totally comprising 1.7 billion parameters, in which 0.5 billion parameters are dedicated to temporal capabilities. The model demonstrates superior performance over state-of-the-art methods across three evaluation metrics. The code and an online demo are available at \url{https://modelscope.cn/models/damo/text-to-video-synthesis/summary}.

Title: LAW-Diffusion: Complex Scene Generation by Diffusion with Layouts. (arXiv:2308.06713v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06713
• Code URL: null
• Copy Paste: [[2308.06713]] LAW-Diffusion: Complex Scene Generation by Diffusion with Layouts(http://arxiv.org/abs/2308.06713)
• Summary:

  Thanks to the rapid development of diffusion models, unprecedented progress has been witnessed in image synthesis. Prior works mostly rely on pre-trained linguistic models, but a text is often too abstract to properly specify all the spatial properties of an image, e.g., the layout configuration of a scene, leading to the sub-optimal results of complex scene generation. In this paper, we achieve accurate complex scene generation by proposing a semantically controllable Layout-AWare diffusion model, termed LAW-Diffusion. Distinct from the previous Layout-to-Image generation (L2I) methods that only explore category-aware relationships, LAW-Diffusion introduces a spatial dependency parser to encode the location-aware semantic coherence across objects as a layout embedding and produces a scene with perceptually harmonious object styles and contextual relations. To be specific, we delicately instantiate each object's regional semantics as an object region map and leverage a location-aware cross-object attention module to capture the spatial dependencies among those disentangled representations. We further propose an adaptive guidance schedule for our layout guidance to mitigate the trade-off between the regional semantic alignment and the texture fidelity of generated objects. Moreover, LAW-Diffusion allows for instance reconfiguration while maintaining the other regions in a synthesized image by introducing a layout-aware latent grafting mechanism to recompose its local regional semantics. To better verify the plausibility of generated scenes, we propose a new evaluation metric for the L2I task, dubbed Scene Relation Score (SRS) to measure how the images preserve the rational and harmonious relations among contextual objects. Comprehensive experiments demonstrate that our LAW-Diffusion yields the state-of-the-art generative performance, especially with coherent object relations.

Title: Size Lowerbounds for Deep Operator Networks. (arXiv:2308.06338v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.06338
• Code URL: null
• Copy Paste: [[2308.06338]] Size Lowerbounds for Deep Operator Networks(http://arxiv.org/abs/2308.06338)
• Summary:

  Deep Operator Networks are an increasingly popular paradigm for solving regression in infinite dimensions and hence solve families of PDEs in one shot. In this work, we aim to establish a first-of-its-kind data-dependent lowerbound on the size of DeepONets required for them to be able to reduce empirical error on noisy data. In particular, we show that for low training errors to be obtained on $n$ data points it is necessary that the common output dimension of the branch and the trunk net be scaling as $\Omega \left ( {\sqrt{n}} \right )$. This inspires our experiments with DeepONets solving the advection-diffusion-reaction PDE, where we demonstrate the possibility that at a fixed model size, to leverage increase in this common output dimension and get monotonic lowering of training error, the size of the training data might necessarily need to scale quadratically with it.

Title: Mirror Diffusion Models. (arXiv:2308.06342v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.06342
• Code URL: null
• Copy Paste: [[2308.06342]] Mirror Diffusion Models(http://arxiv.org/abs/2308.06342)
• Summary:

  Diffusion models have successfully been applied to generative tasks in various continuous domains. However, applying diffusion to discrete categorical data remains a non-trivial task. Moreover, generation in continuous domains often requires clipping in practice, which motivates the need for a theoretical framework for adapting diffusion to constrained domains. Inspired by the mirror Langevin algorithm for the constrained sampling problem, in this theoretical report we propose Mirror Diffusion Models (MDMs). We demonstrate MDMs in the context of simplex diffusion and propose natural extensions to popular domains such as image and text generation.

Title: EquiDiff: A Conditional Equivariant Diffusion Model For Trajectory Prediction. (arXiv:2308.06564v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.06564
• Code URL: null
• Copy Paste: [[2308.06564]] EquiDiff: A Conditional Equivariant Diffusion Model For Trajectory Prediction(http://arxiv.org/abs/2308.06564)
• Summary:

  Accurate trajectory prediction is crucial for the safe and efficient operation of autonomous vehicles. The growing popularity of deep learning has led to the development of numerous methods for trajectory prediction. While deterministic deep learning models have been widely used, deep generative models have gained popularity as they learn data distributions from training data and account for trajectory uncertainties. In this study, we propose EquiDiff, a deep generative model for predicting future vehicle trajectories. EquiDiff is based on the conditional diffusion model, which generates future trajectories by incorporating historical information and random Gaussian noise. The backbone model of EquiDiff is an SO(2)-equivariant transformer that fully utilizes the geometric properties of location coordinates. In addition, we employ Recurrent Neural Networks and Graph Attention Networks to extract social interactions from historical trajectories. To evaluate the performance of EquiDiff, we conduct extensive experiments on the NGSIM dataset. Our results demonstrate that EquiDiff outperforms other baseline models in short-term prediction, but has slightly higher errors for long-term prediction. Furthermore, we conduct an ablation study to investigate the contribution of each component of EquiDiff to the prediction accuracy. Additionally, we present a visualization of the generation process of our diffusion model, providing insights into the uncertainty of the prediction.

Title: Accelerating Diffusion-based Combinatorial Optimization Solvers by Progressive Distillation. (arXiv:2308.06644v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.06644
• Code URL: null
• Copy Paste: [[2308.06644]] Accelerating Diffusion-based Combinatorial Optimization Solvers by Progressive Distillation(http://arxiv.org/abs/2308.06644)
• Summary:

  Graph-based diffusion models have shown promising results in terms of generating high-quality solutions to NP-complete (NPC) combinatorial optimization (CO) problems. However, those models are often inefficient in inference, due to the iterative evaluation nature of the denoising diffusion process. This paper proposes to use progressive distillation to speed up the inference by taking fewer steps (e.g., forecasting two steps ahead within a single step) during the denoising process. Our experimental results show that the progressively distilled model can perform inference 16 times faster with only 0.019% degradation in performance on the TSP-50 dataset.

Title: Law of Balance and Stationary Distribution of Stochastic Gradient Descent. (arXiv:2308.06671v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.06671
• Code URL: null
• Copy Paste: [[2308.06671]] Law of Balance and Stationary Distribution of Stochastic Gradient Descent(http://arxiv.org/abs/2308.06671)
• Summary:

  The stochastic gradient descent (SGD) algorithm is the algorithm we use to train neural networks. However, it remains poorly understood how the SGD navigates the highly nonlinear and degenerate loss landscape of a neural network. In this work, we prove that the minibatch noise of SGD regularizes the solution towards a balanced solution whenever the loss function contains a rescaling symmetry. Because the difference between a simple diffusion process and SGD dynamics is the most significant when symmetries are present, our theory implies that the loss function symmetries constitute an essential probe of how SGD works. We then apply this result to derive the stationary distribution of stochastic gradient flow for a diagonal linear network with arbitrary depth and width. The stationary distribution exhibits complicated nonlinear phenomena such as phase transitions, broken ergodicity, and fluctuation inversion. These phenomena are shown to exist uniquely in deep networks, implying a fundamental difference between deep and shallow models.

Title: Generating observation guided ensembles for data assimilation with denoising diffusion probabilistic model. (arXiv:2308.06708v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.06708
• Code URL: https://github.com/yasahi-hpc/generative-enkf
• Copy Paste: [[2308.06708]] Generating observation guided ensembles for data assimilation with denoising diffusion probabilistic model(http://arxiv.org/abs/2308.06708)
• Summary:

  This paper presents an ensemble data assimilation method using the pseudo ensembles generated by denoising diffusion probabilistic model. Since the model is trained against noisy and sparse observation data, this model can produce divergent ensembles close to observations. Thanks to the variance in generated ensembles, our proposed method displays better performance than the well-established ensemble data assimilation method when the simulation model is imperfect.

noise learning

data-free

transformer

Title: Revisiting Vision Transformer from the View of Path Ensemble. (arXiv:2308.06548v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06548
• Code URL: null
• Copy Paste: [[2308.06548]] Revisiting Vision Transformer from the View of Path Ensemble(http://arxiv.org/abs/2308.06548)
• Summary:

  Vision Transformers (ViTs) are normally regarded as a stack of transformer layers. In this work, we propose a novel view of ViTs showing that they can be seen as ensemble networks containing multiple parallel paths with different lengths. Specifically, we equivalently transform the traditional cascade of multi-head self-attention (MSA) and feed-forward network (FFN) into three parallel paths in each transformer layer. Then, we utilize the identity connection in our new transformer form and further transform the ViT into an explicit multi-path ensemble network. From the new perspective, these paths perform two functions: the first is to provide the feature for the classifier directly, and the second is to provide the lower-level feature representation for subsequent longer paths. We investigate the influence of each path for the final prediction and discover that some paths even pull down the performance. Therefore, we propose the path pruning and EnsembleScale skills for improvement, which cut out the underperforming paths and re-weight the ensemble components, respectively, to optimize the path combination and make the short paths focus on providing high-quality representation for subsequent paths. We also demonstrate that our path combination strategies can help ViTs go deeper and act as high-pass filters to filter out partial low-frequency signals. To further enhance the representation of paths served for subsequent paths, self-distillation is applied to transfer knowledge from the long paths to the short paths. This work calls for more future research to explain and design ViTs from new perspectives.

Title: 3DMOTFormer: Graph Transformer for Online 3D Multi-Object Tracking. (arXiv:2308.06635v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06635
• Code URL: https://github.com/dsx0511/3dmotformer
• Copy Paste: [[2308.06635]] 3DMOTFormer: Graph Transformer for Online 3D Multi-Object Tracking(http://arxiv.org/abs/2308.06635)
• Summary:

  Tracking 3D objects accurately and consistently is crucial for autonomous vehicles, enabling more reliable downstream tasks such as trajectory prediction and motion planning. Based on the substantial progress in object detection in recent years, the tracking-by-detection paradigm has become a popular choice due to its simplicity and efficiency. State-of-the-art 3D multi-object tracking (MOT) approaches typically rely on non-learned model-based algorithms such as Kalman Filter but require many manually tuned parameters. On the other hand, learning-based approaches face the problem of adapting the training to the online setting, leading to inevitable distribution mismatch between training and inference as well as suboptimal performance. In this work, we propose 3DMOTFormer, a learned geometry-based 3D MOT framework building upon the transformer architecture. We use an Edge-Augmented Graph Transformer to reason on the track-detection bipartite graph frame-by-frame and conduct data association via edge classification. To reduce the distribution mismatch between training and inference, we propose a novel online training strategy with an autoregressive and recurrent forward pass as well as sequential batch optimization. Using CenterPoint detections, our approach achieves 71.2% and 68.2% AMOTA on the nuScenes validation and test split, respectively. In addition, a trained 3DMOTFormer model generalizes well across different object detectors. Code is available at: https://github.com/dsx0511/3DMOTFormer.

Title: Isomer: Isomerous Transformer for Zero-shot Video Object Segmentation. (arXiv:2308.06693v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06693
• Code URL: https://github.com/dlut-yyc/isomer
• Copy Paste: [[2308.06693]] Isomer: Isomerous Transformer for Zero-shot Video Object Segmentation(http://arxiv.org/abs/2308.06693)
• Summary:

  Recent leading zero-shot video object segmentation (ZVOS) works devote to integrating appearance and motion information by elaborately designing feature fusion modules and identically applying them in multiple feature stages. Our preliminary experiments show that with the strong long-range dependency modeling capacity of Transformer, simply concatenating the two modality features and feeding them to vanilla Transformers for feature fusion can distinctly benefit the performance but at a cost of heavy computation. Through further empirical analysis, we find that attention dependencies learned in Transformer in different stages exhibit completely different properties: global query-independent dependency in the low-level stages and semantic-specific dependency in the high-level stages. Motivated by the observations, we propose two Transformer variants: i) Context-Sharing Transformer (CST) that learns the global-shared contextual information within image frames with a lightweight computation. ii) Semantic Gathering-Scattering Transformer (SGST) that models the semantic correlation separately for the foreground and background and reduces the computation cost with a soft token merging mechanism. We apply CST and SGST for low-level and high-level feature fusions, respectively, formulating a level-isomerous Transformer framework for ZVOS task. Compared with the baseline that uses vanilla Transformers for multi-stage fusion, ours significantly increase the speed by 13 times and achieves new state-of-the-art ZVOS performance. Code is available at https://github.com/DLUT-yyc/Isomer.

generative

Title: Camouflaged Image Synthesis Is All You Need to Boost Camouflaged Detection. (arXiv:2308.06701v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06701
• Code URL: null
• Copy Paste: [[2308.06701]] Camouflaged Image Synthesis Is All You Need to Boost Camouflaged Detection(http://arxiv.org/abs/2308.06701)
• Summary:

  Camouflaged objects that blend into natural scenes pose significant challenges for deep-learning models to detect and synthesize. While camouflaged object detection is a crucial task in computer vision with diverse real-world applications, this research topic has been constrained by limited data availability. We propose a framework for synthesizing camouflage data to enhance the detection of camouflaged objects in natural scenes. Our approach employs a generative model to produce realistic camouflage images, which can be used to train existing object detection models. Specifically, we use a camouflage environment generator supervised by a camouflage distribution classifier to synthesize the camouflage images, which are then fed into our generator to expand the dataset. Our framework outperforms the current state-of-the-art method on three datasets (COD10k, CAMO, and CHAMELEON), demonstrating its effectiveness in improving camouflaged object detection. This approach can serve as a plug-and-play data generation and augmentation module for existing camouflaged object detection tasks and provides a novel way to introduce more diversity and distributions into current camouflage datasets.

large language model

Title: Large Language Models to Identify Social Determinants of Health in Electronic Health Records. (arXiv:2308.06354v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.06354
• Code URL: null
• Copy Paste: [[2308.06354]] Large Language Models to Identify Social Determinants of Health in Electronic Health Records(http://arxiv.org/abs/2308.06354)
• Summary:

  Social determinants of health (SDoH) have an important impact on patient outcomes but are incompletely collected from the electronic health records (EHR). This study researched the ability of large language models to extract SDoH from free text in EHRs, where they are most commonly documented, and explored the role of synthetic clinical text for improving the extraction of these scarcely documented, yet extremely valuable, clinical data. 800 patient notes were annotated for SDoH categories, and several transformer-based models were evaluated. The study also experimented with synthetic data generation and assessed for algorithmic bias. Our best-performing models were fine-tuned Flan-T5 XL (macro-F1 0.71) for any SDoH, and Flan-T5 XXL (macro-F1 0.70). The benefit of augmenting fine-tuning with synthetic data varied across model architecture and size, with smaller Flan-T5 models (base and large) showing the greatest improvements in performance (delta F1 +0.12 to +0.23). Model performance was similar on the in-hospital system dataset but worse on the MIMIC-III dataset. Our best-performing fine-tuned models outperformed zero- and few-shot performance of ChatGPT-family models for both tasks. These fine-tuned models were less likely than ChatGPT to change their prediction when race/ethnicity and gender descriptors were added to the text, suggesting less algorithmic bias (p<0.05). At the patient-level, our models identified 93.8% of patients with adverse SDoH, while ICD-10 codes captured 2.0%. Our method can effectively extracted SDoH information from clinic notes, performing better compare to GPT zero- and few-shot settings. These models could enhance real-world evidence on SDoH and aid in identifying patients needing social support.

Title: Dynamic Planning with a LLM. (arXiv:2308.06391v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.06391
• Code URL: null
• Copy Paste: [[2308.06391]] Dynamic Planning with a LLM(http://arxiv.org/abs/2308.06391)
• Summary:

  While Large Language Models (LLMs) can solve many NLP tasks in zero-shot settings, applications involving embodied agents remain problematic. In particular, complex plans that require multi-step reasoning become difficult and too costly as the context window grows. Planning requires understanding the likely effects of one's actions and identifying whether the current environment satisfies the goal state. While symbolic planners find optimal solutions quickly, they require a complete and accurate representation of the planning problem, severely limiting their use in practical scenarios. In contrast, modern LLMs cope with noisy observations and high levels of uncertainty when reasoning about a task. Our work presents LLM Dynamic Planner (LLM-DP): a neuro-symbolic framework where an LLM works hand-in-hand with a traditional planner to solve an embodied task. Given action-descriptions, LLM-DP solves Alfworld faster and more efficiently than a naive LLM ReAct baseline.

Title: Three Ways of Using Large Language Models to Evaluate Chat. (arXiv:2308.06502v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.06502
• Code URL: https://github.com/oplatek/chateval-llm
• Copy Paste: [[2308.06502]] Three Ways of Using Large Language Models to Evaluate Chat(http://arxiv.org/abs/2308.06502)
• Summary:

  This paper describes the systems submitted by team6 for ChatEval, the DSTC 11 Track 4 competition. We present three different approaches to predicting turn-level qualities of chatbot responses based on large language models (LLMs). We report improvement over the baseline using dynamic few-shot examples from a vector store for the prompts for ChatGPT. We also analyze the performance of the other two approaches and report needed improvements for future work. We developed the three systems over just two weeks, showing the potential of LLMs for this task. An ablation study conducted after the challenge deadline shows that the new Llama 2 models are closing the performance gap between ChatGPT and open-source LLMs. However, we find that the Llama 2 models do not benefit from few-shot examples in the same way as ChatGPT.

Title: AutoConv: Automatically Generating Information-seeking Conversations with Large Language Models. (arXiv:2308.06507v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.06507
• Code URL: null
• Copy Paste: [[2308.06507]] AutoConv: Automatically Generating Information-seeking Conversations with Large Language Models(http://arxiv.org/abs/2308.06507)
• Summary:

  Information-seeking conversation, which aims to help users gather information through conversation, has achieved great progress in recent years. However, the research is still stymied by the scarcity of training data. To alleviate this problem, we propose AutoConv for synthetic conversation generation, which takes advantage of the few-shot learning ability and generation capacity of large language models (LLM). Specifically, we formulate the conversation generation problem as a language modeling task, then finetune an LLM with a few human conversations to capture the characteristics of the information-seeking process and use it for generating synthetic conversations with high quality. Experimental results on two frequently-used datasets verify that AutoConv has substantial improvements over strong baselines and alleviates the dependence on human annotation. In addition, we also provide several analysis studies to promote future research.

Title: Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation. (arXiv:2308.06610v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.06610
• Code URL: https://github.com/ambroser53/bio-sieve
• Copy Paste: [[2308.06610]] Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation(http://arxiv.org/abs/2308.06610)
• Summary:

  Medical systematic reviews can be very costly and resource intensive. We explore how Large Language Models (LLMs) can support and be trained to perform literature screening when provided with a detailed set of selection criteria. Specifically, we instruction tune LLaMA and Guanaco models to perform abstract screening for medical systematic reviews. Our best model, Bio-SIEVE, outperforms both ChatGPT and trained traditional approaches, and generalises better across medical domains. However, there remains the challenge of adapting the model to safety-first scenarios. We also explore the impact of multi-task training with Bio-SIEVE-Multi, including tasks such as PICO extraction and exclusion reasoning, but find that it is unable to match single-task Bio-SIEVE's performance. We see Bio-SIEVE as an important step towards specialising LLMs for the biomedical systematic review process and explore its future developmental opportunities. We release our models, code and a list of DOIs to reconstruct our dataset for reproducibility.

segmentation

Title: Defensive Perception: Estimation and Monitoring of Neural Network Performance under Deployment. (arXiv:2308.06299v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06299
• Code URL: null
• Copy Paste: [[2308.06299]] Defensive Perception: Estimation and Monitoring of Neural Network Performance under Deployment(http://arxiv.org/abs/2308.06299)
• Summary:

  In this paper, we propose a method for addressing the issue of unnoticed catastrophic deployment and domain shift in neural networks for semantic segmentation in autonomous driving. Our approach is based on the idea that deep learning-based perception for autonomous driving is uncertain and best represented as a probability distribution. As autonomous vehicles' safety is paramount, it is crucial for perception systems to recognize when the vehicle is leaving its operational design domain, anticipate hazardous uncertainty, and reduce the performance of the perception system. To address this, we propose to encapsulate the neural network under deployment within an uncertainty estimation envelope that is based on the epistemic uncertainty estimation through the Monte Carlo Dropout approach. This approach does not require modification of the deployed neural network and guarantees expected model performance. Our defensive perception envelope has the capability to estimate a neural network's performance, enabling monitoring and notification of entering domains of reduced neural network performance under deployment. Furthermore, our envelope is extended by novel methods to improve the application in deployment settings, including reducing compute expenses and confining estimation noise. Finally, we demonstrate the applicability of our method for multiple different potential deployment shifts relevant to autonomous driving, such as transitions into the night, rainy, or snowy domain. Overall, our approach shows great potential for application in deployment settings and enables operational design domain recognition via uncertainty, which allows for defensive perception, safe state triggers, warning notifications, and feedback for testing or development and adaptation of the perception stack.

Title: R2S100K: Road-Region Segmentation Dataset For Semi-Supervised Autonomous Driving in the Wild. (arXiv:2308.06393v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06393
• Code URL: null
• Copy Paste: [[2308.06393]] R2S100K: Road-Region Segmentation Dataset For Semi-Supervised Autonomous Driving in the Wild(http://arxiv.org/abs/2308.06393)
• Summary:

  Semantic understanding of roadways is a key enabling factor for safe autonomous driving. However, existing autonomous driving datasets provide well-structured urban roads while ignoring unstructured roadways containing distress, potholes, water puddles, and various kinds of road patches i.e., earthen, gravel etc. To this end, we introduce Road Region Segmentation dataset (R2S100K) -- a large-scale dataset and benchmark for training and evaluation of road segmentation in aforementioned challenging unstructured roadways. R2S100K comprises 100K images extracted from a large and diverse set of video sequences covering more than 1000 KM of roadways. Out of these 100K privacy respecting images, 14,000 images have fine pixel-labeling of road regions, with 86,000 unlabeled images that can be leveraged through semi-supervised learning methods. Alongside, we present an Efficient Data Sampling (EDS) based self-training framework to improve learning by leveraging unlabeled data. Our experimental results demonstrate that the proposed method significantly improves learning methods in generalizability and reduces the labeling cost for semantic segmentation tasks. Our benchmark will be publicly available to facilitate future research at https://r2s100k.github.io/.

Title: TongueSAM: An Universal Tongue Segmentation Model Based on SAM with Zero-Shot. (arXiv:2308.06444v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06444
• Code URL: https://github.com/cshan-github/tonguesam
• Copy Paste: [[2308.06444]] TongueSAM: An Universal Tongue Segmentation Model Based on SAM with Zero-Shot(http://arxiv.org/abs/2308.06444)
• Summary:

  Tongue segmentation serves as the primary step in automated TCM tongue diagnosis, which plays a significant role in the diagnostic results. Currently, numerous deep learning based methods have achieved promising results. However, most of these methods exhibit mediocre performance on tongues different from the training set. To address this issue, this paper proposes a universal tongue segmentation model named TongueSAM based on SAM (Segment Anything Model). SAM is a large-scale pretrained interactive segmentation model known for its powerful zero-shot generalization capability. Applying SAM to tongue segmentation enables the segmentation of various types of tongue images with zero-shot. In this study, a Prompt Generator based on object detection is integrated into SAM to enable an end-to-end automated tongue segmentation method. Experiments demonstrate that TongueSAM achieves exceptional performance across various of tongue segmentation datasets, particularly under zero-shot. TongueSAM can be directly applied to other datasets without fine-tuning. As far as we know, this is the first application of large-scale pretrained model for tongue segmentation. The project and pretrained model of TongueSAM be publiced in :https://github.com/cshan-github/TongueSAM.

Title: Tiny and Efficient Model for the Edge Detection Generalization. (arXiv:2308.06468v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06468
• Code URL: https://github.com/xavysp/teed
• Copy Paste: [[2308.06468]] Tiny and Efficient Model for the Edge Detection Generalization(http://arxiv.org/abs/2308.06468)
• Summary:

  Most high-level computer vision tasks rely on low-level image operations as their initial processes. Operations such as edge detection, image enhancement, and super-resolution, provide the foundations for higher level image analysis. In this work we address the edge detection considering three main objectives: simplicity, efficiency, and generalization since current state-of-the-art (SOTA) edge detection models are increased in complexity for better accuracy. To achieve this, we present Tiny and Efficient Edge Detector (TEED), a light convolutional neural network with only $58K$ parameters, less than $0.2$% of the state-of-the-art models. Training on the BIPED dataset takes $less than 30 minutes$, with each epoch requiring $less than 5 minutes$. Our proposed model is easy to train and it quickly converges within very first few epochs, while the predicted edge-maps are crisp and of high quality. Additionally, we propose a new dataset to test the generalization of edge detection, which comprises samples from popular images used in edge detection and image segmentation. The source code is available in https://github.com/xavysp/TEED.

Title: Seed Feature Maps-based CNN Models for LEO Satellite Remote Sensing Services. (arXiv:2308.06515v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06515
• Code URL: null
• Copy Paste: [[2308.06515]] Seed Feature Maps-based CNN Models for LEO Satellite Remote Sensing Services(http://arxiv.org/abs/2308.06515)
• Summary:

  Deploying high-performance convolutional neural network (CNN) models on low-earth orbit (LEO) satellites for rapid remote sensing image processing has attracted significant interest from industry and academia. However, the limited resources available on LEO satellites contrast with the demands of resource-intensive CNN models, necessitating the adoption of ground-station server assistance for training and updating these models. Existing approaches often require large floating-point operations (FLOPs) and substantial model parameter transmissions, presenting considerable challenges. To address these issues, this paper introduces a ground-station server-assisted framework. With the proposed framework, each layer of the CNN model contains only one learnable feature map (called the seed feature map) from which other feature maps are generated based on specific rules. The hyperparameters of these rules are randomly generated instead of being trained, thus enabling the generation of multiple feature maps from the seed feature map and significantly reducing FLOPs. Furthermore, since the random hyperparameters can be saved using a few random seeds, the ground station server assistance can be facilitated in updating the CNN model deployed on the LEO satellite. Experimental results on the ISPRS Vaihingen, ISPRS Potsdam, UAVid, and LoveDA datasets for semantic segmentation services demonstrate that the proposed framework outperforms existing state-of-the-art approaches. In particular, the SineFM-based model achieves a higher mIoU than the UNetFormer on the UAVid dataset, with 3.3x fewer parameters and 2.2x fewer FLOPs.

Title: BEV-DG: Cross-Modal Learning under Bird's-Eye View for Domain Generalization of 3D Semantic Segmentation. (arXiv:2308.06530v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06530
• Code URL: null
• Copy Paste: [[2308.06530]] BEV-DG: Cross-Modal Learning under Bird's-Eye View for Domain Generalization of 3D Semantic Segmentation(http://arxiv.org/abs/2308.06530)
• Summary:

  Cross-modal Unsupervised Domain Adaptation (UDA) aims to exploit the complementarity of 2D-3D data to overcome the lack of annotation in a new domain. However, UDA methods rely on access to the target domain during training, meaning the trained model only works in a specific target domain. In light of this, we propose cross-modal learning under bird's-eye view for Domain Generalization (DG) of 3D semantic segmentation, called BEV-DG. DG is more challenging because the model cannot access the target domain during training, meaning it needs to rely on cross-modal learning to alleviate the domain gap. Since 3D semantic segmentation requires the classification of each point, existing cross-modal learning is directly conducted point-to-point, which is sensitive to the misalignment in projections between pixels and points. To this end, our approach aims to optimize domain-irrelevant representation modeling with the aid of cross-modal learning under bird's-eye view. We propose BEV-based Area-to-area Fusion (BAF) to conduct cross-modal learning under bird's-eye view, which has a higher fault tolerance for point-level misalignment. Furthermore, to model domain-irrelevant representations, we propose BEV-driven Domain Contrastive Learning (BDCL) with the help of cross-modal learning under bird's-eye view. We design three domain generalization settings based on three 3D datasets, and BEV-DG significantly outperforms state-of-the-art competitors with tremendous margins in all settings.

Title: SegPrompt: Boosting Open-world Segmentation via Category-level Prompt Learning. (arXiv:2308.06531v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06531
• Code URL: https://github.com/aim-uofa/segprompt
• Copy Paste: [[2308.06531]] SegPrompt: Boosting Open-world Segmentation via Category-level Prompt Learning(http://arxiv.org/abs/2308.06531)
• Summary:

  Current closed-set instance segmentation models rely on pre-defined class labels for each mask during training and evaluation, largely limiting their ability to detect novel objects. Open-world instance segmentation (OWIS) models address this challenge by detecting unknown objects in a class-agnostic manner. However, previous OWIS approaches completely erase category information during training to keep the model's ability to generalize to unknown objects. In this work, we propose a novel training mechanism termed SegPrompt that uses category information to improve the model's class-agnostic segmentation ability for both known and unknown categories. In addition, the previous OWIS training setting exposes the unknown classes to the training set and brings information leakage, which is unreasonable in the real world. Therefore, we provide a new open-world benchmark closer to a real-world scenario by dividing the dataset classes into known-seen-unseen parts. For the first time, we focus on the model's ability to discover objects that never appear in the training set images.

  Experiments show that SegPrompt can improve the overall and unseen detection performance by 5.6% and 6.1% in AR on our new benchmark without affecting the inference efficiency. We further demonstrate the effectiveness of our method on existing cross-dataset transfer and strongly supervised settings, leading to 5.5% and 12.3% relative improvement.

Title: LadleNet: Translating Thermal Infrared Images to Visible Light Images Using A Scalable Two-stage U-Net. (arXiv:2308.06603v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06603
• Code URL: https://github.com/ach-1914/ladlenet
• Copy Paste: [[2308.06603]] LadleNet: Translating Thermal Infrared Images to Visible Light Images Using A Scalable Two-stage U-Net(http://arxiv.org/abs/2308.06603)
• Summary:

  The translation of thermal infrared (TIR) images to visible light (VI) images presents a challenging task with potential applications spanning various domains such as TIR-VI image registration and fusion. Leveraging supplementary information derived from TIR image conversions can significantly enhance model performance and generalization across these applications. However, prevailing issues within this field include suboptimal image fidelity and limited model scalability. In this paper, we introduce an algorithm, LadleNet, based on the U-Net architecture. LadleNet employs a two-stage U-Net concatenation structure, augmented with skip connections and refined feature aggregation techniques, resulting in a substantial enhancement in model performance. Comprising 'Handle' and 'Bowl' modules, LadleNet's Handle module facilitates the construction of an abstract semantic space, while the Bowl module decodes this semantic space to yield mapped VI images. The Handle module exhibits extensibility by allowing the substitution of its network architecture with semantic segmentation networks, thereby establishing more abstract semantic spaces to bolster model performance. Consequently, we propose LadleNet+, which replaces LadleNet's Handle module with the pre-trained DeepLabv3+ network, thereby endowing the model with enhanced semantic space construction capabilities. The proposed method is evaluated and tested on the KAIST dataset, accompanied by quantitative and qualitative analyses. Compared to existing methodologies, our approach achieves state-of-the-art performance in terms of image clarity and perceptual quality. The source code will be made available at https://github.com/Ach-1914/LadleNet/tree/main/.

Title: Unsupervised Adaptation of Polyp Segmentation Models via Coarse-to-Fine Self-Supervision. (arXiv:2308.06665v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.06665
• Code URL: null
• Copy Paste: [[2308.06665]] Unsupervised Adaptation of Polyp Segmentation Models via Coarse-to-Fine Self-Supervision(http://arxiv.org/abs/2308.06665)
• Summary:

  Unsupervised Domain Adaptation~(UDA) has attracted a surge of interest over the past decade but is difficult to be used in real-world applications. Considering the privacy-preservation issues and security concerns, in this work, we study a practical problem of Source-Free Domain Adaptation (SFDA), which eliminates the reliance on annotated source data. Current SFDA methods focus on extracting domain knowledge from the source-trained model but neglects the intrinsic structure of the target domain. Moreover, they typically utilize pseudo labels for self-training in the target domain, but suffer from the notorious error accumulation problem. To address these issues, we propose a new SFDA framework, called Region-to-Pixel Adaptation Network~(RPANet), which learns the region-level and pixel-level discriminative representations through coarse-to-fine self-supervision. The proposed RPANet consists of two modules, Foreground-aware Contrastive Learning (FCL) and Confidence-Calibrated Pseudo-Labeling (CCPL), which explicitly address the key challenges of ``how to distinguish'' and ``how to refine''. To be specific, FCL introduces a supervised contrastive learning paradigm in the region level to contrast different region centroids across different target images, which efficiently involves all pseudo labels while robust to noisy samples. CCPL designs a novel fusion strategy to reduce the overconfidence problem of pseudo labels by fusing two different target predictions without introducing any additional network modules. Extensive experiments on three cross-domain polyp segmentation tasks reveal that RPANet significantly outperforms state-of-the-art SFDA and UDA methods without access to source data, revealing the potential of SFDA in medical applications.