secure

Title: Anatomy of a High-Profile Data Breach: Dissecting the Aftermath of a Crypto-Wallet Case. (arXiv:2308.00375v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.00375
• Code URL: null
• Copy Paste: [[2308.00375] Anatomy of a High-Profile Data Breach: Dissecting the Aftermath of a Crypto-Wallet Case](http://arxiv.org/abs/2308.00375) #secure
• Summary:

  Media reports show an alarming increase of data breaches at providers of cybersecurity products and services. Since the exposed records may reveal security-relevant data, such incidents cause undue burden and create the risk of re-victimization to individuals whose personal data gets exposed. In pursuit of examining a broad spectrum of the downstream effects on victims, we surveyed 104 persons who purchased specialized devices for the secure storage of crypto-assets and later fell victim to a breach of customer data. Our case study reveals common nuisances (i.e., spam, scams, phishing e-mails) as well as previously unseen attack vectors (e.g., involving tampered devices), which are possibly tied to the breach. A few victims report losses of digital assets as a form of the harm. We find that our participants exhibit heightened safety concerns, appear skeptical about litigation efforts, and demonstrate the ability to differentiate between the quality of the security product and the circumstances of the breach. We derive implications for the cybersecurity industry at large, and point out methodological challenges in data breach research.

Title: A First Look at Digital Rights Management Systems for Secure Mobile Content Delivery. (arXiv:2308.00437v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.00437
• Code URL: null
• Copy Paste: [[2308.00437] A First Look at Digital Rights Management Systems for Secure Mobile Content Delivery](http://arxiv.org/abs/2308.00437) #secure
• Summary:

  Digital rights management (DRM) solutions aim to prevent the copying or distribution of copyrighted material. On mobile devices, a variety of DRM technologies have become widely deployed. However, a detailed security study comparing their internal workings, and their strengths and weaknesses, remains missing in the existing literature. In this paper, we present the first detailed security analysis of mobile DRM systems, addressing the modern paradigm of cloud-based content delivery followed by major platforms, such as Netflix, Disney+, and Amazon Prime. We extensively analyse the security of three widely used DRM solutions -- Google Widevine, Apple FairPlay, and Microsoft PlayReady -- deployed on billions of devices worldwide. We then consolidate their features and capabilities, deriving common features and security properties for their evaluation. Furthermore, we identify some design-level shortcomings that render them vulnerable to emerging attacks within the state of the art, including micro-architectural side-channel vulnerabilities and an absence of post-quantum security. Lastly, we propose mitigations and suggest future directions of research.

Title: FLAIRS: FPGA-Accelerated Inference-Resistant & Secure Federated Learning. (arXiv:2308.00553v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.00553
• Code URL: null
• Copy Paste: [[2308.00553] FLAIRS: FPGA-Accelerated Inference-Resistant & Secure Federated Learning](http://arxiv.org/abs/2308.00553) #secure
• Summary:

  Federated Learning (FL) has become very popular since it enables clients to train a joint model collaboratively without sharing their private data. However, FL has been shown to be susceptible to backdoor and inference attacks. While in the former, the adversary injects manipulated updates into the aggregation process; the latter leverages clients' local models to deduce their private data. Contemporary solutions to address the security concerns of FL are either impractical for real-world deployment due to high-performance overheads or are tailored towards addressing specific threats, for instance, privacy-preserving aggregation or backdoor defenses. Given these limitations, our research delves into the advantages of harnessing the FPGA-based computing paradigm to overcome performance bottlenecks of software-only solutions while mitigating backdoor and inference attacks. We utilize FPGA-based enclaves to address inference attacks during the aggregation process of FL. We adopt an advanced backdoor-aware aggregation algorithm on the FPGA to counter backdoor attacks. We implemented and evaluated our method on Xilinx VMK-180, yielding a significant speed-up of around 300 times on the IoT-Traffic dataset and more than 506 times on the CIFAR-10 dataset.

Title: Secure and Trustworthy Computing 2.0 Vision Statement. (arXiv:2308.00623v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.00623
• Code URL: null
• Copy Paste: [[2308.00623] Secure and Trustworthy Computing 2](http://arxiv.org/abs/2308.00623) #secure
• Summary:

  The Secure and Trustworthy Computing (SaTC) program within the National Science Foundation (NSF) program serves as the primary instrument for creating novel fundamental science in security and privacy in the United States with broad impacts that influence the world. The program funds research in a vast array of research topics that span technology, theory, policy, law, and society. In the Spring of 2023, the program managers of SaTC requested that the community prepare a vision for the next ten years of research. This document represents the results of that effort which involved collecting input from numerous members of the security and privacy community, industry, academics, and government. Assembled from that input, this document offers a comprehensive view of general themes and specific areas of focus for future research as envisioned by the community.

security

Title: On the Impact of the Hardware Warm-Up Time on Deep Learning-Based RF Fingerprinting. (arXiv:2308.00156v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.00156
• Code URL: null
• Copy Paste: [[2308.00156] On the Impact of the Hardware Warm-Up Time on Deep Learning-Based RF Fingerprinting](http://arxiv.org/abs/2308.00156) #security
• Summary:

  Deep learning-based RF fingerprinting offers great potential for improving the security robustness of various emerging wireless networks. Although much progress has been done in enhancing fingerprinting methods, the impact of device hardware stabilization and warm-up time on the achievable fingerprinting performances has not received adequate attention. As such, this paper focuses on addressing this gap by investigating and shedding light on what could go wrong if the hardware stabilization aspects are overlooked. Specifically, our experimental results show that when the deep learning models are trained with data samples captured after the hardware stabilizes but tested with data captured right after powering on the devices, the device classification accuracy drops below 37%. However, when both the training and testing data are captured after the stabilization period, the achievable average accuracy exceeds 99%, when the model is trained and tested on the same day, and achieves 88% and 96% when the model is trained on one day but tested on another day, for the wireless and wired scenarios, respectively. Additionally, in this work, we leverage simulation and testbed experimentation to explain the cause behind the I/Q signal behavior observed during the device hardware warm-up time that led to the RF fingerprinting performance degradation. Furthermore, we release a large WiFi dataset, containing both unstable (collected during the warm-up period) and stable (collected after the warm-up period) captures across multiple days. Our work contributes datasets, explanations, and guidelines to enhance the robustness of RF fingerprinting in securing emerging wireless networks.

Title: Verifiable Data Sharing Scheme for Dynamic Multi-Owner Setting. (arXiv:2308.00239v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.00239
• Code URL: null
• Copy Paste: [[2308.00239] Verifiable Data Sharing Scheme for Dynamic Multi-Owner Setting](http://arxiv.org/abs/2308.00239) #security
• Summary:

  One of scenarios in data-sharing applications is that files are managed by multiple owners, and the list of file owners may change dynamically. However, most existing solutions to this problem rely on trusted third parties and have complicated signature permission processes, resulting in additional overhead. Therefore, we propose a verifiable data-sharing scheme (VDS-DM) that can support dynamic multi-owner scenarios. We introduce a management entity that combines linear secret-sharing technology, multi-owner signature generation, and an aggregation technique to allow multi-owner file sharing. Without the help of trusted third parties, VDS-DM can update file signatures for dynamically changing file owners, which helps save communication overhead. Moreover, users independently verify the integrity of files without resorting to a third party. We analyse the security of VDS-DM through a security game. Finally, we conduct enough simulation experiments and the outcomes of experimental demonstrate the feasibility of VDS-DM.

Title: Enhanced Security with Encrypted Vision Transformer in Federated Learning. (arXiv:2308.00271v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.00271
• Code URL: null
• Copy Paste: [[2308.00271] Enhanced Security with Encrypted Vision Transformer in Federated Learning](http://arxiv.org/abs/2308.00271) #security
• Summary:

  Federated learning is a learning method for training models over multiple participants without directly sharing their raw data, and it has been expected to be a privacy protection method for training data. In contrast, attack methods have been studied to restore learning data from model information shared with clients, so enhanced security against attacks has become an urgent problem. Accordingly, in this article, we propose a novel framework of federated learning on the bases of the embedded structure of the vision transformer by using the model information encrypted with a random sequence. In image classification experiments, we verify the effectiveness of the proposed method on the CIFAR-10 dataset in terms of classification accuracy and robustness against attacks.

Title: VulMatch: Binary-level Vulnerability Detection Through Signature. (arXiv:2308.00288v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.00288
• Code URL: null
• Copy Paste: [[2308.00288] VulMatch: Binary-level Vulnerability Detection Through Signature](http://arxiv.org/abs/2308.00288) #security
• Summary:

  Similar vulnerability repeats in real-world software products because of code reuse, especially in wildly reused third-party code and libraries. Detecting repeating vulnerabilities like 1-day and N-day vulnerabilities is an important cyber security task. Unfortunately, the state-of-the-art methods suffer from poor performance because they detect patch existence instead of vulnerability existence and infer the vulnerability signature directly from binary code. In this paper, we propose VulMatch to extract precise vulnerability-related binary instructions to generate the vulnerability-related signature. VulMatch detects vulnerability existence based on binary signatures. Unlike previous approaches, VulMatch accurately locates vulnerability-related instructions by utilizing source and binary codes. Our experiments were conducted using over 1000 vulnerable instances across seven open-source projects. VulMatch significantly outperformed the baseline tools Asm2vec and Palmtree. Besides the performance advantages over the baseline tools, VulMatch offers a better feature by providing explainable reasons during vulnerability detection. Our empirical studies demonstrate that VulMatch detects fine-grained vulnerability that the state-of-the-art tools struggle with. Our experiment on commercial firmware demonstrates VulMatch is able to find vulnerabilities in real-world scenario.

privacy

Title: Towards Equitable Privacy. (arXiv:2308.00004v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.00004
• Code URL: null
• Copy Paste: [[2308.00004] Towards Equitable Privacy](http://arxiv.org/abs/2308.00004) #privacy
• Summary:

  Ensuring equitable privacy experiences remains a challenge, especially for marginalised and vulnerable populations (MVPs) who often hesitate to participate or use digital services due to concerns about the privacy of their sensitive information. In response, security research has emphasised the importance of inclusive security and privacy practices to facilitate meaningful engagement of MVPs online. However, research in this area is still in its early stages, with other MVPs yet to be considered (such as low-income groups, and refugees), novel engagement methods yet to be explored, and limited support for software developers in building applications and services for MVPs. In 2022, we initiated a UK Research Council funded Equitable Privacy project to address these gaps. Our goal is to prioritise the privacy needs and requirements of MVPs in the design and development of software applications and services.

We design and implement a new participatory research approach -- community studybeds -- in collaboration with third-sector organisations that support MVPs to identify and tackle the challenges these groups encounter. In this paper, we share the initial reflections and experiences of the Equitable Privacy project, particularly emphasising the utilisation of our community studybeds.

protect

defense

attack

Title: A Study of Unsupervised Evaluation Metrics for Practical and Automatic Domain Adaptation. (arXiv:2308.00287v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00287
• Code URL: null
• Copy Paste: [[2308.00287] A Study of Unsupervised Evaluation Metrics for Practical and Automatic Domain Adaptation](http://arxiv.org/abs/2308.00287) #attack
• Summary:

  Unsupervised domain adaptation (UDA) methods facilitate the transfer of models to target domains without labels. However, these methods necessitate a labeled target validation set for hyper-parameter tuning and model selection. In this paper, we aim to find an evaluation metric capable of assessing the quality of a transferred model without access to target validation labels. We begin with the metric based on mutual information of the model prediction. Through empirical analysis, we identify three prevalent issues with this metric: 1) It does not account for the source structure. 2) It can be easily attacked. 3) It fails to detect negative transfer caused by the over-alignment of source and target features. To address the first two issues, we incorporate source accuracy into the metric and employ a new MLP classifier that is held out during training, significantly improving the result. To tackle the final issue, we integrate this enhanced metric with data augmentation, resulting in a novel unsupervised UDA metric called the Augmentation Consistency Metric (ACM). Additionally, we empirically demonstrate the shortcomings of previous experiment settings and conduct large-scale experiments to validate the effectiveness of our proposed metric. Furthermore, we employ our metric to automatically search for the optimal hyper-parameter set, achieving superior performance compared to manually tuned sets across four common benchmarks. Codes will be available soon.

Title: How User Language Affects Conflict Fatality Estimates in ChatGPT. (arXiv:2308.00072v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.00072
• Code URL: null
• Copy Paste: [[2308.00072] How User Language Affects Conflict Fatality Estimates in ChatGPT](http://arxiv.org/abs/2308.00072) #attack
• Summary:

  OpenAI's ChatGPT language model has gained popularity as a powerful tool for complex problem-solving and information retrieval. However, concerns arise about the reproduction of biases present in the language-specific training data. In this study, we address this issue in the context of the Israeli-Palestinian and Turkish-Kurdish conflicts. Using GPT-3.5, we employed an automated query procedure to inquire about casualties in specific airstrikes, in both Hebrew and Arabic for the former conflict and Turkish and Kurdish for the latter. Our analysis reveals that GPT-3.5 provides 27$\pm$11 percent lower fatality estimates when queried in the language of the attacker than in the language of the targeted group. Evasive answers denying the existence of such attacks further increase the discrepancy, creating a novel bias mechanism not present in regular search engines. This language bias has the potential to amplify existing media biases and contribute to information bubbles, ultimately reinforcing conflicts.

Title: LimeAttack: Local Explainable Method for Textual Hard-Label Adversarial Attack. (arXiv:2308.00319v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.00319
• Code URL: null
• Copy Paste: [[2308.00319] LimeAttack: Local Explainable Method for Textual Hard-Label Adversarial Attack](http://arxiv.org/abs/2308.00319) #attack
• Summary:

  Natural language processing models are vulnerable to adversarial examples. Previous textual adversarial attacks adopt gradients or confidence scores to calculate word importance ranking and generate adversarial examples. However, this information is unavailable in the real world. Therefore, we focus on a more realistic and challenging setting, named hard-label attack, in which the attacker can only query the model and obtain a discrete prediction label. Existing hard-label attack algorithms tend to initialize adversarial examples by random substitution and then utilize complex heuristic algorithms to optimize the adversarial perturbation. These methods require a lot of model queries and the attack success rate is restricted by adversary initialization. In this paper, we propose a novel hard-label attack algorithm named LimeAttack, which leverages a local explainable method to approximate word importance ranking, and then adopts beam search to find the optimal solution. Extensive experiments show that LimeAttack achieves the better attacking performance compared with existing hard-label attack under the same query budget. In addition, we evaluate the effectiveness of LimeAttack on large language models, and results indicate that adversarial examples remain a significant threat to large language models. The adversarial examples crafted by LimeAttack are highly transferable and effectively improve model robustness in adversarial training.

Title: Using Kernel SHAP XAI Method to optimize the Network Anomaly Detection Model. (arXiv:2308.00074v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00074
• Code URL: null
• Copy Paste: [[2308.00074] Using Kernel SHAP XAI Method to optimize the Network Anomaly Detection Model](http://arxiv.org/abs/2308.00074) #attack
• Summary:

  Anomaly detection and its explanation is important in many research areas such as intrusion detection, fraud detection, unknown attack detection in network traffic and logs. It is challenging to identify the cause or explanation of why one instance is an anomaly? and the other is not due to its unbounded and lack of supervisory nature. The answer to this question is possible with the emerging technique of explainable artificial intelligence (XAI). XAI provides tools and techniques to interpret and explain the output and working of complex models such as Deep Learning (DL). This paper aims to detect and explain network anomalies with XAI, kernelSHAP method. The same approach is used to improve the network anomaly detection model in terms of accuracy, recall, precision and f score. The experiment is conduced with the latest CICIDS2017 dataset. Two models are created (Model_1 and OPT_Model) and compared. The overall accuracy and F score of OPT_Model (when trained in unsupervised way) are 0.90 and 0.76, respectively.

Title: A Novel Deep Learning based Model to Defend Network Intrusion Detection System against Adversarial Attacks. (arXiv:2308.00077v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.00077
• Code URL: null
• Copy Paste: [[2308.00077] A Novel Deep Learning based Model to Defend Network Intrusion Detection System against Adversarial Attacks](http://arxiv.org/abs/2308.00077) #attack
• Summary:

  Network Intrusion Detection System (NIDS) is an essential tool in securing cyberspace from a variety of security risks and unknown cyberattacks. A number of solutions have been implemented for Machine Learning (ML), and Deep Learning (DL) based NIDS. However, all these solutions are vulnerable to adversarial attacks, in which the malicious actor tries to evade or fool the model by injecting adversarial perturbed examples into the system. The main aim of this research work is to study powerful adversarial attack algorithms and their defence method on DL-based NIDS. Fast Gradient Sign Method (FGSM), Jacobian Saliency Map Attack (JSMA), Projected Gradient Descent (PGD) and Carlini & Wagner (C&W) are four powerful adversarial attack methods implemented against the NIDS. As a defence method, Adversarial Training is used to increase the robustness of the NIDS model. The results are summarized in three phases, i.e., 1) before the adversarial attack, 2) after the adversarial attack, and 3) after the adversarial defence. The Canadian Institute for Cybersecurity Intrusion Detection System 2017 (CICIDS-2017) dataset is used for evaluation purposes with various performance measurements like f1-score, accuracy etc.

Title: SF-IDS: An Imbalanced Semi-Supervised Learning Framework for Fine-grained Intrusion Detection. (arXiv:2308.00542v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.00542
• Code URL: null
• Copy Paste: [[2308.00542] SF-IDS: An Imbalanced Semi-Supervised Learning Framework for Fine-grained Intrusion Detection](http://arxiv.org/abs/2308.00542) #attack
• Summary:

  Deep learning-based fine-grained network intrusion detection systems (NIDS) enable different attacks to be responded to in a fast and targeted manner with the help of large-scale labels. However, the cost of labeling causes insufficient labeled samples. Also, the real fine-grained traffic shows a long-tailed distribution with great class imbalance. These two problems often appear simultaneously, posing serious challenges to fine-grained NIDS. In this work, we propose a novel semi-supervised fine-grained intrusion detection framework, SF-IDS, to achieve attack classification in the label-limited and highly class imbalanced case. We design a self-training backbone model called RI-1DCNN to boost the feature extraction by reconstructing the input samples into a multichannel image format. The uncertainty of the generated pseudo-labels is evaluated and used as a reference for pseudo-label filtering in combination with the prediction probability. To mitigate the effects of fine-grained class imbalance, we propose a hybrid loss function combining supervised contrastive loss and multi-weighted classification loss to obtain more compact intra-class features and clearer inter-class intervals. Experiments show that the proposed SF-IDS achieves 3.01% and 2.71% Marco-F1 improvement on two classical datasets with 1% labeled, respectively.

robust

Title: Robust Single-view Cone-beam X-ray Pose Estimation with Neural Tuned Tomography (NeTT) and Masked Neural Radiance Fields (mNeRF). (arXiv:2308.00214v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00214
• Code URL: null
• Copy Paste: [[2308.00214] Robust Single-view Cone-beam X-ray Pose Estimation with Neural Tuned Tomography (NeTT) and Masked Neural Radiance Fields (mNeRF)](http://arxiv.org/abs/2308.00214) #robust
• Summary:

  Many tasks performed in image-guided, mini-invasive, medical procedures can be cast as pose estimation problems, where an X-ray projection is utilized to reach a target in 3D space. Recent advances in the differentiable rendering of optically reflective materials have enabled state-of-the-art performance in RGB camera view synthesis and pose estimation. Expanding on these prior works, we introduce new methods for pose estimation of radiolucent objects using X-ray projections, and we demonstrate the critical role of optimal view synthesis in performing this task. We first develop an algorithm (DiffDRR) that efficiently computes Digitally Reconstructed Radiographs (DRRs) and leverages automatic differentiation within TensorFlow. In conjunction with classic CBCT reconstruction algorithms, we perform pose estimation by gradient descent using a loss function that quantifies the similarity of the DRR synthesized from a randomly initialized pose and the true fluoroscopic image at the target pose. We propose two novel methods for high-fidelity view synthesis, Neural Tuned Tomography (NeTT) and masked Neural Radiance Fields (mNeRF). Both methods rely on classic CBCT; NeTT directly optimizes the CBCT densities, while the non-zero values of mNeRF are constrained by a 3D mask of the anatomic region segmented from CBCT. We demonstrate that both NeTT and mNeRF distinctly improve pose estimation within our framework. By defining a successful pose estimate to be a 3D angle error of less than 3 deg, we find that NeTT and mNeRF can achieve similar results, both with overall success rates more than 93%. Furthermore, we show that a NeTT trained for a single subject can generalize to synthesize high-fidelity DRRs and ensure robust pose estimations for all other subjects. Therefore, we suggest that NeTT is an attractive option for robust pose estimation using fluoroscopic projections.

Title: Using Scene and Semantic Features for Multi-modal Emotion Recognition. (arXiv:2308.00228v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00228
• Code URL: null
• Copy Paste: [[2308.00228] Using Scene and Semantic Features for Multi-modal Emotion Recognition](http://arxiv.org/abs/2308.00228) #robust
• Summary:

  Automatic emotion recognition is a hot topic with a wide range of applications. Much work has been done in the area of automatic emotion recognition in recent years. The focus has been mainly on using the characteristics of a person such as speech, facial expression and pose for this purpose. However, the processing of scene and semantic features for emotion recognition has had limited exploration. In this paper, we propose to use combined scene and semantic features, along with personal features, for multi-modal emotion recognition. Scene features will describe the environment or context in which the target person is operating. The semantic feature can include objects that are present in the environment, as well as their attributes and relationships with the target person. In addition, we use a modified EmbraceNet to extract features from the images, which is trained to learn both the body and pose features simultaneously. By fusing both body and pose features, the EmbraceNet can improve the accuracy and robustness of the model, particularly when dealing with partially missing data. This is because having both body and pose features provides a more complete representation of the subject in the images, which can help the model to make more accurate predictions even when some parts of body are missing. We demonstrate the efficiency of our method on the benchmark EMOTIC dataset. We report an average precision of 40.39\% across the 26 emotion categories, which is a 5\% improvement over previous approaches.

Title: Robust Positive-Unlabeled Learning via Noise Negative Sample Self-correction. (arXiv:2308.00279v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00279
• Code URL: https://github.com/woriazzc/robust-pu
• Copy Paste: [[2308.00279] Robust Positive-Unlabeled Learning via Noise Negative Sample Self-correction](http://arxiv.org/abs/2308.00279) #robust
• Summary:

  Learning from positive and unlabeled data is known as positive-unlabeled (PU) learning in literature and has attracted much attention in recent years. One common approach in PU learning is to sample a set of pseudo-negatives from the unlabeled data using ad-hoc thresholds so that conventional supervised methods can be applied with both positive and negative samples. Owing to the label uncertainty among the unlabeled data, errors of misclassifying unlabeled positive samples as negative samples inevitably appear and may even accumulate during the training processes. Those errors often lead to performance degradation and model instability. To mitigate the impact of label uncertainty and improve the robustness of learning with positive and unlabeled data, we propose a new robust PU learning method with a training strategy motivated by the nature of human learning: easy cases should be learned first. Similar intuition has been utilized in curriculum learning to only use easier cases in the early stage of training before introducing more complex cases. Specifically, we utilize a novel hardness'' measure to distinguish unlabeled samples with a high chance of being negative from unlabeled samples with large label noise. An iterative training strategy is then implemented to fine-tune the selection of negative samples during the training process in an iterative manner to include moreeasy'' samples in the early stage of training. Extensive experimental validations over a wide range of learning tasks show that this approach can effectively improve the accuracy and stability of learning with positive and unlabeled data. Our code is available at https://github.com/woriazzc/Robust-PU

Title: Zero-Shot Learning by Harnessing Adversarial Samples. (arXiv:2308.00313v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00313
• Code URL: https://github.com/uqzhichen/haszsl
• Copy Paste: [[2308.00313] Zero-Shot Learning by Harnessing Adversarial Samples](http://arxiv.org/abs/2308.00313) #robust
• Summary:

  Zero-Shot Learning (ZSL) aims to recognize unseen classes by generalizing the knowledge, i.e., visual and semantic relationships, obtained from seen classes, where image augmentation techniques are commonly applied to improve the generalization ability of a model. However, this approach can also cause adverse effects on ZSL since the conventional augmentation techniques that solely depend on single-label supervision is not able to maintain semantic information and result in the semantic distortion issue consequently. In other words, image argumentation may falsify the semantic (e.g., attribute) information of an image. To take the advantage of image augmentations while mitigating the semantic distortion issue, we propose a novel ZSL approach by Harnessing Adversarial Samples (HAS). HAS advances ZSL through adversarial training which takes into account three crucial aspects: (1) robust generation by enforcing augmentations to be similar to negative classes, while maintaining correct labels, (2) reliable generation by introducing a latent space constraint to avert significant deviations from the original data manifold, and (3) diverse generation by incorporating attribute-based perturbation by adjusting images according to each semantic attribute's localization. Through comprehensive experiments on three prominent zero-shot benchmark datasets, we demonstrate the effectiveness of our adversarial samples approach in both ZSL and Generalized Zero-Shot Learning (GZSL) scenarios. Our source code is available at https://github.com/uqzhichen/HASZSL.

Title: Multiscale Global and Regional Feature Learning Using Co-Tuplet Loss for Offline Handwritten Signature Verification. (arXiv:2308.00428v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00428
• Code URL: https://github.com/ashleyfhh/hansig
• Copy Paste: [[2308.00428] Multiscale Global and Regional Feature Learning Using Co-Tuplet Loss for Offline Handwritten Signature Verification](http://arxiv.org/abs/2308.00428) #robust
• Summary:

  Handwritten signature verification is a significant biometric verification method widely acknowledged by legal and financial institutions. However, the development of automatic signature verification systems poses challenges due to inter-writer similarity, intra-writer variations, and the limited number of signature samples. To address these challenges, we propose a multiscale global and regional feature learning network (MGRNet) with the co-tuplet loss, a new metric learning loss, for offline handwritten signature verification. MGRNet jointly learns global and regional information from various spatial scales and integrates it to generate discriminative features. Consequently, it can capture overall signature stroke information while detecting detailed local differences between genuine and skilled-forged signatures. To enhance the discriminative capability of our network further, we propose the co-tuplet loss, which simultaneously considers multiple positive and negative examples to learn distance metrics. By dealing with inter-writer similarity and intra-writer variations and focusing on informative examples, the co-tuplet loss addresses the limitations of typical metric learning losses. Additionally, we develop HanSig, a large-scale Chinese signature dataset, to facilitate the development of robust systems for this script. The dataset is available at https://github.com/ashleyfhh/HanSig. Experimental results on four benchmark datasets in different languages demonstrate the promising performance of our method in comparison to state-of-the-art approaches.

Title: A Majority Invariant Approach to Patch Robustness Certification for Deep Learning Models. (arXiv:2308.00452v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00452
• Code URL: https://github.com/kio-cs/majorcert
• Copy Paste: [[2308.00452] A Majority Invariant Approach to Patch Robustness Certification for Deep Learning Models](http://arxiv.org/abs/2308.00452) #robust
• Summary:

  Patch robustness certification ensures no patch within a given bound on a sample can manipulate a deep learning model to predict a different label. However, existing techniques cannot certify samples that cannot meet their strict bars at the classifier or patch region levels. This paper proposes MajorCert. MajorCert firstly finds all possible label sets manipulatable by the same patch region on the same sample across the underlying classifiers, then enumerates their combinations element-wise, and finally checks whether the majority invariant of all these combinations is intact to certify samples.

Title: Is Last Layer Re-Training Truly Sufficient for Robustness to Spurious Correlations?. (arXiv:2308.00473v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00473
• Code URL: null
• Copy Paste: [[2308.00473] Is Last Layer Re-Training Truly Sufficient for Robustness to Spurious Correlations?](http://arxiv.org/abs/2308.00473) #robust
• Summary:

  Models trained with empirical risk minimization (ERM) are known to learn to rely on spurious features, i.e., their prediction is based on undesired auxiliary features which are strongly correlated with class labels but lack causal reasoning. This behavior particularly degrades accuracy in groups of samples of the correlated class that are missing the spurious feature or samples of the opposite class but with the spurious feature present. The recently proposed Deep Feature Reweighting (DFR) method improves accuracy of these worst groups. Based on the main argument that ERM mods can learn core features sufficiently well, DFR only needs to retrain the last layer of the classification model with a small group-balanced data set. In this work, we examine the applicability of DFR to realistic data in the medical domain. Furthermore, we investigate the reasoning behind the effectiveness of last-layer retraining and show that even though DFR has the potential to improve the accuracy of the worst group, it remains susceptible to spurious correlations.

Title: Relational Contrastive Learning for Scene Text Recognition. (arXiv:2308.00508v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00508
• Code URL: https://github.com/thundervvv/rclstr
• Copy Paste: [[2308.00508] Relational Contrastive Learning for Scene Text Recognition](http://arxiv.org/abs/2308.00508) #robust
• Summary:

  Context-aware methods achieved great success in supervised scene text recognition via incorporating semantic priors from words. We argue that such prior contextual information can be interpreted as the relations of textual primitives due to the heterogeneous text and background, which can provide effective self-supervised labels for representation learning. However, textual relations are restricted to the finite size of dataset due to lexical dependencies, which causes the problem of over-fitting and compromises representation robustness. To this end, we propose to enrich the textual relations via rearrangement, hierarchy and interaction, and design a unified framework called RCLSTR: Relational Contrastive Learning for Scene Text Recognition. Based on causality, we theoretically explain that three modules suppress the bias caused by the contextual prior and thus guarantee representation robustness. Experiments on representation quality show that our method outperforms state-of-the-art self-supervised STR methods. Code is available at https://github.com/ThunderVVV/RCLSTR.

Title: Point Annotation Probability Map: Towards Dense Object Counting by Tolerating Annotation Noise. (arXiv:2308.00530v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00530
• Code URL: null
• Copy Paste: [[2308.00530] Point Annotation Probability Map: Towards Dense Object Counting by Tolerating Annotation Noise](http://arxiv.org/abs/2308.00530) #robust
• Summary:

  Counting objects in crowded scenes remains a challenge to computer vision. The current deep learning based approach often formulate it as a Gaussian density regression problem. Such a brute-force regression, though effective, may not consider the annotation noise properly which arises from the human annotation process and may lead to different distributions. We conjecture that it would be beneficial to consider the annotation noise in the dense object counting task. To obtain strong robustness against annotation noise, generalized Gaussian distribution (GGD) function with a tunable bandwidth and shape parameter is exploited to form the learning target point annotation probability map, PAPM. Specifically, we first present a hand-designed PAPM method (HD-PAPM), in which we design a function based on GGD to tolerate the annotation noise. For end-to-end training, the hand-designed PAPM may not be optimal for the particular network and dataset. An adaptively learned PAPM method (AL-PAPM) is proposed. To improve the robustness to annotation noise, we design an effective transport cost function based on GGD. With such transport cost constraints, a better PAPM presentation could be adaptively learned with an optimal transport framework from point annotation in an end-to-end manner. Extensive experiments show the superiority of our proposed methods.

Title: Predicting masked tokens in stochastic locations improves masked image modeling. (arXiv:2308.00566v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00566
• Code URL: null
• Copy Paste: [[2308.00566] Predicting masked tokens in stochastic locations improves masked image modeling](http://arxiv.org/abs/2308.00566) #robust
• Summary:

  Self-supervised learning is a promising paradigm in deep learning that enables learning from unlabeled data by constructing pretext tasks that require learning useful representations. In natural language processing, the dominant pretext task has been masked language modeling (MLM), while in computer vision there exists an equivalent called Masked Image Modeling (MIM). However, MIM is challenging because it requires predicting semantic content in accurate locations. E.g, given an incomplete picture of a dog, we can guess that there is a tail, but we cannot determine its exact location. In this work, we propose FlexPredict, a stochastic model that addresses this challenge by incorporating location uncertainty into the model. Specifically, we condition the model on stochastic masked token positions to guide the model toward learning features that are more robust to location uncertainties. Our approach improves downstream performance on a range of tasks, e.g, compared to MIM baselines, FlexPredict boosts ImageNet linear probing by 1.6% with ViT-B and by 2.5% for semi-supervised video segmentation using ViT-L.

Title: Beyond One-Hot-Encoding: Injecting Semantics to Drive Image Classifiers. (arXiv:2308.00607v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00607
• Code URL: https://github.com/s1m0n38/semantic-encodings
• Copy Paste: [[2308.00607] Beyond One-Hot-Encoding: Injecting Semantics to Drive Image Classifiers](http://arxiv.org/abs/2308.00607) #robust
• Summary:

  Images are loaded with semantic information that pertains to real-world ontologies: dog breeds share mammalian similarities, food pictures are often depicted in domestic environments, and so on. However, when training machine learning models for image classification, the relative similarities amongst object classes are commonly paired with one-hot-encoded labels. According to this logic, if an image is labelled as 'spoon', then 'tea-spoon' and 'shark' are equally wrong in terms of training loss. To overcome this limitation, we explore the integration of additional goals that reflect ontological and semantic knowledge, improving model interpretability and trustworthiness. We suggest a generic approach that allows to derive an additional loss term starting from any kind of semantic information about the classification label. First, we show how to apply our approach to ontologies and word embeddings, and discuss how the resulting information can drive a supervised learning process. Second, we use our semantically enriched loss to train image classifiers, and analyse the trade-offs between accuracy, mistake severity, and learned internal representations. Finally, we discuss how this approach can be further exploited in terms of explainability and adversarial robustness. Code repository: https://github.com/S1M0N38/semantic-encodings

Title: Human-M3: A Multi-view Multi-modal Dataset for 3D Human Pose Estimation in Outdoor Scenes. (arXiv:2308.00628v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00628
• Code URL: https://github.com/soullessrobot/human-m3-dataset
• Copy Paste: [[2308.00628] Human-M3: A Multi-view Multi-modal Dataset for 3D Human Pose Estimation in Outdoor Scenes](http://arxiv.org/abs/2308.00628) #robust
• Summary:

  3D human pose estimation in outdoor environments has garnered increasing attention recently. However, prevalent 3D human pose datasets pertaining to outdoor scenes lack diversity, as they predominantly utilize only one type of modality (RGB image or pointcloud), and often feature only one individual within each scene. This limited scope of dataset infrastructure considerably hinders the variability of available data. In this article, we propose Human-M3, an outdoor multi-modal multi-view multi-person human pose database which includes not only multi-view RGB videos of outdoor scenes but also corresponding pointclouds. In order to obtain accurate human poses, we propose an algorithm based on multi-modal data input to generate ground truth annotation. This benefits from robust pointcloud detection and tracking, which solves the problem of inaccurate human localization and matching ambiguity that may exist in previous multi-view RGB videos in outdoor multi-person scenes, and generates reliable ground truth annotations. Evaluation of multiple different modalities algorithms has shown that this database is challenging and suitable for future research. Furthermore, we propose a 3D human pose estimation algorithm based on multi-modal data input, which demonstrates the advantages of multi-modal data input for 3D human pose estimation. Code and data will be released on https://github.com/soullessrobot/Human-M3-Dataset.

Title: AnyLoc: Towards Universal Visual Place Recognition. (arXiv:2308.00688v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00688
• Code URL: https://github.com/AnyLoc/AnyLoc
• Copy Paste: [[2308.00688] AnyLoc: Towards Universal Visual Place Recognition](http://arxiv.org/abs/2308.00688) #robust
• Summary:

  Visual Place Recognition (VPR) is vital for robot localization. To date, the most performant VPR approaches are environment- and task-specific: while they exhibit strong performance in structured environments (predominantly urban driving), their performance degrades severely in unstructured environments, rendering most approaches brittle to robust real-world deployment. In this work, we develop a universal solution to VPR -- a technique that works across a broad range of structured and unstructured environments (urban, outdoors, indoors, aerial, underwater, and subterranean environments) without any re-training or fine-tuning. We demonstrate that general-purpose feature representations derived from off-the-shelf self-supervised models with no VPR-specific training are the right substrate upon which to build such a universal VPR solution. Combining these derived features with unsupervised feature aggregation enables our suite of methods, AnyLoc, to achieve up to 4X significantly higher performance than existing approaches. We further obtain a 6% improvement in performance by characterizing the semantic properties of these features, uncovering unique domains which encapsulate datasets from similar environments. Our detailed experiments and analysis lay a foundation for building VPR solutions that may be deployed anywhere, anytime, and across anyview. We encourage the readers to explore our project page and interactive demos: https://anyloc.github.io/.

Title: Adversarially Robust Neural Legal Judgement Systems. (arXiv:2308.00165v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.00165
• Code URL: null
• Copy Paste: [[2308.00165] Adversarially Robust Neural Legal Judgement Systems](http://arxiv.org/abs/2308.00165) #robust
• Summary:

  Legal judgment prediction is the task of predicting the outcome of court cases on a given text description of facts of cases. These tasks apply Natural Language Processing (NLP) techniques to predict legal judgment results based on facts. Recently, large-scale public datasets and NLP models have increased research in areas related to legal judgment prediction systems. For such systems to be practically helpful, they should be robust from adversarial attacks. Previous works mainly focus on making a neural legal judgement system; however, significantly less or no attention has been given to creating a robust Legal Judgement Prediction(LJP) system. We implemented adversarial attacks on early existing LJP systems and found that none of them could handle attacks. In this work, we proposed an approach for making robust LJP systems. Extensive experiments on three legal datasets show significant improvements in our approach over the state-of-the-art LJP system in handling adversarial attacks. To the best of our knowledge, we are the first to increase the robustness of early-existing LJP systems.

Title: ZRIGF: An Innovative Multimodal Framework for Zero-Resource Image-Grounded Dialogue Generation. (arXiv:2308.00400v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.00400
• Code URL: https://github.com/zhangbo-nlp/zrigf
• Copy Paste: [[2308.00400] ZRIGF: An Innovative Multimodal Framework for Zero-Resource Image-Grounded Dialogue Generation](http://arxiv.org/abs/2308.00400) #robust
• Summary:

  Image-grounded dialogue systems benefit greatly from integrating visual information, resulting in high-quality response generation. However, current models struggle to effectively utilize such information in zero-resource scenarios, mainly due to the disparity between image and text modalities. To overcome this challenge, we propose an innovative multimodal framework, called ZRIGF, which assimilates image-grounded information for dialogue generation in zero-resource situations. ZRIGF implements a two-stage learning strategy, comprising contrastive pre-training and generative pre-training. Contrastive pre-training includes a text-image matching module that maps images and texts into a unified encoded vector space, along with a text-assisted masked image modeling module that preserves pre-training visual features and fosters further multimodal feature alignment. Generative pre-training employs a multimodal fusion module and an information transfer module to produce insightful responses based on harmonized multimodal representations. Comprehensive experiments conducted on both text-based and image-grounded dialogue datasets demonstrate ZRIGF's efficacy in generating contextually pertinent and informative responses. Furthermore, we adopt a fully zero-resource scenario in the image-grounded dialogue dataset to demonstrate our framework's robust generalization capabilities in novel domains. The code is available at https://github.com/zhangbo-nlp/ZRIGF.

Title: Dynamic ensemble selection based on Deep Neural Network Uncertainty Estimation for Adversarial Robustness. (arXiv:2308.00346v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00346
• Code URL: null
• Copy Paste: [[2308.00346] Dynamic ensemble selection based on Deep Neural Network Uncertainty Estimation for Adversarial Robustness](http://arxiv.org/abs/2308.00346) #robust
• Summary:

  The deep neural network has attained significant efficiency in image recognition. However, it has vulnerable recognition robustness under extensive data uncertainty in practical applications. The uncertainty is attributed to the inevitable ambient noise and, more importantly, the possible adversarial attack. Dynamic methods can effectively improve the defense initiative in the arms race of attack and defense of adversarial examples. Different from the previous dynamic method depend on input or decision, this work explore the dynamic attributes in model level through dynamic ensemble selection technology to further protect the model from white-box attacks and improve the robustness. Specifically, in training phase the Dirichlet distribution is apply as prior of sub-models' predictive distribution, and the diversity constraint in parameter space is introduced under the lightweight sub-models to construct alternative ensembel model spaces. In test phase, the certain sub-models are dynamically selected based on their rank of uncertainty value for the final prediction to ensure the majority accurate principle in ensemble robustness and accuracy. Compared with the previous dynamic method and staic adversarial traning model, the presented approach can achieve significant robustness results without damaging accuracy by combining dynamics and diversity property.

Title: Compressed Private Aggregation for Scalable and Robust Federated Learning over Massive Networks. (arXiv:2308.00540v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.00540
• Code URL: https://github.com/langnatalie/CPA
• Copy Paste: [[2308.00540] Compressed Private Aggregation for Scalable and Robust Federated Learning over Massive Networks](http://arxiv.org/abs/2308.00540) #robust
• Summary:

  Federated learning (FL) is an emerging paradigm that allows a central server to train machine learning models using remote users' data. Despite its growing popularity, FL faces challenges in preserving the privacy of local datasets, its sensitivity to poisoning attacks by malicious users, and its communication overhead. The latter is additionally considerably dominant in large-scale networks. These limitations are often individually mitigated by local differential privacy (LDP) mechanisms, robust aggregation, compression, and user selection techniques, which typically come at the cost of accuracy. In this work, we present compressed private aggregation (CPA), that allows massive deployments to simultaneously communicate at extremely low bit rates while achieving privacy, anonymity, and resilience to malicious users. CPA randomizes a codebook for compressing the data into a few bits using nested lattice quantizers, while ensuring anonymity and robustness, with a subsequent perturbation to hold LDP. The proposed CPA is proven to result in FL convergence in the same asymptotic rate as FL without privacy, compression, and robustness considerations, while satisfying both anonymity and LDP requirements. These analytical properties are empirically confirmed in a numerical study, where we demonstrate the performance gains of CPA compared with separate mechanisms for compression and privacy for training different image classification models, as well as its robustness in mitigating the harmful effects of malicious users.

Title: Unsupervised machine learning shock capturing for High-Order CFD solvers. (arXiv:2308.00086v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00086
• Code URL: https://github.com/andres-mg/2023_gmm_shock_sensor
• Copy Paste: [[2308.00086] Unsupervised machine learning shock capturing for High-Order CFD solvers](http://arxiv.org/abs/2308.00086) #robust
• Summary:

  We present a novel unsupervised machine learning shock capturing algorithm based on Gaussian Mixture Models (GMMs). The proposed GMM sensor demonstrates remarkable accuracy in detecting shocks and is robust across diverse test cases without the need for parameter tuning. We compare the GMM-based sensor with state-of-the-art alternatives. All methods are integrated into a high-order compressible discontinuous Galerkin solver where artificial viscosity can be modulated to capture shocks. Supersonic test cases, including high Reynolds numbers, showcase the sensor's performance, demonstrating the same effectiveness as fine-tuned state-of-the-art sensors. %The nodal DG aproach allows for potential applications in sub-cell flux-differencing formulations, supersonic feature detection, and mesh refinement. The adaptive nature and ability to function without extensive training datasets make this GMM-based sensor suitable for complex geometries and varied flow configurations. Our study reveals the potential of unsupervised machine learning methods, exemplified by the GMM sensor, to improve the robustness and efficiency of advanced CFD codes.

Title: Pretrained deep models outperform GBDTs in Learning-To-Rank under label scarcity. (arXiv:2308.00177v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00177
• Code URL: null
• Copy Paste: [[2308.00177] Pretrained deep models outperform GBDTs in Learning-To-Rank under label scarcity](http://arxiv.org/abs/2308.00177) #robust
• Summary:

  While deep learning (DL) models are state-of-the-art in text and image domains, they have not yet consistently outperformed Gradient Boosted Decision Trees (GBDTs) on tabular Learning-To-Rank (LTR) problems. Most of the recent performance gains attained by DL models in text and image tasks have used unsupervised pretraining, which exploits orders of magnitude more unlabeled data than labeled data. To the best of our knowledge, unsupervised pretraining has not been applied to the LTR problem, which often produces vast amounts of unlabeled data. In this work, we study whether unsupervised pretraining can improve LTR performance over GBDTs and other non-pretrained models. Using simple design choices--including SimCLR-Rank, our ranking-specific modification of SimCLR (an unsupervised pretraining method for images)--we produce pretrained deep learning models that soundly outperform GBDTs (and other non-pretrained models) in the case where labeled data is vastly outnumbered by unlabeled data. We also show that pretrained models also often achieve significantly better robustness than non-pretrained models (GBDTs or DL models) in ranking outlier data.

Title: Doubly Robust Instance-Reweighted Adversarial Training. (arXiv:2308.00311v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00311
• Code URL: null
• Copy Paste: [[2308.00311] Doubly Robust Instance-Reweighted Adversarial Training](http://arxiv.org/abs/2308.00311) #robust
• Summary:

  Assigning importance weights to adversarial data has achieved great success in training adversarially robust networks under limited model capacity. However, existing instance-reweighted adversarial training (AT) methods heavily depend on heuristics and/or geometric interpretations to determine those importance weights, making these algorithms lack rigorous theoretical justification/guarantee. Moreover, recent research has shown that adversarial training suffers from a severe non-uniform robust performance across the training distribution, e.g., data points belonging to some classes can be much more vulnerable to adversarial attacks than others. To address both issues, in this paper, we propose a novel doubly-robust instance reweighted AT framework, which allows to obtain the importance weights via exploring distributionally robust optimization (DRO) techniques, and at the same time boosts the robustness on the most vulnerable examples. In particular, our importance weights are obtained by optimizing the KL-divergence regularized loss function, which allows us to devise new algorithms with a theoretical convergence guarantee. Experiments on standard classification datasets demonstrate that our proposed approach outperforms related state-of-the-art baseline methods in terms of average robust performance, and at the same time improves the robustness against attacks on the weakest data points. Codes will be available soon.

biometric

steal

extraction

Title: InFusion: Inject and Attention Fusion for Multi Concept Zero Shot Text based Video Editing. (arXiv:2308.00135v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00135
• Code URL: null
• Copy Paste: [[2308.00135] InFusion: Inject and Attention Fusion for Multi Concept Zero Shot Text based Video Editing](http://arxiv.org/abs/2308.00135) #extraction
• Summary:

  Large text-to-image diffusion models have achieved remarkable success in generating diverse high-quality images that are closely aligned with text prompt. But, when these models applied to video the main challenge is to ensure temporal consistency and coherent editing. In this paper, we proposed InFusion, a framework for zero-shot text-based video editing leveraging large pre-trained image diffusion models. Our framework specifically supports editing of multiple concepts with the pixel level control over diverse concepts mentioned in the editing prompt. Specifically, we inject the difference of features from U-Net residual blocks in decoder layers for source and edit prompt, this when combined with injected attention features make it feasible to query the source contents and scale the edited concepts along with injection of unedited parts. The editing is further controlled in fine-grained manner with mask extraction and attention fusion strategy which cuts the edited part from source and paste it from the denoising pipeline for target prompt. Our framework is a low cost alternative for the one-shot tuned models for editing since it does not require training. We demonstrated the complex concept editing with generalised image model (Stable Diffusion v1.5) using LoRA. Adaptation is compatible with all the existing image diffusion techniques. Extensive experimental results demonstrate the effectiveness over existing methods in rendering high-quality and temporally consistent videos.

Title: Benchmarking Ultra-High-Definition Image Reflection Removal. (arXiv:2308.00265v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00265
• Code URL: https://github.com/liar-zzy/benchmarking-ultra-high-definition-single-image-reflection-removal
• Copy Paste: [[2308.00265] Benchmarking Ultra-High-Definition Image Reflection Removal](http://arxiv.org/abs/2308.00265) #extraction
• Summary:

  Deep learning based methods have achieved significant success in the task of single image reflection removal (SIRR). However, the majority of these methods are focused on High-Definition/Standard-Definition (HD/SD) images, while ignoring higher resolution images such as Ultra-High-Definition (UHD) images. With the increasing prevalence of UHD images captured by modern devices, in this paper, we aim to address the problem of UHD SIRR. Specifically, we first synthesize two large-scale UHD datasets, UHDRR4K and UHDRR8K. The UHDRR4K dataset consists of $2,999$ and $168$ quadruplets of images for training and testing respectively, and the UHDRR8K dataset contains $1,014$ and $105$ quadruplets. To the best of our knowledge, these two datasets are the first largest-scale UHD datasets for SIRR. Then, we conduct a comprehensive evaluation of six state-of-the-art SIRR methods using the proposed datasets. Based on the results, we provide detailed discussions regarding the strengths and limitations of these methods when applied to UHD images. Finally, we present a transformer-based architecture named RRFormer for reflection removal. RRFormer comprises three modules, namely the Prepossessing Embedding Module, Self-attention Feature Extraction Module, and Multi-scale Spatial Feature Extraction Module. These modules extract hypercolumn features, global and partial attention features, and multi-scale spatial features, respectively. To ensure effective training, we utilize three terms in our loss function: pixel loss, feature loss, and adversarial loss. We demonstrate through experimental results that RRFormer achieves state-of-the-art performance on both the non-UHD dataset and our proposed UHDRR datasets. The code and datasets are publicly available at https://github.com/Liar-zzy/Benchmarking-Ultra-High-Definition-Single-Image-Reflection-Removal.

membership infer

federate

Title: Federated Learning for Data and Model Heterogeneity in Medical Imaging. (arXiv:2308.00155v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00155
• Code URL: null
• Copy Paste: [[2308.00155] Federated Learning for Data and Model Heterogeneity in Medical Imaging](http://arxiv.org/abs/2308.00155) #federate
• Summary:

  Federated Learning (FL) is an evolving machine learning method in which multiple clients participate in collaborative learning without sharing their data with each other and the central server. In real-world applications such as hospitals and industries, FL counters the challenges of data heterogeneity and model heterogeneity as an inevitable part of the collaborative training. More specifically, different organizations, such as hospitals, have their own private data and customized models for local training. To the best of our knowledge, the existing methods do not effectively address both problems of model heterogeneity and data heterogeneity in FL. In this paper, we exploit the data and model heterogeneity simultaneously, and propose a method, MDH-FL (Exploiting Model and Data Heterogeneity in FL) to solve such problems to enhance the efficiency of the global model in FL. We use knowledge distillation and a symmetric loss to minimize the heterogeneity and its impact on the model performance. Knowledge distillation is used to solve the problem of model heterogeneity, and symmetric loss tackles with the data and label heterogeneity. We evaluate our method on the medical datasets to conform the real-world scenario of hospitals, and compare with the existing methods. The experimental results demonstrate the superiority of the proposed approach over the other existing methods.

Title: Physics-Driven Spectrum-Consistent Federated Learning for Palmprint Verification. (arXiv:2308.00451v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00451
• Code URL: https://github.com/zi-yuanyang/psfed-palm
• Copy Paste: [[2308.00451] Physics-Driven Spectrum-Consistent Federated Learning for Palmprint Verification](http://arxiv.org/abs/2308.00451) #federate
• Summary:

  Palmprint as biometrics has gained increasing attention recently due to its discriminative ability and robustness. However, existing methods mainly improve palmprint verification within one spectrum, which is challenging to verify across different spectrums. Additionally, in distributed server-client-based deployment, palmprint verification systems predominantly necessitate clients to transmit private data for model training on the centralized server, thereby engendering privacy apprehensions. To alleviate the above issues, in this paper, we propose a physics-driven spectrum-consistent federated learning method for palmprint verification, dubbed as PSFed-Palm. PSFed-Palm draws upon the inherent physical properties of distinct wavelength spectrums, wherein images acquired under similar wavelengths display heightened resemblances. Our approach first partitions clients into short- and long-spectrum groups according to the wavelength range of their local spectrum images. Subsequently, we introduce anchor models for short- and long-spectrum, which constrain the optimization directions of local models associated with long- and short-spectrum images. Specifically, a spectrum-consistent loss that enforces the model parameters and feature representation to align with their corresponding anchor models is designed. Finally, we impose constraints on the local models to ensure their consistency with the global model, effectively preventing model drift. This measure guarantees spectrum consistency while protecting data privacy, as there is no need to share local data. Extensive experiments are conducted to validate the efficacy of our proposed PSFed-Palm approach. The proposed PSFed-Palm demonstrates compelling performance despite only a limited number of training data. The codes will be released at https://github.com/Zi-YuanYang/PSFed-Palm.

Title: AQUILA: Communication Efficient Federated Learning with Adaptive Quantization of Lazily-Aggregated Gradients. (arXiv:2308.00258v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00258
• Code URL: null
• Copy Paste: [[2308.00258] AQUILA: Communication Efficient Federated Learning with Adaptive Quantization of Lazily-Aggregated Gradients](http://arxiv.org/abs/2308.00258) #federate
• Summary:

  The widespread adoption of Federated Learning (FL), a privacy-preserving distributed learning methodology, has been impeded by the challenge of high communication overheads, typically arising from the transmission of large-scale models. Existing adaptive quantization methods, designed to mitigate these overheads, operate under the impractical assumption of uniform device participation in every training round. Additionally, these methods are limited in their adaptability due to the necessity of manual quantization level selection and often overlook biases inherent in local devices' data, thereby affecting the robustness of the global model. In response, this paper introduces AQUILA (adaptive quantization of lazily-aggregated gradients), a novel adaptive framework devised to effectively handle these issues, enhancing the efficiency and robustness of FL. AQUILA integrates a sophisticated device selection method that prioritizes the quality and usefulness of device updates. Utilizing the exact global model stored by devices, it enables a more precise device selection criterion, reduces model deviation, and limits the need for hyperparameter adjustments. Furthermore, AQUILA presents an innovative quantization criterion, optimized to improve communication efficiency while assuring model convergence. Our experiments demonstrate that AQUILA significantly decreases communication costs compared to existing methods, while maintaining comparable model performance across diverse non-homogeneous FL settings, such as Non-IID data and heterogeneous model architectures.

Title: Asynchronous Federated Learning with Bidirectional Quantized Communications and Buffered Aggregation. (arXiv:2308.00263v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00263
• Code URL: null
• Copy Paste: [[2308.00263] Asynchronous Federated Learning with Bidirectional Quantized Communications and Buffered Aggregation](http://arxiv.org/abs/2308.00263) #federate
• Summary:

  Asynchronous Federated Learning with Buffered Aggregation (FedBuff) is a state-of-the-art algorithm known for its efficiency and high scalability. However, it has a high communication cost, which has not been examined with quantized communications. To tackle this problem, we present a new algorithm (QAFeL), with a quantization scheme that establishes a shared "hidden" state between the server and clients to avoid the error propagation caused by direct quantization. This approach allows for high precision while significantly reducing the data transmitted during client-server interactions. We provide theoretical convergence guarantees for QAFeL and corroborate our analysis with experiments on a standard benchmark.

Title: Data Collaboration Analysis applied to Compound Datasets and the Introduction of Projection data to Non-IID settings. (arXiv:2308.00280v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00280
• Code URL: null
• Copy Paste: [[2308.00280] Data Collaboration Analysis applied to Compound Datasets and the Introduction of Projection data to Non-IID settings](http://arxiv.org/abs/2308.00280) #federate
• Summary:

  Given the time and expense associated with bringing a drug to market, numerous studies have been conducted to predict the properties of compounds based on their structure using machine learning. Federated learning has been applied to compound datasets to increase their prediction accuracy while safeguarding potentially proprietary information. However, federated learning is encumbered by low accuracy in not identically and independently distributed (non-IID) settings, i.e., data partitioning has a large label bias, and is considered unsuitable for compound datasets, which tend to have large label bias. To address this limitation, we utilized an alternative method of distributed machine learning to chemical compound data from open sources, called data collaboration analysis (DC). We also proposed data collaboration analysis using projection data (DCPd), which is an improved method that utilizes auxiliary PubChem data. This improves the quality of individual user-side data transformations for the projection data for the creation of intermediate representations. The classification accuracy, i.e., area under the curve in the receiver operating characteristic curve (ROC-AUC) and AUC in the precision-recall curve (PR-AUC), of federated averaging (FedAvg), DC, and DCPd was compared for five compound datasets. We determined that the machine learning performance for non-IID settings was in the order of DCPd, DC, and FedAvg, although they were almost the same in identically and independently distributed (IID) settings. Moreover, the results showed that compared to other methods, DCPd exhibited a negligible decline in classification accuracy in experiments with different degrees of label bias. Thus, DCPd can address the low performance in non-IID settings, which is one of the challenges of federated learning.

Title: Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup. (arXiv:2308.00522v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00522
• Code URL: null
• Copy Paste: [[2308.00522] Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup](http://arxiv.org/abs/2308.00522) #federate
• Summary:

  Adaptive optimization has achieved notable success for distributed learning while extending adaptive optimizer to federated Learning (FL) suffers from severe inefficiency, including (i) rugged convergence due to inaccurate gradient estimation in global adaptive optimizer; (ii) client drifts exacerbated by local over-fitting with the local adaptive optimizer. In this work, we propose a novel momentum-based algorithm via utilizing the global gradient descent and locally adaptive amended optimizer to tackle these difficulties. Specifically, we incorporate a locally amended technique to the adaptive optimizer, named Federated Local ADaptive Amended optimizer (\textit{FedLADA}), which estimates the global average offset in the previous communication round and corrects the local offset through a momentum-like term to further improve the empirical training speed and mitigate the heterogeneous over-fitting. Theoretically, we establish the convergence rate of \textit{FedLADA} with a linear speedup property on the non-convex case under the partial participation settings. Moreover, we conduct extensive experiments on the real-world dataset to demonstrate the efficacy of our proposed \textit{FedLADA}, which could greatly reduce the communication rounds and achieves higher accuracy than several baselines.

fair

Title: Interpretable Stereotype Identification through Reasoning. (arXiv:2308.00071v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.00071
• Code URL: null
• Copy Paste: [[2308.00071] Interpretable Stereotype Identification through Reasoning](http://arxiv.org/abs/2308.00071) #fair
• Summary:

  Given that language models are trained on vast datasets that may contain inherent biases, there is a potential danger of inadvertently perpetuating systemic discrimination. Consequently, it becomes essential to examine and address biases in language models, integrating fairness into their development to ensure these models are equitable and free from bias. In this work, we demonstrate the importance of reasoning in zero-shot stereotype identification based on Vicuna-13B-v1.3. While we do observe improved accuracy by scaling from 13B to 33B, we show that the performance gain from reasoning significantly exceeds the gain from scaling up. Our findings suggest that reasoning could be a key factor that enables LLMs to trescend the scaling law on out-of-domain tasks such as stereotype identification. Additionally, through a qualitative analysis of select reasoning traces, we highlight how reasoning enhances not just accuracy but also the interpretability of the decision.

Title: A Suite of Fairness Datasets for Tabular Classification. (arXiv:2308.00133v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00133
• Code URL: null
• Copy Paste: [[2308.00133] A Suite of Fairness Datasets for Tabular Classification](http://arxiv.org/abs/2308.00133) #fair
• Summary:

  There have been many papers with algorithms for improving fairness of machine-learning classifiers for tabular data. Unfortunately, most use only very few datasets for their experimental evaluation. We introduce a suite of functions for fetching 20 fairness datasets and providing associated fairness metadata. Hopefully, these will lead to more rigorous experimental evaluations in future fairness-aware machine learning research.

interpretability

Title: Copula for Instance-wise Feature Selection and Ranking. (arXiv:2308.00549v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00549
• Code URL: null
• Copy Paste: [[2308.00549] Copula for Instance-wise Feature Selection and Ranking](http://arxiv.org/abs/2308.00549) #interpretability
• Summary:

  Instance-wise feature selection and ranking methods can achieve a good selection of task-friendly features for each sample in the context of neural networks. However, existing approaches that assume feature subsets to be independent are imperfect when considering the dependency between features. To address this limitation, we propose to incorporate the Gaussian copula, a powerful mathematical technique for capturing correlations between variables, into the current feature selection framework with no additional changes needed. Experimental results on both synthetic and real datasets, in terms of performance comparison and interpretability, demonstrate that our method is capable of capturing meaningful correlations.

explainability

watermark

Title: Three Bricks to Consolidate Watermarks for Large Language Models. (arXiv:2308.00113v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.00113
• Code URL: https://github.com/facebookresearch/three_bricks
• Copy Paste: [[2308.00113] Three Bricks to Consolidate Watermarks for Large Language Models](http://arxiv.org/abs/2308.00113) #watermark
• Summary:

  The task of discerning between generated and natural texts is increasingly challenging. In this context, watermarking emerges as a promising technique for ascribing generated text to a specific model. It alters the sampling generation process so as to leave an invisible trace in the generated output, facilitating later detection. This research consolidates watermarks for large language models based on three theoretical and empirical considerations. First, we introduce new statistical tests that offer robust theoretical guarantees which remain valid even at low false-positive rates (less than 10$^{\text{-6}}$). Second, we compare the effectiveness of watermarks using classical benchmarks in the field of natural language processing, gaining insights into their real-world applicability. Third, we develop advanced detection schemes for scenarios where access to the LLM is available, as well as multi-bit watermarking.

Title: Advancing Beyond Identification: Multi-bit Watermark for Language Models. (arXiv:2308.00221v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.00221
• Code URL: null
• Copy Paste: [[2308.00221] Advancing Beyond Identification: Multi-bit Watermark for Language Models](http://arxiv.org/abs/2308.00221) #watermark
• Summary:

  This study aims to proactively tackle misuse of large language models beyond identification of machine-generated text. While existing methods focus on detection, some malicious misuses demand tracing the adversary user for counteracting them. To address this, we propose "Multi-bit Watermark through Color-listing" (COLOR), embedding traceable multi-bit information during language model generation. Leveraging the benefits of zero-bit watermarking (Kirchenbauer et al., 2023a), COLOR enables extraction without model access, on-the-fly embedding, and maintains text quality, while allowing zero-bit detection all at the same time. Preliminary experiments demonstrates successful embedding of 32-bit messages with 91.9% accuracy in moderate-length texts ($\sim$500 tokens). This work advances strategies to counter language model misuse effectively.

diffusion

Title: DAVIS: High-Quality Audio-Visual Separation with Generative Diffusion Models. (arXiv:2308.00122v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00122
• Code URL: null
• Copy Paste: [[2308.00122] DAVIS: High-Quality Audio-Visual Separation with Generative Diffusion Models](http://arxiv.org/abs/2308.00122) #diffusion
• Summary:

  We propose DAVIS, a Diffusion model-based Audio-VIusal Separation framework that solves the audio-visual sound source separation task through a generative manner. While existing discriminative methods that perform mask regression have made remarkable progress in this field, they face limitations in capturing the complex data distribution required for high-quality separation of sounds from diverse categories. In contrast, DAVIS leverages a generative diffusion model and a Separation U-Net to synthesize separated magnitudes starting from Gaussian noises, conditioned on both the audio mixture and the visual footage. With its generative objective, DAVIS is better suited to achieving the goal of high-quality sound separation across diverse categories. We compare DAVIS to existing state-of-the-art discriminative audio-visual separation methods on the domain-specific MUSIC dataset and the open-domain AVE dataset, and results show that DAVIS outperforms other methods in separation quality, demonstrating the advantages of our framework for tackling the audio-visual source separation task.

Title: Diffusion Model for Camouflaged Object Detection. (arXiv:2308.00303v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00303
• Code URL: null
• Copy Paste: [[2308.00303] Diffusion Model for Camouflaged Object Detection](http://arxiv.org/abs/2308.00303) #diffusion
• Summary:

  Camouflaged object detection is a challenging task that aims to identify objects that are highly similar to their background. Due to the powerful noise-to-image denoising capability of denoising diffusion models, in this paper, we propose a diffusion-based framework for camouflaged object detection, termed diffCOD, a new framework that considers the camouflaged object segmentation task as a denoising diffusion process from noisy masks to object masks. Specifically, the object mask diffuses from the ground-truth masks to a random distribution, and the designed model learns to reverse this noising process. To strengthen the denoising learning, the input image prior is encoded and integrated into the denoising diffusion model to guide the diffusion process. Furthermore, we design an injection attention module (IAM) to interact conditional semantic features extracted from the image with the diffusion noise embedding via the cross-attention mechanism to enhance denoising learning. Extensive experiments on four widely used COD benchmark datasets demonstrate that the proposed method achieves favorable performance compared to the existing 11 state-of-the-art methods, especially in the detailed texture segmentation of camouflaged objects. Our code will be made publicly available at: https://github.com/ZNan-Chen/diffCOD.

Title: DiffusAL: Coupling Active Learning with Graph Diffusion for Label-Efficient Node Classification. (arXiv:2308.00146v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00146
• Code URL: https://github.com/lmu-dbs/diffusal
• Copy Paste: [[2308.00146] DiffusAL: Coupling Active Learning with Graph Diffusion for Label-Efficient Node Classification](http://arxiv.org/abs/2308.00146) #diffusion
• Summary:

  Node classification is one of the core tasks on attributed graphs, but successful graph learning solutions require sufficiently labeled data. To keep annotation costs low, active graph learning focuses on selecting the most qualitative subset of nodes that maximizes label efficiency. However, deciding which heuristic is best suited for an unlabeled graph to increase label efficiency is a persistent challenge. Existing solutions either neglect aligning the learned model and the sampling method or focus only on limited selection aspects. They are thus sometimes worse or only equally good as random sampling. In this work, we introduce a novel active graph learning approach called DiffusAL, showing significant robustness in diverse settings. Toward better transferability between different graph structures, we combine three independent scoring functions to identify the most informative node samples for labeling in a parameter-free way: i) Model Uncertainty, ii) Diversity Component, and iii) Node Importance computed via graph diffusion heuristics. Most of our calculations for acquisition and training can be pre-processed, making DiffusAL more efficient compared to approaches combining diverse selection criteria and similarly fast as simpler heuristics. Our experiments on various benchmark datasets show that, unlike previous methods, our approach significantly outperforms random selection in 100% of all datasets and labeling budgets tested.

noise learning

data-free

transformer

Title: Performance Evaluation of Swin Vision Transformer Model using Gradient Accumulation Optimization Technique. (arXiv:2308.00197v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00197
• Code URL: null
• Copy Paste: [[2308.00197] Performance Evaluation of Swin Vision Transformer Model using Gradient Accumulation Optimization Technique](http://arxiv.org/abs/2308.00197) #transformer
• Summary:

  Vision Transformers (ViTs) have emerged as a promising approach for visual recognition tasks, revolutionizing the field by leveraging the power of transformer-based architectures. Among the various ViT models, Swin Transformers have gained considerable attention due to their hierarchical design and ability to capture both local and global visual features effectively. This paper evaluates the performance of Swin ViT model using gradient accumulation optimization (GAO) technique. We investigate the impact of gradient accumulation optimization technique on the model's accuracy and training time. Our experiments show that applying the GAO technique leads to a significant decrease in the accuracy of the Swin ViT model, compared to the standard Swin Transformer model. Moreover, we detect a significant increase in the training time of the Swin ViT model when GAO model is applied. These findings suggest that applying the GAO technique may not be suitable for the Swin ViT model, and concern should be undertaken when using GAO technique for other transformer-based models.

Title: Partitioned Saliency Ranking with Dense Pyramid Transformers. (arXiv:2308.00236v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00236
• Code URL: https://github.com/ssecv/psr
• Copy Paste: [[2308.00236] Partitioned Saliency Ranking with Dense Pyramid Transformers](http://arxiv.org/abs/2308.00236) #transformer
• Summary:

  In recent years, saliency ranking has emerged as a challenging task focusing on assessing the degree of saliency at instance-level. Being subjective, even humans struggle to identify the precise order of all salient instances. Previous approaches undertake the saliency ranking by directly sorting the rank scores of salient instances, which have not explicitly resolved the inherent ambiguities. To overcome this limitation, we propose the ranking by partition paradigm, which segments unordered salient instances into partitions and then ranks them based on the correlations among these partitions. The ranking by partition paradigm alleviates ranking ambiguities in a general sense, as it consistently improves the performance of other saliency ranking models. Additionally, we introduce the Dense Pyramid Transformer (DPT) to enable global cross-scale interactions, which significantly enhances feature interactions with reduced computational burden. Extensive experiments demonstrate that our approach outperforms all existing methods. The code for our method is available at \url{https://github.com/ssecv/PSR}.

Title: LGViT: Dynamic Early Exiting for Accelerating Vision Transformer. (arXiv:2308.00255v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00255
• Code URL: null
• Copy Paste: [[2308.00255] LGViT: Dynamic Early Exiting for Accelerating Vision Transformer](http://arxiv.org/abs/2308.00255) #transformer
• Summary:

  Recently, the efficient deployment and acceleration of powerful vision transformers (ViTs) on resource-limited edge devices for providing multimedia services have become attractive tasks. Although early exiting is a feasible solution for accelerating inference, most works focus on convolutional neural networks (CNNs) and transformer models in natural language processing (NLP).Moreover, the direct application of early exiting methods to ViTs may result in substantial performance degradation. To tackle this challenge, we systematically investigate the efficacy of early exiting in ViTs and point out that the insufficient feature representations in shallow internal classifiers and the limited ability to capture target semantic information in deep internal classifiers restrict the performance of these methods. We then propose an early exiting framework for general ViTs termed LGViT, which incorporates heterogeneous exiting heads, namely, local perception head and global aggregation head, to achieve an efficiency-accuracy trade-off. In particular, we develop a novel two-stage training scheme, including end-to-end training and self-distillation with the backbone frozen to generate early exiting ViTs, which facilitates the fusion of global and local information extracted by the two types of heads. We conduct extensive experiments using three popular ViT backbones on three vision datasets. Results demonstrate that our LGViT can achieve competitive performance with approximately 1.8 $\times$ speed-up.

Title: Improving Pixel-based MIM by Reducing Wasted Modeling Capability. (arXiv:2308.00261v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00261
• Code URL: https://github.com/open-mmlab/mmpretrain
• Copy Paste: [[2308.00261] Improving Pixel-based MIM by Reducing Wasted Modeling Capability](http://arxiv.org/abs/2308.00261) #transformer
• Summary:

  There has been significant progress in Masked Image Modeling (MIM). Existing MIM methods can be broadly categorized into two groups based on the reconstruction target: pixel-based and tokenizer-based approaches. The former offers a simpler pipeline and lower computational cost, but it is known to be biased toward high-frequency details. In this paper, we provide a set of empirical studies to confirm this limitation of pixel-based MIM and propose a new method that explicitly utilizes low-level features from shallow layers to aid pixel reconstruction. By incorporating this design into our base method, MAE, we reduce the wasted modeling capability of pixel-based MIM, improving its convergence and achieving non-trivial improvements across various downstream tasks. To the best of our knowledge, we are the first to systematically investigate multi-level feature fusion for isotropic architectures like the standard Vision Transformer (ViT). Notably, when applied to a smaller model (e.g., ViT-S), our method yields significant performance gains, such as 1.2\% on fine-tuning, 2.8\% on linear probing, and 2.6\% on semantic segmentation. Code and models are available at https://github.com/open-mmlab/mmpretrain.

Title: FLatten Transformer: Vision Transformer using Focused Linear Attention. (arXiv:2308.00442v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00442
• Code URL: https://github.com/leaplabthu/flatten-transformer
• Copy Paste: [[2308.00442] FLatten Transformer: Vision Transformer using Focused Linear Attention](http://arxiv.org/abs/2308.00442) #transformer
• Summary:

  The quadratic computation complexity of self-attention has been a persistent challenge when applying Transformer models to vision tasks. Linear attention, on the other hand, offers a much more efficient alternative with its linear complexity by approximating the Softmax operation through carefully designed mapping functions. However, current linear attention approaches either suffer from significant performance degradation or introduce additional computation overhead from the mapping functions. In this paper, we propose a novel Focused Linear Attention module to achieve both high efficiency and expressiveness. Specifically, we first analyze the factors contributing to the performance degradation of linear attention from two perspectives: the focus ability and feature diversity. To overcome these limitations, we introduce a simple yet effective mapping function and an efficient rank restoration module to enhance the expressiveness of self-attention while maintaining low computation complexity. Extensive experiments show that our linear attention module is applicable to a variety of advanced vision Transformers, and achieves consistently improved performances on multiple benchmarks. Code is available at https://github.com/LeapLabTHU/FLatten-Transformer.

Title: ViT2EEG: Leveraging Hybrid Pretrained Vision Transformers for EEG Data. (arXiv:2308.00454v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00454
• Code URL: https://github.com/ruiqirichard/eegeyenet-vit
• Copy Paste: [[2308.00454] ViT2EEG: Leveraging Hybrid Pretrained Vision Transformers for EEG Data](http://arxiv.org/abs/2308.00454) #transformer
• Summary:

  In this study, we demonstrate the application of a hybrid Vision Transformer (ViT) model, pretrained on ImageNet, on an electroencephalogram (EEG) regression task. Despite being originally trained for image classification tasks, when fine-tuned on EEG data, this model shows a notable increase in performance compared to other models, including an identical architecture ViT trained without the ImageNet weights. This discovery challenges the traditional understanding of model generalization, suggesting that Transformer models pretrained on seemingly unrelated image data can provide valuable priors for EEG regression tasks with an appropriate fine-tuning pipeline.

The success of this approach suggests that the features extracted by ViT models in the context of visual tasks can be readily transformed for the purpose of EEG predictive modeling. We recommend utilizing this methodology not only in neuroscience and related fields, but generally for any task where data collection is limited by practical, financial, or ethical constraints. Our results illuminate the potential of pretrained models on tasks that are clearly distinct from their original purpose.

Title: PVG: Progressive Vision Graph for Vision Recognition. (arXiv:2308.00574v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00574
• Code URL: null
• Copy Paste: [[2308.00574] PVG: Progressive Vision Graph for Vision Recognition](http://arxiv.org/abs/2308.00574) #transformer
• Summary:

  Convolution-based and Transformer-based vision backbone networks process images into the grid or sequence structures, respectively, which are inflexible for capturing irregular objects. Though Vision GNN (ViG) adopts graph-level features for complex images, it has some issues, such as inaccurate neighbor node selection, expensive node information aggregation calculation, and over-smoothing in the deep layers. To address the above problems, we propose a Progressive Vision Graph (PVG) architecture for vision recognition task. Compared with previous works, PVG contains three main components: 1) Progressively Separated Graph Construction (PSGC) to introduce second-order similarity by gradually increasing the channel of the global graph branch and decreasing the channel of local branch as the layer deepens; 2) Neighbor nodes information aggregation and update module by using Max pooling and mathematical Expectation (MaxE) to aggregate rich neighbor information; 3) Graph error Linear Unit (GraphLU) to enhance low-value information in a relaxed form to reduce the compression of image detail information for alleviating the over-smoothing. Extensive experiments on mainstream benchmarks demonstrate the superiority of PVG over state-of-the-art methods, e.g., our PVG-S obtains 83.0% Top-1 accuracy on ImageNet-1K that surpasses GNN-based ViG-S by +0.9 with the parameters reduced by 18.5%, while the largest PVG-B obtains 84.2% that has +0.5 improvement than ViG-B. Furthermore, our PVG-S obtains +1.3 box AP and +0.4 mask AP gains than ViG-S on COCO dataset.

Title: DPBERT: Efficient Inference for BERT based on Dynamic Planning. (arXiv:2308.00108v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.00108
• Code URL: null
• Copy Paste: [[2308.00108] DPBERT: Efficient Inference for BERT based on Dynamic Planning](http://arxiv.org/abs/2308.00108) #transformer
• Summary:

  Large-scale pre-trained language models such as BERT have contributed significantly to the development of NLP. However, those models require large computational resources, making it difficult to be applied to mobile devices where computing power is limited. In this paper we aim to address the weakness of existing input-adaptive inference methods which fail to take full advantage of the structure of BERT. We propose Dynamic Planning in BERT, a novel fine-tuning strategy that can accelerate the inference process of BERT through selecting a subsequence of transformer layers list of backbone as a computational path for an input sample. To do this, our approach adds a planning module to the original BERT model to determine whether a layer is included or bypassed during inference. Experimental results on the GLUE benchmark exhibit that our method reduces latency to 75\% while maintaining 98\% accuracy, yielding a better accuracy-speed trade-off compared to state-of-the-art input-adaptive methods.

Title: EEG-based Cognitive Load Classification using Feature Masked Autoencoding and Emotion Transfer Learning. (arXiv:2308.00246v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00246
• Code URL: null
• Copy Paste: [[2308.00246] EEG-based Cognitive Load Classification using Feature Masked Autoencoding and Emotion Transfer Learning](http://arxiv.org/abs/2308.00246) #transformer
• Summary:

  Cognitive load, the amount of mental effort required for task completion, plays an important role in performance and decision-making outcomes, making its classification and analysis essential in various sensitive domains. In this paper, we present a new solution for the classification of cognitive load using electroencephalogram (EEG). Our model uses a transformer architecture employing transfer learning between emotions and cognitive load. We pre-train our model using self-supervised masked autoencoding on emotion-related EEG datasets and use transfer learning with both frozen weights and fine-tuning to perform downstream cognitive load classification. To evaluate our method, we carry out a series of experiments utilizing two publicly available EEG-based emotion datasets, namely SEED and SEED-IV, for pre-training, while we use the CL-Drive dataset for downstream cognitive load classification. The results of our experiments show that our proposed approach achieves strong results and outperforms conventional single-stage fully supervised learning. Moreover, we perform detailed ablation and sensitivity studies to evaluate the impact of different aspects of our proposed solution. This research contributes to the growing body of literature in affective computing with a focus on cognitive load, and opens up new avenues for future research in the field of cross-domain transfer learning using self-supervised pre-training.

Title: Counterfactual Graph Transformer for Traffic Flow Prediction. (arXiv:2308.00391v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00391
• Code URL: null
• Copy Paste: [[2308.00391] Counterfactual Graph Transformer for Traffic Flow Prediction](http://arxiv.org/abs/2308.00391) #transformer
• Summary:

  Traffic flow prediction (TFP) is a fundamental problem of the Intelligent Transportation System (ITS), as it models the latent spatial-temporal dependency of traffic flow for potential congestion prediction. Recent graph-based models with multiple kinds of attention mechanisms have achieved promising performance. However, existing methods for traffic flow prediction tend to inherit the bias pattern from the dataset and lack interpretability. To this end, we propose a Counterfactual Graph Transformer (CGT) model with an instance-level explainer (e.g., finding the important subgraphs) specifically designed for TFP. We design a perturbation mask generator over input sensor features at the time dimension and the graph structure on the graph transformer module to obtain spatial and temporal counterfactual explanations. By searching the optimal perturbation masks on the input data feature and graph structures, we can obtain the concise and dominant data or graph edge links for the subsequent TFP task. After re-training the utilized graph transformer model after counterfactual perturbation, we can obtain improved and interpretable traffic flow prediction. Extensive results on three real-world public datasets show that CGT can produce reliable explanations and is promising for traffic flow prediction.

generative

Title: Controlling Geometric Abstraction and Texture for Artistic Images. (arXiv:2308.00148v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00148
• Code URL: https://github.com/MartinBuessemeyer/Artistic-Texture-Control
• Copy Paste: [[2308.00148] Controlling Geometric Abstraction and Texture for Artistic Images](http://arxiv.org/abs/2308.00148) #generative
• Summary:

  We present a novel method for the interactive control of geometric abstraction and texture in artistic images. Previous example-based stylization methods often entangle shape, texture, and color, while generative methods for image synthesis generally either make assumptions about the input image, such as only allowing faces or do not offer precise editing controls. By contrast, our holistic approach spatially decomposes the input into shapes and a parametric representation of high-frequency details comprising the image's texture, thus enabling independent control of color and texture. Each parameter in this representation controls painterly attributes of a pipeline of differentiable stylization filters. The proposed decoupling of shape and texture enables various options for stylistic editing, including interactive global and local adjustments of shape, stroke, and painterly attributes such as surface relief and contours. Additionally, we demonstrate optimization-based texture style-transfer in the parametric space using reference images and text prompts, as well as the training of single- and arbitrary style parameter prediction networks for real-time texture decomposition.

Title: Domain Adaptation based on Human Feedback for Enhancing Generative Model Denoising Abilities. (arXiv:2308.00307v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00307
• Code URL: null
• Copy Paste: [[2308.00307] Domain Adaptation based on Human Feedback for Enhancing Generative Model Denoising Abilities](http://arxiv.org/abs/2308.00307) #generative
• Summary:

  How can we apply human feedback into generative model? As answer of this question, in this paper, we show the method applied on denoising problem and domain adaptation using human feedback. Deep generative models have demonstrated impressive results in image denoising. However, current image denoising models often produce inappropriate results when applied to domains different from the ones they were trained on. If there are Good' andBad' result for unseen data, how to raise up quality of `Bad' result. Most methods use an approach based on generalization of model. However, these methods require target image for training or adapting unseen domain. In this paper, to adapting domain, we deal with non-target image for unseen domain, and improve specific failed image. To address this, we propose a method for fine-tuning inappropriate results generated in a different domain by utilizing human feedback. First, we train a generator to denoise images using only the noisy MNIST digit '0' images. The denoising generator trained on the source domain leads to unintended results when applied to target domain images. To achieve domain adaptation, we construct a noise-image denoising generated image data set and train a reward model predict human feedback. Finally, we fine-tune the generator on the different domain using the reward model with auxiliary loss function, aiming to transfer denoising capabilities to target domain. Our approach demonstrates the potential to efficiently fine-tune a generator trained on one domain using human feedback from another domain, thereby enhancing denoising abilities in different domains.

Title: Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?. (arXiv:2308.00189v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00189
• Code URL: null
• Copy Paste: [[2308.00189] Generative Models as a Complex Systems Science: How can we make sense of large language model behavior?](http://arxiv.org/abs/2308.00189) #generative
• Summary:

  Coaxing out desired behavior from pretrained models, while avoiding undesirable ones, has redefined NLP and is reshaping how we interact with computers. What was once a scientific engineering discipline-in which building blocks are stacked one on top of the other-is arguably already a complex systems science, in which emergent behaviors are sought out to support previously unimagined use cases.

Despite the ever increasing number of benchmarks that measure task performance, we lack explanations of what behaviors language models exhibit that allow them to complete these tasks in the first place. We argue for a systematic effort to decompose language model behavior into categories that explain cross-task performance, to guide mechanistic explanations and help future-proof analytic research.

Title: Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges. (arXiv:2308.00031v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00031
• Code URL: null
• Copy Paste: [[2308.00031] Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges](http://arxiv.org/abs/2308.00031) #generative
• Summary:

  Generative Artificial Intelligence (AI) is one of the most exciting developments in Computer Science of the last decade. At the same time, Reinforcement Learning (RL) has emerged as a very successful paradigm for a variety of machine learning tasks. In this survey, we discuss the state of the art, opportunities and open research questions in applying RL to generative AI. In particular, we will discuss three types of applications, namely, RL as an alternative way for generation without specified objectives; as a way for generating outputs while concurrently maximizing an objective function; and, finally, as a way of embedding desired characteristics, which cannot be easily captured by means of an objective function, into the generative process. We conclude the survey with an in-depth discussion of the opportunities and challenges in this fascinating emerging area.

Title: Graph Contrastive Learning with Generative Adversarial Network. (arXiv:2308.00535v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00535
• Code URL: null
• Copy Paste: [[2308.00535] Graph Contrastive Learning with Generative Adversarial Network](http://arxiv.org/abs/2308.00535) #generative
• Summary:

  Graph Neural Networks (GNNs) have demonstrated promising results on exploiting node representations for many downstream tasks through supervised end-to-end training. To deal with the widespread label scarcity issue in real-world applications, Graph Contrastive Learning (GCL) is leveraged to train GNNs with limited or even no labels by maximizing the mutual information between nodes in its augmented views generated from the original graph. However, the distribution of graphs remains unconsidered in view generation, resulting in the ignorance of unseen edges in most existing literature, which is empirically shown to be able to improve GCL's performance in our experiments. To this end, we propose to incorporate graph generative adversarial networks (GANs) to learn the distribution of views for GCL, in order to i) automatically capture the characteristic of graphs for augmentations, and ii) jointly train the graph GAN model and the GCL model. Specifically, we present GACN, a novel Generative Adversarial Contrastive learning Network for graph representation learning. GACN develops a view generator and a view discriminator to generate augmented views automatically in an adversarial style. Then, GACN leverages these views to train a GNN encoder with two carefully designed self-supervised learning losses, including the graph contrastive loss and the Bayesian personalized ranking Loss. Furthermore, we design an optimization framework to train all GACN modules jointly. Extensive experiments on seven real-world datasets show that GACN is able to generate high-quality augmented views for GCL and is superior to twelve state-of-the-art baseline methods. Noticeably, our proposed GACN surprisingly discovers that the generated views in data augmentation finally conform to the well-known preferential attachment rule in online networks.

large language model

Title: Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Models. (arXiv:2308.00675v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.00675
• Code URL: null
• Copy Paste: [[2308.00675] Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Models](http://arxiv.org/abs/2308.00675) #large language model
• Summary:

  Today, large language models (LLMs) are taught to use new tools by providing a few demonstrations of the tool's usage. Unfortunately, demonstrations are hard to acquire, and can result in undesirable biased usage if the wrong demonstration is chosen. Even in the rare scenario that demonstrations are readily available, there is no principled selection protocol to determine how many and which ones to provide. As tasks grow more complex, the selection search grows combinatorially and invariably becomes intractable. Our work provides an alternative to demonstrations: tool documentation. We advocate the use of tool documentation, descriptions for the individual tool usage, over demonstrations. We substantiate our claim through three main empirical findings on 6 tasks across both vision and language modalities. First, on existing benchmarks, zero-shot prompts with only tool documentation are sufficient for eliciting proper tool usage, achieving performance on par with few-shot prompts. Second, on a newly collected realistic tool-use dataset with hundreds of available tool APIs, we show that tool documentation is significantly more valuable than demonstrations, with zero-shot documentation significantly outperforming few-shot without documentation. Third, we highlight the benefits of tool documentations by tackling image generation and video tracking using just-released unseen state-of-the-art models as tools. Finally, we highlight the possibility of using tool documentation to automatically enable new applications: by using nothing more than the documentation of GroundingDino, Stable Diffusion, XMem, and SAM, LLMs can re-invent the functionalities of the just-released Grounded-SAM and Track Anything models.

Title: LISA: Reasoning Segmentation via Large Language Model. (arXiv:2308.00692v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00692
• Code URL: https://github.com/dvlab-research/lisa
• Copy Paste: [[2308.00692] LISA: Reasoning Segmentation via Large Language Model](http://arxiv.org/abs/2308.00692) #large language model
• Summary:

  Although perception systems have made remarkable advancements in recent years, they still rely on explicit human instruction to identify the target objects or categories before executing visual recognition tasks. Such systems lack the ability to actively reason and comprehend implicit user intentions. In this work, we propose a new segmentation task -- reasoning segmentation. The task is designed to output a segmentation mask given a complex and implicit query text. Furthermore, we establish a benchmark comprising over one thousand image-instruction pairs, incorporating intricate reasoning and world knowledge for evaluation purposes. Finally, we present LISA: large Language Instructed Segmentation Assistant, which inherits the language generation capabilities of the multi-modal Large Language Model (LLM) while also possessing the ability to produce segmentation masks. We expand the original vocabulary with a token and propose the embedding-as-mask paradigm to unlock the segmentation capability. Remarkably, LISA can handle cases involving: 1) complex reasoning; 2) world knowledge; 3) explanatory answers; 4) multi-turn conversation. Also, it demonstrates robust zero-shot capability when trained exclusively on reasoning-free datasets. In addition, fine-tuning the model with merely 239 reasoning segmentation image-instruction pairs results in further performance enhancement. Experiments show our method not only unlocks new reasoning segmentation capabilities but also proves effective in both complex reasoning segmentation and standard referring segmentation tasks. Code, models, and demo are at https://github.com/dvlab-research/LISA.

Title: Trustworthiness of Children Stories Generated by Large Language Models. (arXiv:2308.00073v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.00073
• Code URL: null
• Copy Paste: [[2308.00073] Trustworthiness of Children Stories Generated by Large Language Models](http://arxiv.org/abs/2308.00073) #large language model
• Summary:

  Large Language Models (LLMs) have shown a tremendous capacity for generating literary text. However, their effectiveness in generating children's stories has yet to be thoroughly examined. In this study, we evaluate the trustworthiness of children's stories generated by LLMs using various measures, and we compare and contrast our results with both old and new children's stories to better assess their significance. Our findings suggest that LLMs still struggle to generate children's stories at the level of quality and nuance found in actual stories

Title: A Sentence is Worth a Thousand Pictures: Can Large Language Models Understand Human Language?. (arXiv:2308.00109v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.00109
• Code URL: null
• Copy Paste: [[2308.00109] A Sentence is Worth a Thousand Pictures: Can Large Language Models Understand Human Language?](http://arxiv.org/abs/2308.00109) #large language model
• Summary:

  Artificial Intelligence applications show great potential for language-related tasks that rely on next-word prediction. The current generation of large language models have been linked to claims about human-like linguistic performance and their applications are hailed both as a key step towards Artificial General Intelligence and as major advance in understanding the cognitive, and even neural basis of human language. We analyze the contribution of large language models as theoretically informative representations of a target system vs. atheoretical powerful mechanistic tools, and we identify the key abilities that are still missing from the current state of development and exploitation of these models.

Title: Getting pwn'd by AI: Penetration Testing with Large Language Models. (arXiv:2308.00121v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.00121
• Code URL: https://github.com/ipa-lab/hackingBuddyGPT
• Copy Paste: [[2308.00121] Getting pwn'd by AI: Penetration Testing with Large Language Models](http://arxiv.org/abs/2308.00121) #large language model
• Summary:

  The field of software security testing, more specifically penetration testing, is an activity that requires high levels of expertise and involves many manual testing and analysis steps. This paper explores the potential usage of large-language models, such as GPT3.5, to augment penetration testers with AI sparring partners. We explore the feasibility of supplementing penetration testers with AI models for two distinct use cases: high-level task planning for security testing assignments and low-level vulnerability hunting within a vulnerable virtual machine. For the latter, we implemented a closed-feedback loop between LLM-generated low-level actions with a vulnerable virtual machine (connected through SSH) and allowed the LLM to analyze the machine state for vulnerabilities and suggest concrete attack vectors which were automatically executed within the virtual machine. We discuss promising initial results, detail avenues for improvement, and close deliberating on the ethics of providing AI-based sparring partners.

Title: Skills-in-Context Prompting: Unlocking Compositionality in Large Language Models. (arXiv:2308.00304v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.00304
• Code URL: null
• Copy Paste: [[2308.00304] Skills-in-Context Prompting: Unlocking Compositionality in Large Language Models](http://arxiv.org/abs/2308.00304) #large language model
• Summary:

  We consider the problem of eliciting compositional generalization capabilities in large language models (LLMs) with a novel type of prompting strategy. Compositional generalization empowers the LLMs to solve problems that are harder than the ones they have seen (i.e., easy-to-hard generalization), which is a critical reasoning capability of human-like intelligence. However, even the current state-of-the-art LLMs still struggle with this form of reasoning. To bridge this gap, we propose skills-in-context (SKiC) prompting, which instructs LLMs how to compose basic skills to resolve more complex problems. We find that it is crucial to demonstrate both the skills and the compositional examples within the same prompting context. With as few as two examplars, our SKiC prompting initiates strong synergies between skills and their composition capabilities. Notably, it empowers LLMs to solve unseen problems that require innovative skill compositions, achieving near-perfect generalization on a broad range of challenging compositionality tasks. Intriguingly, SKiC prompting unlocks the latent potential of LLMs, enabling them to leverage pre-existing internal skills acquired during earlier pretraining and alignment stages, even when these skills are not explicitly presented in the prompting context. This results in the capability of LLMs to solve unseen complex problems by activating and composing these internal competencies.

Title: Retrieval Augmented Generation and Representative Vector Summarization for large unstructured textual data in Medical Education. (arXiv:2308.00479v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.00479
• Code URL: null
• Copy Paste: [[2308.00479] Retrieval Augmented Generation and Representative Vector Summarization for large unstructured textual data in Medical Education](http://arxiv.org/abs/2308.00479) #large language model
• Summary:

  Large Language Models are increasingly being used for various tasks including content generation and as chatbots. Despite their impressive performances in general tasks, LLMs need to be aligned when applying for domain specific tasks to mitigate the problems of hallucination and producing harmful answers. Retrieval Augmented Generation (RAG) allows to easily attach and manipulate a non-parametric knowledgebases to LLMs. Applications of RAG in the field of medical education are discussed in this paper. A combined extractive and abstractive summarization method for large unstructured textual data using representative vectors is proposed.

Title: JIANG: Chinese Open Foundation Language Model. (arXiv:2308.00624v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.00624
• Code URL: null
• Copy Paste: [[2308.00624] JIANG: Chinese Open Foundation Language Model](http://arxiv.org/abs/2308.00624) #large language model
• Summary:

  With the advancements in large language model technology, it has showcased capabilities that come close to those of human beings across various tasks. This achievement has garnered significant interest from companies and scientific research institutions, leading to substantial investments in the research and development of these models. While numerous large models have emerged during this period, the majority of them have been trained primarily on English data. Although they exhibit decent performance in other languages, such as Chinese, their potential remains limited due to factors like vocabulary design and training corpus. Consequently, their ability to fully express their capabilities in Chinese falls short. To address this issue, we introduce the model named JIANG (Chinese pinyin of ginger) specifically designed for the Chinese language. We have gathered a substantial amount of Chinese corpus to train the model and have also optimized its structure. The extensive experimental results demonstrate the excellent performance of our model.

Title: CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code. (arXiv:2308.00683v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00683
• Code URL: null
• Copy Paste: [[2308.00683] CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code](http://arxiv.org/abs/2308.00683) #large language model
• Summary:

  Recent works have widely adopted large language model pretraining for source code, suggested source code-specific pretraining objectives and investigated the applicability of various Transformer-based language model architectures for source code. This work investigates another important aspect of such models, namely the effect of different subtokenization options, and aims at identifying most effective and length-efficient subtokenizations, taking into account code specifics. We propose subtokenziation that reduces average length by 17% without downstream performance drop, and show that a carefully chosen subtokenization may improve quality by 0.5-2%, possibly with some length increase.

Title: DiviML: A Module-based Heuristic for Mapping Neural Networks onto Heterogeneous Platforms. (arXiv:2308.00127v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.00127
• Code URL: null
• Copy Paste: [[2308.00127] DiviML: A Module-based Heuristic for Mapping Neural Networks onto Heterogeneous Platforms](http://arxiv.org/abs/2308.00127) #large language model
• Summary:

  Datacenters are increasingly becoming heterogeneous, and are starting to include specialized hardware for networking, video processing, and especially deep learning. To leverage the heterogeneous compute capability of modern datacenters, we develop an approach for compiler-level partitioning of deep neural networks (DNNs) onto multiple interconnected hardware devices. We present a general framework for heterogeneous DNN compilation, offering automatic partitioning and device mapping. Our scheduler integrates both an exact solver, through a mixed integer linear programming (MILP) formulation, and a modularity-based heuristic for scalability. Furthermore, we propose a theoretical lower bound formula for the optimal solution, which enables the assessment of the heuristic solutions' quality. We evaluate our scheduler in optimizing both conventional DNNs and randomly-wired neural networks, subject to latency and throughput constraints, on a heterogeneous system comprised of a CPU and two distinct GPUs. Compared to na\"ively running DNNs on the fastest GPU, he proposed framework can achieve more than 3$\times$ times lower latency and up to 2.9$\times$ higher throughput by automatically leveraging both data and model parallelism to deploy DNNs on our sample heterogeneous server node. Moreover, our modularity-based "splitting" heuristic improves the solution runtime up to 395$\times$ without noticeably sacrificing solution quality compared to an exact MILP solution, and outperforms all other heuristics by 30-60% solution quality. Finally, our case study shows how we can extend our framework to schedule large language models across multiple heterogeneous servers by exploiting symmetry in the hardware setup. Our code can be easily plugged in to existing frameworks, and is available at https://github.com/abdelfattah-lab/diviml.

segmentation

Title: Multispectral Image Segmentation in Agriculture: A Comprehensive Study on Fusion Approaches. (arXiv:2308.00159v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00159
• Code URL: https://github.com/cybonic/misagriculture
• Copy Paste: [[2308.00159] Multispectral Image Segmentation in Agriculture: A Comprehensive Study on Fusion Approaches](http://arxiv.org/abs/2308.00159) #segmentation
• Summary:

  Multispectral imagery is frequently incorporated into agricultural tasks, providing valuable support for applications such as image segmentation, crop monitoring, field robotics, and yield estimation. From an image segmentation perspective, multispectral cameras can provide rich spectral information, helping with noise reduction and feature extraction. As such, this paper concentrates on the use of fusion approaches to enhance the segmentation process in agricultural applications. More specifically, in this work, we compare different fusion approaches by combining RGB and NDVI as inputs for crop row detection, which can be useful in autonomous robots operating in the field. The inputs are used individually as well as combined at different times of the process (early and late fusion) to perform classical and DL-based semantic segmentation. In this study, two agriculture-related datasets are subjected to analysis using both deep learning (DL)-based and classical segmentation methodologies. The experiments reveal that classical segmentation methods, utilizing techniques such as edge detection and thresholding, can effectively compete with DL-based algorithms, particularly in tasks requiring precise foreground-background separation. This suggests that traditional methods retain their efficacy in certain specialized applications within the agricultural domain. Moreover, among the fusion strategies examined, late fusion emerges as the most robust approach, demonstrating superiority in adaptability and effectiveness across varying segmentation scenarios. The dataset and code is available at https://github.com/Cybonic/MISAgriculture.git.

Title: Scene Separation & Data Selection: Temporal Segmentation Algorithm for Real-Time Video Stream Analysis. (arXiv:2308.00210v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00210
• Code URL: null
• Copy Paste: [[2308.00210] Scene Separation & Data Selection: Temporal Segmentation Algorithm for Real-Time Video Stream Analysis](http://arxiv.org/abs/2308.00210) #segmentation
• Summary:

  We present 2SDS (Scene Separation and Data Selection algorithm), a temporal segmentation algorithm used in real-time video stream interpretation. It complements CNN-based models to make use of temporal information in videos. 2SDS can detect the change between scenes in a video stream by com-paring the image difference between two frames. It separates a video into segments (scenes), and by combining itself with a CNN model, 2SDS can select the optimal result for each scene. In this paper, we will be discussing some basic methods and concepts behind 2SDS, as well as presenting some preliminary experiment results regarding 2SDS. During these experiments, 2SDS has achieved an overall accuracy of over 90%.

Title: Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understanding. (arXiv:2308.00353v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00353
• Code URL: null
• Copy Paste: [[2308.00353] Lowis3D: Language-Driven Open-World Instance-Level 3D Scene Understanding](http://arxiv.org/abs/2308.00353) #segmentation
• Summary:

  Open-world instance-level scene understanding aims to locate and recognize unseen object categories that are not present in the annotated dataset. This task is challenging because the model needs to both localize novel 3D objects and infer their semantic categories. A key factor for the recent progress in 2D open-world perception is the availability of large-scale image-text pairs from the Internet, which cover a wide range of vocabulary concepts. However, this success is hard to replicate in 3D scenarios due to the scarcity of 3D-text pairs. To address this challenge, we propose to harness pre-trained vision-language (VL) foundation models that encode extensive knowledge from image-text pairs to generate captions for multi-view images of 3D scenes. This allows us to establish explicit associations between 3D shapes and semantic-rich captions. Moreover, to enhance the fine-grained visual-semantic representation learning from captions for object-level categorization, we design hierarchical point-caption association methods to learn semantic-aware embeddings that exploit the 3D geometry between 3D points and multi-view images. In addition, to tackle the localization challenge for novel classes in the open-world setting, we develop debiased instance localization, which involves training object grouping modules on unlabeled data using instance-level pseudo supervision. This significantly improves the generalization capabilities of instance grouping and thus the ability to accurately locate novel objects. We conduct extensive experiments on 3D semantic, instance, and panoptic segmentation tasks, covering indoor and outdoor scenes across three datasets. Our method outperforms baseline methods by a significant margin in semantic segmentation (e.g. 34.5%$\sim$65.3%), instance segmentation (e.g. 21.8%$\sim$54.0%) and panoptic segmentation (e.g. 14.7%$\sim$43.3%). Code will be available.

Title: Shape Completion with Prediction of Uncertain Regions. (arXiv:2308.00377v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00377
• Code URL: https://github.com/dlr-rm/shape-completion
• Copy Paste: [[2308.00377] Shape Completion with Prediction of Uncertain Regions](http://arxiv.org/abs/2308.00377) #segmentation
• Summary:

  Shape completion, i.e., predicting the complete geometry of an object from a partial observation, is highly relevant for several downstream tasks, most notably robotic manipulation. When basing planning or prediction of real grasps on object shape reconstruction, an indication of severe geometric uncertainty is indispensable. In particular, there can be an irreducible uncertainty in extended regions about the presence of entire object parts when given ambiguous object views. To treat this important case, we propose two novel methods for predicting such uncertain regions as straightforward extensions of any method for predicting local spatial occupancy, one through postprocessing occupancy scores, the other through direct prediction of an uncertainty indicator. We compare these methods together with two known approaches to probabilistic shape completion. Moreover, we generate a dataset, derived from ShapeNet, of realistically rendered depth images of object views with ground-truth annotations for the uncertain regions. We train on this dataset and test each method in shape completion and prediction of uncertain regions for known and novel object instances and on synthetic and real data. While direct uncertainty prediction is by far the most accurate in the segmentation of uncertain regions, both novel methods outperform the two baselines in shape completion and uncertain region prediction, and avoiding the predicted uncertain regions increases the quality of grasps for all tested methods. Web: https://github.com/DLR-RM/shape-completion

Title: A Satellite Imagery Dataset for Long-Term Sustainable Development in United States Cities. (arXiv:2308.00465v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00465
• Code URL: https://github.com/axin1301/satellite-imagery-dataset
• Copy Paste: [[2308.00465] A Satellite Imagery Dataset for Long-Term Sustainable Development in United States Cities](http://arxiv.org/abs/2308.00465) #segmentation
• Summary:

  Cities play an important role in achieving sustainable development goals (SDGs) to promote economic growth and meet social needs. Especially satellite imagery is a potential data source for studying sustainable urban development. However, a comprehensive dataset in the United States (U.S.) covering multiple cities, multiple years, multiple scales, and multiple indicators for SDG monitoring is lacking. To support the research on SDGs in U.S. cities, we develop a satellite imagery dataset using deep learning models for five SDGs containing 25 sustainable development indicators. The proposed dataset covers the 100 most populated U.S. cities and corresponding Census Block Groups from 2014 to 2023. Specifically, we collect satellite imagery and identify objects with state-of-the-art object detection and semantic segmentation models to observe cities' bird's-eye view. We further gather population, nighttime light, survey, and built environment data to depict SDGs regarding poverty, health, education, inequality, and living environment. We anticipate the dataset to help urban policymakers and researchers to advance SDGs-related studies, especially applying satellite imagery to monitor long-term and multi-scale SDGs in cities.

Title: MonoNext: A 3D Monocular Object Detection with ConvNext. (arXiv:2308.00596v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.00596
• Code URL: null
• Copy Paste: [[2308.00596] MonoNext: A 3D Monocular Object Detection with ConvNext](http://arxiv.org/abs/2308.00596) #segmentation
• Summary:

  Autonomous driving perception tasks rely heavily on cameras as the primary sensor for Object Detection, Semantic Segmentation, Instance Segmentation, and Object Tracking. However, RGB images captured by cameras lack depth information, which poses a significant challenge in 3D detection tasks. To supplement this missing data, mapping sensors such as LIDAR and RADAR are used for accurate 3D Object Detection. Despite their significant accuracy, the multi-sensor models are expensive and require a high computational demand. In contrast, Monocular 3D Object Detection models are becoming increasingly popular, offering a faster, cheaper, and easier-to-implement solution for 3D detections. This paper introduces a different Multi-Tasking Learning approach called MonoNext that utilizes a spatial grid to map objects in the scene. MonoNext employs a straightforward approach based on the ConvNext network and requires only 3D bounding box annotated data. In our experiments with the KITTI dataset, MonoNext achieved high precision and competitive performance comparable with state-of-the-art approaches. Furthermore, by adding more training data, MonoNext surpassed itself and achieved higher accuracies.