secure

security

privacy

Title: Differential Privacy, Linguistic Fairness, and Training Data Influence: Impossibility and Possibility Theorems for Multilingual Language Models. (arXiv:2308.08774v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.08774
• Code URL: null
• Copy Paste: [[2308.08774]] Differential Privacy, Linguistic Fairness, and Training Data Influence: Impossibility and Possibility Theorems for Multilingual Language Models(http://arxiv.org/abs/2308.08774)
• Summary:

  Language models such as mBERT, XLM-R, and BLOOM aim to achieve multilingual generalization or compression to facilitate transfer to a large number of (potentially unseen) languages. However, these models should ideally also be private, linguistically fair, and transparent, by relating their predictions to training data. Can these requirements be simultaneously satisfied? We show that multilingual compression and linguistic fairness are compatible with differential privacy, but that differential privacy is at odds with training data influence sparsity, an objective for transparency. We further present a series of experiments on two common NLP tasks and evaluate multilingual compression and training data influence sparsity under different privacy guarantees, exploring these trade-offs in more detail. Our results suggest that we need to develop ways to jointly optimize for these objectives in order to find practical trade-offs.

Title: Privacy-Preserving Detection Method for Transmission Line Based on Edge Collaboration. (arXiv:2308.08761v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.08761
• Code URL: null
• Copy Paste: [[2308.08761]] Privacy-Preserving Detection Method for Transmission Line Based on Edge Collaboration(http://arxiv.org/abs/2308.08761)
• Summary:

  Unmanned aerial vehicles (UAVs) are commonly used for edge collaborative computing in current transmission line object detection, where computationally intensive tasks generated by user nodes are offloaded to more powerful edge servers for processing. However, performing edge collaborative processing on transmission line image data may result in serious privacy breaches. To address this issue, we propose a secure single-stage detection model called SecYOLOv7 that preserves the privacy of object detecting. Based on secure multi-party computation (MPC), a series of secure computing protocols are designed for the collaborative execution of Secure Feature Contraction, Secure Bounding-Box Prediction and Secure Object Classification by two non-edge servers. Performance evaluation shows that both computational and communication overhead in this framework as well as calculation error significantly outperform existing works.

Title: APPFLx: Providing Privacy-Preserving Cross-Silo Federated Learning as a Service. (arXiv:2308.08786v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.08786
• Code URL: null
• Copy Paste: [[2308.08786]] APPFLx: Providing Privacy-Preserving Cross-Silo Federated Learning as a Service(http://arxiv.org/abs/2308.08786)
• Summary:

  Cross-silo privacy-preserving federated learning (PPFL) is a powerful tool to collaboratively train robust and generalized machine learning (ML) models without sharing sensitive (e.g., healthcare of financial) local data. To ease and accelerate the adoption of PPFL, we introduce APPFLx, a ready-to-use platform that provides privacy-preserving cross-silo federated learning as a service. APPFLx employs Globus authentication to allow users to easily and securely invite trustworthy collaborators for PPFL, implements several synchronous and asynchronous FL algorithms, streamlines the FL experiment launch process, and enables tracking and visualizing the life cycle of FL experiments, allowing domain experts and ML practitioners to easily orchestrate and evaluate cross-silo FL under one platform. APPFLx is available online at https://appflx.link

Title: Optimal Resource Allocation for U-Shaped Parallel Split Learning. (arXiv:2308.08896v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.08896
• Code URL: null
• Copy Paste: [[2308.08896]] Optimal Resource Allocation for U-Shaped Parallel Split Learning(http://arxiv.org/abs/2308.08896)
• Summary:

  Split learning (SL) has emerged as a promising approach for model training without revealing the raw data samples from the data owners. However, traditional SL inevitably leaks label privacy as the tail model (with the last layers) should be placed on the server. To overcome this limitation, one promising solution is to utilize U-shaped architecture to leave both early layers and last layers on the user side. In this paper, we develop a novel parallel U-shaped split learning and devise the optimal resource optimization scheme to improve the performance of edge networks. In the proposed framework, multiple users communicate with an edge server for SL. We analyze the end-to-end delay of each client during the training process and design an efficient resource allocation algorithm, called LSCRA, which finds the optimal computing resource allocation and split layers. Our experimental results show the effectiveness of LSCRA and that U-shaped PSL can achieve a similar performance with other SL baselines while preserving label privacy. Index Terms: U-shaped network, split learning, label privacy, resource allocation, 5G/6G edge networks.

protect

defense

Title: Towards a Practical Defense against Adversarial Attacks on Deep Learning-based Malware Detectors via Randomized Smoothing. (arXiv:2308.08906v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.08906
• Code URL: null
• Copy Paste: [[2308.08906]] Towards a Practical Defense against Adversarial Attacks on Deep Learning-based Malware Detectors via Randomized Smoothing(http://arxiv.org/abs/2308.08906)
• Summary:

  Malware detectors based on deep learning (DL) have been shown to be susceptible to malware examples that have been deliberately manipulated in order to evade detection, a.k.a. adversarial malware examples. More specifically, it has been show that deep learning detectors are vulnerable to small changes on the input file. Given this vulnerability of deep learning detectors, we propose a practical defense against adversarial malware examples inspired by randomized smoothing. In our work, instead of employing Gaussian or Laplace noise when randomizing inputs, we propose a randomized ablation-based smoothing scheme that ablates a percentage of the bytes within an executable. During training, our randomized ablation-based smoothing scheme trains a base classifier based on ablated versions of the executable files. At test time, the final classification for a given input executable is taken as the class most commonly predicted by the classifier on a set of ablated versions of the original executable. To demonstrate the suitability of our approach we have empirically evaluated the proposed ablation-based model against various state-of-the-art evasion attacks on the BODMAS dataset. Results show greater robustness and generalization capabilities to adversarial malware examples in comparison to a non-smoothed classifier.

attack

Title: A White-Box False Positive Adversarial Attack Method on Contrastive Loss-Based Offline Handwritten Signature Verification Models. (arXiv:2308.08925v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08925
• Code URL: null
• Copy Paste: [[2308.08925]] A White-Box False Positive Adversarial Attack Method on Contrastive Loss-Based Offline Handwritten Signature Verification Models(http://arxiv.org/abs/2308.08925)
• Summary:

  In this paper, we tackle the challenge of white-box false positive adversarial attacks on contrastive loss-based offline handwritten signature verification models. We propose a novel attack method that treats the attack as a style transfer between closely related but distinct writing styles. To guide the generation of deceptive images, we introduce two new loss functions that enhance the attack success rate by perturbing the Euclidean distance between the embedding vectors of the original and synthesized samples, while ensuring minimal perturbations by reducing the difference between the generated image and the original image. Our method demonstrates state-of-the-art performance in white-box attacks on contrastive loss-based offline handwritten signature verification models, as evidenced by our experiments. The key contributions of this paper include a novel false positive attack method, two new loss functions, effective style transfer in handwriting styles, and superior performance in white-box false positive attacks compared to other white-box attack methods.

Title: An Effective Deep Learning Based Multi-Class Classification of DoS and DDoS Attack Detection. (arXiv:2308.08803v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.08803
• Code URL: null
• Copy Paste: [[2308.08803]] An Effective Deep Learning Based Multi-Class Classification of DoS and DDoS Attack Detection(http://arxiv.org/abs/2308.08803)
• Summary:

  In the past few years, cybersecurity is becoming very important due to the rise in internet users. The internet attacks such as Denial of service (DoS) and Distributed Denial of Service (DDoS) attacks severely harm a website or server and make them unavailable to other users. Network Monitoring and control systems have found it challenging to identify the many classes of DoS and DDoS attacks since each operates uniquely. Hence a powerful technique is required for attack detection. Traditional machine learning techniques are inefficient in handling extensive network data and cannot extract high-level features for attack detection. Therefore, an effective deep learning-based intrusion detection system is developed in this paper for DoS and DDoS attack classification. This model includes various phases and starts with the Deep Convolutional Generative Adversarial Networks (DCGAN) based technique to address the class imbalance issue in the dataset. Then a deep learning algorithm based on ResNet-50 extracts the critical features for each class in the dataset. After that, an optimized AlexNet-based classifier is implemented for detecting the attacks separately, and the essential parameters of the classifier are optimized using the Atom search optimization algorithm. The proposed approach was evaluated on benchmark datasets, CCIDS2019 and UNSW-NB15, using key classification metrics and achieved 99.37% accuracy for the UNSW-NB15 dataset and 99.33% for the CICIDS2019 dataset. The investigational results demonstrate that the suggested approach performs superior to other competitive techniques in identifying DoS and DDoS attacks.

robust

Title: Advancements in Repetitive Action Counting: Joint-Based PoseRAC Model With Improved Performance. (arXiv:2308.08632v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08632
• Code URL: null
• Copy Paste: [[2308.08632]] Advancements in Repetitive Action Counting: Joint-Based PoseRAC Model With Improved Performance(http://arxiv.org/abs/2308.08632)
• Summary:

  Repetitive counting (RepCount) is critical in various applications, such as fitness tracking and rehabilitation. Previous methods have relied on the estimation of red-green-and-blue (RGB) frames and body pose landmarks to identify the number of action repetitions, but these methods suffer from a number of issues, including the inability to stably handle changes in camera viewpoints, over-counting, under-counting, difficulty in distinguishing between sub-actions, inaccuracy in recognizing salient poses, etc. In this paper, based on the work done by [1], we integrate joint angles with body pose landmarks to address these challenges and achieve better results than the state-of-the-art RepCount methods, with a Mean Absolute Error (MAE) of 0.211 and an Off-By-One (OBO) counting accuracy of 0.599 on the RepCount data set [2]. Comprehensive experimental results demonstrate the effectiveness and robustness of our method.

Title: Quantifying Overfitting: Introducing the Overfitting Index. (arXiv:2308.08682v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.08682
• Code URL: null
• Copy Paste: [[2308.08682]] Quantifying Overfitting: Introducing the Overfitting Index(http://arxiv.org/abs/2308.08682)
• Summary:

  In the rapidly evolving domain of machine learning, ensuring model generalizability remains a quintessential challenge. Overfitting, where a model exhibits superior performance on training data but falters on unseen data, is a recurrent concern. This paper introduces the Overfitting Index (OI), a novel metric devised to quantitatively assess a model's tendency to overfit. Through extensive experiments on the Breast Ultrasound Images Dataset (BUS) and the MNIST dataset using architectures such as MobileNet, U-Net, ResNet, Darknet, and ViT-32, we illustrate the utility and discernment of the OI. Our results underscore the variable overfitting behaviors across architectures and highlight the mitigative impact of data augmentation, especially on smaller and more specialized datasets. The ViT-32's performance on MNIST further emphasizes the robustness of certain models and the dataset's comprehensive nature. By providing an objective lens to gauge overfitting, the OI offers a promising avenue to advance model optimization and ensure real-world efficacy.

Title: MIPS-Fusion: Multi-Implicit-Submaps for Scalable and Robust Online Neural RGB-D Reconstruction. (arXiv:2308.08741v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08741
• Code URL: null
• Copy Paste: [[2308.08741]] MIPS-Fusion: Multi-Implicit-Submaps for Scalable and Robust Online Neural RGB-D Reconstruction(http://arxiv.org/abs/2308.08741)
• Summary:

  We introduce MIPS-Fusion, a robust and scalable online RGB-D reconstruction method based on a novel neural implicit representation -- multi-implicit-submap. Different from existing neural RGB-D reconstruction methods lacking either flexibility with a single neural map or scalability due to extra storage of feature grids, we propose a pure neural representation tackling both difficulties with a divide-and-conquer design. In our method, neural submaps are incrementally allocated alongside the scanning trajectory and efficiently learned with local neural bundle adjustments. The submaps can be refined individually in a back-end optimization and optimized jointly to realize submap-level loop closure. Meanwhile, we propose a hybrid tracking approach combining randomized and gradient-based pose optimizations. For the first time, randomized optimization is made possible in neural tracking with several key designs to the learning process, enabling efficient and robust tracking even under fast camera motions. The extensive evaluation demonstrates that our method attains higher reconstruction quality than the state of the arts for large-scale scenes and under fast camera motions.

Title: XVTP3D: Cross-view Trajectory Prediction Using Shared 3D Queries for Autonomous Driving. (arXiv:2308.08764v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08764
• Code URL: null
• Copy Paste: [[2308.08764]] XVTP3D: Cross-view Trajectory Prediction Using Shared 3D Queries for Autonomous Driving(http://arxiv.org/abs/2308.08764)
• Summary:

  Trajectory prediction with uncertainty is a critical and challenging task for autonomous driving. Nowadays, we can easily access sensor data represented in multiple views. However, cross-view consistency has not been evaluated by the existing models, which might lead to divergences between the multimodal predictions from different views. It is not practical and effective when the network does not comprehend the 3D scene, which could cause the downstream module in a dilemma. Instead, we predicts multimodal trajectories while maintaining cross-view consistency. We presented a cross-view trajectory prediction method using shared 3D Queries (XVTP3D). We employ a set of 3D queries shared across views to generate multi-goals that are cross-view consistent. We also proposed a random mask method and coarse-to-fine cross-attention to capture robust cross-view features. As far as we know, this is the first work that introduces the outstanding top-down paradigm in BEV detection field to a trajectory prediction problem. The results of experiments on two publicly available datasets show that XVTP3D achieved state-of-the-art performance with consistent cross-view predictions.

Title: URL: Combating Label Noise for Lung Nodule Malignancy Grading. (arXiv:2308.08772v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08772
• Code URL: null
• Copy Paste: [[2308.08772]] URL: Combating Label Noise for Lung Nodule Malignancy Grading(http://arxiv.org/abs/2308.08772)
• Summary:

  Due to the complexity of annotation and inter-annotator variability, most lung nodule malignancy grading datasets contain label noise, which inevitably degrades the performance and generalizability of models. Although researchers adopt the label-noise-robust methods to handle label noise for lung nodule malignancy grading, they do not consider the inherent ordinal relation among classes of this task. To model the ordinal relation among classes to facilitate tackling label noise in this task, we propose a Unimodal-Regularized Label-noise-tolerant (URL) framework. Our URL contains two stages, the Supervised Contrastive Learning (SCL) stage and the Memory pseudo-labels generation and Unimodal regularization (MU) stage. In the SCL stage, we select reliable samples and adopt supervised contrastive learning to learn better representations. In the MU stage, we split samples with multiple annotations into multiple samples with a single annotation and shuffle them into different batches. To handle label noise, pseudo-labels are generated using the similarity between each sample and the central feature of each class, and temporal ensembling is used to obtain memory pseudo-labels that supervise the model training. To model the ordinal relation, we introduce unimodal regularization to keep the ordinal relation among classes in the predictions. Moreover, each lung nodule is characterized by three orthographic views. Experiments conducted on the LIDC-IDRI dataset indicate the superiority of our URL over other competing methods. Code is available at https://github.com/axz520/UR.

Title: Environment Diversification with Multi-head Neural Network for Invariant Learning. (arXiv:2308.08778v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.08778
• Code URL: null
• Copy Paste: [[2308.08778]] Environment Diversification with Multi-head Neural Network for Invariant Learning(http://arxiv.org/abs/2308.08778)
• Summary:

  Neural networks are often trained with empirical risk minimization; however, it has been shown that a shift between training and testing distributions can cause unpredictable performance degradation. On this issue, a research direction, invariant learning, has been proposed to extract invariant features insensitive to the distributional changes. This work proposes EDNIL, an invariant learning framework containing a multi-head neural network to absorb data biases. We show that this framework does not require prior knowledge about environments or strong assumptions about the pre-trained model. We also reveal that the proposed algorithm has theoretical connections to recent studies discussing properties of variant and invariant features. Finally, we demonstrate that models trained with EDNIL are empirically more robust against distributional shifts.

Title: End-to-end Alternating Optimization for Real-World Blind Super Resolution. (arXiv:2308.08816v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08816
• Code URL: https://github.com/greatlog/realdan
• Copy Paste: [[2308.08816]] End-to-end Alternating Optimization for Real-World Blind Super Resolution(http://arxiv.org/abs/2308.08816)
• Summary:

  Blind Super-Resolution (SR) usually involves two sub-problems: 1) estimating the degradation of the given low-resolution (LR) image; 2) super-resolving the LR image to its high-resolution (HR) counterpart. Both problems are ill-posed due to the information loss in the degrading process. Most previous methods try to solve the two problems independently, but often fall into a dilemma: a good super-resolved HR result requires an accurate degradation estimation, which however, is difficult to be obtained without the help of original HR information. To address this issue, instead of considering these two problems independently, we adopt an alternating optimization algorithm, which can estimate the degradation and restore the SR image in a single model. Specifically, we design two convolutional neural modules, namely \textit{Restorer} and \textit{Estimator}. \textit{Restorer} restores the SR image based on the estimated degradation, and \textit{Estimator} estimates the degradation with the help of the restored SR image. We alternate these two modules repeatedly and unfold this process to form an end-to-end trainable network. In this way, both \textit{Restorer} and \textit{Estimator} could get benefited from the intermediate results of each other, and make each sub-problem easier. Moreover, \textit{Restorer} and \textit{Estimator} are optimized in an end-to-end manner, thus they could get more tolerant of the estimation deviations of each other and cooperate better to achieve more robust and accurate final results. Extensive experiments on both synthetic datasets and real-world images show that the proposed method can largely outperform state-of-the-art methods and produce more visually favorable results. The codes are rleased at \url{https://github.com/greatlog/RealDAN.git}.

Title: MV-ROPE: Multi-view Constraints for Robust Category-level Object Pose and Size Estimation. (arXiv:2308.08856v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08856
• Code URL: https://github.com/greatoyster/mv-rope
• Copy Paste: [[2308.08856]] MV-ROPE: Multi-view Constraints for Robust Category-level Object Pose and Size Estimation(http://arxiv.org/abs/2308.08856)
• Summary:

  We propose a novel framework for RGB-based category-level 6D object pose and size estimation. Our approach relies on the prediction of normalized object coordinate space (NOCS), which serves as an efficient and effective object canonical representation that can be extracted from RGB images. Unlike previous approaches that heavily relied on additional depth readings as input, our novelty lies in leveraging multi-view information, which is commonly available in practical scenarios where a moving camera continuously observes the environment. By introducing multi-view constraints, we can obtain accurate camera pose and depth estimation from a monocular dense SLAM framework. Additionally, by incorporating constraints on the camera relative pose, we can apply trimming strategies and robust pose averaging on the multi-view object poses, resulting in more accurate and robust estimations of category-level object poses even in the absence of direct depth readings. Furthermore, we introduce a novel NOCS prediction network that significantly improves performance. Our experimental results demonstrate the strong performance of our proposed method, even comparable to state-of-the-art RGB-D methods across public dataset sequences. Additionally, we showcase the generalization ability of our method by evaluating it on self-collected datasets.

Title: Text-Only Training for Visual Storytelling. (arXiv:2308.08881v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08881
• Code URL: null
• Copy Paste: [[2308.08881]] Text-Only Training for Visual Storytelling(http://arxiv.org/abs/2308.08881)
• Summary:

  Visual storytelling aims to generate a narrative based on a sequence of images, necessitating both vision-language alignment and coherent story generation. Most existing solutions predominantly depend on paired image-text training data, which can be costly to collect and challenging to scale. To address this, we formulate visual storytelling as a visual-conditioned story generation problem and propose a text-only training method that separates the learning of cross-modality alignment and story generation. Our approach specifically leverages the cross-modality pre-trained CLIP model to integrate visual control into a story generator, trained exclusively on text data. Moreover, we devise a training-free visual condition planner that accounts for the temporal structure of the input image sequence while balancing global and local visual content. The distinctive advantage of requiring only text data for training enables our method to learn from external text story data, enhancing the generalization capability of visual storytelling. We conduct extensive experiments on the VIST benchmark, showcasing the effectiveness of our approach in both in-domain and cross-domain settings. Further evaluations on expression diversity and human assessment underscore the superiority of our method in terms of informativeness and robustness.

Title: Automatic Signboard Recognition in Low Quality Night Images. (arXiv:2308.08941v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08941
• Code URL: null
• Copy Paste: [[2308.08941]] Automatic Signboard Recognition in Low Quality Night Images(http://arxiv.org/abs/2308.08941)
• Summary:

  An essential requirement for driver assistance systems and autonomous driving technology is implementing a robust system for detecting and recognizing traffic signs. This system enables the vehicle to autonomously analyze the environment and make appropriate decisions regarding its movement, even when operating at higher frame rates. However, traffic sign images captured in inadequate lighting and adverse weather conditions are poorly visible, blurred, faded, and damaged. Consequently, the recognition of traffic signs in such circumstances becomes inherently difficult. This paper addressed the challenges of recognizing traffic signs from images captured in low light, noise, and blurriness. To achieve this goal, a two-step methodology has been employed. The first step involves enhancing traffic sign images by applying a modified MIRNet model and producing enhanced images. In the second step, the Yolov4 model recognizes the traffic signs in an unconstrained environment. The proposed method has achieved 5.40% increment in mAP@0.5 for low quality images on Yolov4. The overall mAP@0.5 of 96.75% has been achieved on the GTSRB dataset. It has also attained mAP@0.5 of 100% on the GTSDB dataset for the broad categories, comparable with the state-of-the-art work.

Title: Auxiliary Tasks Benefit 3D Skeleton-based Human Motion Prediction. (arXiv:2308.08942v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08942
• Code URL: https://github.com/mediabrain-sjtu/auxformer
• Copy Paste: [[2308.08942]] Auxiliary Tasks Benefit 3D Skeleton-based Human Motion Prediction(http://arxiv.org/abs/2308.08942)
• Summary:

  Exploring spatial-temporal dependencies from observed motions is one of the core challenges of human motion prediction. Previous methods mainly focus on dedicated network structures to model the spatial and temporal dependencies. This paper considers a new direction by introducing a model learning framework with auxiliary tasks. In our auxiliary tasks, partial body joints' coordinates are corrupted by either masking or adding noise and the goal is to recover corrupted coordinates depending on the rest coordinates. To work with auxiliary tasks, we propose a novel auxiliary-adapted transformer, which can handle incomplete, corrupted motion data and achieve coordinate recovery via capturing spatial-temporal dependencies. Through auxiliary tasks, the auxiliary-adapted transformer is promoted to capture more comprehensive spatial-temporal dependencies among body joints' coordinates, leading to better feature learning. Extensive experimental results have shown that our method outperforms state-of-the-art methods by remarkable margins of 7.2%, 3.7%, and 9.4% in terms of 3D mean per joint position error (MPJPE) on the Human3.6M, CMU Mocap, and 3DPW datasets, respectively. We also demonstrate that our method is more robust under data missing cases and noisy data cases. Code is available at https://github.com/MediaBrain-SJTU/AuxFormer.

Title: Linguistically-Informed Neural Architectures for Lexical, Syntactic and Semantic Tasks in Sanskrit. (arXiv:2308.08807v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.08807
• Code URL: null
• Copy Paste: [[2308.08807]] Linguistically-Informed Neural Architectures for Lexical, Syntactic and Semantic Tasks in Sanskrit(http://arxiv.org/abs/2308.08807)
• Summary:

  The primary focus of this thesis is to make Sanskrit manuscripts more accessible to the end-users through natural language technologies. The morphological richness, compounding, free word orderliness, and low-resource nature of Sanskrit pose significant challenges for developing deep learning solutions. We identify four fundamental tasks, which are crucial for developing a robust NLP technology for Sanskrit: word segmentation, dependency parsing, compound type identification, and poetry analysis. The first task, Sanskrit Word Segmentation (SWS), is a fundamental text processing task for any other downstream applications. However, it is challenging due to the sandhi phenomenon that modifies characters at word boundaries. Similarly, the existing dependency parsing approaches struggle with morphologically rich and low-resource languages like Sanskrit. Compound type identification is also challenging for Sanskrit due to the context-sensitive semantic relation between components. All these challenges result in sub-optimal performance in NLP applications like question answering and machine translation. Finally, Sanskrit poetry has not been extensively studied in computational linguistics.

  While addressing these challenges, this thesis makes various contributions: (1) The thesis proposes linguistically-informed neural architectures for these tasks. (2) We showcase the interpretability and multilingual extension of the proposed systems. (3) Our proposed systems report state-of-the-art performance. (4) Finally, we present a neural toolkit named SanskritShala, a web-based application that provides real-time analysis of input for various NLP tasks. Overall, this thesis contributes to making Sanskrit manuscripts more accessible by developing robust NLP technology and releasing various resources, datasets, and web-based toolkit.

Title: Dynamic Neural Network is All You Need: Understanding the Robustness of Dynamic Mechanisms in Neural Networks. (arXiv:2308.08709v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.08709
• Code URL: https://github.com/anonymous2015258/Early_Attack
• Copy Paste: [[2308.08709]] Dynamic Neural Network is All You Need: Understanding the Robustness of Dynamic Mechanisms in Neural Networks(http://arxiv.org/abs/2308.08709)
• Summary:

  Deep Neural Networks (DNNs) have been used to solve different day-to-day problems. Recently, DNNs have been deployed in real-time systems, and lowering the energy consumption and response time has become the need of the hour. To address this scenario, researchers have proposed incorporating dynamic mechanism to static DNNs (SDNN) to create Dynamic Neural Networks (DyNNs) performing dynamic amounts of computation based on the input complexity. Although incorporating dynamic mechanism into SDNNs would be preferable in real-time systems, it also becomes important to evaluate how the introduction of dynamic mechanism impacts the robustness of the models. However, there has not been a significant number of works focusing on the robustness trade-off between SDNNs and DyNNs. To address this issue, we propose to investigate the robustness of dynamic mechanism in DyNNs and how dynamic mechanism design impacts the robustness of DyNNs. For that purpose, we evaluate three research questions. These evaluations are performed on three models and two datasets. Through the studies, we find that attack transferability from DyNNs to SDNNs is higher than attack transferability from SDNNs to DyNNs. Also, we find that DyNNs can be used to generate adversarial samples more efficiently than SDNNs. Then, through research studies, we provide insight into the design choices that can increase robustness of DyNNs against the attack generated using static model. Finally, we propose a novel attack to understand the additional attack surface introduced by the dynamic mechanism and provide design choices to improve robustness against the attack.

Title: Causal Adversarial Perturbations for Individual Fairness and Robustness in Heterogeneous Data Spaces. (arXiv:2308.08938v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.08938
• Code URL: null
• Copy Paste: [[2308.08938]] Causal Adversarial Perturbations for Individual Fairness and Robustness in Heterogeneous Data Spaces(http://arxiv.org/abs/2308.08938)
• Summary:

  As responsible AI gains importance in machine learning algorithms, properties such as fairness, adversarial robustness, and causality have received considerable attention in recent years. However, despite their individual significance, there remains a critical gap in simultaneously exploring and integrating these properties. In this paper, we propose a novel approach that examines the relationship between individual fairness, adversarial robustness, and structural causal models in heterogeneous data spaces, particularly when dealing with discrete sensitive attributes. We use causal structural models and sensitive attributes to create a fair metric and apply it to measure semantic similarity among individuals. By introducing a novel causal adversarial perturbation and applying adversarial training, we create a new regularizer that combines individual fairness, causality, and robustness in the classifier. Our method is evaluated on both real-world and synthetic datasets, demonstrating its effectiveness in achieving an accurate classifier that simultaneously exhibits fairness, adversarial robustness, and causal awareness.

biometric

Title: Deep Ear Biometrics for Gender Classification. (arXiv:2308.08797v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08797
• Code URL: null
• Copy Paste: [[2308.08797]] Deep Ear Biometrics for Gender Classification(http://arxiv.org/abs/2308.08797)
• Summary:

  Human gender classification based on biometric features is a major concern for computer vision due to its vast variety of applications. The human ear is popular among researchers as a soft biometric trait, because it is less affected by age or changing circumstances, and is non-intrusive. In this study, we have developed a deep convolutional neural network (CNN) model for automatic gender classification using the samples of ear images. The performance is evaluated using four cutting-edge pre-trained CNN models. In terms of trainable parameters, the proposed technique requires significantly less computational complexity. The proposed model has achieved 93% accuracy on the EarVN1.0 ear dataset.

steal

extraction

Title: Point-aware Interaction and CNN-induced Refinement Network for RGB-D Salient Object Detection. (arXiv:2308.08930v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08930
• Code URL: https://github.com/rmcong/picr-net_acmmm23
• Copy Paste: [[2308.08930]] Point-aware Interaction and CNN-induced Refinement Network for RGB-D Salient Object Detection(http://arxiv.org/abs/2308.08930)
• Summary:

  By integrating complementary information from RGB image and depth map, the ability of salient object detection (SOD) for complex and challenging scenes can be improved. In recent years, the important role of Convolutional Neural Networks (CNNs) in feature extraction and cross-modality interaction has been fully explored, but it is still insufficient in modeling global long-range dependencies of self-modality and cross-modality. To this end, we introduce CNNs-assisted Transformer architecture and propose a novel RGB-D SOD network with Point-aware Interaction and CNN-induced Refinement (PICR-Net). On the one hand, considering the prior correlation between RGB modality and depth modality, an attention-triggered cross-modality point-aware interaction (CmPI) module is designed to explore the feature interaction of different modalities with positional constraints. On the other hand, in order to alleviate the block effect and detail destruction problems brought by the Transformer naturally, we design a CNN-induced refinement (CNNR) unit for content refinement and supplementation. Extensive experiments on five RGB-D SOD datasets show that the proposed network achieves competitive results in both quantitative and qualitative comparisons.

Title: Enhancing Phrase Representation by Information Bottleneck Guided Text Diffusion Process for Keyphrase Extraction. (arXiv:2308.08739v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.08739
• Code URL: null
• Copy Paste: [[2308.08739]] Enhancing Phrase Representation by Information Bottleneck Guided Text Diffusion Process for Keyphrase Extraction(http://arxiv.org/abs/2308.08739)
• Summary:

  Keyphrase extraction (KPE) is an important task in Natural Language Processing for many scenarios, which aims to extract keyphrases that are present in a given document. Many existing supervised methods treat KPE as sequential labeling, span-level classification, or generative tasks. However, these methods lack the ability to utilize keyphrase information, which may result in biased results. In this study, we propose Diff-KPE, which leverages the supervised Variational Information Bottleneck (VIB) to guide the text diffusion process for generating enhanced keyphrase representations. Diff-KPE first generates the desired keyphrase embeddings conditioned on the entire document and then injects the generated keyphrase embeddings into each phrase representation. A ranking network and VIB are then optimized together with rank loss and classification loss, respectively. This design of Diff-KPE allows us to rank each candidate phrase by utilizing both the information of keyphrases and the document. Experiments show that Diff-KPE outperforms existing KPE methods on a large open domain keyphrase extraction benchmark, OpenKP, and a scientific domain dataset, KP20K.

membership infer

federate

Title: FedPop: Federated Population-based Hyperparameter Tuning. (arXiv:2308.08634v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.08634
• Code URL: null
• Copy Paste: [[2308.08634]] FedPop: Federated Population-based Hyperparameter Tuning(http://arxiv.org/abs/2308.08634)
• Summary:

  Federated Learning (FL) is a distributed machine learning (ML) paradigm, in which multiple clients collaboratively train ML models without centralizing their local data. Similar to conventional ML pipelines, the client local optimization and server aggregation procedure in FL are sensitive to the hyperparameter (HP) selection. Despite extensive research on tuning HPs for centralized ML, these methods yield suboptimal results when employed in FL. This is mainly because their "training-after-tuning" framework is unsuitable for FL with limited client computation power. While some approaches have been proposed for HP-Tuning in FL, they are limited to the HPs for client local updates. In this work, we propose a novel HP-tuning algorithm, called Federated Population-based Hyperparameter Tuning (FedPop), to address this vital yet challenging problem. FedPop employs population-based evolutionary algorithms to optimize the HPs, which accommodates various HP types at both client and server sides. Compared with prior tuning methods, FedPop employs an online "tuning-while-training" framework, offering computational efficiency and enabling the exploration of a broader HP search space. Our empirical validation on the common FL benchmarks and complex real-world FL datasets demonstrates the effectiveness of the proposed method, which substantially outperforms the concurrent state-of-the-art HP tuning methods for FL.

Title: Towards Personalized Federated Learning via Heterogeneous Model Reassembly. (arXiv:2308.08643v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.08643
• Code URL: null
• Copy Paste: [[2308.08643]] Towards Personalized Federated Learning via Heterogeneous Model Reassembly(http://arxiv.org/abs/2308.08643)
• Summary:

  This paper focuses on addressing the practical yet challenging problem of model heterogeneity in federated learning, where clients possess models with different network structures. To track this problem, we propose a novel framework called pFedHR, which leverages heterogeneous model reassembly to achieve personalized federated learning. In particular, we approach the problem of heterogeneous model personalization as a model-matching optimization task on the server side. Moreover, pFedHR automatically and dynamically generates informative and diverse personalized candidates with minimal human intervention. Furthermore, our proposed heterogeneous model reassembly technique mitigates the adverse impact introduced by using public data with different distributions from the client data to a certain extent. Experimental results demonstrate that pFedHR outperforms baselines on three datasets under both IID and Non-IID settings. Additionally, pFedHR effectively reduces the adverse impact of using different public data and dynamically generates diverse personalized models in an automated manner.

Title: Controlling Federated Learning for Covertness. (arXiv:2308.08825v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.08825
• Code URL: null
• Copy Paste: [[2308.08825]] Controlling Federated Learning for Covertness(http://arxiv.org/abs/2308.08825)
• Summary:

  A learner aims to minimize a function $f$ by repeatedly querying a distributed oracle that provides noisy gradient evaluations. At the same time, the learner seeks to hide $\arg\min f$ from a malicious eavesdropper that observes the learner's queries. This paper considers the problem of \textit{covert} or \textit{learner-private} optimization, where the learner has to dynamically choose between learning and obfuscation by exploiting the stochasticity. The problem of controlling the stochastic gradient algorithm for covert optimization is modeled as a Markov decision process, and we show that the dynamic programming operator has a supermodular structure implying that the optimal policy has a monotone threshold structure. A computationally efficient policy gradient algorithm is proposed to search for the optimal querying policy without knowledge of the transition probabilities. As a practical application, our methods are demonstrated on a hate speech classification task in a federated setting where an eavesdropper can use the optimal weights to generate toxic content, which is more easily misclassified. Numerical results show that when the learner uses the optimal policy, an eavesdropper can only achieve a validation accuracy of $52\%$ with no information and $69\%$ when it has a public dataset with 10\% positive samples compared to $83\%$ when the learner employs a greedy policy.

fair

Title: Fair GANs through model rebalancing with synthetic data. (arXiv:2308.08638v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08638
• Code URL: null
• Copy Paste: [[2308.08638]] Fair GANs through model rebalancing with synthetic data(http://arxiv.org/abs/2308.08638)
• Summary:

  Deep generative models require large amounts of training data. This often poses a problem as the collection of datasets can be expensive and difficult, in particular datasets that are representative of the appropriate underlying distribution (e.g. demographic). This introduces biases in datasets which are further propagated in the models. We present an approach to mitigate biases in an existing generative adversarial network by rebalancing the model distribution. We do so by generating balanced data from an existing unbalanced deep generative model using latent space exploration and using this data to train a balanced generative model. Further, we propose a bias mitigation loss function that shows improvements in the fairness metric even when trained with unbalanced datasets. We show results for the Stylegan2 models while training on the FFHQ dataset for racial fairness and see that the proposed approach improves on the fairness metric by almost 5 times, whilst maintaining image quality. We further validate our approach by applying it to an imbalanced Cifar-10 dataset. Lastly, we argue that the traditionally used image quality metrics such as Frechet inception distance (FID) are unsuitable for bias mitigation problems.

interpretability

Title: Discrete Prompt Compression with Reinforcement Learning. (arXiv:2308.08758v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.08758
• Code URL: null
• Copy Paste: [[2308.08758]] Discrete Prompt Compression with Reinforcement Learning(http://arxiv.org/abs/2308.08758)
• Summary:

  Instruction-tuned Language Models (LMs) are widely used by users to address various problems with task-specific prompts. Constraints associated with the context window length and computational costs encourage the development of compressed prompts. Existing methods rely heavily on training embeddings, which are designed to accommodate multiple token meanings. This presents challenges in terms of interpretability, a fixed number of embedding tokens, reusability across different LMs, and inapplicability when interacting with black-box APIs. This study proposes prompt compression with reinforcement learning (PCRL), a novel discrete prompt compression method that addresses these issues. PCRL employs a computationally efficient policy network that directly edits prompts. The PCRL training approach can be flexibly applied to various types of LMs, as well as decoder-only and encoder-decoder architecture, and can be trained without gradient access to LMs or labeled data. PCRL achieves an average reduction of 24.6% in token count across various instruction prompts while preserving performance. Further, we demonstrate that the learned policy can be transferred to larger LMs, and through various analyses, we aid the understanding of token importance within prompts.

explainability

watermark

diffusion

Title: Learning A Coarse-to-Fine Diffusion Transformer for Image Restoration. (arXiv:2308.08730v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08730
• Code URL: https://github.com/wlydlut/c2f-dft
• Copy Paste: [[2308.08730]] Learning A Coarse-to-Fine Diffusion Transformer for Image Restoration(http://arxiv.org/abs/2308.08730)
• Summary:

  Recent years have witnessed the remarkable performance of diffusion models in various vision tasks. However, for image restoration that aims to recover clear images with sharper details from given degraded observations, diffusion-based methods may fail to recover promising results due to inaccurate noise estimation. Moreover, simple constraining noises cannot effectively learn complex degradation information, which subsequently hinders the model capacity. To solve the above problems, we propose a coarse-to-fine diffusion Transformer (C2F-DFT) for image restoration. Specifically, our C2F-DFT contains diffusion self-attention (DFSA) and diffusion feed-forward network (DFN) within a new coarse-to-fine training scheme. The DFSA and DFN respectively capture the long-range diffusion dependencies and learn hierarchy diffusion representation to facilitate better restoration. In the coarse training stage, our C2F-DFT estimates noises and then generates the final clean image by a sampling algorithm. To further improve the restoration quality, we propose a simple yet effective fine training scheme. It first exploits the coarse-trained diffusion model with fixed steps to generate restoration results, which then would be constrained with corresponding ground-truth ones to optimize the models to remedy the unsatisfactory results affected by inaccurate noise estimation. Extensive experiments show that C2F-DFT significantly outperforms diffusion-based restoration method IR-SDE and achieves competitive performance compared with Transformer-based state-of-the-art methods on $3$ tasks, including deraining, deblurring, and real denoising.

Title: Watch Your Steps: Local Image and Scene Editing by Text Instructions. (arXiv:2308.08947v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08947
• Code URL: null
• Copy Paste: [[2308.08947]] Watch Your Steps: Local Image and Scene Editing by Text Instructions(http://arxiv.org/abs/2308.08947)
• Summary:

  Denoising diffusion models have enabled high-quality image generation and editing. We present a method to localize the desired edit region implicit in a text instruction. We leverage InstructPix2Pix (IP2P) and identify the discrepancy between IP2P predictions with and without the instruction. This discrepancy is referred to as the relevance map. The relevance map conveys the importance of changing each pixel to achieve the edits, and is used to to guide the modifications. This guidance ensures that the irrelevant pixels remain unchanged. Relevance maps are further used to enhance the quality of text-guided editing of 3D scenes in the form of neural radiance fields. A field is trained on relevance maps of training views, denoted as the relevance field, defining the 3D region within which modifications should be made. We perform iterative updates on the training views guided by rendered relevance maps from the relevance field. Our method achieves state-of-the-art performance on both image and NeRF editing tasks. Project page: https://ashmrz.github.io/WatchYourSteps/

noise learning

data-free

Title: Feature Enforcing PINN (FE-PINN): A Framework to Learn the Underlying-Physics Features Before Target Task. (arXiv:2308.08873v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.08873
• Code URL: null
• Copy Paste: [[2308.08873]] Feature Enforcing PINN (FE-PINN): A Framework to Learn the Underlying-Physics Features Before Target Task(http://arxiv.org/abs/2308.08873)
• Summary:

  In this work, a new data-free framework called Feature Enforcing Physics Informed Neural Network (FE-PINN) is introduced. This framework is capable of learning the underlying pattern of any problem with low computational cost before the main training loop. The loss function of vanilla PINN due to the existence of two terms of partial differential residuals and boundary condition mean squared error is imbalanced. FE-PINN solves this challenge with just one minute of training instead of time-consuming hyperparameter tuning for loss function that can take hours. The FE-PINN accomplishes this process by performing a sequence of sub-tasks. The first sub-task learns useful features about the underlying physics. Then, the model trains on the target task to refine the calculations. FE-PINN is applied to three benchmarks, flow over a cylinder, 2D heat conduction, and an inverse problem of calculating inlet velocity. FE-PINN can solve each case with, 15x, 2x, and 5x speed up accordingly. Another advantage of FE-PINN is that reaching lower order of value for loss function is systematically possible. In this study, it was possible to reach a loss value near 1e-5 which is challenging for vanilla PINN. FE-PINN also has a smooth convergence process which allows for utilizing higher learning rates in comparison to vanilla PINN. This framework can be used as a fast, accurate tool for solving a wide range of Partial Differential Equations (PDEs) across various fields.

transformer

Title: SkinDistilViT: Lightweight Vision Transformer for Skin Lesion Classification. (arXiv:2308.08669v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08669
• Code URL: null
• Copy Paste: [[2308.08669]] SkinDistilViT: Lightweight Vision Transformer for Skin Lesion Classification(http://arxiv.org/abs/2308.08669)
• Summary:

  Skin cancer is a treatable disease if discovered early. We provide a production-specific solution to the skin cancer classification problem that matches human performance in melanoma identification by training a vision transformer on melanoma medical images annotated by experts. Since inference cost, both time and memory wise is important in practice, we employ knowledge distillation to obtain a model that retains 98.33% of the teacher's balanced multi-class accuracy, at a fraction of the cost. Memory-wise, our model is 49.60% smaller than the teacher. Time-wise, our solution is 69.25% faster on GPU and 97.96% faster on CPU. By adding classification heads at each level of the transformer and employing a cascading distillation process, we improve the balanced multi-class accuracy of the base model by 2.1%, while creating a range of models of various sizes but comparable performance. We provide the code at https://github.com/Longman-Stan/SkinDistilVit.

Title: Long-Range Grouping Transformer for Multi-View 3D Reconstruction. (arXiv:2308.08724v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08724
• Code URL: null
• Copy Paste: [[2308.08724]] Long-Range Grouping Transformer for Multi-View 3D Reconstruction(http://arxiv.org/abs/2308.08724)
• Summary:

  Nowadays, transformer networks have demonstrated superior performance in many computer vision tasks. In a multi-view 3D reconstruction algorithm following this paradigm, self-attention processing has to deal with intricate image tokens including massive information when facing heavy amounts of view input. The curse of information content leads to the extreme difficulty of model learning. To alleviate this problem, recent methods compress the token number representing each view or discard the attention operations between the tokens from different views. Obviously, they give a negative impact on performance. Therefore, we propose long-range grouping attention (LGA) based on the divide-and-conquer principle. Tokens from all views are grouped for separate attention operations. The tokens in each group are sampled from all views and can provide macro representation for the resided view. The richness of feature learning is guaranteed by the diversity among different groups. An effective and efficient encoder can be established which connects inter-view features using LGA and extract intra-view features using the standard self-attention layer. Moreover, a novel progressive upsampling decoder is also designed for voxel generation with relatively high resolution. Hinging on the above, we construct a powerful transformer-based network, called LRGT. Experimental results on ShapeNet verify our method achieves SOTA accuracy in multi-view reconstruction. Code will be available at https://github.com/LiyingCV/Long-Range-Grouping-Transformer.

Title: BOTT: Box Only Transformer Tracker for 3D Object Tracking. (arXiv:2308.08753v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08753
• Code URL: null
• Copy Paste: [[2308.08753]] BOTT: Box Only Transformer Tracker for 3D Object Tracking(http://arxiv.org/abs/2308.08753)
• Summary:

  Tracking 3D objects is an important task in autonomous driving. Classical Kalman Filtering based methods are still the most popular solutions. However, these methods require handcrafted designs in motion modeling and can not benefit from the growing data amounts. In this paper, Box Only Transformer Tracker (BOTT) is proposed to learn to link 3D boxes of the same object from the different frames, by taking all the 3D boxes in a time window as input. Specifically, transformer self-attention is applied to exchange information between all the boxes to learn global-informative box embeddings. The similarity between these learned embeddings can be used to link the boxes of the same object. BOTT can be used for both online and offline tracking modes seamlessly. Its simplicity enables us to significantly reduce engineering efforts required by traditional Kalman Filtering based methods. Experiments show BOTT achieves competitive performance on two largest 3D MOT benchmarks: 69.9 and 66.7 AMOTA on nuScenes validation and test splits, respectively, 56.45 and 59.57 MOTA L2 on Waymo Open Dataset validation and test splits, respectively. This work suggests that tracking 3D objects by learning features directly from 3D boxes using transformers is a simple yet effective way.

Title: Realistic Full-Body Tracking from Sparse Observations via Joint-Level Modeling. (arXiv:2308.08855v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08855
• Code URL: null
• Copy Paste: [[2308.08855]] Realistic Full-Body Tracking from Sparse Observations via Joint-Level Modeling(http://arxiv.org/abs/2308.08855)
• Summary:

  To bridge the physical and virtual worlds for rapidly developed VR/AR applications, the ability to realistically drive 3D full-body avatars is of great significance. Although real-time body tracking with only the head-mounted displays (HMDs) and hand controllers is heavily under-constrained, a carefully designed end-to-end neural network is of great potential to solve the problem by learning from large-scale motion data. To this end, we propose a two-stage framework that can obtain accurate and smooth full-body motions with the three tracking signals of head and hands only. Our framework explicitly models the joint-level features in the first stage and utilizes them as spatiotemporal tokens for alternating spatial and temporal transformer blocks to capture joint-level correlations in the second stage. Furthermore, we design a set of loss terms to constrain the task of a high degree of freedom, such that we can exploit the potential of our joint-level modeling. With extensive experiments on the AMASS motion dataset and real-captured data, we validate the effectiveness of our designs and show our proposed method can achieve more accurate and smooth motion compared to existing approaches.

Title: Decoding Emotions: A comprehensive Multilingual Study of Speech Models for Speech Emotion Recognition. (arXiv:2308.08713v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.08713
• Code URL: https://github.com/95anantsingh/decoding-emotions
• Copy Paste: [[2308.08713]] Decoding Emotions: A comprehensive Multilingual Study of Speech Models for Speech Emotion Recognition(http://arxiv.org/abs/2308.08713)
• Summary:

  Recent advancements in transformer-based speech representation models have greatly transformed speech processing. However, there has been limited research conducted on evaluating these models for speech emotion recognition (SER) across multiple languages and examining their internal representations. This article addresses these gaps by presenting a comprehensive benchmark for SER with eight speech representation models and six different languages. We conducted probing experiments to gain insights into inner workings of these models for SER. We find that using features from a single optimal layer of a speech model reduces the error rate by 32\% on average across seven datasets when compared to systems where features from all layers of speech models are used. We also achieve state-of-the-art results for German and Persian languages. Our probing results indicate that the middle layers of speech models capture the most important emotional information for speech emotion recognition.

Title: PMET: Precise Model Editing in a Transformer. (arXiv:2308.08742v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.08742
• Code URL: null
• Copy Paste: [[2308.08742]] PMET: Precise Model Editing in a Transformer(http://arxiv.org/abs/2308.08742)
• Summary:

  Model editing techniques modify a minor proportion of knowledge in Large Language Models (LLMs) at a relatively low cost, which have demonstrated notable success. Existing methods assume Transformer Layer (TL) hidden states are values of key-value memories of the Feed-Forward Network (FFN). They usually optimize the TL hidden states to memorize target knowledge and use it to update the weights of the FFN in LLMs. However, the information flow of TL hidden states comes from three parts: Multi-Head Self-Attention (MHSA), FFN, and residual connections. Existing methods neglect the fact that the TL hidden states contains information not specifically required for FFN. Consequently, the performance of model editing decreases. To achieve more precise model editing, we analyze hidden states of MHSA and FFN, finding that MHSA encodes certain general knowledge extraction patterns. This implies that MHSA weights do not require updating when new knowledge is introduced. Based on above findings, we introduce PMET, which simultaneously optimizes Transformer Component (TC, namely MHSA and FFN) hidden states, while only using the optimized TC hidden states of FFN to precisely update FFN weights. Our experiments demonstrate that PMET exhibits state-of-the-art performance on both the \textsc{counterfact} and zsRE datasets. Our ablation experiments substantiate the effectiveness of our enhancements, further reinforcing the finding that the MHSA encodes certain general knowledge extraction patterns and indicating its storage of a small amount of factual knowledge. Our code is available at \url{https://github.com/xpq-tech/PMET.git}.

Title: Exploring Demonstration Ensembling for In-context Learning. (arXiv:2308.08780v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.08780
• Code URL: https://github.com/mukhal/icl-ensembling
• Copy Paste: [[2308.08780]] Exploring Demonstration Ensembling for In-context Learning(http://arxiv.org/abs/2308.08780)
• Summary:

  In-context learning (ICL) operates by showing language models (LMs) examples of input-output pairs for a given task, i.e., demonstrations. The standard approach for ICL is to prompt the LM with concatenated demonstrations followed by the test input. This approach suffers from some issues. First, concatenation offers almost no control over the contribution of each demo to the model prediction. This can be sub-optimal when some demonstrations are irrelevant to the test example. Second, due to the input length limit of some transformer models, it might be infeasible to fit many examples into the context, especially when dealing with long-input tasks. In this work, we explore Demonstration Ensembling (DENSE) as an alternative to simple concatenation. \model predicts outputs using subsets (i.e., buckets) of the demonstrations and then combines the output probabilities resulting from each subset to produce the final prediction. We study different ensembling methods using GPT-j and experiment on 12 language tasks. Our experiments show weighted max ensembling to outperform vanilla concatenation by as large as 2.4 average points. Code available at \url{https://github.com/mukhal/icl-ensembling}.

Title: Factuality Detection using Machine Translation -- a Use Case for German Clinical Text. (arXiv:2308.08827v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.08827
• Code URL: null
• Copy Paste: [[2308.08827]] Factuality Detection using Machine Translation -- a Use Case for German Clinical Text(http://arxiv.org/abs/2308.08827)
• Summary:

  Factuality can play an important role when automatically processing clinical text, as it makes a difference if particular symptoms are explicitly not present, possibly present, not mentioned, or affirmed. In most cases, a sufficient number of examples is necessary to handle such phenomena in a supervised machine learning setting. However, as clinical text might contain sensitive information, data cannot be easily shared. In the context of factuality detection, this work presents a simple solution using machine translation to translate English data to German to train a transformer-based factuality detection model.

generative

Title: Fast Inference and Update of Probabilistic Density Estimation on Trajectory Prediction. (arXiv:2308.08824v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08824
• Code URL: https://github.com/meaten/flowchain-iccv2023
• Copy Paste: [[2308.08824]] Fast Inference and Update of Probabilistic Density Estimation on Trajectory Prediction(http://arxiv.org/abs/2308.08824)
• Summary:

  Safety-critical applications such as autonomous vehicles and social robots require fast computation and accurate probability density estimation on trajectory prediction. To address both requirements, this paper presents a new normalizing flow-based trajectory prediction model named FlowChain. FlowChain is a stack of conditional continuously-indexed flows (CIFs) that are expressive and allow analytical probability density computation. This analytical computation is faster than the generative models that need additional approximations such as kernel density estimation. Moreover, FlowChain is more accurate than the Gaussian mixture-based models due to fewer assumptions on the estimated density. FlowChain also allows a rapid update of estimated probability densities. This update is achieved by adopting the \textit{newest observed position} and reusing the flow transformations and its log-det-jacobians that represent the \textit{motion trend}. This update is completed in less than one millisecond because this reuse greatly omits the computational cost. Experimental results showed our FlowChain achieved state-of-the-art trajectory prediction accuracy compared to previous methods. Furthermore, our FlowChain demonstrated superiority in the accuracy and speed of density estimation. Our code is available at \url{https://github.com/meaten/FlowChain-ICCV2023}

Title: Reinforced Self-Training (ReST) for Language Modeling. (arXiv:2308.08998v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.08998
• Code URL: null
• Copy Paste: [[2308.08998]] Reinforced Self-Training (ReST) for Language Modeling(http://arxiv.org/abs/2308.08998)
• Summary:

  Reinforcement learning from human feedback (RLHF) can improve the quality of large language model's (LLM) outputs by aligning them with human preferences. We propose a simple algorithm for aligning LLMs with human preferences inspired by growing batch reinforcement learning (RL), which we call Reinforced Self-Training (ReST). Given an initial LLM policy, ReST produces a dataset by generating samples from the policy, which are then used to improve the LLM policy using offline RL algorithms. ReST is more efficient than typical online RLHF methods because the training dataset is produced offline, which allows data reuse. While ReST is a general approach applicable to all generative learning settings, we focus on its application to machine translation. Our results show that ReST can substantially improve translation quality, as measured by automated metrics and human evaluation on machine translation benchmarks in a compute and sample-efficient manner.

large language model

Title: Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes. (arXiv:2308.08769v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08769
• Code URL: null
• Copy Paste: [[2308.08769]] Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes(http://arxiv.org/abs/2308.08769)
• Summary:

  3D scene understanding has gained significant attention due to its wide range of applications. However, existing methods for 3D scene understanding are limited to specific downstream tasks, which hinders their practicality in real-world applications. This paper presents Chat-3D, which combines the 3D visual perceptual ability of pre-trained 3D representations and the impressive reasoning and conversation capabilities of advanced LLMs to achieve the first universal dialogue systems for 3D scenes. Specifically, we align 3D representations into the feature space of LLMs, thus enabling LLMs to perceive the 3D world. Given the scarcity of 3D scene-text data, we propose a three-stage training strategy to efficiently utilize the available data for better alignment. To enhance the reasoning ability and develop a user-friendly interaction scheme, we further construct a high-quality object-centric 3D instruction dataset and design an associated object-centric prompt. Our experiments show that Chat-3D achieves an impressive ability to comprehend diverse instructions for 3D scenes, engage in intricate spatial reasoning, and incorporate external knowledge into its responses. Chat-3D achieves a 75.6% relative score compared with GPT-4 on the constructed instruction dataset.

Title: Language-enhanced RNR-Map: Querying Renderable Neural Radiance Field maps with natural language. (arXiv:2308.08854v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08854
• Code URL: null
• Copy Paste: [[2308.08854]] Language-enhanced RNR-Map: Querying Renderable Neural Radiance Field maps with natural language(http://arxiv.org/abs/2308.08854)
• Summary:

  We present Le-RNR-Map, a Language-enhanced Renderable Neural Radiance map for Visual Navigation with natural language query prompts. The recently proposed RNR-Map employs a grid structure comprising latent codes positioned at each pixel. These latent codes, which are derived from image observation, enable: i) image rendering given a camera pose, since they are converted to Neural Radiance Field; ii) image navigation and localization with astonishing accuracy. On top of this, we enhance RNR-Map with CLIP-based embedding latent codes, allowing natural language search without additional label data. We evaluate the effectiveness of this map in single and multi-object searches. We also investigate its compatibility with a Large Language Model as an "affordance query resolver". Code and videos are available at https://intelligolabs.github.io/Le-RNR-Map/

Title: FootGPT : A Large Language Model Development Experiment on a Minimal Setting. (arXiv:2308.08610v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.08610
• Code URL: null
• Copy Paste: [[2308.08610]] FootGPT : A Large Language Model Development Experiment on a Minimal Setting(http://arxiv.org/abs/2308.08610)
• Summary:

  With recent empirical observations, it has been argued that the most significant aspect of developing accurate language models may be the proper dataset content and training strategy compared to the number of neural parameters, training duration or dataset size. Following this argument, we opted to fine tune a one billion parameter size trained general purpose causal language model with a dataset curated on team statistics of the Italian football league first ten game weeks, using low rank adaptation. The limited training dataset was compiled based on a framework where a powerful commercial large language model provides distilled paragraphs and question answer pairs as intended. The training duration was kept relatively short to provide a basis for our minimal setting exploration. We share our key observations on the process related to developing a specific purpose language model which is intended to interpret soccer data with constrained resources in this article.

Title: Boosting Logical Reasoning in Large Language Models through a New Framework: The Graph of Thought. (arXiv:2308.08614v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.08614
• Code URL: null
• Copy Paste: [[2308.08614]] Boosting Logical Reasoning in Large Language Models through a New Framework: The Graph of Thought(http://arxiv.org/abs/2308.08614)
• Summary:

  Recent advancements in large-scale models, such as GPT-4, have showcased remarkable capabilities in addressing standard queries. However, when facing complex problems that require multi-step logical reasoning, their accuracy dramatically decreases. Current research has explored the realm of \textit{prompting engineering} to bolster the inferential capacities of these models. Our paper unveils a pioneering prompting technique, dubbed \textit{Graph of Thoughts (GoT)}. Through testing on a trio of escalating challenges: the 24-point game, resolution of high-degree polynomial equations, and derivation of formulas for recursive sequences, our method outperformed GPT-4, achieving accuracy improvements of $89.7\%$, $86\%$, and $56\%$ for each respective task. Moreover, when juxtaposed with the state-of-the-art (SOTA) prompting method, \textit{Tree of Thought (ToT)}, our approach registered an average accuracy boost of $23\%$, $24\%$, and $15\%$.

Title: Large Language Models for Granularized Barrett's Esophagus Diagnosis Classification. (arXiv:2308.08660v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.08660
• Code URL: null
• Copy Paste: [[2308.08660]] Large Language Models for Granularized Barrett's Esophagus Diagnosis Classification(http://arxiv.org/abs/2308.08660)
• Summary:

  Diagnostic codes for Barrett's esophagus (BE), a precursor to esophageal cancer, lack granularity and precision for many research or clinical use cases. Laborious manual chart review is required to extract key diagnostic phenotypes from BE pathology reports. We developed a generalizable transformer-based method to automate data extraction. Using pathology reports from Columbia University Irving Medical Center with gastroenterologist-annotated targets, we performed binary dysplasia classification as well as granularized multi-class BE-related diagnosis classification. We utilized two clinically pre-trained large language models, with best model performance comparable to a highly tailored rule-based system developed using the same data. Binary dysplasia extraction achieves 0.964 F1-score, while the multi-class model achieves 0.911 F1-score. Our method is generalizable and faster to implement as compared to a tailored rule-based approach.

Title: An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning. (arXiv:2308.08747v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.08747
• Code URL: null
• Copy Paste: [[2308.08747]] An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning(http://arxiv.org/abs/2308.08747)
• Summary:

  Catastrophic forgetting (CF) is a phenomenon that occurs in machine learning when a model forgets previously learned information as it learns new information. As large language models (LLMs) have shown excellent performance, it is interesting to uncover whether CF exists in the continual fine-tuning of LLMs. In this study, we empirically evaluate the forgetting phenomenon in LLMs' knowledge, from the perspectives of domain knowledge, reasoning, and reading comprehension. The experiments demonstrate that catastrophic forgetting is generally observed in LLMs ranging from 1b to 7b. Furthermore, as the scale increases, the severity of forgetting also intensifies. Comparing the decoder-only model BLOOMZ with the encoder-decoder model mT0, BLOOMZ suffers less forgetting and maintains more knowledge. We also observe that LLMs can mitigate language bias (e.g. gender bias) during continual fine-tuning. Moreover, we find that ALPACA can maintain more knowledge and capacity compared with LLAMA during the continual fine-tuning, which implies that general instruction tuning can help mitigate the forgetting phenomenon of LLMs in the further fine-tuning process.

Title: CMB: A Comprehensive Medical Benchmark in Chinese. (arXiv:2308.08833v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.08833
• Code URL: https://github.com/FreedomIntelligence/CMB
• Copy Paste: [[2308.08833]] CMB: A Comprehensive Medical Benchmark in Chinese(http://arxiv.org/abs/2308.08833)
• Summary:

  Large Language Models (LLMs) provide a possibility to make a great breakthrough in medicine. The establishment of a standardized medical benchmark becomes a fundamental cornerstone to measure progression. However, medical environments in different regions have their local characteristics, e.g., the ubiquity and significance of traditional Chinese medicine within China. Therefore, merely translating English-based medical evaluation may result in \textit{contextual incongruities} to a local region. To solve the issue, we propose a localized medical benchmark called CMB, a Comprehensive Medical Benchmark in Chinese, designed and rooted entirely within the native Chinese linguistic and cultural framework. While traditional Chinese medicine is integral to this evaluation, it does not constitute its entirety. Using this benchmark, we have evaluated several prominent large-scale LLMs, including ChatGPT, GPT-4, dedicated Chinese LLMs, and LLMs specialized in the medical domain. It is worth noting that our benchmark is not devised as a leaderboard competition but as an instrument for self-assessment of model advancements. We hope this benchmark could facilitate the widespread adoption and enhancement of medical LLMs within China. Check details in \url{https://cmedbenchmark.llmzoo.com/}.

Title: Evaluation of really good grammatical error correction. (arXiv:2308.08982v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.08982
• Code URL: https://github.com/robertostling/gec-evaluation
• Copy Paste: [[2308.08982]] Evaluation of really good grammatical error correction(http://arxiv.org/abs/2308.08982)
• Summary:

  Although rarely stated, in practice, Grammatical Error Correction (GEC) encompasses various models with distinct objectives, ranging from grammatical error detection to improving fluency. Traditional evaluation methods fail to fully capture the full range of system capabilities and objectives. Reference-based evaluations suffer from limitations in capturing the wide variety of possible correction and the biases introduced during reference creation and is prone to favor fixing local errors over overall text improvement. The emergence of large language models (LLMs) has further highlighted the shortcomings of these evaluation strategies, emphasizing the need for a paradigm shift in evaluation methodology. In the current study, we perform a comprehensive evaluation of various GEC systems using a recently published dataset of Swedish learner texts. The evaluation is performed using established evaluation metrics as well as human judges. We find that GPT-3 in a few-shot setting by far outperforms previous grammatical error correction systems for Swedish, a language comprising only 0.11% of its training data. We also found that current evaluation methods contain undesirable biases that a human evaluation is able to reveal. We suggest using human post-editing of GEC system outputs to analyze the amount of change required to reach native-level human performance on the task, and provide a dataset annotated with human post-edits and assessments of grammaticality, fluency and meaning preservation of GEC system outputs.

segmentation

Title: Improving Anomaly Segmentation with Multi-Granularity Cross-Domain Alignment. (arXiv:2308.08696v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08696
• Code URL: null
• Copy Paste: [[2308.08696]] Improving Anomaly Segmentation with Multi-Granularity Cross-Domain Alignment(http://arxiv.org/abs/2308.08696)
• Summary:

  Anomaly segmentation plays a crucial role in identifying anomalous objects within images, which facilitates the detection of road anomalies for autonomous driving. Although existing methods have shown impressive results in anomaly segmentation using synthetic training data, the domain discrepancies between synthetic training data and real test data are often neglected. To address this issue, the Multi-Granularity Cross-Domain Alignment (MGCDA) framework is proposed for anomaly segmentation in complex driving environments. It uniquely combines a new Multi-source Domain Adversarial Training (MDAT) module and a novel Cross-domain Anomaly-aware Contrastive Learning (CACL) method to boost the generality of the model, seamlessly integrating multi-domain data at both scene and sample levels. Multi-source domain adversarial loss and a dynamic label smoothing strategy are integrated into the MDAT module to facilitate the acquisition of domain-invariant features at the scene level, through adversarial training across multiple stages. CACL aligns sample-level representations with contrastive loss on cross-domain data, which utilizes an anomaly-aware sampling strategy to efficiently sample hard samples and anchors. The proposed framework has decent properties of parameter-free during the inference stage and is compatible with other anomaly segmentation networks. Experimental conducted on Fishyscapes and RoadAnomaly datasets demonstrate that the proposed framework achieves state-of-the-art performance.

Title: SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation. (arXiv:2308.08746v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08746
• Code URL: https://github.com/wenxi-yue/surgicalsam
• Copy Paste: [[2308.08746]] SurgicalSAM: Efficient Class Promptable Surgical Instrument Segmentation(http://arxiv.org/abs/2308.08746)
• Summary:

  The Segment Anything Model (SAM) is a powerful foundation model that has revolutionised image segmentation. To apply SAM to surgical instrument segmentation, a common approach is to locate precise points or boxes of instruments and then use them as prompts for SAM in a zero-shot manner. However, we observe two problems with this naive pipeline: (1) the domain gap between natural objects and surgical instruments leads to poor generalisation of SAM; and (2) SAM relies on precise point or box locations for accurate segmentation, requiring either extensive manual guidance or a well-performing specialist detector for prompt preparation, which leads to a complex multi-stage pipeline. To address these problems, we introduce SurgicalSAM, a novel end-to-end efficient-tuning approach for SAM to effectively integrate surgical-specific information with SAM's pre-trained knowledge for improved generalisation. Specifically, we propose a lightweight prototype-based class prompt encoder for tuning, which directly generates prompt embeddings from class prototypes and eliminates the use of explicit prompts for improved robustness and a simpler pipeline. In addition, to address the low inter-class variance among surgical instrument categories, we propose contrastive prototype learning, further enhancing the discrimination of the class prototypes for more accurate class prompting. The results of extensive experiments on both EndoVis2018 and EndoVis2017 datasets demonstrate that SurgicalSAM achieves state-of-the-art performance while only requiring a small number of tunable parameters. The source code will be released at https://github.com/wenxi-yue/SurgicalSAM.

Title: Learning to In-paint: Domain Adaptive Shape Completion for 3D Organ Segmentation. (arXiv:2308.08775v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.08775
• Code URL: null
• Copy Paste: [[2308.08775]] Learning to In-paint: Domain Adaptive Shape Completion for 3D Organ Segmentation(http://arxiv.org/abs/2308.08775)
• Summary:

  We aim at incorporating explicit shape information into current 3D organ segmentation models. Different from previous works, we formulate shape learning as an in-painting task, which is named Masked Label Mask Modeling (MLM). Through MLM, learnable mask tokens are fed into transformer blocks to complete the label mask of organ. To transfer MLM shape knowledge to target, we further propose a novel shape-aware self-distillation with both in-painting reconstruction loss and pseudo loss. Extensive experiments on five public organ segmentation datasets show consistent improvements over prior arts with at least 1.2 points gain in the Dice score, demonstrating the effectiveness of our method in challenging unsupervised domain adaptation scenarios including: (1) In-domain organ segmentation; (2) Unseen domain segmentation and (3) Unseen organ segmentation. We hope this work will advance shape analysis and geometric learning in medical imaging.