secure

Title: Human Gait Recognition using Deep Learning: A Comprehensive Review. (arXiv:2309.10144v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10144
• Code URL: null
• Copy Paste: [[2309.10144]] Human Gait Recognition using Deep Learning: A Comprehensive Review(http://arxiv.org/abs/2309.10144)
• Summary:

  Gait recognition (GR) is a growing biometric modality used for person identification from a distance through visual cameras. GR provides a secure and reliable alternative to fingerprint and face recognition, as it is harder to distinguish between false and authentic signals. Furthermore, its resistance to spoofing makes GR suitable for all types of environments. With the rise of deep learning, steadily improving strides have been made in GR technology with promising results in various contexts. As video surveillance becomes more prevalent, new obstacles arise, such as ensuring uniform performance evaluation across different protocols, reliable recognition despite shifting lighting conditions, fluctuations in gait patterns, and protecting privacy.This survey aims to give an overview of GR and analyze the environmental elements and complications that could affect it in comparison to other biometric recognition systems. The primary goal is to examine the existing deep learning (DL) techniques employed for human GR that may generate new research opportunities.

security

Title: LLM Platform Security: Applying a Systematic Evaluation Framework to OpenAI's ChatGPT Plugins. (arXiv:2309.10254v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.10254
• Code URL: https://github.com/llm-platform-security/chatgpt-plugin-eval
• Copy Paste: [[2309.10254]] LLM Platform Security: Applying a Systematic Evaluation Framework to OpenAI's ChatGPT Plugins(http://arxiv.org/abs/2309.10254)
• Summary:

  Large language model (LLM) platforms, such as ChatGPT, have recently begun offering a plugin ecosystem to interface with third-party services on the internet. While these plugins extend the capabilities of LLM platforms, they are developed by arbitrary third parties and thus cannot be implicitly trusted. Plugins also interface with LLM platforms and users using natural language, which can have imprecise interpretations. In this paper, we propose a framework that lays a foundation for LLM platform designers to analyze and improve the security, privacy, and safety of current and future plugin-integrated LLM platforms. Our framework is a formulation of an attack taxonomy that is developed by iteratively exploring how LLM platform stakeholders could leverage their capabilities and responsibilities to mount attacks against each other. As part of our iterative process, we apply our framework in the context of OpenAI's plugin ecosystem. We uncover plugins that concretely demonstrate the potential for the types of issues that we outline in our attack taxonomy. We conclude by discussing novel challenges and by providing recommendations to improve the security, privacy, and safety of present and future LLM-based computing platforms.

Title: GCNIDS: GCN-based intrusion detection system for CAN Bus. (arXiv:2309.10173v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.10173
• Code URL: null
• Copy Paste: [[2309.10173]] GCNIDS: GCN-based intrusion detection system for CAN Bus(http://arxiv.org/abs/2309.10173)
• Summary:

  The Controller Area Network (CAN) bus serves as a standard protocol for facilitating communication among various electronic control units (ECUs) within contemporary vehicles. However, it has been demonstrated that the CAN bus is susceptible to remote attacks, which pose risks to the vehicle's safety and functionality. To tackle this concern, researchers have introduced intrusion detection systems (IDSs) to identify and thwart such attacks. In this paper, we present an innovative approach to intruder detection within the CAN bus, leveraging Graph Convolutional Network (GCN) techniques as introduced by Zhang, Tong, Xu, and Maciejewski in 2019. By harnessing the capabilities of deep learning, we aim to enhance attack detection accuracy while minimizing the requirement for manual feature engineering. Our experimental findings substantiate that the proposed GCN-based method surpasses existing IDSs in terms of accuracy, precision, and recall. Additionally, our approach demonstrates efficacy in detecting mixed attacks, which are more challenging to identify than single attacks. Furthermore, it reduces the necessity for extensive feature engineering and is particularly well-suited for real-time detection systems. To the best of our knowledge, this represents the pioneering application of GCN to CAN data for intrusion detection. Our proposed approach holds significant potential in fortifying the security and safety of modern vehicles, safeguarding against attacks and preventing them from undermining vehicle functionality.

Title: Trust assumptions in voting systems. (arXiv:2309.10391v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.10391
• Code URL: null
• Copy Paste: [[2309.10391]] Trust assumptions in voting systems(http://arxiv.org/abs/2309.10391)
• Summary:

  Assessing and comparing the security level of different voting systems is non-trivial as the technical means provided for and societal assumptions made about various systems differ significantly. However, trust assumptions concerning the involved parties are present for all voting systems and can be used as a basis for comparison. This paper discusses eight concrete voting systems with different properties, 12 types of parties involved, and seven general security goals set for voting. The emerging trust relations are assessed for their criticality, and the result is used for comparison of the considered systems.

Title: Poster: Control-Flow Integrity in Low-end Embedded Devices. (arXiv:2309.10396v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.10396
• Code URL: null
• Copy Paste: [[2309.10396]] Poster: Control-Flow Integrity in Low-end Embedded Devices(http://arxiv.org/abs/2309.10396)
• Summary:

  Embedded, smart, and IoT devices are increasingly popular in numerous everyday settings. Since lower-end devices have the most strict cost constraints, they tend to have few, if any, security features. This makes them attractive targets for exploits and malware. Prior research proposed various security architectures for enforcing security properties for resource-constrained devices, e.g., via Remote Attestation (RA). Such techniques can (statically) verify software integrity of a remote device and detect compromise. However, run-time (dynamic) security, e.g., via Control-Flow Integrity (CFI), is hard to achieve. This work constructs an architecture that ensures integrity of software execution against run-time attacks, such as Return-Oriented Programming (ROP). It is built atop a recently proposed CASU -- a low-cost active Root-of-Trust (RoT) that guarantees software immutability. We extend CASU to support a shadow stack and a CFI monitor to mitigate run-time attacks. This gives some confidence that CFI can indeed be attained even on low-end devices, with minimal hardware overhead.

Title: Steganography for Neural Radiance Fields by Backdooring. (arXiv:2309.10503v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.10503
• Code URL: null
• Copy Paste: [[2309.10503]] Steganography for Neural Radiance Fields by Backdooring(http://arxiv.org/abs/2309.10503)
• Summary:

  The utilization of implicit representation for visual data (such as images, videos, and 3D models) has recently gained significant attention in computer vision research. In this letter, we propose a novel model steganography scheme with implicit neural representation. The message sender leverages Neural Radiance Fields (NeRF) and its viewpoint synthesis capabilities by introducing a viewpoint as a key. The NeRF model generates a secret viewpoint image, which serves as a backdoor. Subsequently, we train a message extractor using overfitting to establish a one-to-one mapping between the secret message and the secret viewpoint image. The sender delivers the trained NeRF model and the message extractor to the receiver over the open channel, and the receiver utilizes the key shared by both parties to obtain the rendered image in the secret view from the NeRF model, and then obtains the secret message through the message extractor. The inherent complexity of the viewpoint information prevents attackers from stealing the secret message accurately. Experimental results demonstrate that the message extractor trained in this letter achieves high-capacity steganography with fast performance, achieving a 100\% accuracy in message extraction. Furthermore, the extensive viewpoint key space of NeRF ensures the security of the steganography scheme.

Title: A Semi-Supervised Approach for Power System Event Identification. (arXiv:2309.10095v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10095
• Code URL: null
• Copy Paste: [[2309.10095]] A Semi-Supervised Approach for Power System Event Identification(http://arxiv.org/abs/2309.10095)
• Summary:

  Event identification is increasingly recognized as crucial for enhancing the reliability, security, and stability of the electric power system. With the growing deployment of Phasor Measurement Units (PMUs) and advancements in data science, there are promising opportunities to explore data-driven event identification via machine learning classification techniques. However, obtaining accurately-labeled eventful PMU data samples remains challenging due to its labor-intensive nature and uncertainty about the event type (class) in real-time. Thus, it is natural to use semi-supervised learning techniques, which make use of both labeled and unlabeled samples. %We propose a novel semi-supervised framework to assess the effectiveness of incorporating unlabeled eventful samples to enhance existing event identification methodologies. We evaluate three categories of classical semi-supervised approaches: (i) self-training, (ii) transductive support vector machines (TSVM), and (iii) graph-based label spreading (LS) method. Our approach characterizes events using physically interpretable features extracted from modal analysis of synthetic eventful PMU data. In particular, we focus on the identification of four event classes whose identification is crucial for grid operations. We have developed and publicly shared a comprehensive Event Identification package which consists of three aspects: data generation, feature extraction, and event identification with limited labels using semi-supervised methodologies. Using this package, we generate and evaluate eventful PMU data for the South Carolina synthetic network. Our evaluation consistently demonstrates that graph-based LS outperforms the other two semi-supervised methods that we consider, and can noticeably improve event identification performance relative to the setting with only a small number of labeled samples.

privacy

Title: Specification-Driven Video Search via Foundation Models and Formal Verification. (arXiv:2309.10171v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10171
• Code URL: null
• Copy Paste: [[2309.10171]] Specification-Driven Video Search via Foundation Models and Formal Verification(http://arxiv.org/abs/2309.10171)
• Summary:

  The increasing abundance of video data enables users to search for events of interest, e.g., emergency incidents. Meanwhile, it raises new concerns, such as the need for preserving privacy. Existing approaches to video search require either manual inspection or a deep learning model with massive training. We develop a method that uses recent advances in vision and language models, as well as formal methods, to search for events of interest in video clips automatically and efficiently. The method consists of an algorithm to map text-based event descriptions into linear temporal logic over finite traces (LTL$_f$) and an algorithm to construct an automaton encoding the video information. Then, the method formally verifies the automaton representing the video against the LTL$_f$ specifications and adds the pertinent video clips to the search result if the automaton satisfies the specifications. We provide qualitative and quantitative analysis to demonstrate the video-searching capability of the proposed method. It achieves over 90 percent precision in searching over privacy-sensitive videos and a state-of-the-art autonomous driving dataset.

Title: Source-free Active Domain Adaptation for Diabetic Retinopathy Grading Based on Ultra-wide-field Fundus Image. (arXiv:2309.10619v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10619
• Code URL: null
• Copy Paste: [[2309.10619]] Source-free Active Domain Adaptation for Diabetic Retinopathy Grading Based on Ultra-wide-field Fundus Image(http://arxiv.org/abs/2309.10619)
• Summary:

  Domain adaptation (DA) has been widely applied in the diabetic retinopathy (DR) grading of unannotated ultra-wide-field (UWF) fundus images, which can transfer annotated knowledge from labeled color fundus images. However, suffering from huge domain gaps and complex real-world scenarios, the DR grading performance of most mainstream DA is far from that of clinical diagnosis. To tackle this, we propose a novel source-free active domain adaptation (SFADA) in this paper. Specifically, we focus on DR grading problem itself and propose to generate features of color fundus images with continuously evolving relationships of DRs, actively select a few valuable UWF fundus images for labeling with local representation matching, and adapt model on UWF fundus images with DR lesion prototypes. Notably, the SFADA also takes data privacy and computational efficiency into consideration. Extensive experimental results demonstrate that our proposed SFADA achieves state-of-the-art DR grading performance, increasing accuracy by 20.9% and quadratic weighted kappa by 18.63% compared with baseline and reaching 85.36% and 92.38% respectively. These investigations show that the potential of our approach for real clinical practice is promising.

Title: PolicyGPT: Automated Analysis of Privacy Policies with Large Language Models. (arXiv:2309.10238v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.10238
• Code URL: null
• Copy Paste: [[2309.10238]] PolicyGPT: Automated Analysis of Privacy Policies with Large Language Models(http://arxiv.org/abs/2309.10238)
• Summary:

  Privacy policies serve as the primary conduit through which online service providers inform users about their data collection and usage procedures. However, in a bid to be comprehensive and mitigate legal risks, these policy documents are often quite verbose. In practical use, users tend to click the Agree button directly rather than reading them carefully. This practice exposes users to risks of privacy leakage and legal issues. Recently, the advent of Large Language Models (LLM) such as ChatGPT and GPT-4 has opened new possibilities for text analysis, especially for lengthy documents like privacy policies. In this study, we investigate a privacy policy text analysis framework PolicyGPT based on the LLM. This framework was tested using two datasets. The first dataset comprises of privacy policies from 115 websites, which were meticulously annotated by legal experts, categorizing each segment into one of 10 classes. The second dataset consists of privacy policies from 304 popular mobile applications, with each sentence manually annotated and classified into one of another 10 categories. Under zero-shot learning conditions, PolicyGPT demonstrated robust performance. For the first dataset, it achieved an accuracy rate of 97%, while for the second dataset, it attained an 87% accuracy rate, surpassing that of the baseline machine learning and neural network models.

Title: A Neighbourhood-Aware Differential Privacy Mechanism for Static Word Embeddings. (arXiv:2309.10551v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10551
• Code URL: null
• Copy Paste: [[2309.10551]] A Neighbourhood-Aware Differential Privacy Mechanism for Static Word Embeddings(http://arxiv.org/abs/2309.10551)
• Summary:

  We propose a Neighbourhood-Aware Differential Privacy (NADP) mechanism considering the neighbourhood of a word in a pretrained static word embedding space to determine the minimal amount of noise required to guarantee a specified privacy level. We first construct a nearest neighbour graph over the words using their embeddings, and factorise it into a set of connected components (i.e. neighbourhoods). We then separately apply different levels of Gaussian noise to the words in each neighbourhood, determined by the set of words in that neighbourhood. Experiments show that our proposed NADP mechanism consistently outperforms multiple previously proposed DP mechanisms such as Laplacian, Gaussian, and Mahalanobis in multiple downstream tasks, while guaranteeing higher levels of privacy.

Title: Disentangled Information Bottleneck guided Privacy-Protective JSCC for Image Transmission. (arXiv:2309.10263v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.10263
• Code URL: null
• Copy Paste: [[2309.10263]] Disentangled Information Bottleneck guided Privacy-Protective JSCC for Image Transmission(http://arxiv.org/abs/2309.10263)
• Summary:

  Joint source and channel coding (JSCC) has attracted increasing attention due to its robustness and high efficiency. However, JSCC is vulnerable to privacy leakage due to the high relevance between the source image and channel input. In this paper, we propose a disentangled information bottleneck guided privacy-protective JSCC (DIB-PPJSCC) for image transmission, which aims at protecting private information as well as achieving superior communication performance at the legitimate receiver. In particular, we propose a DIB objective to disentangle private and public information. The goal is to compress the private information in the public subcodewords, preserve the private information in the private subcodewords and improve the reconstruction quality simultaneously. In order to optimize JSCC neural networks using the DIB objective, we derive a differentiable estimation of the DIB objective based on the variational approximation and the density-ratio trick. Additionally, we design a password-based privacy-protective (PP) algorithm which can be jointly optimized with JSCC neural networks to encrypt the private subcodewords. Specifically, we employ a private information encryptor to encrypt the private subcodewords before transmission, and a corresponding decryptor to recover the private information at the legitimate receiver. A loss function for jointly training the encryptor, decryptor and JSCC decoder is derived based on the maximum entropy principle, which aims at maximizing the eavesdropping uncertainty as well as improving the reconstruction quality. Experimental results show that DIB-PPJSCC can reduce the eavesdropping accuracy on private information up to $15\%$ and reduce $10\%$ inference time compared to existing privacy-protective JSCC and traditional separate methods.

Title: Love or Hate? Share or Split? Privacy-Preserving Training Using Split Learning and Homomorphic Encryption. (arXiv:2309.10517v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.10517
• Code URL: null
• Copy Paste: [[2309.10517]] Love or Hate? Share or Split? Privacy-Preserving Training Using Split Learning and Homomorphic Encryption(http://arxiv.org/abs/2309.10517)
• Summary:

  Split learning (SL) is a new collaborative learning technique that allows participants, e.g. a client and a server, to train machine learning models without the client sharing raw data. In this setting, the client initially applies its part of the machine learning model on the raw data to generate activation maps and then sends them to the server to continue the training process. Previous works in the field demonstrated that reconstructing activation maps could result in privacy leakage of client data. In addition to that, existing mitigation techniques that overcome the privacy leakage of SL prove to be significantly worse in terms of accuracy. In this paper, we improve upon previous works by constructing a protocol based on U-shaped SL that can operate on homomorphically encrypted data. More precisely, in our approach, the client applies homomorphic encryption on the activation maps before sending them to the server, thus protecting user privacy. This is an important improvement that reduces privacy leakage in comparison to other SL-based works. Finally, our results show that, with the optimum set of parameters, training with HE data in the U-shaped SL setting only reduces accuracy by 2.65% compared to training on plaintext. In addition, raw training data privacy is preserved.

Title: Striking a Balance: An Optimal Mechanism Design for Heterogenous Differentially Private Data Acquisition for Logistic Regression. (arXiv:2309.10340v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10340
• Code URL: null
• Copy Paste: [[2309.10340]] Striking a Balance: An Optimal Mechanism Design for Heterogenous Differentially Private Data Acquisition for Logistic Regression(http://arxiv.org/abs/2309.10340)
• Summary:

  We investigate the problem of performing logistic regression on data collected from privacy-sensitive sellers. Since the data is private, sellers must be incentivized through payments to provide their data. Thus, the goal is to design a mechanism that optimizes a weighted combination of test loss, seller privacy, and payment, i.e., strikes a balance between multiple objectives of interest. We solve the problem by combining ideas from game theory, statistical learning theory, and differential privacy. The buyer's objective function can be highly non-convex. However, we show that, under certain conditions on the problem parameters, the problem can be convexified by using a change of variables. We also provide asymptotic results characterizing the buyer's test error and payments when the number of sellers becomes large. Finally, we demonstrate our ideas by applying them to a real healthcare data set.

protect

defense

Title: Language Guided Adversarial Purification. (arXiv:2309.10348v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10348
• Code URL: null
• Copy Paste: [[2309.10348]] Language Guided Adversarial Purification(http://arxiv.org/abs/2309.10348)
• Summary:

  Adversarial purification using generative models demonstrates strong adversarial defense performance. These methods are classifier and attack-agnostic, making them versatile but often computationally intensive. Recent strides in diffusion and score networks have improved image generation and, by extension, adversarial purification. Another highly efficient class of adversarial defense methods known as adversarial training requires specific knowledge of attack vectors, forcing them to be trained extensively on adversarial examples. To overcome these limitations, we introduce a new framework, namely Language Guided Adversarial Purification (LGAP), utilizing pre-trained diffusion models and caption generators to defend against adversarial attacks. Given an input image, our method first generates a caption, which is then used to guide the adversarial purification process through a diffusion network. Our approach has been evaluated against strong adversarial attacks, proving its effectiveness in enhancing adversarial robustness. Our results indicate that LGAP outperforms most existing adversarial defense techniques without requiring specialized network training. This underscores the generalizability of models trained on large datasets, highlighting a promising direction for further research.

Title: Efficient Low-Rank GNN Defense Against Structural Attacks. (arXiv:2309.10136v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10136
• Code URL: null
• Copy Paste: [[2309.10136]] Efficient Low-Rank GNN Defense Against Structural Attacks(http://arxiv.org/abs/2309.10136)
• Summary:

  Graph Neural Networks (GNNs) have been shown to possess strong representation abilities over graph data. However, GNNs are vulnerable to adversarial attacks, and even minor perturbations to the graph structure can significantly degrade their performance. Existing methods either are ineffective against sophisticated attacks or require the optimization of dense adjacency matrices, which is time-consuming and prone to local minima. To remedy this problem, we propose an Efficient Low-Rank Graph Neural Network (ELR-GNN) defense method, which aims to learn low-rank and sparse graph structures for defending against adversarial attacks, ensuring effective defense with greater efficiency. Specifically, ELR-GNN consists of two modules: a Coarse Low-Rank Estimation Module and a Fine-Grained Estimation Module. The first module adopts the truncated Singular Value Decomposition (SVD) to initialize the low-rank adjacency matrix estimation, which serves as a starting point for optimizing the low-rank matrix. In the second module, the initial estimate is refined by jointly learning a low-rank sparse graph structure with the GNN model. Sparsity is incorporated into the learned low-rank adjacency matrix by pruning weak connections, which can reduce redundant data while maintaining valuable information. As a result, instead of using the dense adjacency matrix directly, ELR-GNN can learn a low-rank and sparse estimate of it in a simple, efficient and easy to optimize manner. The experimental results demonstrate that ELR-GNN outperforms the state-of-the-art GNN defense methods in the literature, in addition to being very efficient and easy to train.

attack

Title: Transferable Adversarial Attack on Image Tampering Localization. (arXiv:2309.10243v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10243
• Code URL: null
• Copy Paste: [[2309.10243]] Transferable Adversarial Attack on Image Tampering Localization(http://arxiv.org/abs/2309.10243)
• Summary:

  It is significant to evaluate the security of existing digital image tampering localization algorithms in real-world applications. In this paper, we propose an adversarial attack scheme to reveal the reliability of such tampering localizers, which would be fooled and fail to predict altered regions correctly. Specifically, the adversarial examples based on optimization and gradient are implemented for white/black-box attacks. Correspondingly, the adversarial example is optimized via reverse gradient propagation, and the perturbation is added adaptively in the direction of gradient rising. The black-box attack is achieved by relying on the transferability of such adversarial examples to different localizers. Extensive evaluations verify that the proposed attack sharply reduces the localization accuracy while preserving high visual quality of the attacked images.

Title: Adversarial Attacks Against Uncertainty Quantification. (arXiv:2309.10586v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10586
• Code URL: null
• Copy Paste: [[2309.10586]] Adversarial Attacks Against Uncertainty Quantification(http://arxiv.org/abs/2309.10586)
• Summary:

  Machine-learning models can be fooled by adversarial examples, i.e., carefully-crafted input perturbations that force models to output wrong predictions. While uncertainty quantification has been recently proposed to detect adversarial inputs, under the assumption that such attacks exhibit a higher prediction uncertainty than pristine data, it has been shown that adaptive attacks specifically aimed at reducing also the uncertainty estimate can easily bypass this defense mechanism. In this work, we focus on a different adversarial scenario in which the attacker is still interested in manipulating the uncertainty estimate, but regardless of the correctness of the prediction; in particular, the goal is to undermine the use of machine-learning models when their outputs are consumed by a downstream module or by a human operator. Following such direction, we: \textit{(i)} design a threat model for attacks targeting uncertainty quantification; \textit{(ii)} devise different attack strategies on conceptually different UQ techniques spanning for both classification and semantic segmentation problems; \textit{(iii)} conduct a first complete and extensive analysis to compare the differences between some of the most employed UQ approaches under attack. Our extensive experimental analysis shows that our attacks are more effective in manipulating uncertainty quantification measures than attacks aimed to also induce misclassifications.

Title: Model Leeching: An Extraction Attack Targeting LLMs. (arXiv:2309.10544v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10544
• Code URL: null
• Copy Paste: [[2309.10544]] Model Leeching: An Extraction Attack Targeting LLMs(http://arxiv.org/abs/2309.10544)
• Summary:

  Model Leeching is a novel extraction attack targeting Large Language Models (LLMs), capable of distilling task-specific knowledge from a target LLM into a reduced parameter model. We demonstrate the effectiveness of our attack by extracting task capability from ChatGPT-3.5-Turbo, achieving 73% Exact Match (EM) similarity, and SQuAD EM and F1 accuracy scores of 75% and 87%, respectively for only $50 in API cost. We further demonstrate the feasibility of adversarial attack transferability from an extracted model extracted via Model Leeching to perform ML attack staging against a target LLM, resulting in an 11% increase to attack success rate when applied to ChatGPT-3.5-Turbo.

Title: Realistic Website Fingerprinting By Augmenting Network Trace. (arXiv:2309.10147v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.10147
• Code URL: https://github.com/spin-umass/realistic-website-fingerprinting-by-augmenting-network-traces
• Copy Paste: [[2309.10147]] Realistic Website Fingerprinting By Augmenting Network Trace(http://arxiv.org/abs/2309.10147)
• Summary:

  Website Fingerprinting (WF) is considered a major threat to the anonymity of Tor users (and other anonymity systems). While state-of-the-art WF techniques have claimed high attack accuracies, e.g., by leveraging Deep Neural Networks (DNN), several recent works have questioned the practicality of such WF attacks in the real world due to the assumptions made in the design and evaluation of these attacks. In this work, we argue that such impracticality issues are mainly due to the attacker's inability in collecting training data in comprehensive network conditions, e.g., a WF classifier may be trained only on samples collected on specific high-bandwidth network links but deployed on connections with different network conditions. We show that augmenting network traces can enhance the performance of WF classifiers in unobserved network conditions. Specifically, we introduce NetAugment, an augmentation technique tailored to the specifications of Tor traces. We instantiate NetAugment through semi-supervised and self-supervised learning techniques. Our extensive open-world and close-world experiments demonstrate that under practical evaluation settings, our WF attacks provide superior performances compared to the state-of-the-art; this is due to their use of augmented network traces for training, which allows them to learn the features of target traffic in unobserved settings. For instance, with a 5-shot learning in a closed-world scenario, our self-supervised WF attack (named NetCLR) reaches up to 80% accuracy when the traces for evaluation are collected in a setting unobserved by the WF adversary. This is compared to an accuracy of 64.4% achieved by the state-of-the-art Triplet Fingerprinting [35]. We believe that the promising results of our work can encourage the use of network trace augmentation in other types of network traffic analysis.

Title: The Impact of Exposed Passwords on Honeyword Efficacy. (arXiv:2309.10323v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.10323
• Code URL: null
• Copy Paste: [[2309.10323]] The Impact of Exposed Passwords on Honeyword Efficacy(http://arxiv.org/abs/2309.10323)
• Summary:

  Honeywords are decoy passwords that can be added to a credential database; if a login attempt uses a honeyword, this indicates that the site's credential database has been leaked. In this paper we explore the basic requirements for honeywords to be effective, in a threat model where the attacker knows passwords for the same users at other sites. First, we show that for user-chosen (vs. algorithmically generated, i.e., by a password manager) passwords, existing honeyword-generation algorithms largely fail to achieve reasonable tradeoffs between false positives and false negatives in this threat model. Second, we show that for users leveraging algorithmically generated passwords, state-of-the-art methods for honeyword generation will produce honeywords that are not sufficiently deceptive, yielding many false negatives. Instead, we find that only a honeyword-generation algorithm that uses the same password generator as the user can provide deceptive honeywords in this case. However, when the defender's ability to infer the generator from the (one) account password is less accurate than the attacker's ability to infer the generator from potentially many, this deception can again wane. Taken together, our results provide a cautionary note for the state of honeyword research and pose new challenges to the field.

Title: Exploring the Dark Side of AI: Advanced Phishing Attack Design and Deployment Using ChatGPT. (arXiv:2309.10463v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.10463
• Code URL: null
• Copy Paste: [[2309.10463]] Exploring the Dark Side of AI: Advanced Phishing Attack Design and Deployment Using ChatGPT(http://arxiv.org/abs/2309.10463)
• Summary:

  This paper explores the possibility of using ChatGPT to develop advanced phishing attacks and automate their large-scale deployment. We make ChatGPT generate the following parts of a phishing attack: i) cloning a targeted website, ii) integrating code for stealing credentials, iii) obfuscating code, iv) automating website deployment on a hosting provider, v) registering a phishing domain name, and vi) integrating the website with a reverse proxy. The initial assessment of the automatically generated phishing kits highlights their rapid generation and deployment process as well as the close resemblance of the resulting pages to the target website. More broadly, we demonstrate that recent advances in AI underscore the potential risks of its misuse in phishing attacks, which can lead to their increased prevalence and severity. This highlights the necessity for enhanced countermeasures within AI systems.

Title: SPFL: A Self-purified Federated Learning Method Against Poisoning Attacks. (arXiv:2309.10607v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2309.10607
• Code URL: null
• Copy Paste: [[2309.10607]] SPFL: A Self-purified Federated Learning Method Against Poisoning Attacks(http://arxiv.org/abs/2309.10607)
• Summary:

  While Federated learning (FL) is attractive for pulling privacy-preserving distributed training data, the credibility of participating clients and non-inspectable data pose new security threats, of which poisoning attacks are particularly rampant and hard to defend without compromising privacy, performance or other desirable properties of FL. To tackle this problem, we propose a self-purified FL (SPFL) method that enables benign clients to exploit trusted historical features of locally purified model to supervise the training of aggregated model in each iteration. The purification is performed by an attention-guided self-knowledge distillation where the teacher and student models are optimized locally for task loss, distillation loss and attention-based loss simultaneously. SPFL imposes no restriction on the communication protocol and aggregator at the server. It can work in tandem with any existing secure aggregation algorithms and protocols for augmented security and privacy guarantee. We experimentally demonstrate that SPFL outperforms state-of-the-art FL defenses against various poisoning attacks. The attack success rate of SPFL trained model is at most 3$\%$ above that of a clean model, even if the poisoning attack is launched in every iteration with all but one malicious clients in the system. Meantime, it improves the model quality on normal inputs compared to FedAvg, either under attack or in the absence of an attack.

robust

Title: Introspective Deep Metric Learning. (arXiv:2309.09982v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.09982
• Code URL: null
• Copy Paste: [[2309.09982]] Introspective Deep Metric Learning(http://arxiv.org/abs/2309.09982)
• Summary:

  This paper proposes an introspective deep metric learning (IDML) framework for uncertainty-aware comparisons of images. Conventional deep metric learning methods focus on learning a discriminative embedding to describe the semantic features of images, which ignore the existence of uncertainty in each image resulting from noise or semantic ambiguity. Training without awareness of these uncertainties causes the model to overfit the annotated labels during training and produce unsatisfactory judgments during inference. Motivated by this, we argue that a good similarity model should consider the semantic discrepancies with awareness of the uncertainty to better deal with ambiguous images for more robust training. To achieve this, we propose to represent an image using not only a semantic embedding but also an accompanying uncertainty embedding, which describes the semantic characteristics and ambiguity of an image, respectively. We further propose an introspective similarity metric to make similarity judgments between images considering both their semantic differences and ambiguities. The gradient analysis of the proposed metric shows that it enables the model to learn at an adaptive and slower pace to deal with the uncertainty during training. The proposed IDML framework improves the performance of deep metric learning through uncertainty modeling and attains state-of-the-art results on the widely used CUB-200-2011, Cars196, and Stanford Online Products datasets for image retrieval and clustering. We further provide an in-depth analysis of our framework to demonstrate the effectiveness and reliability of IDML. Code: https://github.com/wzzheng/IDML.

Title: TCGF: A unified tensorized consensus graph framework for multi-view representation learning. (arXiv:2309.09987v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.09987
• Code URL: null
• Copy Paste: [[2309.09987]] TCGF: A unified tensorized consensus graph framework for multi-view representation learning(http://arxiv.org/abs/2309.09987)
• Summary:

  Multi-view learning techniques have recently gained significant attention in the machine learning domain for their ability to leverage consistency and complementary information across multiple views. However, there remains a lack of sufficient research on generalized multi-view frameworks that unify existing works into a scalable and robust learning framework, as most current works focus on specific styles of multi-view models. Additionally, most multi-view learning works rely heavily on specific-scale scenarios and fail to effectively comprehend multiple scales holistically. These limitations hinder the effective fusion of essential information from multiple views, resulting in poor generalization. To address these limitations, this paper proposes a universal multi-view representation learning framework named Tensorized Consensus Graph Framework (TCGF). Specifically, it first provides a unified framework for existing multi-view works to exploit the representations for individual view, which aims to be suitable for arbitrary assumptions and different-scales datasets. Then, stacks them into a tensor under alignment basics as a high-order representation, allowing for the smooth propagation of consistency and complementary information across all views. Moreover, TCGF proposes learning a consensus embedding shared by adaptively collaborating all views to uncover the essential structure of the multi-view data, which utilizes view-consensus grouping effect to regularize the view-consensus representation. To further facilitate related research, we provide a specific implementation of TCGF for large-scale datasets, which can be efficiently solved by applying the alternating optimization strategy. Experimental results conducted on seven different-scales datasets indicate the superiority of the proposed TCGF against existing state-of-the-art multi-view learning methods.

Title: AR-TTA: A Simple Method for Real-World Continual Test-Time Adaptation. (arXiv:2309.10109v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10109
• Code URL: null
• Copy Paste: [[2309.10109]] AR-TTA: A Simple Method for Real-World Continual Test-Time Adaptation(http://arxiv.org/abs/2309.10109)
• Summary:

  Test-time adaptation is a promising research direction that allows the source model to adapt itself to changes in data distribution without any supervision. Yet, current methods are usually evaluated on benchmarks that are only a simplification of real-world scenarios. Hence, we propose to validate test-time adaptation methods using the recently introduced datasets for autonomous driving, namely CLAD-C and SHIFT. We observe that current test-time adaptation methods struggle to effectively handle varying degrees of domain shift, often resulting in degraded performance that falls below that of the source model. We noticed that the root of the problem lies in the inability to preserve the knowledge of the source model and adapt to dynamically changing, temporally correlated data streams. Therefore, we enhance well-established self-training framework by incorporating a small memory buffer to increase model stability and at the same time perform dynamic adaptation based on the intensity of domain shift. The proposed method, named AR-TTA, outperforms existing approaches on both synthetic and more real-world benchmarks and shows robustness across a variety of TTA scenarios.

Title: RGB-based Category-level Object Pose Estimation via Decoupled Metric Scale Recovery. (arXiv:2309.10255v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10255
• Code URL: null
• Copy Paste: [[2309.10255]] RGB-based Category-level Object Pose Estimation via Decoupled Metric Scale Recovery(http://arxiv.org/abs/2309.10255)
• Summary:

  While showing promising results, recent RGB-D camera-based category-level object pose estimation methods have restricted applications due to the heavy reliance on depth sensors. RGB-only methods provide an alternative to this problem yet suffer from inherent scale ambiguity stemming from monocular observations. In this paper, we propose a novel pipeline that decouples the 6D pose and size estimation to mitigate the influence of imperfect scales on rigid transformations. Specifically, we leverage a pre-trained monocular estimator to extract local geometric information, mainly facilitating the search for inlier 2D-3D correspondence. Meanwhile, a separate branch is designed to directly recover the metric scale of the object based on category-level statistics. Finally, we advocate using the RANSAC-P$n$P algorithm to robustly solve for 6D object pose. Extensive experiments have been conducted on both synthetic and real datasets, demonstrating the superior performance of our method over previous state-of-the-art RGB-based approaches, especially in terms of rotation accuracy.

Title: Improving CLIP Robustness with Knowledge Distillation and Self-Training. (arXiv:2309.10361v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10361
• Code URL: null
• Copy Paste: [[2309.10361]] Improving CLIP Robustness with Knowledge Distillation and Self-Training(http://arxiv.org/abs/2309.10361)
• Summary:

  This paper examines the robustness of a multi-modal computer vision model, CLIP (Contrastive Language-Image Pretraining), in the context of unsupervised learning. The main objective is twofold: first, to evaluate the robustness of CLIP, and second, to explore strategies for augmenting its robustness. To achieve this, we introduce a novel approach named LP-CLIP. This technique involves the distillation of CLIP features through the incorporation of a linear probing layer positioned atop its encoding structure. This newly added layer is trained utilizing pseudo-labels produced by CLIP, coupled with a self-training strategy. The LP-CLIP technique offers a promising approach to enhance the robustness of CLIP without the need for annotations. By leveraging a simple linear probing layer, we aim to improve the model's ability to withstand various uncertainties and challenges commonly encountered in real-world scenarios. Importantly, our approach does not rely on annotated data, which makes it particularly valuable in situations where labeled data might be scarce or costly to obtain. Our proposed approach increases the robustness of CLIP with SOTA results compared to supervised technique on various datasets.

Title: Exploiting Causality Signals in Medical Images: A Pilot Study with Empirical Results. (arXiv:2309.10399v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10399
• Code URL: null
• Copy Paste: [[2309.10399]] Exploiting Causality Signals in Medical Images: A Pilot Study with Empirical Results(http://arxiv.org/abs/2309.10399)
• Summary:

  We present a new method for automatically classifying medical images that uses weak causal signals in the scene to model how the presence of a feature in one part of the image affects the appearance of another feature in a different part of the image. Our method consists of two components: a convolutional neural network backbone and a causality-factors extractor module. The latter computes weights for the feature maps to enhance each feature map according to its causal influence in the image's scene. We can modify the functioning of the causality module by using two external signals, thus obtaining different variants of our method. We evaluate our method on a public dataset of prostate MRI images for prostate cancer diagnosis, using quantitative experiments, qualitative assessment, and ablation studies. Our results show that our method improves classification performance and produces more robust predictions, focusing on relevant parts of the image. That is especially important in medical imaging, where accurate and reliable classifications are essential for effective diagnosis and treatment planning.

Title: Exploring Different Levels of Supervision for Detecting and Localizing Solar Panels on Remote Sensing Imagery. (arXiv:2309.10421v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10421
• Code URL: null
• Copy Paste: [[2309.10421]] Exploring Different Levels of Supervision for Detecting and Localizing Solar Panels on Remote Sensing Imagery(http://arxiv.org/abs/2309.10421)
• Summary:

  This study investigates object presence detection and localization in remote sensing imagery, focusing on solar panel recognition. We explore different levels of supervision, evaluating three models: a fully supervised object detector, a weakly supervised image classifier with CAM-based localization, and a minimally supervised anomaly detector. The classifier excels in binary presence detection (0.79 F1-score), while the object detector (0.72) offers precise localization. The anomaly detector requires more data for viable performance. Fusion of model results shows potential accuracy gains. CAM impacts localization modestly, with GradCAM, GradCAM++, and HiResCAM yielding superior results. Notably, the classifier remains robust with less data, in contrast to the object detector.

Title: Sample-adaptive Augmentation for Point Cloud Recognition Against Real-world Corruptions. (arXiv:2309.10431v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10431
• Code URL: https://github.com/roywangj/adaptpoint
• Copy Paste: [[2309.10431]] Sample-adaptive Augmentation for Point Cloud Recognition Against Real-world Corruptions(http://arxiv.org/abs/2309.10431)
• Summary:

  Robust 3D perception under corruption has become an essential task for the realm of 3D vision. While current data augmentation techniques usually perform random transformations on all point cloud objects in an offline way and ignore the structure of the samples, resulting in over-or-under enhancement. In this work, we propose an alternative to make sample-adaptive transformations based on the structure of the sample to cope with potential corruption via an auto-augmentation framework, named as AdaptPoint. Specially, we leverage a imitator, consisting of a Deformation Controller and a Mask Controller, respectively in charge of predicting deformation parameters and producing a per-point mask, based on the intrinsic structural information of the input point cloud, and then conduct corruption simulations on top. Then a discriminator is utilized to prevent the generation of excessive corruption that deviates from the original data distribution. In addition, a perception-guidance feedback mechanism is incorporated to guide the generation of samples with appropriate difficulty level. Furthermore, to address the paucity of real-world corrupted point cloud, we also introduce a new dataset ScanObjectNN-C, that exhibits greater similarity to actual data in real-world environments, especially when contrasted with preceding CAD datasets. Experiments show that our method achieves state-of-the-art results on multiple corruption benchmarks, including ModelNet-C, our ScanObjectNN-C, and ShapeNet-C.

Title: Posterior sampling algorithms for unsupervised speech enhancement with recurrent variational autoencoder. (arXiv:2309.10439v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10439
• Code URL: null
• Copy Paste: [[2309.10439]] Posterior sampling algorithms for unsupervised speech enhancement with recurrent variational autoencoder(http://arxiv.org/abs/2309.10439)
• Summary:

  In this paper, we address the unsupervised speech enhancement problem based on recurrent variational autoencoder (RVAE). This approach offers promising generalization performance over the supervised counterpart. Nevertheless, the involved iterative variational expectation-maximization (VEM) process at test time, which relies on a variational inference method, results in high computational complexity. To tackle this issue, we present efficient sampling techniques based on Langevin dynamics and Metropolis-Hasting algorithms, adapted to the EM-based speech enhancement with RVAE. By directly sampling from the intractable posterior distribution within the EM process, we circumvent the intricacies of variational inference. We conduct a series of experiments, comparing the proposed methods with VEM and a state-of-the-art supervised speech enhancement approach based on diffusion models. The results reveal that our sampling-based algorithms significantly outperform VEM, not only in terms of computational efficiency but also in overall performance. Furthermore, when compared to the supervised baseline, our methods showcase robust generalization performance in mismatched test conditions.

Title: DCPT: Darkness Clue-Prompted Tracking in Nighttime UAVs. (arXiv:2309.10491v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10491
• Code URL: null
• Copy Paste: [[2309.10491]] DCPT: Darkness Clue-Prompted Tracking in Nighttime UAVs(http://arxiv.org/abs/2309.10491)
• Summary:

  Existing nighttime unmanned aerial vehicle (UAV) trackers follow an "Enhance-then-Track" architecture - first using a light enhancer to brighten the nighttime video, then employing a daytime tracker to locate the object. This separate enhancement and tracking fails to build an end-to-end trainable vision system. To address this, we propose a novel architecture called Darkness Clue-Prompted Tracking (DCPT) that achieves robust UAV tracking at night by efficiently learning to generate darkness clue prompts. Without a separate enhancer, DCPT directly encodes anti-dark capabilities into prompts using a darkness clue prompter (DCP). Specifically, DCP iteratively learns emphasizing and undermining projections for darkness clues. It then injects these learned visual prompts into a daytime tracker with fixed parameters across transformer layers. Moreover, a gated feature aggregation mechanism enables adaptive fusion between prompts and between prompts and the base model. Extensive experiments show state-of-the-art performance for DCPT on multiple dark scenario benchmarks. The unified end-to-end learning of enhancement and tracking in DCPT enables a more trainable system. The darkness clue prompting efficiently injects anti-dark knowledge without extra modules. Code and models will be released.

Title: Unsupervised Landmark Discovery Using Consistency Guided Bottleneck. (arXiv:2309.10518v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10518
• Code URL: https://github.com/mamonaawan/cgb_uld
• Copy Paste: [[2309.10518]] Unsupervised Landmark Discovery Using Consistency Guided Bottleneck(http://arxiv.org/abs/2309.10518)
• Summary:

  We study a challenging problem of unsupervised discovery of object landmarks. Many recent methods rely on bottlenecks to generate 2D Gaussian heatmaps however, these are limited in generating informed heatmaps while training, presumably due to the lack of effective structural cues. Also, it is assumed that all predicted landmarks are semantically relevant despite having no ground truth supervision. In the current work, we introduce a consistency-guided bottleneck in an image reconstruction-based pipeline that leverages landmark consistency, a measure of compatibility score with the pseudo-ground truth to generate adaptive heatmaps. We propose obtaining pseudo-supervision via forming landmark correspondence across images. The consistency then modulates the uncertainty of the discovered landmarks in the generation of adaptive heatmaps which rank consistent landmarks above their noisy counterparts, providing effective structural information for improved robustness. Evaluations on five diverse datasets including MAFL, AFLW, LS3D, Cats, and Shoes demonstrate excellent performance of the proposed approach compared to the existing state-of-the-art methods. Our code is publicly available at https://github.com/MamonaAwan/CGB_ULD.

Title: NDDepth: Normal-Distance Assisted Monocular Depth Estimation. (arXiv:2309.10592v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10592
• Code URL: null
• Copy Paste: [[2309.10592]] NDDepth: Normal-Distance Assisted Monocular Depth Estimation(http://arxiv.org/abs/2309.10592)
• Summary:

  Monocular depth estimation has drawn widespread attention from the vision community due to its broad applications. In this paper, we propose a novel physics (geometry)-driven deep learning framework for monocular depth estimation by assuming that 3D scenes are constituted by piece-wise planes. Particularly, we introduce a new normal-distance head that outputs pixel-level surface normal and plane-to-origin distance for deriving depth at each position. Meanwhile, the normal and distance are regularized by a developed plane-aware consistency constraint. We further integrate an additional depth head to improve the robustness of the proposed framework. To fully exploit the strengths of these two heads, we develop an effective contrastive iterative refinement module that refines depth in a complementary manner according to the depth uncertainty. Extensive experiments indicate that the proposed method exceeds previous state-of-the-art competitors on the NYU-Depth-v2, KITTI and SUN RGB-D datasets. Notably, it ranks 1st among all submissions on the KITTI depth prediction online benchmark at the submission time.

Title: Understanding Catastrophic Forgetting in Language Models via Implicit Inference. (arXiv:2309.10105v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.10105
• Code URL: https://github.com/kothasuhas/understanding-forgetting
• Copy Paste: [[2309.10105]] Understanding Catastrophic Forgetting in Language Models via Implicit Inference(http://arxiv.org/abs/2309.10105)
• Summary:

  Fine-tuning (via methods such as instruction-tuning or reinforcement learning from human feedback) is a crucial step in training language models to robustly carry out tasks of interest. However, we lack a systematic understanding of the effects of fine-tuning, particularly on tasks outside the narrow fine-tuning distribution. In a simplified scenario, we demonstrate that improving performance on tasks within the fine-tuning data distribution comes at the expense of suppressing model capabilities on other tasks. This degradation is especially pronounced for tasks "closest" to the fine-tuning distribution. We hypothesize that language models implicitly infer the task of the prompt corresponds, and the fine-tuning process predominantly skews this task inference towards tasks in the fine-tuning distribution. To test this hypothesis, we propose Conjugate Prompting to see if we can recover pretrained capabilities. Conjugate prompting artificially makes the task look farther from the fine-tuning distribution while requiring the same capability. We find that conjugate prompting systematically recovers some of the pretraining capabilities on our synthetic setup. We then apply conjugate prompting to real-world LLMs using the observation that fine-tuning distributions are typically heavily skewed towards English. We find that simply translating the prompts to different languages can cause the fine-tuned models to respond like their pretrained counterparts instead. This allows us to recover the in-context learning abilities lost via instruction tuning, and more concerningly, to recover harmful content generation suppressed by safety fine-tuning in chatbots like ChatGPT.

Title: Analysis of the Memorization and Generalization Capabilities of AI Agents: Are Continual Learners Robust?. (arXiv:2309.10149v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10149
• Code URL: null
• Copy Paste: [[2309.10149]] Analysis of the Memorization and Generalization Capabilities of AI Agents: Are Continual Learners Robust?(http://arxiv.org/abs/2309.10149)
• Summary:

  In continual learning (CL), an AI agent (e.g., autonomous vehicles or robotics) learns from non-stationary data streams under dynamic environments. For the practical deployment of such applications, it is important to guarantee robustness to unseen environments while maintaining past experiences. In this paper, a novel CL framework is proposed to achieve robust generalization to dynamic environments while retaining past knowledge. The considered CL agent uses a capacity-limited memory to save previously observed environmental information to mitigate forgetting issues. Then, data points are sampled from the memory to estimate the distribution of risks over environmental change so as to obtain predictors that are robust with unseen changes. The generalization and memorization performance of the proposed framework are theoretically analyzed. This analysis showcases the tradeoff between memorization and generalization with the memory size. Experiments show that the proposed algorithm outperforms memory-based CL baselines across all environments while significantly improving the generalization performance on unseen target environments.

Title: Causal Theories and Structural Data Representations for Improving Out-of-Distribution Classification. (arXiv:2309.10211v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10211
• Code URL: null
• Copy Paste: [[2309.10211]] Causal Theories and Structural Data Representations for Improving Out-of-Distribution Classification(http://arxiv.org/abs/2309.10211)
• Summary:

  We consider how human-centered causal theories and tools from the dynamical systems literature can be deployed to guide the representation of data when training neural networks for complex classification tasks. Specifically, we use simulated data to show that training a neural network with a data representation that makes explicit the invariant structural causal features of the data generating process of an epidemic system improves out-of-distribution (OOD) generalization performance on a classification task as compared to a more naive approach to data representation. We take these results to demonstrate that using human-generated causal knowledge to reduce the epistemic uncertainty of ML developers can lead to more well-specified ML pipelines. This, in turn, points to the utility of a dynamical systems approach to the broader effort aimed at improving the robustness and safety of machine learning systems via improved ML system development practices.

Title: Koopman Invertible Autoencoder: Leveraging Forward and Backward Dynamics for Temporal Modeling. (arXiv:2309.10291v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10291
• Code URL: null
• Copy Paste: [[2309.10291]] Koopman Invertible Autoencoder: Leveraging Forward and Backward Dynamics for Temporal Modeling(http://arxiv.org/abs/2309.10291)
• Summary:

  Accurate long-term predictions are the foundations for many machine learning applications and decision-making processes. However, building accurate long-term prediction models remains challenging due to the limitations of existing temporal models like recurrent neural networks (RNNs), as they capture only the statistical connections in the training data and may fail to learn the underlying dynamics of the target system. To tackle this challenge, we propose a novel machine learning model based on Koopman operator theory, which we call Koopman Invertible Autoencoders (KIA), that captures the inherent characteristic of the system by modeling both forward and backward dynamics in the infinite-dimensional Hilbert space. This enables us to efficiently learn low-dimensional representations, resulting in more accurate predictions of long-term system behavior. Moreover, our method's invertibility design guarantees reversibility and consistency in both forward and inverse operations. We illustrate the utility of KIA on pendulum and climate datasets, demonstrating 300% improvements in long-term prediction capability for pendulum while maintaining robustness against noise. Additionally, our method excels in long-term climate prediction, further validating our method's effectiveness.

Title: An Extendable Python Implementation of Robust Optimisation Monte Carlo. (arXiv:2309.10612v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10612
• Code URL: null
• Copy Paste: [[2309.10612]] An Extendable Python Implementation of Robust Optimisation Monte Carlo(http://arxiv.org/abs/2309.10612)
• Summary:

  Performing inference in statistical models with an intractable likelihood is challenging, therefore, most likelihood-free inference (LFI) methods encounter accuracy and efficiency limitations. In this paper, we present the implementation of the LFI method Robust Optimisation Monte Carlo (ROMC) in the Python package ELFI. ROMC is a novel and efficient (highly-parallelizable) LFI framework that provides accurate weighted samples from the posterior. Our implementation can be used in two ways. First, a scientist may use it as an out-of-the-box LFI algorithm; we provide an easy-to-use API harmonized with the principles of ELFI, enabling effortless comparisons with the rest of the methods included in the package. Additionally, we have carefully split ROMC into isolated components for supporting extensibility. A researcher may experiment with novel method(s) for solving part(s) of ROMC without reimplementing everything from scratch. In both scenarios, the ROMC parts can run in a fully-parallelized manner, exploiting all CPU cores. We also provide helpful functionalities for (i) inspecting the inference process and (ii) evaluating the obtained samples. Finally, we test the robustness of our implementation on some typical LFI examples.

biometric

steal

Title: Dual Student Networks for Data-Free Model Stealing. (arXiv:2309.10058v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10058
• Code URL: null
• Copy Paste: [[2309.10058]] Dual Student Networks for Data-Free Model Stealing(http://arxiv.org/abs/2309.10058)
• Summary:

  Existing data-free model stealing methods use a generator to produce samples in order to train a student model to match the target model outputs. To this end, the two main challenges are estimating gradients of the target model without access to its parameters, and generating a diverse set of training samples that thoroughly explores the input space. We propose a Dual Student method where two students are symmetrically trained in order to provide the generator a criterion to generate samples that the two students disagree on. On one hand, disagreement on a sample implies at least one student has classified the sample incorrectly when compared to the target model. This incentive towards disagreement implicitly encourages the generator to explore more diverse regions of the input space. On the other hand, our method utilizes gradients of student models to indirectly estimate gradients of the target model. We show that this novel training objective for the generator network is equivalent to optimizing a lower bound on the generator's loss if we had access to the target model gradients. We show that our new optimization framework provides more accurate gradient estimation of the target model and better accuracies on benchmark classification datasets. Additionally, our approach balances improved query efficiency with training computation cost. Finally, we demonstrate that our method serves as a better proxy model for transfer-based adversarial attacks than existing data-free model stealing methods.

extraction

Title: OccluTrack: Rethinking Awareness of Occlusion for Enhancing Multiple Pedestrian Tracking. (arXiv:2309.10360v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10360
• Code URL: null
• Copy Paste: [[2309.10360]] OccluTrack: Rethinking Awareness of Occlusion for Enhancing Multiple Pedestrian Tracking(http://arxiv.org/abs/2309.10360)
• Summary:

  Multiple pedestrian tracking faces the challenge of tracking pedestrians in the presence of occlusion. Existing methods suffer from inaccurate motion estimation, appearance feature extraction, and association due to occlusion, leading to inadequate Identification F1-Score (IDF1), excessive ID switches (IDSw), and insufficient association accuracy and recall (AssA and AssR). We found that the main reason is abnormal detections caused by partial occlusion. In this paper, we suggest that the key insight is explicit motion estimation, reliable appearance features, and fair association in occlusion scenes. Specifically, we propose an adaptive occlusion-aware multiple pedestrian tracker, OccluTrack. We first introduce an abnormal motion suppression mechanism into the Kalman Filter to adaptively detect and suppress outlier motions caused by partial occlusion. Second, we propose a pose-guided re-ID module to extract discriminative part features for partially occluded pedestrians. Last, we design a new occlusion-aware association method towards fair IoU and appearance embedding distance measurement for occluded pedestrians. Extensive evaluation results demonstrate that our OccluTrack outperforms state-of-the-art methods on MOT-Challenge datasets. Particularly, the improvements on IDF1, IDSw, AssA, and AssR demonstrate the effectiveness of our OccluTrack on tracking and association performance.

Title: Few-shot Object Detection in Remote Sensing: Lifting the Curse of Incompletely Annotated Novel Objects. (arXiv:2309.10588v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10588
• Code URL: null
• Copy Paste: [[2309.10588]] Few-shot Object Detection in Remote Sensing: Lifting the Curse of Incompletely Annotated Novel Objects(http://arxiv.org/abs/2309.10588)
• Summary:

  Object detection is an essential and fundamental task in computer vision and satellite image processing. Existing deep learning methods have achieved impressive performance thanks to the availability of large-scale annotated datasets. Yet, in real-world applications the availability of labels is limited. In this context, few-shot object detection (FSOD) has emerged as a promising direction, which aims at enabling the model to detect novel objects with only few of them annotated. However, many existing FSOD algorithms overlook a critical issue: when an input image contains multiple novel objects and only a subset of them are annotated, the unlabeled objects will be considered as background during training. This can cause confusions and severely impact the model's ability to recall novel objects. To address this issue, we propose a self-training-based FSOD (ST-FSOD) approach, which incorporates the self-training mechanism into the few-shot fine-tuning process. ST-FSOD aims to enable the discovery of novel objects that are not annotated, and take them into account during training. On the one hand, we devise a two-branch region proposal networks (RPN) to separate the proposal extraction of base and novel objects, On another hand, we incorporate the student-teacher mechanism into RPN and the region of interest (RoI) head to include those highly confident yet unlabeled targets as pseudo labels. Experimental results demonstrate that our proposed method outperforms the state-of-the-art in various FSOD settings by a large margin. The codes will be publicly available at https://github.com/zhu-xlab/ST-FSOD.

Title: Hierarchy Builder: Organizing Textual Spans into a Hierarchy to Facilitate Navigation. (arXiv:2309.10057v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.10057
• Code URL: null
• Copy Paste: [[2309.10057]] Hierarchy Builder: Organizing Textual Spans into a Hierarchy to Facilitate Navigation(http://arxiv.org/abs/2309.10057)
• Summary:

  Information extraction systems often produce hundreds to thousands of strings on a specific topic. We present a method that facilitates better consumption of these strings, in an exploratory setting in which a user wants to both get a broad overview of what's available, and a chance to dive deeper on some aspects. The system works by grouping similar items together and arranging the remaining items into a hierarchical navigable DAG structure. We apply the method to medical information extraction.

Title: FRACAS: A FRench Annotated Corpus of Attribution relations in newS. (arXiv:2309.10604v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.10604
• Code URL: null
• Copy Paste: [[2309.10604]] FRACAS: A FRench Annotated Corpus of Attribution relations in newS(http://arxiv.org/abs/2309.10604)
• Summary:

  Quotation extraction is a widely useful task both from a sociological and from a Natural Language Processing perspective. However, very little data is available to study this task in languages other than English. In this paper, we present a manually annotated corpus of 1676 newswire texts in French for quotation extraction and source attribution. We first describe the composition of our corpus and the choices that were made in selecting the data. We then detail the annotation guidelines and annotation process, as well as a few statistics about the final corpus and the obtained balance between quote types (direct, indirect and mixed, which are particularly challenging). We end by detailing our inter-annotator agreement between the 8 annotators who worked on manual labelling, which is substantially high for such a difficult linguistic phenomenon.

membership infer

federate

Title: FRAMU: Attention-based Machine Unlearning using Federated Reinforcement Learning. (arXiv:2309.10283v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10283
• Code URL: null
• Copy Paste: [[2309.10283]] FRAMU: Attention-based Machine Unlearning using Federated Reinforcement Learning(http://arxiv.org/abs/2309.10283)
• Summary:

  Machine Unlearning is an emerging field that addresses data privacy issues by enabling the removal of private or irrelevant data from the Machine Learning process. Challenges related to privacy and model efficiency arise from the use of outdated, private, and irrelevant data. These issues compromise both the accuracy and the computational efficiency of models in both Machine Learning and Unlearning. To mitigate these challenges, we introduce a novel framework, Attention-based Machine Unlearning using Federated Reinforcement Learning (FRAMU). This framework incorporates adaptive learning mechanisms, privacy preservation techniques, and optimization strategies, making it a well-rounded solution for handling various data sources, either single-modality or multi-modality, while maintaining accuracy and privacy. FRAMU's strength lies in its adaptability to fluctuating data landscapes, its ability to unlearn outdated, private, or irrelevant data, and its support for continual model evolution without compromising privacy. Our experiments, conducted on both single-modality and multi-modality datasets, revealed that FRAMU significantly outperformed baseline models. Additional assessments of convergence behavior and optimization strategies further validate the framework's utility in federated learning applications. Overall, FRAMU advances Machine Unlearning by offering a robust, privacy-preserving solution that optimizes model performance while also addressing key challenges in dynamic data environments.

Title: FedWOA: A Federated Learning Model that uses the Whale Optimization Algorithm for Renewable Energy Prediction. (arXiv:2309.10337v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10337
• Code URL: null
• Copy Paste: [[2309.10337]] FedWOA: A Federated Learning Model that uses the Whale Optimization Algorithm for Renewable Energy Prediction(http://arxiv.org/abs/2309.10337)
• Summary:

  Privacy is important when dealing with sensitive personal information in machine learning models, which require large data sets for training. In the energy field, access to household prosumer energy data is crucial for energy predictions to support energy grid management and large-scale adoption of renewables however citizens are often hesitant to grant access to cloud-based machine learning models. Federated learning has been proposed as a solution to privacy challenges however report issues in generating the global prediction model due to data heterogeneity, variations in generation patterns, and the high number of parameters leading to even lower prediction accuracy. This paper addresses these challenges by introducing FedWOA a novel federated learning model that employs the Whale Optimization Algorithm to aggregate global prediction models from the weights of local LTSM neural network models trained on prosumer energy data. The proposed solution identifies the optimal vector of weights in the search spaces of the local models to construct the global shared model and then is subsequently transmitted to the local nodes to improve the prediction quality at the prosumer site while for handling non-IID data K-Means was used for clustering prosumers with similar scale of energy data. The evaluation results on prosumers energy data have shown that FedWOA can effectively enhance the accuracy of energy prediction models accuracy by 25% for MSE and 16% for MAE compared to FedAVG while demonstrating good convergence and reduced loss.

Title: Toward efficient resource utilization at edge nodes in federated learning. (arXiv:2309.10367v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10367
• Code URL: null
• Copy Paste: [[2309.10367]] Toward efficient resource utilization at edge nodes in federated learning(http://arxiv.org/abs/2309.10367)
• Summary:

  Federated learning (FL) enables edge nodes to collaboratively contribute to constructing a global model without sharing their data. This is accomplished by devices computing local, private model updates that are then aggregated by a server. However, computational resource constraints and network communication can become a severe bottleneck for larger model sizes typical for deep learning applications. Edge nodes tend to have limited hardware resources (RAM, CPU), and the network bandwidth and reliability at the edge is a concern for scaling federated fleet applications. In this paper, we propose and evaluate a FL strategy inspired by transfer learning in order to reduce resource utilization on devices, as well as the load on the server and network in each global training round. For each local model update, we randomly select layers to train, freezing the remaining part of the model. In doing so, we can reduce both server load and communication costs per round by excluding all untrained layer weights from being transferred to the server. The goal of this study is to empirically explore the potential trade-off between resource utilization on devices and global model convergence under the proposed strategy. We implement the approach using the federated learning framework FEDn. A number of experiments were carried out over different datasets (CIFAR-10, CASA, and IMDB), performing different tasks using different deep-learning model architectures. Our results show that training the model partially can accelerate the training process, efficiently utilizes resources on-device, and reduce the data transmission by around 75% and 53% when we train 25%, and 50% of the model layers, respectively, without harming the resulting global model accuracy.

Title: Towards Energy-Aware Federated Traffic Prediction for Cellular Networks. (arXiv:2309.10645v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10645
• Code URL: null
• Copy Paste: [[2309.10645]] Towards Energy-Aware Federated Traffic Prediction for Cellular Networks(http://arxiv.org/abs/2309.10645)
• Summary:

  Cellular traffic prediction is a crucial activity for optimizing networks in fifth-generation (5G) networks and beyond, as accurate forecasting is essential for intelligent network design, resource allocation and anomaly mitigation. Although machine learning (ML) is a promising approach to effectively predict network traffic, the centralization of massive data in a single data center raises issues regarding confidentiality, privacy and data transfer demands. To address these challenges, federated learning (FL) emerges as an appealing ML training framework which offers high accurate predictions through parallel distributed computations. However, the environmental impact of these methods is often overlooked, which calls into question their sustainability. In this paper, we address the trade-off between accuracy and energy consumption in FL by proposing a novel sustainability indicator that allows assessing the feasibility of ML models. Then, we comprehensively evaluate state-of-the-art deep learning (DL) architectures in a federated scenario using real-world measurements from base station (BS) sites in the area of Barcelona, Spain. Our findings indicate that larger ML models achieve marginally improved performance but have a significant environmental impact in terms of carbon footprint, which make them impractical for real-world applications.

fair

Title: A multimodal deep learning architecture for smoking detection with a small data approach. (arXiv:2309.10561v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10561
• Code URL: null
• Copy Paste: [[2309.10561]] A multimodal deep learning architecture for smoking detection with a small data approach(http://arxiv.org/abs/2309.10561)
• Summary:

  Introduction: Covert tobacco advertisements often raise regulatory measures. This paper presents that artificial intelligence, particularly deep learning, has great potential for detecting hidden advertising and allows unbiased, reproducible, and fair quantification of tobacco-related media content. Methods: We propose an integrated text and image processing model based on deep learning, generative methods, and human reinforcement, which can detect smoking cases in both textual and visual formats, even with little available training data. Results: Our model can achieve 74\% accuracy for images and 98\% for text. Furthermore, our system integrates the possibility of expert intervention in the form of human reinforcement. Conclusions: Using the pre-trained multimodal, image, and text processing models available through deep learning makes it possible to detect smoking in different media even with few training data.

Title: KFC: Kinship Verification with Fair Contrastive Loss and Multi-Task Learning. (arXiv:2309.10641v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10641
• Code URL: https://github.com/garynlfd/kfc
• Copy Paste: [[2309.10641]] KFC: Kinship Verification with Fair Contrastive Loss and Multi-Task Learning(http://arxiv.org/abs/2309.10641)
• Summary:

  Kinship verification is an emerging task in computer vision with multiple potential applications. However, there's no large enough kinship dataset to train a representative and robust model, which is a limitation for achieving better performance. Moreover, face verification is known to exhibit bias, which has not been dealt with by previous kinship verification works and sometimes even results in serious issues. So we first combine existing kinship datasets and label each identity with the correct race in order to take race information into consideration and provide a larger and complete dataset, called KinRace dataset. Secondly, we propose a multi-task learning model structure with attention module to enhance accuracy, which surpasses state-of-the-art performance. Lastly, our fairness-aware contrastive loss function with adversarial learning greatly mitigates racial bias. We introduce a debias term into traditional contrastive loss and implement gradient reverse in race classification task, which is an innovative idea to mix two fairness methods to alleviate bias. Exhaustive experimental evaluation demonstrates the effectiveness and superior performance of the proposed KFC in both standard deviation and accuracy at the same time.

interpretability

Title: Stochastic Deep Koopman Model for Quality Propagation Analysis in Multistage Manufacturing Systems. (arXiv:2309.10193v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10193
• Code URL: null
• Copy Paste: [[2309.10193]] Stochastic Deep Koopman Model for Quality Propagation Analysis in Multistage Manufacturing Systems(http://arxiv.org/abs/2309.10193)
• Summary:

  The modeling of multistage manufacturing systems (MMSs) has attracted increased attention from both academia and industry. Recent advancements in deep learning methods provide an opportunity to accomplish this task with reduced cost and expertise. This study introduces a stochastic deep Koopman (SDK) framework to model the complex behavior of MMSs. Specifically, we present a novel application of Koopman operators to propagate critical quality information extracted by variational autoencoders. Through this framework, we can effectively capture the general nonlinear evolution of product quality using a transferred linear representation, thus enhancing the interpretability of the data-driven model. To evaluate the performance of the SDK framework, we carried out a comparative study on an open-source dataset. The main findings of this paper are as follows. Our results indicate that SDK surpasses other popular data-driven models in accuracy when predicting stagewise product quality within the MMS. Furthermore, the unique linear propagation property in the stochastic latent space of SDK enables traceability for quality evolution throughout the process, thereby facilitating the design of root cause analysis schemes. Notably, the proposed framework requires minimal knowledge of the underlying physics of production lines. It serves as a virtual metrology tool that can be applied to various MMSs, contributing to the ultimate goal of Zero Defect Manufacturing.

Title: Graph Neural Networks for Dynamic Modeling of Roller Bearing. (arXiv:2309.10418v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10418
• Code URL: null
• Copy Paste: [[2309.10418]] Graph Neural Networks for Dynamic Modeling of Roller Bearing(http://arxiv.org/abs/2309.10418)
• Summary:

  In the presented work, we propose to apply the framework of graph neural networks (GNNs) to predict the dynamics of a rolling element bearing. This approach offers generalizability and interpretability, having the potential for scalable use in real-time operational digital twin systems for monitoring the health state of rotating machines. By representing the bearing's components as nodes in a graph, the GNN can effectively model the complex relationships and interactions among them. We utilize a dynamic spring-mass-damper model of a bearing to generate the training data for the GNN. In this model, discrete masses represent bearing components such as rolling elements, inner raceways, and outer raceways, while a Hertzian contact model is employed to calculate the forces between these components.

  We evaluate the learning and generalization capabilities of the proposed GNN framework by testing different bearing configurations that deviate from the training configurations. Through this approach, we demonstrate the effectiveness of the GNN-based method in accurately predicting the dynamics of rolling element bearings, highlighting its potential for real-time health monitoring of rotating machinery.

explainability

watermark

diffusion

Title: AutoDiffusion: Training-Free Optimization of Time Steps and Architectures for Automated Diffusion Model Acceleration. (arXiv:2309.10438v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10438
• Code URL: null
• Copy Paste: [[2309.10438]] AutoDiffusion: Training-Free Optimization of Time Steps and Architectures for Automated Diffusion Model Acceleration(http://arxiv.org/abs/2309.10438)
• Summary:

  Diffusion models are emerging expressive generative models, in which a large number of time steps (inference steps) are required for a single image generation. To accelerate such tedious process, reducing steps uniformly is considered as an undisputed principle of diffusion models. We consider that such a uniform assumption is not the optimal solution in practice; i.e., we can find different optimal time steps for different models. Therefore, we propose to search the optimal time steps sequence and compressed model architecture in a unified framework to achieve effective image generation for diffusion models without any further training. Specifically, we first design a unified search space that consists of all possible time steps and various architectures. Then, a two stage evolutionary algorithm is introduced to find the optimal solution in the designed search space. To further accelerate the search process, we employ FID score between generated and real samples to estimate the performance of the sampled examples. As a result, the proposed method is (i).training-free, obtaining the optimal time steps and model architecture without any training process; (ii). orthogonal to most advanced diffusion samplers and can be integrated to gain better sample quality. (iii). generalized, where the searched time steps and architectures can be directly applied on different diffusion models with the same guidance scale. Experimental results show that our method achieves excellent performance by using only a few time steps, e.g. 17.86 FID score on ImageNet 64 $\times$ 64 with only four steps, compared to 138.66 with DDIM.

Title: Unsupervised speech enhancement with diffusion-based generative models. (arXiv:2309.10450v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10450
• Code URL: https://github.com/sp-uhh/sgmse
• Copy Paste: [[2309.10450]] Unsupervised speech enhancement with diffusion-based generative models(http://arxiv.org/abs/2309.10450)
• Summary:

  Recently, conditional score-based diffusion models have gained significant attention in the field of supervised speech enhancement, yielding state-of-the-art performance. However, these methods may face challenges when generalising to unseen conditions. To address this issue, we introduce an alternative approach that operates in an unsupervised manner, leveraging the generative power of diffusion models. Specifically, in a training phase, a clean speech prior distribution is learnt in the short-time Fourier transform (STFT) domain using score-based diffusion models, allowing it to unconditionally generate clean speech from Gaussian noise. Then, we develop a posterior sampling methodology for speech enhancement by combining the learnt clean speech prior with a noise model for speech signal inference. The noise parameters are simultaneously learnt along with clean speech estimation through an iterative expectationmaximisation (EM) approach. To the best of our knowledge, this is the first work exploring diffusion-based generative models for unsupervised speech enhancement, demonstrating promising results compared to a recent variational auto-encoder (VAE)-based unsupervised approach and a state-of-the-art diffusion-based supervised method. It thus opens a new direction for future research in unsupervised speech enhancement.

Title: Diffusion-based speech enhancement with a weighted generative-supervised learning loss. (arXiv:2309.10457v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10457
• Code URL: null
• Copy Paste: [[2309.10457]] Diffusion-based speech enhancement with a weighted generative-supervised learning loss(http://arxiv.org/abs/2309.10457)
• Summary:

  Diffusion-based generative models have recently gained attention in speech enhancement (SE), providing an alternative to conventional supervised methods. These models transform clean speech training samples into Gaussian noise centered at noisy speech, and subsequently learn a parameterized model to reverse this process, conditionally on noisy speech. Unlike supervised methods, generative-based SE approaches usually rely solely on an unsupervised loss, which may result in less efficient incorporation of conditioned noisy speech. To address this issue, we propose augmenting the original diffusion training objective with a mean squared error (MSE) loss, measuring the discrepancy between estimated enhanced speech and ground-truth clean speech at each reverse process iteration. Experimental results demonstrate the effectiveness of our proposed methodology.

Title: Forgedit: Text Guided Image Editing via Learning and Forgetting. (arXiv:2309.10556v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10556
• Code URL: https://github.com/witcherofresearch/forgedit
• Copy Paste: [[2309.10556]] Forgedit: Text Guided Image Editing via Learning and Forgetting(http://arxiv.org/abs/2309.10556)
• Summary:

  Text guided image editing on real images given only the image and the target text prompt as inputs, is a very general and challenging problem, which requires the editing model to reason by itself which part of the image should be edited, to preserve the characteristics of original image, and also to perform complicated non-rigid editing. Previous fine-tuning based solutions are time-consuming and vulnerable to overfitting, limiting their editing capabilities. To tackle these issues, we design a novel text guided image editing method, Forgedit. First, we propose a novel fine-tuning framework which learns to reconstruct the given image in less than one minute by vision language joint learning. Then we introduce vector subtraction and vector projection to explore the proper text embedding for editing. We also find a general property of UNet structures in Diffusion Models and inspired by such a finding, we design forgetting strategies to diminish the fatal overfitting issues and significantly boost the editing abilities of Diffusion Models. Our method, Forgedit, implemented with Stable Diffusion, achieves new state-of-the-art results on the challenging text guided image editing benchmark TEdBench, surpassing the previous SOTA method Imagic with Imagen, in terms of both CLIP score and LPIPS score. Codes are available at https://github.com/witcherofresearch/Forgedit.

noise learning

data-free

transformer

Title: Deep Prompt Tuning for Graph Transformers. (arXiv:2309.10131v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10131
• Code URL: null
• Copy Paste: [[2309.10131]] Deep Prompt Tuning for Graph Transformers(http://arxiv.org/abs/2309.10131)
• Summary:

  Graph transformers have gained popularity in various graph-based tasks by addressing challenges faced by traditional Graph Neural Networks. However, the quadratic complexity of self-attention operations and the extensive layering in graph transformer architectures present challenges when applying them to graph based prediction tasks. Fine-tuning, a common approach, is resource-intensive and requires storing multiple copies of large models. We propose a novel approach called deep graph prompt tuning as an alternative to fine-tuning for leveraging large graph transformer models in downstream graph based prediction tasks. Our method introduces trainable feature nodes to the graph and pre-pends task-specific tokens to the graph transformer, enhancing the model's expressive power. By freezing the pre-trained parameters and only updating the added tokens, our approach reduces the number of free parameters and eliminates the need for multiple model copies, making it suitable for small datasets and scalable to large graphs. Through extensive experiments on various-sized datasets, we demonstrate that deep graph prompt tuning achieves comparable or even superior performance to fine-tuning, despite utilizing significantly fewer task-specific parameters. Our contributions include the introduction of prompt tuning for graph transformers, its application to both graph transformers and message passing graph neural networks, improved efficiency and resource utilization, and compelling experimental results. This work brings attention to a promising approach to leverage pre-trained models in graph based prediction tasks and offers new opportunities for exploring and advancing graph representation learning.

Title: RoadFormer: Duplex Transformer for RGB-Normal Semantic Road Scene Parsing. (arXiv:2309.10356v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10356
• Code URL: null
• Copy Paste: [[2309.10356]] RoadFormer: Duplex Transformer for RGB-Normal Semantic Road Scene Parsing(http://arxiv.org/abs/2309.10356)
• Summary:

  The recent advancements in deep convolutional neural networks have shown significant promise in the domain of road scene parsing. Nevertheless, the existing works focus primarily on freespace detection, with little attention given to hazardous road defects that could compromise both driving safety and comfort. In this paper, we introduce RoadFormer, a novel Transformer-based data-fusion network developed for road scene parsing. RoadFormer utilizes a duplex encoder architecture to extract heterogeneous features from both RGB images and surface normal information. The encoded features are subsequently fed into a novel heterogeneous feature synergy block for effective feature fusion and recalibration. The pixel decoder then learns multi-scale long-range dependencies from the fused and recalibrated heterogeneous features, which are subsequently processed by a Transformer decoder to produce the final semantic prediction. Additionally, we release SYN-UDTIRI, the first large-scale road scene parsing dataset that contains over 10,407 RGB images, dense depth images, and the corresponding pixel-level annotations for both freespace and road defects of different shapes and sizes. Extensive experimental evaluations conducted on our SYN-UDTIRI dataset, as well as on three public datasets, including KITTI road, CityScapes, and ORFD, demonstrate that RoadFormer outperforms all other state-of-the-art networks for road scene parsing. Specifically, RoadFormer ranks first on the KITTI road benchmark. Our source code, created dataset, and demo video are publicly available at mias.group/RoadFormer.

Title: LineMarkNet: Line Landmark Detection for Valet Parking. (arXiv:2309.10475v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10475
• Code URL: null
• Copy Paste: [[2309.10475]] LineMarkNet: Line Landmark Detection for Valet Parking(http://arxiv.org/abs/2309.10475)
• Summary:

  We aim for accurate and efficient line landmark detection for valet parking, which is a long-standing yet unsolved problem in autonomous driving. To this end, we present a deep line landmark detection system where we carefully design the modules to be lightweight. Specifically, we first empirically design four general line landmarks including three physical lines and one novel mental line. The four line landmarks are effective for valet parking. We then develop a deep network (LineMarkNet) to detect line landmarks from surround-view cameras where we, via the pre-calibrated homography, fuse context from four separate cameras into the unified bird-eye-view (BEV) space, specifically we fuse the surroundview features and BEV features, then employ the multi-task decoder to detect multiple line landmarks where we apply the center-based strategy for object detection task, and design our graph transformer to enhance the vision transformer with hierarchical level graph reasoning for semantic segmentation task. At last, we further parameterize the detected line landmarks (e.g., intercept-slope form) whereby a novel filtering backend incorporates temporal and multi-view consistency to achieve smooth and stable detection. Moreover, we annotate a large-scale dataset to validate our method. Experimental results show that our framework achieves the enhanced performance compared with several line detection methods and validate the multi-task network's efficiency about the real-time line landmark detection on the Qualcomm 820A platform while meantime keeps superior accuracy, with our deep line landmark detection system.

Title: KoBigBird-large: Transformation of Transformer for Korean Language Understanding. (arXiv:2309.10339v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.10339
• Code URL: null
• Copy Paste: [[2309.10339]] KoBigBird-large: Transformation of Transformer for Korean Language Understanding(http://arxiv.org/abs/2309.10339)
• Summary:

  This work presents KoBigBird-large, a large size of Korean BigBird that achieves state-of-the-art performance and allows long sequence processing for Korean language understanding. Without further pretraining, we only transform the architecture and extend the positional encoding with our proposed Tapered Absolute Positional Encoding Representations (TAPER). In experiments, KoBigBird-large shows state-of-the-art overall performance on Korean language understanding benchmarks and the best performance on document classification and question answering tasks for longer sequences against the competitive baseline models. We publicly release our model here.

Title: Prognosis of Multivariate Battery State of Performance and Health via Transformers. (arXiv:2309.10014v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10014
• Code URL: null
• Copy Paste: [[2309.10014]] Prognosis of Multivariate Battery State of Performance and Health via Transformers(http://arxiv.org/abs/2309.10014)
• Summary:

  Batteries are an essential component in a deeply decarbonized future. Understanding battery performance and "useful life" as a function of design and use is of paramount importance to accelerating adoption. Historically, battery state of health (SOH) was summarized by a single parameter, the fraction of a battery's capacity relative to its initial state. A more useful approach, however, is a comprehensive characterization of its state and complexities, using an interrelated set of descriptors including capacity, energy, ionic and electronic impedances, open circuit voltages, and microstructure metrics. Indeed, predicting across an extensive suite of properties as a function of battery use is a "holy grail" of battery science; it can provide unprecedented insights toward the design of better batteries with reduced experimental effort, and de-risking energy storage investments that are necessary to meet CO2 reduction targets. In this work, we present a first step in that direction via deep transformer networks for the prediction of 28 battery state of health descriptors using two cycling datasets representing six lithium-ion cathode chemistries (LFP, NMC111, NMC532, NMC622, HE5050, and 5Vspinel), multiple electrolyte/anode compositions, and different charge-discharge scenarios. The accuracy of these predictions versus battery life (with an unprecedented mean absolute error of 19 cycles in predicting end of life for an LFP fast-charging dataset) illustrates the promise of deep learning towards providing deeper understanding and control of battery health.

Title: A Configurable Library for Generating and Manipulating Maze Datasets. (arXiv:2309.10498v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10498
• Code URL: https://github.com/understanding-search/maze-dataset
• Copy Paste: [[2309.10498]] A Configurable Library for Generating and Manipulating Maze Datasets(http://arxiv.org/abs/2309.10498)
• Summary:

  Understanding how machine learning models respond to distributional shifts is a key research challenge. Mazes serve as an excellent testbed due to varied generation algorithms offering a nuanced platform to simulate both subtle and pronounced distributional shifts. To enable systematic investigations of model behavior on out-of-distribution data, we present $\texttt{maze-dataset}$, a comprehensive library for generating, processing, and visualizing datasets consisting of maze-solving tasks. With this library, researchers can easily create datasets, having extensive control over the generation algorithm used, the parameters fed to the algorithm of choice, and the filters that generated mazes must satisfy. Furthermore, it supports multiple output formats, including rasterized and text-based, catering to convolutional neural networks and autoregressive transformer models. These formats, along with tools for visualizing and converting between them, ensure versatility and adaptability in research applications.

generative

Title: Looking through the past: better knowledge retention for generative replay in continual learning. (arXiv:2309.10012v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10012
• Code URL: null
• Copy Paste: [[2309.10012]] Looking through the past: better knowledge retention for generative replay in continual learning(http://arxiv.org/abs/2309.10012)
• Summary:

  In this work, we improve the generative replay in a continual learning setting to perform well on challenging scenarios. Current generative rehearsal methods are usually benchmarked on small and simple datasets as they are not powerful enough to generate more complex data with a greater number of classes. We notice that in VAE-based generative replay, this could be attributed to the fact that the generated features are far from the original ones when mapped to the latent space. Therefore, we propose three modifications that allow the model to learn and generate complex data. More specifically, we incorporate the distillation in latent space between the current and previous models to reduce feature drift. Additionally, a latent matching for the reconstruction and original data is proposed to improve generated features alignment. Further, based on the observation that the reconstructions are better for preserving knowledge, we add the cycling of generations through the previously trained model to make them closer to the original data. Our method outperforms other generative replay methods in various scenarios. Code available at https://github.com/valeriya-khan/looking-through-the-past.

Title: Offline Detection of Misspelled Handwritten Words by Convolving Recognition Model Features with Text Labels. (arXiv:2309.10158v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10158
• Code URL: null
• Copy Paste: [[2309.10158]] Offline Detection of Misspelled Handwritten Words by Convolving Recognition Model Features with Text Labels(http://arxiv.org/abs/2309.10158)
• Summary:

  Offline handwriting recognition (HWR) has improved significantly with the advent of deep learning architectures in recent years. Nevertheless, it remains a challenging problem and practical applications often rely on post-processing techniques for restricting the predicted words via lexicons or language models. Despite their enhanced performance, such systems are less usable in contexts where out-of-vocabulary words are anticipated, e.g. for detecting misspelled words in school assessments. To that end, we introduce the task of comparing a handwriting image to text. To solve the problem, we propose an unrestricted binary classifier, consisting of a HWR feature extractor and a multimodal classification head which convolves the feature extractor output with the vector representation of the input text. Our model's classification head is trained entirely on synthetic data created using a state-of-the-art generative adversarial network. We demonstrate that, while maintaining high recall, the classifier can be calibrated to achieve an average precision increase of 19.5% compared to addressing the task by directly using state-of-the-art HWR models. Such massive performance gains can lead to significant productivity increases in applications utilizing human-in-the-loop automation.

Title: 360$^\circ$ Reconstruction From a Single Image Using Space Carved Outpainting. (arXiv:2309.10279v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10279
• Code URL: null
• Copy Paste: [[2309.10279]] 360$^\circ$ Reconstruction From a Single Image Using Space Carved Outpainting(http://arxiv.org/abs/2309.10279)
• Summary:

  We introduce POP3D, a novel framework that creates a full $360^\circ$-view 3D model from a single image. POP3D resolves two prominent issues that limit the single-view reconstruction. Firstly, POP3D offers substantial generalizability to arbitrary categories, a trait that previous methods struggle to achieve. Secondly, POP3D further improves reconstruction fidelity and naturalness, a crucial aspect that concurrent works fall short of. Our approach marries the strengths of four primary components: (1) a monocular depth and normal predictor that serves to predict crucial geometric cues, (2) a space carving method capable of demarcating the potentially unseen portions of the target object, (3) a generative model pre-trained on a large-scale image dataset that can complete unseen regions of the target, and (4) a neural implicit surface reconstruction method tailored in reconstructing objects using RGB images along with monocular geometric cues. The combination of these components enables POP3D to readily generalize across various in-the-wild images and generate state-of-the-art reconstructions, outperforming similar works by a significant margin. Project page: \url{this http URL}

Title: SideGAN: 3D-Aware Generative Model for Improved Side-View Image Synthesis. (arXiv:2309.10388v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10388
• Code URL: null
• Copy Paste: [[2309.10388]] SideGAN: 3D-Aware Generative Model for Improved Side-View Image Synthesis(http://arxiv.org/abs/2309.10388)
• Summary:

  While recent 3D-aware generative models have shown photo-realistic image synthesis with multi-view consistency, the synthesized image quality degrades depending on the camera pose (e.g., a face with a blurry and noisy boundary at a side viewpoint). Such degradation is mainly caused by the difficulty of learning both pose consistency and photo-realism simultaneously from a dataset with heavily imbalanced poses. In this paper, we propose SideGAN, a novel 3D GAN training method to generate photo-realistic images irrespective of the camera pose, especially for faces of side-view angles. To ease the challenging problem of learning photo-realistic and pose-consistent image synthesis, we split the problem into two subproblems, each of which can be solved more easily. Specifically, we formulate the problem as a combination of two simple discrimination problems, one of which learns to discriminate whether a synthesized image looks real or not, and the other learns to discriminate whether a synthesized image agrees with the camera pose. Based on this, we propose a dual-branched discriminator with two discrimination branches. We also propose a pose-matching loss to learn the pose consistency of 3D GANs. In addition, we present a pose sampling strategy to increase learning opportunities for steep angles in a pose-imbalanced dataset. With extensive validation, we demonstrate that our approach enables 3D GANs to generate high-quality geometries and photo-realistic images irrespective of the camera pose.

Title: What is the Best Automated Metric for Text to Motion Generation?. (arXiv:2309.10248v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.10248
• Code URL: null
• Copy Paste: [[2309.10248]] What is the Best Automated Metric for Text to Motion Generation?(http://arxiv.org/abs/2309.10248)
• Summary:

  There is growing interest in generating skeleton-based human motions from natural language descriptions. While most efforts have focused on developing better neural architectures for this task, there has been no significant work on determining the proper evaluation metric. Human evaluation is the ultimate accuracy measure for this task, and automated metrics should correlate well with human quality judgments. Since descriptions are compatible with many motions, determining the right metric is critical for evaluating and designing effective generative models. This paper systematically studies which metrics best align with human evaluations and proposes new metrics that align even better. Our findings indicate that none of the metrics currently used for this task show even a moderate correlation with human judgments on a sample level. However, for assessing average model performance, commonly used metrics such as R-Precision and less-used coordinate errors show strong correlations. Additionally, several recently developed metrics are not recommended due to their low correlation compared to alternatives. We also introduce a novel metric based on a multimodal BERT-like model, MoBERT, which offers strongly human-correlated sample-level evaluations while maintaining near-perfect model-level correlation. Our results demonstrate that this new metric exhibits extensive benefits over all current alternatives.

Title: OpenMSD: Towards Multilingual Scientific Documents Similarity Measurement. (arXiv:2309.10539v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.10539
• Code URL: https://github.com/google-research/google-research
• Copy Paste: [[2309.10539]] OpenMSD: Towards Multilingual Scientific Documents Similarity Measurement(http://arxiv.org/abs/2309.10539)
• Summary:

  We develop and evaluate multilingual scientific documents similarity measurement models in this work. Such models can be used to find related works in different languages, which can help multilingual researchers find and explore papers more efficiently. We propose the first multilingual scientific documents dataset, Open-access Multilingual Scientific Documents (OpenMSD), which has 74M papers in 103 languages and 778M citation pairs. With OpenMSD, we pretrain science-specialized language models, and explore different strategies to derive "related" paper pairs to fine-tune the models, including using a mixture of citation, co-citation, and bibliographic-coupling pairs. To further improve the models' performance for non-English papers, we explore the use of generative language models to enrich the non-English papers with English summaries. This allows us to leverage the models' English capabilities to create better representations for non-English papers. Our best model significantly outperforms strong baselines by 7-16% (in mean average precision).

large language model

Title: Multimodal Foundation Models: From Specialists to General-Purpose Assistants. (arXiv:2309.10020v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10020
• Code URL: null
• Copy Paste: [[2309.10020]] Multimodal Foundation Models: From Specialists to General-Purpose Assistants(http://arxiv.org/abs/2309.10020)
• Summary:

  This paper presents a comprehensive survey of the taxonomy and evolution of multimodal foundation models that demonstrate vision and vision-language capabilities, focusing on the transition from specialist models to general-purpose assistants. The research landscape encompasses five core topics, categorized into two classes. (i) We start with a survey of well-established research areas: multimodal foundation models pre-trained for specific purposes, including two topics -- methods of learning vision backbones for visual understanding and text-to-image generation. (ii) Then, we present recent advances in exploratory, open research areas: multimodal foundation models that aim to play the role of general-purpose assistants, including three topics -- unified vision models inspired by large language models (LLMs), end-to-end training of multimodal LLMs, and chaining multimodal tools with LLMs. The target audiences of the paper are researchers, graduate students, and professionals in computer vision and vision-language multimodal communities who are eager to learn the basics and recent advances in multimodal foundation models.

Title: A novel approach to measuring patent claim scope based on probabilities obtained from (large) language models. (arXiv:2309.10003v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.10003
• Code URL: null
• Copy Paste: [[2309.10003]] A novel approach to measuring patent claim scope based on probabilities obtained from (large) language models(http://arxiv.org/abs/2309.10003)
• Summary:

  This work proposes to measure the scope of a patent claim as the reciprocal of the self-information contained in this claim. Grounded in information theory, this approach is based on the assumption that a rare concept is more informative than a usual concept, inasmuch as it is more surprising. The self-information is calculated from the probability of occurrence of that claim, where the probability is calculated in accordance with a language model. Five language models are considered, ranging from the simplest models (each word or character is drawn from a uniform distribution) to intermediate models (using average word or character frequencies), to a large language model (GPT2). Interestingly, the simplest language models reduce the scope measure to the reciprocal of the word or character count, a metric already used in previous works. Application is made to nine series of patent claims directed to distinct inventions, where the claims in each series have a gradually decreasing scope. The performance of the language models is then assessed with respect to several ad hoc tests. The more sophisticated the model, the better the results. The GPT2 model outperforms models based on word and character frequencies, which are themselves ahead of models based on word and character counts.

Title: SYNDICOM: Improving Conversational Commonsense with Error-Injection and Natural Language Feedback. (arXiv:2309.10015v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.10015
• Code URL: null
• Copy Paste: [[2309.10015]] SYNDICOM: Improving Conversational Commonsense with Error-Injection and Natural Language Feedback(http://arxiv.org/abs/2309.10015)
• Summary:

  Commonsense reasoning is a critical aspect of human communication. Despite recent advances in conversational AI driven by large language models, commonsense reasoning remains a challenging task. In this work, we introduce SYNDICOM - a method for improving commonsense in dialogue response generation. SYNDICOM consists of two components. The first component is a dataset composed of commonsense dialogues created from a knowledge graph and synthesized into natural language. This dataset includes both valid and invalid responses to dialogue contexts, along with natural language feedback (NLF) for the invalid responses. The second contribution is a two-step procedure: training a model to predict natural language feedback (NLF) for invalid responses, and then training a response generation model conditioned on the predicted NLF, the invalid response, and the dialogue. SYNDICOM is scalable and does not require reinforcement learning. Empirical results on three tasks are evaluated using a broad range of metrics. SYNDICOM achieves a relative improvement of 53% over ChatGPT on ROUGE1, and human evaluators prefer SYNDICOM over ChatGPT 57% of the time. We will publicly release the code and the full dataset.

Title: Few-Shot Adaptation for Parsing Contextual Utterances with LLMs. (arXiv:2309.10168v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.10168
• Code URL: https://github.com/microsoft/few_shot_adaptation_for_parsing_contextual_utterances_with_llms
• Copy Paste: [[2309.10168]] Few-Shot Adaptation for Parsing Contextual Utterances with LLMs(http://arxiv.org/abs/2309.10168)
• Summary:

  We evaluate the ability of semantic parsers based on large language models (LLMs) to handle contextual utterances. In real-world settings, there typically exists only a limited number of annotated contextual utterances due to annotation cost, resulting in an imbalance compared to non-contextual utterances. Therefore, parsers must adapt to contextual utterances with a few training examples. We examine four major paradigms for doing so in conversational semantic parsing i.e., Parse-with-Utterance-History, Parse-with-Reference-Program, Parse-then-Resolve, and Rewrite-then-Parse. To facilitate such cross-paradigm comparisons, we construct SMCalFlow-EventQueries, a subset of contextual examples from SMCalFlow with additional annotations. Experiments with in-context learning and fine-tuning suggest that Rewrite-then-Parse is the most promising paradigm when holistically considering parsing accuracy, annotation cost, and error types.

Title: Stabilizing RLHF through Advantage Model and Selective Rehearsal. (arXiv:2309.10202v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.10202
• Code URL: null
• Copy Paste: [[2309.10202]] Stabilizing RLHF through Advantage Model and Selective Rehearsal(http://arxiv.org/abs/2309.10202)
• Summary:

  Large Language Models (LLMs) have revolutionized natural language processing, yet aligning these models with human values and preferences using RLHF remains a significant challenge. This challenge is characterized by various instabilities, such as reward hacking and catastrophic forgetting. In this technical report, we propose two innovations to stabilize RLHF training: 1) Advantage Model, which directly models advantage score i.e., extra reward compared to the expected rewards and regulates score distributions across tasks to prevent reward hacking. 2) Selective Rehearsal, which mitigates catastrophic forgetting by strategically selecting data for PPO training and knowledge rehearsing. Our experimental analysis on public and proprietary datasets reveals that the proposed methods not only increase stability in RLHF training but also achieve higher reward scores and win rates.

Title: Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognition. (arXiv:2309.10294v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.10294
• Code URL: null
• Copy Paste: [[2309.10294]] Leveraging Speech PTM, Text LLM, and Emotional TTS for Speech Emotion Recognition(http://arxiv.org/abs/2309.10294)
• Summary:

  In this paper, we explored how to boost speech emotion recognition (SER) with the state-of-the-art speech pre-trained model (PTM), data2vec, text generation technique, GPT-4, and speech synthesis technique, Azure TTS. First, we investigated the representation ability of different speech self-supervised pre-trained models, and we found that data2vec has a good representation ability on the SER task. Second, we employed a powerful large language model (LLM), GPT-4, and emotional text-to-speech (TTS) model, Azure TTS, to generate emotionally congruent text and speech. We carefully designed the text prompt and dataset construction, to obtain the synthetic emotional speech data with high quality. Third, we studied different ways of data augmentation to promote the SER task with synthetic speech, including random mixing, adversarial training, transfer learning, and curriculum learning. Experiments and ablation studies on the IEMOCAP dataset demonstrate the effectiveness of our method, compared with other data augmentation methods, and data augmentation with other synthetic data.

Title: Baichuan 2: Open Large-scale Language Models. (arXiv:2309.10305v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.10305
• Code URL: https://github.com/baichuan-inc/baichuan2
• Copy Paste: [[2309.10305]] Baichuan 2: Open Large-scale Language Models(http://arxiv.org/abs/2309.10305)
• Summary:

  Large language models (LLMs) have demonstrated remarkable performance on a variety of natural language tasks based on just a few examples of natural language instructions, reducing the need for extensive feature engineering. However, most powerful LLMs are closed-source or limited in their capability for languages other than English. In this technical report, we present Baichuan 2, a series of large-scale multilingual language models containing 7 billion and 13 billion parameters, trained from scratch, on 2.6 trillion tokens. Baichuan 2 matches or outperforms other open-source models of similar size on public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval. Furthermore, Baichuan 2 excels in vertical domains such as medicine and law. We will release all pre-training model checkpoints to benefit the research community in better understanding the training dynamics of Baichuan 2.

Title: Investigating the Catastrophic Forgetting in Multimodal Large Language Models. (arXiv:2309.10313v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.10313
• Code URL: null
• Copy Paste: [[2309.10313]] Investigating the Catastrophic Forgetting in Multimodal Large Language Models(http://arxiv.org/abs/2309.10313)
• Summary:

  Following the success of GPT4, there has been a surge in interest in multimodal large language model (MLLM) research. This line of research focuses on developing general-purpose LLMs through fine-tuning pre-trained LLMs and vision models. However, catastrophic forgetting, a notorious phenomenon where the fine-tuned model fails to retain similar performance compared to the pre-trained model, still remains an inherent problem in multimodal LLMs (MLLM). In this paper, we introduce EMT: Evaluating MulTimodality for evaluating the catastrophic forgetting in MLLMs, by treating each MLLM as an image classifier. We first apply EMT to evaluate several open-source fine-tuned MLLMs and we discover that almost all evaluated MLLMs fail to retain the same performance levels as their vision encoders on standard image classification tasks. Moreover, we continue fine-tuning LLaVA, an MLLM and utilize EMT to assess performance throughout the fine-tuning. Interestingly, our results suggest that early-stage fine-tuning on an image dataset improves performance across other image datasets, by enhancing the alignment of text and visual features. However, as fine-tuning proceeds, the MLLMs begin to hallucinate, resulting in a significant loss of generalizability, even when the image encoder remains frozen. Our results suggest that MLLMs have yet to demonstrate performance on par with their vision models on standard image classification tasks and the current MLLM fine-tuning procedure still has room for improvement.

Title: Explaining Agent Behavior with Large Language Models. (arXiv:2309.10346v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2309.10346
• Code URL: null
• Copy Paste: [[2309.10346]] Explaining Agent Behavior with Large Language Models(http://arxiv.org/abs/2309.10346)
• Summary:

  Intelligent agents such as robots are increasingly deployed in real-world, safety-critical settings. It is vital that these agents are able to explain the reasoning behind their decisions to human counterparts, however, their behavior is often produced by uninterpretable models such as deep neural networks. We propose an approach to generate natural language explanations for an agent's behavior based only on observations of states and actions, agnostic to the underlying model representation. We show how a compact representation of the agent's behavior can be learned and used to produce plausible explanations with minimal hallucination while affording user interaction with a pre-trained large language model. Through user studies and empirical experiments, we show that our approach generates explanations as helpful as those generated by a human domain expert while enabling beneficial interactions such as clarification and counterfactual queries.

Title: Prompt, Condition, and Generate: Classification of Unsupported Claims with In-Context Learning. (arXiv:2309.10359v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.10359
• Code URL: null
• Copy Paste: [[2309.10359]] Prompt, Condition, and Generate: Classification of Unsupported Claims with In-Context Learning(http://arxiv.org/abs/2309.10359)
• Summary:

  Unsupported and unfalsifiable claims we encounter in our daily lives can influence our view of the world. Characterizing, summarizing, and -- more generally -- making sense of such claims, however, can be challenging. In this work, we focus on fine-grained debate topics and formulate a new task of distilling, from such claims, a countable set of narratives. We present a crowdsourced dataset of 12 controversial topics, comprising more than 120k arguments, claims, and comments from heterogeneous sources, each annotated with a narrative label. We further investigate how large language models (LLMs) can be used to synthesise claims using In-Context Learning. We find that generated claims with supported evidence can be used to improve the performance of narrative classification models and, additionally, that the same model can infer the stance and aspect using a few training examples. Such a model can be useful in applications which rely on narratives , e.g. fact-checking.

Title: PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training. (arXiv:2309.10400v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.10400
• Code URL: https://github.com/dwzhu-pku/pose
• Copy Paste: [[2309.10400]] PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training(http://arxiv.org/abs/2309.10400)
• Summary:

  In this paper, we introduce Positional Skip-wisE (PoSE) training for efficient adaptation of large language models~(LLMs) to extremely long context windows. PoSE decouples train length from target context window size by simulating long inputs using a fixed context window with manipulated position indices during training. Concretely, we select several short chunks from a long input sequence, and introduce distinct skipping bias terms to modify the position indices of each chunk. These bias terms, along with the length of each chunk, are altered for each training example, allowing the model to adapt to all positions within the target context window without training on full length inputs. Experiments show that, compared with fine-tuning on the full length, PoSE greatly reduces memory and time overhead with minimal impact on performance. Leveraging this advantage, we have successfully extended the LLaMA model to 128k tokens. Furthermore, we empirically confirm that PoSE is compatible with all RoPE-based LLMs and various position interpolation strategies. Notably, by decoupling fine-tuning length from target context window, PoSE can theoretically extend the context window infinitely, constrained only by memory usage for inference. With ongoing advancements for efficient inference, we believe PoSE holds great promise for scaling the context window even further.

Title: Toward Unified Controllable Text Generation via Regular Expression Instruction. (arXiv:2309.10447v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.10447
• Code URL: https://github.com/mrzhengxin/ctg-regex-instruction
• Copy Paste: [[2309.10447]] Toward Unified Controllable Text Generation via Regular Expression Instruction(http://arxiv.org/abs/2309.10447)
• Summary:

  Controllable text generation is a fundamental aspect of natural language generation, with numerous methods proposed for different constraint types. However, these approaches often require significant architectural or decoding modifications, making them challenging to apply to additional constraints or resolve different constraint combinations. To address this, our paper introduces Regular Expression Instruction (REI), which utilizes an instruction-based mechanism to fully exploit regular expressions' advantages to uniformly model diverse constraints. Specifically, our REI supports all popular fine-grained controllable generation constraints, i.e., lexical, positional, and length, as well as their complex combinations, via regular expression-style instructions. Our method only requires fine-tuning on medium-scale language models or few-shot, in-context learning on large language models, and requires no further adjustment when applied to various constraint combinations. Experiments demonstrate that our straightforward approach yields high success rates and adaptability to various constraints while maintaining competitiveness in automatic metrics and outperforming most previous baselines.

Title: CFGPT: Chinese Financial Assistant with Large Language Model. (arXiv:2309.10654v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2309.10654
• Code URL: null
• Copy Paste: [[2309.10654]] CFGPT: Chinese Financial Assistant with Large Language Model(http://arxiv.org/abs/2309.10654)
• Summary:

  Large language models (LLMs) have demonstrated great potential in natural language processing tasks within the financial domain. In this work, we present a Chinese Financial Generative Pre-trained Transformer framework, named CFGPT, which includes a dataset~(CFData) for pre-training and supervised fine-tuning, a financial LLM~(CFLLM) to adeptly manage financial texts, and a deployment framework~(CFAPP) designed to navigate real-world financial applications. The CFData comprising both a pre-training dataset and a supervised fine-tuning dataset, where the pre-training dataset collates Chinese financial data and analytics, alongside a smaller subset of general-purpose text with 584M documents and 141B tokens in total, and the supervised fine-tuning dataset is tailored for six distinct financial tasks, embodying various facets of financial analysis and decision-making with 1.5M instruction pairs and 1.5B tokens in total. The CFLLM, which is based on InternLM-7B to balance the model capability and size, is trained on CFData in two stage, continued pre-training and supervised fine-tuning. The CFAPP is centered on large language models (LLMs) and augmented with additional modules to ensure multifaceted functionality in real-world application. Our codes are released at https://github.com/TongjiFinLab/CFGPT.

segmentation

Title: An Empirical Study of Attention Networks for Semantic Segmentation. (arXiv:2309.10217v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10217
• Code URL: null
• Copy Paste: [[2309.10217]] An Empirical Study of Attention Networks for Semantic Segmentation(http://arxiv.org/abs/2309.10217)
• Summary:

  Semantic segmentation is a vital problem in computer vision. Recently, a common solution to semantic segmentation is the end-to-end convolution neural network, which is much more accurate than traditional methods.Recently, the decoders based on attention achieve state-of-the-art (SOTA) performance on various datasets. But these networks always are compared with the mIoU of previous SOTA networks to prove their superiority and ignore their characteristics without considering the computation complexity and precision in various categories, which is essential for engineering applications. Besides, the methods to analyze the FLOPs and memory are not consistent between different networks, which makes the comparison hard to be utilized. What's more, various methods utilize attention in semantic segmentation, but the conclusion of these methods is lacking. This paper first conducts experiments to analyze their computation complexity and compare their performance. Then it summarizes suitable scenes for these networks and concludes key points that should be concerned when constructing an attention network. Last it points out some future directions of the attention network.

Title: Multi-level feature fusion network combining attention mechanisms for polyp segmentation. (arXiv:2309.10219v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10219
• Code URL: null
• Copy Paste: [[2309.10219]] Multi-level feature fusion network combining attention mechanisms for polyp segmentation(http://arxiv.org/abs/2309.10219)
• Summary:

  Clinically, automated polyp segmentation techniques have the potential to significantly improve the efficiency and accuracy of medical diagnosis, thereby reducing the risk of colorectal cancer in patients. Unfortunately, existing methods suffer from two significant weaknesses that can impact the accuracy of segmentation. Firstly, features extracted by encoders are not adequately filtered and utilized. Secondly, semantic conflicts and information redundancy caused by feature fusion are not attended to. To overcome these limitations, we propose a novel approach for polyp segmentation, named MLFF-Net, which leverages multi-level feature fusion and attention mechanisms. Specifically, MLFF-Net comprises three modules: Multi-scale Attention Module (MAM), High-level Feature Enhancement Module (HFEM), and Global Attention Module (GAM). Among these, MAM is used to extract multi-scale information and polyp details from the shallow output of the encoder. In HFEM, the deep features of the encoders complement each other by aggregation. Meanwhile, the attention mechanism redistributes the weight of the aggregated features, weakening the conflicting redundant parts and highlighting the information useful to the task. GAM combines features from the encoder and decoder features, as well as computes global dependencies to prevent receptive field locality. Experimental results on five public datasets show that the proposed method not only can segment multiple types of polyps but also has advantages over current state-of-the-art methods in both accuracy and generalization ability.

Title: UPL-SFDA: Uncertainty-aware Pseudo Label Guided Source-Free Domain Adaptation for Medical Image Segmentation. (arXiv:2309.10244v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10244
• Code URL: https://github.com/hilab-git/upl-sfda
• Copy Paste: [[2309.10244]] UPL-SFDA: Uncertainty-aware Pseudo Label Guided Source-Free Domain Adaptation for Medical Image Segmentation(http://arxiv.org/abs/2309.10244)
• Summary:

  Domain Adaptation (DA) is important for deep learning-based medical image segmentation models to deal with testing images from a new target domain. As the source-domain data are usually unavailable when a trained model is deployed at a new center, Source-Free Domain Adaptation (SFDA) is appealing for data and annotation-efficient adaptation to the target domain. However, existing SFDA methods have a limited performance due to lack of sufficient supervision with source-domain images unavailable and target-domain images unlabeled. We propose a novel Uncertainty-aware Pseudo Label guided (UPL) SFDA method for medical image segmentation. Specifically, we propose Target Domain Growing (TDG) to enhance the diversity of predictions in the target domain by duplicating the pre-trained model's prediction head multiple times with perturbations. The different predictions in these duplicated heads are used to obtain pseudo labels for unlabeled target-domain images and their uncertainty to identify reliable pseudo labels. We also propose a Twice Forward pass Supervision (TFS) strategy that uses reliable pseudo labels obtained in one forward pass to supervise predictions in the next forward pass. The adaptation is further regularized by a mean prediction-based entropy minimization term that encourages confident and consistent results in different prediction heads. UPL-SFDA was validated with a multi-site heart MRI segmentation dataset, a cross-modality fetal brain segmentation dataset, and a 3D fetal tissue segmentation dataset. It improved the average Dice by 5.54, 5.01 and 6.89 percentage points for the three tasks compared with the baseline, respectively, and outperformed several state-of-the-art SFDA methods.

Title: Fully automated landmarking and facial segmentation on 3D photographs. (arXiv:2309.10472v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10472
• Code URL: https://github.com/rumc3dlab/3dlandmarkdetection
• Copy Paste: [[2309.10472]] Fully automated landmarking and facial segmentation on 3D photographs(http://arxiv.org/abs/2309.10472)
• Summary:

  Three-dimensional facial stereophotogrammetry provides a detailed representation of craniofacial soft tissue without the use of ionizing radiation. While manual annotation of landmarks serves as the current gold standard for cephalometric analysis, it is a time-consuming process and is prone to human error. The aim in this study was to develop and evaluate an automated cephalometric annotation method using a deep learning-based approach. Ten landmarks were manually annotated on 2897 3D facial photographs by a single observer. The automated landmarking workflow involved two successive DiffusionNet models and additional algorithms for facial segmentation. The dataset was randomly divided into a training and test dataset. The training dataset was used to train the deep learning networks, whereas the test dataset was used to evaluate the performance of the automated workflow. The precision of the workflow was evaluated by calculating the Euclidean distances between the automated and manual landmarks and compared to the intra-observer and inter-observer variability of manual annotation and the semi-automated landmarking method. The workflow was successful in 98.6% of all test cases. The deep learning-based landmarking method achieved precise and consistent landmark annotation. The mean precision of 1.69 (+/-1.15) mm was comparable to the inter-observer variability (1.31 +/-0.91 mm) of manual annotation. The Euclidean distance between the automated and manual landmarks was within 2 mm in 69%. Automated landmark annotation on 3D photographs was achieved with the DiffusionNet-based approach. The proposed method allows quantitative analysis of large datasets and may be used in diagnosis, follow-up, and virtual surgical planning.

Title: RECALL+: Adversarial Web-based Replay for Continual Learning in Semantic Segmentation. (arXiv:2309.10479v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10479
• Code URL: null
• Copy Paste: [[2309.10479]] RECALL+: Adversarial Web-based Replay for Continual Learning in Semantic Segmentation(http://arxiv.org/abs/2309.10479)
• Summary:

  Catastrophic forgetting of previous knowledge is a critical issue in continual learning typically handled through various regularization strategies. However, existing methods struggle especially when several incremental steps are performed. In this paper, we extend our previous approach (RECALL) and tackle forgetting by exploiting unsupervised web-crawled data to retrieve examples of old classes from online databases. Differently from the original approach that did not perform any evaluation of the web data, here we introduce two novel approaches based on adversarial learning and adaptive thresholding to select from web data only samples strongly resembling the statistics of the no longer available training ones. Furthermore, we improved the pseudo-labeling scheme to achieve a more accurate labeling of web data that also consider classes being learned in the current step. Experimental results show that this enhanced approach achieves remarkable results, especially when multiple incremental learning steps are performed.

Title: Single-Image based unsupervised joint segmentation and denoising. (arXiv:2309.10511v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10511
• Code URL: null
• Copy Paste: [[2309.10511]] Single-Image based unsupervised joint segmentation and denoising(http://arxiv.org/abs/2309.10511)
• Summary:

  In this work, we develop an unsupervised method for the joint segmentation and denoising of a single image. To this end, we combine the advantages of a variational segmentation method with the power of a self-supervised, single-image based deep learning approach. One major strength of our method lies in the fact, that in contrast to data-driven methods, where huge amounts of labeled samples are necessary, our model can segment an image into multiple meaningful regions without any training database. Further, we introduce a novel energy functional in which denoising and segmentation are coupled in a way that both tasks benefit from each other. The limitations of existing single-image based variational segmentation methods, which are not capable of dealing with high noise or generic texture, are tackled by this specific combination with self-supervised image denoising. We propose a unified optimisation strategy and show that, especially for very noisy images available in microscopy, our proposed joint approach outperforms its sequential counterpart as well as alternative methods focused purely on denoising or segmentation. Another comparison is conducted with a supervised deep learning approach designed for the same application, highlighting the good performance of our approach.

Title: Uncertainty Estimation in Instance Segmentation with Star-convex Shapes. (arXiv:2309.10513v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10513
• Code URL: null
• Copy Paste: [[2309.10513]] Uncertainty Estimation in Instance Segmentation with Star-convex Shapes(http://arxiv.org/abs/2309.10513)
• Summary:

  Instance segmentation has witnessed promising advancements through deep neural network-based algorithms. However, these models often exhibit incorrect predictions with unwarranted confidence levels. Consequently, evaluating prediction uncertainty becomes critical for informed decision-making. Existing methods primarily focus on quantifying uncertainty in classification or regression tasks, lacking emphasis on instance segmentation. Our research addresses the challenge of estimating spatial certainty associated with the location of instances with star-convex shapes. Two distinct clustering approaches are evaluated which compute spatial and fractional certainty per instance employing samples by the Monte-Carlo Dropout or Deep Ensemble technique. Our study demonstrates that combining spatial and fractional certainty scores yields improved calibrated estimation over individual certainty scores. Notably, our experimental results show that the Deep Ensemble technique alongside our novel radial clustering approach proves to be an effective strategy. Our findings emphasize the significance of evaluating the calibration of estimated certainties for model reliability and decision-making.

Title: Spatial-Assistant Encoder-Decoder Network for Real Time Semantic Segmentation. (arXiv:2309.10519v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10519
• Code URL: https://github.com/cuzaoo/sanet-main
• Copy Paste: [[2309.10519]] Spatial-Assistant Encoder-Decoder Network for Real Time Semantic Segmentation(http://arxiv.org/abs/2309.10519)
• Summary:

  Semantic segmentation is an essential technology for self-driving cars to comprehend their surroundings. Currently, real-time semantic segmentation networks commonly employ either encoder-decoder architecture or two-pathway architecture. Generally speaking, encoder-decoder models tend to be quicker,whereas two-pathway models exhibit higher accuracy. To leverage both strengths, we present the Spatial-Assistant Encoder-Decoder Network (SANet) to fuse the two architectures. In the overall architecture, we uphold the encoder-decoder design while maintaining the feature maps in the middle section of the encoder and utilizing atrous convolution branches for same-resolution feature extraction. Toward the end of the encoder, we integrate the asymmetric pooling pyramid pooling module (APPPM) to optimize the semantic extraction of the feature maps. This module incorporates asymmetric pooling layers that extract features at multiple resolutions. In the decoder, we present a hybrid attention module, SAD, that integrates horizontal and vertical attention to facilitate the combination of various branches. To ascertain the effectiveness of our approach, our SANet model achieved competitive results on the real-time CamVid and cityscape datasets. By employing a single 2080Ti GPU, SANet achieved a 78.4 % mIOU at 65.1 FPS on the Cityscape test dataset and 78.8 % mIOU at 147 FPS on the CamVid test dataset. The training code and model for SANet are available at https://github.com/CuZaoo/SANet-main

Title: Edge-aware Feature Aggregation Network for Polyp Segmentation. (arXiv:2309.10523v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10523
• Code URL: null
• Copy Paste: [[2309.10523]] Edge-aware Feature Aggregation Network for Polyp Segmentation(http://arxiv.org/abs/2309.10523)
• Summary:

  Precise polyp segmentation is vital for the early diagnosis and prevention of colorectal cancer (CRC) in clinical practice. However, due to scale variation and blurry polyp boundaries, it is still a challenging task to achieve satisfactory segmentation performance with different scales and shapes. In this study, we present a novel Edge-aware Feature Aggregation Network (EFA-Net) for polyp segmentation, which can fully make use of cross-level and multi-scale features to enhance the performance of polyp segmentation. Specifically, we first present an Edge-aware Guidance Module (EGM) to combine the low-level features with the high-level features to learn an edge-enhanced feature, which is incorporated into each decoder unit using a layer-by-layer strategy. Besides, a Scale-aware Convolution Module (SCM) is proposed to learn scale-aware features by using dilated convolutions with different ratios, in order to effectively deal with scale variation. Further, a Cross-level Fusion Module (CFM) is proposed to effectively integrate the cross-level features, which can exploit the local and global contextual information. Finally, the outputs of CFMs are adaptively weighted by using the learned edge-aware feature, which are then used to produce multiple side-out segmentation maps. Experimental results on five widely adopted colonoscopy datasets show that our EFA-Net outperforms state-of-the-art polyp segmentation methods in terms of generalization and effectiveness.

Title: SPOT: Scalable 3D Pre-training via Occupancy Prediction for Autonomous Driving. (arXiv:2309.10527v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10527
• Code URL: https://github.com/pjlab-adg/3dtrans
• Copy Paste: [[2309.10527]] SPOT: Scalable 3D Pre-training via Occupancy Prediction for Autonomous Driving(http://arxiv.org/abs/2309.10527)
• Summary:

  Annotating 3D LiDAR point clouds for perception tasks including 3D object detection and LiDAR semantic segmentation is notoriously time-and-energy-consuming. To alleviate the burden from labeling, it is promising to perform large-scale pre-training and fine-tune the pre-trained backbone on different downstream datasets as well as tasks. In this paper, we propose SPOT, namely Scalable Pre-training via Occupancy prediction for learning Transferable 3D representations, and demonstrate its effectiveness on various public datasets with different downstream tasks under the label-efficiency setting. Our contributions are threefold: (1) Occupancy prediction is shown to be promising for learning general representations, which is demonstrated by extensive experiments on plenty of datasets and tasks. (2) SPOT uses beam re-sampling technique for point cloud augmentation and applies class-balancing strategies to overcome the domain gap brought by various LiDAR sensors and annotation strategies in different datasets. (3) Scalable pre-training is observed, that is, the downstream performance across all the experiments gets better with more pre-training data. We believe that our findings can facilitate understanding of LiDAR point clouds and pave the way for future exploration in LiDAR pre-training. Codes and models will be released.

Title: Decoupling the Curve Modeling and Pavement Regression for Lane Detection. (arXiv:2309.10533v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10533
• Code URL: null
• Copy Paste: [[2309.10533]] Decoupling the Curve Modeling and Pavement Regression for Lane Detection(http://arxiv.org/abs/2309.10533)
• Summary:

  The curve-based lane representation is a popular approach in many lane detection methods, as it allows for the representation of lanes as a whole object and maximizes the use of holistic information about the lanes. However, the curves produced by these methods may not fit well with irregular lines, which can lead to gaps in performance compared to indirect representations such as segmentation-based or point-based methods. We have observed that these lanes are not intended to be irregular, but they appear zigzagged in the perspective view due to being drawn on uneven pavement. In this paper, we propose a new approach to the lane detection task by decomposing it into two parts: curve modeling and ground height regression. Specifically, we use a parameterized curve to represent lanes in the BEV space to reflect the original distribution of lanes. For the second part, since ground heights are determined by natural factors such as road conditions and are less holistic, we regress the ground heights of key points separately from the curve modeling. Additionally, we have unified the 2D and 3D lane detection tasks by designing a new framework and a series of losses to guide the optimization of models with or without 3D lane labels. Our experiments on 2D lane detection benchmarks (TuSimple and CULane), as well as the recently proposed 3D lane detection datasets (ONCE-3Dlane and OpenLane), have shown significant improvements. We will make our well-documented source code publicly available.

Title: Intelligent Debris Mass Estimation Model for Autonomous Underwater Vehicle. (arXiv:2309.10617v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10617
• Code URL: null
• Copy Paste: [[2309.10617]] Intelligent Debris Mass Estimation Model for Autonomous Underwater Vehicle(http://arxiv.org/abs/2309.10617)
• Summary:

  Marine debris poses a significant threat to the survival of marine wildlife, often leading to entanglement and starvation, ultimately resulting in death. Therefore, removing debris from the ocean is crucial to restore the natural balance and allow marine life to thrive. Instance segmentation is an advanced form of object detection that identifies objects and precisely locates and separates them, making it an essential tool for autonomous underwater vehicles (AUVs) to navigate and interact with their underwater environment effectively. AUVs use image segmentation to analyze images captured by their cameras to navigate underwater environments. In this paper, we use instance segmentation to calculate the area of individual objects within an image, we use YOLOV7 in Roboflow to generate a set of bounding boxes for each object in the image with a class label and a confidence score for every detection. A segmentation mask is then created for each object by applying a binary mask to the object's bounding box. The masks are generated by applying a binary threshold to the output of a convolutional neural network trained to segment objects from the background. Finally, refining the segmentation mask for each object is done by applying post-processing techniques such as morphological operations and contour detection, to improve the accuracy and quality of the mask. The process of estimating the area of instance segmentation involves calculating the area of each segmented instance separately and then summing up the areas of all instances to obtain the total area. The calculation is carried out using standard formulas based on the shape of the object, such as rectangles and circles. In cases where the object is complex, the Monte Carlo method is used to estimate the area. This method provides a higher degree of accuracy than traditional methods, especially when using a large number of samples.

Title: Cross-modal and Cross-domain Knowledge Transfer for Label-free 3D Segmentation. (arXiv:2309.10649v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2309.10649
• Code URL: null
• Copy Paste: [[2309.10649]] Cross-modal and Cross-domain Knowledge Transfer for Label-free 3D Segmentation(http://arxiv.org/abs/2309.10649)
• Summary:

  Current state-of-the-art point cloud-based perception methods usually rely on large-scale labeled data, which requires expensive manual annotations. A natural option is to explore the unsupervised methodology for 3D perception tasks. However, such methods often face substantial performance-drop difficulties. Fortunately, we found that there exist amounts of image-based datasets and an alternative can be proposed, i.e., transferring the knowledge in the 2D images to 3D point clouds. Specifically, we propose a novel approach for the challenging cross-modal and cross-domain adaptation task by fully exploring the relationship between images and point clouds and designing effective feature alignment strategies. Without any 3D labels, our method achieves state-of-the-art performance for 3D point cloud semantic segmentation on SemanticKITTI by using the knowledge of KITTI360 and GTA5, compared to existing unsupervised and weakly-supervised baselines.