secure

Title: PINE: Efficient Norm-Bound Verification for Secret-Shared Vectors. (arXiv:2311.10237v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.10237
• Code URL: null
• Copy Paste: [[2311.10237]] PINE: Efficient Norm-Bound Verification for Secret-Shared Vectors(http://arxiv.org/abs/2311.10237)
• Summary:

  Secure aggregation of high-dimensional vectors is a fundamental primitive in federated statistics and learning. A two-server system such as PRIO allows for scalable aggregation of secret-shared vectors. Adversarial clients might try to manipulate the aggregate, so it is important to ensure that each (secret-shared) contribution is well-formed. In this work, we focus on the important and well-studied goal of ensuring that each contribution vector has bounded Euclidean norm. Existing protocols for ensuring bounded-norm contributions either incur a large communication overhead, or only allow for approximate verification of the norm bound. We propose Private Inexpensive Norm Enforcement (PINE): a new protocol that allows exact norm verification with little communication overhead. For high-dimensional vectors, our approach has a communication overhead of a few percent, compared to the 16-32x overhead of previous approaches.

Title: Secure Instruction and Data-Level Information Flow Tracking Model for RISC-V. (arXiv:2311.10283v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.10283
• Code URL: null
• Copy Paste: [[2311.10283]] Secure Instruction and Data-Level Information Flow Tracking Model for RISC-V(http://arxiv.org/abs/2311.10283)
• Summary:

  Rising device use and third-party IP integration in semiconductors raise security concerns. Unauthorized access, fault injection, and privacy invasion are potential threats from untrusted actors. Different security techniques have been proposed to provide resilience to secure devices from potential vulnerabilities; however, no one technique can be applied as an overarching solution. We propose an integrated Information Flow Tracking (IFT) technique to enable runtime security to protect system integrity by tracking the flow of data from untrusted communication channels. Existing hardware-based IFT schemes are either fine-, which are resource-intensive, or coarse-grained models, which have minimal precision logic, providing either control flow or data-flow integrity. No current security model provides multi-granularity due to the difficulty in balancing both the flexibility and hardware overheads at the same time. This study proposes a multi-level granularity IFT model that integrates a hardware-based IFT technique with a gate-level-based IFT (GLIFT) technique, along with flexibility, for better precision and assessments. Translation from the instruction level to the data level is based on module instantiation with security-critical data for accurate information flow behaviors without any false conservative flows. A simulation-based IFT model is demonstrated, which translates the architecture-specific extensions into a compiler-specific simulation model with toolchain extensions for Reduced Instruction Set Architecture (RISC-V) to verify the security extensions. This approach provides better precision logic by enhancing the tagged mechanism with 1-bit tags and implementing an optimized shadow logic that eliminates the area overhead by tracking the data for only security-critical modules.

Title: A Survey on Off-chain Networks: Frameworks, Technologies, Solutions and Challenges. (arXiv:2311.10298v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.10298
• Code URL: null
• Copy Paste: [[2311.10298]] A Survey on Off-chain Networks: Frameworks, Technologies, Solutions and Challenges(http://arxiv.org/abs/2311.10298)
• Summary:

  Blockchain has received increasing attention in academia and industry. However, the increasing transaction volumes and limited on-chain storage underscore scalability as a key challenge hindering the widespread adoption of blockchain. Fortunately, off-chain networks that enable transactions outside the blockchain show promising potential to mitigate the scalability challenge. Off-chain solutions that address blockchain scalability hurdles, such as payment channel networks, facilitate secure and fast off-chain transactions, thus relieving the main chain's strain. In this article, we provide a comprehensive review of key technologies, solutions, and challenges of off-chain networks. First, we introduce the background of off-chain networks encompassing design motivation, framework, overview, and application scenarios. We then review the key issues and technologies associated with off-chain networks. Subsequently, we summarize the mainstream solutions for the corresponding key issues. Finally, we discuss some research challenges and open issues in this area.

Title: A Large-Scale Study on the Prevalence and Usage of TEE-based Features on Android. (arXiv:2311.10511v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.10511
• Code URL: null
• Copy Paste: [[2311.10511]] A Large-Scale Study on the Prevalence and Usage of TEE-based Features on Android(http://arxiv.org/abs/2311.10511)
• Summary:

  In the realm of mobile security, where OS-based protections have proven insufficient against robust attackers, Trusted Execution Environments (TEEs) have emerged as a hardware-based security technology. Despite the industry's persistence in advancing TEE technology, the impact on end users and developers remains largely unexplored. This study addresses this gap by conducting a large-scale analysis of TEE utilization in Android applications, focusing on the key areas of cryptography, digital rights management, biometric authentication, and secure dialogs.

  To facilitate our extensive analysis, we introduce Mobsec Analytika, a framework tailored for large-scale app examinations, which we make available to the research community. Through the analysis of 170,550 popular Android apps, our analysis illuminates the implementation of TEE-related features and their contextual usage.

  Our findings reveal that TEE features are predominantly utilized indirectly through third-party libraries, with only 6.7% of apps directly invoking the APIs. Moreover, the study reveals the underutilization of the recent TEE-based UI feature Protected Confirmation.

security

Title: Deep Learning based CNN Model for Classification and Detection of Individuals Wearing Face Mask. (arXiv:2311.10408v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10408
• Code URL: null
• Copy Paste: [[2311.10408]] Deep Learning based CNN Model for Classification and Detection of Individuals Wearing Face Mask(http://arxiv.org/abs/2311.10408)
• Summary:

  In response to the global COVID-19 pandemic, there has been a critical demand for protective measures, with face masks emerging as a primary safeguard. The approach involves a two-fold strategy: first, recognizing the presence of a face by detecting faces, and second, identifying masks on those faces. This project utilizes deep learning to create a model that can detect face masks in real-time streaming video as well as images. Face detection, a facet of object detection, finds applications in diverse fields such as security, biometrics, and law enforcement. Various detector systems worldwide have been developed and implemented, with convolutional neural networks chosen for their superior performance accuracy and speed in object detection. Experimental results attest to the model's excellent accuracy on test data. The primary focus of this research is to enhance security, particularly in sensitive areas. The research paper proposes a rapid image pre-processing method with masks centred on faces. Employing feature extraction and Convolutional Neural Network, the system classifies and detects individuals wearing masks. The research unfolds in three stages: image pre-processing, image cropping, and image classification, collectively contributing to the identification of masked faces. Continuous surveillance through webcams or CCTV cameras ensures constant monitoring, triggering a security alert if a person is detected without a mask.

Title: Practical Cybersecurity Ethics: Mapping CyBOK to Ethical Concerns. (arXiv:2311.10165v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.10165
• Code URL: null
• Copy Paste: [[2311.10165]] Practical Cybersecurity Ethics: Mapping CyBOK to Ethical Concerns(http://arxiv.org/abs/2311.10165)
• Summary:

  Research into the ethics of cybersecurity is an established and growing topic of investigation, however the translation of this research into practice is lacking: there exists a small number of professional codes of ethics or codes of practice in cybersecurity, however these are very broad and do not offer much insight into the ethical dilemmas that can be faced while performing specific cybersecurity activities. In order to address this gap, we leverage ongoing work on the Cyber Security Body of Knowledge (CyBOK) to help elicit and document the responsibilities and ethics of the profession. Based on a literature review of the ethics of cybersecurity, we use CyBOK to frame the exploration of ethical challenges in the cybersecurity profession through a series of 15 interviews with cybersecurity experts. Our approach is qualitative and exploratory, aiming to answer the research question "What ethical challenges, insights, and solutions arise in different areas of cybersecurity?". Our findings indicate that there are broad ethical challenges across the whole of cybersecurity, but also that different areas of cybersecurity can face specific ethical considerations for which more detailed guidance can help professionals in those areas. In particular, our findings indicate that security decision-making is expected of all security professionals, but that this requires them to balance a complex mix of technical, objective and subjective points of view, and that resolving conflicts raises challenging ethical dilemmas. We conclude that more work is needed to explore, map, and integrate ethical considerations into cybersecurity practice; the urgent need to conduct further research into the ethics of cybersecurity AI; and highlight the importance of this work for individuals and professional bodies who seek to develop and mature the cybersecurity profession in a responsible manner.

Title: You Cannot Escape Me: Detecting Evasions of SIEM Rules in Enterprise Networks. (arXiv:2311.10197v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.10197
• Code URL: null
• Copy Paste: [[2311.10197]] You Cannot Escape Me: Detecting Evasions of SIEM Rules in Enterprise Networks(http://arxiv.org/abs/2311.10197)
• Summary:

  Cyberattacks have grown into a major risk for organizations, with common consequences being data theft, sabotage, and extortion. Since preventive measures do not suffice to repel attacks, timely detection of successful intruders is crucial to stop them from reaching their final goals. For this purpose, many organizations utilize Security Information and Event Management (SIEM) systems to centrally collect security-related events and scan them for attack indicators using expert-written detection rules. However, as we show by analyzing a set of widespread SIEM detection rules, adversaries can evade almost half of them easily, allowing them to perform common malicious actions within an enterprise network without being detected. To remedy these critical detection blind spots, we propose the idea of adaptive misuse detection, which utilizes machine learning to compare incoming events to SIEM rules on the one hand and known-benign events on the other hand to discover successful evasions. Based on this idea, we present AMIDES, an open-source proof-of-concept adaptive misuse detection system. Using four weeks of SIEM events from a large enterprise network and more than 500 hand-crafted evasions, we show that AMIDES successfully detects a majority of these evasions without any false alerts. In addition, AMIDES eases alert analysis by assessing which rules were evaded. Its computational efficiency qualifies AMIDES for real-world operation and hence enables organizations to significantly reduce detection blind spots with moderate effort.

Title: Towards Stronger Blockchains: Security Against Front-Running Attacks. (arXiv:2311.10253v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.10253
• Code URL: null
• Copy Paste: [[2311.10253]] Towards Stronger Blockchains: Security Against Front-Running Attacks(http://arxiv.org/abs/2311.10253)
• Summary:

  Blockchains add transactions to a distributed shared ledger by arriving at consensus on sets of transactions contained in blocks. This provides a total ordering on a set of global transactions. However, total ordering is not enough to satisfy application semantics under the Byzantine fault model. This is due to the fact that malicious miners and clients can collaborate to add their own transactions ahead of correct clients' transactions in order to gain application level and financial advantages. These attacks fall under the umbrella of front-running attacks. Therefore, total ordering is not strong enough to preserve application semantics. In this paper, we propose causality preserving total order as a solution to this problem. The resulting Blockchains will be stronger than traditional consensus based blockchains and will provide enhanced security ensuring correct application semantics in a Byzantine setting.

Title: A Novel VAPT Algorithm: Enhancing Web Application Security Trough OWASP top 10 Optimization. (arXiv:2311.10450v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.10450
• Code URL: null
• Copy Paste: [[2311.10450]] A Novel VAPT Algorithm: Enhancing Web Application Security Trough OWASP top 10 Optimization(http://arxiv.org/abs/2311.10450)
• Summary:

  This research study is built upon cybersecurity audits and investigates the optimization of an Open Web Application Security Project (OWASP) Top 10 algorithm for Web Applications (WA) security audits using Vulnerability Assessment and Penetration Testing (VAPT) processes. The study places particular emphasis on enhancing the VAPT process by optimizing the OWASP algorithm. To achieve this, the research utilizes desk documents to gain knowledge of WA cybersecurity audits and their associated tools. It also delves into archives to explore VAPT processes and identify techniques, methods, and tools for VAPT automation. Furthermore, the research proposes a prototype optimization that streamlines the two steps of VAPT using the OWASP Top 10 algorithm through an experimental procedure. The results are obtained within a virtual environment, which employs black box testing methods as the primary means of data acquisition and analysis. In this experimental setting, the OWASP algorithm demonstrates an impressive level of precision, achieving a precision rate exceeding 90%. It effectively covers all researched vulnerabilities, thus justifying its optimization. This research contributes significantly to the enhancement of the OWASP algorithm and benefits the offensive security community. It plays a crucial role in ensuring compliance processes for professionals and analysts in the security and software development fields.

Title: Few-shot Message-Enhanced Contrastive Learning for Graph Anomaly Detection. (arXiv:2311.10370v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10370
• Code URL: null
• Copy Paste: [[2311.10370]] Few-shot Message-Enhanced Contrastive Learning for Graph Anomaly Detection(http://arxiv.org/abs/2311.10370)
• Summary:

  Graph anomaly detection plays a crucial role in identifying exceptional instances in graph data that deviate significantly from the majority. It has gained substantial attention in various domains of information security, including network intrusion, financial fraud, and malicious comments, et al. Existing methods are primarily developed in an unsupervised manner due to the challenge in obtaining labeled data. For lack of guidance from prior knowledge in unsupervised manner, the identified anomalies may prove to be data noise or individual data instances. In real-world scenarios, a limited batch of labeled anomalies can be captured, making it crucial to investigate the few-shot problem in graph anomaly detection. Taking advantage of this potential, we propose a novel few-shot Graph Anomaly Detection model called FMGAD (Few-shot Message-Enhanced Contrastive-based Graph Anomaly Detector). FMGAD leverages a self-supervised contrastive learning strategy within and across views to capture intrinsic and transferable structural representations. Furthermore, we propose the Deep-GNN message-enhanced reconstruction module, which extensively exploits the few-shot label information and enables long-range propagation to disseminate supervision signals to deeper unlabeled nodes. This module in turn assists in the training of self-supervised contrastive learning. Comprehensive experimental results on six real-world datasets demonstrate that FMGAD can achieve better performance than other state-of-the-art methods, regardless of artificially injected anomalies or domain-organic anomalies.

Title: Delete My Account: Impact of Data Deletion on Machine Learning Classifiers. (arXiv:2311.10385v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10385
• Code URL: null
• Copy Paste: [[2311.10385]] Delete My Account: Impact of Data Deletion on Machine Learning Classifiers(http://arxiv.org/abs/2311.10385)
• Summary:

  Users are more aware than ever of the importance of their own data, thanks to reports about security breaches and leaks of private, often sensitive data in recent years. Additionally, the GDPR has been in effect in the European Union for over three years and many people have encountered its effects in one way or another. Consequently, more and more users are actively protecting their personal data. One way to do this is to make of the right to erasure guaranteed in the GDPR, which has potential implications for a number of different fields, such as big data and machine learning.

  Our paper presents an in-depth analysis about the impact of the use of the right to erasure on the performance of machine learning models on classification tasks. We conduct various experiments utilising different datasets as well as different machine learning algorithms to analyse a variety of deletion behaviour scenarios. Due to the lack of credible data on actual user behaviour, we make reasonable assumptions for various deletion modes and biases and provide insight into the effects of different plausible scenarios for right to erasure usage on data quality of machine learning. Our results show that the impact depends strongly on the amount of data deleted, the particular characteristics of the dataset and the bias chosen for deletion and assumptions on user behaviour.

privacy

Title: A2XP: Towards Private Domain Generalization. (arXiv:2311.10339v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10339
• Code URL: null
• Copy Paste: [[2311.10339]] A2XP: Towards Private Domain Generalization(http://arxiv.org/abs/2311.10339)
• Summary:

  Deep Neural Networks (DNNs) have become pivotal in various fields, especially in computer vision, outperforming previous methodologies. A critical challenge in their deployment is the bias inherent in data across different domains, such as image style, and environmental conditions, leading to domain gaps. This necessitates techniques for learning general representations from biased training data, known as domain generalization. This paper presents Attend to eXpert Prompts (A2XP), a novel approach for domain generalization that preserves the privacy and integrity of the network architecture. A2XP consists of two phases: Expert Adaptation and Domain Generalization. In the first phase, prompts for each source domain are optimized to guide the model towards the optimal direction. In the second phase, two embedder networks are trained to effectively amalgamate these expert prompts, aiming for an optimal output. Our extensive experiments demonstrate that A2XP achieves state-of-the-art results over existing non-private domain generalization methods. The experimental results validate that the proposed approach not only tackles the domain generalization challenge in DNNs but also offers a privacy-preserving, efficient solution to the broader field of computer vision.

Title: DeepClean: Machine Unlearning on the Cheap by Resetting Privacy Sensitive Weights using the Fisher Diagonal. (arXiv:2311.10448v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10448
• Code URL: null
• Copy Paste: [[2311.10448]] DeepClean: Machine Unlearning on the Cheap by Resetting Privacy Sensitive Weights using the Fisher Diagonal(http://arxiv.org/abs/2311.10448)
• Summary:

  Machine learning models trained on sensitive or private data can inadvertently memorize and leak that information. Machine unlearning seeks to retroactively remove such details from model weights to protect privacy. We contribute a lightweight unlearning algorithm that leverages the Fisher Information Matrix (FIM) for selective forgetting. Prior work in this area requires full retraining or large matrix inversions, which are computationally expensive. Our key insight is that the diagonal elements of the FIM, which measure the sensitivity of log-likelihood to changes in weights, contain sufficient information for effective forgetting. Specifically, we compute the FIM diagonal over two subsets -- the data to retain and forget -- for all trainable weights. This diagonal representation approximates the complete FIM while dramatically reducing computation. We then use it to selectively update weights to maximize forgetting of the sensitive subset while minimizing impact on the retained subset. Experiments show that our algorithm can successfully forget any randomly selected subsets of training data across neural network architectures. By leveraging the FIM diagonal, our approach provides an interpretable, lightweight, and efficient solution for machine unlearning with practical privacy benefits.

Title: FRCSyn Challenge at WACV 2024:Face Recognition Challenge in the Era of Synthetic Data. (arXiv:2311.10476v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10476
• Code URL: null
• Copy Paste: [[2311.10476]] FRCSyn Challenge at WACV 2024:Face Recognition Challenge in the Era of Synthetic Data(http://arxiv.org/abs/2311.10476)
• Summary:

  Despite the widespread adoption of face recognition technology around the world, and its remarkable performance on current benchmarks, there are still several challenges that must be covered in more detail. This paper offers an overview of the Face Recognition Challenge in the Era of Synthetic Data (FRCSyn) organized at WACV 2024. This is the first international challenge aiming to explore the use of synthetic data in face recognition to address existing limitations in the technology. Specifically, the FRCSyn Challenge targets concerns related to data privacy issues, demographic biases, generalization to unseen scenarios, and performance limitations in challenging scenarios, including significant age disparities between enrollment and testing, pose variations, and occlusions. The results achieved in the FRCSyn Challenge, together with the proposed benchmark, contribute significantly to the application of synthetic data to improve face recognition technology.

Title: Gaussian Differential Privacy on Riemannian Manifolds. (arXiv:2311.10101v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.10101
• Code URL: null
• Copy Paste: [[2311.10101]] Gaussian Differential Privacy on Riemannian Manifolds(http://arxiv.org/abs/2311.10101)
• Summary:

  We develop an advanced approach for extending Gaussian Differential Privacy (GDP) to general Riemannian manifolds. The concept of GDP stands out as a prominent privacy definition that strongly warrants extension to manifold settings, due to its central limit properties. By harnessing the power of the renowned Bishop-Gromov theorem in geometric analysis, we propose a Riemannian Gaussian distribution that integrates the Riemannian distance, allowing us to achieve GDP in Riemannian manifolds with bounded Ricci curvature. To the best of our knowledge, this work marks the first instance of extending the GDP framework to accommodate general Riemannian manifolds, encompassing curved spaces, and circumventing the reliance on tangent space summaries. We provide a simple algorithm to evaluate the privacy budget $\mu$ on any one-dimensional manifold and introduce a versatile Markov Chain Monte Carlo (MCMC)-based algorithm to calculate $\mu$ on any Riemannian manifold with constant curvature. Through simulations on one of the most prevalent manifolds in statistics, the unit sphere $S^d$, we demonstrate the superior utility of our Riemannian Gaussian mechanism in comparison to the previously proposed Riemannian Laplace mechanism for implementing GDP.

Title: From Principle to Practice: Vertical Data Minimization for Machine Learning. (arXiv:2311.10500v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10500
• Code URL: https://github.com/eth-sri/datamin
• Copy Paste: [[2311.10500]] From Principle to Practice: Vertical Data Minimization for Machine Learning(http://arxiv.org/abs/2311.10500)
• Summary:

  Aiming to train and deploy predictive models, organizations collect large amounts of detailed client data, risking the exposure of private information in the event of a breach. To mitigate this, policymakers increasingly demand compliance with the data minimization (DM) principle, restricting data collection to only that data which is relevant and necessary for the task. Despite regulatory pressure, the problem of deploying machine learning models that obey DM has so far received little attention. In this work, we address this challenge in a comprehensive manner. We propose a novel vertical DM (vDM) workflow based on data generalization, which by design ensures that no full-resolution client data is collected during training and deployment of models, benefiting client privacy by reducing the attack surface in case of a breach. We formalize and study the corresponding problem of finding generalizations that both maximize data utility and minimize empirical privacy risk, which we quantify by introducing a diverse set of policy-aligned adversarial scenarios. Finally, we propose a range of baseline vDM algorithms, as well as Privacy-aware Tree (PAT), an especially effective vDM algorithm that outperforms all baselines across several settings. We plan to release our code as a publicly available library, helping advance the standardization of DM for machine learning. Overall, we believe our work can help lay the foundation for further exploration and adoption of DM principles in real-world applications.

protect

defense

Title: QKD Entity Source Authentication: Defense-in-Depth for Post Quantum Cryptography. (arXiv:2311.10636v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.10636
• Code URL: null
• Copy Paste: [[2311.10636]] QKD Entity Source Authentication: Defense-in-Depth for Post Quantum Cryptography(http://arxiv.org/abs/2311.10636)
• Summary:

  Quantum key distribution (QKD) was conceived by Charles Bennett and Gilles Brassard in December of 1984. In the ensuing 39 years QKD systems have been deployed around the world to provide secure encryption for terrestrial as well as satellite communication. In 2016 the National Institute of Standards and Technology (NIST) began a program to standardize a series of quantum resistant algorithms to replace our current encryption standards thereby protecting against future quantum computers breaking public key cryptography. This program is known as post quantum cryptography or PQC. One of the tenets of cybersecurity is to use an approach that simultaneously provides multiple protections known as defense-in-depth. This approach seeks to avoid single points of failure. The goal of this paper is to examine the suitability of a hybrid QKD / PQC defense-in-depth strategy. A focus of the paper will be to examine the sufficiency of initial QKD hardware authentication (entity source authentication) which is necessary to guard against man-in-the-middle attacks.

attack

Title: Breaking Temporal Consistency: Generating Video Universal Adversarial Perturbations Using Image Models. (arXiv:2311.10366v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10366
• Code URL: null
• Copy Paste: [[2311.10366]] Breaking Temporal Consistency: Generating Video Universal Adversarial Perturbations Using Image Models(http://arxiv.org/abs/2311.10366)
• Summary:

  As video analysis using deep learning models becomes more widespread, the vulnerability of such models to adversarial attacks is becoming a pressing concern. In particular, Universal Adversarial Perturbation (UAP) poses a significant threat, as a single perturbation can mislead deep learning models on entire datasets. We propose a novel video UAP using image data and image model. This enables us to take advantage of the rich image data and image model-based studies available for video applications. However, there is a challenge that image models are limited in their ability to analyze the temporal aspects of videos, which is crucial for a successful video attack. To address this challenge, we introduce the Breaking Temporal Consistency (BTC) method, which is the first attempt to incorporate temporal information into video attacks using image models. We aim to generate adversarial videos that have opposite patterns to the original. Specifically, BTC-UAP minimizes the feature similarity between neighboring frames in videos. Our approach is simple but effective at attacking unseen video models. Additionally, it is applicable to videos of varying lengths and invariant to temporal shifts. Our approach surpasses existing methods in terms of effectiveness on various datasets, including ImageNet, UCF-101, and Kinetics-400.

Title: Two-Factor Authentication Approach Based on Behavior Patterns for Defeating Puppet Attacks. (arXiv:2311.10389v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10389
• Code URL: null
• Copy Paste: [[2311.10389]] Two-Factor Authentication Approach Based on Behavior Patterns for Defeating Puppet Attacks(http://arxiv.org/abs/2311.10389)
• Summary:

  Fingerprint traits are widely recognized for their unique qualities and security benefits. Despite their extensive use, fingerprint features can be vulnerable to puppet attacks, where attackers manipulate a reluctant but genuine user into completing the authentication process. Defending against such attacks is challenging due to the coexistence of a legitimate identity and an illegitimate intent. In this paper, we propose PUPGUARD, a solution designed to guard against puppet attacks. This method is based on user behavioral patterns, specifically, the user needs to press the capture device twice successively with different fingers during the authentication process. PUPGUARD leverages both the image features of fingerprints and the timing characteristics of the pressing intervals to establish two-factor authentication. More specifically, after extracting image features and timing characteristics, and performing feature selection on the image features, PUPGUARD fuses these two features into a one-dimensional feature vector, and feeds it into a one-class classifier to obtain the classification result. This two-factor authentication method emphasizes dynamic behavioral patterns during the authentication process, thereby enhancing security against puppet attacks. To assess PUPGUARD's effectiveness, we conducted experiments on datasets collected from 31 subjects, including image features and timing characteristics. Our experimental results demonstrate that PUPGUARD achieves an impressive accuracy rate of 97.87% and a remarkably low false positive rate (FPR) of 1.89%. Furthermore, we conducted comparative experiments to validate the superiority of combining image features and timing characteristics within PUPGUARD for enhancing resistance against puppet attacks.

Title: Hashing it Out: Predicting Unhealthy Conversations on Twitter. (arXiv:2311.10596v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10596
• Code URL: null
• Copy Paste: [[2311.10596]] Hashing it Out: Predicting Unhealthy Conversations on Twitter(http://arxiv.org/abs/2311.10596)
• Summary:

  Personal attacks in the context of social media conversations often lead to fast-paced derailment, leading to even more harmful exchanges being made. State-of-the-art systems for the detection of such conversational derailment often make use of deep learning approaches for prediction purposes. In this paper, we show that an Attention-based BERT architecture, pre-trained on a large Twitter corpus and fine-tuned on our task, is efficient and effective in making such predictions. This model shows clear advantages in performance to the existing LSTM model we use as a baseline. Additionally, we show that this impressive performance can be attained through fine-tuning on a relatively small, novel dataset, particularly after mitigating overfitting issues through synthetic oversampling techniques. By introducing the first transformer based model for forecasting conversational events on Twitter, this work lays the foundation for a practical tool to encourage better interactions on one of the most ubiquitous social media platforms.

Title: A Tale of Unrealized Hope: Hardware Performance Counter Against Cache Attacks. (arXiv:2311.10542v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.10542
• Code URL: null
• Copy Paste: [[2311.10542]] A Tale of Unrealized Hope: Hardware Performance Counter Against Cache Attacks(http://arxiv.org/abs/2311.10542)
• Summary:

  This paper investigates an emerging cache side channel attack defense approach involving the use of hardware performance counters (HPCs). These counters monitor microarchitectural events and analyze statistical deviations to differentiate between malicious and benign software. With numerous proposals and promising reported results, we seek to investigate whether published HPC-based detection methods are evaluated in a proper setting and under the right assumptions, such that their quality can be ensured for real-word deployment against cache side-channel attacks. To achieve this goal, this paper presents a comprehensive evaluation and scrutiny of existing literature on the subject matter in a form of a survey, accompanied by experimental evidences to support our evaluation.

robust

Title: VideoCon: Robust Video-Language Alignment via Contrast Captions. (arXiv:2311.10111v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10111
• Code URL: https://github.com/hritikbansal/videocon
• Copy Paste: [[2311.10111]] VideoCon: Robust Video-Language Alignment via Contrast Captions(http://arxiv.org/abs/2311.10111)
• Summary:

  Despite being (pre)trained on a massive amount of data, state-of-the-art video-language alignment models are not robust to semantically-plausible contrastive changes in the video captions. Our work addresses this by identifying a broad spectrum of contrast misalignments, such as replacing entities, actions, and flipping event order, which alignment models should be robust against. To this end, we introduce the VideoCon, a video-language alignment dataset constructed by a large language model that generates plausible contrast video captions and explanations for differences between original and contrast video captions. Then, a generative video-language model is finetuned with VideoCon to assess video-language entailment and generate explanations. Our VideoCon-based alignment model significantly outperforms current models. It exhibits a 12-point increase in AUC for the video-language alignment task on human-generated contrast captions. Finally, our model sets new state of the art zero-shot performance in temporally-extensive video-language tasks such as text-to-video retrieval (SSv2-Temporal) and video question answering (ATP-Hard). Moreover, our model shows superior performance on novel videos and human-crafted captions and explanations. Our code and data are available at https://github.com/Hritikbansal/videocon.

Title: Video-LLaVA: Learning United Visual Representation by Alignment Before Projection. (arXiv:2311.10122v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10122
• Code URL: https://github.com/PKU-YuanGroup/Video-LLaVA
• Copy Paste: [[2311.10122]] Video-LLaVA: Learning United Visual Representation by Alignment Before Projection(http://arxiv.org/abs/2311.10122)
• Summary:

  The Large Vision-Language Model (LVLM) has enhanced the performance of various downstream tasks in visual-language understanding. Most existing approaches encode images and videos into separate feature spaces, which are then fed as inputs to large language models. However, due to the lack of unified tokenization for images and videos, namely misalignment before projection, it becomes challenging for a Large Language Model (LLM) to learn multi-modal interactions from several poor projection layers. In this work, we unify visual representation into the language feature space to advance the foundational LLM towards a unified LVLM. As a result, we establish a simple but robust LVLM baseline, Video-LLaVA, which learns from a mixed dataset of images and videos, mutually enhancing each other. Video-LLaVA achieves superior performances on a broad range of 9 image benchmarks across 5 image question-answering datasets and 4 image benchmark toolkits. Additionally, our Video-LLaVA also outperforms Video-ChatGPT by 5.8%, 9.9%, 18.6%, and 10.1% on MSRVTT, MSVD, TGIF, and ActivityNet, respectively. Notably, extensive experiments demonstrate that Video-LLaVA mutually benefits images and videos within a unified visual representation, outperforming models designed specifically for images or videos.

Title: Towards Improving Robustness Against Common Corruptions using Mixture of Class Specific Experts. (arXiv:2311.10177v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10177
• Code URL: null
• Copy Paste: [[2311.10177]] Towards Improving Robustness Against Common Corruptions using Mixture of Class Specific Experts(http://arxiv.org/abs/2311.10177)
• Summary:

  Neural networks have demonstrated significant accuracy across various domains, yet their vulnerability to subtle input alterations remains a persistent challenge. Conventional methods like data augmentation, while effective to some extent, fall short in addressing unforeseen corruptions, limiting the adaptability of neural networks in real-world scenarios. In response, this paper introduces a novel paradigm known as the Mixture of Class-Specific Expert Architecture. The approach involves disentangling feature learning for individual classes, offering a nuanced enhancement in scalability and overall performance. By training dedicated network segments for each class and subsequently aggregating their outputs, the proposed architecture aims to mitigate vulnerabilities associated with common neural network structures. The study underscores the importance of comprehensive evaluation methodologies, advocating for the incorporation of benchmarks like the common corruptions benchmark. This inclusion provides nuanced insights into the vulnerabilities of neural networks, especially concerning their generalization capabilities and robustness to unforeseen distortions. The research aligns with the broader objective of advancing the development of highly robust learning systems capable of nuanced reasoning across diverse and challenging real-world scenarios. Through this contribution, the paper aims to foster a deeper understanding of neural network limitations and proposes a practical approach to enhance their resilience in the face of evolving and unpredictable conditions.

Title: Vision meets mmWave Radar: 3D Object Perception Benchmark for Autonomous Driving. (arXiv:2311.10261v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10261
• Code URL: null
• Copy Paste: [[2311.10261]] Vision meets mmWave Radar: 3D Object Perception Benchmark for Autonomous Driving(http://arxiv.org/abs/2311.10261)
• Summary:

  Sensor fusion is crucial for an accurate and robust perception system on autonomous vehicles. Most existing datasets and perception solutions focus on fusing cameras and LiDAR. However, the collaboration between camera and radar is significantly under-exploited. The incorporation of rich semantic information from the camera, and reliable 3D information from the radar can potentially achieve an efficient, cheap, and portable solution for 3D object perception tasks. It can also be robust to different lighting or all-weather driving scenarios due to the capability of mmWave radars. In this paper, we introduce the CRUW3D dataset, including 66K synchronized and well-calibrated camera, radar, and LiDAR frames in various driving scenarios. Unlike other large-scale autonomous driving datasets, our radar data is in the format of radio frequency (RF) tensors that contain not only 3D location information but also spatio-temporal semantic information. This kind of radar format can enable machine learning models to generate more reliable object perception results after interacting and fusing the information or features between the camera and radar.

Title: Hierarchical Pruning of Deep Ensembles with Focal Diversity. (arXiv:2311.10293v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10293
• Code URL: null
• Copy Paste: [[2311.10293]] Hierarchical Pruning of Deep Ensembles with Focal Diversity(http://arxiv.org/abs/2311.10293)
• Summary:

  Deep neural network ensembles combine the wisdom of multiple deep neural networks to improve the generalizability and robustness over individual networks. It has gained increasing popularity to study deep ensemble techniques in the deep learning community. Some mission-critical applications utilize a large number of deep neural networks to form deep ensembles to achieve desired accuracy and resilience, which introduces high time and space costs for ensemble execution. However, it still remains a critical challenge whether a small subset of the entire deep ensemble can achieve the same or better generalizability and how to effectively identify these small deep ensembles for improving the space and time efficiency of ensemble execution. This paper presents a novel deep ensemble pruning approach, which can efficiently identify smaller deep ensembles and provide higher ensemble accuracy than the entire deep ensemble of a large number of member networks. Our hierarchical ensemble pruning approach (HQ) leverages three novel ensemble pruning techniques. First, we show that the focal diversity metrics can accurately capture the complementary capacity of the member networks of an ensemble, which can guide ensemble pruning. Second, we design a focal diversity based hierarchical pruning approach, which will iteratively find high quality deep ensembles with low cost and high accuracy. Third, we develop a focal diversity consensus method to integrate multiple focal diversity metrics to refine ensemble pruning results, where smaller deep ensembles can be effectively identified to offer high accuracy, high robustness and high efficiency. Evaluated using popular benchmark datasets, we demonstrate that the proposed hierarchical ensemble pruning approach can effectively identify high quality deep ensembles with better generalizability while being more time and space efficient in ensemble decision making.

Title: High-fidelity Person-centric Subject-to-Image Synthesis. (arXiv:2311.10329v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10329
• Code URL: null
• Copy Paste: [[2311.10329]] High-fidelity Person-centric Subject-to-Image Synthesis(http://arxiv.org/abs/2311.10329)
• Summary:

  Current subject-driven image generation methods encounter significant challenges in person-centric image generation. The reason is that they learn the semantic scene and person generation by fine-tuning a common pre-trained diffusion, which involves an irreconcilable training imbalance. Precisely, to generate realistic persons, they need to sufficiently tune the pre-trained model, which inevitably causes the model to forget the rich semantic scene prior and makes scene generation over-fit to the training data. Moreover, even with sufficient fine-tuning, these methods can still not generate high-fidelity persons since joint learning of the scene and person generation also lead to quality compromise. In this paper, we propose Face-diffuser, an effective collaborative generation pipeline to eliminate the above training imbalance and quality compromise. Specifically, we first develop two specialized pre-trained diffusion models, i.e., Text-driven Diffusion Model (TDM) and Subject-augmented Diffusion Model (SDM), for scene and person generation, respectively. The sampling process is divided into three sequential stages, i.e., semantic scene construction, subject-scene fusion, and subject enhancement. The first and last stages are performed by TDM and SDM respectively. The subject-scene fusion stage, that is the collaboration achieved through a novel and highly effective mechanism, Saliency-adaptive Noise Fusion (SNF). Specifically, it is based on our key observation that there exists a robust link between classifier-free guidance responses and the saliency of generated images. In each time step, SNF leverages the unique strengths of each model and allows for the spatial blending of predicted noises from both models automatically in a saliency-aware manner. Extensive experiments confirm the impressive effectiveness and robustness of the Face-diffuser.

Title: Single-Shot and Multi-Shot Feature Learning for Multi-Object Tracking. (arXiv:2311.10382v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10382
• Code URL: null
• Copy Paste: [[2311.10382]] Single-Shot and Multi-Shot Feature Learning for Multi-Object Tracking(http://arxiv.org/abs/2311.10382)
• Summary:

  Multi-Object Tracking (MOT) remains a vital component of intelligent video analysis, which aims to locate targets and maintain a consistent identity for each target throughout a video sequence. Existing works usually learn a discriminative feature representation, such as motion and appearance, to associate the detections across frames, which are easily affected by mutual occlusion and background clutter in practice. In this paper, we propose a simple yet effective two-stage feature learning paradigm to jointly learn single-shot and multi-shot features for different targets, so as to achieve robust data association in the tracking process. For the detections without being associated, we design a novel single-shot feature learning module to extract discriminative features of each detection, which can efficiently associate targets between adjacent frames. For the tracklets being lost several frames, we design a novel multi-shot feature learning module to extract discriminative features of each tracklet, which can accurately refind these lost targets after a long period. Once equipped with a simple data association logic, the resulting VisualTracker can perform robust MOT based on the single-shot and multi-shot feature representations. Extensive experimental results demonstrate that our method has achieved significant improvements on MOT17 and MOT20 datasets while reaching state-of-the-art performance on DanceTrack dataset.

Title: Joint covariance property under geometric image transformations for spatio-temporal receptive fields according to generalized Gaussian model for receptive fields. (arXiv:2311.10543v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10543
• Code URL: null
• Copy Paste: [[2311.10543]] Joint covariance property under geometric image transformations for spatio-temporal receptive fields according to generalized Gaussian model for receptive fields(http://arxiv.org/abs/2311.10543)
• Summary:

  The influence of natural image transformations on receptive field responses is crucial for modelling visual operations in computer vision and biological vision. In this regard, covariance properties with respect to geometric image transformations in the earliest layers of the visual hierarchy are essential for expressing robust image operations and for formulating invariant visual operations at higher levels. This paper defines and proves a joint covariance property under compositions of spatial scaling transformations, spatial affine transformations, Galilean transformations and temporal scaling transformations, which makes it possible to characterize how different types of image transformations interact with each other. Specifically, the derived relations show the receptive field parameters need to be transformed, in order to match the output from spatio-temporal receptive fields with the underlying spatio-temporal image transformations.

Title: Latent Feature-based Data Splits to Improve Generalisation Evaluation: A Hate Speech Detection Case Study. (arXiv:2311.10236v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10236
• Code URL: https://github.com/maikezuefle/latent-feature-splits
• Copy Paste: [[2311.10236]] Latent Feature-based Data Splits to Improve Generalisation Evaluation: A Hate Speech Detection Case Study(http://arxiv.org/abs/2311.10236)
• Summary:

  With the ever-growing presence of social media platforms comes the increased spread of harmful content and the need for robust hate speech detection systems. Such systems easily overfit to specific targets and keywords, and evaluating them without considering distribution shifts that might occur between train and test data overestimates their benefit. We challenge hate speech models via new train-test splits of existing datasets that rely on the clustering of models' hidden representations. We present two split variants (Subset-Sum-Split and Closest-Split) that, when applied to two datasets using four pretrained models, reveal how models catastrophically fail on blind spots in the latent space. This result generalises when developing a split with one model and evaluating it on another. Our analysis suggests that there is no clear surface-level property of the data split that correlates with the decreased performance, which underscores that task difficulty is not always humanly interpretable. We recommend incorporating latent feature-based splits in model development and release two splits via the GenBench benchmark.

Title: FedTruth: Byzantine-Robust and Backdoor-Resilient Federated Learning Framework. (arXiv:2311.10248v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10248
• Code URL: null
• Copy Paste: [[2311.10248]] FedTruth: Byzantine-Robust and Backdoor-Resilient Federated Learning Framework(http://arxiv.org/abs/2311.10248)
• Summary:

  Federated Learning (FL) enables collaborative machine learning model training across multiple parties without sharing raw data. However, FL's distributed nature allows malicious clients to impact model training through Byzantine or backdoor attacks, using erroneous model updates. Existing defenses measure the deviation of each update from a 'ground-truth model update.' They often rely on a benign root dataset on the server or use trimmed mean or median for clipping, both methods having limitations.

  We introduce FedTruth, a robust defense against model poisoning in FL. FedTruth doesn't assume specific data distributions nor requires a benign root dataset. It estimates a global model update with dynamic aggregation weights, considering contributions from all benign clients. Empirical studies demonstrate FedTruth's efficacy in mitigating the impacts of poisoned updates from both Byzantine and backdoor attacks.

Title: Surprisal Driven $k$-NN for Robust and Interpretable Nonparametric Learning. (arXiv:2311.10246v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10246
• Code URL: null
• Copy Paste: [[2311.10246]] Surprisal Driven $k$-NN for Robust and Interpretable Nonparametric Learning(http://arxiv.org/abs/2311.10246)
• Summary:

  Nonparametric learning is a fundamental concept in machine learning that aims to capture complex patterns and relationships in data without making strong assumptions about the underlying data distribution. Owing to simplicity and familiarity, one of the most well-known algorithms under this paradigm is the $k$-nearest neighbors ($k$-NN) algorithm. Driven by the usage of machine learning in safety-critical applications, in this work, we shed new light on the traditional nearest neighbors algorithm from the perspective of information theory and propose a robust and interpretable framework for tasks such as classification, regression, and anomaly detection using a single model. Instead of using a traditional distance measure which needs to be scaled and contextualized, we use a novel formulation of \textit{surprisal} (amount of information required to explain the difference between the observed and expected result). Finally, we demonstrate this architecture's capability to perform at-par or above the state-of-the-art on classification, regression, and anomaly detection tasks using a single model with enhanced interpretability by providing novel concepts for characterizing data and predictions.

Title: Multiscale Hodge Scattering Networks for Data Analysis. (arXiv:2311.10270v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10270
• Code URL: null
• Copy Paste: [[2311.10270]] Multiscale Hodge Scattering Networks for Data Analysis(http://arxiv.org/abs/2311.10270)
• Summary:

  We propose new scattering networks for signals measured on simplicial complexes, which we call \emph{Multiscale Hodge Scattering Networks} (MHSNs). Our construction is based on multiscale basis dictionaries on simplicial complexes, i.e., the $\kappa$-GHWT and $\kappa$-HGLET, which we recently developed for simplices of dimension $\kappa \in \N$ in a given simplicial complex by generalizing the node-based Generalized Haar-Walsh Transform (GHWT) and Hierarchical Graph Laplacian Eigen Transform (HGLET). The $\kappa$-GHWT and the $\kk$-HGLET both form redundant sets (i.e., dictionaries) of multiscale basis vectors and the corresponding expansion coefficients of a given signal. Our MHSNs use a layered structure analogous to a convolutional neural network (CNN) to cascade the moments of the modulus of the dictionary coefficients. The resulting features are invariant to reordering of the simplices (i.e., node permutation of the underlying graphs). Importantly, the use of multiscale basis dictionaries in our MHSNs admits a natural pooling operation that is akin to local pooling in CNNs, and which may be performed either locally or per-scale. These pooling operations are harder to define in both traditional scattering networks based on Morlet wavelets, and geometric scattering networks based on Diffusion Wavelets. As a result, we are able to extract a rich set of descriptive yet robust features that can be used along with very simple machine learning methods (i.e., logistic regression or support vector machines) to achieve high-accuracy classification systems with far fewer parameters to train than most modern graph neural networks. Finally, we demonstrate the usefulness of our MHSNs in three distinct types of problems: signal classification, domain (i.e., graph/simplex) classification, and molecular dynamics prediction.

Title: Sobol Sequence Optimization for Hardware-Efficient Vector Symbolic Architectures. (arXiv:2311.10277v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10277
• Code URL: null
• Copy Paste: [[2311.10277]] Sobol Sequence Optimization for Hardware-Efficient Vector Symbolic Architectures(http://arxiv.org/abs/2311.10277)
• Summary:

  Hyperdimensional computing (HDC) is an emerging computing paradigm with significant promise for efficient and robust learning. In HDC, objects are encoded with high-dimensional vector symbolic sequences called hypervectors. The quality of hypervectors, defined by their distribution and independence, directly impacts the performance of HDC systems. Despite a large body of work on the processing parts of HDC systems, little to no attention has been paid to data encoding and the quality of hypervectors. Most prior studies have generated hypervectors using inherent random functions, such as MATLAB`s or Python`s random function. This work introduces an optimization technique for generating hypervectors by employing quasi-random sequences. These sequences have recently demonstrated their effectiveness in achieving accurate and low-discrepancy data encoding in stochastic computing systems. The study outlines the optimization steps for utilizing Sobol sequences to produce high-quality hypervectors in HDC systems. An optimization algorithm is proposed to select the most suitable Sobol sequences for generating minimally correlated hypervectors, particularly in applications related to symbol-oriented architectures. The performance of the proposed technique is evaluated in comparison to two traditional approaches of generating hypervectors based on linear-feedback shift registers and MATLAB random function. The evaluation is conducted for two applications: (i) language and (ii) headline classification. Our experimental results demonstrate accuracy improvements of up to 10.79%, depending on the vector size. Additionally, the proposed encoding hardware exhibits reduced energy consumption and a superior area-delay product.

Title: Graph Neural Networks for Pressure Estimation in Water Distribution Systems. (arXiv:2311.10579v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10579
• Code URL: https://github.com/ditec-project/gnn-pressure-estimation
• Copy Paste: [[2311.10579]] Graph Neural Networks for Pressure Estimation in Water Distribution Systems(http://arxiv.org/abs/2311.10579)
• Summary:

  Pressure and flow estimation in Water Distribution Networks (WDN) allows water management companies to optimize their control operations. For many years, mathematical simulation tools have been the most common approach to reconstructing an estimate of the WDN hydraulics. However, pure physics-based simulations involve several challenges, e.g. partially observable data, high uncertainty, and extensive manual configuration. Thus, data-driven approaches have gained traction to overcome such limitations. In this work, we combine physics-based modeling and Graph Neural Networks (GNN), a data-driven approach, to address the pressure estimation problem. First, we propose a new data generation method using a mathematical simulation but not considering temporal patterns and including some control parameters that remain untouched in previous works; this contributes to a more diverse training data. Second, our training strategy relies on random sensor placement making our GNN-based estimation model robust to unexpected sensor location changes. Third, a realistic evaluation protocol considers real temporal patterns and additionally injects the uncertainties intrinsic to real-world scenarios. Finally, a multi-graph pre-training strategy allows the model to be reused for pressure estimation in unseen target WDNs. Our GNN-based model estimates the pressure of a large-scale WDN in The Netherlands with a MAE of 1.94mH$_2$O and a MAPE of 7%, surpassing the performance of previous studies. Likewise, it outperformed previous approaches on other WDN benchmarks, showing a reduction of absolute error up to approximately 52% in the best cases.

Title: Implicit Maximum a Posteriori Filtering via Adaptive Optimization. (arXiv:2311.10580v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10580
• Code URL: https://github.com/gianlucabencomo/implicitmap
• Copy Paste: [[2311.10580]] Implicit Maximum a Posteriori Filtering via Adaptive Optimization(http://arxiv.org/abs/2311.10580)
• Summary:

  Bayesian filtering approximates the true underlying behavior of a time-varying system by inverting an explicit generative model to convert noisy measurements into state estimates. This process typically requires either storage, inversion, and multiplication of large matrices or Monte Carlo estimation, neither of which are practical in high-dimensional state spaces such as the weight spaces of artificial neural networks. Here, we frame the standard Bayesian filtering problem as optimization over a time-varying objective. Instead of maintaining matrices for the filtering equations or simulating particles, we specify an optimizer that defines the Bayesian filter implicitly. In the linear-Gaussian setting, we show that every Kalman filter has an equivalent formulation using K steps of gradient descent. In the nonlinear setting, our experiments demonstrate that our framework results in filters that are effective, robust, and scalable to high-dimensional systems, comparing well against the standard toolbox of Bayesian filtering solutions. We suggest that it is easier to fine-tune an optimizer than it is to specify the correct filtering equations, making our framework an attractive option for high-dimensional filtering problems.

Title: Predicting the Probability of Collision of a Satellite with Space Debris: A Bayesian Machine Learning Approach. (arXiv:2311.10633v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10633
• Code URL: null
• Copy Paste: [[2311.10633]] Predicting the Probability of Collision of a Satellite with Space Debris: A Bayesian Machine Learning Approach(http://arxiv.org/abs/2311.10633)
• Summary:

  Space is becoming more crowded in Low Earth Orbit due to increased space activity. Such a dense space environment increases the risk of collisions between space objects endangering the whole space population. Therefore, the need to consider collision avoidance as part of routine operations is evident to satellite operators. Current procedures rely on the analysis of multiple collision warnings by human analysts. However, with the continuous growth of the space population, this manual approach may become unfeasible, highlighting the importance of automation in risk assessment. In 2019, ESA launched a competition to study the feasibility of applying machine learning in collision risk estimation and released a dataset that contained sequences of Conjunction Data Messages (CDMs) in support of real close encounters. The competition results showed that the naive forecast and its variants are strong predictors for this problem, which suggests that the CDMs may follow the Markov property. The proposed work investigates this theory by benchmarking Hidden Markov Models (HMM) in predicting the risk of collision between two resident space objects by using one feature of the entire dataset: the sequence of the probability in the CDMs. In addition, Bayesian statistics are used to infer a joint distribution for the parameters of the models, which allows the development of robust and reliable probabilistic predictive models that can incorporate physical or prior knowledge about the problem within a rigorous theoretical framework and provides prediction uncertainties that nicely reflect the accuracy of the predicted risk. This work shows that the implemented HMM outperforms the naive solution in some metrics, which further adds to the idea that the collision warnings may be Markovian and suggests that this is a powerful method to be further explored.

biometric

steal

extraction

Title: The Analysis and Extraction of Structure from Organizational Charts. (arXiv:2311.10234v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10234
• Code URL: null
• Copy Paste: [[2311.10234]] The Analysis and Extraction of Structure from Organizational Charts(http://arxiv.org/abs/2311.10234)
• Summary:

  Organizational charts, also known as org charts, are critical representations of an organization's structure and the hierarchical relationships between its components and positions. However, manually extracting information from org charts can be error-prone and time-consuming. To solve this, we present an automated and end-to-end approach that uses computer vision, deep learning, and natural language processing techniques. Additionally, we propose a metric to evaluate the completeness and hierarchical accuracy of the extracted information. This approach has the potential to improve organizational restructuring and resource utilization by providing a clear and concise representation of the organizational structure. Our study lays a foundation for further research on the topic of hierarchical chart analysis.

Title: BiHRNet: A Binary high-resolution network for Human Pose Estimation. (arXiv:2311.10296v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10296
• Code URL: null
• Copy Paste: [[2311.10296]] BiHRNet: A Binary high-resolution network for Human Pose Estimation(http://arxiv.org/abs/2311.10296)
• Summary:

  Human Pose Estimation (HPE) plays a crucial role in computer vision applications. However, it is difficult to deploy state-of-the-art models on resouce-limited devices due to the high computational costs of the networks. In this work, a binary human pose estimator named BiHRNet(Binary HRNet) is proposed, whose weights and activations are expressed as $\pm$1. BiHRNet retains the keypoint extraction ability of HRNet, while using fewer computing resources by adapting binary neural network (BNN). In order to reduce the accuracy drop caused by network binarization, two categories of techniques are proposed in this work. For optimizing the training process for binary pose estimator, we propose a new loss function combining KL divergence loss with AWing loss, which makes the binary network obtain more comprehensive output distribution from its real-valued counterpart to reduce information loss caused by binarization. For designing more binarization-friendly structures, we propose a new information reconstruction bottleneck called IR Bottleneck to retain more information in the initial stage of the network. In addition, we also propose a multi-scale basic block called MS-Block for information retention. Our work has less computation cost with few precision drop. Experimental results demonstrate that BiHRNet achieves a PCKh of 87.9 on the MPII dataset, which outperforms all binary pose estimation networks. On the challenging of COCO dataset, the proposed method enables the binary neural network to achieve 70.8 mAP, which is better than most tested lightweight full-precision networks.

Title: Dates Fruit Disease Recognition using Machine Learning. (arXiv:2311.10365v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10365
• Code URL: null
• Copy Paste: [[2311.10365]] Dates Fruit Disease Recognition using Machine Learning(http://arxiv.org/abs/2311.10365)
• Summary:

  Many countries such as Saudi Arabia, Morocco and Tunisia are among the top exporters and consumers of palm date fruits. Date fruit production plays a major role in the economies of the date fruit exporting countries. Date fruits are susceptible to disease just like any fruit and early detection and intervention can end up saving the produce. However, with the vast farming lands, it is nearly impossible for farmers to observe date trees on a frequent basis for early disease detection. In addition, even with human observation the process is prone to human error and increases the date fruit cost. With the recent advances in computer vision, machine learning, drone technology, and other technologies; an integrated solution can be proposed for the automatic detection of date fruit disease. In this paper, a hybrid features based method with the standard classifiers is proposed based on the extraction of L*a*b color features, statistical features, and Discrete Wavelet Transform (DWT) texture features for the early detection and classification of date fruit disease. A dataset was developed for this work consisting of 871 images divided into the following classes; Healthy date, Initial stage of disease, Malnourished date, and Parasite infected. The extracted features were input to common classifiers such as the Random Forest (RF), Multilayer Perceptron (MLP), Na\"ive Bayes (NB), and Fuzzy Decision Trees (FDT). The highest average accuracy was achieved when combining the L*a*b, Statistical, and DWT Features.

Title: A Relay System for Semantic Image Transmission based on Shared Feature Extraction and Hyperprior Entropy Compression. (arXiv:2311.10492v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10492
• Code URL: null
• Copy Paste: [[2311.10492]] A Relay System for Semantic Image Transmission based on Shared Feature Extraction and Hyperprior Entropy Compression(http://arxiv.org/abs/2311.10492)
• Summary:

  Nowadays, the need for high-quality image reconstruction and restoration is more and more urgent. However, most image transmission systems may suffer from image quality degradation or transmission interruption in the face of interference such as channel noise and link fading. To solve this problem, a relay communication network for semantic image transmission based on shared feature extraction and hyperprior entropy compression (HEC) is proposed, where the shared feature extraction technology based on Pearson correlation is proposed to eliminate partial shared feature of extracted semantic latent feature. In addition, the HEC technology is used to resist the effect of channel noise and link fading and carried out respectively at the source node and the relay node. Experimental results demonstrate that compared with other recent research methods, the proposed system has lower transmission overhead and higher semantic image transmission performance. Particularly, under the same conditions, the multi-scale structural similarity (MS-SSIM) of this system is superior to the comparison method by approximately 0.2.

Title: SpACNN-LDVAE: Spatial Attention Convolutional Latent Dirichlet Variational Autoencoder for Hyperspectral Pixel Unmixing. (arXiv:2311.10701v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10701
• Code URL: null
• Copy Paste: [[2311.10701]] SpACNN-LDVAE: Spatial Attention Convolutional Latent Dirichlet Variational Autoencoder for Hyperspectral Pixel Unmixing(http://arxiv.org/abs/2311.10701)
• Summary:

  The Hyperspectral Unxming problem is to find the pure spectral signal of the underlying materials (endmembers) and their proportions (abundances). The proposed method builds upon the recently proposed method, Latent Dirichlet Variational Autoencoder (LDVAE). It assumes that abundances can be encoded as Dirichlet Distributions while mixed pixels and endmembers are represented by Multivariate Normal Distributions. However, LDVAE does not leverage spatial information present in an HSI; we propose an Isotropic CNN encoder with spatial attention to solve the hyperspectral unmixing problem. We evaluated our model on Samson, Hydice Urban, Cuprite, and OnTech-HSI-Syn-21 datasets. Our model also leverages the transfer learning paradigm for Cuprite Dataset, where we train the model on synthetic data and evaluate it on real-world data. We are able to observe the improvement in the results for the endmember extraction and abundance estimation by incorporating the spatial information. Code can be found at https://github.com/faisalqureshi/cnn-ldvae

Title: FOAL: Fine-grained Contrastive Learning for Cross-domain Aspect Sentiment Triplet Extraction. (arXiv:2311.10373v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10373
• Code URL: null
• Copy Paste: [[2311.10373]] FOAL: Fine-grained Contrastive Learning for Cross-domain Aspect Sentiment Triplet Extraction(http://arxiv.org/abs/2311.10373)
• Summary:

  Aspect Sentiment Triplet Extraction (ASTE) has achieved promising results while relying on sufficient annotation data in a specific domain. However, it is infeasible to annotate data for each individual domain. We propose to explore ASTE in the cross-domain setting, which transfers knowledge from a resource-rich source domain to a resource-poor target domain, thereby alleviating the reliance on labeled data in the target domain. To effectively transfer the knowledge across domains and extract the sentiment triplets accurately, we propose a method named Fine-grained cOntrAstive Learning (FOAL) to reduce the domain discrepancy and preserve the discriminability of each category. Experiments on six transfer pairs show that FOAL achieves 6% performance gains and reduces the domain discrepancy significantly compared with strong baselines. Our code will be publicly available once accepted.

membership infer

federate

Title: Leveraging Function Space Aggregation for Federated Learning at Scale. (arXiv:2311.10291v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10291
• Code URL: null
• Copy Paste: [[2311.10291]] Leveraging Function Space Aggregation for Federated Learning at Scale(http://arxiv.org/abs/2311.10291)
• Summary:

  The federated learning paradigm has motivated the development of methods for aggregating multiple client updates into a global server model, without sharing client data. Many federated learning algorithms, including the canonical Federated Averaging (FedAvg), take a direct (possibly weighted) average of the client parameter updates, motivated by results in distributed optimization. In this work, we adopt a function space perspective and propose a new algorithm, FedFish, that aggregates local approximations to the functions learned by clients, using an estimate based on their Fisher information. We evaluate FedFish on realistic, large-scale cross-device benchmarks. While the performance of FedAvg can suffer as client models drift further apart, we demonstrate that FedFish is more robust to longer local training. Our evaluation across several settings in image and language benchmarks shows that FedFish outperforms FedAvg as local training epochs increase. Further, FedFish results in global networks that are more amenable to efficient personalization via local fine-tuning on the same or shifted data distributions. For instance, federated pretraining on the C4 dataset, followed by few-shot personalization on Stack Overflow, results in a 7% improvement in next-token prediction by FedFish over FedAvg.

Title: Federated Knowledge Graph Completion via Latent Embedding Sharing and Tensor Factorization. (arXiv:2311.10341v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10341
• Code URL: null
• Copy Paste: [[2311.10341]] Federated Knowledge Graph Completion via Latent Embedding Sharing and Tensor Factorization(http://arxiv.org/abs/2311.10341)
• Summary:

  Knowledge graphs (KGs), which consist of triples, are inherently incomplete and always require completion procedure to predict missing triples. In real-world scenarios, KGs are distributed across clients, complicating completion tasks due to privacy restrictions. Many frameworks have been proposed to address the issue of federated knowledge graph completion. However, the existing frameworks, including FedE, FedR, and FEKG, have certain limitations. = FedE poses a risk of information leakage, FedR's optimization efficacy diminishes when there is minimal overlap among relations, and FKGE suffers from computational costs and mode collapse issues. To address these issues, we propose a novel method, i.e., Federated Latent Embedding Sharing Tensor factorization (FLEST), which is a novel approach using federated tensor factorization for KG completion. FLEST decompose the embedding matrix and enables sharing of latent dictionary embeddings to lower privacy risks. Empirical results demonstrate FLEST's effectiveness and efficiency, offering a balanced solution between performance and privacy. FLEST expands the application of federated tensor factorization in KG completion tasks.

fair

Title: Asymptotically Fair Participation in Machine Learning Models: an Optimal Control Perspective. (arXiv:2311.10223v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10223
• Code URL: null
• Copy Paste: [[2311.10223]] Asymptotically Fair Participation in Machine Learning Models: an Optimal Control Perspective(http://arxiv.org/abs/2311.10223)
• Summary:

  The performance of state-of-the-art machine learning models often deteriorates when testing on demographics that are under-represented in the training dataset. This problem has predominately been studied in a supervised learning setting where the data distribution is static. However, real-world applications often involve distribution shifts caused by the deployed models. For instance, the performance disparity against monitory users can lead to a high customer churn rate, thus the available data provided by active users are skewed due to the lack of minority users. This feedback effect further exacerbates the disparity among different demographic groups in future steps. To address this issue, we propose asymptotically fair participation as a condition to maintain long-term model performance over all demographic groups. In this work, we aim to address the problem of achieving asymptotically fair participation via optimal control formulation. Moreover, we design a surrogate retention system based on existing literature on evolutionary population dynamics to approximate the dynamics of distribution shifts on active user counts, from which the objective of achieving asymptotically fair participation is formulated as an optimal control problem, and the control variables are considered as the model parameters. We apply an efficient implementation of Pontryagin's maximum principle to estimate the optimal control solution. To evaluate the effectiveness of the proposed method, we design a generic simulation environment that simulates the population dynamics of the feedback effect between user retention and model performance. When we deploy the resulting models to the simulation environment, the optimal control solution accounts for long-term planning and leads to superior performance compared with existing baseline methods.

Title: Causal Fairness-Guided Dataset Reweighting using Neural Networks. (arXiv:2311.10512v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10512
• Code URL: null
• Copy Paste: [[2311.10512]] Causal Fairness-Guided Dataset Reweighting using Neural Networks(http://arxiv.org/abs/2311.10512)
• Summary:

  The importance of achieving fairness in machine learning models cannot be overstated. Recent research has pointed out that fairness should be examined from a causal perspective, and several fairness notions based on the on Pearl's causal framework have been proposed. In this paper, we construct a reweighting scheme of datasets to address causal fairness. Our approach aims at mitigating bias by considering the causal relationships among variables and incorporating them into the reweighting process. The proposed method adopts two neural networks, whose structures are intentionally used to reflect the structures of a causal graph and of an interventional graph. The two neural networks can approximate the causal model of the data, and the causal model of interventions. Furthermore, reweighting guided by a discriminator is applied to achieve various fairness notions. Experiments on real-world datasets show that our method can achieve causal fairness on the data while remaining close to the original data for downstream tasks.

interpretability

explainability

watermark

diffusion

Title: MetaDreamer: Efficient Text-to-3D Creation With Disentangling Geometry and Texture. (arXiv:2311.10123v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10123
• Code URL: null
• Copy Paste: [[2311.10123]] MetaDreamer: Efficient Text-to-3D Creation With Disentangling Geometry and Texture(http://arxiv.org/abs/2311.10123)
• Summary:

  Generative models for 3D object synthesis have seen significant advancements with the incorporation of prior knowledge distilled from 2D diffusion models. Nevertheless, challenges persist in the form of multi-view geometric inconsistencies and slow generation speeds within the existing 3D synthesis frameworks. This can be attributed to two factors: firstly, the deficiency of abundant geometric a priori knowledge in optimization, and secondly, the entanglement issue between geometry and texture in conventional 3D generation methods.In response, we introduce MetaDreammer, a two-stage optimization approach that leverages rich 2D and 3D prior knowledge. In the first stage, our emphasis is on optimizing the geometric representation to ensure multi-view consistency and accuracy of 3D objects. In the second stage, we concentrate on fine-tuning the geometry and optimizing the texture, thereby achieving a more refined 3D object. Through leveraging 2D and 3D prior knowledge in two stages, respectively, we effectively mitigate the interdependence between geometry and texture. MetaDreamer establishes clear optimization objectives for each stage, resulting in significant time savings in the 3D generation process. Ultimately, MetaDreamer can generate high-quality 3D objects based on textual prompts within 20 minutes, and to the best of our knowledge, it is the most efficient text-to-3D generation method. Furthermore, we introduce image control into the process, enhancing the controllability of 3D generation. Extensive empirical evidence confirms that our method is not only highly efficient but also achieves a quality level that is at the forefront of current state-of-the-art 3D generation techniques.

Title: Enhancing Object Coherence in Layout-to-Image Synthesis. (arXiv:2311.10522v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10522
• Code URL: null
• Copy Paste: [[2311.10522]] Enhancing Object Coherence in Layout-to-Image Synthesis(http://arxiv.org/abs/2311.10522)
• Summary:

  Layout-to-image synthesis is an emerging technique in conditional image generation. It aims to generate complex scenes, where users require fine control over the layout of the objects in a scene. However, it remains challenging to control the object coherence, including semantic coherence (e.g., the cat looks at the flowers or not) and physical coherence (e.g., the hand and the racket should not be misaligned). In this paper, we propose a novel diffusion model with effective global semantic fusion (GSF) and self-similarity feature enhancement modules to guide the object coherence for this task. For semantic coherence, we argue that the image caption contains rich information for defining the semantic relationship within the objects in the images. Instead of simply employing cross-attention between captions and generated images, which addresses the highly relevant layout restriction and semantic coherence separately and thus leads to unsatisfying results shown in our experiments, we develop GSF to fuse the supervision from the layout restriction and semantic coherence requirement and exploit it to guide the image synthesis process. Moreover, to improve the physical coherence, we develop a Self-similarity Coherence Attention (SCA) module to explicitly integrate local contextual physical coherence into each pixel's generation process. Specifically, we adopt a self-similarity map to encode the coherence restrictions and employ it to extract coherent features from text embedding. Through visualization of our self-similarity map, we explore the essence of SCA, revealing that its effectiveness is not only in capturing reliable physical coherence patterns but also in enhancing complex texture generation. Extensive experiments demonstrate the superiority of our proposed method in both image generation quality and controllability.

Title: Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning. (arXiv:2311.10709v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10709
• Code URL: null
• Copy Paste: [[2311.10709]] Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning(http://arxiv.org/abs/2311.10709)
• Summary:

  We present Emu Video, a text-to-video generation model that factorizes the generation into two steps: first generating an image conditioned on the text, and then generating a video conditioned on the text and the generated image. We identify critical design decisions--adjusted noise schedules for diffusion, and multi-stage training--that enable us to directly generate high quality and high resolution videos, without requiring a deep cascade of models as in prior work. In human evaluations, our generated videos are strongly preferred in quality compared to all prior work--81% vs. Google's Imagen Video, 90% vs. Nvidia's PYOCO, and 96% vs. Meta's Make-A-Video. Our model outperforms commercial solutions such as RunwayML's Gen2 and Pika Labs. Finally, our factorizing approach naturally lends itself to animating images based on a user's text prompt, where our generations are preferred 96% over prior work.

Title: Advancements in Generative AI: A Comprehensive Review of GANs, GPT, Autoencoders, Diffusion Model, and Transformers. (arXiv:2311.10242v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10242
• Code URL: null
• Copy Paste: [[2311.10242]] Advancements in Generative AI: A Comprehensive Review of GANs, GPT, Autoencoders, Diffusion Model, and Transformers(http://arxiv.org/abs/2311.10242)
• Summary:

  The launch of ChatGPT has garnered global attention, marking a significant milestone in the field of Generative Artificial Intelligence. While Generative AI has been in effect for the past decade, the introduction of ChatGPT has ignited a new wave of research and innovation in the AI domain. This surge in interest has led to the development and release of numerous cutting-edge tools, such as Bard, Stable Diffusion, DALL-E, Make-A-Video, Runway ML, and Jukebox, among others. These tools exhibit remarkable capabilities, encompassing tasks ranging from text generation and music composition, image creation, video production, code generation, and even scientific work. They are built upon various state-of-the-art models, including Stable Diffusion, transformer models like GPT-3 (recent GPT-4), variational autoencoders, and generative adversarial networks. This advancement in Generative AI presents a wealth of exciting opportunities and, simultaneously, unprecedented challenges. Throughout this paper, we have explored these state-of-the-art models, the diverse array of tasks they can accomplish, the challenges they pose, and the promising future of Generative Artificial Intelligence.

noise learning

data-free

transformer

Title: Wildfire Smoke Detection with Cross Contrast Patch Embedding. (arXiv:2311.10116v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10116
• Code URL: null
• Copy Paste: [[2311.10116]] Wildfire Smoke Detection with Cross Contrast Patch Embedding(http://arxiv.org/abs/2311.10116)
• Summary:

  The Transformer-based deep networks have increasingly shown significant advantages over CNNs. Some existing work has applied it in the field of wildfire recognition or detection. However, we observed that the vanilla Transformer is not friendly for extracting smoke features. Because low-level information such as color, transparency and texture is very important for smoke recognition, and transformer pays more attention to the semantic relevance between middle- or high-level features, and is not sensitive to the subtle changes of low-level features along the space. To solve this problem, we propose the Cross Contrast Patch Embedding(CCPE) module based on the Swin Transformer, which uses the multi-scales spatial frequency contrast information in both vertical and horizontal directions to improve the discrimination of the network on the underlying details. The fuzzy boundary of smoke makes the positive and negative label assignment for instances in a dilemma, which is another challenge for wildfires detection. To solve this problem, a Separable Negative Sampling Mechanism(SNSM) is proposed. By using two different negative instance sampling strategies on positive images and negative images respectively, the problem of supervision signal confusion caused by label diversity in the process of network training is alleviated. This paper also releases the RealFire Test, the largest real wildfire test set so far, to evaluate the proposed method and promote future research. It contains 50,535 images from 3,649 video clips. The proposed method has been extensively tested and evaluated on RealFire Test dataset, and has a significant performance improvement compared with the baseline detection models.

Title: I&S-ViT: An Inclusive & Stable Method for Pushing the Limit of Post-Training ViTs Quantization. (arXiv:2311.10126v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10126
• Code URL: null
• Copy Paste: [[2311.10126]] I&S-ViT: An Inclusive & Stable Method for Pushing the Limit of Post-Training ViTs Quantization(http://arxiv.org/abs/2311.10126)
• Summary:

  Albeit the scalable performance of vision transformers (ViTs), the dense computational costs (training & inference) undermine their position in industrial applications. Post-training quantization (PTQ), tuning ViTs with a tiny dataset and running in a low-bit format, well addresses the cost issue but unluckily bears more performance drops in lower-bit cases. In this paper, we introduce I&S-ViT, a novel method that regulates the PTQ of ViTs in an inclusive and stable fashion. I&S-ViT first identifies two issues in the PTQ of ViTs: (1) Quantization inefficiency in the prevalent log2 quantizer for post-Softmax activations; (2) Rugged and magnified loss landscape in coarse-grained quantization granularity for post-LayerNorm activations. Then, I&S-ViT addresses these issues by introducing: (1) A novel shift-uniform-log2 quantizer (SULQ) that incorporates a shift mechanism followed by uniform quantization to achieve both an inclusive domain representation and accurate distribution approximation; (2) A three-stage smooth optimization strategy (SOS) that amalgamates the strengths of channel-wise and layer-wise quantization to enable stable learning. Comprehensive evaluations across diverse vision tasks validate I&S-ViT' superiority over existing PTQ of ViTs methods, particularly in low-bit scenarios. For instance, I&S-ViT elevates the performance of 3-bit ViT-B by an impressive 50.68%.

Title: Learning transformer-based heterogeneously salient graph representation for multimodal fusion classification of hyperspectral image and LiDAR data. (arXiv:2311.10320v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10320
• Code URL: null
• Copy Paste: [[2311.10320]] Learning transformer-based heterogeneously salient graph representation for multimodal fusion classification of hyperspectral image and LiDAR data(http://arxiv.org/abs/2311.10320)
• Summary:

  Data collected by different modalities can provide a wealth of complementary information, such as hyperspectral image (HSI) to offer rich spectral-spatial properties, synthetic aperture radar (SAR) to provide structural information about the Earth's surface, and light detection and ranging (LiDAR) to cover altitude information about ground elevation. Therefore, a natural idea is to combine multimodal images for refined and accurate land-cover interpretation. Although many efforts have been attempted to achieve multi-source remote sensing image classification, there are still three issues as follows: 1) indiscriminate feature representation without sufficiently considering modal heterogeneity, 2) abundant features and complex computations associated with modeling long-range dependencies, and 3) overfitting phenomenon caused by sparsely labeled samples. To overcome the above barriers, a transformer-based heterogeneously salient graph representation (THSGR) approach is proposed in this paper. First, a multimodal heterogeneous graph encoder is presented to encode distinctively non-Euclidean structural features from heterogeneous data. Then, a self-attention-free multi-convolutional modulator is designed for effective and efficient long-term dependency modeling. Finally, a mean forward is put forward in order to avoid overfitting. Based on the above structures, the proposed model is able to break through modal gaps to obtain differentiated graph representation with competitive time cost, even for a small fraction of training samples. Experiments and analyses on three benchmark datasets with various state-of-the-art (SOTA) methods show the performance of the proposed approach.

Title: 3D-TexSeg: Unsupervised Segmentation of 3D Texture using Mutual Transformer Learning. (arXiv:2311.10651v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10651
• Code URL: null
• Copy Paste: [[2311.10651]] 3D-TexSeg: Unsupervised Segmentation of 3D Texture using Mutual Transformer Learning(http://arxiv.org/abs/2311.10651)
• Summary:

  Analysis of the 3D Texture is indispensable for various tasks, such as retrieval, segmentation, classification, and inspection of sculptures, knitted fabrics, and biological tissues. A 3D texture is a locally repeated surface variation independent of the surface's overall shape and can be determined using the local neighborhood and its characteristics. Existing techniques typically employ computer vision techniques that analyze a 3D mesh globally, derive features, and then utilize the obtained features for retrieval or classification. Several traditional and learning-based methods exist in the literature, however, only a few are on 3D texture, and nothing yet, to the best of our knowledge, on the unsupervised schemes. This paper presents an original framework for the unsupervised segmentation of the 3D texture on the mesh manifold. We approach this problem as binary surface segmentation, partitioning the mesh surface into textured and non-textured regions without prior annotation. We devise a mutual transformer-based system comprising a label generator and a cleaner. The two models take geometric image representations of the surface mesh facets and label them as texture or non-texture across an iterative mutual learning scheme. Extensive experiments on three publicly available datasets with diverse texture patterns demonstrate that the proposed framework outperforms standard and SOTA unsupervised techniques and competes reasonably with supervised methods.

Title: Diagnosing and Debiasing Corpus-Based Political Bias and Insults in GPT2. (arXiv:2311.10266v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10266
• Code URL: null
• Copy Paste: [[2311.10266]] Diagnosing and Debiasing Corpus-Based Political Bias and Insults in GPT2(http://arxiv.org/abs/2311.10266)
• Summary:

  The training of large language models (LLMs) on extensive, unfiltered corpora sourced from the internet is a common and advantageous practice. Consequently, LLMs have learned and inadvertently reproduced various types of biases, including violent, offensive, and toxic language. However, recent research shows that generative pretrained transformer (GPT) language models can recognize their own biases and detect toxicity in generated content, a process referred to as self-diagnosis. In response, researchers have developed a decoding algorithm that allows LLMs to self-debias, or reduce their likelihood of generating harmful text. This study investigates the efficacy of the diagnosing-debiasing approach in mitigating two additional types of biases: insults and political bias. These biases are often used interchangeably in discourse, despite exhibiting potentially dissimilar semantic and syntactic properties. We aim to contribute to the ongoing effort of investigating the ethical and social implications of human-AI interaction.

Title: Bias A-head? Analyzing Bias in Transformer-Based Language Model Attention Heads. (arXiv:2311.10395v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10395
• Code URL: null
• Copy Paste: [[2311.10395]] Bias A-head? Analyzing Bias in Transformer-Based Language Model Attention Heads(http://arxiv.org/abs/2311.10395)
• Summary:

  Transformer-based pretrained large language models (PLM) such as BERT and GPT have achieved remarkable success in NLP tasks. However, PLMs are prone to encoding stereotypical biases. Although a burgeoning literature has emerged on stereotypical bias mitigation in PLMs, such as work on debiasing gender and racial stereotyping, how such biases manifest and behave internally within PLMs remains largely unknown. Understanding the internal stereotyping mechanisms may allow better assessment of model fairness and guide the development of effective mitigation strategies. In this work, we focus on attention heads, a major component of the Transformer architecture, and propose a bias analysis framework to explore and identify a small set of biased heads that are found to contribute to a PLM's stereotypical bias. We conduct extensive experiments to validate the existence of these biased heads and to better understand how they behave. We investigate gender and racial bias in the English language in two types of Transformer-based PLMs: the encoder-based BERT model and the decoder-based autoregressive GPT model. Overall, the results shed light on understanding the bias behavior in pretrained language models.

Title: Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers. (arXiv:2311.10642v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10642
• Code URL: null
• Copy Paste: [[2311.10642]] Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers(http://arxiv.org/abs/2311.10642)
• Summary:

  This work presents an analysis of the effectiveness of using standard shallow feed-forward networks to mimic the behavior of the attention mechanism in the original Transformer model, a state-of-the-art architecture for sequence-to-sequence tasks. We substitute key elements of the attention mechanism in the Transformer with simple feed-forward networks, trained using the original components via knowledge distillation. Our experiments, conducted on the IWSLT2017 dataset, reveal the capacity of these "attentionless Transformers" to rival the performance of the original architecture. Through rigorous ablation studies, and experimenting with various replacement network types and sizes, we offer insights that support the viability of our approach. This not only sheds light on the adaptability of shallow feed-forward networks in emulating attention mechanisms but also underscores their potential to streamline complex architectures for sequence-to-sequence tasks.

Title: Accommodating Missing Modalities in Time-Continuous Multimodal Emotion Recognition. (arXiv:2311.10119v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10119
• Code URL: null
• Copy Paste: [[2311.10119]] Accommodating Missing Modalities in Time-Continuous Multimodal Emotion Recognition(http://arxiv.org/abs/2311.10119)
• Summary:

  Decades of research indicate that emotion recognition is more effective when drawing information from multiple modalities. But what if some modalities are sometimes missing? To address this problem, we propose a novel Transformer-based architecture for recognizing valence and arousal in a time-continuous manner even with missing input modalities. We use a coupling of cross-attention and self-attention mechanisms to emphasize relationships between modalities during time and enhance the learning process on weak salient inputs. Experimental results on the Ulm-TSST dataset show that our model exhibits an improvement of the concordance correlation coefficient evaluation of 37% when predicting arousal values and 30% when predicting valence values, compared to a late-fusion baseline approach.

Title: Improving Unimodal Inference with Multimodal Transformers. (arXiv:2311.10170v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10170
• Code URL: null
• Copy Paste: [[2311.10170]] Improving Unimodal Inference with Multimodal Transformers(http://arxiv.org/abs/2311.10170)
• Summary:

  This paper proposes an approach for improving performance of unimodal models with multimodal training. Our approach involves a multi-branch architecture that incorporates unimodal models with a multimodal transformer-based branch. By co-training these branches, the stronger multimodal branch can transfer its knowledge to the weaker unimodal branches through a multi-task objective, thereby improving the performance of the resulting unimodal models. We evaluate our approach on tasks of dynamic hand gesture recognition based on RGB and Depth, audiovisual emotion recognition based on speech and facial video, and audio-video-text based sentiment analysis. Our approach outperforms the conventionally trained unimodal counterparts. Interestingly, we also observe that optimization of the unimodal branches improves the multimodal branch, compared to a similar multimodal model trained from scratch.

generative

Title: SelfEval: Leveraging the discriminative nature of generative models for evaluation. (arXiv:2311.10708v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10708
• Code URL: null
• Copy Paste: [[2311.10708]] SelfEval: Leveraging the discriminative nature of generative models for evaluation(http://arxiv.org/abs/2311.10708)
• Summary:

  In this work, we show that text-to-image generative models can be 'inverted' to assess their own text-image understanding capabilities in a completely automated manner.

  Our method, called SelfEval, uses the generative model to compute the likelihood of real images given text prompts, making the generative model directly applicable to discriminative tasks.

  Using SelfEval, we repurpose standard datasets created for evaluating multimodal text-image discriminative models to evaluate generative models in a fine-grained manner: assessing their performance on attribute binding, color recognition, counting, shape recognition, spatial understanding.

  To the best of our knowledge SelfEval is the first automated metric to show a high degree of agreement for measuring text-faithfulness with the gold-standard human evaluations across multiple models and benchmarks.

  Moreover, SelfEval enables us to evaluate generative models on challenging tasks such as Winoground image-score where they demonstrate competitive performance to discriminative models.

  We also show severe drawbacks of standard automated metrics such as CLIP-score to measure text faithfulness on benchmarks such as DrawBench, and how SelfEval sidesteps these issues.

  We hope SelfEval enables easy and reliable automated evaluation for diffusion models.

Title: Supervised structure learning. (arXiv:2311.10300v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10300
• Code URL: null
• Copy Paste: [[2311.10300]] Supervised structure learning(http://arxiv.org/abs/2311.10300)
• Summary:

  This paper concerns structure learning or discovery of discrete generative models. It focuses on Bayesian model selection and the assimilation of training data or content, with a special emphasis on the order in which data are ingested. A key move - in the ensuing schemes - is to place priors on the selection of models, based upon expected free energy. In this setting, expected free energy reduces to a constrained mutual information, where the constraints inherit from priors over outcomes (i.e., preferred outcomes). The resulting scheme is first used to perform image classification on the MNIST dataset to illustrate the basic idea, and then tested on a more challenging problem of discovering models with dynamics, using a simple sprite-based visual disentanglement paradigm and the Tower of Hanoi (cf., blocks world) problem. In these examples, generative models are constructed autodidactically to recover (i.e., disentangle) the factorial structure of latent states - and their characteristic paths or dynamics.

Title: Concept-free Causal Disentanglement with Variational Graph Auto-Encoder. (arXiv:2311.10638v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.10638
• Code URL: null
• Copy Paste: [[2311.10638]] Concept-free Causal Disentanglement with Variational Graph Auto-Encoder(http://arxiv.org/abs/2311.10638)
• Summary:

  In disentangled representation learning, the goal is to achieve a compact representation that consists of all interpretable generative factors in the observational data. Learning disentangled representations for graphs becomes increasingly important as graph data rapidly grows. Existing approaches often rely on Variational Auto-Encoder (VAE) or its causal structure learning-based refinement, which suffer from sub-optimality in VAEs due to the independence factor assumption and unavailability of concept labels, respectively. In this paper, we propose an unsupervised solution, dubbed concept-free causal disentanglement, built on a theoretically provable tight upper bound approximating the optimal factor. This results in an SCM-like causal structure modeling that directly learns concept structures from data. Based on this idea, we propose Concept-free Causal VGAE (CCVGAE) by incorporating a novel causal disentanglement layer into Variational Graph Auto-Encoder. Furthermore, we prove concept consistency under our concept-free causal disentanglement framework, hence employing it to enhance the meta-learning framework, called concept-free causal Meta-Graph (CC-Meta-Graph). We conduct extensive experiments to demonstrate the superiority of the proposed models: CCVGAE and CC-Meta-Graph, reaching up to $29\%$ and $11\%$ absolute improvements over baselines in terms of AUC, respectively.

large language model

Title: UnifiedVisionGPT: Streamlining Vision-Oriented AI through Generalized Multimodal Framework. (arXiv:2311.10125v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10125
• Code URL: https://github.com/lhbuilder/sa-segment-anything
• Copy Paste: [[2311.10125]] UnifiedVisionGPT: Streamlining Vision-Oriented AI through Generalized Multimodal Framework(http://arxiv.org/abs/2311.10125)
• Summary:

  In the current landscape of artificial intelligence, foundation models serve as the bedrock for advancements in both language and vision domains. OpenAI GPT-4 has emerged as the pinnacle in large language models (LLMs), while the computer vision (CV) domain boasts a plethora of state-of-the-art (SOTA) models such as Meta's SAM and DINO, and YOLOS. However, the financial and computational burdens of training new models from scratch remain a significant barrier to progress. In response to this challenge, we introduce UnifiedVisionGPT, a novel framework designed to consolidate and automate the integration of SOTA vision models, thereby facilitating the development of vision-oriented AI. UnifiedVisionGPT distinguishes itself through four key features: (1) provides a versatile multimodal framework adaptable to a wide range of applications, building upon the strengths of multimodal foundation models; (2) seamlessly integrates various SOTA vision models to create a comprehensive multimodal platform, capitalizing on the best components of each model; (3) prioritizes vision-oriented AI, ensuring a more rapid progression in the CV domain compared to the current trajectory of LLMs; and (4) introduces automation in the selection of SOTA vision models, generating optimal results based on diverse multimodal inputs such as text prompts and images. This paper outlines the architecture and capabilities of UnifiedVisionGPT, demonstrating its potential to revolutionize the field of computer vision through enhanced efficiency, versatility, generalization, and performance. Our implementation, along with the unified multimodal framework and comprehensive dataset, is made publicly available at https://github.com/LHBuilder/SA-Segment-Anything.

Title: Predictive Minds: LLMs As Atypical Active Inference Agents. (arXiv:2311.10215v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10215
• Code URL: null
• Copy Paste: [[2311.10215]] Predictive Minds: LLMs As Atypical Active Inference Agents(http://arxiv.org/abs/2311.10215)
• Summary:

  Large language models (LLMs) like GPT are often conceptualized as passive predictors, simulators, or even stochastic parrots. We instead conceptualize LLMs by drawing on the theory of active inference originating in cognitive science and neuroscience. We examine similarities and differences between traditional active inference systems and LLMs, leading to the conclusion that, currently, LLMs lack a tight feedback loop between acting in the world and perceiving the impacts of their actions, but otherwise fit in the active inference paradigm. We list reasons why this loop may soon be closed, and possible consequences of this including enhanced model self-awareness and the drive to minimize prediction error by changing the world.

Title: Exploring the Relationship between In-Context Learning and Instruction Tuning. (arXiv:2311.10367v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10367
• Code URL: null
• Copy Paste: [[2311.10367]] Exploring the Relationship between In-Context Learning and Instruction Tuning(http://arxiv.org/abs/2311.10367)
• Summary:

  In-Context Learning (ICL) and Instruction Tuning (IT) are two primary paradigms of adopting Large Language Models (LLMs) to downstream applications. However, they are significantly different. In ICL, a set of demonstrations are provided at inference time but the LLM's parameters are not updated. In IT, a set of demonstrations are used to tune LLM's parameters in training time but no demonstrations are used at inference time. Although a growing body of literature has explored ICL and IT, studies on these topics have largely been conducted in isolation, leading to a disconnect between these two paradigms. In this work, we explore the relationship between ICL and IT by examining how the hidden states of LLMs change in these two paradigms. Through carefully designed experiments conducted with LLaMA-2 (7B and 13B), we find that ICL is implicit IT. In other words, ICL changes an LLM's hidden states as if the demonstrations were used to instructionally tune the model. Furthermore, the convergence between ICL and IT is largely contingent upon several factors related to the provided demonstrations. Overall, this work offers a unique perspective to explore the connection between ICL and IT and sheds light on understanding the behaviors of LLM.

Title: MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning. (arXiv:2311.10537v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10537
• Code URL: https://github.com/gersteinlab/medagents
• Copy Paste: [[2311.10537]] MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning(http://arxiv.org/abs/2311.10537)
• Summary:

  Large Language Models (LLMs), despite their remarkable progress across various general domains, encounter significant barriers in medicine and healthcare. This field faces unique challenges such as domain-specific terminologies and the reasoning over specialized knowledge. To address these obstinate issues, we propose a novel Multi-disciplinary Collaboration (MC) framework for the medical domain that leverages role-playing LLM-based agents who participate in a collaborative multi-round discussion, thereby enhancing LLM proficiency and reasoning capabilities. This training-free and interpretable framework encompasses five critical steps: gathering domain experts, proposing individual analyses, summarising these analyses into a report, iterating over discussions until a consensus is reached, and ultimately making a decision. Our work particularly focuses on the zero-shot scenario, our results on nine data sets (MedQA, MedMCQA, PubMedQA, and six subtasks from MMLU) establish that our proposed MC framework excels at mining and harnessing the medical expertise in LLMs, as well as extending its reasoning abilities. Based on these outcomes, we further conduct a human evaluation to pinpoint and categorize common errors within our method, as well as ablation studies aimed at understanding the impact of various factors on overall performance. Our code can be found at \url{https://github.com/gersteinlab/MedAgents}.

Title: A Self-enhancement Approach for Domain-specific Chatbot Training via Knowledge Mining and Digest. (arXiv:2311.10614v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10614
• Code URL: null
• Copy Paste: [[2311.10614]] A Self-enhancement Approach for Domain-specific Chatbot Training via Knowledge Mining and Digest(http://arxiv.org/abs/2311.10614)
• Summary:

  Large Language Models (LLMs), despite their great power in language generation, often encounter challenges when dealing with intricate and knowledge-demanding queries in specific domains. This paper introduces a novel approach to enhance LLMs by effectively extracting the relevant knowledge from domain-specific textual sources, and the adaptive training of a chatbot with domain-specific inquiries. Our two-step approach starts from training a knowledge miner, namely LLMiner, which autonomously extracts Question-Answer pairs from relevant documents through a chain-of-thought reasoning process. Subsequently, we blend the mined QA pairs with a conversational dataset to fine-tune the LLM as a chatbot, thereby enriching its domain-specific expertise and conversational capabilities. We also developed a new evaluation benchmark which comprises four domain-specific text corpora and associated human-crafted QA pairs for testing. Our model shows remarkable performance improvement over generally aligned LLM and surpasses domain-adapted models directly fine-tuned on domain corpus. In particular, LLMiner achieves this with minimal human intervention, requiring only 600 seed instances, thereby providing a pathway towards self-improvement of LLMs through model-synthesized training data.

Title: PEFT-MedAware: Large Language Model for Medical Awareness. (arXiv:2311.10697v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10697
• Code URL: null
• Copy Paste: [[2311.10697]] PEFT-MedAware: Large Language Model for Medical Awareness(http://arxiv.org/abs/2311.10697)
• Summary:

  Chat models are capable of answering a wide range of questions, however, the accuracy of their responses is highly uncertain. In this research, we propose a specialized PEFT-MedAware model where we utilize parameter-efficient fine-tuning (PEFT) to enhance the Falcon-1b large language model on specialized MedQuAD data consisting of 16,407 medical QA pairs, leveraging only 0.44% of its trainable parameters to enhance computational efficiency. The paper adopts data preprocessing and PEFT to optimize model performance, complemented by a BitsAndBytesConfig for efficient transformer training. The resulting model was capable of outperforming other LLMs in medical question-answering tasks in specific domains with greater accuracy utilizing limited computational resources making it suitable for deployment in resource-constrained environments. We propose further improvements through expanded datasets, larger models, and feedback mechanisms for sustained medical relevancy. Our work highlights the efficiency gains and specialized capabilities of PEFT in medical AI, outpacing standard models in precision without extensive resource demands. The proposed model and data are released for research purposes only.

Title: Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2. (arXiv:2311.10702v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.10702
• Code URL: null
• Copy Paste: [[2311.10702]] Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2(http://arxiv.org/abs/2311.10702)
• Summary:

  Since the release of T\"ULU [Wang et al., 2023b], open resources for instruction tuning have developed quickly, from better base models to new finetuning techniques. We test and incorporate a number of these advances into T\"ULU, resulting in T\"ULU 2, a suite of improved T\"ULU models for advancing the understanding and best practices of adapting pretrained language models to downstream tasks and user preferences. Concretely, we release: (1) T\"ULU-V2-mix, an improved collection of high-quality instruction datasets; (2) T\"ULU 2, LLAMA-2 models finetuned on the V2 mixture; (3) T\"ULU 2+DPO, T\"ULU 2 models trained with direct preference optimization (DPO), including the largest DPO-trained model to date (T\"ULU 2+DPO 70B); (4) CODE T\"ULU 2, CODE LLAMA models finetuned on our V2 mix that outperform CODE LLAMA and its instruction-tuned variant, CODE LLAMA-Instruct. Our evaluation from multiple perspectives shows that the T\"ULU 2 suite achieves state-of-the-art performance among open models and matches or exceeds the performance of GPT-3.5-turbo-0301 on several benchmarks. We release all the checkpoints, data, training and evaluation code to facilitate future open efforts on adapting large language models.

segmentation

Title: Smart Traffic Management of Vehicles using Faster R-CNN based Deep Learning Method. (arXiv:2311.10099v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10099
• Code URL: null
• Copy Paste: [[2311.10099]] Smart Traffic Management of Vehicles using Faster R-CNN based Deep Learning Method(http://arxiv.org/abs/2311.10099)
• Summary:

  With constant growth of civilization and modernization of cities all across the world since past few centuries smart traffic management of vehicles is one of the most sorted after problem by research community. It is a challenging problem in computer vision and artificial intelligence domain. Smart traffic management basically involves segmentation of vehicles, estimation of traffic density and tracking of vehicles. The vehicle segmentation from traffic videos helps realization of niche applications such as monitoring of speed and estimation of traffic. When occlusions, background with clutters and traffic with density variations are present, this problem becomes more intractable in nature. Keeping this motivation in this research work, we investigate Faster R-CNN based deep learning method towards segmentation of vehicles. This problem is addressed in four steps viz minimization with adaptive background model, Faster R-CNN based subnet operation, Faster R-CNN initial refinement and result optimization with extended topological active nets. The computational framework uses ideas of adaptive background modeling. It also addresses shadow and illumination related issues. Higher segmentation accuracy is achieved through topological active net deformable models. The topological and extended topological active nets help to achieve stated deformations. Mesh deformation is achieved with minimization of energy. The segmentation accuracy is improved with modified version of extended topological active net. The experimental results demonstrate superiority of this computational framework

Title: Slide-SAM: Medical SAM Meets Sliding Window. (arXiv:2311.10121v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10121
• Code URL: null
• Copy Paste: [[2311.10121]] Slide-SAM: Medical SAM Meets Sliding Window(http://arxiv.org/abs/2311.10121)
• Summary:

  Segment Anything Model (SAM) achieves remarkable results in 2D image segmentation of natural images. However, the huge gap between medical images and natural images prevents it directly applied to medical image segmentation tasks. Especially in 3D medical image, SAM cannot learn the contextual relationship between slices, which limites application in real scenarios. In addition, recent research shows that applying 2D SAM to 3D images requires prompting the entire volume, which is time and label comsuming. In order to solve the above problems, we introduced Slide-SAM which extended SAM to 3D medical images. Specifically, you only need to use a single slice prompt to segement the entire volume, which greatly reduces the prompt workload for professionals. Secondly, unlike traditional 3D medical image segmentation, we are free from the influence of computing resources and can still use high resolution (H$ \times $W = 1024$ \times $1024) for training in 3D images to achieve optimal learning for small targets. This is to combine the entire 3D volume is beyond the reach of training. Finally, we collected a large number of 3D images from large-scale 3D public and private datasets, and extended SAM to 3D medical image segmentation involving bounding box and point prompts. Finally, we perform a comprehensive evaluation and analysis investigating the performance of Slide-SAM in medical image segmentation of different modalities, anatomy, and organs. We have verified Slide-SAM's segmentation capabilities on multiple datasets, achieving the most advanced 3D segmentation performance while maintaining the minimum prompt. Code will be open source soon.

Title: Segment Anything in Defect Detection. (arXiv:2311.10245v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10245
• Code URL: null
• Copy Paste: [[2311.10245]] Segment Anything in Defect Detection(http://arxiv.org/abs/2311.10245)
• Summary:

  Defect detection plays a crucial role in infrared non-destructive testing systems, offering non-contact, safe, and efficient inspection capabilities. However, challenges such as low resolution, high noise, and uneven heating in infrared thermal images hinder comprehensive and accurate defect detection. In this study, we propose DefectSAM, a novel approach for segmenting defects on highly noisy thermal images based on the widely adopted model, Segment Anything (SAM)\cite{kirillov2023segany}. Harnessing the power of a meticulously curated dataset generated through labor-intensive lab experiments and valuable prompts from experienced experts, DefectSAM surpasses existing state-of-the-art segmentation algorithms and achieves significant improvements in defect detection rates. Notably, DefectSAM excels in detecting weaker and smaller defects on complex and irregular surfaces, reducing the occurrence of missed detections and providing more accurate defect size estimations. Experimental studies conducted on various materials have validated the effectiveness of our solutions in defect detection, which hold significant potential to expedite the evolution of defect detection tools, enabling enhanced inspection capabilities and accuracy in defect identification.

Title: SSASS: Semi-Supervised Approach for Stenosis Segmentation. (arXiv:2311.10281v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10281
• Code URL: null
• Copy Paste: [[2311.10281]] SSASS: Semi-Supervised Approach for Stenosis Segmentation(http://arxiv.org/abs/2311.10281)
• Summary:

  Coronary artery stenosis is a critical health risk, and its precise identification in Coronary Angiography (CAG) can significantly aid medical practitioners in accurately evaluating the severity of a patient's condition. The complexity of coronary artery structures combined with the inherent noise in X-ray images poses a considerable challenge to this task. To tackle these obstacles, we introduce a semi-supervised approach for cardiovascular stenosis segmentation. Our strategy begins with data augmentation, specifically tailored to replicate the structural characteristics of coronary arteries. We then apply a pseudo-label-based semi-supervised learning technique that leverages the data generated through our augmentation process. Impressively, our approach demonstrated an exceptional performance in the Automatic Region-based Coronary Artery Disease diagnostics using x-ray angiography imagEs (ARCADE) Stenosis Detection Algorithm challenge by utilizing a single model instead of relying on an ensemble of multiple models. This success emphasizes our method's capability and efficiency in providing an automated solution for accurately assessing stenosis severity from medical imaging data.

Title: Shifting to Machine Supervision: Annotation-Efficient Semi and Self-Supervised Learning for Automatic Medical Image Segmentation and Classification. (arXiv:2311.10319v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10319
• Code URL: null
• Copy Paste: [[2311.10319]] Shifting to Machine Supervision: Annotation-Efficient Semi and Self-Supervised Learning for Automatic Medical Image Segmentation and Classification(http://arxiv.org/abs/2311.10319)
• Summary:

  Advancements in clinical treatment and research are limited by supervised learning techniques that rely on large amounts of annotated data, an expensive task requiring many hours of clinical specialists' time. In this paper, we propose using self-supervised and semi-supervised learning. These techniques perform an auxiliary task that is label-free, scaling up machine-supervision is easier compared with fully-supervised techniques. This paper proposes S4MI (Self-Supervision and Semi-Supervision for Medical Imaging), our pipeline to leverage advances in self and semi-supervision learning. We benchmark them on three medical imaging datasets to analyze their efficacy for classification and segmentation. This advancement in self-supervised learning with 10% annotation performed better than 100% annotation for the classification of most datasets. The semi-supervised approach yielded favorable outcomes for segmentation, outperforming the fully-supervised approach by using 50% fewer labels in all three datasets.

Title: MSE-Nets: Multi-annotated Semi-supervised Ensemble Networks for Improving Segmentation of Medical Image with Ambiguous Boundaries. (arXiv:2311.10380v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10380
• Code URL: null
• Copy Paste: [[2311.10380]] MSE-Nets: Multi-annotated Semi-supervised Ensemble Networks for Improving Segmentation of Medical Image with Ambiguous Boundaries(http://arxiv.org/abs/2311.10380)
• Summary:

  Medical image segmentation annotations exhibit variations among experts due to the ambiguous boundaries of segmented objects and backgrounds in medical images. Although using multiple annotations for each image in the fully-supervised has been extensively studied for training deep models, obtaining a large amount of multi-annotated data is challenging due to the substantial time and manpower costs required for segmentation annotations, resulting in most images lacking any annotations. To address this, we propose Multi-annotated Semi-supervised Ensemble Networks (MSE-Nets) for learning segmentation from limited multi-annotated and abundant unannotated data. Specifically, we introduce the Network Pairwise Consistency Enhancement (NPCE) module and Multi-Network Pseudo Supervised (MNPS) module to enhance MSE-Nets for the segmentation task by considering two major factors: (1) to optimize the utilization of all accessible multi-annotated data, the NPCE separates (dis)agreement annotations of multi-annotated data at the pixel level and handles agreement and disagreement annotations in different ways, (2) to mitigate the introduction of imprecise pseudo-labels, the MNPS extends the training data by leveraging consistent pseudo-labels from unannotated data. Finally, we improve confidence calibration by averaging the predictions of base networks. Experiments on the ISIC dataset show that we reduced the demand for multi-annotated data by 97.75\% and narrowed the gap with the best fully-supervised baseline to just a Jaccard index of 4\%. Furthermore, compared to other semi-supervised methods that rely only on a single annotation or a combined fusion approach, the comprehensive experimental results on ISIC and RIGA datasets demonstrate the superior performance of our proposed method in medical image segmentation with ambiguous boundaries.

Title: A Framework of Landsat-8 Band Selection based on UMDA for Deforestation Detection. (arXiv:2311.10513v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10513
• Code URL: null
• Copy Paste: [[2311.10513]] A Framework of Landsat-8 Band Selection based on UMDA for Deforestation Detection(http://arxiv.org/abs/2311.10513)
• Summary:

  The conservation of tropical forests is a current subject of social and ecological relevance due to their crucial role in the global ecosystem. Unfortunately, millions of hectares are deforested and degraded each year. Therefore, government or private initiatives are needed for monitoring tropical forests. In this sense, this work proposes a novel framework, which uses of distribution estimation algorithm (UMDA) to select spectral bands from Landsat-8 that yield a better representation of deforestation areas to guide a semantic segmentation architecture called DeepLabv3+. In performed experiments, it was possible to find several compositions that reach balanced accuracy superior to 90% in segment classification tasks. Furthermore, the best composition (651) found by UMDA algorithm fed the DeepLabv3+ architecture and surpassed in efficiency and effectiveness all compositions compared in this work.

Title: Segment Anything Model with Uncertainty Rectification for Auto-Prompting Medical Image Segmentation. (arXiv:2311.10529v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10529
• Code URL: null
• Copy Paste: [[2311.10529]] Segment Anything Model with Uncertainty Rectification for Auto-Prompting Medical Image Segmentation(http://arxiv.org/abs/2311.10529)
• Summary:

  The introduction of the Segment Anything Model (SAM) has marked a significant advancement in prompt-driven image segmentation. However, SAM's application to medical image segmentation requires manual prompting of target structures to obtain acceptable performance, which is still labor-intensive. Despite attempts of auto-prompting to turn SAM into a fully automatic manner, it still exhibits subpar performance and lacks of reliability in the field of medical imaging. In this paper, we propose UR-SAM, an uncertainty rectified SAM framework to enhance the robustness and reliability for auto-prompting medical image segmentation. Our method incorporates a prompt augmentation module to estimate the distribution of predictions and generate uncertainty maps, and an uncertainty-based rectification module to further enhance the performance of SAM. Extensive experiments on two public 3D medical datasets covering the segmentation of 35 organs demonstrate that without supplementary training or fine-tuning, our method further improves the segmentation performance with up to 10.7 % and 13.8 % in dice similarity coefficient, demonstrating efficiency and broad capabilities for medical image segmentation without manual prompting.

Title: Self-trained Panoptic Segmentation. (arXiv:2311.10648v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10648
• Code URL: null
• Copy Paste: [[2311.10648]] Self-trained Panoptic Segmentation(http://arxiv.org/abs/2311.10648)
• Summary:

  Panoptic segmentation is an important computer vision task which combines semantic and instance segmentation. It plays a crucial role in domains of medical image analysis, self-driving vehicles, and robotics by providing a comprehensive understanding of visual environments. Traditionally, deep learning panoptic segmentation models have relied on dense and accurately annotated training data, which is expensive and time consuming to obtain. Recent advancements in self-supervised learning approaches have shown great potential in leveraging synthetic and unlabelled data to generate pseudo-labels using self-training to improve the performance of instance and semantic segmentation models. The three available methods for self-supervised panoptic segmentation use proposal-based transformer architectures which are computationally expensive, complicated and engineered for specific tasks. The aim of this work is to develop a framework to perform embedding-based self-supervised panoptic segmentation using self-training in a synthetic-to-real domain adaptation problem setting.

Title: Versatile Medical Image Segmentation Learned from Multi-Source Datasets via Model Self-Disambiguation. (arXiv:2311.10696v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.10696
• Code URL: null
• Copy Paste: [[2311.10696]] Versatile Medical Image Segmentation Learned from Multi-Source Datasets via Model Self-Disambiguation(http://arxiv.org/abs/2311.10696)
• Summary:

  A versatile medical image segmentation model applicable to imaging data collected with diverse equipment and protocols can facilitate model deployment and maintenance. However, building such a model typically requires a large, diverse, and fully annotated dataset, which is rarely available due to the labor-intensive and costly data curation. In this study, we develop a cost-efficient method by harnessing readily available data with partially or even sparsely annotated segmentation labels. We devise strategies for model self-disambiguation, prior knowledge incorporation, and imbalance mitigation to address challenges associated with inconsistently labeled data from various sources, including label ambiguity and imbalances across modalities, datasets, and segmentation labels. Experimental results on a multi-modal dataset compiled from eight different sources for abdominal organ segmentation have demonstrated our method's effectiveness and superior performance over alternative state-of-the-art methods, highlighting its potential for optimizing the use of existing annotated data and reducing the annotation efforts for new data to further enhance model capability.