secure

Title: IoT Security: On-Chip Secure Deletion Scheme using ECC Modulation in IoT Appliances. (arXiv:2308.05225v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.05225
• Code URL: null
• Copy Paste: [[2308.05225]] IoT Security: On-Chip Secure Deletion Scheme using ECC Modulation in IoT Appliances(http://arxiv.org/abs/2308.05225)
• Summary:

  NAND flash memory-based IoT devices inherently suffer from data retention issues. In IoT security, these retention issues are significant and require a robust solution for secure deletion. Secure deletion methods can be categorized into off-chip and on-chip schemes. Off-chip secure deletion schemes, based on block-level erasure operations, are unable to perform real-time trim operations. Consequently, they are vulnerable to hacking threats. On the other hand, on-chip secure deletion schemes enable real-time trim operations by performing deletion on a page-by-page basis. However, the on-chip scheme introduces a challenge of program disturbance for neighboring page data. The proposed on-chip deletion scheme tackles this problem by utilizing ECC code modulation through a partial program operation. This approach significantly reduces the program disturbance issue associated with neighboring page data. Moreover, the proposed code modulation secure deletion scheme allows for real-time verification of the deletion of original data.

security

Title: Data-Driven Intelligence can Revolutionize Today's Cybersecurity World: A Position Paper. (arXiv:2308.05126v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.05126
• Code URL: null
• Copy Paste: [[2308.05126]] Data-Driven Intelligence can Revolutionize Today's Cybersecurity World: A Position Paper(http://arxiv.org/abs/2308.05126)
• Summary:

  As cyber threats evolve and grow progressively more sophisticated, cyber security is becoming a more significant concern in today's digital era. Traditional security measures tend to be insufficient to defend against these persistent and dynamic threats because they are mainly intuitional. One of the most promising ways to handle this ongoing problem is utilizing the potential of data-driven intelligence, by leveraging AI and machine learning techniques. It can improve operational efficiency and saves response times by automating repetitive operations, enabling real-time threat detection, and facilitating incident response. In addition, it augments human expertise with insightful information, predictive analytics, and enhanced decision-making, enabling them to better understand and address evolving problems. Thus, data-driven intelligence could significantly improve real-world cybersecurity solutions in a wide range of application areas like critical infrastructure, smart cities, digital twin, industrial control systems and so on. In this position paper, we argue that data-driven intelligence can revolutionize the realm of cybersecurity, offering not only large-scale task automation but also assist human experts for better situation awareness and decision-making in real-world scenarios.

Title: DCM: A Developers Certification Model for Mobile Ecosystems. (arXiv:2308.05278v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.05278
• Code URL: null
• Copy Paste: [[2308.05278]] DCM: A Developers Certification Model for Mobile Ecosystems(http://arxiv.org/abs/2308.05278)
• Summary:

  This article introduces a distributed model of trust for app developers in Android and iOS mobile ecosystems. The model aims to allow the co-existence of multiple app stores and distribution channels while retaining a high level of safety for mobile device users and minimum changes to current mobile operating systems. The Developers Certification Model (DCM) is a trust model for Android and iOS that aims to distinguish legit applications from security threats to user safeness by answering the question: "is the developer of this app trustable"? It proposes security by design, where safety relies on a chain of trust mapping real-world levels of trust across organizations. For the technical implementation, DCM is heavily inspired by SSL/TLS certification protocol, as a proven model that has been working for over 30 years.

Title: FINER: Enhancing State-of-the-art Classifiers with Feature Attribution to Facilitate Security Analysis. (arXiv:2308.05362v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.05362
• Code URL: https://github.com/e0hyl/finer-explain
• Copy Paste: [[2308.05362]] FINER: Enhancing State-of-the-art Classifiers with Feature Attribution to Facilitate Security Analysis(http://arxiv.org/abs/2308.05362)
• Summary:

  Deep learning classifiers achieve state-of-the-art performance in various risk detection applications. They explore rich semantic representations and are supposed to automatically discover risk behaviors. However, due to the lack of transparency, the behavioral semantics cannot be conveyed to downstream security experts to reduce their heavy workload in security analysis. Although feature attribution (FA) methods can be used to explain deep learning, the underlying classifier is still blind to what behavior is suspicious, and the generated explanation cannot adapt to downstream tasks, incurring poor explanation fidelity and intelligibility. In this paper, we propose FINER, the first framework for risk detection classifiers to generate high-fidelity and high-intelligibility explanations. The high-level idea is to gather explanation efforts from model developer, FA designer, and security experts. To improve fidelity, we fine-tune the classifier with an explanation-guided multi-task learning strategy. To improve intelligibility, we engage task knowledge to adjust and ensemble FA methods. Extensive evaluations show that FINER improves explanation quality for risk detection. Moreover, we demonstrate that FINER outperforms a state-of-the-art tool in facilitating malware analysis.

privacy

Title: Decentralized Finance (DeFi): A Survey. (arXiv:2308.05282v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.05282
• Code URL: null
• Copy Paste: [[2308.05282]] Decentralized Finance (DeFi): A Survey(http://arxiv.org/abs/2308.05282)
• Summary:

  Decentralized Finance (DeFi) is a new paradigm in the creation, distribution, and utilization of financial services via the integration of blockchain technology. Our research conducts a comprehensive introduction and meticulous classification of various DeFi applications. Beyond that, we thoroughly analyze these risks from both technical and economic perspectives, spanning multiple layers. Lastly, we point out research directions in DeFi, encompassing areas of technological advancements, innovative economics, and privacy optimization.

Title: Your DRM Can Watch You Too: Exploring the Privacy Implications of Browsers (mis)Implementations of Widevine EME. (arXiv:2308.05416v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.05416
• Code URL: null
• Copy Paste: [[2308.05416]] Your DRM Can Watch You Too: Exploring the Privacy Implications of Browsers (mis)Implementations of Widevine EME(http://arxiv.org/abs/2308.05416)
• Summary:

  Thanks to HTML5, users can now view videos on Web browsers without installing plug-ins or relying on specific devices. In 2017, W3C published Encrypted Media Extensions (EME) as the first official Web standard for Digital Rights Management (DRM), with the overarching goal of allowing seamless integration of DRM systems on browsers. EME has prompted numerous voices of dissent with respect to the inadequate protection of users. Of particular interest, privacy concerns were articulated, especially that DRM systems inherently require uniquely identifying information on users' devices to control content distribution better. Despite this anecdotal evidence, we lack a comprehensive overview of how browsers have supported EME in practice and what privacy implications are caused by their implementations. In this paper, we fill this gap by investigating privacy leakage caused by EME relying on proprietary and closed-source DRM systems. We focus on Google Widevine because of its versatility and wide adoption. We conduct empirical experiments to show that browsers diverge when complying EME privacy guidelines, which might undermine users' privacy. For instance, we find that many browsers gladly give away the identifying Widevine Client ID with no or little explicit consent from users. Moreover, we characterize the privacy risks of users tracking when browsers miss applying EME guidelines regarding privacy. Because of being closed-source, our work involves reverse engineering to dissect the contents of EME messages as instantiated by Widevine. Finally, we implement EME Track, a tool that automatically exploits bad Widevine-based implementations to break privacy.

Title: A Homomorphic Encryption Framework for Privacy-Preserving Spiking Neural Networks. (arXiv:2308.05636v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.05636
• Code URL: null
• Copy Paste: [[2308.05636]] A Homomorphic Encryption Framework for Privacy-Preserving Spiking Neural Networks(http://arxiv.org/abs/2308.05636)
• Summary:

  Machine learning (ML) is widely used today, especially through deep neural networks (DNNs), however, increasing computational load and resource requirements have led to cloud-based solutions. To address this problem, a new generation of networks called Spiking Neural Networks (SNN) has emerged, which mimic the behavior of the human brain to improve efficiency and reduce energy consumption. These networks often process large amounts of sensitive information, such as confidential data, and thus privacy issues arise. Homomorphic encryption (HE) offers a solution, allowing calculations to be performed on encrypted data without decrypting it. This research compares traditional DNNs and SNNs using the Brakerski/Fan-Vercauteren (BFV) encryption scheme. The LeNet-5 model, a widely-used convolutional architecture, is used for both DNN and SNN models based on the LeNet-5 architecture, and the networks are trained and compared using the FashionMNIST dataset. The results show that SNNs using HE achieve up to 40% higher accuracy than DNNs for low values of the plaintext modulus t, although their execution time is longer due to their time-coding nature with multiple time-steps.

Title: The Privacy-Value-App Relationship and the Value-Centered Privacy Assistant. (arXiv:2308.05700v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.05700
• Code URL: null
• Copy Paste: [[2308.05700]] The Privacy-Value-App Relationship and the Value-Centered Privacy Assistant(http://arxiv.org/abs/2308.05700)
• Summary:

  Many of us make quick decisions that affect our data privacy on our smartphones without due consideration of our values. One such decision point is establishing whether to download a smartphone app or not. In this work, we aim to better understand the relationship between our values, our privacy preferences, and our app choices, as well as explore the effectiveness of a smartphone value-centered privacy assistant (VcPA) at promoting value-centered app selection. To do this, we conducted a mixed-methods study that involved two phases. The first was an online survey of 273 smartphone user's values and privacy preferences when considering whether to download one of two apps (Lose It! and OpenLitterMap). Our results suggest that values and privacy preferences are related in an app or context-dependent manner. The second phase was testing the VcPA with 77 users in a synthetic Mock App Store setting. We established usability of a VcPA, with the VcPA helping some users more than others with selecting apps consistent with their selected value profile. Future qualitative and context-specific explorations of user perspectives could contribute to adequately capturing the specific role of values for privacy decision-making and improving the VcPA.

Title: ReLU and Addition-based Gated RNN. (arXiv:2308.05629v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.05629
• Code URL: null
• Copy Paste: [[2308.05629]] ReLU and Addition-based Gated RNN(http://arxiv.org/abs/2308.05629)
• Summary:

  We replace the multiplication and sigmoid function of the conventional recurrent gate with addition and ReLU activation. This mechanism is designed to maintain long-term memory for sequence processing but at a reduced computational cost, thereby opening up for more efficient execution or larger models on restricted hardware. Recurrent Neural Networks (RNNs) with gating mechanisms such as LSTM and GRU have been widely successful in learning from sequential data due to their ability to capture long-term dependencies. Conventionally, the update based on current inputs and the previous state history is each multiplied with dynamic weights and combined to compute the next state. However, multiplication can be computationally expensive, especially for certain hardware architectures or alternative arithmetic systems such as homomorphic encryption. It is demonstrated that the novel gating mechanism can capture long-term dependencies for a standard synthetic sequence learning task while significantly reducing computational costs such that execution time is reduced by half on CPU and by one-third under encryption. Experimental results on handwritten text recognition tasks furthermore show that the proposed architecture can be trained to achieve comparable accuracy to conventional GRU and LSTM baselines. The gating mechanism introduced in this paper may enable privacy-preserving AI applications operating under homomorphic encryption by avoiding the multiplication of encrypted variables. It can also support quantization in (unencrypted) plaintext applications, with the potential for substantial performance gains since the addition-based formulation can avoid the expansion to double precision often required for multiplication.

protect

Title: Advancing Early Detection of Virus Yellows: Developing a Hybrid Convolutional Neural Network for Automatic Aphid Counting in Sugar Beet Fields. (arXiv:2308.05257v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05257
• Code URL: https://github.com/junfenggaolab/counting-aphids
• Copy Paste: [[2308.05257]] Advancing Early Detection of Virus Yellows: Developing a Hybrid Convolutional Neural Network for Automatic Aphid Counting in Sugar Beet Fields(http://arxiv.org/abs/2308.05257)
• Summary:

  Aphids are efficient vectors to transmit virus yellows in sugar beet fields. Timely monitoring and control of their populations are thus critical to prevent the large-scale outbreak of virus yellows. However, the manual counting of aphids, which is the most common practice, is labor-intensive and time-consuming. Additionally, two of the biggest challenges in aphid counting are that aphids are small objects and their density distributions are varied in different areas of the field. To address these challenges, we proposed a hybrid automatic aphid counting network architecture which integrates the detection network and the density map estimation network. When the distribution density of aphids is low, it utilizes an improved Yolov5 to count aphids. Conversely, when the distribution density of aphids is high, its witches to CSRNet to count aphids. To the best of our knowledge, this is the first framework integrating the detection network and the density map estimation network for counting tasks. Through comparison experiments of counting aphids, it verified that our proposed approach outperforms all other methods in counting aphids. It achieved the lowest MAE and RMSE values for both the standard and high-density aphid datasets: 2.93 and 4.01 (standard), and 34.19 and 38.66 (high-density), respectively. Moreover, the AP of the improved Yolov5 is 5% higher than that of the original Yolov5. Especially for extremely small aphids and densely distributed aphids, the detection performance of the improved Yolov5 is significantly better than the original Yolov5. This work provides an effective early warning for the virus yellows risk caused by aphids in sugar beet fields, offering protection for sugar beet growth and ensuring sugar beet yield. The datasets and project code are released at: https://github.com/JunfengGaolab/Counting-Aphids.

Title: Benchmarking Algorithmic Bias in Face Recognition: An Experimental Approach Using Synthetic Faces and Human Evaluation. (arXiv:2308.05441v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05441
• Code URL: null
• Copy Paste: [[2308.05441]] Benchmarking Algorithmic Bias in Face Recognition: An Experimental Approach Using Synthetic Faces and Human Evaluation(http://arxiv.org/abs/2308.05441)
• Summary:

  We propose an experimental method for measuring bias in face recognition systems. Existing methods to measure bias depend on benchmark datasets that are collected in the wild and annotated for protected (e.g., race, gender) and non-protected (e.g., pose, lighting) attributes. Such observational datasets only permit correlational conclusions, e.g., "Algorithm A's accuracy is different on female and male faces in dataset X.". By contrast, experimental methods manipulate attributes individually and thus permit causal conclusions, e.g., "Algorithm A's accuracy is affected by gender and skin color."

  Our method is based on generating synthetic faces using a neural face generator, where each attribute of interest is modified independently while leaving all other attributes constant. Human observers crucially provide the ground truth on perceptual identity similarity between synthetic image pairs. We validate our method quantitatively by evaluating race and gender biases of three research-grade face recognition models. Our synthetic pipeline reveals that for these algorithms, accuracy is lower for Black and East Asian population subgroups. Our method can also quantify how perceptual changes in attributes affect face identity distances reported by these models. Our large synthetic dataset, consisting of 48,000 synthetic face image pairs (10,200 unique synthetic faces) and 555,000 human annotations (individual attributes and pairwise identity comparisons) is available to researchers in this important area.

Title: Accountability of Things: Large-Scale Tamper-Evident Logging for Smart Devices. (arXiv:2308.05557v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.05557
• Code URL: null
• Copy Paste: [[2308.05557]] Accountability of Things: Large-Scale Tamper-Evident Logging for Smart Devices(http://arxiv.org/abs/2308.05557)
• Summary:

  Our modern world relies on a growing number of interconnected and interacting devices, leading to a plethora of logs establishing audit trails for all kinds of events. Simultaneously, logs become increasingly important for forensic investigations, and thus, an adversary will aim to alter logs to avoid culpability, e.g., by compromising devices that generate and store logs. Thus, it is essential to ensure that no one can tamper with any logs without going undetected. However, existing approaches to establish tamper evidence of logs do not scale and cannot protect the increasingly large number of devices found today, as they impose large storage or network overheads. Additionally, most schemes do not provide an efficient mechanism to prove that individual events have been logged to establish accountability when different devices interact.

  This paper introduces a novel scheme for practical large-scale tamper-evident logging with the help of a trusted third party. To achieve this, we present a new binary hash tree construction designed around timestamps to achieve constant storage overhead with a configured temporal resolution. Additionally, our design enables the efficient construction of shareable proofs, proving that an event was indeed logged. Our evaluation shows that - using practical parameters - our scheme can localize any tampering of logs with a sub-second resolution, with a constant overhead of ~8KB per hour per device.

defense

Title: Critical Points ++: An Agile Point Cloud Importance Measure for Robust Classification, Adversarial Defense and Explainable AI. (arXiv:2308.05525v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05525
• Code URL: null
• Copy Paste: [[2308.05525]] Critical Points ++: An Agile Point Cloud Importance Measure for Robust Classification, Adversarial Defense and Explainable AI(http://arxiv.org/abs/2308.05525)
• Summary:

  The ability to cope accurately and fast with Out-Of-Distribution (OOD) samples is crucial in real-world safety demanding applications. In this work we first study the interplay between critical points of 3D point clouds and OOD samples. Our findings are that common corruptions and outliers are often interpreted as critical points. We generalize the notion of critical points into importance measures. We show that training a classification network based only on less important points dramatically improves robustness, at a cost of minor performance loss on the clean set. We observe that normalized entropy is highly informative for corruption analysis. An adaptive threshold based on normalized entropy is suggested for selecting the set of uncritical points. Our proposed importance measure is extremely fast to compute. We show it can be used for a variety of applications, such as Explainable AI (XAI), Outlier Removal, Uncertainty Estimation, Robust Classification and Adversarial Defense. We reach SOTA results on the two latter tasks.

Title: Analysis of the LockBit 3.0 and its infiltration into Advanced's infrastructure crippling NHS services. (arXiv:2308.05565v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.05565
• Code URL: null
• Copy Paste: [[2308.05565]] Analysis of the LockBit 3(http://arxiv.org/abs/2308.05565)
• Summary:

  The LockBit 3.0 ransomware variant is arguably the most threatening of malware in recent times. With no regard for a victim's industry, the ransomware has undergone several evolutions to arrive at an adaptable and evasive variant which has been a menace to governments and organisations, recently infiltrating Advanced Computer Software group. Previous LockBit studies mostly concentrated on measuring the impact of the ransomware attack, prevention, encryption detection, decryption, or data recovery, thereby providing little or no benefit to the less tech savvy populace as a detailed breakdown of the mode of attack is rarely examined. This article analyses the LockBit 3.0 attack techniques with a contextual illustration of the attack on Advanced Computer Software group. With the NHS being a major client of the organisation, and its services alongside 15 other clients being crippled for hours during the attack, attention is drawn to how dreadful such disruption may be in critical organisations. We observed that the upgrade of Lockbit based on releasing newer versions is in a bid to continuously ensure the malware's efficiency - a virtue that keeps it at the zenith - by staying ahead of improved defenses. Our study highlights social engineering as a vibrant portal to Lockbit's maliciousness and indicates an investment in detection systems may profit more than in prevention systems. Therefore, further research should consider improving detection systems against Lockbit 3.0.

Title: Symmetry Defense Against XGBoost Adversarial Perturbation Attacks. (arXiv:2308.05575v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.05575
• Code URL: null
• Copy Paste: [[2308.05575]] Symmetry Defense Against XGBoost Adversarial Perturbation Attacks(http://arxiv.org/abs/2308.05575)
• Summary:

  We examine whether symmetry can be used to defend tree-based ensemble classifiers such as gradient-boosting decision trees (GBDTs) against adversarial perturbation attacks. The idea is based on a recent symmetry defense for convolutional neural network classifiers (CNNs) that utilizes CNNs' lack of invariance with respect to symmetries. CNNs lack invariance because they can classify a symmetric sample, such as a horizontally flipped image, differently from the original sample. CNNs' lack of invariance also means that CNNs can classify symmetric adversarial samples differently from the incorrect classification of adversarial samples. Using CNNs' lack of invariance, the recent CNN symmetry defense has shown that the classification of symmetric adversarial samples reverts to the correct sample classification. In order to apply the same symmetry defense to GBDTs, we examine GBDT invariance and are the first to show that GBDTs also lack invariance with respect to symmetries. We apply and evaluate the GBDT symmetry defense for nine datasets against six perturbation attacks with a threat model that ranges from zero-knowledge to perfect-knowledge adversaries. Using the feature inversion symmetry against zero-knowledge adversaries, we achieve up to 100% accuracy on adversarial samples even when default and robust classifiers have 0% accuracy. Using the feature inversion and horizontal flip symmetries against perfect-knowledge adversaries, we achieve up to over 95% accuracy on adversarial samples for the GBDT classifier of the F-MNIST dataset even when default and robust classifiers have 0% accuracy.

attack

Title: Data-Free Model Extraction Attacks in the Context of Object Detection. (arXiv:2308.05127v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.05127
• Code URL: null
• Copy Paste: [[2308.05127]] Data-Free Model Extraction Attacks in the Context of Object Detection(http://arxiv.org/abs/2308.05127)
• Summary:

  A significant number of machine learning models are vulnerable to model extraction attacks, which focus on stealing the models by using specially curated queries against the target model. This task is well accomplished by using part of the training data or a surrogate dataset to train a new model that mimics a target model in a white-box environment. In pragmatic situations, however, the target models are trained on private datasets that are inaccessible to the adversary. The data-free model extraction technique replaces this problem when it comes to using queries artificially curated by a generator similar to that used in Generative Adversarial Nets. We propose for the first time, to the best of our knowledge, an adversary black box attack extending to a regression problem for predicting bounding box coordinates in object detection. As part of our study, we found that defining a loss function and using a novel generator setup is one of the key aspects in extracting the target model. We find that the proposed model extraction method achieves significant results by using reasonable queries. The discovery of this object detection vulnerability will support future prospects for securing such models.

Title: Adv-Inpainting: Generating Natural and Transferable Adversarial Patch via Attention-guided Feature Fusion. (arXiv:2308.05320v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05320
• Code URL: null
• Copy Paste: [[2308.05320]] Adv-Inpainting: Generating Natural and Transferable Adversarial Patch via Attention-guided Feature Fusion(http://arxiv.org/abs/2308.05320)
• Summary:

  The rudimentary adversarial attacks utilize additive noise to attack facial recognition (FR) models. However, because manipulating the total face is impractical in the physical setting, most real-world FR attacks are based on adversarial patches, which limit perturbations to a small area. Previous adversarial patch attacks often resulted in unnatural patterns and clear boundaries that were easily noticeable. In this paper, we argue that generating adversarial patches with plausible content can result in stronger transferability than using additive noise or directly sampling from the latent space. To generate natural-looking and highly transferable adversarial patches, we propose an innovative two-stage coarse-to-fine attack framework called Adv-Inpainting. In the first stage, we propose an attention-guided StyleGAN (Att-StyleGAN) that adaptively combines texture and identity features based on the attention map to generate high-transferable and natural adversarial patches. In the second stage, we design a refinement network with a new boundary variance loss to further improve the coherence between the patch and its surrounding area. Experiment results demonstrate that Adv-Inpainting is stealthy and can produce adversarial patches with stronger transferability and improved visual quality than previous adversarial patch attacks.

Title: Hard No-Box Adversarial Attack on Skeleton-Based Human Action Recognition with Skeleton-Motion-Informed Gradient. (arXiv:2308.05681v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05681
• Code URL: null
• Copy Paste: [[2308.05681]] Hard No-Box Adversarial Attack on Skeleton-Based Human Action Recognition with Skeleton-Motion-Informed Gradient(http://arxiv.org/abs/2308.05681)
• Summary:

  Recently, methods for skeleton-based human activity recognition have been shown to be vulnerable to adversarial attacks. However, these attack methods require either the full knowledge of the victim (i.e. white-box attacks), access to training data (i.e. transfer-based attacks) or frequent model queries (i.e. black-box attacks). All their requirements are highly restrictive, raising the question of how detrimental the vulnerability is. In this paper, we show that the vulnerability indeed exists. To this end, we consider a new attack task: the attacker has no access to the victim model or the training data or labels, where we coin the term hard no-box attack. Specifically, we first learn a motion manifold where we define an adversarial loss to compute a new gradient for the attack, named skeleton-motion-informed (SMI) gradient. Our gradient contains information of the motion dynamics, which is different from existing gradient-based attack methods that compute the loss gradient assuming each dimension in the data is independent. The SMI gradient can augment many gradient-based attack methods, leading to a new family of no-box attack methods. Extensive evaluation and comparison show that our method imposes a real threat to existing classifiers. They also show that the SMI gradient improves the transferability and imperceptibility of adversarial samples in both no-box and transfer-based black-box settings.

robust

Title: Robust Object Modeling for Visual Tracking. (arXiv:2308.05140v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05140
• Code URL: https://github.com/dawnyc/romtrack
• Copy Paste: [[2308.05140]] Robust Object Modeling for Visual Tracking(http://arxiv.org/abs/2308.05140)
• Summary:

  Object modeling has become a core part of recent tracking frameworks. Current popular tackers use Transformer attention to extract the template feature separately or interactively with the search region. However, separate template learning lacks communication between the template and search regions, which brings difficulty in extracting discriminative target-oriented features. On the other hand, interactive template learning produces hybrid template features, which may introduce potential distractors to the template via the cluttered search regions. To enjoy the merits of both methods, we propose a robust object modeling framework for visual tracking (ROMTrack), which simultaneously models the inherent template and the hybrid template features. As a result, harmful distractors can be suppressed by combining the inherent features of target objects with search regions' guidance. Target-related features can also be extracted using the hybrid template, thus resulting in a more robust object modeling framework. To further enhance robustness, we present novel variation tokens to depict the ever-changing appearance of target objects. Variation tokens are adaptable to object deformation and appearance variations, which can boost overall performance with negligible computation. Experiments show that our ROMTrack sets a new state-of-the-art on multiple benchmarks.

Title: RLSAC: Reinforcement Learning enhanced Sample Consensus for End-to-End Robust Estimation. (arXiv:2308.05318v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05318
• Code URL: https://github.com/irmvlab/rlsac
• Copy Paste: [[2308.05318]] RLSAC: Reinforcement Learning enhanced Sample Consensus for End-to-End Robust Estimation(http://arxiv.org/abs/2308.05318)
• Summary:

  Robust estimation is a crucial and still challenging task, which involves estimating model parameters in noisy environments. Although conventional sampling consensus-based algorithms sample several times to achieve robustness, these algorithms cannot use data features and historical information effectively. In this paper, we propose RLSAC, a novel Reinforcement Learning enhanced SAmple Consensus framework for end-to-end robust estimation. RLSAC employs a graph neural network to utilize both data and memory features to guide exploring directions for sampling the next minimum set. The feedback of downstream tasks serves as the reward for unsupervised training. Therefore, RLSAC can avoid differentiating to learn the features and the feedback of downstream tasks for end-to-end robust estimation. In addition, RLSAC integrates a state transition module that encodes both data and memory features. Our experimental results demonstrate that RLSAC can learn from features to gradually explore a better hypothesis. Through analysis, it is apparent that RLSAC can be easily transferred to other sampling consensus-based robust estimation tasks. To the best of our knowledge, RLSAC is also the first method that uses reinforcement learning to sample consensus for end-to-end robust estimation. We release our codes at https://github.com/IRMVLab/RLSAC.

Title: Robust Localization with Visual-Inertial Odometry Constraints for Markerless Mobile AR. (arXiv:2308.05394v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05394
• Code URL: null
• Copy Paste: [[2308.05394]] Robust Localization with Visual-Inertial Odometry Constraints for Markerless Mobile AR(http://arxiv.org/abs/2308.05394)
• Summary:

  Visual Inertial Odometry (VIO) is an essential component of modern Augmented Reality (AR) applications. However, VIO only tracks the relative pose of the device, leading to drift over time. Absolute pose estimation methods infer the device's absolute pose, but their accuracy depends on the input quality. This paper introduces VIO-APR, a new framework for markerless mobile AR that combines an absolute pose regressor (APR) with a local VIO tracking system. VIO-APR uses VIO to assess the reliability of the APR and the APR to identify and compensate for VIO drift. This feedback loop results in more accurate positioning and more stable AR experiences. To evaluate VIO-APR, we created a dataset that combines camera images with ARKit's VIO system output for six indoor and outdoor scenes of various scales. Over this dataset, VIO-APR improves the median accuracy of popular APR by up to 36\% in position and 29\% in orientation, increases the percentage of frames in the high ($0.25 m, 2^{\circ}$) accuracy level by up to 112\% and reduces the percentage of frames predicted below the low ($5 m, 10^\circ$) accuracy greatly. We implement VIO-APR into a mobile AR application using Unity to demonstrate its capabilities. VIO-APR results in noticeably more accurate localization and a more stable overall experience.

Title: SC3K: Self-supervised and Coherent 3D Keypoints Estimation from Rotated, Noisy, and Decimated Point Cloud Data. (arXiv:2308.05410v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05410
• Code URL: null
• Copy Paste: [[2308.05410]] SC3K: Self-supervised and Coherent 3D Keypoints Estimation from Rotated, Noisy, and Decimated Point Cloud Data(http://arxiv.org/abs/2308.05410)
• Summary:

  This paper proposes a new method to infer keypoints from arbitrary object categories in practical scenarios where point cloud data (PCD) are noisy, down-sampled and arbitrarily rotated. Our proposed model adheres to the following principles: i) keypoints inference is fully unsupervised (no annotation given), ii) keypoints position error should be low and resilient to PCD perturbations (robustness), iii) keypoints should not change their indexes for the intra-class objects (semantic coherence), iv) keypoints should be close to or proximal to PCD surface (compactness). We achieve these desiderata by proposing a new self-supervised training strategy for keypoints estimation that does not assume any a priori knowledge of the object class, and a model architecture with coupled auxiliary losses that promotes the desired keypoints properties. We compare the keypoints estimated by the proposed approach with those of the state-of-the-art unsupervised approaches. The experiments show that our approach outperforms by estimating keypoints with improved coverage (+9.41%) while being semantically consistent (+4.66%) that best characterizes the object's 3D shape for downstream tasks. Code and data are available at: https://github.com/IITPAVIS/SC3K

Title: Deep Fusion Transformer Network with Weighted Vector-Wise Keypoints Voting for Robust 6D Object Pose Estimation. (arXiv:2308.05438v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05438
• Code URL: https://github.com/junzastar/dftr_voting
• Copy Paste: [[2308.05438]] Deep Fusion Transformer Network with Weighted Vector-Wise Keypoints Voting for Robust 6D Object Pose Estimation(http://arxiv.org/abs/2308.05438)
• Summary:

  One critical challenge in 6D object pose estimation from a single RGBD image is efficient integration of two different modalities, i.e., color and depth. In this work, we tackle this problem by a novel Deep Fusion Transformer~(DFTr) block that can aggregate cross-modality features for improving pose estimation. Unlike existing fusion methods, the proposed DFTr can better model cross-modality semantic correlation by leveraging their semantic similarity, such that globally enhanced features from different modalities can be better integrated for improved information extraction. Moreover, to further improve robustness and efficiency, we introduce a novel weighted vector-wise voting algorithm that employs a non-iterative global optimization strategy for precise 3D keypoint localization while achieving near real-time inference. Extensive experiments show the effectiveness and strong generalization capability of our proposed 3D keypoint voting algorithm. Results on four widely used benchmarks also demonstrate that our method outperforms the state-of-the-art methods by large margins.

Title: KS-APR: Keyframe Selection for Robust Absolute Pose Regression. (arXiv:2308.05459v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05459
• Code URL: null
• Copy Paste: [[2308.05459]] KS-APR: Keyframe Selection for Robust Absolute Pose Regression(http://arxiv.org/abs/2308.05459)
• Summary:

  Markerless Mobile Augmented Reality (AR) aims to anchor digital content in the physical world without using specific 2D or 3D objects. Absolute Pose Regressors (APR) are end-to-end machine learning solutions that infer the device's pose from a single monocular image. Thanks to their low computation cost, they can be directly executed on the constrained hardware of mobile AR devices. However, APR methods tend to yield significant inaccuracies for input images that are too distant from the training set. This paper introduces KS-APR, a pipeline that assesses the reliability of an estimated pose with minimal overhead by combining the inference results of the APR and the prior images in the training set. Mobile AR systems tend to rely upon visual-inertial odometry to track the relative pose of the device during the experience. As such, KS-APR favours reliability over frequency, discarding unreliable poses. This pipeline can integrate most existing APR methods to improve accuracy by filtering unreliable images with their pose estimates. We implement the pipeline on three types of APR models on indoor and outdoor datasets. The median error on position and orientation is reduced for all models, and the proportion of large errors is minimized across datasets. Our method enables state-of-the-art APRs such as DFNetdm to outperform single-image and sequential APR methods. These results demonstrate the scalability and effectiveness of KS-APR for visual localization tasks that do not require one-shot decisions.

Title: Robust Asymmetric Loss for Multi-Label Long-Tailed Learning. (arXiv:2308.05542v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05542
• Code URL: https://github.com/kalelpark/ral
• Copy Paste: [[2308.05542]] Robust Asymmetric Loss for Multi-Label Long-Tailed Learning(http://arxiv.org/abs/2308.05542)
• Summary:

  In real medical data, training samples typically show long-tailed distributions with multiple labels. Class distribution of the medical data has a long-tailed shape, in which the incidence of different diseases is quite varied, and at the same time, it is not unusual for images taken from symptomatic patients to be multi-label diseases. Therefore, in this paper, we concurrently address these two issues by putting forth a robust asymmetric loss on the polynomial function. Since our loss tackles both long-tailed and multi-label classification problems simultaneously, it leads to a complex design of the loss function with a large number of hyper-parameters. Although a model can be highly fine-tuned due to a large number of hyper-parameters, it is difficult to optimize all hyper-parameters at the same time, and there might be a risk of overfitting a model. Therefore, we regularize the loss function using the Hill loss approach, which is beneficial to be less sensitive against the numerous hyper-parameters so that it reduces the risk of overfitting the model. For this reason, the proposed loss is a generic method that can be applied to most medical image classification tasks and does not make the training process more time-consuming. We demonstrate that the proposed robust asymmetric loss performs favorably against the long-tailed with multi-label medical image classification in addition to the various long-tailed single-label datasets. Notably, our method achieves Top-5 results on the CXR-LT dataset of the ICCV CVAMD 2023 competition. We opensource our implementation of the robust asymmetric loss in the public repository: https://github.com/kalelpark/RAL.

Title: Test-Time Selection for Robust Skin Lesion Analysis. (arXiv:2308.05595v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05595
• Code URL: null
• Copy Paste: [[2308.05595]] Test-Time Selection for Robust Skin Lesion Analysis(http://arxiv.org/abs/2308.05595)
• Summary:

  Skin lesion analysis models are biased by artifacts placed during image acquisition, which influence model predictions despite carrying no clinical information. Solutions that address this problem by regularizing models to prevent learning those spurious features achieve only partial success, and existing test-time debiasing techniques are inappropriate for skin lesion analysis due to either making unrealistic assumptions on the distribution of test data or requiring laborious annotation from medical practitioners. We propose TTS (Test-Time Selection), a human-in-the-loop method that leverages positive (e.g., lesion area) and negative (e.g., artifacts) keypoints in test samples. TTS effectively steers models away from exploiting spurious artifact-related correlations without retraining, and with less annotation requirements. Our solution is robust to a varying availability of annotations, and different levels of bias. We showcase on the ISIC2019 dataset (for which we release a subset of annotated images) how our model could be deployed in the real-world for mitigating bias.

Title: FrozenRecon: Pose-free 3D Scene Reconstruction with Frozen Depth Models. (arXiv:2308.05733v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05733
• Code URL: null
• Copy Paste: [[2308.05733]] FrozenRecon: Pose-free 3D Scene Reconstruction with Frozen Depth Models(http://arxiv.org/abs/2308.05733)
• Summary:

  3D scene reconstruction is a long-standing vision task. Existing approaches can be categorized into geometry-based and learning-based methods. The former leverages multi-view geometry but can face catastrophic failures due to the reliance on accurate pixel correspondence across views. The latter was proffered to mitigate these issues by learning 2D or 3D representation directly. However, without a large-scale video or 3D training data, it can hardly generalize to diverse real-world scenarios due to the presence of tens of millions or even billions of optimization parameters in the deep network. Recently, robust monocular depth estimation models trained with large-scale datasets have been proven to possess weak 3D geometry prior, but they are insufficient for reconstruction due to the unknown camera parameters, the affine-invariant property, and inter-frame inconsistency. Here, we propose a novel test-time optimization approach that can transfer the robustness of affine-invariant depth models such as LeReS to challenging diverse scenes while ensuring inter-frame consistency, with only dozens of parameters to optimize per video frame. Specifically, our approach involves freezing the pre-trained affine-invariant depth model's depth predictions, rectifying them by optimizing the unknown scale-shift values with a geometric consistency alignment module, and employing the resulting scale-consistent depth maps to robustly obtain camera poses and achieve dense scene reconstruction, even in low-texture regions. Experiments show that our method achieves state-of-the-art cross-dataset reconstruction on five zero-shot testing datasets.

Title: MapTRv2: An End-to-End Framework for Online Vectorized HD Map Construction. (arXiv:2308.05736v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05736
• Code URL: https://github.com/hustvl/maptr
• Copy Paste: [[2308.05736]] MapTRv2: An End-to-End Framework for Online Vectorized HD Map Construction(http://arxiv.org/abs/2308.05736)
• Summary:

  High-definition (HD) map provides abundant and precise static environmental information of the driving scene, serving as a fundamental and indispensable component for planning in autonomous driving system. In this paper, we present \textbf{Map} \textbf{TR}ansformer, an end-to-end framework for online vectorized HD map construction. We propose a unified permutation-equivalent modeling approach, \ie, modeling map element as a point set with a group of equivalent permutations, which accurately describes the shape of map element and stabilizes the learning process. We design a hierarchical query embedding scheme to flexibly encode structured map information and perform hierarchical bipartite matching for map element learning. To speed up convergence, we further introduce auxiliary one-to-many matching and dense supervision. The proposed method well copes with various map elements with arbitrary shapes. It runs at real-time inference speed and achieves state-of-the-art performance on both nuScenes and Argoverse2 datasets. Abundant qualitative results show stable and robust map construction quality in complex and various driving scenes. Code and more demos are available at \url{https://github.com/hustvl/MapTR} for facilitating further studies and applications.

Title: PlankAssembly: Robust 3D Reconstruction from Three Orthographic Views with Learnt Shape Programs. (arXiv:2308.05744v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05744
• Code URL: https://github.com/manycore-research/PlankAssembly
• Copy Paste: [[2308.05744]] PlankAssembly: Robust 3D Reconstruction from Three Orthographic Views with Learnt Shape Programs(http://arxiv.org/abs/2308.05744)
• Summary:

  In this paper, we develop a new method to automatically convert 2D line drawings from three orthographic views into 3D CAD models. Existing methods for this problem reconstruct 3D models by back-projecting the 2D observations into 3D space while maintaining explicit correspondence between the input and output. Such methods are sensitive to errors and noises in the input, thus often fail in practice where the input drawings created by human designers are imperfect. To overcome this difficulty, we leverage the attention mechanism in a Transformer-based sequence generation model to learn flexible mappings between the input and output. Further, we design shape programs which are suitable for generating the objects of interest to boost the reconstruction accuracy and facilitate CAD modeling applications. Experiments on a new benchmark dataset show that our method significantly outperforms existing ones when the inputs are noisy or incomplete.

Title: Finding Already Debunked Narratives via Multistage Retrieval: Enabling Cross-Lingual, Cross-Dataset and Zero-Shot Learning. (arXiv:2308.05680v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.05680
• Code URL: null
• Copy Paste: [[2308.05680]] Finding Already Debunked Narratives via Multistage Retrieval: Enabling Cross-Lingual, Cross-Dataset and Zero-Shot Learning(http://arxiv.org/abs/2308.05680)
• Summary:

  The task of retrieving already debunked narratives aims to detect stories that have already been fact-checked. The successful detection of claims that have already been debunked not only reduces the manual efforts of professional fact-checkers but can also contribute to slowing the spread of misinformation. Mainly due to the lack of readily available data, this is an understudied problem, particularly when considering the cross-lingual task, i.e. the retrieval of fact-checking articles in a language different from the language of the online post being checked. This paper fills this gap by (i) creating a novel dataset to enable research on cross-lingual retrieval of already debunked narratives, using tweets as queries to a database of fact-checking articles; (ii) presenting an extensive experiment to benchmark fine-tuned and off-the-shelf multilingual pre-trained Transformer models for this task; and (iii) proposing a novel multistage framework that divides this cross-lingual debunk retrieval task into refinement and re-ranking stages. Results show that the task of cross-lingual retrieval of already debunked narratives is challenging and off-the-shelf Transformer models fail to outperform a strong lexical-based baseline (BM25). Nevertheless, our multistage retrieval framework is robust, outperforming BM25 in most scenarios and enabling cross-domain and zero-shot learning, without significantly harming the model's performance.

Title: Byzantine-Robust Decentralized Stochastic Optimization with Stochastic Gradient Noise-Independent Learning Error. (arXiv:2308.05292v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.05292
• Code URL: null
• Copy Paste: [[2308.05292]] Byzantine-Robust Decentralized Stochastic Optimization with Stochastic Gradient Noise-Independent Learning Error(http://arxiv.org/abs/2308.05292)
• Summary:

  This paper studies Byzantine-robust stochastic optimization over a decentralized network, where every agent periodically communicates with its neighbors to exchange local models, and then updates its own local model by stochastic gradient descent (SGD). The performance of such a method is affected by an unknown number of Byzantine agents, which conduct adversarially during the optimization process. To the best of our knowledge, there is no existing work that simultaneously achieves a linear convergence speed and a small learning error. We observe that the learning error is largely dependent on the intrinsic stochastic gradient noise. Motivated by this observation, we introduce two variance reduction methods, stochastic average gradient algorithm (SAGA) and loopless stochastic variance-reduced gradient (LSVRG), to Byzantine-robust decentralized stochastic optimization for eliminating the negative effect of the stochastic gradient noise. The two resulting methods, BRAVO-SAGA and BRAVO-LSVRG, enjoy both linear convergence speeds and stochastic gradient noise-independent learning errors. Such learning errors are optimal for a class of methods based on total variation (TV)-norm regularization and stochastic subgradient update. We conduct extensive numerical experiments to demonstrate their effectiveness under various Byzantine attacks.

Title: AutoGluon-TimeSeries: AutoML for Probabilistic Time Series Forecasting. (arXiv:2308.05566v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.05566
• Code URL: null
• Copy Paste: [[2308.05566]] AutoGluon-TimeSeries: AutoML for Probabilistic Time Series Forecasting(http://arxiv.org/abs/2308.05566)
• Summary:

  We introduce AutoGluon-TimeSeries - an open-source AutoML library for probabilistic time series forecasting. Focused on ease of use and robustness, AutoGluon-TimeSeries enables users to generate accurate point and quantile forecasts with just 3 lines of Python code. Built on the design philosophy of AutoGluon, AutoGluon-TimeSeries leverages ensembles of diverse forecasting models to deliver high accuracy within a short training time. AutoGluon-TimeSeries combines both conventional statistical models, machine-learning based forecasting approaches, and ensembling techniques. In our evaluation on 29 benchmark datasets, AutoGluon-TimeSeries demonstrates strong empirical performance, outperforming a range of forecasting methods in terms of both point and quantile forecast accuracy, and often even improving upon the best-in-hindsight combination of prior methods.

biometric

steal

extraction

Title: Spatial Gated Multi-Layer Perceptron for Land Use and Land Cover Mapping. (arXiv:2308.05235v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05235
• Code URL: https://github.com/aj1365/sgumlp
• Copy Paste: [[2308.05235]] Spatial Gated Multi-Layer Perceptron for Land Use and Land Cover Mapping(http://arxiv.org/abs/2308.05235)
• Summary:

  Convolutional Neural Networks (CNNs) are models that are utilized extensively for the hierarchical extraction of features. Vision transformers (ViTs), through the use of a self-attention mechanism, have recently achieved superior modeling of global contextual information compared to CNNs. However, to realize their image classification strength, ViTs require substantial training datasets. Where the available training data are limited, current advanced multi-layer perceptrons (MLPs) can provide viable alternatives to both deep CNNs and ViTs. In this paper, we developed the SGU-MLP, a learning algorithm that effectively uses both MLPs and spatial gating units (SGUs) for precise land use land cover (LULC) mapping. Results illustrated the superiority of the developed SGU-MLP classification algorithm over several CNN and CNN-ViT-based models, including HybridSN, ResNet, iFormer, EfficientFormer and CoAtNet. The proposed SGU-MLP algorithm was tested through three experiments in Houston, USA, Berlin, Germany and Augsburg, Germany. The SGU-MLP classification model was found to consistently outperform the benchmark CNN and CNN-ViT-based algorithms. For example, for the Houston experiment, SGU-MLP significantly outperformed HybridSN, CoAtNet, Efficientformer, iFormer and ResNet by approximately 15%, 19%, 20%, 21%, and 25%, respectively, in terms of average accuracy. The code will be made publicly available at https://github.com/aj1365/SGUMLP

Title: HGDNet: A Height-Hierarchy Guided Dual-Decoder Network for Single View Building Extraction and Height Estimation. (arXiv:2308.05387v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05387
• Code URL: null
• Copy Paste: [[2308.05387]] HGDNet: A Height-Hierarchy Guided Dual-Decoder Network for Single View Building Extraction and Height Estimation(http://arxiv.org/abs/2308.05387)
• Summary:

  Unifying the correlative single-view satellite image building extraction and height estimation tasks indicates a promising way to share representations and acquire generalist model for large-scale urban 3D reconstruction. However, the common spatial misalignment between building footprints and stereo-reconstructed nDSM height labels incurs degraded performance on both tasks. To address this issue, we propose a Height-hierarchy Guided Dual-decoder Network (HGDNet) to estimate building height. Under the guidance of synthesized discrete height-hierarchy nDSM, auxiliary height-hierarchical building extraction branch enhance the height estimation branch with implicit constraints, yielding an accuracy improvement of more than 6% on the DFC 2023 track2 dataset. Additional two-stage cascade architecture is adopted to achieve more accurate building extraction. Experiments on the DFC 2023 Track 2 dataset shows the superiority of the proposed method in building height estimation ({\delta}1:0.8012), instance extraction (AP50:0.7730), and the final average score 0.7871 ranks in the first place in test phase.

Title: Learning Gabor Texture Features for Fine-Grained Recognition. (arXiv:2308.05396v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05396
• Code URL: null
• Copy Paste: [[2308.05396]] Learning Gabor Texture Features for Fine-Grained Recognition(http://arxiv.org/abs/2308.05396)
• Summary:

  Extracting and using class-discriminative features is critical for fine-grained recognition. Existing works have demonstrated the possibility of applying deep CNNs to exploit features that distinguish similar classes. However, CNNs suffer from problems including frequency bias and loss of detailed local information, which restricts the performance of recognizing fine-grained categories. To address the challenge, we propose a novel texture branch as complimentary to the CNN branch for feature extraction. We innovatively utilize Gabor filters as a powerful extractor to exploit texture features, motivated by the capability of Gabor filters in effectively capturing multi-frequency features and detailed local information. We implement several designs to enhance the effectiveness of Gabor filters, including imposing constraints on parameter values and developing a learning method to determine the optimal parameters. Moreover, we introduce a statistical feature extractor to utilize informative statistical information from the signals captured by Gabor filters, and a gate selection mechanism to enable efficient computation by only considering qualified regions as input for texture extraction. Through the integration of features from the Gabor-filter-based texture branch and CNN-based semantic branch, we achieve comprehensive information extraction. We demonstrate the efficacy of our method on multiple datasets, including CUB-200-2011, NA-bird, Stanford Dogs, and GTOS-mobile. State-of-the-art performance is achieved using our approach.

Title: A Generalized Physical-knowledge-guided Dynamic Model for Underwater Image Enhancement. (arXiv:2308.05447v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05447
• Code URL: null
• Copy Paste: [[2308.05447]] A Generalized Physical-knowledge-guided Dynamic Model for Underwater Image Enhancement(http://arxiv.org/abs/2308.05447)
• Summary:

  Underwater images often suffer from color distortion and low contrast resulting in various image types, due to the scattering and absorption of light by water. While it is difficult to obtain high-quality paired training samples with a generalized model. To tackle these challenges, we design a Generalized Underwater image enhancement method via a Physical-knowledge-guided Dynamic Model (short for GUPDM), consisting of three parts: Atmosphere-based Dynamic Structure (ADS), Transmission-guided Dynamic Structure (TDS), and Prior-based Multi-scale Structure (PMS). In particular, to cover complex underwater scenes, this study changes the global atmosphere light and the transmission to simulate various underwater image types (e.g., the underwater image color ranging from yellow to blue) through the formation model. We then design ADS and TDS that use dynamic convolutions to adaptively extract prior information from underwater images and generate parameters for PMS. These two modules enable the network to select appropriate parameters for various water types adaptively. Besides, the multi-scale feature extraction module in PMS uses convolution blocks with different kernel sizes and obtains weights for each feature map via channel attention block and fuses them to boost the receptive field of the network. The source code will be available at \href{https://github.com/shiningZZ/GUPDM}{https://github.com/shiningZZ/GUPDM}.

Title: Self-Supervised Monocular Depth Estimation by Direction-aware Cumulative Convolution Network. (arXiv:2308.05605v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05605
• Code URL: https://github.com/wencheng256/daccn
• Copy Paste: [[2308.05605]] Self-Supervised Monocular Depth Estimation by Direction-aware Cumulative Convolution Network(http://arxiv.org/abs/2308.05605)
• Summary:

  Monocular depth estimation is known as an ill-posed task in which objects in a 2D image usually do not contain sufficient information to predict their depth. Thus, it acts differently from other tasks (e.g., classification and segmentation) in many ways. In this paper, we find that self-supervised monocular depth estimation shows a direction sensitivity and environmental dependency in the feature representation. But the current backbones borrowed from other tasks pay less attention to handling different types of environmental information, limiting the overall depth accuracy. To bridge this gap, we propose a new Direction-aware Cumulative Convolution Network (DaCCN), which improves the depth feature representation in two aspects. First, we propose a direction-aware module, which can learn to adjust the feature extraction in each direction, facilitating the encoding of different types of information. Secondly, we design a new cumulative convolution to improve the efficiency for aggregating important environmental information. Experiments show that our method achieves significant improvements on three widely used benchmarks, KITTI, Cityscapes, and Make3D, setting a new state-of-the-art performance on the popular benchmarks with all three types of self-supervision.

Title: LASIGE and UNICAGE solution to the NASA LitCoin NLP Competition. (arXiv:2308.05609v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.05609
• Code URL: null
• Copy Paste: [[2308.05609]] LASIGE and UNICAGE solution to the NASA LitCoin NLP Competition(http://arxiv.org/abs/2308.05609)
• Summary:

  Biomedical Natural Language Processing (NLP) tends to become cumbersome for most researchers, frequently due to the amount and heterogeneity of text to be processed. To address this challenge, the industry is continuously developing highly efficient tools and creating more flexible engineering solutions. This work presents the integration between industry data engineering solutions for efficient data processing and academic systems developed for Named Entity Recognition (LasigeUnicage\_NER) and Relation Extraction (BiOnt). Our design reflects an integration of those components with external knowledge in the form of additional training data from other datasets and biomedical ontologies. We used this pipeline in the 2022 LitCoin NLP Challenge, where our team LasigeUnicage was awarded the 7th Prize out of approximately 200 participating teams, reflecting a successful collaboration between the academia (LASIGE) and the industry (Unicage). The software supporting this work is available at \url{https://github.com/lasigeBioTM/Litcoin-Lasige_Unicage}.

membership infer

federate

Title: Balancing Accuracy and Training Time in Federated Learning for Violence Detection in Surveillance Videos: A Study of Neural Network Architectures. (arXiv:2308.05106v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05106
• Code URL: null
• Copy Paste: [[2308.05106]] Balancing Accuracy and Training Time in Federated Learning for Violence Detection in Surveillance Videos: A Study of Neural Network Architectures(http://arxiv.org/abs/2308.05106)
• Summary:

  This paper presents an investigation into machine learning techniques for violence detection in videos and their adaptation to a federated learning context. The study includes experiments with spatio-temporal features extracted from benchmark video datasets, comparison of different methods, and proposal of a modified version of the "Flow-Gated" architecture called "Diff-Gated." Additionally, various machine learning techniques, including super-convergence and transfer learning, are explored, and a method for adapting centralized datasets to a federated learning context is developed. The research achieves better accuracy results compared to state-of-the-art models by training the best violence detection model in a federated learning context.

Title: Federated Online/Offline Remote Data Inspection for Distributed Edge Computing. (arXiv:2308.05198v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2308.05198
• Code URL: null
• Copy Paste: [[2308.05198]] Federated Online/Offline Remote Data Inspection for Distributed Edge Computing(http://arxiv.org/abs/2308.05198)
• Summary:

  In edge computing environments, app vendors can cache their data to be shared with their users in many geographically distributed edge servers. However, the cached data is particularly vulnerable to several intentional attacks or unintentional events. Given the limited resources of edge servers and prohibitive storage costs incurred by app vendors, designing an efficient approach to inspect and maintain the data over tremendous edge servers is a critical issue. To tackle the problem, we design a novel data inspection approach, named ${\text{O}^2\text{DI}}$, that provides the following services: i) using ${\text{O}^2\text{DI}}$, app vendors can inspect the data cached in edge servers without having the original data, which reduces the incurred I/O and storage overhead significantly; ii) computational operations conducted by both edge servers and app vendors are highly efficient because of a novel online/offline technique; iii) many data files cached in different edge servers can be verified quickly and at once by using a novel batch verification method; iv) corrupted data in edge servers can be localized and repaired efficiently. We analyze the security and performance of ${\text{O}^2\text{DI}}$. We see that it is secure in the random oracle model, much faster, and more cost-effective compared to state-of-the-art approaches.

fair

Title: TrainFors: A Large Benchmark Training Dataset for Image Manipulation Detection and Localization. (arXiv:2308.05264v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05264
• Code URL: null
• Copy Paste: [[2308.05264]] TrainFors: A Large Benchmark Training Dataset for Image Manipulation Detection and Localization(http://arxiv.org/abs/2308.05264)
• Summary:

  The evaluation datasets and metrics for image manipulation detection and localization (IMDL) research have been standardized. But the training dataset for such a task is still nonstandard. Previous researchers have used unconventional and deviating datasets to train neural networks for detecting image forgeries and localizing pixel maps of manipulated regions. For a fair comparison, the training set, test set, and evaluation metrics should be persistent. Hence, comparing the existing methods may not seem fair as the results depend heavily on the training datasets as well as the model architecture. Moreover, none of the previous works release the synthetic training dataset used for the IMDL task. We propose a standardized benchmark training dataset for image splicing, copy-move forgery, removal forgery, and image enhancement forgery. Furthermore, we identify the problems with the existing IMDL datasets and propose the required modifications. We also train the state-of-the-art IMDL methods on our proposed TrainFors1 dataset for a fair evaluation and report the actual performance of these methods under similar conditions.

Title: Is there progress in activity progress prediction?. (arXiv:2308.05533v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05533
• Code URL: https://github.com/frans-db/progress-prediction
• Copy Paste: [[2308.05533]] Is there progress in activity progress prediction?(http://arxiv.org/abs/2308.05533)
• Summary:

  Activity progress prediction aims to estimate what percentage of an activity has been completed. Currently this is done with machine learning approaches, trained and evaluated on complicated and realistic video datasets. The videos in these datasets vary drastically in length and appearance. And some of the activities have unanticipated developments, making activity progression difficult to estimate. In this work, we examine the results obtained by existing progress prediction methods on these datasets. We find that current progress prediction methods seem not to extract useful visual information for the progress prediction task. Therefore, these methods fail to exceed simple frame-counting baselines. We design a precisely controlled dataset for activity progress prediction and on this synthetic dataset we show that the considered methods can make use of the visual information, when this directly relates to the progress prediction. We conclude that the progress prediction task is ill-posed on the currently used real-world datasets. Moreover, to fairly measure activity progression we advise to consider a, simple but effective, frame-counting baseline.

interpretability

Title: Can Attention Be Used to Explain EHR-Based Mortality Prediction Tasks: A Case Study on Hemorrhagic Stroke. (arXiv:2308.05110v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.05110
• Code URL: null
• Copy Paste: [[2308.05110]] Can Attention Be Used to Explain EHR-Based Mortality Prediction Tasks: A Case Study on Hemorrhagic Stroke(http://arxiv.org/abs/2308.05110)
• Summary:

  Stroke is a significant cause of mortality and morbidity, necessitating early predictive strategies to minimize risks. Traditional methods for evaluating patients, such as Acute Physiology and Chronic Health Evaluation (APACHE II, IV) and Simplified Acute Physiology Score III (SAPS III), have limited accuracy and interpretability. This paper proposes a novel approach: an interpretable, attention-based transformer model for early stroke mortality prediction. This model seeks to address the limitations of previous predictive models, providing both interpretability (providing clear, understandable explanations of the model) and fidelity (giving a truthful explanation of the model's dynamics from input to output). Furthermore, the study explores and compares fidelity and interpretability scores using Shapley values and attention-based scores to improve model explainability. The research objectives include designing an interpretable attention-based transformer model, evaluating its performance compared to existing models, and providing feature importance derived from the model.

explainability

watermark

diffusion

Title: Masked Diffusion as Self-supervised Representation Learner. (arXiv:2308.05695v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05695
• Code URL: null
• Copy Paste: [[2308.05695]] Masked Diffusion as Self-supervised Representation Learner(http://arxiv.org/abs/2308.05695)
• Summary:

  Denoising diffusion probabilistic models have recently demonstrated state-of-the-art generative performance and been used as strong pixel-level representation learners. This paper decomposes the interrelation between the generative capability and representation learning ability inherent in diffusion models. We present masked diffusion model (MDM), a scalable self-supervised representation learner that substitutes the conventional additive Gaussian noise of traditional diffusion with a masking mechanism. Our proposed approach convincingly surpasses prior benchmarks, demonstrating remarkable advancements in both medical and natural image semantic segmentation tasks, particularly within the context of few-shot scenario.

Title: PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solvers. (arXiv:2308.05732v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.05732
• Code URL: null
• Copy Paste: [[2308.05732]] PDE-Refiner: Achieving Accurate Long Rollouts with Neural PDE Solvers(http://arxiv.org/abs/2308.05732)
• Summary:

  Time-dependent partial differential equations (PDEs) are ubiquitous in science and engineering. Recently, mostly due to the high computational cost of traditional solution techniques, deep neural network based surrogates have gained increased interest. The practical utility of such neural PDE solvers relies on their ability to provide accurate, stable predictions over long time horizons, which is a notoriously hard problem. In this work, we present a large-scale analysis of common temporal rollout strategies, identifying the neglect of non-dominant spatial frequency information, often associated with high frequencies in PDE solutions, as the primary pitfall limiting stable, accurate rollout performance. Based on these insights, we draw inspiration from recent advances in diffusion models to introduce PDE-Refiner; a novel model class that enables more accurate modeling of all frequency components via a multistep refinement process. We validate PDE-Refiner on challenging benchmarks of complex fluid dynamics, demonstrating stable and accurate rollouts that consistently outperform state-of-the-art models, including neural, numerical, and hybrid neural-numerical architectures. We further demonstrate that PDE-Refiner greatly enhances data efficiency, since the denoising objective implicitly induces a novel form of spectral data augmentation. Finally, PDE-Refiner's connection to diffusion models enables an accurate and efficient assessment of the model's predictive uncertainty, allowing us to estimate when the surrogate becomes inaccurate.

noise learning

data-free

Title: NUPES : Non-Uniform Post-Training Quantization via Power Exponent Search. (arXiv:2308.05600v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.05600
• Code URL: null
• Copy Paste: [[2308.05600]] NUPES : Non-Uniform Post-Training Quantization via Power Exponent Search(http://arxiv.org/abs/2308.05600)
• Summary:

  Deep neural network (DNN) deployment has been confined to larger hardware devices due to their expensive computational requirements. This challenge has recently reached another scale with the emergence of large language models (LLMs). In order to reduce both their memory footprint and latency, a promising technique is quantization. It consists in converting floating point representations to low bit-width fixed point representations, usually by assuming a uniform mapping onto a regular grid. This process, referred to in the literature as uniform quantization, may however be ill-suited as most DNN weights and activations follow a bell-shaped distribution. This is even worse on LLMs whose weight distributions are known to exhibit large, high impact, outlier values. In this work, we propose an improvement over the most commonly adopted way to tackle this limitation in deep learning models quantization, namely, non-uniform quantization. NUPES leverages automorphisms to preserve the scalar multiplications. Such transformations are derived from power functions. However, the optimization of the exponent parameter and weight values remains a challenging and novel problem which could not be solved with previous post training optimization techniques which only learn to round up or down weight values in order to preserve the predictive function. We circumvent this limitation with a new paradigm: learning new quantized weights over the entire quantized space. Similarly, we enable the optimization of the power exponent, i.e. the optimization of the quantization operator itself during training by alleviating all the numerical instabilities. The resulting predictive function is compatible with integer-only low-bit inference. We show the ability of the method to achieve state-of-the-art compression rates in both, data-free and data-driven configurations.

transformer

Title: Informative Scene Graph Generation via Debiasing. (arXiv:2308.05286v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05286
• Code URL: null
• Copy Paste: [[2308.05286]] Informative Scene Graph Generation via Debiasing(http://arxiv.org/abs/2308.05286)
• Summary:

  Scene graph generation aims to detect visual relationship triplets, (subject, predicate, object). Due to biases in data, current models tend to predict common predicates, e.g. "on" and "at", instead of informative ones, e.g. "standing on" and "looking at". This tendency results in the loss of precise information and overall performance. If a model only uses "stone on road" rather than "stone blocking road" to describe an image, it may be a grave misunderstanding. We argue that this phenomenon is caused by two imbalances: semantic space level imbalance and training sample level imbalance. For this problem, we propose DB-SGG, an effective framework based on debiasing but not the conventional distribution fitting. It integrates two components: Semantic Debiasing (SD) and Balanced Predicate Learning (BPL), for these imbalances. SD utilizes a confusion matrix and a bipartite graph to construct predicate relationships. BPL adopts a random undersampling strategy and an ambiguity removing strategy to focus on informative predicates. Benefiting from the model-agnostic process, our method can be easily applied to SGG models and outperforms Transformer by 136.3%, 119.5%, and 122.6% on mR@20 at three SGG sub-tasks on the SGG-VG dataset. Our method is further verified on another complex SGG dataset (SGG-GQA) and two downstream tasks (sentence-to-graph retrieval and image captioning).

Title: Double-chain Constraints for 3D Human Pose Estimation in Images and Videos. (arXiv:2308.05298v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05298
• Code URL: https://github.com/KHB1698/DC-GCT
• Copy Paste: [[2308.05298]] Double-chain Constraints for 3D Human Pose Estimation in Images and Videos(http://arxiv.org/abs/2308.05298)
• Summary:

  Reconstructing 3D poses from 2D poses lacking depth information is particularly challenging due to the complexity and diversity of human motion. The key is to effectively model the spatial constraints between joints to leverage their inherent dependencies. Thus, we propose a novel model, called Double-chain Graph Convolutional Transformer (DC-GCT), to constrain the pose through a double-chain design consisting of local-to-global and global-to-local chains to obtain a complex representation more suitable for the current human pose. Specifically, we combine the advantages of GCN and Transformer and design a Local Constraint Module (LCM) based on GCN and a Global Constraint Module (GCM) based on self-attention mechanism as well as a Feature Interaction Module (FIM). The proposed method fully captures the multi-level dependencies between human body joints to optimize the modeling capability of the model. Moreover, we propose a method to use temporal information into the single-frame model by guiding the video sequence embedding through the joint embedding of the target frame, with negligible increase in computational cost. Experimental results demonstrate that DC-GCT achieves state-of-the-art performance on two challenging datasets (Human3.6M and MPI-INF-3DHP). Notably, our model achieves state-of-the-art performance on all action categories in the Human3.6M dataset using detected 2D poses from CPN, and our code is available at: https://github.com/KHB1698/DC-GCT.

Title: Interaction-aware Joint Attention Estimation Using People Attributes. (arXiv:2308.05382v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05382
• Code URL: https://github.com/chihina/pjae
• Copy Paste: [[2308.05382]] Interaction-aware Joint Attention Estimation Using People Attributes(http://arxiv.org/abs/2308.05382)
• Summary:

  This paper proposes joint attention estimation in a single image. Different from related work in which only the gaze-related attributes of people are independently employed, (I) their locations and actions are also employed as contextual cues for weighting their attributes, and (ii) interactions among all of these attributes are explicitly modeled in our method. For the interaction modeling, we propose a novel Transformer-based attention network to encode joint attention as low-dimensional features. We introduce a specialized MLP head with positional embedding to the Transformer so that it predicts pixelwise confidence of joint attention for generating the confidence heatmap. This pixelwise prediction improves the heatmap accuracy by avoiding the ill-posed problem in which the high-dimensional heatmap is predicted from the low-dimensional features. The estimated joint attention is further improved by being integrated with general image-based attention estimation. Our method outperforms SOTA methods quantitatively in comparative experiments. Code: https://anonymous.4open.science/r/anonymized_codes-ECA4.

Title: Category Feature Transformer for Semantic Segmentation. (arXiv:2308.05581v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05581
• Code URL: null
• Copy Paste: [[2308.05581]] Category Feature Transformer for Semantic Segmentation(http://arxiv.org/abs/2308.05581)
• Summary:

  Aggregation of multi-stage features has been revealed to play a significant role in semantic segmentation. Unlike previous methods employing point-wise summation or concatenation for feature aggregation, this study proposes the Category Feature Transformer (CFT) that explores the flow of category embedding and transformation among multi-stage features through the prevalent multi-head attention mechanism. CFT learns unified feature embeddings for individual semantic categories from high-level features during each aggregation process and dynamically broadcasts them to high-resolution features. Integrating the proposed CFT into a typical feature pyramid structure exhibits superior performance over a broad range of backbone networks. We conduct extensive experiments on popular semantic segmentation benchmarks. Specifically, the proposed CFT obtains a compelling 55.1% mIoU with greatly reduced model parameters and computations on the challenging ADE20K dataset.

Title: Object Goal Navigation with Recursive Implicit Maps. (arXiv:2308.05602v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05602
• Code URL: null
• Copy Paste: [[2308.05602]] Object Goal Navigation with Recursive Implicit Maps(http://arxiv.org/abs/2308.05602)
• Summary:

  Object goal navigation aims to navigate an agent to locations of a given object category in unseen environments. Classical methods explicitly build maps of environments and require extensive engineering while lacking semantic information for object-oriented exploration. On the other hand, end-to-end learning methods alleviate manual map design and predict actions using implicit representations. Such methods, however, lack an explicit notion of geometry and may have limited ability to encode navigation history. In this work, we propose an implicit spatial map for object goal navigation. Our implicit map is recursively updated with new observations at each step using a transformer. To encourage spatial reasoning, we introduce auxiliary tasks and train our model to reconstruct explicit maps as well as to predict visual features, semantic labels and actions. Our method significantly outperforms the state of the art on the challenging MP3D dataset and generalizes well to the HM3D dataset. We successfully deploy our model on a real robot and achieve encouraging object goal navigation results in real scenes using only a few real-world demonstrations. Code, trained models and videos are available at \url{https://www.di.ens.fr/willow/research/onav_rim/}.

Title: IIHT: Medical Report Generation with Image-to-Indicator Hierarchical Transformer. (arXiv:2308.05633v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05633
• Code URL: null
• Copy Paste: [[2308.05633]] IIHT: Medical Report Generation with Image-to-Indicator Hierarchical Transformer(http://arxiv.org/abs/2308.05633)
• Summary:

  Automated medical report generation has become increasingly important in medical analysis. It can produce computer-aided diagnosis descriptions and thus significantly alleviate the doctors' work. Inspired by the huge success of neural machine translation and image captioning, various deep learning methods have been proposed for medical report generation. However, due to the inherent properties of medical data, including data imbalance and the length and correlation between report sequences, the generated reports by existing methods may exhibit linguistic fluency but lack adequate clinical accuracy. In this work, we propose an image-to-indicator hierarchical transformer (IIHT) framework for medical report generation. It consists of three modules, i.e., a classifier module, an indicator expansion module and a generator module. The classifier module first extracts image features from the input medical images and produces disease-related indicators with their corresponding states. The disease-related indicators are subsequently utilised as input for the indicator expansion module, incorporating the "data-text-data" strategy. The transformer-based generator then leverages these extracted features along with image features as auxiliary information to generate final reports. Furthermore, the proposed IIHT method is feasible for radiologists to modify disease indicators in real-world scenarios and integrate the operations into the indicator expansion module for fluent and accurate medical report generation. Extensive experiments and comparisons with state-of-the-art methods under various evaluation metrics demonstrate the great performance of the proposed method.

Title: 2D3D-MATR: 2D-3D Matching Transformer for Detection-free Registration between Images and Point Clouds. (arXiv:2308.05667v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05667
• Code URL: null
• Copy Paste: [[2308.05667]] 2D3D-MATR: 2D-3D Matching Transformer for Detection-free Registration between Images and Point Clouds(http://arxiv.org/abs/2308.05667)
• Summary:

  The commonly adopted detect-then-match approach to registration finds difficulties in the cross-modality cases due to the incompatible keypoint detection and inconsistent feature description. We propose, 2D3D-MATR, a detection-free method for accurate and robust registration between images and point clouds. Our method adopts a coarse-to-fine pipeline where it first computes coarse correspondences between downsampled patches of the input image and the point cloud and then extends them to form dense correspondences between pixels and points within the patch region. The coarse-level patch matching is based on transformer which jointly learns global contextual constraints with self-attention and cross-modality correlations with cross-attention. To resolve the scale ambiguity in patch matching, we construct a multi-scale pyramid for each image patch and learn to find for each point patch the best matching image patch at a proper resolution level. Extensive experiments on two public benchmarks demonstrate that 2D3D-MATR outperforms the previous state-of-the-art P2-Net by around $20$ percentage points on inlier ratio and over $10$ points on registration recall. Our code and models are available at \url{https://github.com/minhaolee/2D3DMATR}.

Title: Deformable Mixer Transformer with Gating for Multi-Task Learning of Dense Prediction. (arXiv:2308.05721v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05721
• Code URL: https://github.com/yangyangxu0/demtg
• Copy Paste: [[2308.05721]] Deformable Mixer Transformer with Gating for Multi-Task Learning of Dense Prediction(http://arxiv.org/abs/2308.05721)
• Summary:

  CNNs and Transformers have their own advantages and both have been widely used for dense prediction in multi-task learning (MTL). Most of the current studies on MTL solely rely on CNN or Transformer. In this work, we present a novel MTL model by combining both merits of deformable CNN and query-based Transformer with shared gating for multi-task learning of dense prediction. This combination may offer a simple and efficient solution owing to its powerful and flexible task-specific learning and advantages of lower cost, less complexity and smaller parameters than the traditional MTL methods. We introduce deformable mixer Transformer with gating (DeMTG), a simple and effective encoder-decoder architecture up-to-date that incorporates the convolution and attention mechanism in a unified network for MTL. It is exquisitely designed to use advantages of each block, and provide deformable and comprehensive features for all tasks from local and global perspective. First, the deformable mixer encoder contains two types of operators: the channel-aware mixing operator leveraged to allow communication among different channels, and the spatial-aware deformable operator with deformable convolution applied to efficiently sample more informative spatial locations. Second, the task-aware gating transformer decoder is used to perform the task-specific predictions, in which task interaction block integrated with self-attention is applied to capture task interaction features, and the task query block integrated with gating attention is leveraged to select corresponding task-specific features. Further, the experiment results demonstrate that the proposed DeMTG uses fewer GFLOPs and significantly outperforms current Transformer-based and CNN-based competitive models on a variety of metrics on three dense prediction datasets. Our code and models are available at https://github.com/yangyangxu0/DeMTG.

Title: Decoding Layer Saliency in Language Transformers. (arXiv:2308.05219v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.05219
• Code URL: null
• Copy Paste: [[2308.05219]] Decoding Layer Saliency in Language Transformers(http://arxiv.org/abs/2308.05219)
• Summary:

  In this paper, we introduce a strategy for identifying textual saliency in large-scale language models applied to classification tasks. In visual networks where saliency is more well-studied, saliency is naturally localized through the convolutional layers of the network; however, the same is not true in modern transformer-stack networks used to process natural language. We adapt gradient-based saliency methods for these networks, propose a method for evaluating the degree of semantic coherence of each layer, and demonstrate consistent improvement over numerous other methods for textual saliency on multiple benchmark classification datasets. Our approach requires no additional training or access to labelled data, and is comparatively very computationally efficient.

Title: Exploring Machine Learning and Transformer-based Approaches for Deceptive Text Classification: A Comparative Analysis. (arXiv:2308.05476v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.05476
• Code URL: null
• Copy Paste: [[2308.05476]] Exploring Machine Learning and Transformer-based Approaches for Deceptive Text Classification: A Comparative Analysis(http://arxiv.org/abs/2308.05476)
• Summary:

  Deceptive text classification is a critical task in natural language processing that aims to identify deceptive or fraudulent content. This study presents a comparative analysis of machine learning and transformer-based approaches for deceptive text classification. We investigate the effectiveness of traditional machine learning algorithms and state-of-the-art transformer models, such as BERT, XLNET, DistilBERT, and RoBERTa, in detecting deceptive text. A labeled dataset consisting of deceptive and non-deceptive texts is used for training and evaluation purposes. Through extensive experimentation, we compare the performance metrics, including accuracy, precision, recall, and F1 score, of the different approaches. The results of this study shed light on the strengths and limitations of machine learning and transformer-based methods for deceptive text classification, enabling researchers and practitioners to make informed decisions when dealing with deceptive content

Title: Bringing order into the realm of Transformer-based language models for artificial intelligence and law. (arXiv:2308.05502v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.05502
• Code URL: null
• Copy Paste: [[2308.05502]] Bringing order into the realm of Transformer-based language models for artificial intelligence and law(http://arxiv.org/abs/2308.05502)
• Summary:

  Transformer-based language models (TLMs) have widely been recognized to be a cutting-edge technology for the successful development of deep-learning-based solutions to problems and applications that require natural language processing and understanding. Like for other textual domains, TLMs have indeed pushed the state-of-the-art of AI approaches for many tasks of interest in the legal domain. Despite the first Transformer model being proposed about six years ago, there has been a rapid progress of this technology at an unprecedented rate, whereby BERT and related models represent a major reference, also in the legal domain. This article provides the first systematic overview of TLM-based methods for AI-driven problems and tasks in the legal sphere. A major goal is to highlight research advances in this field so as to understand, on the one hand, how the Transformers have contributed to the success of AI in supporting legal processes, and on the other hand, what are the current limitations and opportunities for further research development.

Title: AST-MHSA : Code Summarization using Multi-Head Self-Attention. (arXiv:2308.05646v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.05646
• Code URL: null
• Copy Paste: [[2308.05646]] AST-MHSA : Code Summarization using Multi-Head Self-Attention(http://arxiv.org/abs/2308.05646)
• Summary:

  Code summarization aims to generate concise natural language descriptions for source code. The prevailing approaches adopt transformer-based encoder-decoder architectures, where the Abstract Syntax Tree (AST) of the source code is utilized for encoding structural information. However, ASTs are much longer than the corresponding source code, and existing methods ignore this size constraint by directly feeding the entire linearized AST into the encoders. This simplistic approach makes it challenging to extract truly valuable dependency relations from the overlong input sequence and leads to significant computational overhead due to self-attention applied to all nodes in the AST.

  To address this issue effectively and efficiently, we present a model, AST-MHSA that uses multi-head attention to extract the important semantic information from the AST. The model consists of two main components: an encoder and a decoder. The encoder takes as input the abstract syntax tree (AST) of the code and generates a sequence of hidden states. The decoder then takes these hidden states as input and generates a natural language summary of the code.

  The multi-head attention mechanism allows the model to learn different representations of the input code, which can be combined to generate a more comprehensive summary. The model is trained on a dataset of code and summaries, and the parameters of the model are optimized to minimize the loss between the generated summaries and the ground-truth summaries.

generative

Title: Hierarchical Representations for Spatio-Temporal Visual Attention Modeling and Understanding. (arXiv:2308.05189v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05189
• Code URL: null
• Copy Paste: [[2308.05189]] Hierarchical Representations for Spatio-Temporal Visual Attention Modeling and Understanding(http://arxiv.org/abs/2308.05189)
• Summary:

  This PhD. Thesis concerns the study and development of hierarchical representations for spatio-temporal visual attention modeling and understanding in video sequences. More specifically, we propose two computational models for visual attention. First, we present a generative probabilistic model for context-aware visual attention modeling and understanding. Secondly, we develop a deep network architecture for visual attention modeling, which first estimates top-down spatio-temporal visual attention, and ultimately serves for modeling attention in the temporal domain.

Title: Vector quantization loss analysis in VQGANs: a single-GPU ablation study for image-to-image synthesis. (arXiv:2308.05242v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05242
• Code URL: https://github.com/luv91/vqgan_project
• Copy Paste: [[2308.05242]] Vector quantization loss analysis in VQGANs: a single-GPU ablation study for image-to-image synthesis(http://arxiv.org/abs/2308.05242)
• Summary:

  This study performs an ablation analysis of Vector Quantized Generative Adversarial Networks (VQGANs), concentrating on image-to-image synthesis utilizing a single NVIDIA A100 GPU. The current work explores the nuanced effects of varying critical parameters including the number of epochs, image count, and attributes of codebook vectors and latent dimensions, specifically within the constraint of limited resources. Notably, our focus is pinpointed on the vector quantization loss, keeping other hyperparameters and loss components (GAN loss) fixed. This was done to delve into a deeper understanding of the discrete latent space, and to explore how varying its size affects the reconstruction. Though, our results do not surpass the existing benchmarks, however, our findings shed significant light on VQGAN's behaviour for a smaller dataset, particularly concerning artifacts, codebook size optimization, and comparative analysis with Principal Component Analysis (PCA). The study also uncovers the promising direction by introducing 2D positional encodings, revealing a marked reduction in artifacts and insights into balancing clarity and overfitting.

Title: Shadow Datasets, New challenging datasets for Causal Representation Learning. (arXiv:2308.05707v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.05707
• Code URL: https://github.com/Jiagengzhu/Shadow-dataset-for-crl
• Copy Paste: [[2308.05707]] Shadow Datasets, New challenging datasets for Causal Representation Learning(http://arxiv.org/abs/2308.05707)
• Summary:

  Discovering causal relations among semantic factors is an emergent topic in representation learning. Most causal representation learning (CRL) methods are fully supervised, which is impractical due to costly labeling. To resolve this restriction, weakly supervised CRL methods were introduced. To evaluate CRL performance, four existing datasets, Pendulum, Flow, CelebA(BEARD) and CelebA(SMILE), are utilized. However, existing CRL datasets are limited to simple graphs with few generative factors. Thus we propose two new datasets with a larger number of diverse generative factors and more sophisticated causal graphs. In addition, current real datasets, CelebA(BEARD) and CelebA(SMILE), the originally proposed causal graphs are not aligned with the dataset distributions. Thus, we propose modifications to them.

Title: Neural Progressive Meshes. (arXiv:2308.05741v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05741
• Code URL: null
• Copy Paste: [[2308.05741]] Neural Progressive Meshes(http://arxiv.org/abs/2308.05741)
• Summary:

  The recent proliferation of 3D content that can be consumed on hand-held devices necessitates efficient tools for transmitting large geometric data, e.g., 3D meshes, over the Internet. Detailed high-resolution assets can pose a challenge to storage as well as transmission bandwidth, and level-of-detail techniques are often used to transmit an asset using an appropriate bandwidth budget. It is especially desirable for these methods to transmit data progressively, improving the quality of the geometry with more data. Our key insight is that the geometric details of 3D meshes often exhibit similar local patterns even across different shapes, and thus can be effectively represented with a shared learned generative space. We learn this space using a subdivision-based encoder-decoder architecture trained in advance on a large collection of surfaces. We further observe that additional residual features can be transmitted progressively between intermediate levels of subdivision that enable the client to control the tradeoff between bandwidth cost and quality of reconstruction, providing a neural progressive mesh representation. We evaluate our method on a diverse set of complex 3D shapes and demonstrate that it outperforms baselines in terms of compression ratio and reconstruction quality.

Title: Classification of Human- and AI-Generated Texts: Investigating Features for ChatGPT. (arXiv:2308.05341v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.05341
• Code URL: null
• Copy Paste: [[2308.05341]] Classification of Human- and AI-Generated Texts: Investigating Features for ChatGPT(http://arxiv.org/abs/2308.05341)
• Summary:

  Recently, generative AIs like ChatGPT have become available to the wide public. These tools can for instance be used by students to generate essays or whole theses. But how does a teacher know whether a text is written by a student or an AI? In our work, we explore traditional and new features to (1) detect text generated by AI from scratch and (2) text rephrased by AI. Since we found that classification is more difficult when the AI has been instructed to create the text in a way that a human would not recognize that it was generated by an AI, we also investigate this more advanced case. For our experiments, we produced a new text corpus covering 10 school topics. Our best systems to classify basic and advanced human-generated/AI-generated texts have F1-scores of over 96%. Our best systems for classifying basic and advanced human-generated/AI-rephrased texts have F1-scores of more than 78%. The systems use a combination of perplexity, semantic, list lookup, error-based, readability, AI feedback, and text vector features. Our results show that the new features substantially help to improve the performance of many classifiers. Our best basic text rephrasing detection system even outperforms GPTZero by 183.8% relative in F1-score.

Title: $\mathcal{G}^2Pxy$: Generative Open-Set Node Classification on Graphs with Proxy Unknowns. (arXiv:2308.05463v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.05463
• Code URL: null
• Copy Paste: [[2308.05463]] $\mathcal{G}^2Pxy$: Generative Open-Set Node Classification on Graphs with Proxy Unknowns(http://arxiv.org/abs/2308.05463)
• Summary:

  Node classification is the task of predicting the labels of unlabeled nodes in a graph. State-of-the-art methods based on graph neural networks achieve excellent performance when all labels are available during training. But in real-life, models are often applied on data with new classes, which can lead to massive misclassification and thus significantly degrade performance. Hence, developing open-set classification methods is crucial to determine if a given sample belongs to a known class. Existing methods for open-set node classification generally use transductive learning with part or all of the features of real unseen class nodes to help with open-set classification. In this paper, we propose a novel generative open-set node classification method, i.e. $\mathcal{G}^2Pxy$, which follows a stricter inductive learning setting where no information about unknown classes is available during training and validation. Two kinds of proxy unknown nodes, inter-class unknown proxies and external unknown proxies are generated via mixup to efficiently anticipate the distribution of novel classes. Using the generated proxies, a closed-set classifier can be transformed into an open-set one, by augmenting it with an extra proxy classifier. Under the constraints of both cross entropy loss and complement entropy loss, $\mathcal{G}^2Pxy$ achieves superior effectiveness for unknown class detection and known class classification, which is validated by experiments on benchmark graph datasets. Moreover, $\mathcal{G}^2Pxy$ does not have specific requirement on the GNN architecture and shows good generalizations.

large language model

Title: Metacognitive Prompting Improves Understanding in Large Language Models. (arXiv:2308.05342v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.05342
• Code URL: null
• Copy Paste: [[2308.05342]] Metacognitive Prompting Improves Understanding in Large Language Models(http://arxiv.org/abs/2308.05342)
• Summary:

  In Large Language Models (LLMs), there have been consistent advancements in task-specific performance, largely influenced by effective prompt design. While recent research on prompting has enhanced the reasoning capabilities of LLMs, a gap remains in further improving their understanding abilities. In this study, we introduce metacognitive prompting (MP), a strategy inspired by human introspective reasoning processes. Using MP, LLMs undergo a systematic series of structured, self-aware evaluations, drawing on both their vast inherent knowledge and new insights. Our experiments involve five prevalent LLMs: Llama2, Vicuna, PaLM, GPT-3.5, and GPT-4, all of which span various general natural language understanding (NLU) tasks from the GLUE and SuperGLUE benchmarks. Results indicate that, although GPT-4 consistently excels in most tasks, PaLM, when equipped with MP, approaches its performance level. Furthermore, across models and datasets, MP consistently outperforms existing prompting methods, including standard and chain-of-thought prompting. This study underscores the potential to amplify the understanding abilities of LLMs and highlights the benefits of mirroring human introspective reasoning in NLU tasks.

Title: WeaverBird: Empowering Financial Decision-Making with Large Language Model, Knowledge Base, and Search Engine. (arXiv:2308.05361v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.05361
• Code URL: null
• Copy Paste: [[2308.05361]] WeaverBird: Empowering Financial Decision-Making with Large Language Model, Knowledge Base, and Search Engine(http://arxiv.org/abs/2308.05361)
• Summary:

  We present WeaverBird, an intelligent dialogue system designed specifically for the finance domain. Our system harnesses a large language model of GPT architecture that has been tuned using extensive corpora of finance-related text. As a result, our system possesses the capability to understand complex financial queries, such as "How should I manage my investments during inflation?", and provide informed responses. Furthermore, our system incorporates a local knowledge base and a search engine to retrieve relevant information. The final responses are conditioned on the search results and include proper citations to the sources, thus enjoying an enhanced credibility. Through a range of finance-related questions, we have demonstrated the superior performance of our system compared to other models. To experience our system firsthand, users can interact with our live demo at https://weaverbird.ttic.edu, as well as watch our 2-min video illustration at https://www.youtube.com/watch?v=yofgeqnlrMc.

Title: You Only Prompt Once: On the Capabilities of Prompt Learning on Large Language Models to Tackle Toxic Content. (arXiv:2308.05596v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.05596
• Code URL: null
• Copy Paste: [[2308.05596]] You Only Prompt Once: On the Capabilities of Prompt Learning on Large Language Models to Tackle Toxic Content(http://arxiv.org/abs/2308.05596)
• Summary:

  The spread of toxic content online is an important problem that has adverse effects on user experience online and in our society at large. Motivated by the importance and impact of the problem, research focuses on developing solutions to detect toxic content, usually leveraging machine learning (ML) models trained on human-annotated datasets. While these efforts are important, these models usually do not generalize well and they can not cope with new trends (e.g., the emergence of new toxic terms). Currently, we are witnessing a shift in the approach to tackling societal issues online, particularly leveraging large language models (LLMs) like GPT-3 or T5 that are trained on vast corpora and have strong generalizability. In this work, we investigate how we can use LLMs and prompt learning to tackle the problem of toxic content, particularly focusing on three tasks; 1) Toxicity Classification, 2) Toxic Span Detection, and 3) Detoxification. We perform an extensive evaluation over five model architectures and eight datasets demonstrating that LLMs with prompt learning can achieve similar or even better performance compared to models trained on these specific tasks. We find that prompt learning achieves around 10\% improvement in the toxicity classification task compared to the baselines, while for the toxic span detection task we find better performance to the best baseline (0.643 vs. 0.640 in terms of $F_1$-score). Finally, for the detoxification task, we find that prompt learning can successfully reduce the average toxicity score (from 0.775 to 0.213) while preserving semantic meaning.

Title: A Preliminary Study of the Intrinsic Relationship between Complexity and Alignment. (arXiv:2308.05696v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2308.05696
• Code URL: https://github.com/alibabaresearch/damo-convai
• Copy Paste: [[2308.05696]] A Preliminary Study of the Intrinsic Relationship between Complexity and Alignment(http://arxiv.org/abs/2308.05696)
• Summary:

  Training large language models (LLMs) with open-domain instruction data has yielded remarkable success in aligning to end tasks and user preferences. Extensive research has highlighted that enhancing the quality and diversity of instruction data consistently improves performance. However, the impact of data complexity, as a crucial metric, remains relatively unexplored in three aspects: (1) scaling law, where the sustainability of performance improvements with increasing complexity is uncertain, (2) additional tokens, whether the improvement brought by complexity comes from introducing more training tokens, and (3) curriculum tuning, where the potential advantages of incorporating instructions ranging from easy to difficult are not yet fully understood. In this paper, we propose \textit{tree-instruct} to systematically enhance the complexity of instruction data in a controllable manner. This approach adds a specified number of nodes into the instruction semantic tree, yielding new instruction data based on the modified tree. By adjusting the number of added nodes, we can control the difficulty level in the modified instruction data. Our preliminary experiments reveal the following insights: (1) Increasing complexity consistently leads to sustained performance improvements. For instance, using 1,000 instruction data and 10 nodes resulted in a substantial 24\% increase in win rate. (2) Under the same token budget, a few complex instructions outperform diverse yet simple instructions. (3) Curriculum instruction tuning might not yield the anticipated results; focusing on increasing complexity appears to be the key.

Title: RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model. (arXiv:2308.05345v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2308.05345
• Code URL: null
• Copy Paste: [[2308.05345]] RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model(http://arxiv.org/abs/2308.05345)
• Summary:

  Inspired by the recent success of large language models (LLMs) like ChatGPT, researchers start to explore the adoption of LLMs for agile hardware design, such as generating design RTL based on natural-language instructions. However, in existing works, their target designs are all relatively simple and in a small scale, and proposed by the authors themselves, making a fair comparison among different LLM solutions challenging. In addition, many prior works only focus on the design correctness, without evaluating the design qualities of generated design RTL. In this work, we propose an open-source benchmark named RTLLM, for generating design RTL with natural language instructions. To systematically evaluate the auto-generated design RTL, we summarized three progressive goals, named syntax goal, functionality goal, and design quality goal. This benchmark can automatically provide a quantitative evaluation of any given LLM-based solution. Furthermore, we propose an easy-to-use yet surprisingly effective prompt engineering technique named self-planning, which proves to significantly boost the performance of GPT-3.5 in our proposed benchmark.

segmentation

Title: Discrepancy-based Active Learning for Weakly Supervised Bleeding Segmentation in Wireless Capsule Endoscopy Images. (arXiv:2308.05137v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05137
• Code URL: null
• Copy Paste: [[2308.05137]] Discrepancy-based Active Learning for Weakly Supervised Bleeding Segmentation in Wireless Capsule Endoscopy Images(http://arxiv.org/abs/2308.05137)
• Summary:

  Weakly supervised methods, such as class activation maps (CAM) based, have been applied to achieve bleeding segmentation with low annotation efforts in Wireless Capsule Endoscopy (WCE) images. However, the CAM labels tend to be extremely noisy, and there is an irreparable gap between CAM labels and ground truths for medical images. This paper proposes a new Discrepancy-basEd Active Learning (DEAL) approach to bridge the gap between CAMs and ground truths with a few annotations. Specifically, to liberate labor, we design a novel discrepancy decoder model and a CAMPUS (CAM, Pseudo-label and groUnd-truth Selection) criterion to replace the noisy CAMs with accurate model predictions and a few human labels. The discrepancy decoder model is trained with a unique scheme to generate standard, coarse and fine predictions. And the CAMPUS criterion is proposed to predict the gaps between CAMs and ground truths based on model divergence and CAM divergence. We evaluate our method on the WCE dataset and results show that our method outperforms the state-of-the-art active learning methods and reaches comparable performance to those trained with full annotated datasets with only 10% of the training data labeled.

Title: A Unified Interactive Model Evaluation for Classification, Object Detection, and Instance Segmentation in Computer Vision. (arXiv:2308.05168v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05168
• Code URL: null
• Copy Paste: [[2308.05168]] A Unified Interactive Model Evaluation for Classification, Object Detection, and Instance Segmentation in Computer Vision(http://arxiv.org/abs/2308.05168)
• Summary:

  Existing model evaluation tools mainly focus on evaluating classification models, leaving a gap in evaluating more complex models, such as object detection. In this paper, we develop an open-source visual analysis tool, Uni-Evaluator, to support a unified model evaluation for classification, object detection, and instance segmentation in computer vision. The key idea behind our method is to formulate both discrete and continuous predictions in different tasks as unified probability distributions. Based on these distributions, we develop 1) a matrix-based visualization to provide an overview of model performance; 2) a table visualization to identify the problematic data subsets where the model performs poorly; 3) a grid visualization to display the samples of interest. These visualizations work together to facilitate the model evaluation from a global overview to individual samples. Two case studies demonstrate the effectiveness of Uni-Evaluator in evaluating model performance and making informed improvements.

Title: SegMatch: A semi-supervised learning method for surgical instrument segmentation. (arXiv:2308.05232v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05232
• Code URL: null
• Copy Paste: [[2308.05232]] SegMatch: A semi-supervised learning method for surgical instrument segmentation(http://arxiv.org/abs/2308.05232)
• Summary:

  Surgical instrument segmentation is recognised as a key enabler to provide advanced surgical assistance and improve computer assisted interventions. In this work, we propose SegMatch, a semi supervised learning method to reduce the need for expensive annotation for laparoscopic and robotic surgical images. SegMatch builds on FixMatch, a widespread semi supervised classification pipeline combining consistency regularization and pseudo labelling, and adapts it for the purpose of segmentation. In our proposed SegMatch, the unlabelled images are weakly augmented and fed into the segmentation model to generate a pseudo-label to enforce the unsupervised loss against the output of the model for the adversarial augmented image on the pixels with a high confidence score. Our adaptation for segmentation tasks includes carefully considering the equivariance and invariance properties of the augmentation functions we rely on. To increase the relevance of our augmentations, we depart from using only handcrafted augmentations and introduce a trainable adversarial augmentation strategy. Our algorithm was evaluated on the MICCAI Instrument Segmentation Challenge datasets Robust-MIS 2019 and EndoVis 2017. Our results demonstrate that adding unlabelled data for training purposes allows us to surpass the performance of fully supervised approaches which are limited by the availability of training data in these challenges. SegMatch also outperforms a range of state-of-the-art semi-supervised learning semantic segmentation models in different labelled to unlabelled data ratios.

Title: Deep Semantic Graph Matching for Large-scale Outdoor Point Clouds Registration. (arXiv:2308.05314v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05314
• Code URL: null
• Copy Paste: [[2308.05314]] Deep Semantic Graph Matching for Large-scale Outdoor Point Clouds Registration(http://arxiv.org/abs/2308.05314)
• Summary:

  The current point cloud registration methods are mainly based on geometric information and usually ignore the semantic information in the point clouds. In this paper, we treat the point cloud registration problem as semantic instance matching and registration task, and propose a deep semantic graph matching method for large-scale outdoor point cloud registration. Firstly, the semantic category labels of 3D point clouds are obtained by utilizing large-scale point cloud semantic segmentation network. The adjacent points with the same category labels are then clustered together by using Euclidean clustering algorithm to obtain the semantic instances. Secondly, the semantic adjacency graph is constructed based on the spatial adjacency relation of semantic instances. Three kinds of high-dimensional features including geometric shape features, semantic categorical features and spatial distribution features are learned through graph convolutional network, and enhanced based on attention mechanism. Thirdly, the semantic instance matching problem is modeled as an optimal transport problem, and solved through an optimal matching layer. Finally, according to the matched semantic instances, the geometric transformation matrix between two point clouds is first obtained by SVD algorithm and then refined by ICP algorithm. The experiments are cconducted on the KITTI Odometry dataset, and the average relative translation error and average relative rotation error of the proposed method are 6.6cm and 0.229{\deg} respectively.

Title: Fine-grained building roof instance segmentation based on domain adapted pretraining and composite dual-backbone. (arXiv:2308.05358v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05358
• Code URL: null
• Copy Paste: [[2308.05358]] Fine-grained building roof instance segmentation based on domain adapted pretraining and composite dual-backbone(http://arxiv.org/abs/2308.05358)
• Summary:

  The diversity of building architecture styles of global cities situated on various landforms, the degraded optical imagery affected by clouds and shadows, and the significant inter-class imbalance of roof types pose challenges for designing a robust and accurate building roof instance segmentor. To address these issues, we propose an effective framework to fulfill semantic interpretation of individual buildings with high-resolution optical satellite imagery. Specifically, the leveraged domain adapted pretraining strategy and composite dual-backbone greatly facilitates the discriminative feature learning. Moreover, new data augmentation pipeline, stochastic weight averaging (SWA) training and instance segmentation based model ensemble in testing are utilized to acquire additional performance boost. Experiment results show that our approach ranks in the first place of the 2023 IEEE GRSS Data Fusion Contest (DFC) Track 1 test phase ($mAP_{50}$:50.6\%). Note-worthily, we have also explored the potential of multimodal data fusion with both optical satellite imagery and SAR data.

Title: Pseudo-label Alignment for Semi-supervised Instance Segmentation. (arXiv:2308.05359v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05359
• Code URL: null
• Copy Paste: [[2308.05359]] Pseudo-label Alignment for Semi-supervised Instance Segmentation(http://arxiv.org/abs/2308.05359)
• Summary:

  Pseudo-labeling is significant for semi-supervised instance segmentation, which generates instance masks and classes from unannotated images for subsequent training. However, in existing pipelines, pseudo-labels that contain valuable information may be directly filtered out due to mismatches in class and mask quality. To address this issue, we propose a novel framework, called pseudo-label aligning instance segmentation (PAIS), in this paper. In PAIS, we devise a dynamic aligning loss (DALoss) that adjusts the weights of semi-supervised loss terms with varying class and mask score pairs. Through extensive experiments conducted on the COCO and Cityscapes datasets, we demonstrate that PAIS is a promising framework for semi-supervised instance segmentation, particularly in cases where labeled data is severely limited. Notably, with just 1\% labeled data, PAIS achieves 21.2 mAP (based on Mask-RCNN) and 19.9 mAP (based on K-Net) on the COCO dataset, outperforming the current state-of-the-art model, \ie, NoisyBoundary with 7.7 mAP, by a margin of over 12 points. Code is available at: \url{https://github.com/hujiecpp/PAIS}.

Title: Adaptive Low Rank Adaptation of Segment Anything to Salient Object Detection. (arXiv:2308.05426v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05426
• Code URL: null
• Copy Paste: [[2308.05426]] Adaptive Low Rank Adaptation of Segment Anything to Salient Object Detection(http://arxiv.org/abs/2308.05426)
• Summary:

  Foundation models, such as OpenAI's GPT-3 and GPT-4, Meta's LLaMA, and Google's PaLM2, have revolutionized the field of artificial intelligence. A notable paradigm shift has been the advent of the Segment Anything Model (SAM), which has exhibited a remarkable capability to segment real-world objects, trained on 1 billion masks and 11 million images. Although SAM excels in general object segmentation, it lacks the intrinsic ability to detect salient objects, resulting in suboptimal performance in this domain. To address this challenge, we present the Segment Salient Object Model (SSOM), an innovative approach that adaptively fine-tunes SAM for salient object detection by harnessing the low-rank structure inherent in deep learning. Comprehensive qualitative and quantitative evaluations across five challenging RGB benchmark datasets demonstrate the superior performance of our approach, surpassing state-of-the-art methods.

Title: Look at the Neighbor: Distortion-aware Unsupervised Domain Adaptation for Panoramic Semantic Segmentation. (arXiv:2308.05493v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2308.05493
• Code URL: null
• Copy Paste: [[2308.05493]] Look at the Neighbor: Distortion-aware Unsupervised Domain Adaptation for Panoramic Semantic Segmentation(http://arxiv.org/abs/2308.05493)
• Summary:

  Endeavors have been recently made to transfer knowledge from the labeled pinhole image domain to the unlabeled panoramic image domain via Unsupervised Domain Adaptation (UDA). The aim is to tackle the domain gaps caused by the style disparities and distortion problem from the non-uniformly distributed pixels of equirectangular projection (ERP). Previous works typically focus on transferring knowledge based on geometric priors with specially designed multi-branch network architectures. As a result, considerable computational costs are induced, and meanwhile, their generalization abilities are profoundly hindered by the variation of distortion among pixels. In this paper, we find that the pixels' neighborhood regions of the ERP indeed introduce less distortion. Intuitively, we propose a novel UDA framework that can effectively address the distortion problems for panoramic semantic segmentation. In comparison, our method is simpler, easier to implement, and more computationally efficient. Specifically, we propose distortion-aware attention (DA) capturing the neighboring pixel distribution without using any geometric constraints. Moreover, we propose a class-wise feature aggregation (CFA) module to iteratively update the feature representations with a memory bank. As such, the feature similarity between two domains can be consistently optimized. Extensive experiments show that our method achieves new state-of-the-art performance while remarkably reducing 80% parameters.