secure

Title: Orion: A Fully Homomorphic Encryption Compiler for Private Deep Neural Network Inference. (arXiv:2311.03470v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.03470
• Code URL: null
• Copy Paste: [[2311.03470]] Orion: A Fully Homomorphic Encryption Compiler for Private Deep Neural Network Inference(http://arxiv.org/abs/2311.03470)
• Summary:

  Fully Homomorphic Encryption (FHE) has the potential to substantially improve privacy and security by enabling computation on encrypted data. This is especially true with deep learning, as today many popular user services are powered by neural networks. One of the major challenges facing wide-scale deployment of FHE-secured neural inference is effectively mapping them to the FHE domain. FHE poses many programming challenges including packing large vectors, handling expensive rotations, and correctly implementing complex strided convolutions. This makes programming FHE inferences prone to poor performance and errors. In this paper we overcome these challenges with Orion, an automated optimizing FHE compiler for neural inference. Orion automatically maps PyTorch-specified networks to FHE, handling common layer types and arbitrary tensor shapes and strides. Moreover, we develop novel optimizations that balance dense FHE vector packing, efficient rotations, and minimize operations to improve performance. We have implemented Orion, which will be open sourced, and evaluated it on common benchmarks used by the FHE deep learning community. We compare Orion to multiple state-of-the-art solutions and report iso-accuracy speedups ranging from 2.7$\times$ to 20.5$\times$.

Title: DonationChain: A New Platform for Blockchain-Based Donation-Tracking System. (arXiv:2311.03573v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.03573
• Code URL: null
• Copy Paste: [[2311.03573]] DonationChain: A New Platform for Blockchain-Based Donation-Tracking System(http://arxiv.org/abs/2311.03573)
• Summary:

  A donation-tracking system using smart contracts and blockchain technology has the potential to revolutionize the way charitable giving is tracked and managed. This article explores how smart contracts and blockchain can be used to create a transparent and secure ledger for tracking charitable donations. We discuss the limitations of traditional donation systems and how a blockchain-based system can help overcome these challenges. We describe how smart contracts work, how they can be used in donation tracking, and the benefits they offer, including automated processes, reduced transaction fees, and increased accountability. We also discuss how blockchain technology provides a decentralized and tamper-proof ledger that can increase transparency and help prevent fraud. Finally, we examine some of the challenges that must be addressed when implementing a smart contract-based donation tracking system, such as the need for technical expertise and the potential for security breaches. Overall, a donation-tracking system using smart contracts and blockchain has the potential to increase trust and accountability in the donation process, which can ultimately help ensure that donations are used for their intended purposes.

Title: IC-SECURE: Intelligent System for Assisting Security Experts in Generating Playbooks for Automated Incident Response. (arXiv:2311.03825v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.03825
• Code URL: null
• Copy Paste: [[2311.03825]] IC-SECURE: Intelligent System for Assisting Security Experts in Generating Playbooks for Automated Incident Response(http://arxiv.org/abs/2311.03825)
• Summary:

  Security orchestration, automation, and response (SOAR) systems ingest alerts from security information and event management (SIEM) system, and then trigger relevant playbooks that automate and orchestrate the execution of a sequence of security activities. SOAR systems have two major limitations: (i) security analysts need to define, create and change playbooks manually, and (ii) the choice between multiple playbooks that could be triggered is based on rules defined by security analysts. To address these limitations, recent studies in the field of artificial intelligence for cybersecurity suggested the task of interactive playbook creation. In this paper, we propose IC-SECURE, an interactive playbook creation solution based on a novel deep learning-based approach that provides recommendations to security analysts during the playbook creation process. IC-SECURE captures the context in the form of alert data and current status of incomplete playbook, required to make reasonable recommendation for next module that should be included in the new playbook being created. We created three evaluation datasets, each of which involved a combination of a set of alert rules and a set of playbooks from a SOAR platform. We evaluated IC-SECURE under various settings, and compared our results with two state-of-the-art recommender system methods. In our evaluation IC-SECURE demonstrated superior performance compared to other methods by consistently recommending the correct security module, achieving precision@1 > 0.8 and recall@3 > 0.92

security

Title: CapST: An Enhanced and Lightweight Method for Deepfake Video Classification. (arXiv:2311.03782v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03782
• Code URL: null
• Copy Paste: [[2311.03782]] CapST: An Enhanced and Lightweight Method for Deepfake Video Classification(http://arxiv.org/abs/2311.03782)
• Summary:

  The proliferation of deepfake videos, synthetic media produced through advanced Artificial Intelligence techniques has raised significant concerns across various sectors, encompassing realms such as politics, entertainment, and security. In response, this research introduces an innovative and streamlined model designed to classify deepfake videos generated by five distinct encoders adeptly. Our approach not only achieves state of the art performance but also optimizes computational resources. At its core, our solution employs part of a VGG19bn as a backbone to efficiently extract features, a strategy proven effective in image-related tasks. We integrate a Capsule Network coupled with a Spatial Temporal attention mechanism to bolster the model's classification capabilities while conserving resources. This combination captures intricate hierarchies among features, facilitating robust identification of deepfake attributes. Delving into the intricacies of our innovation, we introduce an existing video level fusion technique that artfully capitalizes on temporal attention mechanisms. This mechanism serves to handle concatenated feature vectors, capitalizing on the intrinsic temporal dependencies embedded within deepfake videos. By aggregating insights across frames, our model gains a holistic comprehension of video content, resulting in more precise predictions. Experimental results on an extensive benchmark dataset of deepfake videos called DFDM showcase the efficacy of our proposed method. Notably, our approach achieves up to a 4 percent improvement in accurately categorizing deepfake videos compared to baseline models, all while demanding fewer computational resources.

Title: Performance Analysis of Security Certificate Management System in Vehicle-to-Everything (V2X). (arXiv:2311.03360v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.03360
• Code URL: null
• Copy Paste: [[2311.03360]] Performance Analysis of Security Certificate Management System in Vehicle-to-Everything (V2X)(http://arxiv.org/abs/2311.03360)
• Summary:

  In Vehicle-to-Everything (V2X) communications, providing accurate information and safeguarding the privacy of end entities is one of the crucial information security issues. Therefore, several international standardization organizations have begun to develop V2X communication security standards in recent years. For instance, the IEEE 1609.2.1 standard designs a Security Credential Management System (SCMS) that specifies certificate application and issuance processes, as well as certificate revocation processes. Furthermore, the IEEE 1609.2 standard defines certificate formats and Secure Protocol Data Units (SPDUs) for secure data transmission based on these standards. As a result, end entity manufacturers and SCMS providers worldwide have started building V2X security systems in accordance with these standards and conducting interoperability testing. Although international standards mainly employ Elliptic-Curve Cryptography (ECC) for signature/verification and encryption/decryption functions, performance analysis remains a crucial issue for the practical deployment of these systems. Therefore, this study implements end entities and a SCMS conforming to IEEE 1609.2 and IEEE 1609.2.1 standards. It measures the computation and transmission times for each security communication action within the system from the perspective of end entities and identifies potential system bottlenecks. In the experimental results, this study analyzes the most performance-intensive actions and provides relevant suggestions for enhancing system efficiency for SCMS developers to reference.

Title: OpenBSD formal driver verification with SeL4. (arXiv:2311.03585v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.03585
• Code URL: null
• Copy Paste: [[2311.03585]] OpenBSD formal driver verification with SeL4(http://arxiv.org/abs/2311.03585)
• Summary:

  The seL4 microkernel is currently the only kernel that has been fully formally verified. In general, the increased interest in ensuring the security of a kernel's code results from its important role in the entire operating system. One of the basic features of an operating system is that it abstracts the handling of devices. This abstraction is represented by device drivers - the software that manages the hardware. A proper verification of the software component could ensure that the device would work properly unless there is a hardware failure.In this paper, we choose to model the behavior of a device driver and build the proof that the code implementation matches the expected behavior. The proof was written in Isabelle/HOL, the code translation from C to Isabelle was done automatically by the use of the C-to-Isabelle Parser and AutoCorres tools. We choose Isabelle theorem prover because its efficiency was already shown through the verification of seL4 microkernel.

Title: SoK: Security Below the OS -- A Security Analysis of UEFI. (arXiv:2311.03809v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.03809
• Code URL: null
• Copy Paste: [[2311.03809]] SoK: Security Below the OS -- A Security Analysis of UEFI(http://arxiv.org/abs/2311.03809)
• Summary:

  The Unified Extensible Firmware Interface (UEFI) is a linchpin of modern computing systems, governing secure system initialization and booting. This paper is urgently needed because of the surge in UEFI-related attacks and vulnerabilities in recent years. Motivated by this urgent concern, we undertake an extensive exploration of the UEFI landscape, dissecting its distribution supply chain, booting process, and security features. We carefully study a spectrum of UEFI-targeted attacks and proofs of concept (PoCs) for exploiting UEFI-related vulnerabilities. Building upon these insights, we construct a comprehensive attack threat model encompassing threat actors, attack vectors, attack types, vulnerabilities, attack capabilities, and attacker objectives. Drawing inspiration from the MITRE ATT&CK framework, we present a MITRE ATT&CK-like taxonomy delineating tactics, techniques, and sub-techniques in the context of UEFI attacks. This taxonomy can provide a road map for identifying existing gaps and developing new techniques for rootkit prevention, detection, and removal. Finally, the paper discusses existing countermeasures against UEFI attacks including a variety of technical and operational measures that can be implemented to lower the risk of UEFI attacks to an acceptable level. This paper seeks to clarify the complexities of UEFI and equip the cybersecurity community with the necessary knowledge to strengthen the security of this critical component against a growing threat landscape.

Title: Theoretical Patchability Quantification for IP-Level Hardware Patching Designs. (arXiv:2311.03818v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.03818
• Code URL: null
• Copy Paste: [[2311.03818]] Theoretical Patchability Quantification for IP-Level Hardware Patching Designs(http://arxiv.org/abs/2311.03818)
• Summary:

  As the complexity of System-on-Chip (SoC) designs continues to increase, ensuring thorough verification becomes a significant challenge for system integrators. The complexity of verification can result in undetected bugs. Unlike software or firmware bugs, hardware bugs are hard to fix after deployment and they require additional logic, i.e., patching logic integrated with the design in advance in order to patch. However, the absence of a standardized metric for defining "patchability" leaves system integrators relying on their understanding of each IP and security requirements to engineer ad hoc patching designs. In this paper, we propose a theoretical patchability quantification method to analyze designs at the Register Transfer Level (RTL) with provided patching options. Our quantification defines patchability as a combination of observability and controllability so that we can analyze and compare the patchability of IP variations. This quantification is a systematic approach to estimate each patching architecture's ability to patch at run-time and complements existing patching works. In experiments, we compare several design options of the same patching architecture and discuss their differences in terms of theoretical patchability and how many potential weaknesses can be mitigated.

Title: Unveiling the Invisible: Detection and Evaluation of Prototype Pollution Gadgets with Dynamic Taint Analysis. (arXiv:2311.03919v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.03919
• Code URL: null
• Copy Paste: [[2311.03919]] Unveiling the Invisible: Detection and Evaluation of Prototype Pollution Gadgets with Dynamic Taint Analysis(http://arxiv.org/abs/2311.03919)
• Summary:

  For better or worse, JavaScript is the cornerstone of modern Web. Prototype-based languages like JavaScript are susceptible to prototype pollution vulnerabilities, enabling an attacker to inject arbitrary properties into an object's prototype. The attacker can subsequently capitalize on the injected properties by executing otherwise benign pieces of code, so-called gadgets, that perform security-sensitive operations. The success of an attack largely depends on the presence of gadgets, leading to high-profile exploits such as privilege escalation and arbitrary code execution (ACE).

  This paper proposes Dasty, the first semi-automated pipeline to help developers identify gadgets in their applications' software supply chain. Dasty targets server-side Node.js applications and relies on an enhancement of dynamic taint analysis which we implement with the dynamic AST-level instrumentation. Moreover, Dasty provides support for visualization of code flows with an IDE, thus facilitating the subsequent manual analysis for building proof-of-concept exploits. To illustrate the danger of gadgets, we use Dasty in a study of the most dependent-upon NPM packages to analyze the presence of gadgets leading to ACE. Dasty identifies 1,269 server-side packages, of which 631 have code flows that may reach dangerous sinks. We manually prioritize and verify the candidate flows to build proof-of-concept exploits for 49 NPM packages, including popular packages such as ejs, nodemailer and workerpool. To investigate how Dasty integrates with existing tools to find end-to-end exploits, we conduct an in-depth analysis of a popular data visualization dashboard to find one high-severity CVE-2023-31415 leading to remote code execution. For the first time, our results systematically demonstrate the dangers of server-side gadgets and call for further research to solve the problem.

privacy

Title: DAO Decentralization: Voting-Bloc Entropy, Bribery, and Dark DAOs. (arXiv:2311.03530v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.03530
• Code URL: null
• Copy Paste: [[2311.03530]] DAO Decentralization: Voting-Bloc Entropy, Bribery, and Dark DAOs(http://arxiv.org/abs/2311.03530)
• Summary:

  Decentralized Autonomous Organizations (DAOs) use smart contracts to foster communities working toward common goals. Existing definitions of decentralization, however-the 'D' in DAO-fall short of capturing key properties characteristic of diverse and equitable participation. We propose a new metric called Voting-Bloc Entropy (VBE, pronounced ''vibe'') that formalizes a broad notion of decentralization in voting on DAO proposals. VBE measures the similarity of participants' utility functions across a set of proposals. We use VBE to prove a number of results about the decentralizing effects of vote delegation, proposal bundling, bribery, and quadratic voting. Our results lead to practical suggestions for enhancing DAO decentralization. One of our results highlights the risk of systemic bribery with increasing DAO decentralization. To show that this threat is realistic, we present the first practical realization of a Dark DAO, a proposed mechanism for privacy-preserving corruption of identity systems, including those used in DAO voting. Our Dark-DAO prototype uses trusted execution environments (TEEs) in the Oasis Sapphire blockchain for attacks on Ethereum DAOs. It demonstrates that Dark DAOs constitute a realistic future concern for DAO governance.

Title: User-level Differentially Private Stochastic Convex Optimization: Efficient Algorithms with Optimal Rates. (arXiv:2311.03797v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.03797
• Code URL: null
• Copy Paste: [[2311.03797]] User-level Differentially Private Stochastic Convex Optimization: Efficient Algorithms with Optimal Rates(http://arxiv.org/abs/2311.03797)
• Summary:

  We study differentially private stochastic convex optimization (DP-SCO) under user-level privacy, where each user may hold multiple data items. Existing work for user-level DP-SCO either requires super-polynomial runtime [Ghazi et al. (2023)] or requires the number of users to grow polynomially with the dimensionality of the problem with additional strict assumptions [Bassily et al. (2023)]. We develop new algorithms for user-level DP-SCO that obtain optimal rates for both convex and strongly convex functions in polynomial time and require the number of users to grow only logarithmically in the dimension. Moreover, our algorithms are the first to obtain optimal rates for non-smooth functions in polynomial time. These algorithms are based on multiple-pass DP-SGD, combined with a novel private mean estimation procedure for concentrated data, which applies an outlier removal step before estimating the mean of the gradients.

Title: Communication Efficient and Privacy-Preserving Federated Learning Based on Evolution Strategies. (arXiv:2311.03405v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.03405
• Code URL: https://github.com/Eric-Lan0/FedES
• Copy Paste: [[2311.03405]] Communication Efficient and Privacy-Preserving Federated Learning Based on Evolution Strategies(http://arxiv.org/abs/2311.03405)
• Summary:

  Federated learning (FL) is an emerging paradigm for training deep neural networks (DNNs) in distributed manners. Current FL approaches all suffer from high communication overhead and information leakage. In this work, we present a federated learning algorithm based on evolution strategies (FedES), a zeroth-order training method. Instead of transmitting model parameters, FedES only communicates loss values, and thus has very low communication overhead. Moreover, a third party is unable to estimate gradients without knowing the pre-shared seed, which protects data privacy. Experimental results demonstrate FedES can achieve the above benefits while keeping convergence performance the same as that with back propagation methods.

Title: DP-DCAN: Differentially Private Deep Contrastive Autoencoder Network for Single-cell Clustering. (arXiv:2311.03410v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.03410
• Code URL: null
• Copy Paste: [[2311.03410]] DP-DCAN: Differentially Private Deep Contrastive Autoencoder Network for Single-cell Clustering(http://arxiv.org/abs/2311.03410)
• Summary:

  Single-cell RNA sequencing (scRNA-seq) is important to transcriptomic analysis of gene expression. Recently, deep learning has facilitated the analysis of high-dimensional single-cell data. Unfortunately, deep learning models may leak sensitive information about users. As a result, Differential Privacy (DP) is increasingly used to protect privacy. However, existing DP methods usually perturb whole neural networks to achieve differential privacy, and hence result in great performance overheads. To address this challenge, in this paper, we take advantage of the uniqueness of the autoencoder that it outputs only the dimension-reduced vector in the middle of the network, and design a Differentially Private Deep Contrastive Autoencoder Network (DP-DCAN) by partial network perturbation for single-cell clustering. Since only partial network is added with noise, the performance improvement is obvious and twofold: one part of network is trained with less noise due to a bigger privacy budget, and the other part is trained without any noise. Experimental results of six datasets have verified that DP-DCAN is superior to the traditional DP scheme with whole network perturbation. Moreover, DP-DCAN demonstrates strong robustness to adversarial attacks. The code is available at https://github.com/LFD-byte/DP-DCAN.

protect

defense

attack

Title: Image Generation and Learning Strategy for Deep Document Forgery Detection. (arXiv:2311.03650v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03650
• Code URL: null
• Copy Paste: [[2311.03650]] Image Generation and Learning Strategy for Deep Document Forgery Detection(http://arxiv.org/abs/2311.03650)
• Summary:

  In recent years, document processing has flourished and brought numerous benefits. However, there has been a significant rise in reported cases of forged document images. Specifically, recent advancements in deep neural network (DNN) methods for generative tasks may amplify the threat of document forgery. Traditional approaches for forged document images created by prevalent copy-move methods are unsuitable against those created by DNN-based methods, as we have verified. To address this issue, we construct a training dataset of document forgery images, named FD-VIED, by emulating possible attacks, such as text addition, removal, and replacement with recent DNN-methods. Additionally, we introduce an effective pre-training approach through self-supervised learning with both natural images and document images. In our experiments, we demonstrate that our approach enhances detection performance.

Title: Unscrambling the Rectification of Adversarial Attacks Transferability across Computer Networks. (arXiv:2311.03373v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.03373
• Code URL: null
• Copy Paste: [[2311.03373]] Unscrambling the Rectification of Adversarial Attacks Transferability across Computer Networks(http://arxiv.org/abs/2311.03373)
• Summary:

  Convolutional neural networks (CNNs) models play a vital role in achieving state-of-the-art performances in various technological fields. CNNs are not limited to Natural Language Processing (NLP) or Computer Vision (CV) but also have substantial applications in other technological domains, particularly in cybersecurity. The reliability of CNN's models can be compromised because of their susceptibility to adversarial attacks, which can be generated effortlessly, easily applied, and transferred in real-world scenarios.

  In this paper, we present a novel and comprehensive method to improve the strength of attacks and assess the transferability of adversarial examples in CNNs when such strength changes, as well as whether the transferability property issue exists in computer network applications. In the context of our study, we initially examined six distinct modes of attack: the Carlini and Wagner (C&W), Fast Gradient Sign Method (FGSM), Iterative Fast Gradient Sign Method (I-FGSM), Jacobian-based Saliency Map (JSMA), Limited-memory Broyden fletcher Goldfarb Shanno (L-BFGS), and Projected Gradient Descent (PGD) attack. We applied these attack techniques on two popular datasets: the CIC and UNSW datasets. The outcomes of our experiment demonstrate that an improvement in transferability occurs in the targeted scenarios for FGSM, JSMA, LBFGS, and other attacks. Our findings further indicate that the threats to security posed by adversarial examples, even in computer network applications, necessitate the development of novel defense mechanisms to enhance the security of DL-based techniques.

Title: MIRAGE: Multi-Binary Image Risk Assessment with Attack Graph Employment. (arXiv:2311.03565v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2311.03565
• Code URL: null
• Copy Paste: [[2311.03565]] MIRAGE: Multi-Binary Image Risk Assessment with Attack Graph Employment(http://arxiv.org/abs/2311.03565)
• Summary:

  Attackers can exploit known vulnerabilities to infiltrate a device's firmware and the communication between firmware binaries, in order to pass between them. To improve cybersecurity, organizations must identify and mitigate the risks of the firmware they use. An attack graph (AG) can be used to assess and visually display firmware's risks by organizing the identified vulnerabilities into attack paths composed of sequences of actions attackers may perform to compromise firmware images. In this paper, we utilize AGs for firmware risk assessment. We propose MIRAGE (Multi-binary Image Risk Assessment with Attack Graph Employment), a framework for identifying potential attack vectors and vulnerable interactions between firmware binaries; MIRAGE accomplishes this by generating AGs for firmware inter-binary communication. The use cases of the proposed firmware AG generation framework include the identification of risky external interactions, supply chain risk assessment, and security analysis with digital twins. To evaluate the MIRAGE framework, we collected a dataset of 703 firmware images. We also propose a model for examining the risks of firmware binaries, demonstrate the model's implementation on the dataset of firmware images, and list the riskiest binaries.

Title: FD-MIA: Efficient Attacks on Fairness-enhanced Models. (arXiv:2311.03865v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.03865
• Code URL: null
• Copy Paste: [[2311.03865]] FD-MIA: Efficient Attacks on Fairness-enhanced Models(http://arxiv.org/abs/2311.03865)
• Summary:

  Previous studies have developed fairness methods for biased models that exhibit discriminatory behaviors towards specific subgroups. While these models have shown promise in achieving fair predictions, recent research has identified their potential vulnerability to score-based membership inference attacks (MIAs). In these attacks, adversaries can infer whether a particular data sample was used during training by analyzing the model's prediction scores. However, our investigations reveal that these score-based MIAs are ineffective when targeting fairness-enhanced models in binary classifications. The attack models trained to launch the MIAs degrade into simplistic threshold models, resulting in lower attack performance. Meanwhile, we observe that fairness methods often lead to prediction performance degradation for the majority subgroups of the training data. This raises the barrier to successful attacks and widens the prediction gaps between member and non-member data. Building upon these insights, we propose an efficient MIA method against fairness-enhanced models based on fairness discrepancy results (FD-MIA). It leverages the difference in the predictions from both the original and fairness-enhanced models and exploits the observed prediction gaps as attack clues. We also explore potential strategies for mitigating privacy leakages. Extensive experiments validate our findings and demonstrate the efficacy of the proposed method.

robust

Title: DeepInspect: An AI-Powered Defect Detection for Manufacturing Industries. (arXiv:2311.03725v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03725
• Code URL: null
• Copy Paste: [[2311.03725]] DeepInspect: An AI-Powered Defect Detection for Manufacturing Industries(http://arxiv.org/abs/2311.03725)
• Summary:

  Utilizing Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Generative Adversarial Networks (GANs), our system introduces an innovative approach to defect detection in manufacturing. This technology excels in precisely identifying faults by extracting intricate details from product photographs, utilizing RNNs to detect evolving errors and generating synthetic defect data to bolster the model's robustness and adaptability across various defect scenarios. The project leverages a deep learning framework to automate real-time flaw detection in the manufacturing process. It harnesses extensive datasets of annotated images to discern complex defect patterns. This integrated system seamlessly fits into production workflows, thereby boosting efficiency and elevating product quality. As a result, it reduces waste and operational costs, ultimately enhancing market competitiveness.

Title: 3DifFusionDet: Diffusion Model for 3D Object Detection with Robust LiDAR-Camera Fusion. (arXiv:2311.03742v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03742
• Code URL: null
• Copy Paste: [[2311.03742]] 3DifFusionDet: Diffusion Model for 3D Object Detection with Robust LiDAR-Camera Fusion(http://arxiv.org/abs/2311.03742)
• Summary:

  Good 3D object detection performance from LiDAR-Camera sensors demands seamless feature alignment and fusion strategies. We propose the 3DifFusionDet framework in this paper, which structures 3D object detection as a denoising diffusion process from noisy 3D boxes to target boxes. In this framework, ground truth boxes diffuse in a random distribution for training, and the model learns to reverse the noising process. During inference, the model gradually refines a set of boxes that were generated at random to the outcomes. Under the feature align strategy, the progressive refinement method could make a significant contribution to robust LiDAR-Camera fusion. The iterative refinement process could also demonstrate great adaptability by applying the framework to various detecting circumstances where varying levels of accuracy and speed are required. Extensive experiments on KITTI, a benchmark for real-world traffic object identification, revealed that 3DifFusionDet is able to perform favorably in comparison to earlier, well-respected detectors.

Title: UP-NeRF: Unconstrained Pose-Prior-Free Neural Radiance Fields. (arXiv:2311.03784v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03784
• Code URL: null
• Copy Paste: [[2311.03784]] UP-NeRF: Unconstrained Pose-Prior-Free Neural Radiance Fields(http://arxiv.org/abs/2311.03784)
• Summary:

  Neural Radiance Field (NeRF) has enabled novel view synthesis with high fidelity given images and camera poses. Subsequent works even succeeded in eliminating the necessity of pose priors by jointly optimizing NeRF and camera pose. However, these works are limited to relatively simple settings such as photometrically consistent and occluder-free image collections or a sequence of images from a video. So they have difficulty handling unconstrained images with varying illumination and transient occluders. In this paper, we propose \textbf{UP-NeRF} (\textbf{U}nconstrained \textbf{P}ose-prior-free \textbf{Ne}ural \textbf{R}adiance \textbf{F}ields) to optimize NeRF with unconstrained image collections without camera pose prior. We tackle these challenges with surrogate tasks that optimize color-insensitive feature fields and a separate module for transient occluders to block their influence on pose estimation. In addition, we introduce a candidate head to enable more robust pose estimation and transient-aware depth supervision to minimize the effect of incorrect prior. Our experiments verify the superior performance of our method compared to the baselines including BARF and its variants in a challenging internet photo collection, \textit{Phototourism} dataset. The code of UP-NeRF is available at \url{https://github.com/mlvlab/UP-NeRF}.

Title: RobustMat: Neural Diffusion for Street Landmark Patch Matching under Challenging Environments. (arXiv:2311.03904v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03904
• Code URL: https://github.com/ai-it-avs/robustmat
• Copy Paste: [[2311.03904]] RobustMat: Neural Diffusion for Street Landmark Patch Matching under Challenging Environments(http://arxiv.org/abs/2311.03904)
• Summary:

  For autonomous vehicles (AVs), visual perception techniques based on sensors like cameras play crucial roles in information acquisition and processing. In various computer perception tasks for AVs, it may be helpful to match landmark patches taken by an onboard camera with other landmark patches captured at a different time or saved in a street scene image database. To perform matching under challenging driving environments caused by changing seasons, weather, and illumination, we utilize the spatial neighborhood information of each patch. We propose an approach, named RobustMat, which derives its robustness to perturbations from neural differential equations. A convolutional neural ODE diffusion module is used to learn the feature representation for the landmark patches. A graph neural PDE diffusion module then aggregates information from neighboring landmark patches in the street scene. Finally, feature similarity learning outputs the final matching score. Our approach is evaluated on several street scene datasets and demonstrated to achieve state-of-the-art matching results under environmental perturbations.

Title: Measuring Adversarial Datasets. (arXiv:2311.03566v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.03566
• Code URL: null
• Copy Paste: [[2311.03566]] Measuring Adversarial Datasets(http://arxiv.org/abs/2311.03566)
• Summary:

  In the era of widespread public use of AI systems across various domains, ensuring adversarial robustness has become increasingly vital to maintain safety and prevent undesirable errors. Researchers have curated various adversarial datasets (through perturbations) for capturing model deficiencies that cannot be revealed in standard benchmark datasets. However, little is known about how these adversarial examples differ from the original data points, and there is still no methodology to measure the intended and unintended consequences of those adversarial transformations. In this research, we conducted a systematic survey of existing quantifiable metrics that describe text instances in NLP tasks, among dimensions of difficulty, diversity, and disagreement. We selected several current adversarial effect datasets and compared the distributions between the original and their adversarial counterparts. The results provide valuable insights into what makes these datasets more challenging from a metrics perspective and whether they align with underlying assumptions.

Title: Dimensions of Online Conflict: Towards Modeling Agonism. (arXiv:2311.03584v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.03584
• Code URL: null
• Copy Paste: [[2311.03584]] Dimensions of Online Conflict: Towards Modeling Agonism(http://arxiv.org/abs/2311.03584)
• Summary:

  Agonism plays a vital role in democratic dialogue by fostering diverse perspectives and robust discussions. Within the realm of online conflict there is another type: hateful antagonism, which undermines constructive dialogue. Detecting conflict online is central to platform moderation and monetization. It is also vital for democratic dialogue, but only when it takes the form of agonism. To model these two types of conflict, we collected Twitter conversations related to trending controversial topics. We introduce a comprehensive annotation schema for labelling different dimensions of conflict in the conversations, such as the source of conflict, the target, and the rhetorical strategies deployed. Using this schema, we annotated approximately 4,000 conversations with multiple labels. We then trained both logistic regression and transformer-based models on the dataset, incorporating context from the conversation, including the number of participants and the structure of the interactions. Results show that contextual labels are helpful in identifying conflict and make the models robust to variations in topic. Our research contributes a conceptualization of different dimensions of conflict, a richly annotated dataset, and promising results that can contribute to content moderation.

Title: Counterfactual Data Augmentation with Contrastive Learning. (arXiv:2311.03630v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.03630
• Code URL: null
• Copy Paste: [[2311.03630]] Counterfactual Data Augmentation with Contrastive Learning(http://arxiv.org/abs/2311.03630)
• Summary:

  Statistical disparity between distinct treatment groups is one of the most significant challenges for estimating Conditional Average Treatment Effects (CATE). To address this, we introduce a model-agnostic data augmentation method that imputes the counterfactual outcomes for a selected subset of individuals. Specifically, we utilize contrastive learning to learn a representation space and a similarity measure such that in the learned representation space close individuals identified by the learned similarity measure have similar potential outcomes. This property ensures reliable imputation of counterfactual outcomes for the individuals with close neighbors from the alternative treatment group. By augmenting the original dataset with these reliable imputations, we can effectively reduce the discrepancy between different treatment groups, while inducing minimal imputation error. The augmented dataset is subsequently employed to train CATE estimation models. Theoretical analysis and experimental studies on synthetic and semi-synthetic benchmarks demonstrate that our method achieves significant improvements in both performance and robustness to overfitting across state-of-the-art models.

Title: Stable Modular Control via Contraction Theory for Reinforcement Learning. (arXiv:2311.03669v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.03669
• Code URL: null
• Copy Paste: [[2311.03669]] Stable Modular Control via Contraction Theory for Reinforcement Learning(http://arxiv.org/abs/2311.03669)
• Summary:

  We propose a novel way to integrate control techniques with reinforcement learning (RL) for stability, robustness, and generalization: leveraging contraction theory to realize modularity in neural control, which ensures that combining stable subsystems can automatically preserve the stability. We realize such modularity via signal composition and dynamic decomposition. Signal composition creates the latent space, within which RL applies to maximizing rewards. Dynamic decomposition is realized by coordinate transformation that creates an auxiliary space, within which the latent signals are coupled in the way that their combination can preserve stability provided each signal, that is, each subsystem, has stable self-feedbacks. Leveraging modularity, the nonlinear stability problem is deconstructed into algebraically solvable ones, the stability of the subsystems in the auxiliary space, yielding linear constraints on the input gradients of control networks that can be as simple as switching the signs of network weights. This minimally invasive method for stability allows arguably easy integration into the modular neural architectures in machine learning, like hierarchical RL, and improves their performance. We demonstrate in simulation the necessity and the effectiveness of our method: the necessity for robustness and generalization, and the effectiveness in improving hierarchical RL for manipulation learning.

biometric

Title: Multimodal Stress Detection Using Facial Landmarks and Biometric Signals. (arXiv:2311.03606v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03606
• Code URL: null
• Copy Paste: [[2311.03606]] Multimodal Stress Detection Using Facial Landmarks and Biometric Signals(http://arxiv.org/abs/2311.03606)
• Summary:

  The development of various sensing technologies is improving measurements of stress and the well-being of individuals. Although progress has been made with single signal modalities like wearables and facial emotion recognition, integrating multiple modalities provides a more comprehensive understanding of stress, given that stress manifests differently across different people. Multi-modal learning aims to capitalize on the strength of each modality rather than relying on a single signal. Given the complexity of processing and integrating high-dimensional data from limited subjects, more research is needed. Numerous research efforts have been focused on fusing stress and emotion signals at an early stage, e.g., feature-level fusion using basic machine learning methods and 1D-CNN Methods. This paper proposes a multi-modal learning approach for stress detection that integrates facial landmarks and biometric signals. We test this multi-modal integration with various early-fusion and late-fusion techniques to integrate the 1D-CNN model from biometric signals and 2-D CNN using facial landmarks. We evaluate these architectures using a rigorous test of models' generalizability using the leave-one-subject-out mechanism, i.e., all samples related to a single subject are left out to train the model. Our findings show that late-fusion achieved 94.39\% accuracy, and early-fusion surpassed it with a 98.38\% accuracy rate. This research contributes valuable insights into enhancing stress detection through a multi-modal approach. The proposed research offers important knowledge in improving stress detection using a multi-modal approach.

steal

extraction

Title: Multi Loss-based Feature Fusion and Top Two Voting Ensemble Decision Strategy for Facial Expression Recognition in the Wild. (arXiv:2311.03478v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03478
• Code URL: null
• Copy Paste: [[2311.03478]] Multi Loss-based Feature Fusion and Top Two Voting Ensemble Decision Strategy for Facial Expression Recognition in the Wild(http://arxiv.org/abs/2311.03478)
• Summary:

  Facial expression recognition (FER) in the wild is a challenging task affected by the image quality and has attracted broad interest in computer vision. There is no research using feature fusion and ensemble strategy for FER simultaneously. Different from previous studies, this paper applies both internal feature fusion for a single model and feature fusion among multiple networks, as well as the ensemble strategy. This paper proposes one novel single model named R18+FAML, as well as one ensemble model named R18+FAML-FGA-T2V to improve the performance of the FER in the wild. Based on the structure of ResNet18 (R18), R18+FAML combines internal Feature fusion and three Attention blocks using Multiple Loss functions (FAML) to improve the diversity of the feature extraction. To improve the performance of R18+FAML, we propose a Feature fusion among networks based on the Genetic Algorithm (FGA), which can fuse the convolution kernels for feature extraction of multiple networks. On the basis of R18+FAML and FGA, we propose one ensemble strategy, i.e., the Top Two Voting (T2V) to support the classification of FER, which can consider more classification information comprehensively. Combining the above strategies, R18+FAML-FGA-T2V can focus on the main expression-aware areas. Extensive experiments demonstrate that our single model R18+FAML and the ensemble model R18+FAML-FGA-T2V achieve the accuracies of $\left( 90.32, 62.17, 65.83 \right)\%$ and $\left( 91.59, 63.27, 66.63 \right)\%$ on three challenging unbalanced FER datasets RAF-DB, AffectNet-8 and AffectNet-7 respectively, both outperforming the state-of-the-art results.

Title: Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models. (arXiv:2311.03799v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03799
• Code URL: null
• Copy Paste: [[2311.03799]] Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models(http://arxiv.org/abs/2311.03799)
• Summary:

  Human-object interaction (HOI) detection aims to comprehend the intricate relationships between humans and objects, predicting $<human, action, object>$ triplets, and serving as the foundation for numerous computer vision tasks. The complexity and diversity of human-object interactions in the real world, however, pose significant challenges for both annotation and recognition, particularly in recognizing interactions within an open world context. This study explores the universal interaction recognition in an open-world setting through the use of Vision-Language (VL) foundation models and large language models (LLMs). The proposed method is dubbed as \emph{\textbf{UniHOI}}. We conduct a deep analysis of the three hierarchical features inherent in visual HOI detectors and propose a method for high-level relation extraction aimed at VL foundation models, which we call HO prompt-based learning. Our design includes an HO Prompt-guided Decoder (HOPD), facilitates the association of high-level relation representations in the foundation model with various HO pairs within the image. Furthermore, we utilize a LLM (\emph{i.e.} GPT) for interaction interpretation, generating a richer linguistic understanding for complex HOIs. For open-category interaction recognition, our method supports either of two input types: interaction phrase or interpretive sentence. Our efficient architecture design and learning methods effectively unleash the potential of the VL foundation models and LLMs, allowing UniHOI to surpass all existing methods with a substantial margin, under both supervised and zero-shot settings. The code and pre-trained weights are available at: \url{https://github.com/Caoyichao/UniHOI}.

Title: Spoken Dialogue System for Medical Prescription Acquisition on Smartphone: Development, Corpus and Evaluation. (arXiv:2311.03510v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.03510
• Code URL: null
• Copy Paste: [[2311.03510]] Spoken Dialogue System for Medical Prescription Acquisition on Smartphone: Development, Corpus and Evaluation(http://arxiv.org/abs/2311.03510)
• Summary:

  Hospital information systems (HIS) have become an essential part of healthcare institutions and now incorporate prescribing support software. Prescription support software allows for structured information capture, which improves the safety, appropriateness and efficiency of prescriptions and reduces the number of adverse drug events (ADEs). However, such a system increases the amount of time physicians spend at a computer entering information instead of providing medical care. In addition, any new visiting clinician must learn to manage complex interfaces since each HIS has its own interfaces. In this paper, we present a natural language interface for e-prescribing software in the form of a spoken dialogue system accessible on a smartphone. This system allows prescribers to record their prescriptions verbally, a form of interaction closer to their usual practice. The system extracts the formal representation of the prescription ready to be checked by the prescribing software and uses the dialogue to request mandatory information, correct errors or warn of particular situations. Since, to the best of our knowledge, there is no existing voice-based prescription dialogue system, we present the system developed in a low-resource environment, focusing on dialogue modeling, semantic extraction and data augmentation. The system was evaluated in the wild with 55 participants. This evaluation showed that our system has an average prescription time of 66.15 seconds for physicians and 35.64 seconds for other experts, and a task success rate of 76\% for physicians and 72\% for other experts. All evaluation data were recorded and annotated to form PxCorpus, the first spoken drug prescription corpus that has been made fully available to the community (\url{https://doi.org/10.5281/zenodo.6524162}).

Title: Generalization of NLP Models: Notion and Causation. (arXiv:2311.03663v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.03663
• Code URL: null
• Copy Paste: [[2311.03663]] Generalization of NLP Models: Notion and Causation(http://arxiv.org/abs/2311.03663)
• Summary:

  The NLP community typically relies on performance of a model on a held-out test set to assess generalization. Performance drops observed in datasets outside of official test sets are generally attributed to "out-of-distribution'' effects. Here, we explore the foundations of generalizability and study the various factors that affect it, articulating generalizability lessons from clinical studies. In clinical research generalizability depends on (a) internal validity of experiments to ensure controlled measurement of cause and effect, and (b) external validity or transportability of the results to the wider population. We present the need to ensure internal validity when building machine learning models in natural language processing, especially where results may be impacted by spurious correlations in the data. We demonstrate how spurious factors, such as the distance between entities in relation extraction tasks, can affect model internal validity and in turn adversely impact generalization. We also offer guidance on how to analyze generalization failures.

Title: Leveraging Structured Information for Explainable Multi-hop Question Answering and Reasoning. (arXiv:2311.03734v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.03734
• Code URL: null
• Copy Paste: [[2311.03734]] Leveraging Structured Information for Explainable Multi-hop Question Answering and Reasoning(http://arxiv.org/abs/2311.03734)
• Summary:

  Neural models, including large language models (LLMs), achieve superior performance on multi-hop question-answering. To elicit reasoning capabilities from LLMs, recent works propose using the chain-of-thought (CoT) mechanism to generate both the reasoning chain and the answer, which enhances the model's capabilities in conducting multi-hop reasoning. However, several challenges still remain: such as struggling with inaccurate reasoning, hallucinations, and lack of interpretability. On the other hand, information extraction (IE) identifies entities, relations, and events grounded to the text. The extracted structured information can be easily interpreted by humans and machines (Grishman, 2019). In this work, we investigate constructing and leveraging extracted semantic structures (graphs) for multi-hop question answering, especially the reasoning process. Empirical results and human evaluations show that our framework: generates more faithful reasoning chains and substantially improves the QA performance on two benchmark datasets. Moreover, the extracted structures themselves naturally provide grounded explanations that are preferred by humans, as compared to the generated reasoning chains and saliency-based explanations.

Title: Unified Low-Resource Sequence Labeling by Sample-Aware Dynamic Sparse Finetuning. (arXiv:2311.03748v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.03748
• Code URL: https://github.com/psunlpgroup/fish-dip
• Copy Paste: [[2311.03748]] Unified Low-Resource Sequence Labeling by Sample-Aware Dynamic Sparse Finetuning(http://arxiv.org/abs/2311.03748)
• Summary:

  Unified Sequence Labeling that articulates different sequence labeling problems such as Named Entity Recognition, Relation Extraction, Semantic Role Labeling, etc. in a generalized sequence-to-sequence format opens up the opportunity to make the maximum utilization of large language model knowledge toward structured prediction. Unfortunately, this requires formatting them into specialized augmented format unknown to the base pretrained language model (PLMs) necessitating finetuning to the target format. This significantly bounds its usefulness in data-limited settings where finetuning large models cannot properly generalize to the target format. To address this challenge and leverage PLM knowledge effectively, we propose FISH-DIP, a sample-aware dynamic sparse finetuning strategy that selectively focuses on a fraction of parameters, informed by feedback from highly regressing examples, during the fine-tuning process. By leveraging the dynamism of sparsity, our approach mitigates the impact of well-learned samples and prioritizes underperforming instances for improvement in generalization. Across five tasks of sequence labeling, we demonstrate that FISH-DIP can smoothly optimize the model in low resource settings offering upto 40% performance improvements over full fine-tuning depending on target evaluation settings. Also, compared to in-context learning and other parameter-efficient fine-tuning approaches, FISH-DIP performs comparably or better, notably in extreme low-resource settings.

Title: iACOS: Advancing Implicit Sentiment Extraction with Informative and Adaptive Negative Examples. (arXiv:2311.03896v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.03896
• Code URL: null
• Copy Paste: [[2311.03896]] iACOS: Advancing Implicit Sentiment Extraction with Informative and Adaptive Negative Examples(http://arxiv.org/abs/2311.03896)
• Summary:

  Aspect-based sentiment analysis (ABSA) have been extensively studied, but little light has been shed on the quadruple extraction consisting of four fundamental elements: aspects, categories, opinions and sentiments, especially with implicit aspects and opinions. In this paper, we propose a new method iACOS for extracting Implicit Aspects with Categories and Opinions with Sentiments. First, iACOS appends two implicit tokens at the end of a text to capture the context-aware representation of all tokens including implicit aspects and opinions. Second, iACOS develops a sequence labeling model over the context-aware token representation to co-extract explicit and implicit aspects and opinions. Third, iACOS devises a multi-label classifier with a specialized multi-head attention for discovering aspect-opinion pairs and predicting their categories and sentiments simultaneously. Fourth, iACOS leverages informative and adaptive negative examples to jointly train the multi-label classifier and the other two classifiers on categories and sentiments by multi-task learning. Finally, the experimental results show that iACOS significantly outperforms other quadruple extraction baselines according to the F1 score on two public benchmark datasets.

membership infer

federate

Title: Can We Trust the Similarity Measurement in Federated Learning?. (arXiv:2311.03369v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.03369
• Code URL: null
• Copy Paste: [[2311.03369]] Can We Trust the Similarity Measurement in Federated Learning?(http://arxiv.org/abs/2311.03369)
• Summary:

  Is it secure to measure the reliability of local models by similarity in federated learning (FL)? This paper delves into an unexplored security threat concerning applying similarity metrics, such as the L_2 norm, Euclidean distance, and cosine similarity, in protecting FL. We first uncover the deficiencies of similarity metrics that high-dimensional local models, including benign and poisoned models, may be evaluated to have the same similarity while being significantly different in the parameter values. We then leverage this finding to devise a novel untargeted model poisoning attack, Faker, which launches the attack by simultaneously maximizing the evaluated similarity of the poisoned local model and the difference in the parameter values. Experimental results based on seven datasets and eight defenses show that Faker outperforms the state-of-the-art benchmark attacks by 1.1-9.0X in reducing accuracy and 1.2-8.0X in saving time cost, which even holds for the case of a single malicious client with limited knowledge about the FL system. Moreover, Faker can degrade the performance of the global model by attacking only once. We also preliminarily explore extending Faker to other attacks, such as backdoor attacks and Sybil attacks. Lastly, we provide a model evaluation strategy, called the similarity of partial parameters (SPP), to defend against Faker. Given that numerous mechanisms in FL utilize similarity metrics to assess local models, this work suggests that we should be vigilant regarding the potential risks of using these metrics.

Title: Differentially Private Pre-Trained Model Fusion using Decentralized Federated Graph Matching. (arXiv:2311.03396v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.03396
• Code URL: null
• Copy Paste: [[2311.03396]] Differentially Private Pre-Trained Model Fusion using Decentralized Federated Graph Matching(http://arxiv.org/abs/2311.03396)
• Summary:

  Model fusion is becoming a crucial component in the context of model-as-a-service scenarios, enabling the delivery of high-quality model services to local users. However, this approach introduces privacy risks and imposes certain limitations on its applications. Ensuring secure model exchange and knowledge fusion among users becomes a significant challenge in this setting. To tackle this issue, we propose PrivFusion, a novel architecture that preserves privacy while facilitating model fusion under the constraints of local differential privacy. PrivFusion leverages a graph-based structure, enabling the fusion of models from multiple parties without necessitating retraining. By employing randomized mechanisms, PrivFusion ensures privacy guarantees throughout the fusion process. To enhance model privacy, our approach incorporates a hybrid local differentially private mechanism and decentralized federated graph matching, effectively protecting both activation values and weights. Additionally, we introduce a perturbation filter adapter to alleviate the impact of randomized noise, thereby preserving the utility of the fused model. Through extensive experiments conducted on diverse image datasets and real-world healthcare applications, we provide empirical evidence showcasing the effectiveness of PrivFusion in maintaining model performance while preserving privacy. Our contributions offer valuable insights and practical solutions for secure and collaborative data analysis within the domain of privacy-preserving model fusion.

Title: Federated Learning for Clinical Structured Data: A Benchmark Comparison of Engineering and Statistical Approaches. (arXiv:2311.03417v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.03417
• Code URL: https://github.com/nliulab/fl-benchmark
• Copy Paste: [[2311.03417]] Federated Learning for Clinical Structured Data: A Benchmark Comparison of Engineering and Statistical Approaches(http://arxiv.org/abs/2311.03417)
• Summary:

  Federated learning (FL) has shown promising potential in safeguarding data privacy in healthcare collaborations. While the term "FL" was originally coined by the engineering community, the statistical field has also explored similar privacy-preserving algorithms. Statistical FL algorithms, however, remain considerably less recognized than their engineering counterparts. Our goal was to bridge the gap by presenting the first comprehensive comparison of FL frameworks from both engineering and statistical domains. We evaluated five FL frameworks using both simulated and real-world data. The results indicate that statistical FL algorithms yield less biased point estimates for model coefficients and offer convenient confidence interval estimations. In contrast, engineering-based methods tend to generate more accurate predictions, sometimes surpassing central pooled and statistical FL models. This study underscores the relative strengths and weaknesses of both types of methods, emphasizing the need for increased awareness and their integration in future FL applications.

Title: Asynchronous Local Computations in Distributed Bayesian Learning. (arXiv:2311.03496v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.03496
• Code URL: null
• Copy Paste: [[2311.03496]] Asynchronous Local Computations in Distributed Bayesian Learning(http://arxiv.org/abs/2311.03496)
• Summary:

  Due to the expanding scope of machine learning (ML) to the fields of sensor networking, cooperative robotics and many other multi-agent systems, distributed deployment of inference algorithms has received a lot of attention. These algorithms involve collaboratively learning unknown parameters from dispersed data collected by multiple agents. There are two competing aspects in such algorithms, namely, intra-agent computation and inter-agent communication. Traditionally, algorithms are designed to perform both synchronously. However, certain circumstances need frugal use of communication channels as they are either unreliable, time-consuming, or resource-expensive. In this paper, we propose gossip-based asynchronous communication to leverage fast computations and reduce communication overhead simultaneously. We analyze the effects of multiple (local) intra-agent computations by the active agents between successive inter-agent communications. For local computations, Bayesian sampling via unadjusted Langevin algorithm (ULA) MCMC is utilized. The communication is assumed to be over a connected graph (e.g., as in decentralized learning), however, the results can be extended to coordinated communication where there is a central server (e.g., federated learning). We theoretically quantify the convergence rates in the process. To demonstrate the efficacy of the proposed algorithm, we present simulations on a toy problem as well as on real world data sets to train ML models to perform classification tasks. We observe faster initial convergence and improved performance accuracy, especially in the low data range. We achieve on average 78% and over 90% classification accuracy respectively on the Gamma Telescope and mHealth data sets from the UCI ML repository.

Title: CAFE: Carbon-Aware Federated Learning in Geographically Distributed Data Centers. (arXiv:2311.03615v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.03615
• Code URL: null
• Copy Paste: [[2311.03615]] CAFE: Carbon-Aware Federated Learning in Geographically Distributed Data Centers(http://arxiv.org/abs/2311.03615)
• Summary:

  Training large-scale artificial intelligence (AI) models demands significant computational power and energy, leading to increased carbon footprint with potential environmental repercussions. This paper delves into the challenges of training AI models across geographically distributed (geo-distributed) data centers, emphasizing the balance between learning performance and carbon footprint. We consider Federated Learning (FL) as a solution, which prioritizes model parameter exchange over raw data, ensuring data privacy and compliance with local regulations. Given the variability in carbon intensity across regions, we propose a new framework called CAFE (short for Carbon-Aware Federated Learning) to optimize training within a fixed carbon footprint budget. Our approach incorporates coreset selection to assess learning performance, employs the Lyapunov drift-plus-penalty framework to address the unpredictability of future carbon intensity, and devises an efficient algorithm to address the combinatorial complexity of the data center selection. Through extensive simulations using real-world carbon intensity data, we demonstrate the efficacy of our algorithm, highlighting its superiority over existing methods in optimizing learning performance while minimizing environmental impact.

fair

Title: Bias and Diversity in Synthetic-based Face Recognition. (arXiv:2311.03970v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03970
• Code URL: null
• Copy Paste: [[2311.03970]] Bias and Diversity in Synthetic-based Face Recognition(http://arxiv.org/abs/2311.03970)
• Summary:

  Synthetic data is emerging as a substitute for authentic data to solve ethical and legal challenges in handling authentic face data. The current models can create real-looking face images of people who do not exist. However, it is a known and sensitive problem that face recognition systems are susceptible to bias, i.e. performance differences between different demographic and non-demographics attributes, which can lead to unfair decisions. In this work, we investigate how the diversity of synthetic face recognition datasets compares to authentic datasets, and how the distribution of the training data of the generative models affects the distribution of the synthetic data. To do this, we looked at the distribution of gender, ethnicity, age, and head position. Furthermore, we investigated the concrete bias of three recent synthetic-based face recognition models on the studied attributes in comparison to a baseline model trained on authentic data. Our results show that the generator generate a similar distribution as the used training data in terms of the different attributes. With regard to bias, it can be seen that the synthetic-based models share a similar bias behavior with the authentic-based models. However, with the uncovered lower intra-identity attribute consistency seems to be beneficial in reducing bias.

Title: The Fairness Stitch: Unveiling the Potential of Model Stitching in Neural Network De-Biasing. (arXiv:2311.03532v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.03532
• Code URL: https://github.com/modar7/the_fairness_stitch
• Copy Paste: [[2311.03532]] The Fairness Stitch: Unveiling the Potential of Model Stitching in Neural Network De-Biasing(http://arxiv.org/abs/2311.03532)
• Summary:

  The pursuit of fairness in machine learning models has emerged as a critical research challenge in different applications ranging from bank loan approval to face detection. Despite the widespread adoption of artificial intelligence algorithms across various domains, concerns persist regarding the presence of biases and discrimination within these models. To address this pressing issue, this study introduces a novel method called "The Fairness Stitch (TFS)" to enhance fairness in deep learning models. This method combines model stitching and training jointly, while incorporating fairness constraints. In this research, we assess the effectiveness of our proposed method by conducting a comprehensive evaluation of two well-known datasets, CelebA and UTKFace. We systematically compare the performance of our approach with the existing baseline method. Our findings reveal a notable improvement in achieving a balanced trade-off between fairness and performance, highlighting the promising potential of our method to address bias-related challenges and foster equitable outcomes in machine learning models. This paper poses a challenge to the conventional wisdom of the effectiveness of the last layer in deep learning models for de-biasing.

Title: Loss Balancing for Fair Supervised Learning. (arXiv:2311.03714v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.03714
• Code URL: https://github.com/khalilimahdi/loss_balancing_icml2023
• Copy Paste: [[2311.03714]] Loss Balancing for Fair Supervised Learning(http://arxiv.org/abs/2311.03714)
• Summary:

  Supervised learning models have been used in various domains such as lending, college admission, face recognition, natural language processing, etc. However, they may inherit pre-existing biases from training data and exhibit discrimination against protected social groups. Various fairness notions have been proposed to address unfairness issues. In this work, we focus on Equalized Loss (EL), a fairness notion that requires the expected loss to be (approximately) equalized across different groups. Imposing EL on the learning process leads to a non-convex optimization problem even if the loss function is convex, and the existing fair learning algorithms cannot properly be adopted to find the fair predictor under the EL constraint. This paper introduces an algorithm that can leverage off-the-shelf convex programming tools (e.g., CVXPY) to efficiently find the global optimum of this non-convex optimization. In particular, we propose the ELminimizer algorithm, which finds the optimal fair predictor under EL by reducing the non-convex optimization to a sequence of convex optimization problems. We theoretically prove that our algorithm finds the global optimal solution under certain conditions. Then, we support our theoretical results through several empirical studies.

interpretability

Title: PowerFlowNet: Leveraging Message Passing GNNs for Improved Power Flow Approximation. (arXiv:2311.03415v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.03415
• Code URL: null
• Copy Paste: [[2311.03415]] PowerFlowNet: Leveraging Message Passing GNNs for Improved Power Flow Approximation(http://arxiv.org/abs/2311.03415)
• Summary:

  Accurate and efficient power flow (PF) analysis is crucial in modern electrical networks' efficient operation and planning. Therefore, there is a need for scalable algorithms capable of handling large-scale power networks that can provide accurate and fast solutions. Graph Neural Networks (GNNs) have emerged as a promising approach for enhancing the speed of PF approximations by leveraging their ability to capture distinctive features from the underlying power network graph. In this study, we introduce PowerFlowNet, a novel GNN architecture for PF approximation that showcases similar performance with the traditional Newton-Raphson method but achieves it 4 times faster in the simple IEEE 14-bus system and 145 times faster in the realistic case of the French high voltage network (6470rte). Meanwhile, it significantly outperforms other traditional approximation methods, such as the DC relaxation method, in terms of performance and execution time; therefore, making PowerFlowNet a highly promising solution for real-world PF analysis. Furthermore, we verify the efficacy of our approach by conducting an in-depth experimental evaluation, thoroughly examining the performance, scalability, interpretability, and architectural dependability of PowerFlowNet. The evaluation provides insights into the behavior and potential applications of GNNs in power system analysis.

explainability

Title: Which is better? Exploring Prompting Strategy For LLM-based Metrics. (arXiv:2311.03754v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.03754
• Code URL: null
• Copy Paste: [[2311.03754]] Which is better? Exploring Prompting Strategy For LLM-based Metrics(http://arxiv.org/abs/2311.03754)
• Summary:

  This paper describes the DSBA submissions to the Prompting Large Language Models as Explainable Metrics shared task, where systems were submitted to two tracks: small and large summarization tracks. With advanced Large Language Models (LLMs) such as GPT-4, evaluating the quality of Natural Language Generation (NLG) has become increasingly paramount. Traditional similarity-based metrics such as BLEU and ROUGE have shown to misalign with human evaluation and are ill-suited for open-ended generation tasks. To address this issue, we explore the potential capability of LLM-based metrics, especially leveraging open-source LLMs. In this study, wide range of prompts and prompting techniques are systematically analyzed with three approaches: prompting strategy, score aggregation, and explainability. Our research focuses on formulating effective prompt templates, determining the granularity of NLG quality scores and assessing the impact of in-context examples on LLM-based evaluation. Furthermore, three aggregation strategies are compared to identify the most reliable method for aggregating NLG quality scores. To examine explainability, we devise a strategy that generates rationales for the scores and analyzes the characteristics of the explanation produced by the open-source LLMs. Extensive experiments provide insights regarding evaluation capabilities of open-source LLMs and suggest effective prompting strategies.

watermark

diffusion

Title: Reducing Spatial Fitting Error in Distillation of Denoising Diffusion Models. (arXiv:2311.03830v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03830
• Code URL: null
• Copy Paste: [[2311.03830]] Reducing Spatial Fitting Error in Distillation of Denoising Diffusion Models(http://arxiv.org/abs/2311.03830)
• Summary:

  Denoising Diffusion models have exhibited remarkable capabilities in image generation. However, generating high-quality samples requires a large number of iterations. Knowledge distillation for diffusion models is an effective method to address this limitation with a shortened sampling process but causes degraded generative quality. Based on our analysis with bias-variance decomposition and experimental observations, we attribute the degradation to the spatial fitting error occurring in the training of both the teacher and student model. Accordingly, we propose $\textbf{S}$patial $\textbf{F}$itting-$\textbf{E}$rror $\textbf{R}$eduction $\textbf{D}$istillation model ($\textbf{SFERD}$). SFERD utilizes attention guidance from the teacher model and a designed semantic gradient predictor to reduce the student's fitting error. Empirically, our proposed model facilitates high-quality sample generation in a few function evaluations. We achieve an FID of 5.31 on CIFAR-10 and 9.39 on ImageNet 64$\times$64 with only one step, outperforming existing diffusion methods. Our study provides a new perspective on diffusion distillation by highlighting the intrinsic denoising ability of models.

Title: Learning Decentralized Traffic Signal Controllers with Multi-Agent Graph Reinforcement Learning. (arXiv:2311.03756v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.03756
• Code URL: null
• Copy Paste: [[2311.03756]] Learning Decentralized Traffic Signal Controllers with Multi-Agent Graph Reinforcement Learning(http://arxiv.org/abs/2311.03756)
• Summary:

  This paper considers optimal traffic signal control in smart cities, which has been taken as a complex networked system control problem. Given the interacting dynamics among traffic lights and road networks, attaining controller adaptivity and scalability stands out as a primary challenge. Capturing the spatial-temporal correlation among traffic lights under the framework of Multi-Agent Reinforcement Learning (MARL) is a promising solution. Nevertheless, existing MARL algorithms ignore effective information aggregation which is fundamental for improving the learning capacity of decentralized agents. In this paper, we design a new decentralized control architecture with improved environmental observability to capture the spatial-temporal correlation. Specifically, we first develop a topology-aware information aggregation strategy to extract correlation-related information from unstructured data gathered in the road network. Particularly, we transfer the road network topology into a graph shift operator by forming a diffusion process on the topology, which subsequently facilitates the construction of graph signals. A diffusion convolution module is developed, forming a new MARL algorithm, which endows agents with the capabilities of graph learning. Extensive experiments based on both synthetic and real-world datasets verify that our proposal outperforms existing decentralized algorithms.

Title: Formulating Discrete Probability Flow Through Optimal Transport. (arXiv:2311.03886v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.03886
• Code URL: https://github.com/pangzecheung/discrete-probability-flow
• Copy Paste: [[2311.03886]] Formulating Discrete Probability Flow Through Optimal Transport(http://arxiv.org/abs/2311.03886)
• Summary:

  Continuous diffusion models are commonly acknowledged to display a deterministic probability flow, whereas discrete diffusion models do not. In this paper, we aim to establish the fundamental theory for the probability flow of discrete diffusion models. Specifically, we first prove that the continuous probability flow is the Monge optimal transport map under certain conditions, and also present an equivalent evidence for discrete cases. In view of these findings, we are then able to define the discrete probability flow in line with the principles of optimal transport. Finally, drawing upon our newly established definitions, we propose a novel sampling method that surpasses previous discrete diffusion models in its ability to generate more certain outcomes. Extensive experiments on the synthetic toy dataset and the CIFAR-10 dataset have validated the effectiveness of our proposed discrete probability flow. Code is released at: https://github.com/PangzeCheung/Discrete-Probability-Flow.

noise learning

data-free

transformer

Title: GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Values. (arXiv:2311.03426v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.03426
• Code URL: null
• Copy Paste: [[2311.03426]] GQKVA: Efficient Pre-training of Transformers by Grouping Queries, Keys, and Values(http://arxiv.org/abs/2311.03426)
• Summary:

  Massive transformer-based models face several challenges, including slow and computationally intensive pre-training and over-parametrization. This paper addresses these challenges by proposing a versatile method called GQKVA, which generalizes query, key, and value grouping techniques. GQKVA is designed to speed up transformer pre-training while reducing the model size. Our experiments with various GQKVA variants highlight a clear trade-off between performance and model size, allowing for customized choices based on resource and time limitations. Our findings also indicate that the conventional multi-head attention approach is not always the best choice, as there are lighter and faster alternatives available. We tested our method on ViT, which achieved an approximate 0.3% increase in accuracy while reducing the model size by about 4% in the task of image classification. Additionally, our most aggressive model reduction experiment resulted in a reduction of approximately 15% in model size, with only around a 1% drop in accuracy.

Title: TSP-Transformer: Task-Specific Prompts Boosted Transformer for Holistic Scene Understanding. (arXiv:2311.03427v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03427
• Code URL: https://github.com/tb2-sy/tsp-transformer
• Copy Paste: [[2311.03427]] TSP-Transformer: Task-Specific Prompts Boosted Transformer for Holistic Scene Understanding(http://arxiv.org/abs/2311.03427)
• Summary:

  Holistic scene understanding includes semantic segmentation, surface normal estimation, object boundary detection, depth estimation, etc. The key aspect of this problem is to learn representation effectively, as each subtask builds upon not only correlated but also distinct attributes. Inspired by visual-prompt tuning, we propose a Task-Specific Prompts Transformer, dubbed TSP-Transformer, for holistic scene understanding. It features a vanilla transformer in the early stage and tasks-specific prompts transformer encoder in the lateral stage, where tasks-specific prompts are augmented. By doing so, the transformer layer learns the generic information from the shared parts and is endowed with task-specific capacity. First, the tasks-specific prompts serve as induced priors for each task effectively. Moreover, the task-specific prompts can be seen as switches to favor task-specific representation learning for different tasks. Extensive experiments on NYUD-v2 and PASCAL-Context show that our method achieves state-of-the-art performance, validating the effectiveness of our method for holistic scene understanding. We also provide our code in the following link https://github.com/tb2-sy/TSP-Transformer.

Title: High-resolution power equipment recognition based on improved self-attention. (arXiv:2311.03518v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03518
• Code URL: null
• Copy Paste: [[2311.03518]] High-resolution power equipment recognition based on improved self-attention(http://arxiv.org/abs/2311.03518)
• Summary:

  The current trend of automating inspections at substations has sparked a surge in interest in the field of transformer image recognition. However, due to restrictions in the number of parameters in existing models, high-resolution images can't be directly applied, leaving significant room for enhancing recognition accuracy. Addressing this challenge, the paper introduces a novel improvement on deep self-attention networks tailored for this issue. The proposed model comprises four key components: a foundational network, a region proposal network, a module for extracting and segmenting target areas, and a final prediction network. The innovative approach of this paper differentiates itself by decoupling the processes of part localization and recognition, initially using low-resolution images for localization followed by high-resolution images for recognition. Moreover, the deep self-attention network's prediction mechanism uniquely incorporates the semantic context of images, resulting in substantially improved recognition performance. Comparative experiments validate that this method outperforms the two other prevalent target recognition models, offering a groundbreaking perspective for automating electrical equipment inspections.

Title: Cal-DETR: Calibrated Detection Transformer. (arXiv:2311.03570v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03570
• Code URL: https://github.com/akhtarvision/cal-detr
• Copy Paste: [[2311.03570]] Cal-DETR: Calibrated Detection Transformer(http://arxiv.org/abs/2311.03570)
• Summary:

  Albeit revealing impressive predictive performance for several computer vision tasks, deep neural networks (DNNs) are prone to making overconfident predictions. This limits the adoption and wider utilization of DNNs in many safety-critical applications. There have been recent efforts toward calibrating DNNs, however, almost all of them focus on the classification task. Surprisingly, very little attention has been devoted to calibrating modern DNN-based object detectors, especially detection transformers, which have recently demonstrated promising detection performance and are influential in many decision-making systems. In this work, we address the problem by proposing a mechanism for calibrated detection transformers (Cal-DETR), particularly for Deformable-DETR, UP-DETR and DINO. We pursue the train-time calibration route and make the following contributions. First, we propose a simple yet effective approach for quantifying uncertainty in transformer-based object detectors. Second, we develop an uncertainty-guided logit modulation mechanism that leverages the uncertainty to modulate the class logits. Third, we develop a logit mixing approach that acts as a regularizer with detection-specific losses and is also complementary to the uncertainty-guided logit modulation technique to further improve the calibration performance. Lastly, we conduct extensive experiments across three in-domain and four out-domain scenarios. Results corroborate the effectiveness of Cal-DETR against the competing train-time methods in calibrating both in-domain and out-domain detections while maintaining or even improving the detection performance. Our codebase and pre-trained models can be accessed at \url{https://github.com/akhtarvision/cal-detr}.

Title: FusionViT: Hierarchical 3D Object Detection via LiDAR-Camera Vision Transformer Fusion. (arXiv:2311.03620v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03620
• Code URL: null
• Copy Paste: [[2311.03620]] FusionViT: Hierarchical 3D Object Detection via LiDAR-Camera Vision Transformer Fusion(http://arxiv.org/abs/2311.03620)
• Summary:

  For 3D object detection, both camera and lidar have been demonstrated to be useful sensory devices for providing complementary information about the same scenery with data representations in different modalities, e.g., 2D RGB image vs 3D point cloud. An effective representation learning and fusion of such multi-modal sensor data is necessary and critical for better 3D object detection performance. To solve the problem, in this paper, we will introduce a novel vision transformer-based 3D object detection model, namely FusionViT. Different from the existing 3D object detection approaches, FusionViT is a pure-ViT based framework, which adopts a hierarchical architecture by extending the transformer model to embed both images and point clouds for effective representation learning. Such multi-modal data embedding representations will be further fused together via a fusion vision transformer model prior to feeding the learned features to the object detection head for both detection and localization of the 3D objects in the input scenery. To demonstrate the effectiveness of FusionViT, extensive experiments have been done on real-world traffic object detection benchmark datasets KITTI and Waymo Open. Notably, our FusionViT model can achieve state-of-the-art performance and outperforms not only the existing baseline methods that merely rely on camera images or lidar point clouds, but also the latest multi-modal image-point cloud deep fusion approaches.

Title: SBCFormer: Lightweight Network Capable of Full-size ImageNet Classification at 1 FPS on Single Board Computers. (arXiv:2311.03747v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03747
• Code URL: https://github.com/xyonglu/sbcformer
• Copy Paste: [[2311.03747]] SBCFormer: Lightweight Network Capable of Full-size ImageNet Classification at 1 FPS on Single Board Computers(http://arxiv.org/abs/2311.03747)
• Summary:

  Computer vision has become increasingly prevalent in solving real-world problems across diverse domains, including smart agriculture, fishery, and livestock management. These applications may not require processing many image frames per second, leading practitioners to use single board computers (SBCs). Although many lightweight networks have been developed for mobile/edge devices, they primarily target smartphones with more powerful processors and not SBCs with the low-end CPUs. This paper introduces a CNN-ViT hybrid network called SBCFormer, which achieves high accuracy and fast computation on such low-end CPUs. The hardware constraints of these CPUs make the Transformer's attention mechanism preferable to convolution. However, using attention on low-end CPUs presents a challenge: high-resolution internal feature maps demand excessive computational resources, but reducing their resolution results in the loss of local image details. SBCFormer introduces an architectural design to address this issue. As a result, SBCFormer achieves the highest trade-off between accuracy and speed on a Raspberry Pi 4 Model B with an ARM-Cortex A72 CPU. For the first time, it achieves an ImageNet-1K top-1 accuracy of around 80% at a speed of 1.0 frame/sec on the SBC. Code is available at https://github.com/xyongLu/SBCFormer.

Title: Lightweight Portrait Matting via Regional Attention and Refinement. (arXiv:2311.03770v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03770
• Code URL: null
• Copy Paste: [[2311.03770]] Lightweight Portrait Matting via Regional Attention and Refinement(http://arxiv.org/abs/2311.03770)
• Summary:

  We present a lightweight model for high resolution portrait matting. The model does not use any auxiliary inputs such as trimaps or background captures and achieves real time performance for HD videos and near real time for 4K. Our model is built upon a two-stage framework with a low resolution network for coarse alpha estimation followed by a refinement network for local region improvement. However, a naive implementation of the two-stage model suffers from poor matting quality if not utilizing any auxiliary inputs. We address the performance gap by leveraging the vision transformer (ViT) as the backbone of the low resolution network, motivated by the observation that the tokenization step of ViT can reduce spatial resolution while retain as much pixel information as possible. To inform local regions of the context, we propose a novel cross region attention (CRA) module in the refinement network to propagate the contextual information across the neighboring regions. We demonstrate that our method achieves superior results and outperforms other baselines on three benchmark datasets while only uses $1/20$ of the FLOPS compared to the existing state-of-the-art model.

Title: Mini but Mighty: Finetuning ViTs with Mini Adapters. (arXiv:2311.03873v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03873
• Code URL: https://github.com/iemprog/mimi
• Copy Paste: [[2311.03873]] Mini but Mighty: Finetuning ViTs with Mini Adapters(http://arxiv.org/abs/2311.03873)
• Summary:

  Vision Transformers (ViTs) have become one of the dominant architectures in computer vision, and pre-trained ViT models are commonly adapted to new tasks via fine-tuning. Recent works proposed several parameter-efficient transfer learning methods, such as adapters, to avoid the prohibitive training and storage cost of finetuning. In this work, we observe that adapters perform poorly when the dimension of adapters is small, and we propose MiMi, a training framework that addresses this issue. We start with large adapters which can reach high performance, and iteratively reduce their size. To enable automatic estimation of the hidden dimension of every adapter, we also introduce a new scoring function, specifically designed for adapters, that compares the neuron importance across layers. Our method outperforms existing methods in finding the best trade-off between accuracy and trained parameters across the three dataset benchmarks DomainNet, VTAB, and Multi-task, for a total of 29 datasets.

Title: FLORA: Fine-grained Low-Rank Architecture Search for Vision Transformer. (arXiv:2311.03912v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03912
• Code URL: https://github.com/shadowpa0327/flora
• Copy Paste: [[2311.03912]] FLORA: Fine-grained Low-Rank Architecture Search for Vision Transformer(http://arxiv.org/abs/2311.03912)
• Summary:

  Vision Transformers (ViT) have recently demonstrated success across a myriad of computer vision tasks. However, their elevated computational demands pose significant challenges for real-world deployment. While low-rank approximation stands out as a renowned method to reduce computational loads, efficiently automating the target rank selection in ViT remains a challenge. Drawing from the notable similarity and alignment between the processes of rank selection and One-Shot NAS, we introduce FLORA, an end-to-end automatic framework based on NAS. To overcome the design challenge of supernet posed by vast search space, FLORA employs a low-rank aware candidate filtering strategy. This method adeptly identifies and eliminates underperforming candidates, effectively alleviating potential undertraining and interference among subnetworks. To further enhance the quality of low-rank supernets, we design a low-rank specific training paradigm. First, we propose weight inheritance to construct supernet and enable gradient sharing among low-rank modules. Secondly, we adopt low-rank aware sampling to strategically allocate training resources, taking into account inherited information from pre-trained models. Empirical results underscore FLORA's efficacy. With our method, a more fine-grained rank configuration can be generated automatically and yield up to 33% extra FLOPs reduction compared to a simple uniform configuration. More specific, FLORA-DeiT-B/FLORA-Swin-B can save up to 55%/42% FLOPs almost without performance degradtion. Importantly, FLORA boasts both versatility and orthogonality, offering an extra 21%-26% FLOPs reduction when integrated with leading compression techniques or compact hybrid structures. Our code is publicly available at https://github.com/shadowpa0327/FLORA.

Title: Character-Level Bangla Text-to-IPA Transcription Using Transformer Architecture with Sequence Alignment. (arXiv:2311.03792v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.03792
• Code URL: null
• Copy Paste: [[2311.03792]] Character-Level Bangla Text-to-IPA Transcription Using Transformer Architecture with Sequence Alignment(http://arxiv.org/abs/2311.03792)
• Summary:

  The International Phonetic Alphabet (IPA) is indispensable in language learning and understanding, aiding users in accurate pronunciation and comprehension. Additionally, it plays a pivotal role in speech therapy, linguistic research, accurate transliteration, and the development of text-to-speech systems, making it an essential tool across diverse fields. Bangla being 7th as one of the widely used languages, gives rise to the need for IPA in its domain. Its IPA mapping is too diverse to be captured manually giving the need for Artificial Intelligence and Machine Learning in this field. In this study, we have utilized a transformer-based sequence-to-sequence model at the letter and symbol level to get the IPA of each Bangla word as the variation of IPA in association of different words is almost null. Our transformer model only consisted of 8.5 million parameters with only a single decoder and encoder layer. Additionally, to handle the punctuation marks and the occurrence of foreign languages in the text, we have utilized manual mapping as the model won't be able to learn to separate them from Bangla words while decreasing our required computational resources. Finally, maintaining the relative position of the sentence component IPAs and generation of the combined IPA has led us to achieve the top position with a word error rate of 0.10582 in the public ranking of DataVerse Challenge - ITVerse 2023 (https://www.kaggle.com/competitions/dataverse_2023/).

generative

Title: An attempt to generate new bridge types from latent space of variational autoencoder. (arXiv:2311.03380v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.03380
• Code URL: https://github.com/QQ583304953/Bridge-VAE
• Copy Paste: [[2311.03380]] An attempt to generate new bridge types from latent space of variational autoencoder(http://arxiv.org/abs/2311.03380)
• Summary:

  Try to generate new bridge types using generative artificial intelligence technology. The grayscale images of the bridge facade with the change of component width was rendered by 3dsMax animation software, and then the OpenCV module performed an appropriate amount of geometric transformation (rotation, horizontal scale, vertical scale) to obtain the image dataset of three-span beam bridge, arch bridge, cable-stayed bridge and suspension bridge. Based on Python programming language, TensorFlow and Keras deep learning platform framework, variational autoencoder was constructed and trained, and low-dimensional bridge-type latent space that is convenient for vector operations was obtained. Variational autoencoder can combine two bridge types on the basis of the original of human into one that is a new bridge type. Generative artificial intelligence technology can assist bridge designers in bridge-type innovation, and can be used as copilot.

Title: Unsupervised Video Summarization. (arXiv:2311.03745v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03745
• Code URL: null
• Copy Paste: [[2311.03745]] Unsupervised Video Summarization(http://arxiv.org/abs/2311.03745)
• Summary:

  This paper introduces a new, unsupervised method for automatic video summarization using ideas from generative adversarial networks but eliminating the discriminator, having a simple loss function, and separating training of different parts of the model. An iterative training strategy is also applied by alternately training the reconstructor and the frame selector for multiple iterations. Furthermore, a trainable mask vector is added to the model in summary generation during training and evaluation. The method also includes an unsupervised model selection algorithm. Results from experiments on two public datasets (SumMe and TVSum) and four datasets we created (Soccer, LoL, MLB, and ShortMLB) demonstrate the effectiveness of each component on the model performance, particularly the iterative training strategy. Evaluations and comparisons with the state-of-the-art methods highlight the advantages of the proposed method in performance, stability, and training efficiency.

Title: SCONE-GAN: Semantic Contrastive learning-based Generative Adversarial Network for an end-to-end image translation. (arXiv:2311.03866v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03866
• Code URL: null
• Copy Paste: [[2311.03866]] SCONE-GAN: Semantic Contrastive learning-based Generative Adversarial Network for an end-to-end image translation(http://arxiv.org/abs/2311.03866)
• Summary:

  SCONE-GAN presents an end-to-end image translation, which is shown to be effective for learning to generate realistic and diverse scenery images. Most current image-to-image translation approaches are devised as two mappings: a translation from the source to target domain and another to represent its inverse. While successful in many applications, these approaches may suffer from generating trivial solutions with limited diversity. That is because these methods learn more frequent associations rather than the scene structures. To mitigate the problem, we propose SCONE-GAN that utilises graph convolutional networks to learn the objects dependencies, maintain the image structure and preserve its semantics while transferring images into the target domain. For more realistic and diverse image generation we introduce style reference image. We enforce the model to maximize the mutual information between the style image and output. The proposed method explicitly maximizes the mutual information between the related patches, thus encouraging the generator to produce more diverse images. We validate the proposed algorithm for image-to-image translation and stylizing outdoor images. Both qualitative and quantitative results demonstrate the effectiveness of our approach on four dataset.

Title: Improving the Effectiveness of Deep Generative Data. (arXiv:2311.03959v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03959
• Code URL: null
• Copy Paste: [[2311.03959]] Improving the Effectiveness of Deep Generative Data(http://arxiv.org/abs/2311.03959)
• Summary:

  Recent deep generative models (DGMs) such as generative adversarial networks (GANs) and diffusion probabilistic models (DPMs) have shown their impressive ability in generating high-fidelity photorealistic images. Although looking appealing to human eyes, training a model on purely synthetic images for downstream image processing tasks like image classification often results in an undesired performance drop compared to training on real data. Previous works have demonstrated that enhancing a real dataset with synthetic images from DGMs can be beneficial. However, the improvements were subjected to certain circumstances and yet were not comparable to adding the same number of real images. In this work, we propose a new taxonomy to describe factors contributing to this commonly observed phenomenon and investigate it on the popular CIFAR-10 dataset. We hypothesize that the Content Gap accounts for a large portion of the performance drop when using synthetic images from DGM and propose strategies to better utilize them in downstream tasks. Extensive experiments on multiple datasets showcase that our method outperforms baselines on downstream classification tasks both in case of training on synthetic only (Synthetic-to-Real) and training on a mix of real and synthetic data (Data Augmentation), particularly in the data-scarce scenario.

Title: Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Mining. (arXiv:2311.03964v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03964
• Code URL: null
• Copy Paste: [[2311.03964]] Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Mining(http://arxiv.org/abs/2311.03964)
• Summary:

  Contemporary large-scale visual language models (VLMs) exhibit strong representation capacities, making them ubiquitous for enhancing image and text understanding tasks. They are often trained in a contrastive manner on a large and diverse corpus of images and corresponding text captions scraped from the internet. Despite this, VLMs often struggle with compositional reasoning tasks which require a fine-grained understanding of the complex interactions of objects and their attributes. This failure can be attributed to two main factors: 1) Contrastive approaches have traditionally focused on mining negative examples from existing datasets. However, the mined negative examples might not be difficult for the model to discriminate from the positive. An alternative to mining would be negative sample generation 2) But existing generative approaches primarily focus on generating hard negative texts associated with a given image. Mining in the other direction, i.e., generating negative image samples associated with a given text has been ignored. To overcome both these limitations, we propose a framework that not only mines in both directions but also generates challenging negative samples in both modalities, i.e., images and texts. Leveraging these generative hard negative samples, we significantly enhance VLMs' performance in tasks involving multimodal compositional reasoning. Our code and dataset are released at https://ugorsahin.github.io/enhancing-multimodal-compositional-reasoning-of-vlm.html.

Title: Training Multi-layer Neural Networks on Ising Machine. (arXiv:2311.03408v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.03408
• Code URL: null
• Copy Paste: [[2311.03408]] Training Multi-layer Neural Networks on Ising Machine(http://arxiv.org/abs/2311.03408)
• Summary:

  As a dedicated quantum device, Ising machines could solve large-scale binary optimization problems in milliseconds. There is emerging interest in utilizing Ising machines to train feedforward neural networks due to the prosperity of generative artificial intelligence. However, existing methods can only train single-layer feedforward networks because of the complex nonlinear network topology. This paper proposes an Ising learning algorithm to train quantized neural network (QNN), by incorporating two essential techinques, namely binary representation of topological network and order reduction of loss function. As far as we know, this is the first algorithm to train multi-layer feedforward networks on Ising machines, providing an alternative to gradient-based backpropagation. Firstly, training QNN is formulated as a quadratic constrained binary optimization (QCBO) problem by representing neuron connection and activation function as equality constraints. All quantized variables are encoded by binary bits based on binary encoding protocol. Secondly, QCBO is converted to a quadratic unconstrained binary optimization (QUBO) problem, that can be efficiently solved on Ising machines. The conversion leverages both penalty function and Rosenberg order reduction, who together eliminate equality constraints and reduce high-order loss function into a quadratic one. With some assumptions, theoretical analysis shows the space complexity of our algorithm is $\mathcal{O}(H^2L + HLN\log H)$, quantifying the required number of Ising spins. Finally, the algorithm effectiveness is validated with a simulated Ising machine on MNIST dataset. After annealing 700 ms, the classification accuracy achieves 98.3%. Among 100 runs, the success probability of finding the optimal solution is 72%. Along with the increasing number of spins on Ising machine, our algorithm has the potential to train deeper neural networks.

Title: A Generative Neural Network Approach for 3D Multi-Criteria Design Generation and Optimization of an Engine Mount for an Unmanned Air Vehicle. (arXiv:2311.03414v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2311.03414
• Code URL: null
• Copy Paste: [[2311.03414]] A Generative Neural Network Approach for 3D Multi-Criteria Design Generation and Optimization of an Engine Mount for an Unmanned Air Vehicle(http://arxiv.org/abs/2311.03414)
• Summary:

  One of the most promising developments in computer vision in recent years is the use of generative neural networks for functionality condition-based 3D design reconstruction and generation. Here, neural networks learn dependencies between functionalities and a geometry in a very effective way. For a neural network the functionalities are translated in conditions to a certain geometry. But the more conditions the design generation needs to reflect, the more difficult it is to learn clear dependencies. This leads to a multi criteria design problem due various conditions, which are not considered in the neural network structure so far.

  In this paper, we address this multi-criteria challenge for a 3D design use case related to an unmanned aerial vehicle (UAV) motor mount. We generate 10,000 abstract 3D designs and subject them all to simulations for three physical disciplines: mechanics, thermodynamics, and aerodynamics. Then, we train a Conditional Variational Autoencoder (CVAE) using the geometry and corresponding multicriteria functional constraints as input. We use our trained CVAE as well as the Marching cubes algorithm to generate meshes for simulation based evaluation. The results are then evaluated with the generated UAV designs. Subsequently, we demonstrate the ability to generate optimized designs under self-defined functionality conditions using the trained neural network.

large language model

Title: LLM as an Art Director (LaDi): Using LLMs to improve Text-to-Media Generators. (arXiv:2311.03716v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.03716
• Code URL: null
• Copy Paste: [[2311.03716]] LLM as an Art Director (LaDi): Using LLMs to improve Text-to-Media Generators(http://arxiv.org/abs/2311.03716)
• Summary:

  Recent advancements in text-to-image generation have revolutionized numerous fields, including art and cinema, by automating the generation of high-quality, context-aware images and video. However, the utility of these technologies is often limited by the inadequacy of text prompts in guiding the generator to produce artistically coherent and subject-relevant images. In this paper, We describe the techniques that can be used to make Large Language Models (LLMs) act as Art Directors that enhance image and video generation. We describe our unified system for this called "LaDi". We explore how LaDi integrates multiple techniques for augmenting the capabilities of text-to-image generators (T2Is) and text-to-video generators (T2Vs), with a focus on constrained decoding, intelligent prompting, fine-tuning, and retrieval. LaDi and these techniques are being used today in apps and platforms developed by Plai Labs.

Title: In-Context Exemplars as Clues to Retrieving from Large Associative Memory. (arXiv:2311.03498v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.03498
• Code URL: null
• Copy Paste: [[2311.03498]] In-Context Exemplars as Clues to Retrieving from Large Associative Memory(http://arxiv.org/abs/2311.03498)
• Summary:

  Recently, large language models (LLMs) have made remarkable progress in natural language processing. The most representative ability of LLMs is in-context learning (ICL), which enables LLMs to learn patterns from in-context exemplars without training. The performance of ICL greatly depends on the exemplars used. However, how to choose exemplars remains unclear due to the lack of understanding of how in-context learning works. In this paper, we present a novel perspective on ICL by conceptualizing it as contextual retrieval from a model of associative memory. We establish a theoretical framework of ICL based on Hopfield Networks. Based on our framework, we look into how in-context exemplars influence the performance of ICL and propose more efficient active exemplar selection. Our study sheds new light on the mechanism of ICL by connecting it to memory retrieval, with potential implications for advancing the understanding of LLMs.

Title: Quantifying Uncertainty in Natural Language Explanations of Large Language Models. (arXiv:2311.03533v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.03533
• Code URL: null
• Copy Paste: [[2311.03533]] Quantifying Uncertainty in Natural Language Explanations of Large Language Models(http://arxiv.org/abs/2311.03533)
• Summary:

  Large Language Models (LLMs) are increasingly used as powerful tools for several high-stakes natural language processing (NLP) applications. Recent prompting works claim to elicit intermediate reasoning steps and key tokens that serve as proxy explanations for LLM predictions. However, there is no certainty whether these explanations are reliable and reflect the LLMs behavior. In this work, we make one of the first attempts at quantifying the uncertainty in explanations of LLMs. To this end, we propose two novel metrics -- $\textit{Verbalized Uncertainty}$ and $\textit{Probing Uncertainty}$ -- to quantify the uncertainty of generated explanations. While verbalized uncertainty involves prompting the LLM to express its confidence in its explanations, probing uncertainty leverages sample and model perturbations as a means to quantify the uncertainty. Our empirical analysis of benchmark datasets reveals that verbalized uncertainty is not a reliable estimate of explanation confidence. Further, we show that the probing uncertainty estimates are correlated with the faithfulness of an explanation, with lower uncertainty corresponding to explanations with higher faithfulness. Our study provides insights into the challenges and opportunities of quantifying uncertainty in LLM explanations, contributing to the broader discussion of the trustworthiness of foundation models.

Title: Context Unlocks Emotions: Text-based Emotion Classification Dataset Auditing with Large Language Models. (arXiv:2311.03551v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.03551
• Code URL: null
• Copy Paste: [[2311.03551]] Context Unlocks Emotions: Text-based Emotion Classification Dataset Auditing with Large Language Models(http://arxiv.org/abs/2311.03551)
• Summary:

  The lack of contextual information in text data can make the annotation process of text-based emotion classification datasets challenging. As a result, such datasets often contain labels that fail to consider all the relevant emotions in the vocabulary. This misalignment between text inputs and labels can degrade the performance of machine learning models trained on top of them. As re-annotating entire datasets is a costly and time-consuming task that cannot be done at scale, we propose to use the expressive capabilities of large language models to synthesize additional context for input text to increase its alignment with the annotated emotional labels. In this work, we propose a formal definition of textual context to motivate a prompting strategy to enhance such contextual information. We provide both human and empirical evaluation to demonstrate the efficacy of the enhanced context. Our method improves alignment between inputs and their human-annotated labels from both an empirical and human-evaluated standpoint.

Title: The Linear Representation Hypothesis and the Geometry of Large Language Models. (arXiv:2311.03658v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.03658
• Code URL: null
• Copy Paste: [[2311.03658]] The Linear Representation Hypothesis and the Geometry of Large Language Models(http://arxiv.org/abs/2311.03658)
• Summary:

  Informally, the 'linear representation hypothesis' is the idea that high-level concepts are represented linearly as directions in some representation space. In this paper, we address two closely related questions: What does "linear representation" actually mean? And, how do we make sense of geometric notions (e.g., cosine similarity or projection) in the representation space? To answer these, we use the language of counterfactuals to give two formalizations of "linear representation", one in the output (word) representation space, and one in the input (sentence) space. We then prove these connect to linear probing and model steering, respectively. To make sense of geometric notions, we use the formalization to identify a particular (non-Euclidean) inner product that respects language structure in a sense we make precise. Using this causal inner product, we show how to unify all notions of linear representation. In particular, this allows the construction of probes and steering vectors using counterfactual pairs. Experiments with LLaMA-2 demonstrate the existence of linear representations of concepts, the connection to interpretation and control, and the fundamental role of the choice of inner product.

Title: A Survey of Large Language Models Attribution. (arXiv:2311.03731v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.03731
• Code URL: null
• Copy Paste: [[2311.03731]] A Survey of Large Language Models Attribution(http://arxiv.org/abs/2311.03731)
• Summary:

  Open-domain generative systems have gained significant attention in the field of conversational AI (e.g., generative search engines). This paper presents a comprehensive review of the attribution mechanisms employed by these systems, particularly large language models. Though attribution or citation improve the factuality and verifiability, issues like ambiguous knowledge reservoirs, inherent biases, and the drawbacks of excessive attribution can hinder the effectiveness of these systems. The aim of this survey is to provide valuable insights for researchers, aiding in the refinement of attribution methodologies to enhance the reliability and veracity of responses generated by open-domain generative systems. We believe that this field is still in its early stages; hence, we maintain a repository to keep track of ongoing studies at https://github.com/HITsz-TMG/awesome-llm-attributions.

Title: Conversations in Galician: a Large Language Model for an Underrepresented Language. (arXiv:2311.03812v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.03812
• Code URL: https://gitlab.irlab.org/irlab/cabuxa
• Copy Paste: [[2311.03812]] Conversations in Galician: a Large Language Model for an Underrepresented Language(http://arxiv.org/abs/2311.03812)
• Summary:

  The recent proliferation of Large Conversation Language Models has highlighted the economic significance of widespread access to this type of AI technologies in the current information age. Nevertheless, prevailing models have primarily been trained on corpora consisting of documents written in popular languages. The dearth of such cutting-edge tools for low-resource languages further exacerbates their underrepresentation in the current economic landscape, thereby impacting their native speakers. This paper introduces two novel resources designed to enhance Natural Language Processing (NLP) for the Galician language. We present a Galician adaptation of the Alpaca dataset, comprising 52,000 instructions and demonstrations. This dataset proves invaluable for enhancing language models by fine-tuning them to more accurately adhere to provided instructions. Additionally, as a demonstration of the dataset utility, we fine-tuned LLaMA-7B to comprehend and respond in Galician, a language not originally supported by the model, by following the Alpaca format. This work contributes to the research on multilingual models tailored for low-resource settings, a crucial endeavor in ensuring the inclusion of all linguistic communities in the development of Large Language Models. Another noteworthy aspect of this research is the exploration of how knowledge of a closely related language, in this case, Portuguese, can assist in generating coherent text when training resources are scarce. Both the Galician Alpaca dataset and Cabuxa-7B are publicly accessible on our Huggingface Hub, and we have made the source code available to facilitate replication of this experiment and encourage further advancements for underrepresented languages.

Title: Aspects of human memory and Large Language Models. (arXiv:2311.03839v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2311.03839
• Code URL: https://github.com/rmldj/memory-llm-paper
• Copy Paste: [[2311.03839]] Aspects of human memory and Large Language Models(http://arxiv.org/abs/2311.03839)
• Summary:

  Large Language Models (LLMs) are huge artificial neural networks which primarily serve to generate text, but also provide a very sophisticated probabilistic model of language use. Since generating a semantically consistent text requires a form of effective memory, we investigate the memory properties of LLMs and find surprising similarities with key characteristics of human memory. This result strongly suggests that the biological features of human memory leave an imprint on the way that we structure our textual narratives.

segmentation

Title: Leveraging point annotations in segmentation learning with boundary loss. (arXiv:2311.03537v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03537
• Code URL: null
• Copy Paste: [[2311.03537]] Leveraging point annotations in segmentation learning with boundary loss(http://arxiv.org/abs/2311.03537)
• Summary:

  This paper investigates the combination of intensity-based distance maps with boundary loss for point-supervised semantic segmentation. By design the boundary loss imposes a stronger penalty on the false positives the farther away from the object they occur. Hence it is intuitively inappropriate for weak supervision, where the ground truth label may be much smaller than the actual object and a certain amount of false positives (w.r.t. the weak ground truth) is actually desirable. Using intensity-aware distances instead may alleviate this drawback, allowing for a certain amount of false positives without a significant increase to the training loss. The motivation for applying the boundary loss directly under weak supervision lies in its great success for fully supervised segmentation tasks, but also in not requiring extra priors or outside information that is usually required -- in some form -- with existing weakly supervised methods in the literature. This formulation also remains potentially more attractive than existing CRF-based regularizers, due to its simplicity and computational efficiency. We perform experiments on two multi-class datasets; ACDC (heart segmentation) and POEM (whole-body abdominal organ segmentation). Preliminary results are encouraging and show that this supervision strategy has great potential. On ACDC it outperforms the CRF-loss based approach, and on POEM data it performs on par with it. The code for all our experiments is openly available.

Title: Unsupervised Region-Growing Network for Object Segmentation in Atmospheric Turbulence. (arXiv:2311.03572v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03572
• Code URL: null
• Copy Paste: [[2311.03572]] Unsupervised Region-Growing Network for Object Segmentation in Atmospheric Turbulence(http://arxiv.org/abs/2311.03572)
• Summary:

  In this paper, we present a two-stage unsupervised foreground object segmentation network tailored for dynamic scenes affected by atmospheric turbulence. In the first stage, we utilize averaged optical flow from turbulence-distorted image sequences to feed a novel region-growing algorithm, crafting preliminary masks for each moving object in the video. In the second stage, we employ a U-Net architecture with consistency and grouping losses to further refine these masks optimizing their spatio-temporal alignment. Our approach does not require labeled training data and works across varied turbulence strengths for long-range video. Furthermore, we release the first moving object segmentation dataset of turbulence-affected videos, complete with manually annotated ground truth masks. Our method, evaluated on this new dataset, demonstrates superior segmentation accuracy and robustness as compared to current state-of-the-art unsupervised methods.

Title: Instruct Me More! Random Prompting for Visual In-Context Learning. (arXiv:2311.03648v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03648
• Code URL: null
• Copy Paste: [[2311.03648]] Instruct Me More! Random Prompting for Visual In-Context Learning(http://arxiv.org/abs/2311.03648)
• Summary:

  Large-scale models trained on extensive datasets, have emerged as the preferred approach due to their high generalizability across various tasks. In-context learning (ICL), a popular strategy in natural language processing, uses such models for different tasks by providing instructive prompts but without updating model parameters. This idea is now being explored in computer vision, where an input-output image pair (called an in-context pair) is supplied to the model with a query image as a prompt to exemplify the desired output. The efficacy of visual ICL often depends on the quality of the prompts. We thus introduce a method coined Instruct Me More (InMeMo), which augments in-context pairs with a learnable perturbation (prompt), to explore its potential. Our experiments on mainstream tasks reveal that InMeMo surpasses the current state-of-the-art performance. Specifically, compared to the baseline without learnable prompt, InMeMo boosts mIoU scores by 7.35 and 15.13 for foreground segmentation and single object detection tasks, respectively. Our findings suggest that InMeMo offers a versatile and efficient way to enhance the performance of visual ICL with lightweight training. Code is available at https://github.com/Jackieam/InMeMo.

Title: Multiclass Segmentation using Teeth Attention Modules for Dental X-ray Images. (arXiv:2311.03749v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03749
• Code URL: null
• Copy Paste: [[2311.03749]] Multiclass Segmentation using Teeth Attention Modules for Dental X-ray Images(http://arxiv.org/abs/2311.03749)
• Summary:

  This paper proposed a cutting-edge multiclass teeth segmentation architecture that integrates an M-Net-like structure with Swin Transformers and a novel component named Teeth Attention Block (TAB). Existing teeth image segmentation methods have issues with less accurate and unreliable segmentation outcomes due to the complex and varying morphology of teeth, although teeth segmentation in dental panoramic images is essential for dental disease diagnosis. We propose a novel teeth segmentation model incorporating an M-Net-like structure with Swin Transformers and TAB. The proposed TAB utilizes a unique attention mechanism that focuses specifically on the complex structures of teeth. The attention mechanism in TAB precisely highlights key elements of teeth features in panoramic images, resulting in more accurate segmentation outcomes. The proposed architecture effectively captures local and global contextual information, accurately defining each tooth and its surrounding structures. Furthermore, we employ a multiscale supervision strategy, which leverages the left and right legs of the U-Net structure, boosting the performance of the segmentation with enhanced feature representation. The squared Dice loss is utilized to tackle the class imbalance issue, ensuring accurate segmentation across all classes. The proposed method was validated on a panoramic teeth X-ray dataset, which was taken in a real-world dental diagnosis. The experimental results demonstrate the efficacy of our proposed architecture for tooth segmentation on multiple benchmark dental image datasets, outperforming existing state-of-the-art methods in objective metrics and visual examinations. This study has the potential to significantly enhance dental image analysis and contribute to advances in dental applications.

Title: Meta-Adapter: An Online Few-shot Learner for Vision-Language Model. (arXiv:2311.03774v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03774
• Code URL: null
• Copy Paste: [[2311.03774]] Meta-Adapter: An Online Few-shot Learner for Vision-Language Model(http://arxiv.org/abs/2311.03774)
• Summary:

  The contrastive vision-language pre-training, known as CLIP, demonstrates remarkable potential in perceiving open-world visual concepts, enabling effective zero-shot image recognition. Nevertheless, few-shot learning methods based on CLIP typically require offline fine-tuning of the parameters on few-shot samples, resulting in longer inference time and the risk of over-fitting in certain domains. To tackle these challenges, we propose the Meta-Adapter, a lightweight residual-style adapter, to refine the CLIP features guided by the few-shot samples in an online manner. With a few training samples, our method can enable effective few-shot learning capabilities and generalize to unseen data or tasks without additional fine-tuning, achieving competitive performance and high efficiency. Without bells and whistles, our approach outperforms the state-of-the-art online few-shot learning method by an average of 3.6\% on eight image classification datasets with higher inference speed. Furthermore, our model is simple and flexible, serving as a plug-and-play module directly applicable to downstream tasks. Without further fine-tuning, Meta-Adapter obtains notable performance improvements in open-vocabulary object detection and segmentation tasks.

Title: A Comparative Study of Knowledge Transfer Methods for Misaligned Urban Building Labels. (arXiv:2311.03867v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2311.03867
• Code URL: null
• Copy Paste: [[2311.03867]] A Comparative Study of Knowledge Transfer Methods for Misaligned Urban Building Labels(http://arxiv.org/abs/2311.03867)
• Summary:

  Misalignment in Earth observation (EO) images and building labels impact the training of accurate convolutional neural networks (CNNs) for semantic segmentation of building footprints. Recently, three Teacher-Student knowledge transfer methods have been introduced to address this issue: supervised domain adaptation (SDA), knowledge distillation (KD), and deep mutual learning (DML). However, these methods are merely studied for different urban buildings (low-rise, mid-rise, high-rise, and skyscrapers), where misalignment increases with building height and spatial resolution. In this study, we present a workflow for the systematic comparative study of the three methods. The workflow first identifies the best (with the highest evaluation scores) hyperparameters, lightweight CNNs for the Student (among 43 CNNs from Computer Vision), and encoder-decoder networks (EDNs) for both Teachers and Students. Secondly, three building footprint datasets are developed to train and evaluate the identified Teachers and Students in the three transfer methods. The results show that U-Net with VGG19 (U-VGG19) is the best Teacher, and U-EfficientNetv2B3 and U-EfficientNet-lite0 are among the best Students. With these Teacher-Student pairs, SDA could yield upto 0.943, 0.868, 0.912, and 0.697 F1 scores in the low-rise, mid-rise, high-rise, and skyscrapers respectively. KD and DML provide model compression of upto 82%, despite marginal loss in performance. This new comparison concludes that SDA is the most effective method to address the misalignment problem, while KD and DML can efficiently compress network size without significant loss in performance. The 158 experiments and datasets developed in this study will be valuable to minimise the misaligned labels.