2023-12-21

secure

Title: DynamiQS: Quantum Secure Authentication for Dynamic Charging of Electric Vehicles. (arXiv:2312.12879v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.12879
• Code URL: null
• Copy Paste: [[2312.12879]] DynamiQS: Quantum Secure Authentication for Dynamic Charging of Electric Vehicles(http://arxiv.org/abs/2312.12879)
• Summary:

  Dynamic Wireless Power Transfer (DWPT) is a novel technology that allows charging an electric vehicle while driving thanks to a dedicated road infrastructure. DWPT's capabilities in automatically establishing charging sessions and billing without users' intervention make it prone to cybersecurity attacks. Hence, security is essential in preventing fraud, impersonation, and user tracking. To this aim, researchers proposed different solutions for authenticating users. However, recent advancements in quantum computing jeopardize classical public key cryptography, making currently existing solutions in DWPT authentication nonviable. To avoid the resource burden imposed by technology upgrades, it is essential to develop post-quantum-resistant solutions. In this paper, we propose DynamiQS, the first post-quantum secure authentication protocol for dynamic wireless charging. DynamiQS is privacy-preserving and secure against attacks on the DWPT. We leverage an Identity-Based Encryption with Lattices in the Ring Learning With Error framework. Furthermore, we show the possibility of using DynamiQS in a real environment, leveraging the results of cryptographic computation on real constrained devices and simulations. DynamiQS reaches a total time cost of around 281 ms, which is practicable in dynamic charging settings (car and charging infrastructure).

Title: Secure Authentication Mechanism for Cluster based Vehicular Adhoc Network (VANET): A Survey. (arXiv:2312.12925v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.12925
• Code URL: null
• Copy Paste: [[2312.12925]] Secure Authentication Mechanism for Cluster based Vehicular Adhoc Network (VANET): A Survey(http://arxiv.org/abs/2312.12925)
• Summary:

  Vehicular Ad Hoc Networks (VANETs) play a crucial role in Intelligent Transportation Systems (ITS) by facilitating communication between vehicles and infrastructure. This communication aims to enhance road safety, improve traffic efficiency, and enhance passenger comfort. The secure and reliable exchange of information is paramount to ensure the integrity and confidentiality of data, while the authentication of vehicles and messages is essential to prevent unauthorized access and malicious activities. This survey paper presents a comprehensive analysis of existing authentication mechanisms proposed for cluster-based VANETs. The strengths, weaknesses, and suitability of these mechanisms for various scenarios are carefully examined. Additionally, the integration of secure key management techniques is discussed to enhance the overall authentication process. Cluster-based VANETs are formed by dividing the network into smaller groups or clusters, with designated cluster heads comprising one or more vehicles. Furthermore, this paper identifies gaps in the existing literature through an exploration of previous surveys. Several schemes based on different methods are critically evaluated, considering factors such as throughput, detection rate, security, packet delivery ratio, and end-to-end delay. To provide optimal solutions for authentication in cluster-based VANETs, this paper highlights AI- and ML-based routing-based schemes. These approaches leverage artificial intelligence and machine learning techniques to enhance authentication within the cluster-based VANET network. Finally, this paper explores the open research challenges that exist in the realm of authentication for cluster-based Vehicular Adhoc Networks, shedding light on areas that require further investigation and development.

Title: CARGO: Crypto-Assisted Differentially Private Triangle Counting without Trusted Servers. (arXiv:2312.12938v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.12938
• Code URL: null
• Copy Paste: [[2312.12938]] CARGO: Crypto-Assisted Differentially Private Triangle Counting without Trusted Servers(http://arxiv.org/abs/2312.12938)
• Summary:

  Differentially private triangle counting in graphs is essential for analyzing connection patterns and calculating clustering coefficients while protecting sensitive individual information. Previous works have relied on either central or local models to enforce differential privacy. However, a significant utility gap exists between the central and local models of differentially private triangle counting, depending on whether or not a trusted server is needed. In particular, the central model provides a high accuracy but necessitates a trusted server. The local model does not require a trusted server but suffers from limited accuracy. Our paper introduces a crypto-assisted differentially private triangle counting system, named CARGO, leveraging cryptographic building blocks to improve the effectiveness of differentially private triangle counting without assumption of trusted servers. It achieves high utility similar to the central model but without the need for a trusted server like the local model. CARGO consists of three main components. First, we introduce a similarity-based projection method that reduces the global sensitivity while preserving more triangles via triangle homogeneity. Second, we present a triangle counting scheme based on the additive secret sharing that securely and accurately computes the triangles while protecting sensitive information. Third, we design a distributed perturbation algorithm that perturbs the triangle count with minimal but sufficient noise. We also provide a comprehensive theoretical and empirical analysis of our proposed methods. Extensive experiments demonstrate that our CARGO significantly outperforms the local model in terms of utility and achieves high-utility triangle counting comparable to the central model.

Title: HeisenTrojans: They Are Not There Until They Are Triggered. (arXiv:2312.13190v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.13190
• Code URL: null
• Copy Paste: [[2312.13190]] HeisenTrojans: They Are Not There Until They Are Triggered(http://arxiv.org/abs/2312.13190)
• Summary:

  The hardware security community has made significant advances in detecting Hardware Trojan vulnerabilities using software fuzzing-inspired automated analysis. However, the Electronic Design Automation (EDA) code base itself remains under-examined by the same techniques. Our experiments in fuzzing EDA tools demonstrate that, indeed, they are prone to software bugs. As a consequence, this paper unveils HeisenTrojan attacks, a new hardware attack that does not generate harmful hardware, but rather, exploits software vulnerabilities in the EDA tools themselves. A key feature of HeisenTrojan attacks is that they are capable of deploying a malicious payload on the system hosting the EDA tools without triggering verification tools because HeisenTrojan attacks do not rely on superfluous or malicious hardware that would otherwise be noticeable. The aim of a HeisenTrojan attack is to execute arbitrary code on the system on which the vulnerable EDA tool is hosted, thereby establishing a permanent presence and providing a beachhead for intrusion into that system. Our analysis reveals 83% of the EDA tools analyzed have exploitable bugs. In what follows, we demonstrate an end- to-end attack and provide analysis on the existing capabilities of fuzzers to find HeisenTrojan attacks in order to emphasize their practicality and the need to secure EDA tools against them.

Title: Near-Optimal Resilient Aggregation Rules for Distributed Learning Using 1-Center and 1-Mean Clustering with Outliers. (arXiv:2312.12835v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12835
• Code URL: https://github.com/jerry907/aaai24-rashb
• Copy Paste: [[2312.12835]] Near-Optimal Resilient Aggregation Rules for Distributed Learning Using 1-Center and 1-Mean Clustering with Outliers(http://arxiv.org/abs/2312.12835)
• Summary:

  Byzantine machine learning has garnered considerable attention in light of the unpredictable faults that can occur in large-scale distributed learning systems. The key to secure resilience against Byzantine machines in distributed learning is resilient aggregation mechanisms. Although abundant resilient aggregation rules have been proposed, they are designed in ad-hoc manners, imposing extra barriers on comparing, analyzing, and improving the rules across performance criteria. This paper studies near-optimal aggregation rules using clustering in the presence of outliers. Our outlier-robust clustering approach utilizes geometric properties of the update vectors provided by workers. Our analysis show that constant approximations to the 1-center and 1-mean clustering problems with outliers provide near-optimal resilient aggregators for metric-based criteria, which have been proven to be crucial in the homogeneous and heterogeneous cases respectively. In addition, we discuss two contradicting types of attacks under which no single aggregation rule is guaranteed to improve upon the naive average. Based on the discussion, we propose a two-phase resilient aggregation framework. We run experiments for image classification using a non-convex loss function. The proposed algorithms outperform previously known aggregation rules by a large margin with both homogeneous and heterogeneous data distributions among non-faulty workers. Code and appendix are available at https://github.com/jerry907/AAAI24-RASHB.

security

Title: Foreseeing Reconstruction Quality of Gradient Inversion: An Optimization Perspective. (arXiv:2312.12488v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12488
• Code URL: null
• Copy Paste: [[2312.12488]] Foreseeing Reconstruction Quality of Gradient Inversion: An Optimization Perspective(http://arxiv.org/abs/2312.12488)
• Summary:

  Gradient inversion attacks can leak data privacy when clients share weight updates with the server in federated learning (FL). Existing studies mainly use L2 or cosine distance as the loss function for gradient matching in the attack. Our empirical investigation shows that the vulnerability ranking varies with the loss function used. Gradient norm, which is commonly used as a vulnerability proxy for gradient inversion attack, cannot explain this as it remains constant regardless of the loss function for gradient matching. In this paper, we propose a loss-aware vulnerability proxy (LAVP) for the first time. LAVP refers to either the maximum or minimum eigenvalue of the Hessian with respect to gradient matching loss at ground truth. This suggestion is based on our theoretical findings regarding the local optimization of the gradient inversion in proximity to the ground truth, which corresponds to the worst case attack scenario. We demonstrate the effectiveness of LAVP on various architectures and datasets, showing its consistent superiority over the gradient norm in capturing sample vulnerabilities. The performance of each proxy is measured in terms of Spearman's rank correlation with respect to several similarity scores. This work will contribute to enhancing FL security against any potential loss functions beyond L2 or cosine distance in the future.

Title: Prometheus: Infrastructure Security Posture Analysis with AI-generated Attack Graphs. (arXiv:2312.13119v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.13119
• Code URL: null
• Copy Paste: [[2312.13119]] Prometheus: Infrastructure Security Posture Analysis with AI-generated Attack Graphs(http://arxiv.org/abs/2312.13119)
• Summary:

  The rampant occurrence of cybersecurity breaches imposes substantial limitations on the progress of network infrastructures, leading to compromised data, financial losses, potential harm to individuals, and disruptions in essential services. The current security landscape demands the urgent development of a holistic security assessment solution that encompasses vulnerability analysis and investigates the potential exploitation of these vulnerabilities as attack paths. In this paper, we propose Prometheus, an advanced system designed to provide a detailed analysis of the security posture of computing infrastructures. Using user-provided information, such as device details and software versions, Prometheus performs a comprehensive security assessment. This assessment includes identifying associated vulnerabilities and constructing potential attack graphs that adversaries can exploit. Furthermore, Prometheus evaluates the exploitability of these attack paths and quantifies the overall security posture through a scoring mechanism. The system takes a holistic approach by analyzing security layers encompassing hardware, system, network, and cryptography. Furthermore, Prometheus delves into the interconnections between these layers, exploring how vulnerabilities in one layer can be leveraged to exploit vulnerabilities in others. In this paper, we present the end-to-end pipeline implemented in Prometheus, showcasing the systematic approach adopted for conducting this thorough security analysis.

Title: SoK: Security of Cross-chain Bridges: Attack Surfaces, Defenses, and Open Problems. (arXiv:2312.12573v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.12573
• Code URL: null
• Copy Paste: [[2312.12573]] SoK: Security of Cross-chain Bridges: Attack Surfaces, Defenses, and Open Problems(http://arxiv.org/abs/2312.12573)
• Summary:

  Cross-chain bridges are used to facilitate token and data exchanges across blockchains. Although bridges are becoming increasingly popular, they are still in their infancy and have been attacked multiple times recently, causing significant financial loss. Although there are numerous reports online explaining each of the incidents on cross-chain bridges, they are scattered over the Internet, and there is no work that analyzes the security landscape of cross-chain bridges in a holistic manner. To fill the gap, in this paper, we performed a systematic study of cross-chain bridge security issues. First, we summarize the characteristics of existing cross-chain bridges, including their usages, verification mechanisms, communication models, and three categorizations. Based on these characteristics, we identify 12 potential attack vectors that attackers may exploit. Next, we introduce a taxonomy that categorizes cross-chain attacks in the past two years into 10 distinct types, and then provide explanations for each vulnerability type, accompanied by Solidity code examples. We also discuss existing and potential defenses, as well as open questions and future research directions on cross-chain bridges. We believe that this systematization can shed light on designing and implementing cross-chain bridges with higher security and, more importantly, facilitating future research on building a better cross-chain bridge ecosystem.

Title: Can Large Language Models Identify And Reason About Security Vulnerabilities? Not Yet. (arXiv:2312.12575v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.12575
• Code URL: null
• Copy Paste: [[2312.12575]] Can Large Language Models Identify And Reason About Security Vulnerabilities? Not Yet(http://arxiv.org/abs/2312.12575)
• Summary:

  Large Language Models (LLMs) have been suggested for use in automated vulnerability repair, but benchmarks showing they can consistently identify security-related bugs are lacking. We thus perform the most detailed investigation to date on whether LLMs can reliably identify security-related bugs. We construct a series of 228 code scenarios and analyze eight of the most capable LLMs across eight different investigative dimensions in an automated framework. Our evaluation shows LLMs provide non-deterministic responses, incorrect and unfaithful reasoning, and perform poorly in real-world scenarios outside their knowledge cut-off date. Most importantly, our findings reveal significant non-robustness in even the most advanced models like `PaLM2' and `GPT-4': by merely changing function or variable names, or by the addition of library functions in the source code, these models can yield incorrect answers in 26% and 17% of cases, respectively. These findings demonstrate that further LLM advances are needed before LLMs can be used as general purpose security assistants.

Title: Discovering Malicious Signatures in Software from Structural Interactions. (arXiv:2312.12667v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.12667
• Code URL: null
• Copy Paste: [[2312.12667]] Discovering Malicious Signatures in Software from Structural Interactions(http://arxiv.org/abs/2312.12667)
• Summary:

  Malware represents a significant security concern in today's digital landscape, as it can destroy or disable operating systems, steal sensitive user information, and occupy valuable disk space. However, current malware detection methods, such as static-based and dynamic-based approaches, struggle to identify newly developed (``zero-day") malware and are limited by customized virtual machine (VM) environments. To overcome these limitations, we propose a novel malware detection approach that leverages deep learning, mathematical techniques, and network science. Our approach focuses on static and dynamic analysis and utilizes the Low-Level Virtual Machine (LLVM) to profile applications within a complex network. The generated network topologies are input into the GraphSAGE architecture to efficiently distinguish between benign and malicious software applications, with the operation names denoted as node features. Importantly, the GraphSAGE models analyze the network's topological geometry to make predictions, enabling them to detect state-of-the-art malware and prevent potential damage during execution in a VM. To evaluate our approach, we conduct a study on a dataset comprising source code from 24,376 applications, specifically written in C/C++, sourced directly from widely-recognized malware and various types of benign software. The results show a high detection performance with an Area Under the Receiver Operating Characteristic Curve (AUROC) of 99.85%. Our approach marks a substantial improvement in malware detection, providing a notably more accurate and efficient solution when compared to current state-of-the-art malware detection methods.

Title: Symbolic Security Verification of Mesh Commissioning Protocol in Thread (extended version). (arXiv:2312.12958v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.12958
• Code URL: null
• Copy Paste: [[2312.12958]] Symbolic Security Verification of Mesh Commissioning Protocol in Thread (extended version)(http://arxiv.org/abs/2312.12958)
• Summary:

  The Thread protocol (or simply Thread ) is a popular networking protocol for the Internet of Things (IoT). It allows seamless integration of a set of applications and protocols, hence reducing the risk of incompatibility among different applications or user protocols. Thread has been deployed in many popular smart home products by the majority of IoT manufacturers, such as Apple TV, Apple HomePod mini, eero 6, Nest Hub, and Nest Wifi. Despite a few empirical analyses on the security of Thread, there is still a lack of formal analysis on this infrastructure of the booming IoT ecosystem. In this work, we performed a formal symbolic analysis of the security properties of Thread. Our main focus is on MeshCoP (Mesh Commissioning Protocol), the main subprotocol in Thread for secure authentication and commissioning of new, untrusted devices inside an existing Thread network. This case study presents the challenges and proposed solutions in modeling MeshCoP. We use ProVerif, a symbolic verification tool of {\pi}-calculus models, for verifying the security properties of MeshCoP.

Title: Advancing SQL Injection Detection for High-Speed Data Centers: A Novel Approach Using Cascaded NLP. (arXiv:2312.13041v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.13041
• Code URL: null
• Copy Paste: [[2312.13041]] Advancing SQL Injection Detection for High-Speed Data Centers: A Novel Approach Using Cascaded NLP(http://arxiv.org/abs/2312.13041)
• Summary:

  Detecting SQL Injection (SQLi) attacks is crucial for web-based data center security, but it is challenging to balance accuracy and computational efficiency, especially in high-speed networks. Traditional methods struggle with this balance, while NLP-based approaches, although accurate, are computationally intensive.

  We introduce a novel cascade SQLi detection method, blending classical and transformer-based NLP models, achieving a 99.86% detection accuracy with significantly lower computational demands-20 times faster than using transformer-based models alone. Our approach is tested in a realistic setting and compared with 35 other methods, including Machine Learning-based and transformer models like BERT, on a dataset of over 30,000 SQL sentences.

  Our results show that this hybrid method effectively detects SQLi in high-traffic environments, offering efficient and accurate protection against SQLi vulnerabilities with computational efficiency. The code is available at https://github.com/gdrlab/cascaded-sqli-detection .

privacy

Title: SCoTTi: Save Computation at Training Time with an adaptive framework. (arXiv:2312.12483v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12483
• Code URL: https://github.com/liziyu403/scotti-save-computation-at-training-time-with-an-adaptive-framework
• Copy Paste: [[2312.12483]] SCoTTi: Save Computation at Training Time with an adaptive framework(http://arxiv.org/abs/2312.12483)
• Summary:

  On-device training is an emerging approach in machine learning where models are trained on edge devices, aiming to enhance privacy protection and real-time performance. However, edge devices typically possess restricted computational power and resources, making it challenging to perform computationally intensive model training tasks. Consequently, reducing resource consumption during training has become a pressing concern in this field. To this end, we propose SCoTTi (Save Computation at Training Time), an adaptive framework that addresses the aforementioned challenge. It leverages an optimizable threshold parameter to effectively reduce the number of neuron updates during training which corresponds to a decrease in memory and computation footprint. Our proposed approach demonstrates superior performance compared to the state-of-the-art methods regarding computational resource savings on various commonly employed benchmarks and popular architectures, including ResNets, MobileNet, and Swin-T.

Title: A self-attention-based differentially private tabular GAN with high data utility. (arXiv:2312.13031v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.13031
• Code URL: null
• Copy Paste: [[2312.13031]] A self-attention-based differentially private tabular GAN with high data utility(http://arxiv.org/abs/2312.13031)
• Summary:

  Generative Adversarial Networks (GANs) have become a ubiquitous technology for data generation, with their prowess in image generation being well-established. However, their application in generating tabular data has been less than ideal. Furthermore, attempting to incorporate differential privacy technology into these frameworks has often resulted in a degradation of data utility. To tackle these challenges, this paper introduces DP-SACTGAN, a novel Conditional Generative Adversarial Network (CGAN) framework for differentially private tabular data generation, aiming to surmount these obstacles. Experimental findings demonstrate that DP-SACTGAN not only accurately models the distribution of the original data but also effectively satisfies the requirements of differential privacy.

protect

Title: Comprehensive Validation on Reweighting Samples for Bias Mitigation via AIF360. (arXiv:2312.12560v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12560
• Code URL: null
• Copy Paste: [[2312.12560]] Comprehensive Validation on Reweighting Samples for Bias Mitigation via AIF360(http://arxiv.org/abs/2312.12560)
• Summary:

  Fairness AI aims to detect and alleviate bias across the entire AI development life cycle, encompassing data curation, modeling, evaluation, and deployment-a pivotal aspect of ethical AI implementation. Addressing data bias, particularly concerning sensitive attributes like gender and race, reweighting samples proves efficient for fairness AI. This paper contributes a systematic examination of reweighting samples for traditional machine learning (ML) models, employing five models for binary classification on the Adult Income and COMPUS datasets with various protected attributes. The study evaluates prediction results using five fairness metrics, uncovering the nuanced and model-specific nature of reweighting sample effectiveness in achieving fairness in traditional ML models, as well as revealing the complexity of bias dynamics.

defense

Title: Progressive Poisoned Data Isolation for Training-time Backdoor Defense. (arXiv:2312.12724v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.12724
• Code URL: null
• Copy Paste: [[2312.12724]] Progressive Poisoned Data Isolation for Training-time Backdoor Defense(http://arxiv.org/abs/2312.12724)
• Summary:

  Deep Neural Networks (DNN) are susceptible to backdoor attacks where malicious attackers manipulate the model's predictions via data poisoning. It is hence imperative to develop a strategy for training a clean model using a potentially poisoned dataset. Previous training-time defense mechanisms typically employ an one-time isolation process, often leading to suboptimal isolation outcomes. In this study, we present a novel and efficacious defense method, termed Progressive Isolation of Poisoned Data (PIPD), that progressively isolates poisoned data to enhance the isolation accuracy and mitigate the risk of benign samples being misclassified as poisoned ones. Once the poisoned portion of the dataset has been identified, we introduce a selective training process to train a clean model. Through the implementation of these techniques, we ensure that the trained model manifests a significantly diminished attack success rate against the poisoned data. Extensive experiments on multiple benchmark datasets and DNN models, assessed against nine state-of-the-art backdoor attacks, demonstrate the superior performance of our PIPD method for backdoor defense. For instance, our PIPD achieves an average True Positive Rate (TPR) of 99.95% and an average False Positive Rate (FPR) of 0.06% for diverse attacks over CIFAR-10 dataset, markedly surpassing the performance of state-of-the-art methods.

Title: LRS: Enhancing Adversarial Transferability through Lipschitz Regularized Surrogate. (arXiv:2312.13118v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.13118
• Code URL: https://github.com/trustaiot/lrs
• Copy Paste: [[2312.13118]] LRS: Enhancing Adversarial Transferability through Lipschitz Regularized Surrogate(http://arxiv.org/abs/2312.13118)
• Summary:

  The transferability of adversarial examples is of central importance to transfer-based black-box adversarial attacks. Previous works for generating transferable adversarial examples focus on attacking \emph{given} pretrained surrogate models while the connections between surrogate models and adversarial trasferability have been overlooked. In this paper, we propose {\em Lipschitz Regularized Surrogate} (LRS) for transfer-based black-box attacks, a novel approach that transforms surrogate models towards favorable adversarial transferability. Using such transformed surrogate models, any existing transfer-based black-box attack can run without any change, yet achieving much better performance. Specifically, we impose Lipschitz regularization on the loss landscape of surrogate models to enable a smoother and more controlled optimization process for generating more transferable adversarial examples. In addition, this paper also sheds light on the connection between the inner properties of surrogate models and adversarial transferability, where three factors are identified: smaller local Lipschitz constant, smoother loss landscape, and stronger adversarial robustness. We evaluate our proposed LRS approach by attacking state-of-the-art standard deep neural networks and defense models. The results demonstrate significant improvement on the attack success rates and transferability. Our code is available at https://github.com/TrustAIoT/LRS.

attack

Title: Mutual-modality Adversarial Attack with Semantic Perturbation. (arXiv:2312.12768v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12768
• Code URL: null
• Copy Paste: [[2312.12768]] Mutual-modality Adversarial Attack with Semantic Perturbation(http://arxiv.org/abs/2312.12768)
• Summary:

  Adversarial attacks constitute a notable threat to machine learning systems, given their potential to induce erroneous predictions and classifications. However, within real-world contexts, the essential specifics of the deployed model are frequently treated as a black box, consequently mitigating the vulnerability to such attacks. Thus, enhancing the transferability of the adversarial samples has become a crucial area of research, which heavily relies on selecting appropriate surrogate models. To address this challenge, we propose a novel approach that generates adversarial attacks in a mutual-modality optimization scheme. Our approach is accomplished by leveraging the pre-trained CLIP model. Firstly, we conduct a visual attack on the clean image that causes semantic perturbations on the aligned embedding space with the other textual modality. Then, we apply the corresponding defense on the textual modality by updating the prompts, which forces the re-matching on the perturbed embedding space. Finally, to enhance the attack transferability, we utilize the iterative training strategy on the visual attack and the textual defense, where the two processes optimize from each other. We evaluate our approach on several benchmark datasets and demonstrate that our mutual-modal attack strategy can effectively produce high-transferable attacks, which are stable regardless of the target networks. Our approach outperforms state-of-the-art attack methods and can be readily deployed as a plug-and-play solution.

Title: SkyMask: Attack-agnostic Robust Federated Learning with Fine-grained Learnable Masks. (arXiv:2312.12484v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.12484
• Code URL: null
• Copy Paste: [[2312.12484]] SkyMask: Attack-agnostic Robust Federated Learning with Fine-grained Learnable Masks(http://arxiv.org/abs/2312.12484)
• Summary:

  Federated Learning (FL) is becoming a popular paradigm for leveraging distributed data and preserving data privacy. However, due to the distributed characteristic, FL systems are vulnerable to Byzantine attacks that compromised clients attack the global model by uploading malicious model updates. Most existing Byzantine-robust FL systems statistically analyze the weights of whole individual model updates uploaded by clients to defend against Byzantine attacks. With the development of layer-level and parameter-level fine-grained attacks, the attacks' stealthiness and effectiveness have been significantly improved. Due to unawareness or overreaction, the existing model-level defense methods degrade the training efficiency and model performance. To address this problem, we propose SkyMask, a new attack-agnostic robust FL system that leverages fine-grained learnable masks to identify malicious model updates at the parameter-level. Specifically, the FL server applies parameter-level masks to model updates uploaded by clients and trains the masks over a small clean dataset (i.e., root dataset) to learn the subtle difference between benign and malicious model updates in a high-dimension space. Our extensive experiments involve different models on three public datasets under state-of-the-art (SOTA) attacks, where the results show that SkyMask achieves up to 10% higher testing accuracy compared with SOTA defense strategies and successfully defends against attacks with malicious clients of a high fraction up to 80%. In the meantime, the experimental results demonstrate the scalability of our approach and the weak dependence on the data distribution of the root dataset.

Title: BadRL: Sparse Targeted Backdoor Attack Against Reinforcement Learning. (arXiv:2312.12585v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12585
• Code URL: null
• Copy Paste: [[2312.12585]] BadRL: Sparse Targeted Backdoor Attack Against Reinforcement Learning(http://arxiv.org/abs/2312.12585)
• Summary:

  Backdoor attacks in reinforcement learning (RL) have previously employed intense attack strategies to ensure attack success. However, these methods suffer from high attack costs and increased detectability. In this work, we propose a novel approach, BadRL, which focuses on conducting highly sparse backdoor poisoning efforts during training and testing while maintaining successful attacks. Our algorithm, BadRL, strategically chooses state observations with high attack values to inject triggers during training and testing, thereby reducing the chances of detection. In contrast to the previous methods that utilize sample-agnostic trigger patterns, BadRL dynamically generates distinct trigger patterns based on targeted state observations, thereby enhancing its effectiveness. Theoretical analysis shows that the targeted backdoor attack is always viable and remains stealthy under specific assumptions. Empirical results on various classic RL tasks illustrate that BadRL can substantially degrade the performance of a victim agent with minimal poisoning efforts 0.003% of total training steps) during training and infrequent attacks during testing.

Title: Trust, But Verify: A Survey of Randomized Smoothing Techniques. (arXiv:2312.12608v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12608
• Code URL: null
• Copy Paste: [[2312.12608]] Trust, But Verify: A Survey of Randomized Smoothing Techniques(http://arxiv.org/abs/2312.12608)
• Summary:

  Machine learning models have demonstrated remarkable success across diverse domains but remain vulnerable to adversarial attacks. Empirical defence mechanisms often fall short, as new attacks constantly emerge, rendering existing defences obsolete. A paradigm shift from empirical defences to certification-based defences has been observed in response. Randomized smoothing has emerged as a promising technique among notable advancements. This study reviews the theoretical foundations, empirical effectiveness, and applications of randomized smoothing in verifying machine learning classifiers. We provide an in-depth exploration of the fundamental concepts underlying randomized smoothing, highlighting its theoretical guarantees in certifying robustness against adversarial perturbations. Additionally, we discuss the challenges of existing methodologies and offer insightful perspectives on potential solutions. This paper is novel in its attempt to systemise the existing knowledge in the context of randomized smoothing.

Title: When Memory Mappings Attack: On the (Mis)use of the ARM Cortex-M FPB Unit. (arXiv:2312.13189v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.13189
• Code URL: null
• Copy Paste: [[2312.13189]] When Memory Mappings Attack: On the (Mis)use of the ARM Cortex-M FPB Unit(http://arxiv.org/abs/2312.13189)
• Summary:

  In recent years we have seen an explosion in the usage of low-cost, low-power microcontrollers (MCUs) in embedded devices around us due to the popularity of Internet of Things (IoT) devices. Although this is good from an economics perspective, it has also been detrimental for security as microcontroller-based systems are now a viable attack target. In response, researchers have developed various protection mechanisms dedicated to improve security in these resource-constrained embedded systems. We demonstrate in this paper these defenses fall short when we leverage benign memory mapped design-for-debug (DfD) structures added by MCU vendors in their products. In particular, we utilize the Flash Patch and Breakpoint (FPB) unit present in the ARM Cortex-M family to build new attack primitives which can be used to bypass common defenses for embedded devices. Our work serves as a warning and a call in balancing security and debug structures in modern microcontrollers.

Title: PGN: A perturbation generation network against deep reinforcement learning. (arXiv:2312.12904v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12904
• Code URL: null
• Copy Paste: [[2312.12904]] PGN: A perturbation generation network against deep reinforcement learning(http://arxiv.org/abs/2312.12904)
• Summary:

  Deep reinforcement learning has advanced greatly and applied in many areas. In this paper, we explore the vulnerability of deep reinforcement learning by proposing a novel generative model for creating effective adversarial examples to attack the agent. Our proposed model can achieve both targeted attacks and untargeted attacks. Considering the specificity of deep reinforcement learning, we propose the action consistency ratio as a measure of stealthiness, and a new measurement index of effectiveness and stealthiness. Experiment results show that our method can ensure the effectiveness and stealthiness of attack compared with other algorithms. Moreover, our methods are considerably faster and thus can achieve rapid and efficient verification of the vulnerability of deep reinforcement learning.

robust

Title: What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?. (arXiv:2312.12444v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12444
• Code URL: null
• Copy Paste: [[2312.12444]] What Makes Pre-Trained Visual Representations Successful for Robust Manipulation?(http://arxiv.org/abs/2312.12444)
• Summary:

  Inspired by the success of transfer learning in computer vision, roboticists have investigated visual pre-training as a means to improve the learning efficiency and generalization ability of policies learned from pixels. To that end, past work has favored large object interaction datasets, such as first-person videos of humans completing diverse tasks, in pursuit of manipulation-relevant features. Although this approach improves the efficiency of policy learning, it remains unclear how reliable these representations are in the presence of distribution shifts that arise commonly in robotic applications. Surprisingly, we find that visual representations designed for manipulation and control tasks do not necessarily generalize under subtle changes in lighting and scene texture or the introduction of distractor objects. To understand what properties do lead to robust representations, we compare the performance of 15 pre-trained vision models under different visual appearances. We find that emergent segmentation ability is a strong predictor of out-of-distribution generalization among ViT models. The rank order induced by this metric is more predictive than metrics that have previously guided generalization research within computer vision and machine learning, such as downstream ImageNet accuracy, in-domain accuracy, or shape-bias as evaluated by cue-conflict performance. We test this finding extensively on a suite of distribution shifts in ten tasks across two simulated manipulation environments. On the ALOHA setup, segmentation score predicts real-world performance after offline training with 50 demonstrations.

Title: ProS: Prompting-to-simulate Generalized knowledge for Universal Cross-Domain Retrieval. (arXiv:2312.12478v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12478
• Code URL: null
• Copy Paste: [[2312.12478]] ProS: Prompting-to-simulate Generalized knowledge for Universal Cross-Domain Retrieval(http://arxiv.org/abs/2312.12478)
• Summary:

  The goal of Universal Cross-Domain Retrieval (UCDR) is to achieve robust performance in generalized test scenarios, wherein data may belong to strictly unknown domains and categories during training. Recently, pre-trained models with prompt tuning have shown strong generalization capabilities and attained noteworthy achievements in various downstream tasks, such as few-shot learning and video-text retrieval. However, applying them directly to UCDR may not sufficiently to handle both domain shift (i.e., adapting to unfamiliar domains) and semantic shift (i.e., transferring to unknown categories). To this end, we propose Prompting-to-Simulate (ProS), the first method to apply prompt tuning for UCDR. ProS employs a two-step process to simulate Content-aware Dynamic Prompts (CaDP) which can impact models to produce generalized features for UCDR. Concretely, in Prompt Units Learning stage, we introduce two Prompt Units to individually capture domain and semantic knowledge in a mask-and-align way. Then, in Context-aware Simulator Learning stage, we train a Content-aware Prompt Simulator under a simulated test scenarios to produce the corresponding CaDP. Extensive experiments conducted on three benchmark datasets show that our method achieves new state-of-the-art performance without bringing excessive parameters. Our method is publicly available at https://anonymous.4open.science/r/ProS

Title: IS-DARTS: Stabilizing DARTS through Precise Measurement on Candidate Importance. (arXiv:2312.12648v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12648
• Code URL: https://github.com/hy-he/is-darts
• Copy Paste: [[2312.12648]] IS-DARTS: Stabilizing DARTS through Precise Measurement on Candidate Importance(http://arxiv.org/abs/2312.12648)
• Summary:

  Among existing Neural Architecture Search methods, DARTS is known for its efficiency and simplicity. This approach applies continuous relaxation of network representation to construct a weight-sharing supernet and enables the identification of excellent subnets in just a few GPU days. However, performance collapse in DARTS results in deteriorating architectures filled with parameter-free operations and remains a great challenge to the robustness. To resolve this problem, we reveal that the fundamental reason is the biased estimation of the candidate importance in the search space through theoretical and experimental analysis, and more precisely select operations via information-based measurements. Furthermore, we demonstrate that the excessive concern over the supernet and inefficient utilization of data in bi-level optimization also account for suboptimal results. We adopt a more realistic objective focusing on the performance of subnets and simplify it with the help of the information-based measurements. Finally, we explain theoretically why progressively shrinking the width of the supernet is necessary and reduce the approximation error of optimal weights in DARTS. Our proposed method, named IS-DARTS, comprehensively improves DARTS and resolves the aforementioned problems. Extensive experiments on NAS-Bench-201 and DARTS-based search space demonstrate the effectiveness of IS-DARTS.

Title: Trajectory Approximation of Video Based on Phase Correlation for Forward Facing Camera. (arXiv:2312.12680v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12680
• Code URL: null
• Copy Paste: [[2312.12680]] Trajectory Approximation of Video Based on Phase Correlation for Forward Facing Camera(http://arxiv.org/abs/2312.12680)
• Summary:

  In this paper, we introduce an innovative approach for extracting trajectories from a camera sensor in GPS-denied environments, leveraging visual odometry. The system takes video footage captured by a forward-facing camera mounted on a vehicle as input, with the output being a chain code representing the camera's trajectory. The proposed methodology involves several key steps. Firstly, we employ phase correlation between consecutive frames of the video to extract essential information. Subsequently, we introduce a novel chain code method termed "dynamic chain code," which is based on the x-shift values derived from the phase correlation. The third step involves determining directional changes (forward, left, right) by establishing thresholds and extracting the corresponding chain code. This extracted code is then stored in a buffer for further processing. Notably, our system outperforms traditional methods reliant on spatial features, exhibiting greater speed and robustness in noisy environments. Importantly, our approach operates without external camera calibration information. Moreover, by incorporating visual odometry, our system enhances its accuracy in estimating camera motion, providing a more comprehensive understanding of trajectory dynamics. Finally, the system culminates in the visualization of the normalized camera motion trajectory.

Title: AdvST: Revisiting Data Augmentations for Single Domain Generalization. (arXiv:2312.12720v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12720
• Code URL: https://github.com/gtzheng/advst
• Copy Paste: [[2312.12720]] AdvST: Revisiting Data Augmentations for Single Domain Generalization(http://arxiv.org/abs/2312.12720)
• Summary:

  Single domain generalization (SDG) aims to train a robust model against unknown target domain shifts using data from a single source domain. Data augmentation has been proven an effective approach to SDG. However, the utility of standard augmentations, such as translate, or invert, has not been fully exploited in SDG; practically, these augmentations are used as a part of a data preprocessing procedure. Although it is intuitive to use many such augmentations to boost the robustness of a model to out-of-distribution domain shifts, we lack a principled approach to harvest the benefit brought from multiple these augmentations. Here, we conceptualize standard data augmentations with learnable parameters as semantics transformations that can manipulate certain semantics of a sample, such as the geometry or color of an image. Then, we propose Adversarial learning with Semantics Transformations (AdvST) that augments the source domain data with semantics transformations and learns a robust model with the augmented data. We theoretically show that AdvST essentially optimizes a distributionally robust optimization objective defined on a set of semantics distributions induced by the parameters of semantics transformations. We demonstrate that AdvST can produce samples that expand the coverage on target domain data. Compared with the state-of-the-art methods, AdvST, despite being a simple method, is surprisingly competitive and achieves the best average SDG performance on the Digits, PACS, and DomainNet datasets. Our code is available at https://github.com/gtzheng/AdvST.

Title: No More Shortcuts: Realizing the Potential of Temporal Self-Supervision. (arXiv:2312.13008v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.13008
• Code URL: null
• Copy Paste: [[2312.13008]] No More Shortcuts: Realizing the Potential of Temporal Self-Supervision(http://arxiv.org/abs/2312.13008)
• Summary:

  Self-supervised approaches for video have shown impressive results in video understanding tasks. However, unlike early works that leverage temporal self-supervision, current state-of-the-art methods primarily rely on tasks from the image domain (e.g., contrastive learning) that do not explicitly promote the learning of temporal features. We identify two factors that limit existing temporal self-supervision: 1) tasks are too simple, resulting in saturated training performance, and 2) we uncover shortcuts based on local appearance statistics that hinder the learning of high-level features. To address these issues, we propose 1) a more challenging reformulation of temporal self-supervision as frame-level (rather than clip-level) recognition tasks and 2) an effective augmentation strategy to mitigate shortcuts. Our model extends a representation of single video frames, pre-trained through contrastive learning, with a transformer that we train through temporal self-supervision. We demonstrate experimentally that our more challenging frame-level task formulations and the removal of shortcuts drastically improve the quality of features learned through temporal self-supervision. The generalization capability of our self-supervised video method is evidenced by its state-of-the-art performance in a wide range of high-level semantic tasks, including video retrieval, action classification, and video attribute recognition (such as object and scene identification), as well as low-level temporal correspondence tasks like video object segmentation and pose tracking. Additionally, we show that the video representations learned through our method exhibit increased robustness to the input perturbations.

Title: SEER-ZSL: Semantic Encoder-Enhanced Representations for Generalized Zero-Shot Learning. (arXiv:2312.13100v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.13100
• Code URL: https://github.com/william-heyden/seer-zeroshotlearning
• Copy Paste: [[2312.13100]] SEER-ZSL: Semantic Encoder-Enhanced Representations for Generalized Zero-Shot Learning(http://arxiv.org/abs/2312.13100)
• Summary:

  Generalized Zero-Shot Learning (GZSL) recognizes unseen classes by transferring knowledge from the seen classes, depending on the inherent interactions between visual and semantic data. However, the discrepancy between well-prepared training data and unpredictable real-world test scenarios remains a significant challenge. This paper introduces a dual strategy to address the generalization gap. Firstly, we incorporate semantic information through an innovative encoder. This encoder effectively integrates class-specific semantic information by targeting the performance disparity, enhancing the produced features to enrich the semantic space for class-specific attributes. Secondly, we refine our generative capabilities using a novel compositional loss function. This approach generates discriminative classes, effectively classifying both seen and unseen classes. In addition, we extend the exploitation of the learned latent space by utilizing controlled semantic inputs, ensuring the robustness of the model in varying environments. This approach yields a model that outperforms the state-of-the-art models in terms of both generalization and diverse settings, notably without requiring hyperparameter tuning or domain-specific adaptations. We also propose a set of novel evaluation metrics to provide a more detailed assessment of the reliability and reproducibility of the results. The complete code is made available on https://github.com/william-heyden/SEER-ZeroShotLearning/.

Title: Investigating Color Illusions from the Perspective of Computational Color Constancy. (arXiv:2312.13114v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.13114
• Code URL: null
• Copy Paste: [[2312.13114]] Investigating Color Illusions from the Perspective of Computational Color Constancy(http://arxiv.org/abs/2312.13114)
• Summary:

  Color constancy and color illusion perception are two phenomena occurring in the human visual system, which can help us reveal unknown mechanisms of human perception. For decades computer vision scientists have developed numerous color constancy methods, which estimate the reflectance of the surface by discounting the illuminant. However, color illusions have not been analyzed in detail in the field of computational color constancy, which we find surprising since the relationship they share is significant and may let us design more robust systems. We argue that any model that can reproduce our sensation on color illusions should also be able to provide pixel-wise estimates of the light source. In other words, we suggest that the analysis of color illusions helps us to improve the performance of the existing global color constancy methods, and enable them to provide pixel-wise estimates for scenes illuminated by multiple light sources. In this study, we share the outcomes of our investigation in which we take several color constancy methods and modify them to reproduce the behavior of the human visual system on color illusions. Also, we show that parameters purely extracted from illusions are able to improve the performance of color constancy methods. A noteworthy outcome is that our strategy based on the investigation of color illusions outperforms the state-of-the-art methods that are specifically designed to transform global color constancy algorithms into multi-illuminant algorithms.

Title: Assaying on the Robustness of Zero-Shot Machine-Generated Text Detectors. (arXiv:2312.12918v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.12918
• Code URL: https://github.com/yfzhang114/robustness-detection
• Copy Paste: [[2312.12918]] Assaying on the Robustness of Zero-Shot Machine-Generated Text Detectors(http://arxiv.org/abs/2312.12918)
• Summary:

  To combat the potential misuse of Natural Language Generation (NLG) technology, a variety of algorithms have been developed for the detection of AI-generated texts. Traditionally, this task is treated as a binary classification problem. Although supervised learning has demonstrated promising results, acquiring labeled data for detection purposes poses real-world challenges and the risk of overfitting. In an effort to address these issues, we delve into the realm of zero-shot machine-generated text detection. Existing zero-shot detectors, typically designed for specific tasks or topics, often assume uniform testing scenarios, limiting their practicality. In our research, we explore various advanced Large Language Models (LLMs) and their specialized variants, contributing to this field in several ways. In empirical studies, we uncover a significant correlation between topics and detection performance. Secondly, we delve into the influence of topic shifts on zero-shot detectors. These investigations shed light on the adaptability and robustness of these detection methods across diverse topics.

Title: AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation. (arXiv:2312.13010v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.13010
• Code URL: null
• Copy Paste: [[2312.13010]] AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation(http://arxiv.org/abs/2312.13010)
• Summary:

  The advancement of natural language processing (NLP) has been significantly boosted by the development of transformer-based large language models (LLMs). These models have revolutionized NLP tasks, particularly in code generation, aiding developers in creating software with enhanced efficiency. Despite their advancements, challenges in balancing code snippet generation with effective test case generation and execution persist. To address these issues, this paper introduces Multi-Agent Assistant Code Generation (AgentCoder), a novel solution comprising a multi-agent framework with specialized agents: the programmer agent, the test designer agent, and the test executor agent. During the coding procedure, the programmer agent will focus on the code generation and refinement based on the test executor agent's feedback. The test designer agent will generate test cases for the generated code, and the test executor agent will run the code with the test cases and write the feedback to the programmer. This collaborative system ensures robust code generation, surpassing the limitations of single-agent models and traditional methodologies. Our extensive experiments on 9 code generation models and 12 enhancement approaches showcase AgentCoder's superior performance over existing code generation models and prompt engineering techniques across various benchmarks. For example, AgentCoder achieves 77.4% and 89.1% pass@1 in HumanEval-ET and MBPP-ET with GPT-3.5, while SOTA baselines obtain only 69.5% and 63.0%.

Title: Scaling Compute Is Not All You Need for Adversarial Robustness. (arXiv:2312.13131v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.13131
• Code URL: null
• Copy Paste: [[2312.13131]] Scaling Compute Is Not All You Need for Adversarial Robustness(http://arxiv.org/abs/2312.13131)
• Summary:

  The last six years have witnessed significant progress in adversarially robust deep learning. As evidenced by the CIFAR-10 dataset category in RobustBench benchmark, the accuracy under $\ell_\infty$ adversarial perturbations improved from 44\% in \citet{Madry2018Towards} to 71\% in \citet{peng2023robust}. Although impressive, existing state-of-the-art is still far from satisfactory. It is further observed that best-performing models are often very large models adversarially trained by industrial labs with significant computational budgets. In this paper, we aim to understand: ``how much longer can computing power drive adversarial robustness advances?" To answer this question, we derive \emph{scaling laws for adversarial robustness} which can be extrapolated in the future to provide an estimate of how much cost we would need to pay to reach a desired level of robustness. We show that increasing the FLOPs needed for adversarial training does not bring as much advantage as it does for standard training in terms of performance improvements. Moreover, we find that some of the top-performing techniques are difficult to exactly reproduce, suggesting that they are not robust enough for minor changes in the training setup. Our analysis also uncovers potentially worthwhile directions to pursue in future research. Finally, we make our benchmarking framework (built on top of \texttt{timm}~\citep{rw2019timm}) publicly available to facilitate future analysis in efficient robust deep learning.

Title: Robust Machine Learning by Transforming and Augmenting Imperfect Training Data. (arXiv:2312.12597v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12597
• Code URL: null
• Copy Paste: [[2312.12597]] Robust Machine Learning by Transforming and Augmenting Imperfect Training Data(http://arxiv.org/abs/2312.12597)
• Summary:

  Machine Learning (ML) is an expressive framework for turning data into computer programs. Across many problem domains -- both in industry and policy settings -- the types of computer programs needed for accurate prediction or optimal control are difficult to write by hand. On the other hand, collecting instances of desired system behavior may be relatively more feasible. This makes ML broadly appealing, but also induces data sensitivities that often manifest as unexpected failure modes during deployment. In this sense, the training data available tend to be imperfect for the task at hand. This thesis explores several data sensitivities of modern machine learning and how to address them. We begin by discussing how to prevent ML from codifying prior human discrimination measured in the training data, where we take a fair representation learning approach. We then discuss the problem of learning from data containing spurious features, which provide predictive fidelity during training but are unreliable upon deployment. Here we observe that insofar as standard training methods tend to learn such features, this propensity can be leveraged to search for partitions of training data that expose this inconsistency, ultimately promoting learning algorithms invariant to spurious features. Finally, we turn our attention to reinforcement learning from data with insufficient coverage over all possible states and actions. To address the coverage issue, we discuss how causal priors can be used to model the single-step dynamics of the setting where data are collected. This enables a new type of data augmentation where observed trajectories are stitched together to produce new but plausible counterfactual trajectories.

Title: Robustly Improving Bandit Algorithms with Confounded and Selection Biased Offline Data: A Causal Approach. (arXiv:2312.12731v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12731
• Code URL: null
• Copy Paste: [[2312.12731]] Robustly Improving Bandit Algorithms with Confounded and Selection Biased Offline Data: A Causal Approach(http://arxiv.org/abs/2312.12731)
• Summary:

  This paper studies bandit problems where an agent has access to offline data that might be utilized to potentially improve the estimation of each arm's reward distribution. A major obstacle in this setting is the existence of compound biases from the observational data. Ignoring these biases and blindly fitting a model with the biased data could even negatively affect the online learning phase. In this work, we formulate this problem from a causal perspective. First, we categorize the biases into confounding bias and selection bias based on the causal structure they imply. Next, we extract the causal bound for each arm that is robust towards compound biases from biased observational data. The derived bounds contain the ground truth mean reward and can effectively guide the bandit agent to learn a nearly-optimal decision policy. We also conduct regret analysis in both contextual and non-contextual bandit settings and show that prior causal bounds could help consistently reduce the asymptotic regret.

Title: BSL: Understanding and Improving Softmax Loss for Recommendation. (arXiv:2312.12882v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12882
• Code URL: https://github.com/junkangwu/bsl
• Copy Paste: [[2312.12882]] BSL: Understanding and Improving Softmax Loss for Recommendation(http://arxiv.org/abs/2312.12882)
• Summary:

  Loss functions steer the optimization direction of recommendation models and are critical to model performance, but have received relatively little attention in recent recommendation research. Among various losses, we find Softmax loss (SL) stands out for not only achieving remarkable accuracy but also better robustness and fairness. Nevertheless, the current literature lacks a comprehensive explanation for the efficacy of SL. Toward addressing this research gap, we conduct theoretical analyses on SL and uncover three insights: 1) Optimizing SL is equivalent to performing Distributionally Robust Optimization (DRO) on the negative data, thereby learning against perturbations on the negative distribution and yielding robustness to noisy negatives. 2) Comparing with other loss functions, SL implicitly penalizes the prediction variance, resulting in a smaller gap between predicted values and and thus producing fairer results. Building on these insights, we further propose a novel loss function Bilateral SoftMax Loss (BSL) that extends the advantage of SL to both positive and negative sides. BSL augments SL by applying the same Log-Expectation-Exp structure to positive examples as is used for negatives, making the model robust to the noisy positives as well. Remarkably, BSL is simple and easy-to-implement -- requiring just one additional line of code compared to SL. Experiments on four real-world datasets and three representative backbones demonstrate the effectiveness of our proposal. The code is available at https://github.com/junkangwu/BSL

Title: Robust Loss Functions for Training Decision Trees with Noisy Labels. (arXiv:2312.12937v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12937
• Code URL: null
• Copy Paste: [[2312.12937]] Robust Loss Functions for Training Decision Trees with Noisy Labels(http://arxiv.org/abs/2312.12937)
• Summary:

  We consider training decision trees using noisily labeled data, focusing on loss functions that can lead to robust learning algorithms. Our contributions are threefold. First, we offer novel theoretical insights on the robustness of many existing loss functions in the context of decision tree learning. We show that some of the losses belong to a class of what we call conservative losses, and the conservative losses lead to an early stopping behavior during training and noise-tolerant predictions during testing. Second, we introduce a framework for constructing robust loss functions, called distribution losses. These losses apply percentile-based penalties based on an assumed margin distribution, and they naturally allow adapting to different noise rates via a robustness parameter. In particular, we introduce a new loss called the negative exponential loss, which leads to an efficient greedy impurity-reduction learning algorithm. Lastly, our experiments on multiple datasets and noise settings validate our theoretical insight and the effectiveness of our adaptive negative exponential loss.

biometric

steal

extraction

Title: Zero-shot Building Attribute Extraction from Large-Scale Vision and Language Models. (arXiv:2312.12479v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12479
• Code URL: null
• Copy Paste: [[2312.12479]] Zero-shot Building Attribute Extraction from Large-Scale Vision and Language Models(http://arxiv.org/abs/2312.12479)
• Summary:

  Existing building recognition methods, exemplified by BRAILS, utilize supervised learning to extract information from satellite and street-view images for classification and segmentation. However, each task module requires human-annotated data, hindering the scalability and robustness to regional variations and annotation imbalances. In response, we propose a new zero-shot workflow for building attribute extraction that utilizes large-scale vision and language models to mitigate reliance on external annotations. The proposed workflow contains two key components: image-level captioning and segment-level captioning for the building images based on the vocabularies pertinent to structural and civil engineering. These two components generate descriptive captions by computing feature representations of the image and the vocabularies, and facilitating a semantic match between the visual and textual representations. Consequently, our framework offers a promising avenue to enhance AI-driven captioning for building attribute extraction in the structural and civil engineering domains, ultimately reducing reliance on human annotations while bolstering performance and adaptability.

Title: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation. (arXiv:2312.12480v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12480
• Code URL: null
• Copy Paste: [[2312.12480]] Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation(http://arxiv.org/abs/2312.12480)
• Summary:

  Continual Test-Time Adaptation (CTTA) is proposed to migrate a source pre-trained model to continually changing target distributions, addressing real-world dynamism. Existing CTTA methods mainly rely on entropy minimization or teacher-student pseudo-labeling schemes for knowledge extraction in unlabeled target domains. However, dynamic data distributions cause miscalibrated predictions and noisy pseudo-labels in existing self-supervised learning methods, hindering the effective mitigation of error accumulation and catastrophic forgetting problems during the continual adaptation process. To tackle these issues, we propose a continual self-supervised method, Adaptive Distribution Masked Autoencoders (ADMA), which enhances the extraction of target domain knowledge while mitigating the accumulation of distribution shifts. Specifically, we propose a Distribution-aware Masking (DaM) mechanism to adaptively sample masked positions, followed by establishing consistency constraints between the masked target samples and the original target samples. Additionally, for masked tokens, we utilize an efficient decoder to reconstruct a hand-crafted feature descriptor (e.g., Histograms of Oriented Gradients), leveraging its invariant properties to boost task-relevant representations. Through conducting extensive experiments on four widely recognized benchmarks, our proposed method attains state-of-the-art performance in both classification and segmentation CTTA tasks.

Title: Unveiling Spaces: Architecturally meaningful semantic descriptions from images of interior spaces. (arXiv:2312.12481v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12481
• Code URL: null
• Copy Paste: [[2312.12481]] Unveiling Spaces: Architecturally meaningful semantic descriptions from images of interior spaces(http://arxiv.org/abs/2312.12481)
• Summary:

  There has been a growing adoption of computer vision tools and technologies in architectural design workflows over the past decade. Notable use cases include point cloud generation, visual content analysis, and spatial awareness for robotic fabrication. Multiple image classification, object detection, and semantic pixel segmentation models have become popular for the extraction of high-level symbolic descriptions and semantic content from two-dimensional images and videos. However, a major challenge in this regard has been the extraction of high-level architectural structures (walls, floors, ceilings windows etc.) from diverse imagery where parts of these elements are occluded by furniture, people, or other non-architectural elements. This project aims to tackle this problem by proposing models that are capable of extracting architecturally meaningful semantic descriptions from two-dimensional scenes of populated interior spaces. 1000 virtual classrooms are parametrically generated, randomized along key spatial parameters such as length, width, height, and door/window positions. The positions of cameras, and non-architectural visual obstructions (furniture/objects) are also randomized. A Generative Adversarial Network (GAN) for image-to-image translation (Pix2Pix) is trained on synthetically generated rendered images of these enclosures, along with corresponding image abstractions representing high-level architectural structure. The model is then tested on unseen synthetic imagery of new enclosures, and outputs are compared to ground truth using pixel-wise comparison for evaluation. A similar model evaluation is also carried out on photographs of existing indoor enclosures, to measure its performance in real-world settings.

Title: Produce Once, Utilize Twice for Anomaly Detection. (arXiv:2312.12913v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12913
• Code URL: null
• Copy Paste: [[2312.12913]] Produce Once, Utilize Twice for Anomaly Detection(http://arxiv.org/abs/2312.12913)
• Summary:

  Visual anomaly detection aims at classifying and locating the regions that deviate from the normal appearance. Embedding-based methods and reconstruction-based methods are two main approaches for this task. However, they are either not efficient or not precise enough for the industrial detection. To deal with this problem, we derive POUTA (Produce Once Utilize Twice for Anomaly detection), which improves both the accuracy and efficiency by reusing the discriminant information potential in the reconstructive network. We observe that the encoder and decoder representations of the reconstructive network are able to stand for the features of the original and reconstructed image respectively. And the discrepancies between the symmetric reconstructive representations provides roughly accurate anomaly information. To refine this information, a coarse-to-fine process is proposed in POUTA, which calibrates the semantics of each discriminative layer by the high-level representations and supervision loss. Equipped with the above modules, POUTA is endowed with the ability to provide a more precise anomaly location than the prior arts. Besides, the representation reusage also enables to exclude the feature extraction process in the discriminative network, which reduces the parameters and improves the efficiency. Extensive experiments show that, POUTA is superior or comparable to the prior methods with even less cost. Furthermore, POUTA also achieves better performance than the state-of-the-art few-shot anomaly detection methods without any special design, showing that POUTA has strong ability to learn representations inherent in the training data.

Title: Rule-Extraction Methods From Feedforward Neural Networks: A Systematic Literature Review. (arXiv:2312.12878v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12878
• Code URL: null
• Copy Paste: [[2312.12878]] Rule-Extraction Methods From Feedforward Neural Networks: A Systematic Literature Review(http://arxiv.org/abs/2312.12878)
• Summary:

  Motivated by the interpretability question in ML models as a crucial element for the successful deployment of AI systems, this paper focuses on rule extraction as a means for neural networks interpretability. Through a systematic literature review, different approaches for extracting rules from feedforward neural networks, an important block in deep learning models, are identified and explored. The findings reveal a range of methods developed for over two decades, mostly suitable for shallow neural networks, with recent developments to meet deep learning models' challenges. Rules offer a transparent and intuitive means of explaining neural networks, making this study a comprehensive introduction for researchers interested in the field. While the study specifically addresses feedforward networks with supervised learning and crisp rules, future work can extend to other network types, machine learning methods, and fuzzy rule extraction.

membership infer

federate

Title: FedA3I: Annotation Quality-Aware Aggregation for Federated Medical Image Segmentation Against Heterogeneous Annotation Noise. (arXiv:2312.12838v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12838
• Code URL: null
• Copy Paste: [[2312.12838]] FedA3I: Annotation Quality-Aware Aggregation for Federated Medical Image Segmentation Against Heterogeneous Annotation Noise(http://arxiv.org/abs/2312.12838)
• Summary:

  Federated learning (FL) has emerged as a promising paradigm for training segmentation models on decentralized medical data, owing to its privacy-preserving property. However, existing research overlooks the prevalent annotation noise encountered in real-world medical datasets, which limits the performance ceilings of FL. In this paper, we, for the first time, identify and tackle this problem. For problem formulation, we propose a contour evolution for modeling non-independent and identically distributed (Non-IID) noise across pixels within each client and then extend it to the case of multi-source data to form a heterogeneous noise model (\textit{i.e.}, Non-IID annotation noise across clients). For robust learning from annotations with such two-level Non-IID noise, we emphasize the importance of data quality in model aggregation, allowing high-quality clients to have a greater impact on FL. To achieve this, we propose \textbf{Fed}erated learning with \textbf{A}nnotation qu\textbf{A}lity-aware \textbf{A}ggregat\textbf{I}on, named \textbf{FedA$^3$I}, by introducing a quality factor based on client-wise noise estimation. Specifically, noise estimation at each client is accomplished through the Gaussian mixture model and then incorporated into model aggregation in a layer-wise manner to up-weight high-quality clients. Extensive experiments on two real-world medical image segmentation datasets demonstrate the superior performance of FedA$^3$I against the state-of-the-art approaches in dealing with cross-client annotation noise. The code is available at \color{blue}{https://github.com/wnn2000/FedAAAI}.

Title: Blood Glucose Level Prediction: A Graph-based Explainable Method with Federated Learning. (arXiv:2312.12541v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12541
• Code URL: null
• Copy Paste: [[2312.12541]] Blood Glucose Level Prediction: A Graph-based Explainable Method with Federated Learning(http://arxiv.org/abs/2312.12541)
• Summary:

  In the UK, approximately 400,000 people with type 1 diabetes (T1D) rely on insulin delivery due to insufficient pancreatic insulin production. Managing blood glucose (BG) levels is crucial, with continuous glucose monitoring (CGM) playing a key role. CGM, tracking BG every 5 minutes, enables effective blood glucose level prediction (BGLP) by considering factors like carbohydrate intake and insulin delivery.

  Recent research has focused on developing sequential models for BGLP using historical BG data, incorporating additional attributes such as carbohydrate intake, insulin delivery, and time. These methods have shown notable success in BGLP, with some providing temporal explanations. However, they often lack clear correlations between attributes and their impact on BGLP. Additionally, some methods raise privacy concerns by aggregating participant data to learn population patterns.

  Addressing these limitations, we introduced a graph attentive memory (GAM) model, combining a graph attention network (GAT) with a gated recurrent unit (GRU). GAT applies graph attention to model attribute correlations, offering transparent, dynamic attribute relationships. Attention weights dynamically gauge attribute significance over time. To ensure privacy, we employed federated learning (FL), facilitating secure population pattern analysis.

  Our method was validated using the OhioT1DM'18 and OhioT1DM'20 datasets from 12 participants, focusing on 6 key attributes. We demonstrated our model's stability and effectiveness through hyperparameter impact analysis.

Title: Incremental Semi-supervised Federated Learning for Health Inference via Mobile Sensing. (arXiv:2312.12666v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12666
• Code URL: null
• Copy Paste: [[2312.12666]] Incremental Semi-supervised Federated Learning for Health Inference via Mobile Sensing(http://arxiv.org/abs/2312.12666)
• Summary:

  Mobile sensing appears as a promising solution for health inference problem (e.g., influenza-like symptom recognition) by leveraging diverse smart sensors to capture fine-grained information about human behaviors and ambient contexts. Centralized training of machine learning models can place mobile users' sensitive information under privacy risks due to data breach and misexploitation. Federated Learning (FL) enables mobile devices to collaboratively learn global models without the exposure of local private data. However, there are challenges of on-device FL deployment using mobile sensing: 1) long-term and continuously collected mobile sensing data may exhibit domain shifts as sensing objects (e.g. humans) have varying behaviors as a result of internal and/or external stimulus; 2) model retraining using all available data may increase computation and memory burden; and 3) the sparsity of annotated crowd-sourced data causes supervised FL to lack robustness. In this work, we propose FedMobile, an incremental semi-supervised federated learning algorithm, to train models semi-supervisedly and incrementally in a decentralized online fashion. We evaluate FedMobile using a real-world mobile sensing dataset for influenza-like symptom recognition. Our empirical results show that FedMobile-trained models achieve the best results in comparison to the selected baseline methods.

Title: On the Role of Server Momentum in Federated Learning. (arXiv:2312.12670v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12670
• Code URL: null
• Copy Paste: [[2312.12670]] On the Role of Server Momentum in Federated Learning(http://arxiv.org/abs/2312.12670)
• Summary:

  Federated Averaging (FedAvg) is known to experience convergence issues when encountering significant clients system heterogeneity and data heterogeneity. Server momentum has been proposed as an effective mitigation. However, existing server momentum works are restrictive in the momentum formulation, do not properly schedule hyperparameters and focus only on system homogeneous settings, which leaves the role of server momentum still an under-explored problem. In this paper, we propose a general framework for server momentum, that (a) covers a large class of momentum schemes that are unexplored in federated learning (FL), (b) enables a popular stagewise hyperparameter scheduler, (c) allows heterogeneous and asynchronous local computing. We provide rigorous convergence analysis for the proposed framework. To our best knowledge, this is the first work that thoroughly analyzes the performances of server momentum with a hyperparameter scheduler and system heterogeneity. Extensive experiments validate the effectiveness of our proposed framework.

Title: Federated Learning with Extremely Noisy Clients via Negative Distillation. (arXiv:2312.12703v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12703
• Code URL: https://github.com/linchen99/fedned
• Copy Paste: [[2312.12703]] Federated Learning with Extremely Noisy Clients via Negative Distillation(http://arxiv.org/abs/2312.12703)
• Summary:

  Federated learning (FL) has shown remarkable success in cooperatively training deep models, while typically struggling with noisy labels. Advanced works propose to tackle label noise by a re-weighting strategy with a strong assumption, i.e., mild label noise. However, it may be violated in many real-world FL scenarios because of highly contaminated clients, resulting in extreme noise ratios, e.g., $>$90%. To tackle extremely noisy clients, we study the robustness of the re-weighting strategy, showing a pessimistic conclusion: minimizing the weight of clients trained over noisy data outperforms re-weighting strategies. To leverage models trained on noisy clients, we propose a novel approach, called negative distillation (FedNed). FedNed first identifies noisy clients and employs rather than discards the noisy clients in a knowledge distillation manner. In particular, clients identified as noisy ones are required to train models using noisy labels and pseudo-labels obtained by global models. The model trained on noisy labels serves as a `bad teacher' in knowledge distillation, aiming to decrease the risk of providing incorrect information. Meanwhile, the model trained on pseudo-labels is involved in model aggregation if not identified as a noisy client. Consequently, through pseudo-labeling, FedNed gradually increases the trustworthiness of models trained on noisy clients, while leveraging all clients for model aggregation through negative distillation. To verify the efficacy of FedNed, we conduct extensive experiments under various settings, demonstrating that FedNed can consistently outperform baselines and achieve state-of-the-art performance. Our code is available at https://github.com/linChen99/FedNed.

fair

Title: FairFlow Protocol: Equitable Maximal Extractable Value (MEV) mitigation in Ethereum. (arXiv:2312.12654v1 [cs.CR])

• Paper URL: http://arxiv.org/abs/2312.12654
• Code URL: null
• Copy Paste: [[2312.12654]] FairFlow Protocol: Equitable Maximal Extractable Value (MEV) mitigation in Ethereum(http://arxiv.org/abs/2312.12654)
• Summary:

  Ethereum has emerged as a leading platform for decentralized applications (dApps) due to its robust smart contract capabilities. One of the critical issues in the Ethereum ecosystem is Maximal Extractable Value (MEV), a concept that has gained significant attention in the blockchain community. However, MEV has remained a major challenge with significant implications for the platform's operation and integrity. This paper introduces the FairFlow protocol, a novel framework designed to mitigate the effects of MEV within Ethereum's existing infrastructure. The protocol aims to provide a more equitable environment, preventing exploitation by miners or validators, and protecting user data. The combined approach of auction-based block space allocation and randomized transaction ordering significantly reduces the potential for MEV exploitation.

Title: Learning Fair Policies for Multi-stage Selection Problems from Observational Data. (arXiv:2312.13173v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.13173
• Code URL: null
• Copy Paste: [[2312.13173]] Learning Fair Policies for Multi-stage Selection Problems from Observational Data(http://arxiv.org/abs/2312.13173)
• Summary:

  We consider the problem of learning fair policies for multi-stage selection problems from observational data. This problem arises in several high-stakes domains such as company hiring, loan approval, or bail decisions where outcomes (e.g., career success, loan repayment, recidivism) are only observed for those selected. We propose a multi-stage framework that can be augmented with various fairness constraints, such as demographic parity or equal opportunity. This problem is a highly intractable infinite chance-constrained program involving the unknown joint distribution of covariates and outcomes. Motivated by the potential impact of selection decisions on people's lives and livelihoods, we propose to focus on interpretable linear selection rules. Leveraging tools from causal inference and sample average approximation, we obtain an asymptotically consistent solution to this selection problem by solving a mixed binary conic optimization problem, which can be solved using standard off-the-shelf solvers. We conduct extensive computational experiments on a variety of datasets adapted from the UCI repository on which we show that our proposed approaches can achieve an 11.6% improvement in precision and a 38% reduction in the measure of unfairness compared to the existing selection policy.

interpretability

Title: Integration and Performance Analysis of Artificial Intelligence and Computer Vision Based on Deep Learning Algorithms. (arXiv:2312.12872v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12872
• Code URL: null
• Copy Paste: [[2312.12872]] Integration and Performance Analysis of Artificial Intelligence and Computer Vision Based on Deep Learning Algorithms(http://arxiv.org/abs/2312.12872)
• Summary:

  This paper focuses on the analysis of the application effectiveness of the integration of deep learning and computer vision technologies. Deep learning achieves a historic breakthrough by constructing hierarchical neural networks, enabling end-to-end feature learning and semantic understanding of images. The successful experiences in the field of computer vision provide strong support for training deep learning algorithms. The tight integration of these two fields has given rise to a new generation of advanced computer vision systems, significantly surpassing traditional methods in tasks such as machine vision image classification and object detection. In this paper, typical image classification cases are combined to analyze the superior performance of deep neural network models while also pointing out their limitations in generalization and interpretability, proposing directions for future improvements. Overall, the efficient integration and development trend of deep learning with massive visual data will continue to drive technological breakthroughs and application expansion in the field of computer vision, making it possible to build truly intelligent machine vision systems. This deepening fusion paradigm will powerfully promote unprecedented tasks and functions in computer vision, providing stronger development momentum for related disciplines and industries.

Title: Stability of Graph Convolutional Neural Networks through the lens of small perturbation analysis. (arXiv:2312.12934v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12934
• Code URL: null
• Copy Paste: [[2312.12934]] Stability of Graph Convolutional Neural Networks through the lens of small perturbation analysis(http://arxiv.org/abs/2312.12934)
• Summary:

  In this work, we study the problem of stability of Graph Convolutional Neural Networks (GCNs) under random small perturbations in the underlying graph topology, i.e. under a limited number of insertions or deletions of edges. We derive a novel bound on the expected difference between the outputs of unperturbed and perturbed GCNs. The proposed bound explicitly depends on the magnitude of the perturbation of the eigenpairs of the Laplacian matrix, and the perturbation explicitly depends on which edges are inserted or deleted. Then, we provide a quantitative characterization of the effect of perturbing specific edges on the stability of the network. We leverage tools from small perturbation analysis to express the bounds in closed, albeit approximate, form, in order to enhance interpretability of the results, without the need to compute any perturbed shift operator. Finally, we numerically evaluate the effectiveness of the proposed bound.

Title: AutoXPCR: Automated Multi-Objective Model Selection for Time Series Forecasting. (arXiv:2312.13038v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.13038
• Code URL: null
• Copy Paste: [[2312.13038]] AutoXPCR: Automated Multi-Objective Model Selection for Time Series Forecasting(http://arxiv.org/abs/2312.13038)
• Summary:

  Automated machine learning (AutoML) streamlines the creation of ML models. While most methods select the "best" model based on predictive quality, it's crucial to acknowledge other aspects, such as interpretability and resource consumption. This holds particular importance in the context of deep neural networks (DNNs), as these models are often perceived as computationally intensive black boxes. In the challenging domain of time series forecasting, DNNs achieve stunning results, but specialized approaches for automatically selecting models are scarce. In this paper, we propose AutoXPCR - a novel method for automated and explainable multi-objective model selection. Our approach leverages meta-learning to estimate any model's performance along PCR criteria, which encompass (P)redictive error, (C)omplexity, and (R)esource demand. Explainability is addressed on multiple levels, as our interactive framework can prioritize less complex models and provide by-product explanations of recommendations. We demonstrate practical feasibility by deploying AutoXPCR on over 1000 configurations across 114 data sets from various domains. Our method clearly outperforms other model selection approaches - on average, it only requires 20% of computation costs for recommending models with 90% of the best-possible quality.

explainability

Title: ALMANACS: A Simulatability Benchmark for Language Model Explainability. (arXiv:2312.12747v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12747
• Code URL: https://github.com/edmundmills/almanacs
• Copy Paste: [[2312.12747]] ALMANACS: A Simulatability Benchmark for Language Model Explainability(http://arxiv.org/abs/2312.12747)
• Summary:

  How do we measure the efficacy of language model explainability methods? While many explainability methods have been developed, they are typically evaluated on bespoke tasks, preventing an apples-to-apples comparison. To help fill this gap, we present ALMANACS, a language model explainability benchmark. ALMANACS scores explainability methods on simulatability, i.e., how well the explanations improve behavior prediction on new inputs. The ALMANACS scenarios span twelve safety-relevant topics such as ethical reasoning and advanced AI behaviors; they have idiosyncratic premises to invoke model-specific behavior; and they have a train-test distributional shift to encourage faithful explanations. By using another language model to predict behavior based on the explanations, ALMANACS is a fully automated benchmark. We use ALMANACS to evaluate counterfactuals, rationalizations, attention, and Integrated Gradients explanations. Our results are sobering: when averaged across all topics, no explanation method outperforms the explanation-free control. We conclude that despite modest successes in prior work, developing an explanation method that aids simulatability in ALMANACS remains an open challenge.

Title: Survey on Trustworthy Graph Neural Networks: From A Causal Perspective. (arXiv:2312.12477v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12477
• Code URL: https://github.com/usail-hkust/causality-inspired-gnns
• Copy Paste: [[2312.12477]] Survey on Trustworthy Graph Neural Networks: From A Causal Perspective(http://arxiv.org/abs/2312.12477)
• Summary:

  Graph Neural Networks (GNNs) have emerged as powerful representation learning tools for capturing complex dependencies within diverse graph-structured data. Despite their success in a wide range of graph mining tasks, GNNs have raised serious concerns regarding their trustworthiness, including susceptibility to distribution shift, biases towards certain populations, and lack of explainability. Recently, integrating causal learning techniques into GNNs has sparked numerous ground-breaking studies since most of the trustworthiness issues can be alleviated by capturing the underlying data causality rather than superficial correlations. In this survey, we provide a comprehensive review of recent research efforts on causality-inspired GNNs. Specifically, we first present the key trustworthy risks of existing GNN models through the lens of causality. Moreover, we introduce a taxonomy of Causality-Inspired GNNs (CIGNNs) based on the type of causal learning capability they are equipped with, i.e., causal reasoning and causal representation learning. Besides, we systematically discuss typical methods within each category and demonstrate how they mitigate trustworthiness risks. Finally, we summarize useful resources and discuss several future directions, hoping to shed light on new research opportunities in this emerging field. The representative papers, along with open-source data and codes, are available in https://github.com/usail-hkust/Causality-Inspired-GNNs.

watermark

diffusion

Title: DiffSpectralNet : Unveiling the Potential of Diffusion Models for Hyperspectral Image Classification. (arXiv:2312.12441v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12441
• Code URL: null
• Copy Paste: [[2312.12441]] DiffSpectralNet : Unveiling the Potential of Diffusion Models for Hyperspectral Image Classification(http://arxiv.org/abs/2312.12441)
• Summary:

  Hyperspectral images (HSI) have become popular for analysing remotely sensed images in multiple domain like agriculture, medical. However, existing models struggle with complex relationships and characteristics of spectral-spatial data due to the multi-band nature and data redundancy of hyperspectral data. To address this limitation, we propose a new network called DiffSpectralNet, which combines diffusion and transformer techniques. Our approach involves a two-step process. First, we use an unsupervised learning framework based on the diffusion model to extract both high-level and low-level spectral-spatial features. The diffusion method is capable of extracting diverse and meaningful spectral-spatial features, leading to improvement in HSI classification. Then, we employ a pretrained denoising U-Net to extract intermediate hierarchical features for classification. Finally, we use a supervised transformer-based classifier to perform the HSI classification. Through comprehensive experiments on HSI datasets, we evaluate the classification performance of DiffSpectralNet. The results demonstrate that our framework significantly outperforms existing approaches, achieving state-of-the-art performance.

Title: Atlantis: Enabling Underwater Depth Estimation with Stable Diffusion. (arXiv:2312.12471v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12471
• Code URL: https://github.com/zkawfanx/atlantis
• Copy Paste: [[2312.12471]] Atlantis: Enabling Underwater Depth Estimation with Stable Diffusion(http://arxiv.org/abs/2312.12471)
• Summary:

  Monocular depth estimation has experienced significant progress on terrestrial images in recent years, largely due to deep learning advancements. However, it remains inadequate for underwater scenes, primarily because of data scarcity. Given the inherent challenges of light attenuation and backscattering in water, acquiring clear underwater images or precise depth information is notably difficult and costly. Consequently, learning-based approaches often rely on synthetic data or turn to unsupervised or self-supervised methods to mitigate this lack of data. Nonetheless, the performance of these methods is often constrained by the domain gap and looser constraints. In this paper, we propose a novel pipeline for generating photorealistic underwater images using accurate terrestrial depth data. This approach facilitates the training of supervised models for underwater depth estimation, effectively reducing the performance disparity between terrestrial and underwater environments. Contrary to prior synthetic datasets that merely apply style transfer to terrestrial images without altering the scene content, our approach uniquely creates vibrant, non-existent underwater scenes by leveraging terrestrial depth data through the innovative Stable Diffusion model. Specifically, we introduce a unique Depth2Underwater ControlNet, trained on specially prepared \{Underwater, Depth, Text\} data triplets, for this generation task. Our newly developed dataset enables terrestrial depth estimation models to achieve considerable improvements, both quantitatively and qualitatively, on unseen underwater images, surpassing their terrestrial pre-trained counterparts. Moreover, the enhanced depth accuracy for underwater scenes also aids underwater image restoration techniques that rely on depth maps, further demonstrating our dataset's utility. The dataset will be available at https://github.com/zkawfanx/Atlantis.

Title: InstructVideo: Instructing Video Diffusion Models with Human Feedback. (arXiv:2312.12490v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12490
• Code URL: null
• Copy Paste: [[2312.12490]] InstructVideo: Instructing Video Diffusion Models with Human Feedback(http://arxiv.org/abs/2312.12490)
• Summary:

  Diffusion models have emerged as the de facto paradigm for video generation. However, their reliance on web-scale data of varied quality often yields results that are visually unappealing and misaligned with the textual prompts. To tackle this problem, we propose InstructVideo to instruct text-to-video diffusion models with human feedback by reward fine-tuning. InstructVideo has two key ingredients: 1) To ameliorate the cost of reward fine-tuning induced by generating through the full DDIM sampling chain, we recast reward fine-tuning as editing. By leveraging the diffusion process to corrupt a sampled video, InstructVideo requires only partial inference of the DDIM sampling chain, reducing fine-tuning cost while improving fine-tuning efficiency. 2) To mitigate the absence of a dedicated video reward model for human preferences, we repurpose established image reward models, e.g., HPSv2. To this end, we propose Segmental Video Reward, a mechanism to provide reward signals based on segmental sparse sampling, and Temporally Attenuated Reward, a method that mitigates temporal modeling degradation during fine-tuning. Extensive experiments, both qualitative and quantitative, validate the practicality and efficacy of using image reward models in InstructVideo, significantly enhancing the visual quality of generated videos without compromising generalization capabilities. Code and models will be made publicly available.

Title: StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation. (arXiv:2312.12491v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12491
• Code URL: null
• Copy Paste: [[2312.12491]] StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation(http://arxiv.org/abs/2312.12491)
• Summary:

  We introduce StreamDiffusion, a real-time diffusion pipeline designed for interactive image generation. Existing diffusion models are adept at creating images from text or image prompts, yet they often fall short in real-time interaction. This limitation becomes particularly evident in scenarios involving continuous input, such as Metaverse, live video streaming, and broadcasting, where high throughput is imperative. To address this, we present a novel approach that transforms the original sequential denoising into the batching denoising process. Stream Batch eliminates the conventional wait-and-interact approach and enables fluid and high throughput streams. To handle the frequency disparity between data input and model throughput, we design a novel input-output queue for parallelizing the streaming process. Moreover, the existing diffusion pipeline uses classifier-free guidance(CFG), which requires additional U-Net computation. To mitigate the redundant computations, we propose a novel residual classifier-free guidance (RCFG) algorithm that reduces the number of negative conditional denoising steps to only one or even zero. Besides, we introduce a stochastic similarity filter(SSF) to optimize power consumption. Our Stream Batch achieves around 1.5x speedup compared to the sequential denoising method at different denoising levels. The proposed RCFG leads to speeds up to 2.05x higher than the conventional CFG. Combining the proposed strategies and existing mature acceleration tools makes the image-to-image generation achieve up-to 91.07fps on one RTX4090, improving the throughputs of AutoPipline developed by Diffusers over 59.56x. Furthermore, our proposed StreamDiffusion also significantly reduces the energy consumption by 2.39x on one RTX3060 and 1.99x on one RTX4090, respectively.

Title: Fixed-point Inversion for Text-to-image diffusion models. (arXiv:2312.12540v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12540
• Code URL: null
• Copy Paste: [[2312.12540]] Fixed-point Inversion for Text-to-image diffusion models(http://arxiv.org/abs/2312.12540)
• Summary:

  Text-guided diffusion models offer powerful new ways to generate and manipulate images. Several applications of these models, including image editing interpolation, and semantic augmentation, require diffusion inversion. This is the process of finding a noise seed that can be used to generate a given image. Current techniques for inverting a given image can be slow or inaccurate. The technical challenge for inverting the diffusion process arises from an implicit equation over the latent that cannot be solved in closed form. Previous approaches proposed to solve this issue by approximation or various learning schemes. Here, we formulate the problem as a fixed-point equation problem and solve it using fixed-point iterations, a well-studied approach in numerical analysis. We further identify a source of inconsistency that significantly hurts the inversion of real images encoded to the latent space. We show how to correct it by applying a prompt-aware adjustment of the encoding. Our solution, Fixed-point inversion, is much faster than previous techniques like EDICT and Null-text, with similar inversion quality. It can be combined with any pretrained diffusion model and requires no model training, prompt tuning, or additional parameters. In a series of experiments, we find that Fixed-point inversion shows improved results in several downstream tasks: image editing, image interpolation, and generation of rare objects.

Title: RealCraft: Attention Control as A Solution for Zero-shot Long Video Editing. (arXiv:2312.12635v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12635
• Code URL: null
• Copy Paste: [[2312.12635]] RealCraft: Attention Control as A Solution for Zero-shot Long Video Editing(http://arxiv.org/abs/2312.12635)
• Summary:

  Although large-scale text-to-image generative models have shown promising performance in synthesizing high-quality images, directly applying these models to image editing remains a significant challenge. This challenge is further amplified in video editing due to the additional dimension of time. Especially for editing real videos as it necessitates maintaining a stable semantic layout across the frames while executing localized edits precisely without disrupting the existing backgrounds. In this paper, we propose \textit{RealCraft}, an attention-control-based method for zero-shot editing in real videos. By employing the object-centric manipulation of cross-attention between prompts and frames and spatial-temporal attention within the frames, we achieve precise shape-wise editing along with enhanced consistency. Our model can be used directly with Stable Diffusion and operates without the need for additional localized information. We showcase our zero-shot attention-control-based method across a range of videos, demonstrating localized, high-fidelity, shape-precise and time-consistent editing in videos of various lengths, up to 64 frames.

Title: AMD:Anatomical Motion Diffusion with Interpretable Motion Decomposition and Fusion. (arXiv:2312.12763v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12763
• Code URL: null
• Copy Paste: [[2312.12763]] AMD:Anatomical Motion Diffusion with Interpretable Motion Decomposition and Fusion(http://arxiv.org/abs/2312.12763)
• Summary:

  Generating realistic human motion sequences from text descriptions is a challenging task that requires capturing the rich expressiveness of both natural language and human motion.Recent advances in diffusion models have enabled significant progress in human motion synthesis.However, existing methods struggle to handle text inputs that describe complex or long motions.In this paper, we propose the Adaptable Motion Diffusion (AMD) model, which leverages a Large Language Model (LLM) to parse the input text into a sequence of concise and interpretable anatomical scripts that correspond to the target motion.This process exploits the LLM's ability to provide anatomical guidance for complex motion synthesis.We then devise a two-branch fusion scheme that balances the influence of the input text and the anatomical scripts on the inverse diffusion process, which adaptively ensures the semantic fidelity and diversity of the synthesized motion.Our method can effectively handle texts with complex or long motion descriptions, where existing methods often fail. Experiments on datasets with relatively more complex motions, such as CLCD1 and CLCD2, demonstrate that our AMD significantly outperforms existing state-of-the-art models.

Title: All but One: Surgical Concept Erasing with Model Preservation in Text-to-Image Diffusion Models. (arXiv:2312.12807v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12807
• Code URL: null
• Copy Paste: [[2312.12807]] All but One: Surgical Concept Erasing with Model Preservation in Text-to-Image Diffusion Models(http://arxiv.org/abs/2312.12807)
• Summary:

  Text-to-Image models such as Stable Diffusion have shown impressive image generation synthesis, thanks to the utilization of large-scale datasets. However, these datasets may contain sexually explicit, copyrighted, or undesirable content, which allows the model to directly generate them. Given that retraining these large models on individual concept deletion requests is infeasible, fine-tuning algorithms have been developed to tackle concept erasing in diffusion models. While these algorithms yield good concept erasure, they all present one of the following issues: 1) the corrupted feature space yields synthesis of disintegrated objects, 2) the initially synthesized content undergoes a divergence in both spatial structure and semantics in the generated images, and 3) sub-optimal training updates heighten the model's susceptibility to utility harm. These issues severely degrade the original utility of generative models. In this work, we present a new approach that solves all of these challenges. We take inspiration from the concept of classifier guidance and propose a surgical update on the classifier guidance term while constraining the drift of the unconditional score term. Furthermore, our algorithm empowers the user to select an alternative to the erasing concept, allowing for more controllability. Our experimental results show that our algorithm not only erases the target concept effectively but also preserves the model's generation capability.

Title: ReCo-Diff: Explore Retinex-Based Condition Strategy in Diffusion Model for Low-Light Image Enhancement. (arXiv:2312.12826v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12826
• Code URL: null
• Copy Paste: [[2312.12826]] ReCo-Diff: Explore Retinex-Based Condition Strategy in Diffusion Model for Low-Light Image Enhancement(http://arxiv.org/abs/2312.12826)
• Summary:

  Low-light image enhancement (LLIE) has achieved promising performance by employing conditional diffusion models. In this study, we propose ReCo-Diff, a novel approach that incorporates Retinex-based prior as an additional pre-processing condition to regulate the generating capabilities of the diffusion model. ReCo-Diff first leverages a pre-trained decomposition network to produce initial reflectance and illumination maps of the low-light image. Then, an adjustment network is introduced to suppress the noise in the reflectance map and brighten the illumination map, thus forming the learned Retinex-based condition. The condition is integrated into a refinement network, implementing Retinex-based conditional modules that offer sufficient guidance at both feature- and image-levels. By treating Retinex theory as a condition, ReCo-Diff presents a unique perspective for establishing an LLIE-specific diffusion model. Extensive experiments validate the rationality and superiority of our ReCo-Diff approach. The code will be made publicly available.

Title: RadEdit: stress-testing biomedical vision models via diffusion image editing. (arXiv:2312.12865v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12865
• Code URL: null
• Copy Paste: [[2312.12865]] RadEdit: stress-testing biomedical vision models via diffusion image editing(http://arxiv.org/abs/2312.12865)
• Summary:

  Biomedical imaging datasets are often small and biased, meaning that real-world performance of predictive models can be substantially lower than expected from internal testing. This work proposes using generative image editing to simulate dataset shifts and diagnose failure modes of biomedical vision models; this can be used in advance of deployment to assess readiness, potentially reducing cost and patient harm. Existing editing methods can produce undesirable changes, with spurious correlations learned due to the co-occurrence of disease and treatment interventions, limiting practical applicability. To address this, we train a text-to-image diffusion model on multiple chest X-ray datasets and introduce a new editing method RadEdit that uses multiple masks, if present, to constrain changes and ensure consistency in the edited images. We consider three types of dataset shifts: acquisition shift, manifestation shift, and population shift, and demonstrate that our approach can diagnose failures and quantify model robustness without additional data collection, complementing more qualitative tools for explainable AI.

Title: DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis. (arXiv:2312.13016v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.13016
• Code URL: https://github.com/FreedomGu/DiffPortrait3D
• Copy Paste: [[2312.13016]] DiffPortrait3D: Controllable Diffusion for Zero-Shot Portrait View Synthesis(http://arxiv.org/abs/2312.13016)
• Summary:

  We present DiffPortrait3D, a conditional diffusion model that is capable of synthesizing 3D-consistent photo-realistic novel views from as few as a single in-the-wild portrait. Specifically, given a single RGB input, we aim to synthesize plausible but consistent facial details rendered from novel camera views with retained both identity and facial expression. In lieu of time-consuming optimization and fine-tuning, our zero-shot method generalizes well to arbitrary face portraits with unposed camera views, extreme facial expressions, and diverse artistic depictions. At its core, we leverage the generative prior of 2D diffusion models pre-trained on large-scale image datasets as our rendering backbone, while the denoising is guided with disentangled attentive control of appearance and camera pose. To achieve this, we first inject the appearance context from the reference image into the self-attention layers of the frozen UNets. The rendering view is then manipulated with a novel conditional control module that interprets the camera pose by watching a condition image of a crossed subject from the same view. Furthermore, we insert a trainable cross-view attention module to enhance view consistency, which is further strengthened with a novel 3D-aware noise generation process during inference. We demonstrate state-of-the-art results both qualitatively and quantitatively on our challenging in-the-wild and multi-view benchmarks.

Title: Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models. (arXiv:2312.12487v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12487
• Code URL: null
• Copy Paste: [[2312.12487]] Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models(http://arxiv.org/abs/2312.12487)
• Summary:

  This paper presents a comprehensive study on the role of Classifier-Free Guidance (CFG) in text-conditioned diffusion models from the perspective of inference efficiency. In particular, we relax the default choice of applying CFG in all diffusion steps and instead search for efficient guidance policies. We formulate the discovery of such policies in the differentiable Neural Architecture Search framework. Our findings suggest that the denoising steps proposed by CFG become increasingly aligned with simple conditional steps, which renders the extra neural network evaluation of CFG redundant, especially in the second half of the denoising process. Building upon this insight, we propose "Adaptive Guidance" (AG), an efficient variant of CFG, that adaptively omits network evaluations when the denoising process displays convergence. Our experiments demonstrate that AG preserves CFG's image quality while reducing computation by 25%. Thus, AG constitutes a plug-and-play alternative to Guidance Distillation, achieving 50% of the speed-ups of the latter while being training-free and retaining the capacity to handle negative prompts. Finally, we uncover further redundancies of CFG in the first half of the diffusion process, showing that entire neural function evaluations can be replaced by simple affine transformations of past score estimates. This method, termed LinearAG, offers even cheaper inference at the cost of deviating from the baseline model. Our findings provide insights into the efficiency of the conditional denoising process that contribute to more practical and swift deployment of text-conditioned diffusion models.

noise learning

data-free

transformer

Title: Hierarchical Classification System for Breast Cancer Specimen Report (HCSBC) -- an end-to-end model for characterizing severity and diagnosis. (arXiv:2312.12442v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12442
• Code URL: null
• Copy Paste: [[2312.12442]] Hierarchical Classification System for Breast Cancer Specimen Report (HCSBC) -- an end-to-end model for characterizing severity and diagnosis(http://arxiv.org/abs/2312.12442)
• Summary:

  Automated classification of cancer pathology reports can extract information from unstructured reports and categorize each report into structured diagnosis and severity categories. Thus, such system can reduce the burden for populating tumor registries, help registration for clinical trial as well as developing large dataset for deep learning model development using true pathologic ground truth. However, the content of breast pathology reports can be difficult for categorize due to the high linguistic variability in content and wide variety of potential diagnoses >50. Existing NLP models are primarily focused on developing classifier for primary breast cancer types (e.g. IDC, DCIS, ILC) and tumor characteristics, and ignore the rare diagnosis of cancer subtypes. We then developed a hierarchical hybrid transformer-based pipeline (59 labels) - Hierarchical Classification System for Breast Cancer Specimen Report (HCSBC), which utilizes the potential of the transformer context-preserving NLP technique and compared our model to several state of the art ML and DL models. We trained the model on the EUH data and evaluated our model's performance on two external datasets - MGH and Mayo Clinic. We publicly release the code and a live application under Huggingface spaces repository

Title: Open Vocabulary Semantic Scene Sketch Understanding. (arXiv:2312.12463v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12463
• Code URL: null
• Copy Paste: [[2312.12463]] Open Vocabulary Semantic Scene Sketch Understanding(http://arxiv.org/abs/2312.12463)
• Summary:

  We study the underexplored but fundamental vision problem of machine understanding of abstract freehand scene sketches. We introduce a sketch encoder that results in semantically-aware feature space, which we evaluate by testing its performance on a semantic sketch segmentation task. To train our model we rely only on the availability of bitmap sketches with their brief captions and do not require any pixel-level annotations. To obtain generalization to a large set of sketches and categories, we build on a vision transformer encoder pretrained with the CLIP model. We freeze the text encoder and perform visual-prompt tuning of the visual encoder branch while introducing a set of critical modifications. Firstly, we augment the classical key-query (k-q) self-attention blocks with value-value (v-v) self-attention blocks. Central to our model is a two-level hierarchical network design that enables efficient semantic disentanglement: The first level ensures holistic scene sketch encoding, and the second level focuses on individual categories. We, then, in the second level of the hierarchy, introduce a cross-attention between textual and visual branches. Our method outperforms zero-shot CLIP pixel accuracy of segmentation results by 37 points, reaching an accuracy of $85.5\%$ on the FS-COCO sketch dataset. Finally, we conduct a user study that allows us to identify further improvements needed over our method to reconcile machine and human understanding of scene sketches.

Title: MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers. (arXiv:2312.12468v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12468
• Code URL: null
• Copy Paste: [[2312.12468]] MaskINT: Video Editing via Interpolative Non-autoregressive Masked Transformers(http://arxiv.org/abs/2312.12468)
• Summary:

  Recent advances in generative AI have significantly enhanced image and video editing, particularly in the context of text prompt control. State-of-the-art approaches predominantly rely on diffusion models to accomplish these tasks. However, the computational demands of diffusion-based methods are substantial, often necessitating large-scale paired datasets for training, and therefore challenging the deployment in practical applications. This study addresses this challenge by breaking down the text-based video editing process into two separate stages. In the first stage, we leverage an existing text-to-image diffusion model to simultaneously edit a few keyframes without additional fine-tuning. In the second stage, we introduce an efficient model called MaskINT, which is built on non-autoregressive masked generative transformers and specializes in frame interpolation between the keyframes, benefiting from structural guidance provided by intermediate frames. Our comprehensive set of experiments illustrates the efficacy and efficiency of MaskINT when compared to other diffusion-based methodologies. This research offers a practical solution for text-based video editing and showcases the potential of non-autoregressive masked generative transformers in this domain.

Title: Hierarchical Vision Transformers for Context-Aware Prostate Cancer Grading in Whole Slide Images. (arXiv:2312.12619v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12619
• Code URL: https://github.com/computationalpathologygroup/hvit
• Copy Paste: [[2312.12619]] Hierarchical Vision Transformers for Context-Aware Prostate Cancer Grading in Whole Slide Images(http://arxiv.org/abs/2312.12619)
• Summary:

  Vision Transformers (ViTs) have ushered in a new era in computer vision, showcasing unparalleled performance in many challenging tasks. However, their practical deployment in computational pathology has largely been constrained by the sheer size of whole slide images (WSIs), which result in lengthy input sequences. Transformers faced a similar limitation when applied to long documents, and Hierarchical Transformers were introduced to circumvent it. Given the analogous challenge with WSIs and their inherent hierarchical structure, Hierarchical Vision Transformers (H-ViTs) emerge as a promising solution in computational pathology. This work delves into the capabilities of H-ViTs, evaluating their efficiency for prostate cancer grading in WSIs. Our results show that they achieve competitive performance against existing state-of-the-art solutions.

Title: Cached Transformers: Improving Transformers with Differentiable Memory Cache. (arXiv:2312.12742v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12742
• Code URL: null
• Copy Paste: [[2312.12742]] Cached Transformers: Improving Transformers with Differentiable Memory Cache(http://arxiv.org/abs/2312.12742)
• Summary:

  This work introduces a new Transformer model called Cached Transformer, which uses Gated Recurrent Cached (GRC) attention to extend the self-attention mechanism with a differentiable memory cache of tokens. GRC attention enables attending to both past and current tokens, increasing the receptive field of attention and allowing for exploring long-range dependencies. By utilizing a recurrent gating unit to continuously update the cache, our model achieves significant advancements in \textbf{six} language and vision tasks, including language modeling, machine translation, ListOPs, image classification, object detection, and instance segmentation. Furthermore, our approach surpasses previous memory-based techniques in tasks such as language modeling and displays the ability to be applied to a broader range of situations.

Title: Sign Language Production with Latent Motion Transformer. (arXiv:2312.12917v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12917
• Code URL: null
• Copy Paste: [[2312.12917]] Sign Language Production with Latent Motion Transformer(http://arxiv.org/abs/2312.12917)
• Summary:

  Sign Language Production (SLP) is the tough task of turning sign language into sign videos. The main goal of SLP is to create these videos using a sign gloss. In this research, we've developed a new method to make high-quality sign videos without using human poses as a middle step. Our model works in two main parts: first, it learns from a generator and the video's hidden features, and next, it uses another model to understand the order of these hidden features. To make this method even better for sign videos, we make several significant improvements. (i) In the first stage, we take an improved 3D VQ-GAN to learn downsampled latent representations. (ii) In the second stage, we introduce sequence-to-sequence attention to better leverage conditional information. (iii) The separated two-stage training discards the realistic visual semantic of the latent codes in the second stage. To endow the latent sequences semantic information, we extend the token-level autoregressive latent codes learning with perceptual loss and reconstruction loss for the prior model with visual perception. Compared with previous state-of-the-art approaches, our model performs consistently better on two word-level sign language datasets, i.e., WLASL and NMFs-CSL.

Title: D3Former: Jointly Learning Repeatable Dense Detectors and Feature-enhanced Descriptors via Saliency-guided Transformer. (arXiv:2312.12970v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12970
• Code URL: null
• Copy Paste: [[2312.12970]] D3Former: Jointly Learning Repeatable Dense Detectors and Feature-enhanced Descriptors via Saliency-guided Transformer(http://arxiv.org/abs/2312.12970)
• Summary:

  Establishing accurate and representative matches is a crucial step in addressing the point cloud registration problem. A commonly employed approach involves detecting keypoints with salient geometric features and subsequently mapping these keypoints from one frame of the point cloud to another. However, methods within this category are hampered by the repeatability of the sampled keypoints. In this paper, we introduce a saliency-guided trans\textbf{former}, referred to as \textit{D3Former}, which entails the joint learning of repeatable \textbf{D}ense \textbf{D}etectors and feature-enhanced \textbf{D}escriptors. The model comprises a Feature Enhancement Descriptor Learning (FEDL) module and a Repetitive Keypoints Detector Learning (RKDL) module. The FEDL module utilizes a region attention mechanism to enhance feature distinctiveness, while the RKDL module focuses on detecting repeatable keypoints to enhance matching capabilities. Extensive experimental results on challenging indoor and outdoor benchmarks demonstrate that our proposed method consistently outperforms state-of-the-art point cloud matching methods. Notably, tests on 3DLoMatch, even with a low overlap ratio, show that our method consistently outperforms recently published approaches such as RoReg and RoITr. For instance, with the number of extracted keypoints reduced to 250, the registration recall scores for RoReg, RoITr, and our method are 64.3\%, 73.6\%, and 76.5\%, respectively.

Title: Can Transformers Learn Sequential Function Classes In Context?. (arXiv:2312.12655v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12655
• Code URL: null
• Copy Paste: [[2312.12655]] Can Transformers Learn Sequential Function Classes In Context?(http://arxiv.org/abs/2312.12655)
• Summary:

  In-context learning (ICL) has revolutionized the capabilities of transformer models in NLP. In our project, we extend the understanding of the mechanisms underpinning ICL by exploring whether transformers can learn from sequential, non-textual function class data distributions. We introduce a novel sliding window sequential function class and employ toy-sized transformers with a GPT-2 architecture to conduct our experiments. Our analysis indicates that these models can indeed leverage ICL when trained on non-textual sequential function classes. Additionally, our experiments with randomized y-label sequences highlights that transformers retain some ICL capabilities even when the label associations are obfuscated. We provide evidence that transformers can reason with and understand sequentiality encoded within function classes, as reflected by the effective learning of our proposed tasks. Our results also show that the performance deteriorated with increasing randomness in the labels, though not to the extent one might expect, implying a potential robustness of learned sequentiality against label noise. Future research may want to look into how previous explanations of transformers, such as induction heads and task vectors, relate to sequentiality in ICL in these toy examples. Our investigation lays the groundwork for further research into how transformers process and perceive sequential data.

Title: DSFormer: Effective Compression of Text-Transformers by Dense-Sparse Weight Factorization. (arXiv:2312.13211v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.13211
• Code URL: null
• Copy Paste: [[2312.13211]] DSFormer: Effective Compression of Text-Transformers by Dense-Sparse Weight Factorization(http://arxiv.org/abs/2312.13211)
• Summary:

  With the tremendous success of large transformer models in natural language understanding, down-sizing them for cost-effective deployments has become critical. Recent studies have explored the low-rank weight factorization techniques which are efficient to train, and apply out-of-the-box to any transformer architecture. Unfortunately, the low-rank assumption tends to be over-restrictive and hinders the expressiveness of the compressed model. This paper proposes, DSFormer, a simple alternative factorization scheme which expresses a target weight matrix as the product of a small dense and a semi-structured sparse matrix. The resulting approximation is more faithful to the weight distribution in transformers and therefore achieves a stronger efficiency-accuracy trade-off. Another concern with existing factorizers is their dependence on a task-unaware initialization step which degrades the accuracy of the resulting model. DSFormer addresses this issue through a novel Straight-Through Factorizer (STF) algorithm that jointly learns all the weight factorizations to directly maximize the final task accuracy. Extensive experiments on multiple natural language understanding benchmarks demonstrate that DSFormer obtains up to 40% better compression than the state-of-the-art low-rank factorizers, leading semi-structured sparsity baselines and popular knowledge distillation approaches. Our approach is also orthogonal to mainstream compressors and offers up to 50% additional compression when added to popular distilled, layer-shared and quantized transformers. We empirically evaluate the benefits of STF over conventional optimization practices.

Title: Learning Flexible Body Collision Dynamics with Hierarchical Contact Mesh Transformer. (arXiv:2312.12467v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12467
• Code URL: null
• Copy Paste: [[2312.12467]] Learning Flexible Body Collision Dynamics with Hierarchical Contact Mesh Transformer(http://arxiv.org/abs/2312.12467)
• Summary:

  Recently, many mesh-based graph neural network (GNN) models have been proposed for modeling complex high-dimensional physical systems. Remarkable achievements have been made in significantly reducing the solving time compared to traditional numerical solvers. These methods are typically designed to i) reduce the computational cost in solving physical dynamics and/or ii) propose techniques to enhance the solution accuracy in fluid and rigid body dynamics. However, it remains under-explored whether they are effective in addressing the challenges of flexible body dynamics, where instantaneous collisions occur within a very short timeframe. In this paper, we present Hierarchical Contact Mesh Transformer (HCMT), which uses hierarchical mesh structures and can learn long-range dependencies (occurred by collisions) among spatially distant positions of a body -- two close positions in a higher-level mesh corresponds to two distant positions in a lower-level mesh. HCMT enables long-range interactions, and the hierarchical mesh structure quickly propagates collision effects to faraway positions. To this end, it consists of a contact mesh Transformer and a hierarchical mesh Transformer (CMT and HMT, respectively). Lastly, we propose a flexible body dynamics dataset, consisting of trajectories that reflect experimental settings frequently used in the display industry for product designs. We also compare the performance of several baselines using well-known benchmark datasets. Our results show that HCMT provides significant performance improvements over existing methods.

generative

Title: How Good Are Deep Generative Models for Solving Inverse Problems?. (arXiv:2312.12691v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12691
• Code URL: null
• Copy Paste: [[2312.12691]] How Good Are Deep Generative Models for Solving Inverse Problems?(http://arxiv.org/abs/2312.12691)
• Summary:

  Deep generative models, such as diffusion models, GANs, and IMLE, have shown impressive capability in tackling inverse problems. However, the validity of model-generated solutions w.r.t. the forward problem and the reliability of associated uncertainty estimates remain understudied. This study evaluates recent diffusion-based, GAN-based, and IMLE-based methods on three inverse problems, i.e., $16\times$ super-resolution, colourization, and image decompression. We assess the validity of these models' outputs as solutions to the inverse problems and conduct a thorough analysis of the reliability of the models' estimates of uncertainty over the solution. Overall, we find that the IMLE-based CHIMLE method outperforms other methods in terms of producing valid solutions and reliable uncertainty estimates.

Title: Quantifying Bias in Text-to-Image Generative Models. (arXiv:2312.13053v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.13053
• Code URL: null
• Copy Paste: [[2312.13053]] Quantifying Bias in Text-to-Image Generative Models(http://arxiv.org/abs/2312.13053)
• Summary:

  Bias in text-to-image (T2I) models can propagate unfair social representations and may be used to aggressively market ideas or push controversial agendas. Existing T2I model bias evaluation methods only focus on social biases. We look beyond that and instead propose an evaluation methodology to quantify general biases in T2I generative models, without any preconceived notions. We assess four state-of-the-art T2I models and compare their baseline bias characteristics to their respective variants (two for each), where certain biases have been intentionally induced. We propose three evaluation metrics to assess model biases including: (i) Distribution bias, (ii) Jaccard hallucination and (iii) Generative miss-rate. We conduct two evaluation studies, modelling biases under general, and task-oriented conditions, using a marketing scenario as the domain for the latter. We also quantify social biases to compare our findings to related works. Finally, our methodology is transferred to evaluate captioned-image datasets and measure their bias. Our approach is objective, domain-agnostic and consistently measures different forms of T2I model biases. We have developed a web application and practical implementation of what has been proposed in this work, which is at https://huggingface.co/spaces/JVice/try-before-you-bias. A video series with demonstrations is available at https://www.youtube.com/channel/UCk-0xyUyT0MSd_hkp4jQt1Q

Title: Building a Llama2-finetuned LLM for Odia Language Utilizing Domain Knowledge Instruction Set. (arXiv:2312.12624v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.12624
• Code URL: null
• Copy Paste: [[2312.12624]] Building a Llama2-finetuned LLM for Odia Language Utilizing Domain Knowledge Instruction Set(http://arxiv.org/abs/2312.12624)
• Summary:

  Building LLMs for languages other than English is in great demand due to the unavailability and performance of multilingual LLMs, such as understanding the local context. The problem is critical for low-resource languages due to the need for instruction sets. In a multilingual country like India, there is a need for LLMs supporting Indic languages to provide generative AI and LLM-based technologies and services to its citizens.

  This paper presents our approach of i) generating a large Odia instruction set, including domain knowledge data suitable for LLM fine-tuning, and ii) building a Llama2-finetuned model tailored for enhanced performance in the Odia domain. The proposed work will help researchers build an instruction set and LLM, particularly for Indic languages. We will release the model and instruction set for the public for research and noncommercial purposes.

Title: Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?. (arXiv:2312.12683v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.12683
• Code URL: https://github.com/zurichnlp/multilingual-instruction-tuning
• Copy Paste: [[2312.12683]] Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?(http://arxiv.org/abs/2312.12683)
• Summary:

  The vast majority of today's large language models are English-centric, having been pretrained predominantly on English text. Yet, in order to meet user expectations, models need to be able to respond appropriately in multiple languages once deployed in downstream applications. Given limited exposure to other languages during pretraining, cross-lingual transfer is important for achieving decent performance in non-English settings. In this work, we investigate just how much multilinguality is required during finetuning to elicit strong cross-lingual generalisation across a range of tasks and target languages. We find that, compared to English-only finetuning, multilingual instruction tuning with as few as three languages significantly improves a model's cross-lingual transfer abilities on generative tasks that assume input/output language agreement, while being of less importance for highly structured tasks. Our code and data is available at https://github.com/ZurichNLP/multilingual-instruction-tuning.

Title: In Generative AI we Trust: Can Chatbots Effectively Verify Political Information?. (arXiv:2312.13096v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.13096
• Code URL: null
• Copy Paste: [[2312.13096]] In Generative AI we Trust: Can Chatbots Effectively Verify Political Information?(http://arxiv.org/abs/2312.13096)
• Summary:

  This article presents a comparative analysis of the ability of two large language model (LLM)-based chatbots, ChatGPT and Bing Chat, recently rebranded to Microsoft Copilot, to detect veracity of political information. We use AI auditing methodology to investigate how chatbots evaluate true, false, and borderline statements on five topics: COVID-19, Russian aggression against Ukraine, the Holocaust, climate change, and LGBTQ+ related debates. We compare how the chatbots perform in high- and low-resource languages by using prompts in English, Russian, and Ukrainian. Furthermore, we explore the ability of chatbots to evaluate statements according to political communication concepts of disinformation, misinformation, and conspiracy theory, using definition-oriented prompts. We also systematically test how such evaluations are influenced by source bias which we model by attributing specific claims to various political and social actors. The results show high performance of ChatGPT for the baseline veracity evaluation task, with 72 percent of the cases evaluated correctly on average across languages without pre-training. Bing Chat performed worse with a 67 percent accuracy. We observe significant disparities in how chatbots evaluate prompts in high- and low-resource languages and how they adapt their evaluations to political communication concepts with ChatGPT providing more nuanced outputs than Bing Chat. Finally, we find that for some veracity detection-related tasks, the performance of chatbots varied depending on the topic of the statement or the source to which it is attributed. These findings highlight the potential of LLM-based chatbots in tackling different forms of false information in online environments, but also points to the substantial variation in terms of how such potential is realized due to specific factors, such as language of the prompt or the topic.

Title: FSscore: A Machine Learning-based Synthetic Feasibility Score Leveraging Human Expertise. (arXiv:2312.12737v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12737
• Code URL: null
• Copy Paste: [[2312.12737]] FSscore: A Machine Learning-based Synthetic Feasibility Score Leveraging Human Expertise(http://arxiv.org/abs/2312.12737)
• Summary:

  Determining whether a molecule can be synthesized is crucial for many aspects of chemistry and drug discovery, allowing prioritization of experimental work and ranking molecules in de novo design tasks. Existing scoring approaches to assess synthetic feasibility struggle to extrapolate to out-of-distribution chemical spaces or fail to discriminate based on minor differences such as chirality that might be obvious to trained chemists. This work aims to address these limitations by introducing the Focused Synthesizability score (FSscore), which learns to rank structures based on binary preferences using a graph attention network. First, a baseline trained on an extensive set of reactant-product pairs is established that subsequently is fine-tuned with expert human feedback on a chemical space of interest. Fine-tuning on focused datasets improves performance on these chemical scopes over the pre-trained model exhibiting moderate performance and generalizability. This enables distinguishing hard- from easy-to-synthesize molecules and improving the synthetic accessibility of generative model outputs. On very complex scopes with limited labels achieving satisfactory gains remains challenging. The FSscore showcases how human expert feedback can be utilized to optimize the assessment of synthetic feasibility for a variety of applications.

Title: Class Conditional Time Series Generation with Structured Noise Space GAN. (arXiv:2312.12946v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12946
• Code URL: null
• Copy Paste: [[2312.12946]] Class Conditional Time Series Generation with Structured Noise Space GAN(http://arxiv.org/abs/2312.12946)
• Summary:

  This paper introduces Structured Noise Space GAN (SNS-GAN), a novel approach in the field of generative modeling specifically tailored for class-conditional generation in both image and time series data. It addresses the challenge of effectively integrating class labels into generative models without requiring structural modifications to the network. The SNS-GAN method embeds class conditions within the generator's noise space, simplifying the training process and enhancing model versatility. The model's efficacy is demonstrated through qualitative validations in the image domain and superior performance in time series generation compared to baseline models. This research opens new avenues for the application of GANs in various domains, including but not limited to time series and image data generation.

Title: Pre-training of Molecular GNNs as Conditional Boltzmann Generator. (arXiv:2312.13110v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.13110
• Code URL: null
• Copy Paste: [[2312.13110]] Pre-training of Molecular GNNs as Conditional Boltzmann Generator(http://arxiv.org/abs/2312.13110)
• Summary:

  Learning representations of molecular structures using deep learning is a fundamental problem in molecular property prediction tasks. Molecules inherently exist in the real world as three-dimensional structures; furthermore, they are not static but in continuous motion in the 3D Euclidean space, forming a potential energy surface. Therefore, it is desirable to generate multiple conformations in advance and extract molecular representations using a 4D-QSAR model that incorporates multiple conformations. However, this approach is impractical for drug and material discovery tasks because of the computational cost of obtaining multiple conformations. To address this issue, we propose a pre-training method for molecular GNNs using an existing dataset of molecular conformations to generate a latent vector universal to multiple conformations from a 2D molecular graph. Our method, called Boltzmann GNN, is formulated by maximizing the conditional marginal likelihood of a conditional generative model for conformations generation. We show that our model has a better prediction performance for molecular properties than existing pre-training methods using molecular graphs and three-dimensional molecular structures.

Title: Neural Stochastic Differential Equations with Change Points: A Generative Adversarial Approach. (arXiv:2312.13152v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.13152
• Code URL: null
• Copy Paste: [[2312.13152]] Neural Stochastic Differential Equations with Change Points: A Generative Adversarial Approach(http://arxiv.org/abs/2312.13152)
• Summary:

  Stochastic differential equations (SDEs) have been widely used to model real world random phenomena. Existing works mainly focus on the case where the time series is modeled by a single SDE, which might be restrictive for modeling time series with distributional shift. In this work, we propose a change point detection algorithm for time series modeled as neural SDEs. Given a time series dataset, the proposed method jointly learns the unknown change points and the parameters of distinct neural SDE models corresponding to each change point. Specifically, the SDEs are learned under the framework of generative adversarial networks (GANs) and the change points are detected based on the output of the GAN discriminator in a forward pass. At each step of the proposed algorithm, the change points and the SDE model parameters are updated in an alternating fashion. Numerical results on both synthetic and real datasets are provided to validate the performance of our algorithm in comparison to classical change point detection benchmarks, standard GAN-based neural SDEs, and other state-of-the-art deep generative models for time series data.

large language model

Title: Exploring Multimodal Large Language Models for Radiology Report Error-checking. (arXiv:2312.13103v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.13103
• Code URL: null
• Copy Paste: [[2312.13103]] Exploring Multimodal Large Language Models for Radiology Report Error-checking(http://arxiv.org/abs/2312.13103)
• Summary:

  This paper proposes one of the first clinical applications of multimodal large language models (LLMs) as an assistant for radiologists to check errors in their reports. We created an evaluation dataset from two real-world radiology datasets (MIMIC-CXR and IU-Xray), with 1,000 subsampled reports each. A subset of original reports was modified to contain synthetic errors by introducing various type of mistakes. The evaluation contained two difficulty levels: SIMPLE for binary error-checking and COMPLEX for identifying error types. LLaVA (Large Language and Visual Assistant) variant models, including our instruction-tuned model, were used for the evaluation. Additionally, a domain expert evaluation was conducted on a small test set. At the SIMPLE level, the LLaVA v1.5 model outperformed other publicly available models. Instruction tuning significantly enhanced performance by 47.4% and 25.4% on MIMIC-CXR and IU-Xray data, respectively. The model also surpassed the domain experts accuracy in the MIMIC-CXR dataset by 1.67%. Notably, among the subsets (N=21) of the test set where a clinician did not achieve the correct conclusion, the LLaVA ensemble mode correctly identified 71.4% of these cases. This study marks a promising step toward utilizing multi-modal LLMs to enhance diagnostic accuracy in radiology. The ensemble model demonstrated comparable performance to clinicians, even capturing errors overlooked by humans. Nevertheless, future work is needed to improve the model ability to identify the types of inconsistency.

Title: ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation. (arXiv:2312.13108v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.13108
• Code URL: null
• Copy Paste: [[2312.13108]] ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation(http://arxiv.org/abs/2312.13108)
• Summary:

  Graphical User Interface (GUI) automation holds significant promise for assisting users with complex tasks, thereby boosting human productivity. Existing works leveraging Large Language Model (LLM) or LLM-based AI agents have shown capabilities in automating tasks on Android and Web platforms. However, these tasks are primarily aimed at simple device usage and entertainment operations. This paper presents a novel benchmark, AssistGUI, to evaluate whether models are capable of manipulating the mouse and keyboard on the Windows platform in response to user-requested tasks. We carefully collected a set of 100 tasks from nine widely-used software applications, such as, After Effects and MS Word, each accompanied by the necessary project files for better evaluation. Moreover, we propose an advanced Actor-Critic Embodied Agent framework, which incorporates a sophisticated GUI parser driven by an LLM-agent and an enhanced reasoning mechanism adept at handling lengthy procedural tasks. Our experimental results reveal that our GUI Parser and Reasoning mechanism outshine existing methods in performance. Nevertheless, the potential remains substantial, with the best model attaining only a 46% success rate on our benchmark. We conclude with a thorough analysis of the current methods' limitations, setting the stage for future breakthroughs in this domain.

Title: Towards Better Serialization of Tabular Data for Few-shot Classification. (arXiv:2312.12464v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12464
• Code URL: null
• Copy Paste: [[2312.12464]] Towards Better Serialization of Tabular Data for Few-shot Classification(http://arxiv.org/abs/2312.12464)
• Summary:

  We present a study on the integration of Large Language Models (LLMs) in tabular data classification, emphasizing an efficient framework. Building upon existing work done in TabLLM (arXiv:2210.10723), we introduce three novel serialization techniques, including the standout LaTeX serialization method. This method significantly boosts the performance of LLMs in processing domain-specific datasets, Our method stands out for its memory efficiency and ability to fully utilize complex data structures. Through extensive experimentation, including various serialization approaches like feature combination and importance, we demonstrate our work's superiority in accuracy and efficiency over traditional models.

Title: Mini-GPTs: Efficient Large Language Models through Contextual Pruning. (arXiv:2312.12682v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.12682
• Code URL: null
• Copy Paste: [[2312.12682]] Mini-GPTs: Efficient Large Language Models through Contextual Pruning(http://arxiv.org/abs/2312.12682)
• Summary:

  In AI research, the optimization of Large Language Models (LLMs) remains a significant challenge, crucial for advancing the field's practical applications and sustainability. Building upon the foundational work of Professor Song Han's lab at MIT, this paper introduces a novel approach in developing Mini-GPTs via contextual pruning. Our methodology strategically prunes the computational architecture of traditional LLMs, like Phi-1.5, focusing on retaining core functionalities while drastically reducing model sizes. We employ the technique across diverse and complex datasets, including US law, Medical Q&A, Skyrim dialogue, English-Taiwanese translation, and Economics articles. The results underscore the efficiency and effectiveness of contextual pruning, not merely as a theoretical concept but as a practical tool in developing domain-specific, resource-efficient LLMs. Contextual pruning is a promising method for building domain-specific LLMs, and this research is a building block towards future development with more hardware compute, refined fine-tuning, and quantization.

Title: Learning and Forgetting Unsafe Examples in Large Language Models. (arXiv:2312.12736v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.12736
• Code URL: null
• Copy Paste: [[2312.12736]] Learning and Forgetting Unsafe Examples in Large Language Models(http://arxiv.org/abs/2312.12736)
• Summary:

  As the number of large language models (LLMs) released to the public grows, there is a pressing need to understand the safety implications associated with these models learning from third-party custom finetuning data. We explore the behavior of LLMs finetuned on noisy custom data containing unsafe content, represented by datasets that contain biases, toxicity, and harmfulness, finding that while aligned LLMs can readily learn this unsafe content, they also tend to forget it more significantly than other examples when subsequently finetuned on safer content. Drawing inspiration from the discrepancies in forgetting, we introduce the "ForgetFilter" algorithm, which filters unsafe data based on how strong the model's forgetting signal is for that data. We demonstrate that the ForgetFilter algorithm ensures safety in customized finetuning without compromising downstream task performance, unlike sequential safety finetuning. ForgetFilter outperforms alternative strategies like replay and moral self-correction in curbing LLMs' ability to assimilate unsafe content during custom finetuning, e.g. 75% lower than not applying any safety measures and 62% lower than using self-correction in toxicity score.

Title: Fine-tuning Large Language Models for Adaptive Machine Translation. (arXiv:2312.12740v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.12740
• Code URL: null
• Copy Paste: [[2312.12740]] Fine-tuning Large Language Models for Adaptive Machine Translation(http://arxiv.org/abs/2312.12740)
• Summary:

  This paper presents the outcomes of fine-tuning Mistral 7B, a general-purpose large language model (LLM), for adaptive machine translation (MT). The fine-tuning process involves utilising a combination of zero-shot and one-shot translation prompts within the medical domain. The primary objective is to enhance real-time adaptive MT capabilities of Mistral 7B, enabling it to adapt translations to the required domain at inference time. The results, particularly for Spanish-to-English MT, showcase the efficacy of the fine-tuned model, demonstrating quality improvements in both zero-shot and one-shot translation scenarios, surpassing Mistral 7B's baseline performance. Notably, the fine-tuned Mistral outperforms ChatGPT "gpt-3.5-turbo" in zero-shot translation while achieving comparable one-shot translation quality. Moreover, the zero-shot translation of the fine-tuned Mistral matches NLLB 3.3B's performance, and its one-shot translation quality surpasses that of NLLB 3.3B. These findings emphasise the significance of fine-tuning efficient LLMs like Mistral 7B to yield high-quality zero-shot translations comparable to task-oriented models like NLLB 3.3B. Additionally, the adaptive gains achieved in one-shot translation are comparable to those of commercial LLMs such as ChatGPT. Our experiments demonstrate that, with a relatively small dataset of 20,000 segments that incorporate a mix of zero-shot and one-shot prompts, fine-tuning significantly enhances Mistral's in-context learning ability, especially for real-time adaptive MT.

Title: MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models. (arXiv:2312.12806v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.12806
• Code URL: null
• Copy Paste: [[2312.12806]] MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models(http://arxiv.org/abs/2312.12806)
• Summary:

  The emergence of various medical large language models (LLMs) in the medical domain has highlighted the need for unified evaluation standards, as manual evaluation of LLMs proves to be time-consuming and labor-intensive. To address this issue, we introduce MedBench, a comprehensive benchmark for the Chinese medical domain, comprising 40,041 questions sourced from authentic examination exercises and medical reports of diverse branches of medicine. In particular, this benchmark is composed of four key components: the Chinese Medical Licensing Examination, the Resident Standardization Training Examination, the Doctor In-Charge Qualification Examination, and real-world clinic cases encompassing examinations, diagnoses, and treatments. MedBench replicates the educational progression and clinical practice experiences of doctors in Mainland China, thereby establishing itself as a credible benchmark for assessing the mastery of knowledge and reasoning abilities in medical language learning models. We perform extensive experiments and conduct an in-depth analysis from diverse perspectives, which culminate in the following findings: (1) Chinese medical LLMs underperform on this benchmark, highlighting the need for significant advances in clinical knowledge and diagnostic precision. (2) Several general-domain LLMs surprisingly possess considerable medical knowledge. These findings elucidate both the capabilities and limitations of LLMs within the context of MedBench, with the ultimate goal of aiding the medical research community.

Title: Turning Dust into Gold: Distilling Complex Reasoning Capabilities from LLMs by Leveraging Negative Data. (arXiv:2312.12832v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.12832
• Code URL: null
• Copy Paste: [[2312.12832]] Turning Dust into Gold: Distilling Complex Reasoning Capabilities from LLMs by Leveraging Negative Data(http://arxiv.org/abs/2312.12832)
• Summary:

  Large Language Models (LLMs) have performed well on various reasoning tasks, but their inaccessibility and numerous parameters hinder wide application in practice. One promising way is distilling the reasoning ability from LLMs to small models by the generated chain-of-thought reasoning paths. In some cases, however, LLMs may produce incorrect reasoning chains, especially when facing complex mathematical problems. Previous studies only transfer knowledge from positive samples and drop the synthesized data with wrong answers. In this work, we illustrate the merit of negative data and propose a model specialization framework to distill LLMs with negative samples besides positive ones. The framework consists of three progressive steps, covering from training to inference stages, to absorb knowledge from negative data. We conduct extensive experiments across arithmetic reasoning tasks to demonstrate the role of negative data in distillation from LLM.

Title: Language Resources for Dutch Large Language Modelling. (arXiv:2312.12852v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.12852
• Code URL: https://github.com/bramvanroy/dutch-instruction-datasets
• Copy Paste: [[2312.12852]] Language Resources for Dutch Large Language Modelling(http://arxiv.org/abs/2312.12852)
• Summary:

  Despite the rapid expansion of types of large language models, there remains a notable gap in models specifically designed for the Dutch language. This gap is not only a shortage in terms of pretrained Dutch models but also in terms of data, and benchmarks and leaderboards. This work provides a small step to improve the situation. First, we introduce two fine-tuned variants of the Llama 2 13B model. We first fine-tuned Llama 2 using Dutch-specific web-crawled data and subsequently refined this model further on multiple synthetic instruction and chat datasets. These datasets as well as the model weights are made available. In addition, we provide a leaderboard to keep track of the performance of (Dutch) models on a number of generation tasks, and we include results of a number of state-of-the-art models, including our own. Finally we provide a critical conclusion on what we believe is needed to push forward Dutch language models and the whole eco-system around the models.

Title: CORECODE: A Common Sense Annotated Dialogue Dataset with Benchmark Tasks for Chinese Large Language Models. (arXiv:2312.12853v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.12853
• Code URL: null
• Copy Paste: [[2312.12853]] CORECODE: A Common Sense Annotated Dialogue Dataset with Benchmark Tasks for Chinese Large Language Models(http://arxiv.org/abs/2312.12853)
• Summary:

  As an indispensable ingredient of intelligence, commonsense reasoning is crucial for large language models (LLMs) in real-world scenarios. In this paper, we propose CORECODE, a dataset that contains abundant commonsense knowledge manually annotated on dyadic dialogues, to evaluate the commonsense reasoning and commonsense conflict detection capabilities of Chinese LLMs. We categorize commonsense knowledge in everyday conversations into three dimensions: entity, event, and social interaction. For easy and consistent annotation, we standardize the form of commonsense knowledge annotation in open-domain dialogues as "domain: slot = value". A total of 9 domains and 37 slots are defined to capture diverse commonsense knowledge. With these pre-defined domains and slots, we collect 76,787 commonsense knowledge annotations from 19,700 dialogues through crowdsourcing. To evaluate and enhance the commonsense reasoning capability for LLMs on the curated dataset, we establish a series of dialogue-level reasoning and detection tasks, including commonsense knowledge filling, commonsense knowledge generation, commonsense conflict phrase detection, domain identification, slot identification, and event causal inference. A wide variety of existing open-source Chinese LLMs are evaluated with these tasks on our dataset. Experimental results demonstrate that these models are not competent to predict CORECODE's plentiful reasoning content, and even ChatGPT could only achieve 0.275 and 0.084 accuracy on the domain identification and slot identification tasks under the zero-shot setting. We release the data and codes of CORECODE at https://github.com/danshi777/CORECODE to promote commonsense reasoning evaluation and study of LLMs in the context of daily conversations.

Title: Machine Mindset: An MBTI Exploration of Large Language Models. (arXiv:2312.12999v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.12999
• Code URL: https://github.com/pku-yuangroup/machine-mindset
• Copy Paste: [[2312.12999]] Machine Mindset: An MBTI Exploration of Large Language Models(http://arxiv.org/abs/2312.12999)
• Summary:

  We present a novel approach for integrating Myers-Briggs Type Indicator (MBTI) personality traits into large language models (LLMs), addressing the challenges of personality consistency in personalized AI. Our method, "Machine Mindset," involves a two-phase fine-tuning and Direct Preference Optimization (DPO) to embed MBTI traits into LLMs. This approach ensures that models internalize these traits, offering a stable and consistent personality profile. We demonstrate the effectiveness of our models across various domains, showing alignment between model performance and their respective MBTI traits. The paper highlights significant contributions in the development of personality datasets and a new training methodology for personality integration in LLMs, enhancing the potential for personalized AI applications. We also open-sourced our model and part of the data at \url{https://github.com/PKU-YuanGroup/Machine-Mindset}.

Title: Retrieval-augmented Multilingual Knowledge Editing. (arXiv:2312.13040v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.13040
• Code URL: https://github.com/vicky-wil/remake
• Copy Paste: [[2312.13040]] Retrieval-augmented Multilingual Knowledge Editing(http://arxiv.org/abs/2312.13040)
• Summary:

  Knowledge represented in Large Language Models (LLMs) is quite often incorrect and can also become obsolete over time. Updating knowledge via fine-tuning is computationally resource-hungry and not reliable, and so knowledge editing (KE) has developed as an effective and economical alternative to inject new knowledge or to fix factual errors in LLMs. Although there has been considerable interest in this area, current KE research exclusively focuses on the monolingual setting, typically in English. However, what happens if the new knowledge is supplied in one language, but we would like to query the LLM in a different language? To address the problem of multilingual knowledge editing, we propose Retrieval-augmented Multilingual Knowledge Editor (ReMaKE) to update new knowledge in LLMs. ReMaKE can perform model-agnostic knowledge editing in multilingual settings. ReMaKE concatenates the new knowledge retrieved from a multilingual knowledge base with prompts. Our experimental results show that ReMaKE outperforms baseline knowledge editing methods by a significant margin and is the first KE method to work in a multilingual setting. We provide our multilingual knowledge editing dataset (MzsRE) in 12 languages, which along with code, and additional project information is available at https://github.com/Vicky-Wil/ReMaKE.

Title: Contextual Code Switching for Machine Translation using Language Models. (arXiv:2312.13179v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.13179
• Code URL: null
• Copy Paste: [[2312.13179]] Contextual Code Switching for Machine Translation using Language Models(http://arxiv.org/abs/2312.13179)
• Summary:

  Large language models (LLMs) have exerted a considerable impact on diverse language-related tasks in recent years. Their demonstrated state-of-the-art performance is achieved through methodologies such as zero-shot or few-shot prompting. These models undergo training on extensive datasets that encompass segments of the Internet and subsequently undergo fine-tuning tailored to specific tasks. Notably, they exhibit proficiency in tasks such as translation, summarization, question answering, and creative writing, even in the absence of explicit training for those particular tasks. While they have shown substantial improvement in the multilingual tasks their performance in the code switching, especially for machine translation remains relatively uncharted. In this paper, we present an extensive study on the code switching task specifically for the machine translation task comparing multiple LLMs. Our results indicate that despite the LLMs having promising results in the certain tasks, the models with relatively lesser complexity outperform the multilingual large language models in the machine translation task. We posit that the efficacy of multilingual large language models in contextual code switching is constrained by their training methodologies. In contrast, relatively smaller models, when trained and fine-tuned on bespoke datasets, may yield superior results in comparison to the majority of multilingual models.

Title: LlaMaVAE: Guiding Large Language Model Generation via Continuous Latent Sentence Spaces. (arXiv:2312.13208v1 [cs.CL])

• Paper URL: http://arxiv.org/abs/2312.13208
• Code URL: null
• Copy Paste: [[2312.13208]] LlaMaVAE: Guiding Large Language Model Generation via Continuous Latent Sentence Spaces(http://arxiv.org/abs/2312.13208)
• Summary:

  Deep generative neural networks, such as Variational AutoEncoders (VAEs), offer an opportunity to better understand and control language models from the perspective of sentence-level latent spaces. To combine the controllability of VAE latent spaces with the state-of-the-art performance of recent large language models (LLMs), we present in this work LlaMaVAE, which combines expressive encoder and decoder models (sentenceT5 and LlaMA) with a VAE architecture, aiming to provide better text generation control to LLMs. In addition, to conditionally guide the VAE generation, we investigate a new approach based on flow-based invertible neural networks (INNs) named Invertible CVAE. Experimental results reveal that LlaMaVAE can outperform the previous state-of-the-art VAE language model, Optimus, across various tasks, including language modelling, semantic textual similarity and definition modelling. Qualitative analysis on interpolation and traversal experiments also indicates an increased degree of semantic clustering and geometric consistency, which enables better generation control.

Title: PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU. (arXiv:2312.12456v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12456
• Code URL: null
• Copy Paste: [[2312.12456]] PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU(http://arxiv.org/abs/2312.12456)
• Summary:

  This paper introduces PowerInfer, a high-speed Large Language Model (LLM) inference engine on a personal computer (PC) equipped with a single consumer-grade GPU. The key underlying the design of PowerInfer is exploiting the high locality inherent in LLM inference, characterized by a power-law distribution in neuron activation. This distribution indicates that a small subset of neurons, termed hot neurons, are consistently activated across inputs, while the majority, cold neurons, vary based on specific inputs. PowerInfer exploits such an insight to design a GPU-CPU hybrid inference engine: hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons are computed on the CPU, thus significantly reducing GPU memory demands and CPU-GPU data transfers. PowerInfer further integrates adaptive predictors and neuron-aware sparse operators, optimizing the efficiency of neuron activation and computational sparsity. Evaluation shows that PowerInfer attains an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs (including OPT-175B) on a single NVIDIA RTX 4090 GPU, only 18% lower than that achieved by a top-tier server-grade A100 GPU. This significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy.

Title: A Performance Evaluation of a Quantized Large Language Model on Various Smartphones. (arXiv:2312.12472v1 [cs.LG])

• Paper URL: http://arxiv.org/abs/2312.12472
• Code URL: null
• Copy Paste: [[2312.12472]] A Performance Evaluation of a Quantized Large Language Model on Various Smartphones(http://arxiv.org/abs/2312.12472)
• Summary:

  This paper explores the feasibility and performance of on-device large language model (LLM) inference on various Apple iPhone models. Amidst the rapid evolution of generative AI, on-device LLMs offer solutions to privacy, security, and connectivity challenges inherent in cloud-based models. Leveraging existing literature on running multi-billion parameter LLMs on resource-limited devices, our study examines the thermal effects and interaction speeds of a high-performing LLM across different smartphone generations. We present real-world performance results, providing insights into on-device inference capabilities.

segmentation

Title: Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentation. (arXiv:2312.12470v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12470
• Code URL: https://github.com/lsan2401/rmsin
• Copy Paste: [[2312.12470]] Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentation(http://arxiv.org/abs/2312.12470)
• Summary:

  Referring Remote Sensing Image Segmentation (RRSIS) is a new challenge that combines computer vision and natural language processing, delineating specific regions in aerial images as described by textual queries. Traditional Referring Image Segmentation (RIS) approaches have been impeded by the complex spatial scales and orientations found in aerial imagery, leading to suboptimal segmentation results. To address these challenges, we introduce the Rotated Multi-Scale Interaction Network (RMSIN), an innovative approach designed for the unique demands of RRSIS. RMSIN incorporates an Intra-scale Interaction Module (IIM) to effectively address the fine-grained detail required at multiple scales and a Cross-scale Interaction Module (CIM) for integrating these details coherently across the network. Furthermore, RMSIN employs an Adaptive Rotated Convolution (ARC) to account for the diverse orientations of objects, a novel contribution that significantly enhances segmentation accuracy. To assess the efficacy of RMSIN, we have curated an expansive dataset comprising 17,402 image-caption-mask triplets, which is unparalleled in terms of scale and variety. This dataset not only presents the model with a wide range of spatial and rotational scenarios but also establishes a stringent benchmark for the RRSIS task, ensuring a rigorous evaluation of performance. Our experimental evaluations demonstrate the exceptional performance of RMSIN, surpassing existing state-of-the-art models by a significant margin. All datasets and code are made available at https://github.com/Lsan2401/RMSIN.

Title: DDOS: The Drone Depth and Obstacle Segmentation Dataset. (arXiv:2312.12494v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12494
• Code URL: null
• Copy Paste: [[2312.12494]] DDOS: The Drone Depth and Obstacle Segmentation Dataset(http://arxiv.org/abs/2312.12494)
• Summary:

  Accurate depth and semantic segmentation are crucial for various computer vision tasks. However, the scarcity of annotated real-world aerial datasets poses a significant challenge for training and evaluating robust models. Additionally, the detection and segmentation of thin objects, such as wires, cables, and fences, present a critical concern for ensuring the safe operation of drones. To address these limitations, we present a novel synthetic dataset specifically designed for depth and semantic segmentation tasks in aerial views. Leveraging photo-realistic rendering techniques, our dataset provides a valuable resource for training models using a synthetic-supervision training scheme while introducing new drone-specific metrics for depth accuracy.

Title: Segment Anything Model Meets Image Harmonization. (arXiv:2312.12729v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12729
• Code URL: null
• Copy Paste: [[2312.12729]] Segment Anything Model Meets Image Harmonization(http://arxiv.org/abs/2312.12729)
• Summary:

  Image harmonization is a crucial technique in image composition that aims to seamlessly match the background by adjusting the foreground of composite images. Current methods adopt either global-level or pixel-level feature matching. Global-level feature matching ignores the proximity prior, treating foreground and background as separate entities. On the other hand, pixel-level feature matching loses contextual information. Therefore, it is necessary to use the information from semantic maps that describe different objects to guide harmonization. In this paper, we propose Semantic-guided Region-aware Instance Normalization (SRIN) that can utilize the semantic segmentation maps output by a pre-trained Segment Anything Model (SAM) to guide the visual consistency learning of foreground and background features. Abundant experiments demonstrate the superiority of our method for image harmonization over state-of-the-art methods.

Title: MetaSegNet: Metadata-collaborative Vision-Language Representation Learning for Semantic Segmentation of Remote Sensing Images. (arXiv:2312.12735v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12735
• Code URL: null
• Copy Paste: [[2312.12735]] MetaSegNet: Metadata-collaborative Vision-Language Representation Learning for Semantic Segmentation of Remote Sensing Images(http://arxiv.org/abs/2312.12735)
• Summary:

  Semantic segmentation of remote sensing images plays a vital role in a wide range of Earth Observation (EO) applications, such as land use land cover mapping, environment monitoring, and sustainable development. Driven by rapid developments in Artificial Intelligence (AI), deep learning (DL) has emerged as the mainstream tool for semantic segmentation and achieved many breakthroughs in the field of remote sensing. However, the existing DL-based methods mainly focus on unimodal visual data while ignoring the rich multimodal information involved in the real world, usually demonstrating weak reliability and generlization. Inspired by the success of Vision Transformers and large language models, we propose a novel metadata-collaborative multimodal segmentation network (MetaSegNet) that applies vision-language representation learning for semantic segmentation of remote sensing images. Unlike the common model structure that only uses unimodal visual data, we extract the key characteristic (i.e. the climate zone) from freely available remote sensing image metadata and transfer it into knowledge-based text prompts via the generic ChatGPT. Then, we construct an image encoder, a text encoder and a crossmodal attention fusion subnetwork to extract the image and text feature and apply image-text interaction. Benefiting from such a design, the proposed MetaSegNet demonstrates superior generalization and achieves competitive accuracy with state-of-the-art semantic segmentation methods on the large-scale OpenEarthMap dataset (68.6% mIoU) and Potsdam dataset (93.3% mean F1 score) as well as LoveDA dataset (52.2% mIoU).

Title: PointeNet: A Lightweight Framework for Effective and Efficient Point Cloud Analysis. (arXiv:2312.12743v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12743
• Code URL: null
• Copy Paste: [[2312.12743]] PointeNet: A Lightweight Framework for Effective and Efficient Point Cloud Analysis(http://arxiv.org/abs/2312.12743)
• Summary:

  Current methodologies in point cloud analysis predominantly explore 3D geometries, often achieved through the introduction of intricate learnable geometric extractors in the encoder or by deepening networks with repeated blocks. However, these approaches inevitably lead to a significant number of learnable parameters, resulting in substantial computational costs and imposing memory burdens on CPU/GPU. Additionally, the existing strategies are primarily tailored for object-level point cloud classification and segmentation tasks, with limited extensions to crucial scene-level applications, such as autonomous driving. In response to these limitations, we introduce PointeNet, an efficient network designed specifically for point cloud analysis. PointeNet distinguishes itself with its lightweight architecture, low training cost, and plug-and-play capability, effectively capturing representative features. The network consists of a Multivariate Geometric Encoding (MGE) module and an optional Distance-aware Semantic Enhancement (DSE) module. The MGE module employs operations of sampling, grouping, and multivariate geometric aggregation to lightweightly capture and adaptively aggregate multivariate geometric features, providing a comprehensive depiction of 3D geometries. The DSE module, designed for real-world autonomous driving scenarios, enhances the semantic perception of point clouds, particularly for distant points. Our method demonstrates flexibility by seamlessly integrating with a classification/segmentation head or embedding into off-the-shelf 3D object detection networks, achieving notable performance improvements at a minimal cost. Extensive experiments on object-level datasets, including ModelNet40, ScanObjectNN, ShapeNetPart, and the scene-level dataset KITTI, demonstrate the superior performance of PointeNet over state-of-the-art methods in point cloud analysis.

Title: Spectral Prompt Tuning:Unveiling Unseen Classes for Zero-Shot Semantic Segmentation. (arXiv:2312.12754v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12754
• Code URL: https://github.com/clearxu/spt
• Copy Paste: [[2312.12754]] Spectral Prompt Tuning:Unveiling Unseen Classes for Zero-Shot Semantic Segmentation(http://arxiv.org/abs/2312.12754)
• Summary:

  Recently, CLIP has found practical utility in the domain of pixel-level zero-shot segmentation tasks. The present landscape features two-stage methodologies beset by issues such as intricate pipelines and elevated computational costs. While current one-stage approaches alleviate these concerns and incorporate Visual Prompt Training (VPT) to uphold CLIP's generalization capacity, they still fall short in fully harnessing CLIP's potential for pixel-level unseen class demarcation and precise pixel predictions. To further stimulate CLIP's zero-shot dense prediction capability, we propose SPT-SEG, a one-stage approach that improves CLIP's adaptability from image to pixel. Specifically, we initially introduce Spectral Prompt Tuning (SPT), incorporating spectral prompts into the CLIP visual encoder's shallow layers to capture structural intricacies of images, thereby enhancing comprehension of unseen classes. Subsequently, we introduce the Spectral Guided Decoder (SGD), utilizing both high and low-frequency information to steer the network's spatial focus towards more prominent classification features, enabling precise pixel-level prediction outcomes. Through extensive experiments on two public datasets, we demonstrate the superiority of our method over state-of-the-art approaches, performing well across all classes and particularly excelling in handling unseen classes. Code is available at:https://github.com/clearxu/SPT.

Title: Segmenting Messy Text: Detecting Boundaries in Text Derived from Historical Newspaper Images. (arXiv:2312.12773v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12773
• Code URL: null
• Copy Paste: [[2312.12773]] Segmenting Messy Text: Detecting Boundaries in Text Derived from Historical Newspaper Images(http://arxiv.org/abs/2312.12773)
• Summary:

  Text segmentation, the task of dividing a document into sections, is often a prerequisite for performing additional natural language processing tasks. Existing text segmentation methods have typically been developed and tested using clean, narrative-style text with segments containing distinct topics. Here we consider a challenging text segmentation task: dividing newspaper marriage announcement lists into units of one announcement each. In many cases the information is not structured into sentences, and adjacent segments are not topically distinct from each other. In addition, the text of the announcements, which is derived from images of historical newspapers via optical character recognition, contains many typographical errors. As a result, these announcements are not amenable to segmentation with existing techniques. We present a novel deep learning-based model for segmenting such text and show that it significantly outperforms an existing state-of-the-art method on our task.

Title: OCTOPUS: Open-vocabulary Content Tracking and Object Placement Using Semantic Understanding in Mixed Reality. (arXiv:2312.12815v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12815
• Code URL: null
• Copy Paste: [[2312.12815]] OCTOPUS: Open-vocabulary Content Tracking and Object Placement Using Semantic Understanding in Mixed Reality(http://arxiv.org/abs/2312.12815)
• Summary:

  One key challenge in augmented reality is the placement of virtual content in natural locations. Existing automated techniques are only able to work with a closed-vocabulary, fixed set of objects. In this paper, we introduce a new open-vocabulary method for object placement. Our eight-stage pipeline leverages recent advances in segmentation models, vision-language models, and LLMs to place any virtual object in any AR camera frame or scene. In a preliminary user study, we show that our method performs at least as well as human experts 57% of the time.

Title: TagCLIP: A Local-to-Global Framework to Enhance Open-Vocabulary Multi-Label Classification of CLIP Without Training. (arXiv:2312.12828v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.12828
• Code URL: https://github.com/linyq2117/tagclip
• Copy Paste: [[2312.12828]] TagCLIP: A Local-to-Global Framework to Enhance Open-Vocabulary Multi-Label Classification of CLIP Without Training(http://arxiv.org/abs/2312.12828)
• Summary:

  Contrastive Language-Image Pre-training (CLIP) has demonstrated impressive capabilities in open-vocabulary classification. The class token in the image encoder is trained to capture the global features to distinguish different text descriptions supervised by contrastive loss, making it highly effective for single-label classification. However, it shows poor performance on multi-label datasets because the global feature tends to be dominated by the most prominent class and the contrastive nature of softmax operation aggravates it. In this study, we observe that the multi-label classification results heavily rely on discriminative local features but are overlooked by CLIP. As a result, we dissect the preservation of patch-wise spatial information in CLIP and proposed a local-to-global framework to obtain image tags. It comprises three steps: (1) patch-level classification to obtain coarse scores; (2) dual-masking attention refinement (DMAR) module to refine the coarse scores; (3) class-wise reidentification (CWR) module to remedy predictions from a global perspective. This framework is solely based on frozen CLIP and significantly enhances its multi-label classification performance on various benchmarks without dataset-specific training. Besides, to comprehensively assess the quality and practicality of generated tags, we extend their application to the downstream task, i.e., weakly supervised semantic segmentation (WSSS) with generated tags as image-level pseudo labels. Experiments demonstrate that this classify-then-segment paradigm dramatically outperforms other annotation-free segmentation methods and validates the effectiveness of generated tags. Our code is available at https://github.com/linyq2117/TagCLIP.

Title: BEVSeg2TP: Surround View Camera Bird's-Eye-View Based Joint Vehicle Segmentation and Ego Vehicle Trajectory Prediction. (arXiv:2312.13081v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.13081
• Code URL: null
• Copy Paste: [[2312.13081]] BEVSeg2TP: Surround View Camera Bird's-Eye-View Based Joint Vehicle Segmentation and Ego Vehicle Trajectory Prediction(http://arxiv.org/abs/2312.13081)
• Summary:

  Trajectory prediction is, naturally, a key task for vehicle autonomy. While the number of traffic rules is limited, the combinations and uncertainties associated with each agent's behaviour in real-world scenarios are nearly impossible to encode. Consequently, there is a growing interest in learning-based trajectory prediction. The proposed method in this paper predicts trajectories by considering perception and trajectory prediction as a unified system. In considering them as unified tasks, we show that there is the potential to improve the performance of perception. To achieve these goals, we present BEVSeg2TP - a surround-view camera bird's-eye-view-based joint vehicle segmentation and ego vehicle trajectory prediction system for autonomous vehicles. The proposed system uses a network trained on multiple camera views. The images are transformed using several deep learning techniques to perform semantic segmentation of objects, including other vehicles, in the scene. The segmentation outputs are fused across the camera views to obtain a comprehensive representation of the surrounding vehicles from the bird's-eye-view perspective. The system further predicts the future trajectory of the ego vehicle using a spatiotemporal probabilistic network (STPN) to optimize trajectory prediction. This network leverages information from encoder-decoder transformers and joint vehicle segmentation.

Title: VSR-Net: Vessel-like Structure Rehabilitation Network with Graph Clustering. (arXiv:2312.13116v1 [cs.CV])

• Paper URL: http://arxiv.org/abs/2312.13116
• Code URL: null
• Copy Paste: [[2312.13116]] VSR-Net: Vessel-like Structure Rehabilitation Network with Graph Clustering(http://arxiv.org/abs/2312.13116)
• Summary:

  The morphologies of vessel-like structures, such as blood vessels and nerve fibres, play significant roles in disease diagnosis, e.g., Parkinson's disease. Deep network-based refinement segmentation methods have recently achieved promising vessel-like structure segmentation results. There are still two challenges: (1) existing methods have limitations in rehabilitating subsection ruptures in segmented vessel-like structures; (2) they are often overconfident in predicted segmentation results. To tackle these two challenges, this paper attempts to leverage the potential of spatial interconnection relationships among subsection ruptures from the structure rehabilitation perspective. Based on this, we propose a novel Vessel-like Structure Rehabilitation Network (VSR-Net) to rehabilitate subsection ruptures and improve the model calibration based on coarse vessel-like structure segmentation results. VSR-Net first constructs subsection rupture clusters with Curvilinear Clustering Module (CCM). Then, the well-designed Curvilinear Merging Module (CMM) is applied to rehabilitate the subsection ruptures to obtain the refined vessel-like structures. Extensive experiments on five 2D/3D medical image datasets show that VSR-Net significantly outperforms state-of-the-art (SOTA) refinement segmentation methods with lower calibration error. Additionally, we provide quantitative analysis to explain the morphological difference between the rehabilitation results of VSR-Net and ground truth (GT), which is smaller than SOTA methods and GT, demonstrating that our method better rehabilitates vessel-like structures by restoring subsection ruptures.