2024-02-05

Title: Security and Privacy Challenges of Large Language Models: A Survey

• Authors: Badhan Chandra Das, M. Hadi Amini, Yanzhao Wu
• Subjects: cs.CL, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.00888
• Pdf URL: https://arxiv.org/pdf/2402.00888
• Copy Paste: [[2402.00888]] Security and Privacy Challenges of Large Language Models: A Survey(https://arxiv.org/abs/2402.00888)
• Keywords: security, privacy, defense, attack, large language model
• Abstract: Large Language Models (LLMs) have demonstrated extraordinary capabilities and contributed to multiple fields, such as generating and summarizing text, language translation, and question-answering. Nowadays, LLM is becoming a very popular tool in computerized language processing tasks, with the capability to analyze complicated linguistic patterns and provide relevant and appropriate responses depending on the context. While offering significant advantages, these models are also vulnerable to security and privacy attacks, such as jailbreaking attacks, data poisoning attacks, and Personally Identifiable Information (PII) leakage attacks. This survey provides a thorough review of the security and privacy challenges of LLMs for both training data and users, along with the application-based risks in various domains, such as transportation, education, and healthcare. We assess the extent of LLM vulnerabilities, investigate emerging security and privacy attacks for LLMs, and review the potential defense mechanisms. Additionally, the survey outlines existing research gaps in this domain and highlights future research directions.

Title: Utilizing Large Language Models to Translate RFC Protocol Specifications to CPSA Definitions

• Authors: Martin Duclos, Ivan A. Fernandez, Kaneesha Moore, Sudip Mittal, Edward Zieglar
• Subjects: cs.CR, cs.NI, cs.SE
• Abstract URL: https://arxiv.org/abs/2402.00890
• Pdf URL: https://arxiv.org/pdf/2402.00890
• Copy Paste: [[2402.00890]] Utilizing Large Language Models to Translate RFC Protocol Specifications to CPSA Definitions(https://arxiv.org/abs/2402.00890)
• Keywords: security, large language model
• Abstract: This paper proposes the use of Large Language Models (LLMs) for translating Request for Comments (RFC) protocol specifications into a format compatible with the Cryptographic Protocol Shapes Analyzer (CPSA). This novel approach aims to reduce the complexities and efforts involved in protocol analysis, by offering an automated method for translating protocol specifications into structured models suitable for CPSA. In this paper we discuss the implementation of an RFC Protocol Translator, its impact on enhancing the accessibility of formal methods analysis, and its potential for improving the security of internet protocols.

Title: Large Language Models in Cybersecurity: State-of-the-Art

• Authors: Farzad Nourmohammadzadeh Motlagh, Mehrdad Hajizadeh, Mehryar Majd, Pejman Najafi, Feng Cheng, Christoph Meinel
• Subjects: cs.CR, cs.AI, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00891
• Pdf URL: https://arxiv.org/pdf/2402.00891
• Copy Paste: [[2402.00891]] Large Language Models in Cybersecurity: State-of-the-Art(https://arxiv.org/abs/2402.00891)
• Keywords: security, large language model
• Abstract: The rise of Large Language Models (LLMs) has revolutionized our comprehension of intelligence bringing us closer to Artificial Intelligence. Since their introduction, researchers have actively explored the applications of LLMs across diverse fields, significantly elevating capabilities. Cybersecurity, traditionally resistant to data-driven solutions and slow to embrace machine learning, stands out as a domain. This study examines the existing literature, providing a thorough characterization of both defensive and adversarial applications of LLMs within the realm of cybersecurity. Our review not only surveys and categorizes the current landscape but also identifies critical research gaps. By evaluating both offensive and defensive applications, we aim to provide a holistic understanding of the potential risks and opportunities associated with LLM-driven cybersecurity.

Title: MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts

• Authors: Zhitian Xie, Yinger Zhang, Chenyi Zhuang, Qitao Shi, Zhining Liu, Jinjie Gu, Guannan Zhang
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00893
• Pdf URL: https://arxiv.org/pdf/2402.00893
• Copy Paste: [[2402.00893]] MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts(https://arxiv.org/abs/2402.00893)
• Keywords: robust
• Abstract: The application of mixture-of-experts (MoE) is gaining popularity due to its ability to improve model's performance. In an MoE structure, the gate layer plays a significant role in distinguishing and routing input features to different experts. This enables each expert to specialize in processing their corresponding sub-tasks. However, the gate's routing mechanism also gives rise to narrow vision: the individual MoE's expert fails to use more samples in learning the allocated sub-task, which in turn limits the MoE to further improve its generalization ability. To effectively address this, we propose a method called Mixture-of-Distilled-Expert (MoDE), which applies moderate mutual distillation among experts to enable each expert to pick up more features learned by other experts and gain more accurate perceptions on their original allocated sub-tasks. We conduct plenty experiments including tabular, NLP and CV datasets, which shows MoDE's effectiveness, universality and robustness. Furthermore, we develop a parallel study through innovatively constructing "expert probing", to experimentally prove why MoDE works: moderate distilling knowledge can improve each individual expert's test performances on their assigned tasks, leading to MoE's overall performance improvement.

Title: Privacy and Security Implications of Cloud-Based AI Services : A Survey

• Authors: Alka Luqman, Riya Mahesh, Anupam Chattopadhyay
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00896
• Pdf URL: https://arxiv.org/pdf/2402.00896
• Copy Paste: [[2402.00896]] Privacy and Security Implications of Cloud-Based AI Services : A Survey(https://arxiv.org/abs/2402.00896)
• Keywords: secure, security, privacy, robust
• Abstract: This paper details the privacy and security landscape in today's cloud ecosystem and identifies that there is a gap in addressing the risks introduced by machine learning models. As machine learning algorithms continue to evolve and find applications across diverse domains, the need to categorize and quantify privacy and security risks becomes increasingly critical. With the emerging trend of AI-as-a-Service (AIaaS), machine learned AI models (or ML models) are deployed on the cloud by model providers and used by model consumers. We first survey the AIaaS landscape to document the various kinds of liabilities that ML models, especially Deep Neural Networks pose and then introduce a taxonomy to bridge this gap by holistically examining the risks that creators and consumers of ML models are exposed to and their known defences till date. Such a structured approach will be beneficial for ML model providers to create robust solutions. Likewise, ML model consumers will find it valuable to evaluate such solutions and understand the implications of their engagement with such services. The proposed taxonomies provide a foundational basis for solutions in private, secure and robust ML, paving the way for more transparent and resilient AI systems.

Title: An Early Categorization of Prompt Injection Attacks on Large Language Models

• Authors: Sippo Rossi, Alisia Marianne Michel, Raghava Rao Mukkamala, Jason Bennett Thatcher
• Subjects: cs.CR, cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00898
• Pdf URL: https://arxiv.org/pdf/2402.00898
• Copy Paste: [[2402.00898]] An Early Categorization of Prompt Injection Attacks on Large Language Models(https://arxiv.org/abs/2402.00898)
• Keywords: attack, large language model
• Abstract: Large language models and AI chatbots have been at the forefront of democratizing artificial intelligence. However, the releases of ChatGPT and other similar tools have been followed by growing concerns regarding the difficulty of controlling large language models and their outputs. Currently, we are witnessing a cat-and-mouse game where users attempt to misuse the models with a novel attack called prompt injections. In contrast, the developers attempt to discover the vulnerabilities and block the attacks simultaneously. In this paper, we provide an overview of these emergent threats and present a categorization of prompt injections, which can guide future research on prompt injections and act as a checklist of vulnerabilities in the development of LLM interfaces. Moreover, based on previous literature and our own empirical research, we discuss the implications of prompt injections to LLM end users, developers, and researchers.

Title: BrainLeaks: On the Privacy-Preserving Properties of Neuromorphic Architectures against Model Inversion Attacks

• Authors: Hamed Poursiami, Ihsen Alouani, Maryam Parsa
• Subjects: cs.CR, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2402.00906
• Pdf URL: https://arxiv.org/pdf/2402.00906
• Copy Paste: [[2402.00906]] BrainLeaks: On the Privacy-Preserving Properties of Neuromorphic Architectures against Model Inversion Attacks(https://arxiv.org/abs/2402.00906)
• Keywords: security, privacy, attack
• Abstract: With the mainstream integration of machine learning into security-sensitive domains such as healthcare and finance, concerns about data privacy have intensified. Conventional artificial neural networks (ANNs) have been found vulnerable to several attacks that can leak sensitive data. Particularly, model inversion (MI) attacks enable the reconstruction of data samples that have been used to train the model. Neuromorphic architectures have emerged as a paradigm shift in neural computing, enabling asynchronous and energy-efficient computation. However, little to no existing work has investigated the privacy of neuromorphic architectures against model inversion. Our study is motivated by the intuition that the non-differentiable aspect of spiking neural networks (SNNs) might result in inherent privacy-preserving properties, especially against gradient-based attacks. To investigate this hypothesis, we propose a thorough exploration of SNNs' privacy-preserving capabilities. Specifically, we develop novel inversion attack strategies that are comprehensively designed to target SNNs, offering a comparative analysis with their conventional ANN counterparts. Our experiments, conducted on diverse event-based and static datasets, demonstrate the effectiveness of the proposed attack strategies and therefore questions the assumption of inherent privacy-preserving in neuromorphic architectures.

Title: Addressing Bias Through Ensemble Learning and Regularized Fine-Tuning

• Authors: Ahmed Radwan, Layan Zaafarani, Jetana Abudawood, Faisal AlZahrani, Fares Fourat
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00910
• Pdf URL: https://arxiv.org/pdf/2402.00910
• Copy Paste: [[2402.00910]] Addressing Bias Through Ensemble Learning and Regularized Fine-Tuning(https://arxiv.org/abs/2402.00910)
• Keywords: fair
• Abstract: Addressing biases in AI models is crucial for ensuring fair and accurate predictions. However, obtaining large, unbiased datasets for training can be challenging. This paper proposes a comprehensive approach using multiple methods to remove bias in AI models, with only a small dataset and a potentially biased pretrained model. We train multiple models with the counter-bias of the pre-trained model through data splitting, local training, and regularized fine-tuning, gaining potentially counter-biased models. Then, we employ ensemble learning for all models to reach unbiased predictions. To further accelerate the inference time of our ensemble model, we conclude our solution with knowledge distillation that results in a single unbiased neural network. We demonstrate the effectiveness of our approach through experiments on the CIFAR10 and HAM10000 datasets, showcasing promising results. This work contributes to the ongoing effort to create more unbiased and reliable AI models, even with limited data availability.

Title: Can we Constrain Concept Bottleneck Models to Learn Semantically Meaningful Input Features?

• Authors: Jack Furby, Daniel Cunnington, Dave Braines, Alun Preece
• Subjects: cs.LG, cs.AI, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00912
• Pdf URL: https://arxiv.org/pdf/2402.00912
• Copy Paste: [[2402.00912]] Can we Constrain Concept Bottleneck Models to Learn Semantically Meaningful Input Features?(https://arxiv.org/abs/2402.00912)
• Keywords: interpretability
• Abstract: Concept Bottleneck Models (CBMs) are considered inherently interpretable because they first predict a set of human-defined concepts before using these concepts to predict the output of a downstream task. For inherent interpretability to be fully realised, and ensure trust in a model's output, we need to guarantee concepts are predicted based on semantically mapped input features. For example, one might expect the pixels representing a broken bone in an image to be used for the prediction of a fracture. However, current literature indicates this is not the case, as concept predictions are often mapped to irrelevant input features. We hypothesise that this occurs when concept annotations are inaccurate or how input features should relate to concepts is unclear. In general, the effect of dataset labelling on concept representations in CBMs remains an understudied area. Therefore, in this paper, we examine how CBMs learn concepts from datasets with fine-grained concept annotations. We demonstrate that CBMs can learn concept representations with semantic mapping to input features by removing problematic concept correlations, such as two concepts always appearing together. To support our evaluation, we introduce a new synthetic image dataset based on a playing cards domain, which we hope will serve as a benchmark for future CBM research. For validation, we provide empirical evidence on a real-world dataset of chest X-rays, to demonstrate semantically meaningful concepts can be learned in real-world applications.

Title: Institutional Platform for Secure Self-Service Large Language Model Exploration

• Authors: V. K. Cody Bumgardner, Mitchell A. Klusty, W. Vaiden Logan, Samuel E. Armstrong, Caylin Hickey, Jeff Talbert
• Subjects: cs.CR, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00913
• Pdf URL: https://arxiv.org/pdf/2402.00913
• Copy Paste: [[2402.00913]] Institutional Platform for Secure Self-Service Large Language Model Exploration(https://arxiv.org/abs/2402.00913)
• Keywords: secure, extraction, large language model
• Abstract: This paper introduces a user-friendly platform developed by the University of Kentucky Center for Applied AI, designed to make large, customized language models (LLMs) more accessible. By capitalizing on recent advancements in multi-LoRA inference, the system efficiently accommodates custom adapters for a diverse range of users and projects. The paper outlines the system's architecture and key features, encompassing dataset curation, model training, secure inference, and text-based feature extraction. We illustrate the establishment of a tenant-aware computational network using agent-based methods, securely utilizing islands of isolated resources as a unified system. The platform strives to deliver secure LLM services, emphasizing process and data isolation, end-to-end encryption, and role-based resource authentication. This contribution aligns with the overarching goal of enabling simplified access to cutting-edge AI models and technology in support of scientific discovery.

Title: MUSTAN: Multi-scale Temporal Context as Attention for Robust Video Foreground Segmentation

• Authors: Praveen Kumar Pokala, Jaya Sai Kiran Patibandla, Naveen Kumar Pandey, Balakrishna Reddy Pailla
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.00918
• Pdf URL: https://arxiv.org/pdf/2402.00918
• Copy Paste: [[2402.00918]] MUSTAN: Multi-scale Temporal Context as Attention for Robust Video Foreground Segmentation(https://arxiv.org/abs/2402.00918)
• Keywords: robust, segmentation
• Abstract: Video foreground segmentation (VFS) is an important computer vision task wherein one aims to segment the objects under motion from the background. Most of the current methods are image-based, i.e., rely only on spatial cues while ignoring motion cues. Therefore, they tend to overfit the training data and don't generalize well to out-of-domain (OOD) distribution. To solve the above problem, prior works exploited several cues such as optical flow, background subtraction mask, etc. However, having a video data with annotations like optical flow is a challenging task. In this paper, we utilize the temporal information and the spatial cues from the video data to improve OOD performance. However, the challenge lies in how we model the temporal information given the video data in an interpretable way creates a very noticeable difference. We therefore devise a strategy that integrates the temporal context of the video in the development of VFS. Our approach give rise to deep learning architectures, namely MUSTAN1 and MUSTAN2 and they are based on the idea of multi-scale temporal context as an attention, i.e., aids our models to learn better representations that are beneficial for VFS. Further, we introduce a new video dataset, namely Indoor Surveillance Dataset (ISD) for VFS. It has multiple annotations on a frame level such as foreground binary mask, depth map, and instance semantic annotations. Therefore, ISD can benefit other computer vision tasks. We validate the efficacy of our architectures and compare the performance with baselines. We demonstrate that proposed methods significantly outperform the benchmark methods on OOD. In addition, the performance of MUSTAN2 is significantly improved on certain video categories on OOD data due to ISD.

Title: Deep Learning Approaches for Network Traffic Classification in the Internet of Things (IoT): A Survey

• Authors: Jawad Hussain Kalwar, Sania Bhatti
• Subjects: cs.CR, cs.LG, cs.NI
• Abstract URL: https://arxiv.org/abs/2402.00920
• Pdf URL: https://arxiv.org/pdf/2402.00920
• Copy Paste: [[2402.00920]] Deep Learning Approaches for Network Traffic Classification in the Internet of Things (IoT): A Survey(https://arxiv.org/abs/2402.00920)
• Keywords: security
• Abstract: The Internet of Things (IoT) has witnessed unprecedented growth, resulting in a massive influx of diverse network traffic from interconnected devices. Effectively classifying this network traffic is crucial for optimizing resource allocation, enhancing security measures, and ensuring efficient network management in IoT systems. Deep learning has emerged as a powerful technique for network traffic classification due to its ability to automatically learn complex patterns and representations from raw data. This survey paper aims to provide a comprehensive overview of the existing deep learning approaches employed in network traffic classification specifically tailored for IoT environments. By systematically analyzing and categorizing the latest research contributions in this domain, we explore the strengths and limitations of various deep learning models in handling the unique challenges posed by IoT network traffic. Through this survey, we aim to offer researchers and practitioners valuable insights, identify research gaps, and provide directions for future research to further enhance the effectiveness and efficiency of deep learning-based network traffic classification in IoT.

Title: Towards post-quantum blockchain: A review on blockchain cryptography resistant to quantum computing attacks

• Authors: Tiago M. Fernandez-Carames, Paula Fraga-Lamas
• Subjects: cs.CR, quant-ph
• Abstract URL: https://arxiv.org/abs/2402.00922
• Pdf URL: https://arxiv.org/pdf/2402.00922
• Copy Paste: [[2402.00922]] Towards post-quantum blockchain: A review on blockchain cryptography resistant to quantum computing attacks(https://arxiv.org/abs/2402.00922)
• Keywords: security, attack
• Abstract: Blockchain and other Distributed Ledger Technologies (DLTs) have evolved significantly in the last years and their use has been suggested for numerous applications due to their ability to provide transparency, redundancy and accountability. In the case of blockchain, such characteristics are provided through public-key cryptography and hash functions. However, the fast progress of quantum computing has opened the possibility of performing attacks based on Grover's and Shor's algorithms in the near future. Such algorithms threaten both public-key cryptography and hash functions, forcing to redesign blockchains to make use of cryptosystems that withstand quantum attacks, thus creating which are known as post-quantum, quantum-proof, quantum-safe or quantum-resistant cryptosystems. For such a purpose, this article first studies current state of the art on post-quantum cryptosystems and how they can be applied to blockchains and DLTs. Moreover, the most relevant post-quantum blockchain systems are studied, as well as their main challenges. Furthermore, extensive comparisons are provided on the characteristics and performance of the most promising post-quantum public-key encryption and digital signature schemes for blockchains. Thus, this article seeks to provide a broad view and useful guidelines on post-quantum blockchain security to future blockchain researchers and developers.

Title: A Review on Blockchain Technologies for an Advanced and Cyber-Resilient Automotive Industry

• Authors: Paula Fraga-Lamas, Tiago M. Fernandez-Carames
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.00954
• Pdf URL: https://arxiv.org/pdf/2402.00954
• Copy Paste: [[2402.00954]] A Review on Blockchain Technologies for an Advanced and Cyber-Resilient Automotive Industry(https://arxiv.org/abs/2402.00954)
• Keywords: security, privacy, robust
• Abstract: In the last century the automotive industry has arguably transformed society, being one of the most complex, sophisticated and technologically advanced industries, with innovations ranging from hybrid, electric and self-driving smart cars to the development of IoT-connected cars. Due to its complexity, it requires the involvement of many Industry 4.0 technologies, like robotics, advanced manufacturing systems, cyber-physical systems or augmented reality. One of the latest technologies that can benefit the automotive industry is blockchain, which can enhance its data security, privacy, anonymity, traceability, accountability, integrity, robustness, transparency, trustworthiness and authentication, as well as provide long-term sustainability and a higher operational efficiency to the whole industry. This review analyzes the great potential of applying blockchain technologies to the automotive industry emphasizing its cybersecurity features. Thus, the applicability of blockchain is evaluated after examining the state-of-the-art and devising the main stakeholders' current challenges. Furthermore, the article describes the most relevant use cases, since the broad adoption of blockchain unlocks a wide area of short- and medium-term promising automotive applications that can create new business models and even disrupt the car-sharing economy as we know it. Finally, after a Strengths, Weaknesses, Opportunities, and Threats (SWOT) analysis, some recommendations are enumerated with the aim of guiding researchers and companies in future cyber-resilient automotive industry developments.

Title: FairEHR-CLP: Towards Fairness-Aware Clinical Predictions with Contrastive Learning in Multimodal Electronic Health Records

• Authors: Yuqing Wang, Malvika Pillai, Yun Zhao, Catherine Curtin, Tina Hernandez-Boussard
• Subjects: cs.LG, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.00955
• Pdf URL: https://arxiv.org/pdf/2402.00955
• Copy Paste: [[2402.00955]] FairEHR-CLP: Towards Fairness-Aware Clinical Predictions with Contrastive Learning in Multimodal Electronic Health Records(https://arxiv.org/abs/2402.00955)
• Keywords: fair
• Abstract: In the high-stakes realm of healthcare, ensuring fairness in predictive models is crucial. Electronic Health Records (EHRs) have become integral to medical decision-making, yet existing methods for enhancing model fairness restrict themselves to unimodal data and fail to address the multifaceted social biases intertwined with demographic factors in EHRs. To mitigate these biases, we present FairEHR-CLP: a general framework for Fairness-aware Clinical Predictions with Contrastive Learning in EHRs. FairEHR-CLP operates through a two-stage process, utilizing patient demographics, longitudinal data, and clinical notes. First, synthetic counterparts are generated for each patient, allowing for diverse demographic identities while preserving essential health information. Second, fairness-aware predictions employ contrastive learning to align patient representations across sensitive attributes, jointly optimized with an MLP classifier with a softmax layer for clinical classification tasks. Acknowledging the unique challenges in EHRs, such as varying group sizes and class imbalance, we introduce a novel fairness metric to effectively measure error rate disparities across subgroups. Extensive experiments on three diverse EHR datasets on three tasks demonstrate the effectiveness of FairEHR-CLP in terms of fairness and utility compared with competitive baselines. FairEHR-CLP represents an advancement towards ensuring both accuracy and equity in predictive healthcare models.

Title: Exploring Spatial Schema Intuitions in Large Language and Vision Models

• Authors: Philipp Wicke, Lennart Wachowiak
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.00956
• Pdf URL: https://arxiv.org/pdf/2402.00956
• Copy Paste: [[2402.00956]] Exploring Spatial Schema Intuitions in Large Language and Vision Models(https://arxiv.org/abs/2402.00956)
• Keywords: large language model
• Abstract: Despite the ubiquity of large language models (LLMs) in AI research, the question of embodiment in LLMs remains underexplored, distinguishing them from embodied systems in robotics where sensory perception directly informs physical action. Our investigation navigates the intriguing terrain of whether LLMs, despite their non-embodied nature, effectively capture implicit human intuitions about fundamental, spatial building blocks of language. We employ insights from spatial cognitive foundations developed through early sensorimotor experiences, guiding our exploration through the reproduction of three psycholinguistic experiments. Surprisingly, correlations between model outputs and human responses emerge, revealing adaptability without a tangible connection to embodied experiences. Notable distinctions include polarized language model responses and reduced correlations in vision language models. This research contributes to a nuanced understanding of the interplay between language, spatial experiences, and the computations made by large language models. More at https://cisnlp.github.io/Spatial_Schemas/

Title: FuseFormer: A Transformer for Visual and Thermal Image Fusion

• Authors: Aytekin Erdogan, Erdem Akagunduz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.00971
• Pdf URL: https://arxiv.org/pdf/2402.00971
• Copy Paste: [[2402.00971]] FuseFormer: A Transformer for Visual and Thermal Image Fusion(https://arxiv.org/abs/2402.00971)
• Keywords: transformer
• Abstract: Image fusion is the process of combining images from different sensors into a single image that incorporates all relevant information. The majority of state-of-the-art image fusion techniques use deep learning methods to extract meaningful features; however, they primarily integrate local features without considering the image's broader context. To overcome this limitation, Transformer-based models have emerged as a promising solution, aiming to capture general context dependencies through attention mechanisms. Since there is no ground truth for image fusion, the loss functions are structured based on evaluation metrics, such as the structural similarity index measure (SSIM). By doing so, we create a bias towards the SSIM and, therefore, the input visual band image. The objective of this study is to propose a novel methodology for image fusion that mitigates the limitations associated with using evaluation metrics as loss functions. Our approach integrates a transformer-based multi-scale fusion strategy, which adeptly addresses both local and global context information. This integration not only refines the individual components of the image fusion process but also significantly enhances the overall efficacy of the method. Our proposed method follows a two-stage training approach, where an auto-encoder is initially trained to extract deep features at multiple scales at the first stage. For the second stage, we integrate our fusion block and change the loss function as mentioned. The multi-scale features are fused using a combination of Convolutional Neural Networks (CNNs) and Transformers. The CNNs are utilized to capture local features, while the Transformer handles the integration of general context features.

Title: Recurrent Transformers with Dynamic Halt

• Authors: Jishnu Ray Chowdhury, Cornelia Caragea
• Subjects: cs.LG, cs.AI, cs.NE
• Abstract URL: https://arxiv.org/abs/2402.00976
• Pdf URL: https://arxiv.org/pdf/2402.00976
• Copy Paste: [[2402.00976]] Recurrent Transformers with Dynamic Halt(https://arxiv.org/abs/2402.00976)
• Keywords: transformer
• Abstract: In this paper, we study the inductive biases of two major approaches to augmenting Transformers with a recurrent mechanism - (1) the approach of incorporating a depth-wise recurrence similar to Universal Transformers; and (2) the approach of incorporating a chunk-wise temporal recurrence like Temporal Latent Bottleneck. Furthermore, we propose and investigate novel ways to extend and combine the above methods - for example, we propose a global mean-based dynamic halting mechanism for Universal Transformer and an augmentation of Temporal Latent Bottleneck with elements from Universal Transformer. We compare the models and probe their inductive biases in several diagnostic tasks such as Long Range Arena (LRA), flip-flop language modeling, ListOps, and Logical Inference.

Title: Enhanced fringe-to-phase framework using deep learning

• Authors: Won-Hoe Kim, Bongjoong Kim, Hyung-Gun Chi, Jae-Sang Hyun
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.00977
• Pdf URL: https://arxiv.org/pdf/2402.00977
• Copy Paste: [[2402.00977]] Enhanced fringe-to-phase framework using deep learning(https://arxiv.org/abs/2402.00977)
• Keywords: robust
• Abstract: In Fringe Projection Profilometry (FPP), achieving robust and accurate 3D reconstruction with a limited number of fringe patterns remains a challenge in structured light 3D imaging. Conventional methods require a set of fringe images, but using only one or two patterns complicates phase recovery and unwrapping. In this study, we introduce SFNet, a symmetric fusion network that transforms two fringe images into an absolute phase. To enhance output reliability, Our framework predicts refined phases by incorporating information from fringe images of a different frequency than those used as input. This allows us to achieve high accuracy with just two images. Comparative experiments and ablation studies validate the effectiveness of our proposed method. The dataset and code are publicly accessible on our project page https://wonhoe-kim.github.io/SFNet.

Title: Self-Supervised Contrastive Pre-Training for Multivariate Point Processes

• Authors: Xiao Shou, Dharmashankar Subramanian, Debarun Bhattacharjya, Tian Gao, Kristin P. Bennet
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00987
• Pdf URL: https://arxiv.org/pdf/2402.00987
• Copy Paste: [[2402.00987]] Self-Supervised Contrastive Pre-Training for Multivariate Point Processes(https://arxiv.org/abs/2402.00987)
• Keywords: transformer, large language model
• Abstract: Self-supervision is one of the hallmarks of representation learning in the increasingly popular suite of foundation models including large language models such as BERT and GPT-3, but it has not been pursued in the context of multivariate event streams, to the best of our knowledge. We introduce a new paradigm for self-supervised learning for multivariate point processes using a transformer encoder. Specifically, we design a novel pre-training strategy for the encoder where we not only mask random event epochs but also insert randomly sampled "void" epochs where an event does not occur; this differs from the typical discrete-time pretext tasks such as word-masking in BERT but expands the effectiveness of masking to better capture continuous-time dynamics. To improve downstream tasks, we introduce a contrasting module that compares real events to simulated void instances. The pre-trained model can subsequently be fine-tuned on a potentially much smaller event dataset, similar conceptually to the typical transfer of popular pre-trained language models. We demonstrate the effectiveness of our proposed paradigm on the next-event prediction task using synthetic datasets and 3 real applications, observing a relative performance boost of as high as up to 20% compared to state-of-the-art models.

Title: A Cost-Efficient Approach for Creating Virtual Fitting Room using Generative Adversarial Networks (GANs)

• Authors: Kirolos Attallah, Girgis Zaky, Nourhan Abdelrhim, Kyrillos Botros, Amjad Dife, Nermin Negied
• Subjects: cs.CV, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.00994
• Pdf URL: https://arxiv.org/pdf/2402.00994
• Copy Paste: [[2402.00994]] A Cost-Efficient Approach for Creating Virtual Fitting Room using Generative Adversarial Networks (GANs)(https://arxiv.org/abs/2402.00994)
• Keywords: generative
• Abstract: Customers all over the world want to see how the clothes fit them or not before purchasing. Therefore, customers by nature prefer brick-and-mortar clothes shopping so they can try on products before purchasing them. But after the Pandemic of COVID19 many sellers either shifted to online shopping or closed their fitting rooms which made the shopping process hesitant and doubtful. The fact that the clothes may not be suitable for their buyers after purchase led us to think about using new AI technologies to create an online platform or a virtual fitting room (VFR) in the form of a mobile application and a deployed model using a webpage that can be embedded later to any online store where they can try on any number of cloth items without physically trying them. Besides, it will save much searching time for their needs. Furthermore, it will reduce the crowding and headache in the physical shops by applying the same technology using a special type of mirror that will enable customers to try on faster. On the other hand, from business owners' perspective, this project will highly increase their online sales, besides, it will save the quality of the products by avoiding physical trials issues. The main approach used in this work is applying Generative Adversarial Networks (GANs) combined with image processing techniques to generate one output image from two input images which are the person image and the cloth image. This work achieved results that outperformed the state-of-the-art approaches found in literature.

Title: mmID: High-Resolution mmWave Imaging for Human Identification

• Authors: Sakila S. Jayaweera, Sai Deepika Regani, Yuqian Hu, Beibei Wang, K. J. Ray Liu
• Subjects: cs.CV, eess.SP
• Abstract URL: https://arxiv.org/abs/2402.00996
• Pdf URL: https://arxiv.org/pdf/2402.00996
• Copy Paste: [[2402.00996]] mmID: High-Resolution mmWave Imaging for Human Identification(https://arxiv.org/abs/2402.00996)
• Keywords: generative
• Abstract: Achieving accurate human identification through RF imaging has been a persistent challenge, primarily attributed to the limited aperture size and its consequent impact on imaging resolution. The existing imaging solution enables tasks such as pose estimation, activity recognition, and human tracking based on deep neural networks by estimating skeleton joints. In contrast to estimating joints, this paper proposes to improve imaging resolution by estimating the human figure as a whole using conditional generative adversarial networks (cGAN). In order to reduce training complexity, we use an estimated spatial spectrum using the MUltiple SIgnal Classification (MUSIC) algorithm as input to the cGAN. Our system generates environmentally independent, high-resolution images that can extract unique physical features useful for human identification. We use a simple convolution layers-based classification network to obtain the final identification result. From the experimental results, we show that resolution of the image produced by our trained generator is high enough to enable human identification. Our finding indicates high-resolution accuracy with 5% mean silhouette difference to the Kinect device. Extensive experiments in different environments on multiple testers demonstrate that our system can achieve 93% overall test accuracy in unseen environments for static human target identification.

Title: AI-generated faces free from racial and gender stereotypes

• Authors: Nouar AlDahoul, Talal Rahwan, Yasir Zaki
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01002
• Pdf URL: https://arxiv.org/pdf/2402.01002
• Copy Paste: [[2402.01002]] AI-generated faces free from racial and gender stereotypes(https://arxiv.org/abs/2402.01002)
• Keywords: diffusion, generative
• Abstract: Text-to-image generative AI models such as Stable Diffusion are used daily by millions worldwide. However, many have raised concerns regarding how these models amplify racial and gender stereotypes. To study this phenomenon, we develop a classifier to predict the race, gender, and age group of any given face image, and show that it achieves state-of-the-art performance. Using this classifier, we quantify biases in Stable Diffusion across six races, two genders, five age groups, 32 professions, and eight attributes. We then propose novel debiasing solutions that outperform state-of-the-art alternatives. Additionally, we examine the degree to which Stable Diffusion depicts individuals of the same race as being similar to one another. This analysis reveals a high degree of stereotyping, e.g., depicting most middle eastern males as being dark-skinned, bearded, and wearing a traditional headdress. We address these limitations by proposing yet another novel solution that increases facial diversity across genders and racial groups. Our solutions are open-sourced and made publicly available.

Title: Municipal cyber risk modeling using cryptographic computing to inform cyber policymaking

• Authors: Avital Baral, Taylor Reynolds, Lawrence Susskind, Daniel J. Weitzner, Angelina Wu
• Subjects: cs.CR, econ.GN
• Abstract URL: https://arxiv.org/abs/2402.01007
• Pdf URL: https://arxiv.org/pdf/2402.01007
• Copy Paste: [[2402.01007]] Municipal cyber risk modeling using cryptographic computing to inform cyber policymaking(https://arxiv.org/abs/2402.01007)
• Keywords: secure, security, attack
• Abstract: Municipalities are vulnerable to cyberattacks with devastating consequences, but they lack key information to evaluate their own risk and compare their security posture to peers. Using data from 83 municipalities collected via a cryptographically secure computation platform about their security posture, incidents, security control failures, and losses, we build data-driven cyber risk models and cyber security benchmarks for municipalities. We produce benchmarks of the security posture in a sector, the frequency of cyber incidents, forecasted annual losses for organizations based on their defensive posture, and a weighting of cyber controls based on their individual failure rates and associated losses. Combined, these four items can help guide cyber policymaking by quantifying the cyber risk in a sector, identifying gaps that need to be addressed, prioritizing policy interventions, and tracking progress of those interventions over time. In the case of the municipalities, these newly derived risk measures highlight the need for continuous measured improvement of cybersecurity readiness, show clear areas of weakness and strength, and provide governments with some early targets for policy focus such as security education, incident response, and focusing efforts first on municipalities at the lowest security levels that have the highest risk reduction per security dollar invested.

Title: algoXSSF: Detection and analysis of cross-site request forgery (XSRF) and cross-site scripting (XSS) attacks via Machine learning algorithms

• Authors: Naresh Kshetri, Dilip Kumar, James Hutson, Navneet Kaur, Omar Faruq Osama
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.01012
• Pdf URL: https://arxiv.org/pdf/2402.01012
• Copy Paste: [[2402.01012]] algoXSSF: Detection and analysis of cross-site request forgery (XSRF) and cross-site scripting (XSS) attacks via Machine learning algorithms(https://arxiv.org/abs/2402.01012)
• Keywords: security, protect, defense, attack
• Abstract: The global rise of online users and online devices has ultimately given rise to the global internet population apart from several cybercrimes and cyberattacks. The combination of emerging new technology and powerful algorithms (of Artificial Intelligence, Deep Learning, and Machine Learning) is needed to counter defense web security including attacks on several search engines and websites. The unprecedented increase rate of cybercrime and website attacks urged for new technology consideration to protect data and information online. There have been recent and continuous cyberattacks on websites, web domains with ongoing data breaches including - GitHub account hack, data leaks on Twitter, malware in WordPress plugins, vulnerability in Tomcat server to name just a few. We have investigated with an in-depth study apart from the detection and analysis of two major cyberattacks (although there are many more types): cross-site request forgery (XSRF) and cross-site scripting (XSS) attacks. The easy identification of cyber trends and patterns with continuous improvement is possible within the edge of machine learning and AI algorithms. The use of machine learning algorithms would be extremely helpful to counter (apart from detection) the XSRF and XSS attacks. We have developed the algorithm and cyber defense framework - algoXSSF with machine learning algorithms embedded to combat malicious attacks (including Man-in-the-Middle attacks) on websites for detection and analysis.

Title: HR-MultiWOZ: A Task Oriented Dialogue (TOD) Dataset for HR LLM Agent

• Authors: Weijie Xu, Zicheng Huang, Wenxiang Hu, Xi Fang, Rajesh Kumar Cherukuri, Naumaan Nayyar, Lorenzo Malandri, Srinivasan H. Sengamedu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01018
• Pdf URL: https://arxiv.org/pdf/2402.01018
• Copy Paste: [[2402.01018]] HR-MultiWOZ: A Task Oriented Dialogue (TOD) Dataset for HR LLM Agent(https://arxiv.org/abs/2402.01018)
• Keywords: privacy, large language model
• Abstract: Recent advancements in Large Language Models (LLMs) have been reshaping Natural Language Processing (NLP) task in several domains. Their use in the field of Human Resources (HR) has still room for expansions and could be beneficial for several time consuming tasks. Examples such as time-off submissions, medical claims filing, and access requests are noteworthy, but they are by no means the sole instances. However, the aforementioned developments must grapple with the pivotal challenge of constructing a high-quality training dataset. On one hand, most conversation datasets are solving problems for customers not employees. On the other hand, gathering conversations with HR could raise privacy concerns. To solve it, we introduce HR-Multiwoz, a fully-labeled dataset of 550 conversations spanning 10 HR domains to evaluate LLM Agent. Our work has the following contributions: (1) It is the first labeled open-sourced conversation dataset in the HR domain for NLP research. (2) It provides a detailed recipe for the data generation procedure along with data analysis and human evaluations. The data generation pipeline is transferable and can be easily adapted for labeled conversation data generation in other domains. (3) The proposed data-collection pipeline is mostly based on LLMs with minimal human involvement for annotation, which is time and cost-efficient.

Title: Domain-Independent Deception: A New Taxonomy and Linguistic Analysis

• Authors: Rakesh M. Verma, Nachum Dershowitz, Victor Zeng, Dainis Boumber, Xuting Liu
• Subjects: cs.CL, cs.CR, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.01019
• Pdf URL: https://arxiv.org/pdf/2402.01019
• Copy Paste: [[2402.01019]] Domain-Independent Deception: A New Taxonomy and Linguistic Analysis(https://arxiv.org/abs/2402.01019)
• Keywords: attack
• Abstract: Internet-based economies and societies are drowning in deceptive attacks. These attacks take many forms, such as fake news, phishing, and job scams, which we call ``domains of deception.'' Machine-learning and natural-language-processing researchers have been attempting to ameliorate this precarious situation by designing domain-specific detectors. Only a few recent works have considered domain-independent deception. We collect these disparate threads of research and investigate domain-independent deception. First, we provide a new computational definition of deception and break down deception into a new taxonomy. Then, we analyze the debate on linguistic cues for deception and supply guidelines for systematic reviews. Finally, we investigate common linguistic features and give evidence for knowledge transfer across different forms of deception.

Title: Executable Code Actions Elicit Better LLM Agents

• Authors: Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, Heng Ji
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01030
• Pdf URL: https://arxiv.org/pdf/2402.01030
• Copy Paste: [[2402.01030]] Executable Code Actions Elicit Better LLM Agents(https://arxiv.org/abs/2402.01030)
• Keywords: large language model
• Abstract: Large Language Model (LLM) agents, capable of performing a broad range of actions, such as invoking tools and controlling robots, show great potential in tackling real-world challenges. LLM agents are typically prompted to produce actions by generating JSON or text in a pre-defined format, which is usually limited by constrained action space (e.g., the scope of pre-defined tools) and restricted flexibility (e.g., inability to compose multiple tools). This work proposes to use executable Python code to consolidate LLM agents' actions into a unified action space (CodeAct). Integrated with a Python interpreter, CodeAct can execute code actions and dynamically revise prior actions or emit new actions upon new observations through multi-turn interactions. Our extensive analysis of 17 LLMs on API-Bank and a newly curated benchmark shows that CodeAct outperforms widely used alternatives (up to 20% higher success rate). The encouraging performance of CodeAct motivates us to build an open-source LLM agent that interacts with environments by executing interpretable code and collaborates with users using natural language. To this end, we collect an instruction-tuning dataset CodeActInstruct that consists of 7k multi-turn interactions using CodeAct. We show that it can be used with existing data to improve models in agent-oriented tasks without compromising their general capability. CodeActAgent, finetuned from Llama2 and Mistral, is integrated with Python interpreter and uniquely tailored to perform sophisticated tasks (e.g., model training) using existing libraries and autonomously self-debug.

Title: Repeat After Me: Transformers are Better than State Space Models at Copying

• Authors: Samy Jelassi, David Brandfonbrener, Sham M. Kakade, Eran Malach
• Subjects: cs.LG, cs.AI, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01032
• Pdf URL: https://arxiv.org/pdf/2402.01032
• Copy Paste: [[2402.01032]] Repeat After Me: Transformers are Better than State Space Models at Copying(https://arxiv.org/abs/2402.01032)
• Keywords: transformer, large language model
• Abstract: Transformers are the dominant architecture for sequence modeling, but there is growing interest in models that use a fixed-size latent state that does not depend on the sequence length, which we refer to as "generalized state space models" (GSSMs). In this paper we show that while GSSMs are promising in terms of inference-time efficiency, they are limited compared to transformer models on tasks that require copying from the input context. We start with a theoretical analysis of the simple task of string copying and prove that a two layer transformer can copy strings of exponential length while GSSMs are fundamentally limited by their fixed-size latent state. Empirically, we find that transformers outperform GSSMs in terms of efficiency and generalization on synthetic tasks that require copying the context. Finally, we evaluate pretrained large language models and find that transformer models dramatically outperform state space models at copying and retrieving information from context. Taken together, these results suggest a fundamental gap between transformers and GSSMs on tasks of practical interest.

Title: Ultra Fast Transformers on FPGAs for Particle Physics Experiments

• Authors: Zhixing Jiang, Dennis Yin, Elham E Khoda, Vladimir Loncar, Ekaterina Govorkova, Eric Moreno, Philip Harris, Scott Hauck, Shih-Chieh Hsu
• Subjects: cs.LG, cs.AR, hep-ex
• Abstract URL: https://arxiv.org/abs/2402.01047
• Pdf URL: https://arxiv.org/pdf/2402.01047
• Copy Paste: [[2402.01047]] Ultra Fast Transformers on FPGAs for Particle Physics Experiments(https://arxiv.org/abs/2402.01047)
• Keywords: transformer
• Abstract: This work introduces a highly efficient implementation of the transformer architecture on a Field-Programmable Gate Array (FPGA) by using the \texttt{hls4ml} tool. Given the demonstrated effectiveness of transformer models in addressing a wide range of problems, their application in experimental triggers within particle physics becomes a subject of significant interest. In this work, we have implemented critical components of a transformer model, such as multi-head attention and softmax layers. To evaluate the effectiveness of our implementation, we have focused on a particle physics jet flavor tagging problem, employing a public dataset. We recorded latency under 2 $\mu$s on the Xilinx UltraScale+ FPGA, which is compatible with hardware trigger requirements at the CERN Large Hadron Collider experiments.

Title: IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition

• Authors: Zikang Leng, Amitrajit Bhattacharjee, Hrudhai Rajasekhar, Lizhe Zhang, Elizabeth Bruda, Hyeokhyen Kwon, Thomas Plötz
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01049
• Pdf URL: https://arxiv.org/pdf/2402.01049
• Copy Paste: [[2402.01049]] IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition(https://arxiv.org/abs/2402.01049)
• Keywords: robust, generative, large language model
• Abstract: One of the primary challenges in the field of human activity recognition (HAR) is the lack of large labeled datasets. This hinders the development of robust and generalizable models. Recently, cross modality transfer approaches have been explored that can alleviate the problem of data scarcity. These approaches convert existing datasets from a source modality, such as video, to a target modality (IMU). With the emergence of generative AI models such as large language models (LLMs) and text-driven motion synthesis models, language has become a promising source data modality as well as shown in proof of concepts such as IMUGPT. In this work, we conduct a large-scale evaluation of language-based cross modality transfer to determine their effectiveness for HAR. Based on this study, we introduce two new extensions for IMUGPT that enhance its use for practical HAR application scenarios: a motion filter capable of filtering out irrelevant motion sequences to ensure the relevance of the generated virtual IMU data, and a set of metrics that measure the diversity of the generated data facilitating the determination of when to stop generating virtual IMU data for both effective and efficient processing. We demonstrate that our diversity metrics can reduce the effort needed for the generation of virtual IMU data by at least 50%, which open up IMUGPT for practical use cases beyond a mere proof of concept.

Title: Plan-Grounded Large Language Models for Dual Goal Conversational Settings

• Authors: Diogo Glória-Silva, Rafael Ferreira, Diogo Tavares, David Semedo, João Magalhães
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01053
• Pdf URL: https://arxiv.org/pdf/2402.01053
• Copy Paste: [[2402.01053]] Plan-Grounded Large Language Models for Dual Goal Conversational Settings(https://arxiv.org/abs/2402.01053)
• Keywords: large language model
• Abstract: Training Large Language Models (LLMs) to follow user instructions has been shown to supply the LLM with ample capacity to converse fluently while being aligned with humans. Yet, it is not completely clear how an LLM can lead a plan-grounded conversation in mixed-initiative settings where instructions flow in both directions of the conversation, i.e. both the LLM and the user provide instructions to one another. In this paper, we tackle a dual goal mixed-initiative conversational setting where the LLM not only grounds the conversation on an arbitrary plan but also seeks to satisfy both a procedural plan and user instructions. The LLM is then responsible for guiding the user through the plan and, at the same time, adapting to new circumstances, answering questions, and activating safety guardrails when needed. We propose a novel LLM that grounds the dialogue on a procedural plan, can take the dialogue initiative, and enforces guardrails on the system's behavior, while also improving the LLM's responses to unexpected user behavior. Experiments in controlled settings and with real users show that the best-performing model, which we call PlanLLM, achieves a 2.1x improvement over a strong baseline. Moreover, experiments also show good generalization to unseen domains.

Title: Evaluation Methodology for Large Language Models for Multilingual Document Question and Answer

• Authors: Adar Kahana, Jaya Susan Mathew, Said Bleik, Jeremy Reynolds, Oren Elisha
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01065
• Pdf URL: https://arxiv.org/pdf/2402.01065
• Copy Paste: [[2402.01065]] Evaluation Methodology for Large Language Models for Multilingual Document Question and Answer(https://arxiv.org/abs/2402.01065)
• Keywords: large language model
• Abstract: With the widespread adoption of Large Language Models (LLMs), in this paper we investigate the multilingual capability of these models. Our preliminary results show that, translating the native language context, question and answer into a high resource language produced the best results.

Title: FedShift: Tackling Dual Heterogeneity Problem of Federated Learning via Weight Shift Aggregation

• Authors: Jungwon Seo, Chunming Rong, Minhoe Kim
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.01070
• Pdf URL: https://arxiv.org/pdf/2402.01070
• Copy Paste: [[2402.01070]] FedShift: Tackling Dual Heterogeneity Problem of Federated Learning via Weight Shift Aggregation(https://arxiv.org/abs/2402.01070)
• Keywords: privacy, federate
• Abstract: Federated Learning (FL) offers a compelling method for training machine learning models with a focus on preserving data privacy. The presence of system heterogeneity and statistical heterogeneity, recognized challenges in FL, arises from the diversity of client hardware, network, and dataset distribution. This diversity can critically affect the training pace and the performance of models. While many studies address either system or statistical heterogeneity by introducing communication-efficient or stable convergence algorithms, addressing these challenges in isolation often leads to compromises due to unaddressed heterogeneity. In response, this paper introduces FedShift, a novel algorithm designed to enhance both the training speed and the models' accuracy in a dual heterogeneity scenario. Our solution can improve client engagement through quantization and mitigate the adverse effects on performance typically associated with quantization by employing a shifting technique. This technique has proven to enhance accuracy by an average of 3.9% in diverse heterogeneity environments.

Title: Chameleon: Foundation Models for Fairness-aware Multi-modal Data Augmentation to Enhance Coverage of Minorities

• Authors: Mahdi Erfanian, H. V. Jagadish, Abolfazl Asudeh
• Subjects: cs.LG, cs.CY, cs.DB
• Abstract URL: https://arxiv.org/abs/2402.01071
• Pdf URL: https://arxiv.org/pdf/2402.01071
• Copy Paste: [[2402.01071]] Chameleon: Foundation Models for Fairness-aware Multi-modal Data Augmentation to Enhance Coverage of Minorities(https://arxiv.org/abs/2402.01071)
• Keywords: fair, generative, large language model
• Abstract: The potential harms of the under-representation of minorities in training data, particularly in multi-modal settings, is a well-recognized concern. While there has been extensive effort in detecting such under-representation, resolution has remained a challenge. With recent advancements in generative AI, large language models and foundation models have emerged as versatile tools across various domains. In this paper, we propose Chameleon, a system that efficiently utilizes these tools to augment a data set with a minimal addition of synthetically generated tuples, in order to enhance the coverage of the under-represented groups. Our system follows a rejection sampling approach to ensure the generated tuples have a high quality and follow the underlying distribution. In order to minimize the rejection chance of the generated tuples, we propose multiple strategies for providing a guide for the foundation model. Our experiment results, in addition to confirming the efficiency of our proposed algorithms, illustrate the effectiveness of our approach, as the unfairness of the model in a downstream task significantly dropped after data repair using Chameleon.

Title: Salsa Fresca: Angular Embeddings and Pre-Training for ML Attacks on Learning With Errors

• Authors: Samuel Stevens, Emily Wenger, Cathy Li, Niklas Nolte, Eshika Saxena, François Charton, Kristin Lauter
• Subjects: cs.CR, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.01082
• Pdf URL: https://arxiv.org/pdf/2402.01082
• Copy Paste: [[2402.01082]] Salsa Fresca: Angular Embeddings and Pre-Training for ML Attacks on Learning With Errors(https://arxiv.org/abs/2402.01082)
• Keywords: attack
• Abstract: Learning with Errors (LWE) is a hard math problem underlying recently standardized post-quantum cryptography (PQC) systems for key exchange and digital signatures. Prior work proposed new machine learning (ML)-based attacks on LWE problems with small, sparse secrets, but these attacks require millions of LWE samples to train on and take days to recover secrets. We propose three key methods -- better preprocessing, angular embeddings and model pre-training -- to improve these attacks, speeding up preprocessing by $25\times$ and improving model sample efficiency by $10\times$. We demonstrate for the first time that pre-training improves and reduces the cost of ML attacks on LWE. Our architecture improvements enable scaling to larger-dimension LWE problems: this work is the first instance of ML attacks recovering sparse binary secrets in dimension $n=1024$, the smallest dimension used in practice for homomorphic encryption applications of LWE where sparse binary secrets are proposed.

Title: Specialized Language Models with Cheap Inference from Limited Domain Data

• Authors: David Grangier, Angelos Katharopoulos, Pierre Ablin, Awni Hannun
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01093
• Pdf URL: https://arxiv.org/pdf/2402.01093
• Copy Paste: [[2402.01093]] Specialized Language Models with Cheap Inference from Limited Domain Data(https://arxiv.org/abs/2402.01093)
• Keywords: transformer, large language model
• Abstract: Large language models have emerged as a versatile tool but are challenging to apply to tasks lacking large inference budgets and large in-domain training sets. This work formalizes these constraints and distinguishes four important variables: the pretraining budget (for training before the target domain is known), the specialization budget (for training after the target domain is known), the inference budget, and the in-domain training set size. Across these settings, we compare different approaches from the machine learning literature. Limited by inference cost, we find better alternatives to the standard practice of training very large vanilla transformer models. In particular, we show that hyper-networks and mixture of experts have better perplexity for large pretraining budgets, while small models trained on importance sampled datasets are attractive for large specialization budgets.

Title: How many views does your deep neural network use for prediction?

• Authors: Keisuke Kawano, Takuro Kutsuna, Keisuke Sano
• Subjects: cs.LG, cs.AI, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.01095
• Pdf URL: https://arxiv.org/pdf/2402.01095
• Copy Paste: [[2402.01095]] How many views does your deep neural network use for prediction?(https://arxiv.org/abs/2402.01095)
• Keywords: transformer
• Abstract: The generalization ability of Deep Neural Networks (DNNs) is still not fully understood, despite numerous theoretical and empirical analyses. Recently, Allen-Zhu & Li (2023) introduced the concept of multi-views to explain the generalization ability of DNNs, but their main target is ensemble or distilled models, and no method for estimating multi-views used in a prediction of a specific input is discussed. In this paper, we propose Minimal Sufficient Views (MSVs), which is similar to multi-views but can be efficiently computed for real images. MSVs is a set of minimal and distinct features in an input, each of which preserves a model's prediction for the input. We empirically show that there is a clear relationship between the number of MSVs and prediction accuracy across models, including convolutional and transformer models, suggesting that a multi-view like perspective is also important for understanding the generalization ability of (non-ensemble or non-distilled) DNNs.

Title: Trustworthy Distributed AI Systems: Robustness, Privacy, and Governance

• Authors: Wenqi Wei, Ling Liu
• Subjects: cs.LG, cs.AI, cs.CR, cs.DC
• Abstract URL: https://arxiv.org/abs/2402.01096
• Pdf URL: https://arxiv.org/pdf/2402.01096
• Copy Paste: [[2402.01096]] Trustworthy Distributed AI Systems: Robustness, Privacy, and Governance(https://arxiv.org/abs/2402.01096)
• Keywords: security, privacy, protect, attack, robust, fair
• Abstract: Emerging Distributed AI systems are revolutionizing big data computing and data processing capabilities with growing economic and societal impact. However, recent studies have identified new attack surfaces and risks caused by security, privacy, and fairness issues in AI systems. In this paper, we review representative techniques, algorithms, and theoretical foundations for trustworthy distributed AI through robustness guarantee, privacy protection, and fairness awareness in distributed learning. We first provide a brief overview of alternative architectures for distributed learning, discuss inherent vulnerabilities for security, privacy, and fairness of AI algorithms in distributed learning, and analyze why these problems are present in distributed learning regardless of specific architectures. Then we provide a unique taxonomy of countermeasures for trustworthy distributed AI, covering (1) robustness to evasion attacks and irregular queries at inference, and robustness to poisoning attacks, Byzantine attacks, and irregular data distribution during training; (2) privacy protection during distributed learning and model inference at deployment; and (3) AI fairness and governance with respect to both data and models. We conclude with a discussion on open challenges and future research directions toward trustworthy distributed AI, such as the need for trustworthy AI policy guidelines, the AI responsibility-utility co-design, and incentives and compliance.

Title: Compositional Generative Modeling: A Single Model is Not All You Need

• Authors: Yilun Du, Leslie Kaelbling
• Subjects: cs.LG, cs.AI, cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.01103
• Pdf URL: https://arxiv.org/pdf/2402.01103
• Copy Paste: [[2402.01103]] Compositional Generative Modeling: A Single Model is Not All You Need(https://arxiv.org/abs/2402.01103)
• Keywords: generative
• Abstract: Large monolithic generative models trained on massive amounts of data have become an increasingly dominant approach in AI research. In this paper, we argue that we should instead construct large generative systems by composing smaller generative models together. We show how such a compositional generative approach enables us to learn distributions in a more data-efficient manner, enabling generalization to parts of the data distribution unseen at training time. We further show how this enables us to program and construct new generative models for tasks completely unseen at training. Finally, we show that in many cases, we can discover separate compositional components from data.

Title: A Survey for Foundation Models in Autonomous Driving

• Authors: Haoxiang Gao, Yaqian Li, Kaiwen Long, Ming Yang, Yiqing Shen
• Subjects: cs.LG, cs.CV, cs.RO
• Abstract URL: https://arxiv.org/abs/2402.01105
• Pdf URL: https://arxiv.org/pdf/2402.01105
• Copy Paste: [[2402.01105]] A Survey for Foundation Models in Autonomous Driving(https://arxiv.org/abs/2402.01105)
• Keywords: large language model
• Abstract: The advent of foundation models has revolutionized the fields of natural language processing and computer vision, paving the way for their application in autonomous driving (AD). This survey presents a comprehensive review of more than 40 research papers, demonstrating the role of foundation models in enhancing AD. Large language models contribute to planning and simulation in AD, particularly through their proficiency in reasoning, code generation and translation. In parallel, vision foundation models are increasingly adapted for critical tasks such as 3D object detection and tracking, as well as creating realistic driving scenarios for simulation and testing. Multi-modal foundation models, integrating diverse inputs, exhibit exceptional visual understanding and spatial reasoning, crucial for end-to-end AD. This survey not only provides a structured taxonomy, categorizing foundation models based on their modalities and functionalities within the AD domain but also delves into the methods employed in current research. It identifies the gaps between existing foundation models and cutting-edge AD approaches, thereby charting future research directions and proposing a roadmap for bridging these gaps.

Title: Simulation of Graph Algorithms with Looped Transformers

• Authors: Artur Back de Luca, Kimon Fountoulakis
• Subjects: cs.LG, cs.AI, cs.DS
• Abstract URL: https://arxiv.org/abs/2402.01107
• Pdf URL: https://arxiv.org/pdf/2402.01107
• Copy Paste: [[2402.01107]] Simulation of Graph Algorithms with Looped Transformers(https://arxiv.org/abs/2402.01107)
• Keywords: transformer
• Abstract: The execution of graph algorithms using neural networks has recently attracted significant interest due to promising empirical progress. This motivates further understanding of how neural networks can replicate reasoning steps with relational data. In this work, we study the ability of transformer networks to simulate algorithms on graphs from a theoretical perspective. The architecture that we utilize is a looped transformer with extra attention heads that interact with the graph. We prove by construction that this architecture can simulate algorithms such as Dijkstra's shortest path algorithm, Breadth- and Depth-First Search, and Kosaraju's strongly connected components algorithm. The width of the network does not increase with the size of the input graph, which implies that the network can simulate the above algorithms for any graph. Despite this property, we show that there is a limit to simulation in our solution due to finite precision. Finally, we show a Turing Completeness result with constant width when the extra attention heads are utilized.

Title: Reasoning Capacity in Multi-Agent Systems: Limitations, Challenges and Human-Centered Solutions

• Authors: Pouya Pezeshkpour, Eser Kandogan, Nikita Bhutani, Sajjadur Rahman, Tom Mitchell, Estevam Hruschka
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.01108
• Pdf URL: https://arxiv.org/pdf/2402.01108
• Copy Paste: [[2402.01108]] Reasoning Capacity in Multi-Agent Systems: Limitations, Challenges and Human-Centered Solutions(https://arxiv.org/abs/2402.01108)
• Keywords: large language model
• Abstract: Remarkable performance of large language models (LLMs) in a variety of tasks brings forth many opportunities as well as challenges of utilizing them in production settings. Towards practical adoption of LLMs, multi-agent systems hold great promise to augment, integrate, and orchestrate LLMs in the larger context of enterprise platforms that use existing proprietary data and models to tackle complex real-world tasks. Despite the tremendous success of these systems, current approaches rely on narrow, single-focus objectives for optimization and evaluation, often overlooking potential constraints in real-world scenarios, including restricted budgets, resources and time. Furthermore, interpreting, analyzing, and debugging these systems requires different components to be evaluated in relation to one another. This demand is currently not feasible with existing methodologies. In this postion paper, we introduce the concept of reasoning capacity as a unifying criterion to enable integration of constraints during optimization and establish connections among different components within the system, which also enable a more holistic and comprehensive approach to evaluation. We present a formal definition of reasoning capacity and illustrate its utility in identifying limitations within each component of the system. We then argue how these limitations can be addressed with a self-reflective process wherein human-feedback is used to alleviate shortcomings in reasoning and enhance overall consistency of the system.

Title: Vaccine: Perturbation-aware Alignment for Large Language Model

• Authors: Tiansheng Huang, Sihao Hu, Ling Liu
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01109
• Pdf URL: https://arxiv.org/pdf/2402.01109
• Copy Paste: [[2402.01109]] Vaccine: Perturbation-aware Alignment for Large Language Model(https://arxiv.org/abs/2402.01109)
• Keywords: security, attack, robust, large language model
• Abstract: The new paradigm of finetuning-as-a-service introduces a new attack surface for Large Language Models (LLMs): a few harmful data uploaded by users can easily trick the finetuning to produce an alignment-broken model. We conduct an empirical analysis and uncover a \textit{harmful embedding drift} phenomenon, showing a probable cause of the alignment-broken effect. Inspired by our findings, we propose Vaccine, a perturbation-aware alignment technique to mitigate the security risk of users finetuning. The core idea of Vaccine is to produce invariant hidden embeddings by progressively adding crafted perturbation to them in the alignment phase. This enables the embeddings to withstand harmful perturbation from un-sanitized user data in the finetuning phase. Our results on open source mainstream LLMs (e.g., Llama2, Opt, Vicuna) demonstrate that Vaccine can boost the robustness of alignment against harmful prompts induced embedding drift while reserving reasoning ability towards benign prompts. Our code is available at \url{https://github.com/git-disl/Vaccine}.

Title: Double-Dip: Thwarting Label-Only Membership Inference Attacks with Transfer Learning and Randomization

• Authors: Arezoo Rajabi, Reeya Pimple, Aiswarya Janardhanan, Surudhi Asokraj, Bhaskar Ramasubramanian, Radha Poovendran
• Subjects: cs.LG, cs.AI, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.01114
• Pdf URL: https://arxiv.org/pdf/2402.01114
• Copy Paste: [[2402.01114]] Double-Dip: Thwarting Label-Only Membership Inference Attacks with Transfer Learning and Randomization(https://arxiv.org/abs/2402.01114)
• Keywords: privacy, attack, membership infer
• Abstract: Transfer learning (TL) has been demonstrated to improve DNN model performance when faced with a scarcity of training samples. However, the suitability of TL as a solution to reduce vulnerability of overfitted DNNs to privacy attacks is unexplored. A class of privacy attacks called membership inference attacks (MIAs) aim to determine whether a given sample belongs to the training dataset (member) or not (nonmember). We introduce Double-Dip, a systematic empirical study investigating the use of TL (Stage-1) combined with randomization (Stage-2) to thwart MIAs on overfitted DNNs without degrading classification accuracy. Our study examines the roles of shared feature space and parameter values between source and target models, number of frozen layers, and complexity of pretrained models. We evaluate Double-Dip on three (Target, Source) dataset paris: (i) (CIFAR-10, ImageNet), (ii) (GTSRB, ImageNet), (iii) (CelebA, VGGFace2). We consider four publicly available pretrained DNNs: (a) VGG-19, (b) ResNet-18, (c) Swin-T, and (d) FaceNet. Our experiments demonstrate that Stage-1 reduces adversary success while also significantly increasing classification accuracy of nonmembers against an adversary with either white-box or black-box DNN model access, attempting to carry out SOTA label-only MIAs. After Stage-2, success of an adversary carrying out a label-only MIA is further reduced to near 50%, bringing it closer to a random guess and showing the effectiveness of Double-Dip. Stage-2 of Double-Dip also achieves lower ASR and higher classification accuracy than regularization and differential privacy-based methods.

Title: Interpretation of Intracardiac Electrograms Through Textual Representations

• Authors: William Jongwon Han, Diana Gomez, Avi Alok, Chaojing Duan, Michael A. Rosenberg, Douglas Weber, Emerson Liu, Ding Zhao
• Subjects: cs.CL, eess.SP
• Abstract URL: https://arxiv.org/abs/2402.01115
• Pdf URL: https://arxiv.org/pdf/2402.01115
• Copy Paste: [[2402.01115]] Interpretation of Intracardiac Electrograms Through Textual Representations(https://arxiv.org/abs/2402.01115)
• Keywords: interpretability
• Abstract: Understanding the irregular electrical activity of atrial fibrillation (AFib) has been a key challenge in electrocardiography. For serious cases of AFib, catheter ablations are performed to collect intracardiac electrograms (EGMs). EGMs offer intricately detailed and localized electrical activity of the heart and are an ideal modality for interpretable cardiac studies. Recent advancements in artificial intelligence (AI) has allowed some works to utilize deep learning frameworks to interpret EGMs during AFib. Additionally, language models (LMs) have shown exceptional performance in being able to generalize to unseen domains, especially in healthcare. In this study, we are the first to leverage pretrained LMs for finetuning of EGM interpolation and AFib classification via masked language modeling. We formulate the EGM as a textual sequence and present competitive performances on AFib classification compared against other representations. Lastly, we provide a comprehensive interpretability study to provide a multi-perspective intuition of the model's behavior, which could greatly benefit the clinical use.

Title: DTS-SQL: Decomposed Text-to-SQL with Small Large Language Models

• Authors: Mohammadreza Pourreza, Davood Rafiei
• Subjects: cs.CL, cs.DB, cs.HC
• Abstract URL: https://arxiv.org/abs/2402.01117
• Pdf URL: https://arxiv.org/pdf/2402.01117
• Copy Paste: [[2402.01117]] DTS-SQL: Decomposed Text-to-SQL with Small Large Language Models(https://arxiv.org/abs/2402.01117)
• Keywords: privacy, large language model
• Abstract: Leading models for the text-to-SQL task heavily rely on proprietary Large Language Models (LLMs), posing concerns over data privacy. Closing the performance gap between small open-source models and large proprietary models is crucial to mitigate this reliance. To this end, we introduce a novel two-stage fine-tuning approach that decomposes the task into two simpler tasks. Through comprehensive evaluation on two large cross-domain datasets and two small LLMs, we show that this approach improves execution accuracy by 3 to 7 percent, effectively aligning the performance of open-source models with their proprietary counterparts.

Title: A Single Simple Patch is All You Need for AI-generated Image Detection

• Authors: Jiaxuan Chen, Jieteng Yao, Li Niu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01123
• Pdf URL: https://arxiv.org/pdf/2402.01123
• Copy Paste: [[2402.01123]] A Single Simple Patch is All You Need for AI-generated Image Detection(https://arxiv.org/abs/2402.01123)
• Keywords: robust, generative
• Abstract: The recent development of generative models unleashes the potential of generating hyper-realistic fake images. To prevent the malicious usage of fake images, AI-generated image detection aims to distinguish fake images from real images. Nevertheless, existing methods usually suffer from poor generalizability across different generators. In this work, we propose an embarrassingly simple approach named SSP, i.e., feeding the noise pattern of a Single Simple Patch (SSP) to a binary classifier, which could achieve 14.6% relative improvement over the recent method on GenImage dataset. Our SSP method is very robust and generalizable, which could serve as a simple and competitive baseline for the future methods.

Title: Seeing Objects in a Cluttered World: Computational Objectness from Motion in Video

• Authors: Douglas Poland, Amar Saini
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01126
• Pdf URL: https://arxiv.org/pdf/2402.01126
• Copy Paste: [[2402.01126]] Seeing Objects in a Cluttered World: Computational Objectness from Motion in Video(https://arxiv.org/abs/2402.01126)
• Keywords: robust
• Abstract: Perception of the visually disjoint surfaces of our cluttered world as whole objects, physically distinct from those overlapping them, is a cognitive phenomenon called objectness that forms the basis of our visual perception. Shared by all vertebrates and present at birth in humans, it enables object-centric representation and reasoning about the visual world. We present a computational approach to objectness that leverages motion cues and spatio-temporal attention using a pair of supervised spatio-temporal R(2+1)U-Nets. The first network detects motion boundaries and classifies the pixels at those boundaries in terms of their local foreground-background sense. This motion boundary sense (MBS) information is passed, along with a spatio-temporal object attention cue, to an attentional surface perception (ASP) module which infers the form of the attended object over a sequence of frames and classifies its 'pixels' as visible or obscured. The spatial form of the attention cue is flexible, but it must loosely track the attended object which need not be visible. We demonstrate the ability of this simple but novel approach to infer objectness from phenomenology without object models, and show that it delivers robust perception of individual attended objects in cluttered scenes, even with blur and camera shake. We show that our data diversity and augmentation minimizes bias and facilitates transfer to real video. Finally, we describe how this computational objectness capability can grow in sophistication and anchor a robust modular video object perception framework.

Title: DeepAAT: Deep Automated Aerial Triangulation for Fast UAV-based Mapping

• Authors: Zequan Chen, Jianping Li, Qusheng Li, Bisheng Yang, Zhen Dong
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01134
• Pdf URL: https://arxiv.org/pdf/2402.01134
• Copy Paste: [[2402.01134]] DeepAAT: Deep Automated Aerial Triangulation for Fast UAV-based Mapping(https://arxiv.org/abs/2402.01134)
• Keywords: robust
• Abstract: Automated Aerial Triangulation (AAT), aiming to restore image pose and reconstruct sparse points simultaneously, plays a pivotal role in earth observation. With its rich research heritage spanning several decades in photogrammetry, AAT has evolved into a fundamental process widely applied in large-scale Unmanned Aerial Vehicle (UAV) based mapping. Despite its advancements, classic AAT methods still face challenges like low efficiency and limited robustness. This paper introduces DeepAAT, a deep learning network designed specifically for AAT of UAV imagery. DeepAAT considers both spatial and spectral characteristics of imagery, enhancing its capability to resolve erroneous matching pairs and accurately predict image poses. DeepAAT marks a significant leap in AAT's efficiency, ensuring thorough scene coverage and precision. Its processing speed outpaces incremental AAT methods by hundreds of times and global AAT methods by tens of times while maintaining a comparable level of reconstruction accuracy. Additionally, DeepAAT's scene clustering and merging strategy facilitate rapid localization and pose determination for large-scale UAV images, even under constrained computing resources. The experimental results demonstrate DeepAAT's substantial improvements over conventional AAT methods, highlighting its potential in the efficiency and accuracy of UAV-based 3D reconstruction tasks. To benefit the photogrammetry society, the code of DeepAAT will be released at: https://github.com/WHU-USI3DV/DeepAAT.

Title: Learning Network Representations with Disentangled Graph Auto-Encoder

• Authors: Di Fan, Chuanhou Gao
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.01143
• Pdf URL: https://arxiv.org/pdf/2402.01143
• Copy Paste: [[2402.01143]] Learning Network Representations with Disentangled Graph Auto-Encoder(https://arxiv.org/abs/2402.01143)
• Keywords: generative
• Abstract: The (variational) graph auto-encoder is extensively employed for learning representations of graph-structured data. However, the formation of real-world graphs is a complex and heterogeneous process influenced by latent factors. Existing encoders are fundamentally holistic, neglecting the entanglement of latent factors. This not only makes graph analysis tasks less effective but also makes it harder to understand and explain the representations. Learning disentangled graph representations with (variational) graph auto-encoder poses significant challenges, and remains largely unexplored in the existing literature. In this article, we introduce the Disentangled Graph Auto-Encoder (DGA) and Disentangled Variational Graph Auto-Encoder (DVGA), approaches that leverage generative models to learn disentangled representations. Specifically, we first design a disentangled graph convolutional network with multi-channel message-passing layers, as the encoder aggregating information related to each disentangled latent factor. Subsequently, a component-wise flow is applied to each channel to enhance the expressive capabilities of disentangled variational graph auto-encoder. Additionally, we design a factor-wise decoder, considering the characteristics of disentangled representations. In order to further enhance the independence among representations, we introduce independence constraints on mapping channels for different latent factors. Empirical experiments on both synthetic and real-world datasets show the superiority of our proposed method compared to several state-of-the-art baselines.

Title: Scale Equalization for Multi-Level Feature Fusion

• Authors: Bum Jun Kim, Sang Woo Kim
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01149
• Pdf URL: https://arxiv.org/pdf/2402.01149
• Copy Paste: [[2402.01149]] Scale Equalization for Multi-Level Feature Fusion(https://arxiv.org/abs/2402.01149)
• Keywords: segmentation
• Abstract: Deep neural networks have exhibited remarkable performance in a variety of computer vision fields, especially in semantic segmentation tasks. Their success is often attributed to multi-level feature fusion, which enables them to understand both global and local information from an image. However, we found that multi-level features from parallel branches are on different scales. The scale disequilibrium is a universal and unwanted flaw that leads to detrimental gradient descent, thereby degrading performance in semantic segmentation. We discover that scale disequilibrium is caused by bilinear upsampling, which is supported by both theoretical and empirical evidence. Based on this observation, we propose injecting scale equalizers to achieve scale equilibrium across multi-level features after bilinear upsampling. Our proposed scale equalizers are easy to implement, applicable to any architecture, hyperparameter-free, implementable without requiring extra computational cost, and guarantee scale equilibrium for any dataset. Experiments showed that adopting scale equalizers consistently improved the mIoU index across various target datasets, including ADE20K, PASCAL VOC 2012, and Cityscapes, as well as various decoder choices, including UPerHead, PSPHead, ASPPHead, SepASPPHead, and FCNHead.

Title: Towards Quantum-Safe Federated Learning via Homomorphic Encryption: Learning with Gradients

• Authors: Guangfeng Yan, Shanxiang Lyu, Hanxu Hou, Zhiyong Zheng, Linqi Song
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.01154
• Pdf URL: https://arxiv.org/pdf/2402.01154
• Copy Paste: [[2402.01154]] Towards Quantum-Safe Federated Learning via Homomorphic Encryption: Learning with Gradients(https://arxiv.org/abs/2402.01154)
• Keywords: security, privacy, federate
• Abstract: This paper introduces a privacy-preserving distributed learning framework via private-key homomorphic encryption. Thanks to the randomness of the quantization of gradients, our learning with error (LWE) based encryption can eliminate the error terms, thus avoiding the issue of error expansion in conventional LWE-based homomorphic encryption. The proposed system allows a large number of learning participants to engage in neural network-based deep learning collaboratively over an honest-but-curious server, while ensuring the cryptographic security of participants' uploaded gradients.

Title: CABINET: Content Relevance based Noise Reduction for Table Question Answering

• Authors: Sohan Patnaik, Heril Changwal, Milan Aggarwal, Sumita Bhatia, Yaman Kumar, Balaji Krishnamurthy
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01155
• Pdf URL: https://arxiv.org/pdf/2402.01155
• Copy Paste: [[2402.01155]] CABINET: Content Relevance based Noise Reduction for Table Question Answering(https://arxiv.org/abs/2402.01155)
• Keywords: robust, large language model
• Abstract: Table understanding capability of Large Language Models (LLMs) has been extensively studied through the task of question-answering (QA) over tables. Typically, only a small part of the whole table is relevant to derive the answer for a given question. The irrelevant parts act as noise and are distracting information, resulting in sub-optimal performance due to the vulnerability of LLMs to noise. To mitigate this, we propose CABINET (Content RelevAnce-Based NoIse ReductioN for TablE QuesTion-Answering) - a framework to enable LLMs to focus on relevant tabular data by suppressing extraneous information. CABINET comprises an Unsupervised Relevance Scorer (URS), trained differentially with the QA LLM, that weighs the table content based on its relevance to the input question before feeding it to the question-answering LLM (QA LLM). To further aid the relevance scorer, CABINET employs a weakly supervised module that generates a parsing statement describing the criteria of rows and columns relevant to the question and highlights the content of corresponding table cells. CABINET significantly outperforms various tabular LLM baselines, as well as GPT3-based in-context learning methods, is more robust to noise, maintains outperformance on tables of varying sizes, and establishes new SoTA performance on WikiTQ, FeTaQA, and WikiSQL datasets. We release our code and datasets at https://github.com/Sohanpatnaik106/CABINET_QA.

Title: LLM-Detector: Improving AI-Generated Chinese Text Detection with Open-Source LLM Instruction Tuning

• Authors: Rongsheng Wang, Haoming Chen, Ruizhe Zhou, Han Ma, Yaofei Duan, Yanlan Kang, Songhua Yang, Baoyu Fan, Tao Tan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01158
• Pdf URL: https://arxiv.org/pdf/2402.01158
• Copy Paste: [[2402.01158]] LLM-Detector: Improving AI-Generated Chinese Text Detection with Open-Source LLM Instruction Tuning(https://arxiv.org/abs/2402.01158)
• Keywords: large language model
• Abstract: ChatGPT and other general large language models (LLMs) have achieved remarkable success, but they have also raised concerns about the misuse of AI-generated texts. Existing AI-generated text detection models, such as based on BERT and RoBERTa, are prone to in-domain over-fitting, leading to poor out-of-domain (OOD) detection performance. In this paper, we first collected Chinese text responses generated by human experts and 9 types of LLMs, for which to multiple domains questions, and further created a dataset that mixed human-written sentences and sentences polished by LLMs. We then proposed LLM-Detector, a novel method for both document-level and sentence-level text detection through Instruction Tuning of LLMs. Our method leverages the wealth of knowledge LLMs acquire during pre-training, enabling them to detect the text they generate. Instruction tuning aligns the model's responses with the user's expected text detection tasks. Experimental results show that previous methods struggle with sentence-level AI-generated text detection and OOD detection. In contrast, our proposed method not only significantly outperforms baseline methods in both sentence-level and document-level text detection but also demonstrates strong generalization capabilities. Furthermore, since LLM-Detector is trained based on open-source LLMs, it is easy to customize for deployment.

Title: A Comprehensive Survey on 3D Content Generation

• Authors: Jian Liu, Xiaoshui Huang, Tianyu Huang, Lu Chen, Yuenan Hou, Shixiang Tang, Ziwei Liu, Wanli Ouyang, Wangmeng Zuo, Junjun Jiang, Xianming Liu
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01166
• Pdf URL: https://arxiv.org/pdf/2402.01166
• Copy Paste: [[2402.01166]] A Comprehensive Survey on 3D Content Generation(https://arxiv.org/abs/2402.01166)
• Keywords: generative
• Abstract: Recent years have witnessed remarkable advances in artificial intelligence generated content(AIGC), with diverse input modalities, e.g., text, image, video, audio and 3D. The 3D is the most close visual modality to real-world 3D environment and carries enormous knowledge. The 3D content generation shows both academic and practical values while also presenting formidable technical challenges. This review aims to consolidate developments within the burgeoning domain of 3D content generation. Specifically, a new taxonomy is proposed that categorizes existing approaches into three types: 3D native generative methods, 2D prior-based 3D generative methods, and hybrid 3D generative methods. The survey covers approximately 60 papers spanning the major techniques. Besides, we discuss limitations of current 3D content generation techniques, and point out open challenges as well as promising directions for future work. Accompanied with this survey, we have established a project website where the resources on 3D content generation research are provided. The project page is available at https://github.com/hitcslj/Awesome-AIGC-3D.

Title: Faster Inference of Integer SWIN Transformer by Removing the GELU Activation

• Authors: Mohammadreza Tayaranian, Seyyed Hasan Mozafari, James J. Clark, Brett Meyer, Warren Gross
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01169
• Pdf URL: https://arxiv.org/pdf/2402.01169
• Copy Paste: [[2402.01169]] Faster Inference of Integer SWIN Transformer by Removing the GELU Activation(https://arxiv.org/abs/2402.01169)
• Keywords: transformer
• Abstract: SWIN transformer is a prominent vision transformer model that has state-of-the-art accuracy in image classification tasks. Despite this success, its unique architecture causes slower inference compared with similar deep neural networks. Integer quantization of the model is one of the methods used to improve its inference latency. However, state-of-the-art has not been able to fully quantize the model. In this work, we improve upon the inference latency of the state-of-the-art methods by removing the floating-point operations, which are associated with the GELU activation in Swin Transformer. While previous work proposed to replace the non-integer operations with linear approximation functions, we propose to replace GELU with ReLU activation. The advantage of ReLU over previous methods is its low memory and computation complexity. We use iterative knowledge distillation to compensate for the lost accuracy due to replacing GELU with ReLU. We quantize our GELU-less SWIN transformer and show that on an RTX 4090 NVIDIA GPU we can improve the inference latency of the quantized SWIN transformer by at least $11\%$ while maintaining an accuracy drop of under $0.5\%$ on the ImageNet evaluation dataset.

Title: Streaming Sequence Transduction through Dynamic Compression

• Authors: Weiting Tan, Yunmo Chen, Tongfei Chen, Guanghui Qin, Haoran Xu, Heidi C. Zhang, Benjamin Van Durme, Philipp Koehn
• Subjects: cs.CL, cs.SD, eess.AS
• Abstract URL: https://arxiv.org/abs/2402.01172
• Pdf URL: https://arxiv.org/pdf/2402.01172
• Copy Paste: [[2402.01172]] Streaming Sequence Transduction through Dynamic Compression(https://arxiv.org/abs/2402.01172)
• Keywords: transformer, segmentation
• Abstract: We introduce STAR (Stream Transduction with Anchor Representations), a novel Transformer-based model designed for efficient sequence-to-sequence transduction over streams. STAR dynamically segments input streams to create compressed anchor representations, achieving nearly lossless compression (12x) in Automatic Speech Recognition (ASR) and outperforming existing methods. Moreover, STAR demonstrates superior segmentation and latency-quality trade-offs in simultaneous speech-to-text tasks, optimizing latency, memory footprint, and quality.

Title: Efficient Prompt Caching via Embedding Similarity

• Authors: Hanlin Zhu, Banghua Zhu, Jiantao Jiao
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.01173
• Pdf URL: https://arxiv.org/pdf/2402.01173
• Copy Paste: [[2402.01173]] Efficient Prompt Caching via Embedding Similarity(https://arxiv.org/abs/2402.01173)
• Keywords: large language model
• Abstract: Large language models (LLMs) have achieved huge success in numerous natural language process (NLP) tasks. However, it faces the challenge of significant resource consumption during inference. In this paper, we aim to improve the inference efficiency of LLMs by prompt caching, i.e., if the current prompt can be answered by the same response of a previous prompt, one can directly utilize that previous response without calling the LLM. Specifically, we focus on the prediction accuracy of prompt caching for single-round question-answering tasks via embedding similarity. The existing embeddings of prompts mostly focus on whether two prompts are semantically similar, which is not necessarily equivalent to whether the same response can answer them. Therefore, we propose a distillation-based method to fine-tune the existing embeddings for better caching prediction. Theoretically, we provide finite-sample guarantees for the convergence of our method under different types of loss functions. Empirically, we carefully construct a hard dataset based on Kwiatkowski et al. (2019) where the existing embedding model (Wang et al., 2022) only achieves an AUC of 0.51. We then fine-tune the above embedding model, which significantly improves the AUC of caching prediction from 0.51 to 0.81. We also conduct simulations demonstrating that our trained models achieve better caching efficiency than the previous embedding model.

Title: Towards a Unified Language Model for Knowledge-Intensive Tasks Utilizing External Corpus

• Authors: Xiaoxi Li, Zhicheng Dou, Yujia Zhou, Fangchao Liu
• Subjects: cs.CL, cs.IR
• Abstract URL: https://arxiv.org/abs/2402.01176
• Pdf URL: https://arxiv.org/pdf/2402.01176
• Copy Paste: [[2402.01176]] Towards a Unified Language Model for Knowledge-Intensive Tasks Utilizing External Corpus(https://arxiv.org/abs/2402.01176)
• Keywords: generative, large language model
• Abstract: The advent of large language models (LLMs) has showcased their efficacy across various domains, yet they often hallucinate, especially in knowledge-intensive tasks that require external knowledge sources. To improve factual accuracy of language models, retrieval-augmented generation (RAG) has emerged as a popular solution. However, traditional retrieval modules often rely on large-scale document indexes, which can be disconnected from generative tasks. Through generative retrieval (GR) approach, language models can achieve superior retrieval performance by directly generating relevant document identifiers (DocIDs). However, the relationship between GR and downstream tasks, as well as the potential of LLMs in GR, remains unexplored. In this paper, we present a unified language model that utilizes external corpus to handle various knowledge-intensive tasks by seamlessly integrating generative retrieval, closed-book generation, and RAG. In order to achieve effective retrieval and generation through a unified continuous decoding process, we introduce the following mechanisms: (1) a ranking-oriented DocID decoding strategy, which improves ranking ability by directly learning from a DocID ranking list; (2) a continuous generation strategy to facilitate effective and efficient RAG; (3) well-designed auxiliary DocID understanding tasks to enhance the model's comprehension of DocIDs and their relevance to downstream tasks. Our approach is evaluated on the widely used KILT benchmark using two variants of backbone models: an encoder-decoder T5 model and a decoder-only LLM, Llama2. Experimental results showcase the superior performance of our models in both retrieval and downstream knowledge-intensive tasks.

Title: DeepBranchTracer: A Generally-Applicable Approach to Curvilinear Structure Reconstruction Using Multi-Feature Learning

• Authors: Chao Liu, Ting Zhao, Nenggan Zheng
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01187
• Pdf URL: https://arxiv.org/pdf/2402.01187
• Copy Paste: [[2402.01187]] DeepBranchTracer: A Generally-Applicable Approach to Curvilinear Structure Reconstruction Using Multi-Feature Learning(https://arxiv.org/abs/2402.01187)
• Keywords: extraction, segmentation
• Abstract: Curvilinear structures, which include line-like continuous objects, are fundamental geometrical elements in image-based applications. Reconstructing these structures from images constitutes a pivotal research area in computer vision. However, the complex topology and ambiguous image evidence render this process a challenging task. In this paper, we introduce DeepBranchTracer, a novel method that learns both external image features and internal geometric characteristics to reconstruct curvilinear structures. Firstly, we formulate the curvilinear structures extraction as a geometric attribute estimation problem. Then, a curvilinear structure feature learning network is designed to extract essential branch attributes, including the image features of centerline and boundary, and the geometric features of direction and radius. Finally, utilizing a multi-feature fusion tracing strategy, our model iteratively traces the entire branch by integrating the extracted image and geometric features. We extensively evaluated our model on both 2D and 3D datasets, demonstrating its superior performance over existing segmentation and reconstruction methods in terms of accuracy and continuity.

Title: Segment Any Change

• Authors: Zhuo Zheng, Yanfei Zhong, Liangpei Zhang, Stefano Ermon
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01188
• Pdf URL: https://arxiv.org/pdf/2402.01188
• Copy Paste: [[2402.01188]] Segment Any Change(https://arxiv.org/abs/2402.01188)
• Keywords: segmentation
• Abstract: Visual foundation models have achieved remarkable results in zero-shot image classification and segmentation, but zero-shot change detection remains an open problem. In this paper, we propose the segment any change models (AnyChange), a new type of change detection model that supports zero-shot prediction and generalization on unseen change types and data distributions. AnyChange is built on the segment anything model (SAM) via our training-free adaptation method, bitemporal latent matching. By revealing and exploiting intra-image and inter-image semantic similarities in SAM's latent space, bitemporal latent matching endows SAM with zero-shot change detection capabilities in a training-free way. We also propose a point query mechanism to enable AnyChange's zero-shot object-centric change detection capability. We perform extensive experiments to confirm the effectiveness of AnyChange for zero-shot change detection. AnyChange sets a new record on the SECOND benchmark for unsupervised change detection, exceeding the previous SOTA by up to 4.4% F$_1$ score, and achieving comparable accuracy with negligible manual annotations (1 pixel per image) for supervised change detection.

Title: Unsupervised Generation of Pseudo Normal PET from MRI with Diffusion Model for Epileptic Focus Localization

• Authors: Wentao Chen, Jiwei Li, Xichen Xu, Hui Huang, Siyu Yuan, Miao Zhang, Tianming Xu, Jie Luo, Weimin Zhou
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01191
• Pdf URL: https://arxiv.org/pdf/2402.01191
• Copy Paste: [[2402.01191]] Unsupervised Generation of Pseudo Normal PET from MRI with Diffusion Model for Epileptic Focus Localization(https://arxiv.org/abs/2402.01191)
• Keywords: diffusion
• Abstract: [$^{18}$F]fluorodeoxyglucose (FDG) positron emission tomography (PET) has emerged as a crucial tool in identifying the epileptic focus, especially in cases where magnetic resonance imaging (MRI) diagnosis yields indeterminate results. FDG PET can provide the metabolic information of glucose and help identify abnormal areas that are not easily found through MRI. However, the effectiveness of FDG PET-based assessment and diagnosis depends on the selection of a healthy control group. The healthy control group typically consists of healthy individuals similar to epilepsy patients in terms of age, gender, and other aspects for providing normal FDG PET data, which will be used as a reference for enhancing the accuracy and reliability of the epilepsy diagnosis. However, significant challenges arise when a healthy PET control group is unattainable. Yaakub \emph{et al.} have previously introduced a Pix2PixGAN-based method for MRI to PET translation. This method used paired MRI and FDG PET scans from healthy individuals for training, and produced pseudo normal FDG PET images from patient MRIs that are subsequently used for lesion detection. However, this approach requires a large amount of high-quality, paired MRI and PET images from healthy control subjects, which may not always be available. In this study, we investigated unsupervised learning methods for unpaired MRI to PET translation for generating pseudo normal FDG PET for epileptic focus localization. Two deep learning methods, CycleGAN and SynDiff, were employed, and we found that diffusion-based method achieved improved performance in accurately localizing the epileptic focus.

Title: Conditional Normalizing Flows for Active Learning of Coarse-Grained Molecular Representations

• Authors: Henrik Schopmans, Pascal Friederich
• Subjects: cs.LG, cs.AI, physics.chem-ph
• Abstract URL: https://arxiv.org/abs/2402.01195
• Pdf URL: https://arxiv.org/pdf/2402.01195
• Copy Paste: [[2402.01195]] Conditional Normalizing Flows for Active Learning of Coarse-Grained Molecular Representations(https://arxiv.org/abs/2402.01195)
• Keywords: generative
• Abstract: Efficient sampling of the Boltzmann distribution of molecular systems is a long-standing challenge. Recently, instead of generating long molecular dynamics simulations, generative machine learning methods such as normalizing flows have been used to learn the Boltzmann distribution directly, without samples. However, this approach is susceptible to mode collapse and thus often does not explore the full configurational space. In this work, we address this challenge by separating the problem into two levels, the fine-grained and coarse-grained degrees of freedom. A normalizing flow conditioned on the coarse-grained space yields a probabilistic connection between the two levels. To explore the configurational space, we employ coarse-grained simulations with active learning which allows us to update the flow and make all-atom potential energy evaluations only when necessary. Using alanine dipeptide as an example, we show that our methods obtain a speedup to molecular dynamics simulations of approximately 15.9 to 216.2 compared to the speedup of 4.5 of the current state-of-the-art machine learning approach.

Title: Structured World Modeling via Semantic Vector Quantization

• Authors: Yi-Fu Wu, Minseung Lee, Sungjin Ahn
• Subjects: cs.LG, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01203
• Pdf URL: https://arxiv.org/pdf/2402.01203
• Copy Paste: [[2402.01203]] Structured World Modeling via Semantic Vector Quantization(https://arxiv.org/abs/2402.01203)
• Keywords: generative
• Abstract: Neural discrete representations are crucial components of modern neural networks. However, their main limitation is that the primary strategies such as VQ-VAE can only provide representations at the patch level. Therefore, one of the main goals of representation learning, acquiring structured, semantic, and compositional abstractions such as the color and shape of an object, remains elusive. In this paper, we present the first approach to semantic neural discrete representation learning. The proposed model, called Semantic Vector-Quantized Variational Autoencoder (SVQ), leverages recent advances in unsupervised object-centric learning to address this limitation. Specifically, we observe that a simple approach quantizing at the object level poses a significant challenge and propose constructing scene representations hierarchically, from low-level discrete concept schemas to object representations. Additionally, we suggest a novel method for structured semantic world modeling by training a prior over these representations, enabling the ability to generate images by sampling the semantic properties of the objects in the scene. In experiments on various 2D and 3D object-centric datasets, we find that our model achieves superior generation performance compared to non-semantic vector quantization methods such as VQ-VAE and previous object-centric generative models. Furthermore, we find that the semantic discrete representations can solve downstream scene understanding tasks that require reasoning about the properties of different objects in the scene.

Title: A Survey on Self-Supervised Learning for Non-Sequential Tabular Data

• Authors: Wei-Yao Wang, Wei-Wei Du, Derek Xu, Wei Wang, Wen-Chih Peng
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01204
• Pdf URL: https://arxiv.org/pdf/2402.01204
• Copy Paste: [[2402.01204]] A Survey on Self-Supervised Learning for Non-Sequential Tabular Data(https://arxiv.org/abs/2402.01204)
• Keywords: robust
• Abstract: Self-supervised learning (SSL) has been incorporated into many state-of-the-art models in various domains, where SSL defines pretext tasks based on unlabeled datasets to learn contextualized and robust representations. Recently, SSL has been a new trend in exploring the representation learning capability in the realm of tabular data, which is more challenging due to not having explicit relations for learning descriptive representations. This survey aims to systematically review and summarize the recent progress and challenges of SSL for non-sequential tabular data (SSL4NS-TD). We first present a formal definition of NS-TD and clarify its correlation to related studies. Then, these approaches are categorized into three groups -- predictive learning, contrastive learning, and hybrid learning, with their motivations and strengths of representative methods within each direction. On top of this, application issues of SSL4NS-TD are presented, including automatic data engineering, cross-table transferability, and domain knowledge integration. In addition, we elaborate on existing benchmarks and datasets for NS-TD applications to discuss the performance of existing tabular models. Finally, we discuss the challenges of SSL4NS-TD and provide potential directions for future research. We expect our work to be useful in terms of encouraging more research on lowering the barrier to entry SSL for the tabular domain and improving the foundations for implicit tabular data.

Title: Efficient Causal Graph Discovery Using Large Language Models

• Authors: Thomas Jiralerspong, Xiaoyin Chen, Yash More, Vedant Shah, Yoshua Bengio
• Subjects: cs.LG, cs.AI, stat.ME
• Abstract URL: https://arxiv.org/abs/2402.01207
• Pdf URL: https://arxiv.org/pdf/2402.01207
• Copy Paste: [[2402.01207]] Efficient Causal Graph Discovery Using Large Language Models(https://arxiv.org/abs/2402.01207)
• Keywords: large language model
• Abstract: We propose a novel framework that leverages LLMs for full causal graph discovery. While previous LLM-based methods have used a pairwise query approach, this requires a quadratic number of queries which quickly becomes impractical for larger causal graphs. In contrast, the proposed framework uses a breadth-first search (BFS) approach which allows it to use only a linear number of queries. We also show that the proposed method can easily incorporate observational data when available, to improve performance. In addition to being more time and data-efficient, the proposed framework achieves state-of-the-art results on real-world causal graphs of varying sizes. The results demonstrate the effectiveness and efficiency of the proposed method in discovering causal relationships, showcasing its potential for broad applicability in causal graph discovery tasks across different domains.

Title: TSJNet: A Multi-modality Target and Semantic Awareness Joint-driven Image Fusion Network

• Authors: Yuchan Jie, Yushen Xu, Xiaosong Li, Haishu Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01212
• Pdf URL: https://arxiv.org/pdf/2402.01212
• Copy Paste: [[2402.01212]] TSJNet: A Multi-modality Target and Semantic Awareness Joint-driven Image Fusion Network(https://arxiv.org/abs/2402.01212)
• Keywords: extraction, segmentation
• Abstract: Multi-modality image fusion involves integrating complementary information from different modalities into a single image. Current methods primarily focus on enhancing image fusion with a single advanced task such as incorporating semantic or object-related information into the fusion process. This method creates challenges in achieving multiple objectives simultaneously. We introduce a target and semantic awareness joint-driven fusion network called TSJNet. TSJNet comprises fusion, detection, and segmentation subnetworks arranged in a series structure. It leverages object and semantically relevant information derived from dual high-level tasks to guide the fusion network. Additionally, We propose a local significant feature extraction module with a double parallel branch structure to fully capture the fine-grained features of cross-modal images and foster interaction among modalities, targets, and segmentation information. We conducted extensive experiments on four publicly available datasets (MSRS, M3FD, RoadScene, and LLVIP). The results demonstrate that TSJNet can generate visually pleasing fused results, achieving an average increase of 2.84% and 7.47% in object detection and segmentation mAP @0.5 and mIoU, respectively, compared to the state-of-the-art methods.

Title: Taming Uncertainty in Sparse-view Generalizable NeRF via Indirect Diffusion Guidance

• Authors: Yaokun Li, Chao Gou, Guang Tan
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01217
• Pdf URL: https://arxiv.org/pdf/2402.01217
• Copy Paste: [[2402.01217]] Taming Uncertainty in Sparse-view Generalizable NeRF via Indirect Diffusion Guidance(https://arxiv.org/abs/2402.01217)
• Keywords: diffusion, generative
• Abstract: Neural Radiance Fields (NeRF) have demonstrated effectiveness in synthesizing novel views. However, their reliance on dense inputs and scene-specific optimization has limited their broader applicability. Generalizable NeRFs (Gen-NeRF), while intended to address this, often produce blurring artifacts in unobserved regions with sparse inputs, which are full of uncertainty. In this paper, we aim to diminish the uncertainty in Gen-NeRF for plausible renderings. We assume that NeRF's inability to effectively mitigate this uncertainty stems from its inherent lack of generative capacity. Therefore, we innovatively propose an Indirect Diffusion-guided NeRF framework, termed ID-NeRF, to address this uncertainty from a generative perspective by leveraging a distilled diffusion prior as guidance. Specifically, to avoid model confusion caused by directly regularizing with inconsistent samplings as in previous methods, our approach introduces a strategy to indirectly inject the inherently missing imagination into the learned implicit function through a diffusion-guided latent space. Empirical evaluation across various benchmarks demonstrates the superior performance of our approach in handling uncertainty with sparse inputs.

Title: AI Code Generators for Security: Friend or Foe?

• Authors: Roberto Natella, Pietro Liguori, Cristina Improta, Bojan Cukic, Domenico Cotroneo
• Subjects: cs.CR, cs.AI, cs.SE
• Abstract URL: https://arxiv.org/abs/2402.01219
• Pdf URL: https://arxiv.org/pdf/2402.01219
• Copy Paste: [[2402.01219]] AI Code Generators for Security: Friend or Foe?(https://arxiv.org/abs/2402.01219)
• Keywords: security
• Abstract: Recent advances of artificial intelligence (AI) code generators are opening new opportunities in software security research, including misuse by malicious actors. We review use cases for AI code generators for security and introduce an evaluation benchmark.

Title: Delving into Decision-based Black-box Attacks on Semantic Segmentation

• Authors: Zhaoyu Chen, Zhengyang Shan, Jingwen Chang, Kaixun Jiang, Dingkang Yang, Yiting Cheng, Wenqiang Zhang
• Subjects: cs.CV, cs.CR
• Abstract URL: https://arxiv.org/abs/2402.01220
• Pdf URL: https://arxiv.org/pdf/2402.01220
• Copy Paste: [[2402.01220]] Delving into Decision-based Black-box Attacks on Semantic Segmentation(https://arxiv.org/abs/2402.01220)
• Keywords: security, attack, robust, segmentation
• Abstract: Semantic segmentation is a fundamental visual task that finds extensive deployment in applications with security-sensitive considerations. Nonetheless, recent work illustrates the adversarial vulnerability of semantic segmentation models to white-box attacks. However, its adversarial robustness against black-box attacks has not been fully explored. In this paper, we present the first exploration of black-box decision-based attacks on semantic segmentation. First, we analyze the challenges that semantic segmentation brings to decision-based attacks through the case study. Then, to address these challenges, we first propose a decision-based attack on semantic segmentation, called Discrete Linear Attack (DLA). Based on random search and proxy index, we utilize the discrete linear noises for perturbation exploration and calibration to achieve efficient attack efficiency. We conduct adversarial robustness evaluation on 5 models from Cityscapes and ADE20K under 8 attacks. DLA shows its formidable power on Cityscapes by dramatically reducing PSPNet's mIoU from an impressive 77.83% to a mere 2.14% with just 50 queries.

Title: HW-SW Optimization of DNNs for Privacy-preserving People Counting on Low-resolution Infrared Arrays

• Authors: Matteo Risso, Chen Xie, Francesco Daghero, Alessio Burrello, Seyedmorteza Mollaei, Marco Castellano, Enrico Macii, Massimo Poncino, Daniele Jahier Pagliari
• Subjects: cs.LG, cs.AR
• Abstract URL: https://arxiv.org/abs/2402.01226
• Pdf URL: https://arxiv.org/pdf/2402.01226
• Copy Paste: [[2402.01226]] HW-SW Optimization of DNNs for Privacy-preserving People Counting on Low-resolution Infrared Arrays(https://arxiv.org/abs/2402.01226)
• Keywords: privacy
• Abstract: Low-resolution infrared (IR) array sensors enable people counting applications such as monitoring the occupancy of spaces and people flows while preserving privacy and minimizing energy consumption. Deep Neural Networks (DNNs) have been shown to be well-suited to process these sensor data in an accurate and efficient manner. Nevertheless, the space of DNNs' architectures is huge and its manual exploration is burdensome and often leads to sub-optimal solutions. To overcome this problem, in this work, we propose a highly automated full-stack optimization flow for DNNs that goes from neural architecture search, mixed-precision quantization, and post-processing, down to the realization of a new smart sensor prototype, including a Microcontroller with a customized instruction set. Integrating these cross-layer optimizations, we obtain a large set of Pareto-optimal solutions in the 3D-space of energy, memory, and accuracy. Deploying such solutions on our hardware platform, we improve the state-of-the-art achieving up to 4.2x model size reduction, 23.8x code size reduction, and 15.38x energy reduction at iso-accuracy.

Title: Flexible Variational Information Bottleneck: Achieving Diverse Compression with a Single Training

• Authors: Sota Kudo, Naoaki Ono, Shigehiko Kanaya, Ming Huang
• Subjects: cs.LG, cs.AI, cs.IT
• Abstract URL: https://arxiv.org/abs/2402.01238
• Pdf URL: https://arxiv.org/pdf/2402.01238
• Copy Paste: [[2402.01238]] Flexible Variational Information Bottleneck: Achieving Diverse Compression with a Single Training(https://arxiv.org/abs/2402.01238)
• Keywords: extraction
• Abstract: Information Bottleneck (IB) is a widely used framework that enables the extraction of information related to a target random variable from a source random variable. In the objective function, IB controls the trade-off between data compression and predictiveness through the Lagrange multiplier $\beta$. Traditionally, to find the trade-off to be learned, IB requires a search for $\beta$ through multiple training cycles, which is computationally expensive. In this study, we introduce Flexible Variational Information Bottleneck (FVIB), an innovative framework for classification task that can obtain optimal models for all values of $\beta$ with single, computationally efficient training. We theoretically demonstrate that across all values of reasonable $\beta$, FVIB can simultaneously maximize an approximation of the objective function for Variational Information Bottleneck (VIB), the conventional IB method. Then we empirically show that FVIB can learn the VIB objective as effectively as VIB. Furthermore, in terms of calibration performance, FVIB outperforms other IB and calibration methods by enabling continuous optimization of $\beta$. Our codes are available at https://github.com/sotakudo/fvib.

Title: PRIME: Protect Your Videos From Malicious Editing

• Authors: Guanlin Li, Shuai Yang, Jie Zhang, Tianwei Zhang
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01239
• Pdf URL: https://arxiv.org/pdf/2402.01239
• Copy Paste: [[2402.01239]] PRIME: Protect Your Videos From Malicious Editing(https://arxiv.org/abs/2402.01239)
• Keywords: privacy, protect, generative
• Abstract: With the development of generative models, the quality of generated content keeps increasing. Recently, open-source models have made it surprisingly easy to manipulate and edit photos and videos, with just a few simple prompts. While these cutting-edge technologies have gained popularity, they have also given rise to concerns regarding the privacy and portrait rights of individuals. Malicious users can exploit these tools for deceptive or illegal purposes. Although some previous works focus on protecting photos against generative models, we find there are still gaps between protecting videos and images in the aspects of efficiency and effectiveness. Therefore, we introduce our protection method, PRIME, to significantly reduce the time cost and improve the protection performance. Moreover, to evaluate our proposed protection method, we consider both objective metrics and human subjective metrics. Our evaluation results indicate that PRIME only costs 8.3% GPU hours of the cost of the previous state-of-the-art method and achieves better protection results on both human evaluation and objective metrics. Code can be found in https://github.com/GuanlinLee/prime.

Title: Beyond the Request: Harnessing HTTP Response Headers for Cross-Browser Web Tracker Classification in an Imbalanced Setting

• Authors: Wolf Rieder, Philip Raschke, Thomas Cory
• Subjects: cs.CR, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.01240
• Pdf URL: https://arxiv.org/pdf/2402.01240
• Copy Paste: [[2402.01240]] Beyond the Request: Harnessing HTTP Response Headers for Cross-Browser Web Tracker Classification in an Imbalanced Setting(https://arxiv.org/abs/2402.01240)
• Keywords: security, privacy
• Abstract: The World Wide Web's connectivity is greatly attributed to the HTTP protocol, with HTTP messages offering informative header fields that appeal to disciplines like web security and privacy, especially concerning web tracking. Despite existing research employing HTTP/S request messages to identify web trackers, HTTP/S response headers are often overlooked. This study endeavors to design effective machine learning classifiers for web tracker detection using HTTP/S response headers. Data from the Chrome, Firefox, and Brave browsers, obtained through the traffic monitoring browser extension T.EX, serves as our data set. Eleven supervised models were trained on Chrome data and tested across all browsers. The results demonstrated high accuracy, F1-score, precision, recall, and minimal log-loss error for Chrome and Firefox, but subpar performance on Brave, potentially due to its distinct data distribution and feature set. The research suggests that these classifiers are viable for detecting web trackers in Chrome and Firefox. However, real-world application testing remains pending, and the distinction between tracker types and broader label sources could be explored in future studies.

Title: Can Shape-Infused Joint Embeddings Improve Image-Conditioned 3D Diffusion?

• Authors: Cristian Sbrolli, Paolo Cudrano, Matteo Matteucci
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01241
• Pdf URL: https://arxiv.org/pdf/2402.01241
• Copy Paste: [[2402.01241]] Can Shape-Infused Joint Embeddings Improve Image-Conditioned 3D Diffusion?(https://arxiv.org/abs/2402.01241)
• Keywords: diffusion, generative
• Abstract: Recent advancements in deep generative models, particularly with the application of CLIP (Contrastive Language Image Pretraining) to Denoising Diffusion Probabilistic Models (DDPMs), have demonstrated remarkable effectiveness in text to image generation. The well structured embedding space of CLIP has also been extended to image to shape generation with DDPMs, yielding notable results. Despite these successes, some fundamental questions arise: Does CLIP ensure the best results in shape generation from images? Can we leverage conditioning to bring explicit 3D knowledge into the generative process and obtain better quality? This study introduces CISP (Contrastive Image Shape Pre training), designed to enhance 3D shape synthesis guided by 2D images. CISP aims to enrich the CLIP framework by aligning 2D images with 3D shapes in a shared embedding space, specifically capturing 3D characteristics potentially overlooked by CLIP's text image focus. Our comprehensive analysis assesses CISP's guidance performance against CLIP guided models, focusing on generation quality, diversity, and coherence of the produced shapes with the conditioning image. We find that, while matching CLIP in generation quality and diversity, CISP substantially improves coherence with input images, underscoring the value of incorporating 3D knowledge into generative models. These findings suggest a promising direction for advancing the synthesis of 3D visual content by integrating multimodal systems with 3D representations.

Title: Two Heads Are Better Than One: Boosting Graph Sparse Training via Semantic and Topological Awareness

• Authors: Guibin Zhang, Yanwei Yue, Kun Wang, Junfeng Fang, Yongduo Sui, Kai Wang, Yuxuan Liang, Dawei Cheng, Shirui Pan, Tianlong Chen
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.01242
• Pdf URL: https://arxiv.org/pdf/2402.01242
• Copy Paste: [[2402.01242]] Two Heads Are Better Than One: Boosting Graph Sparse Training via Semantic and Topological Awareness(https://arxiv.org/abs/2402.01242)
• Keywords: defense
• Abstract: Graph Neural Networks (GNNs) excel in various graph learning tasks but face computational challenges when applied to large-scale graphs. A promising solution is to remove non-essential edges to reduce the computational overheads in GNN. Previous literature generally falls into two categories: topology-guided and semantic-guided. The former maintains certain graph topological properties yet often underperforms on GNNs due to low integration with neural network training. The latter performs well at lower sparsity on GNNs but faces performance collapse at higher sparsity levels. With this in mind, we take the first step to propose a new research line and concept termed Graph Sparse Training (GST), which dynamically manipulates sparsity at the data level. Specifically, GST initially constructs a topology & semantic anchor at a low training cost, followed by performing dynamic sparse training to align the sparse graph with the anchor. We introduce the Equilibria Sparsification Principle to guide this process, effectively balancing the preservation of both topological and semantic information. Ultimately, GST produces a sparse graph with maximum topological integrity and no performance degradation. Extensive experiments on 6 datasets and 5 backbones showcase that GST (I) identifies subgraphs at higher graph sparsity levels (1.67%~15.85% $\uparrow$) than state-of-the-art sparsification methods, (II) preserves more key spectral properties, (III) achieves 1.27-3.42$\times$ speedup in GNN inference and (IV) successfully helps graph adversarial defense and graph lottery tickets.

Title: Spectrum-guided Feature Enhancement Network for Event Person Re-Identification

• Authors: Hongchen Tan, Yi Zhang, Xiuping Liu, Baocai Yin, Nan Ma, Xin Li, Huchuan Lu
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01269
• Pdf URL: https://arxiv.org/pdf/2402.01269
• Copy Paste: [[2402.01269]] Spectrum-guided Feature Enhancement Network for Event Person Re-Identification(https://arxiv.org/abs/2402.01269)
• Keywords: privacy, robust
• Abstract: As a cutting-edge biosensor, the event camera holds significant potential in the field of computer vision, particularly regarding privacy preservation. However, compared to traditional cameras, event streams often contain noise and possess extremely sparse semantics, posing a formidable challenge for event-based person re-identification (event Re-ID). To address this, we introduce a novel event person re-identification network: the Spectrum-guided Feature Enhancement Network (SFE-Net). This network consists of two innovative components: the Multi-grain Spectrum Attention Mechanism (MSAM) and the Consecutive Patch Dropout Module (CPDM). MSAM employs a fourier spectrum transform strategy to filter event noise, while also utilizing an event-guided multi-granularity attention strategy to enhance and capture discriminative person semantics. CPDM employs a consecutive patch dropout strategy to generate multiple incomplete feature maps, encouraging the deep Re-ID model to equally perceive each effective region of the person's body and capture robust person descriptors. Extensive experiments on Event Re-ID datasets demonstrate that our SFE-Net achieves the best performance in this task.

Title: Can MLLMs Perform Text-to-Image In-Context Learning?

• Authors: Yuchen Zeng, Wonjun Kang, Yicong Chen, Hyung Il Koo, Kangwook Lee
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01293
• Pdf URL: https://arxiv.org/pdf/2402.01293
• Copy Paste: [[2402.01293]] Can MLLMs Perform Text-to-Image In-Context Learning?(https://arxiv.org/abs/2402.01293)
• Keywords: large language model
• Abstract: The evolution from Large Language Models (LLMs) to Multimodal Large Language Models (MLLMs) has spurred research into extending In-Context Learning (ICL) to its multimodal counterpart. Existing such studies have primarily concentrated on image-to-text ICL. However, the Text-to-Image ICL (T2I-ICL), with its unique characteristics and potential applications, remains underexplored. To address this gap, we formally define the task of T2I-ICL and present CoBSAT, the first T2I-ICL benchmark dataset, encompassing ten tasks. Utilizing our dataset to benchmark six state-of-the-art MLLMs, we uncover considerable difficulties MLLMs encounter in solving T2I-ICL. We identify the primary challenges as the inherent complexity of multimodality and image generation. To overcome these challenges, we explore strategies like fine-tuning and Chain-of-Thought prompting, demonstrating notable improvements. Our code and dataset are available at \url{https://github.com/UW-Madison-Lee-Lab/CoBSAT}.

Title: ExtremeCast: Boosting Extreme Value Prediction for Global Weather Forecast

• Authors: Wanghan Xu, Kang Chen, Tao Han, Hao Chen, Wanli Ouyang, Lei Bai
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01295
• Pdf URL: https://arxiv.org/pdf/2402.01295
• Copy Paste: [[2402.01295]] ExtremeCast: Boosting Extreme Value Prediction for Global Weather Forecast(https://arxiv.org/abs/2402.01295)
• Keywords: robust
• Abstract: Data-driven weather forecast based on machine learning (ML) has experienced rapid development and demonstrated superior performance in the global medium-range forecast compared to traditional physics-based dynamical models. However, most of these ML models struggle with accurately predicting extreme weather, which is closely related to the extreme value prediction. Through mathematical analysis, we prove that the use of symmetric losses, such as the Mean Squared Error (MSE), leads to biased predictions and underestimation of extreme values. To address this issue, we introduce Exloss, a novel loss function that performs asymmetric optimization and highlights extreme values to obtain accurate extreme weather forecast. Furthermore, we introduce a training-free extreme value enhancement strategy named ExEnsemble, which increases the variance of pixel values and improves the forecast robustness. Combined with an advanced global weather forecast model, extensive experiments show that our solution can achieve state-of-the-art performance in extreme weather prediction, while maintaining the overall forecast accuracy comparable to the top medium-range forecast models.

Title: Bi-CryptoNets: Leveraging Different-Level Privacy for Encrypted Inference

• Authors: Man-Jie Yuan, Zheng Zou, Wei Gao
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01296
• Pdf URL: https://arxiv.org/pdf/2402.01296
• Copy Paste: [[2402.01296]] Bi-CryptoNets: Leveraging Different-Level Privacy for Encrypted Inference(https://arxiv.org/abs/2402.01296)
• Keywords: security, privacy
• Abstract: Privacy-preserving neural networks have attracted increasing attention in recent years, and various algorithms have been developed to keep the balance between accuracy, computational complexity and information security from the cryptographic view. This work takes a different view from the input data and structure of neural networks. We decompose the input data (e.g., some images) into sensitive and insensitive segments according to importance and privacy. The sensitive segment includes some important and private information such as human faces and we take strong homomorphic encryption to keep security, whereas the insensitive one contains some background and we add perturbations. We propose the bi-CryptoNets, i.e., plaintext and ciphertext branches, to deal with two segments, respectively, and ciphertext branch could utilize the information from plaintext branch by unidirectional connections. We adopt knowledge distillation for our bi-CryptoNets by transferring representations from a well-trained teacher neural network. Empirical studies show the effectiveness and decrease of inference latency for our bi-CryptoNets.

Title: Two Approaches to Diachronic Normalization of Polish Texts

• Authors: Kacper Dudzic, Filip Graliński, Krzysztof Jassem, Marek Kubis, Piotr Wierzchoń
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01300
• Pdf URL: https://arxiv.org/pdf/2402.01300
• Copy Paste: [[2402.01300]] Two Approaches to Diachronic Normalization of Polish Texts(https://arxiv.org/abs/2402.01300)
• Keywords: transformer
• Abstract: This paper discusses two approaches to the diachronic normalization of Polish texts: a rule-based solution that relies on a set of handcrafted patterns, and a neural normalization model based on the text-to-text transfer transformer architecture. The training and evaluation data prepared for the task are discussed in detail, along with experiments conducted to compare the proposed normalization solutions. A quantitative and qualitative analysis is made. It is shown that at the current stage of inquiry into the problem, the rule-based solution outperforms the neural one on 3 out of 4 variants of the prepared dataset, although in practice both approaches have distinct advantages and disadvantages.

Title: Deep Multimodal Fusion of Data with Heterogeneous Dimensionality via Projective Networks

• Authors: José Morano, Guilherme Aresta, Christoph Grechenig, Ursula Schmidt-Erfurth, Hrvoje Bogunović
• Subjects: cs.CV, eess.IV
• Abstract URL: https://arxiv.org/abs/2402.01311
• Pdf URL: https://arxiv.org/pdf/2402.01311
• Copy Paste: [[2402.01311]] Deep Multimodal Fusion of Data with Heterogeneous Dimensionality via Projective Networks(https://arxiv.org/abs/2402.01311)
• Keywords: segmentation
• Abstract: The use of multimodal imaging has led to significant improvements in the diagnosis and treatment of many diseases. Similar to clinical practice, some works have demonstrated the benefits of multimodal fusion for automatic segmentation and classification using deep learning-based methods. However, current segmentation methods are limited to fusion of modalities with the same dimensionality (e.g., 3D+3D, 2D+2D), which is not always possible, and the fusion strategies implemented by classification methods are incompatible with localization tasks. In this work, we propose a novel deep learning-based framework for the fusion of multimodal data with heterogeneous dimensionality (e.g., 3D+2D) that is compatible with localization tasks. The proposed framework extracts the features of the different modalities and projects them into the common feature subspace. The projected features are then fused and further processed to obtain the final prediction. The framework was validated on the following tasks: segmentation of geographic atrophy (GA), a late-stage manifestation of age-related macular degeneration, and segmentation of retinal blood vessels (RBV) in multimodal retinal imaging. Our results show that the proposed method outperforms the state-of-the-art monomodal methods on GA and RBV segmentation by up to 3.10% and 4.64% Dice, respectively.

Title: Supervised Algorithmic Fairness in Distribution Shifts: A Survey

• Authors: Yujie Lin, Dong Li, Chen Zhao, Xintao Wu, Qin Tian, Minglai Shao
• Subjects: cs.LG, cs.AI, cs.CY
• Abstract URL: https://arxiv.org/abs/2402.01327
• Pdf URL: https://arxiv.org/pdf/2402.01327
• Copy Paste: [[2402.01327]] Supervised Algorithmic Fairness in Distribution Shifts: A Survey(https://arxiv.org/abs/2402.01327)
• Keywords: fair
• Abstract: Supervised fairness-aware machine learning under distribution shifts is an emerging field that addresses the challenge of maintaining equitable and unbiased predictions when faced with changes in data distributions from source to target domains. In real-world applications, machine learning models are often trained on a specific dataset but deployed in environments where the data distribution may shift over time due to various factors. This shift can lead to unfair predictions, disproportionately affecting certain groups characterized by sensitive attributes, such as race and gender. In this survey, we provide a summary of various types of distribution shifts and comprehensively investigate existing methods based on these shifts, highlighting six commonly used approaches in the literature. Additionally, this survey lists publicly available datasets and evaluation metrics for empirical studies. We further explore the interconnection with related research fields, discuss the significant challenges, and identify potential directions for future studies.

Title: Simulator-Free Visual Domain Randomization via Video Games

• Authors: Chintan Trivedi, Nemanja Rašajski, Konstantinos Makantasis, Antonios Liapis, Georgios N. Yannakakis
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01335
• Pdf URL: https://arxiv.org/pdf/2402.01335
• Copy Paste: [[2402.01335]] Simulator-Free Visual Domain Randomization via Video Games(https://arxiv.org/abs/2402.01335)
• Keywords: robust
• Abstract: Domain randomization is an effective computer vision technique for improving transferability of vision models across visually distinct domains exhibiting similar content. Existing approaches, however, rely extensively on tweaking complex and specialized simulation engines that are difficult to construct, subsequently affecting their feasibility and scalability. This paper introduces BehAVE, a video understanding framework that uniquely leverages the plethora of existing commercial video games for domain randomization, without requiring access to their simulation engines. Under BehAVE (1) the inherent rich visual diversity of video games acts as the source of randomization and (2) player behavior -- represented semantically via textual descriptions of actions -- guides the *alignment* of videos with similar content. We test BehAVE on 25 games of the first-person shooter (FPS) genre across various video and text foundation models and we report its robustness for domain randomization. BehAVE successfully aligns player behavioral patterns and is able to zero-shot transfer them to multiple unseen FPS games when trained on just one FPS game. In a more challenging setting, BehAVE manages to improve the zero-shot transferability of foundation models to unseen FPS games (up to 22%) even when trained on a game of a different genre (Minecraft). Code and dataset can be found at https://github.com/nrasajski/BehAVE.

Title: SignSGD with Federated Defense: Harnessing Adversarial Attacks through Gradient Sign Decoding

• Authors: Chanho Park, Namyoon Lee
• Subjects: cs.LG, cs.CR, eess.SP
• Abstract URL: https://arxiv.org/abs/2402.01340
• Pdf URL: https://arxiv.org/pdf/2402.01340
• Copy Paste: [[2402.01340]] SignSGD with Federated Defense: Harnessing Adversarial Attacks through Gradient Sign Decoding(https://arxiv.org/abs/2402.01340)
• Keywords: defense, attack, federate
• Abstract: Distributed learning is an effective approach to accelerate model training using multiple workers. However, substantial communication delays emerge between workers and a parameter server due to massive costs associated with communicating gradients. SignSGD with majority voting (signSGD-MV) is a simple yet effective optimizer that reduces communication costs through one-bit quantization, yet the convergence rates considerably decrease as adversarial workers increase. In this paper, we show that the convergence rate is invariant as the number of adversarial workers increases, provided that the number of adversarial workers is smaller than that of benign workers. The key idea showing this counter-intuitive result is our novel signSGD with federated defense (signSGD-FD). Unlike the traditional approaches, signSGD-FD exploits the gradient information sent by adversarial workers with the proper weights, which are obtained through gradient sign decoding. Experimental results demonstrate signSGD-FD achieves superior convergence rates over traditional algorithms in various adversarial attack scenarios.

Title: Training-time Neuron Alignment through Permutation Subspace for Improving Linear Mode Connectivity and Model Fusion

• Authors: Zexi Li, Zhiqi Li, Jie Lin, Tao Shen, Tao Lin, Chao Wu
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.01342
• Pdf URL: https://arxiv.org/pdf/2402.01342
• Copy Paste: [[2402.01342]] Training-time Neuron Alignment through Permutation Subspace for Improving Linear Mode Connectivity and Model Fusion(https://arxiv.org/abs/2402.01342)
• Keywords: federate, transformer
• Abstract: In deep learning, stochastic gradient descent often yields functionally similar yet widely scattered solutions in the weight space even under the same initialization, causing barriers in the Linear Mode Connectivity (LMC) landscape. Overcoming these barriers is crucial for understanding deep learning dynamics and enhancing model-fusion algorithms. Previous studies highlight the role of permutation symmetry in reducing post-training barriers through network permutation. However, these post-hoc methods, demanding extra computations, are less effective for larger, complex models (e.g., ViT, LLM) due to numerous permutation matrices. Thus, in this paper, we study training-time neuron alignment. Our hypothesis suggests that training-time permutation subspace can reduce LMC barriers for free. We find that pruning at initialization supports this. Beyond pruning, we introduce TNA-PFN, a simple yet lossless algorithm using a partial gradient mask during training. TNA-PFN is theoretically and empirically validated for reducing LMC barriers. It excels in wide model fusion applications, especially in federated learning, two algorithms based on TNA-FPN that are proposed to show its prospects even under heterogeneous datasets. Moreover, TNA-PFN can enhance the generalization of model soup for vision transformers and ColD fusion for pretrained language models.

Title: Shapelet-based Model-agnostic Counterfactual Local Explanations for Time Series Classification

• Authors: Qi Huang, Wei Chen, Thomas Bäck, Niki van Stein
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.01343
• Pdf URL: https://arxiv.org/pdf/2402.01343
• Copy Paste: [[2402.01343]] Shapelet-based Model-agnostic Counterfactual Local Explanations for Time Series Classification(https://arxiv.org/abs/2402.01343)
• Keywords: explainability
• Abstract: In this work, we propose a model-agnostic instance-based post-hoc explainability method for time series classification. The proposed algorithm, namely Time-CF, leverages shapelets and TimeGAN to provide counterfactual explanations for arbitrary time series classifiers. We validate the proposed method on several real-world univariate time series classification tasks from the UCR Time Series Archive. The results indicate that the counterfactual instances generated by Time-CF when compared to state-of-the-art methods, demonstrate better performance in terms of four explainability metrics: closeness, sensibility, plausibility, and sparsity.

Title: Beyond the Answers: Reviewing the Rationality of Multiple Choice Question Answering for the Evaluation of Large Language Models

• Authors: Haochun Wang, Sendong Zhao, Zewen Qiang, Bing Qin, Ting Liu
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01349
• Pdf URL: https://arxiv.org/pdf/2402.01349
• Copy Paste: [[2402.01349]] Beyond the Answers: Reviewing the Rationality of Multiple Choice Question Answering for the Evaluation of Large Language Models(https://arxiv.org/abs/2402.01349)
• Keywords: robust, large language model
• Abstract: In the field of natural language processing (NLP), Large Language Models (LLMs) have precipitated a paradigm shift, markedly enhancing performance in natural language generation tasks. Despite these advancements, the comprehensive evaluation of LLMs remains an inevitable challenge for the community. Recently, the utilization of Multiple Choice Question Answering (MCQA) as a benchmark for LLMs has gained considerable traction. This study investigates the rationality of MCQA as an evaluation method for LLMs. If LLMs genuinely understand the semantics of questions, their performance should exhibit consistency across the varied configurations derived from the same questions. Contrary to this expectation, our empirical findings suggest a notable disparity in the consistency of LLM responses, which we define as REsponse VAriability Syndrome (REVAS) of the LLMs, indicating that current MCQA-based benchmarks may not adequately capture the true capabilities of LLMs, which underscores the need for more robust evaluation mechanisms in assessing the performance of LLMs.

Title: FedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learning

• Authors: Liping Yi, Han Yu, Chao Ren, Heng Zhang, Gang Wang, Xiaoguang Liu, Xiaoxiao Li
• Subjects: cs.LG, cs.DC
• Abstract URL: https://arxiv.org/abs/2402.01350
• Pdf URL: https://arxiv.org/pdf/2402.01350
• Copy Paste: [[2402.01350]] FedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learning(https://arxiv.org/abs/2402.01350)
• Keywords: privacy, extraction, federate, large language model
• Abstract: Federated learning (FL) is widely employed for collaborative training on decentralized data but faces challenges like data, system, and model heterogeneity. This prompted the emergency of model-heterogeneous personalized federated learning (MHPFL). However, concerns persist regarding data and model privacy, model performance, communication, and computational costs in current MHPFL methods. To tackle these concerns, we propose a novel model-heterogeneous personalized Federated learning algorithm (FedMoE) with the Mixture of Experts (MoE), renowned for enhancing large language models (LLMs). It assigns a shared homogeneous small feature extractor and a local gating network for each client's local heterogeneous large model. (1) During local training, the local heterogeneous model's feature extractor acts as a local expert for personalized feature (representation) extraction, while the shared homogeneous small feature extractor serves as a global expert for generalized feature extraction. The local gating network produces personalized weights for extracted representations from both experts on each data sample. The three models form a local heterogeneous MoE. The weighted mixed representation fuses global generalized and local personalized features and is processed by the local heterogeneous large model's header with personalized prediction information for output. The MoE and prediction header are updated synchronously. (2) The trained local homogeneous small feature extractors are sent to the server for cross-client information fusion via aggregation. Briefly, FedMoE first enhances local model personalization at a fine-grained data level while supporting model heterogeneity.

Title: TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time (Extended Version)

• Authors: Zeliang Kan, Shae McFadden, Daniel Arp, Feargus Pendlebury, Roberto Jordaney, Johannes Kinder, Fabio Pierazzi, Lorenzo Cavallaro
• Subjects: cs.LG, cs.CR, cs.PF
• Abstract URL: https://arxiv.org/abs/2402.01359
• Pdf URL: https://arxiv.org/pdf/2402.01359
• Copy Paste: [[2402.01359]] TESSERACT: Eliminating Experimental Bias in Malware Classification across Space and Time (Extended Version)(https://arxiv.org/abs/2402.01359)
• Keywords: attack, robust, fair
• Abstract: Machine learning (ML) plays a pivotal role in detecting malicious software. Despite the high F1-scores reported in numerous studies reaching upwards of 0.99, the issue is not completely solved. Malware detectors often experience performance decay due to constantly evolving operating systems and attack methods, which can render previously learned knowledge insufficient for accurate decision-making on new inputs. This paper argues that commonly reported results are inflated due to two pervasive sources of experimental bias in the detection task: spatial bias caused by data distributions that are not representative of a real-world deployment; and temporal bias caused by incorrect time splits of data, leading to unrealistic configurations. To address these biases, we introduce a set of constraints for fair experiment design, and propose a new metric, AUT, for classifier robustness in real-world settings. We additionally propose an algorithm designed to tune training data to enhance classifier performance. Finally, we present TESSERACT, an open-source framework for realistic classifier comparison. Our evaluation encompasses both traditional ML and deep learning methods, examining published works on an extensive Android dataset with 259,230 samples over a five-year span. Additionally, we conduct case studies in the Windows PE and PDF domains. Our findings identify the existence of biases in previous studies and reveal that significant performance enhancements are possible through appropriate, periodic tuning. We explore how mitigation strategies may support in achieving a more stable and better performance over time by employing multiple strategies to delay performance decay.

Title: Bribe & Fork: Cheap Bribing Attacks via Forking Threat

• Authors: Zeta Avarikioti, Paweł Kędzior, Tomasz Lizurej, Tomasz Michalak
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.01363
• Pdf URL: https://arxiv.org/pdf/2402.01363
• Copy Paste: [[2402.01363]] Bribe & Fork: Cheap Bribing Attacks via Forking Threat(https://arxiv.org/abs/2402.01363)
• Keywords: attack, robust
• Abstract: In this work, we reexamine the vulnerability of Payment Channel Networks (PCNs) to bribing attacks, where an adversary incentivizes blockchain miners to deliberately ignore a specific transaction to undermine the punishment mechanism of PCNs. While previous studies have posited a prohibitive cost for such attacks, we show that this cost may be dramatically reduced (to approximately \$125), thereby increasing the likelihood of these attacks. To this end, we introduce Bribe & Fork, a modified bribing attack that leverages the threat of a so-called feather fork which we analyze with a novel formal model for the mining game with forking. We empirically analyze historical data of some real-world blockchain implementations to evaluate the scale of this cost reduction. Our findings shed more light on the potential vulnerability of PCNs and highlight the need for robust solutions.

Title: Continual Learning for Large Language Models: A Survey

• Authors: Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, Gholamreza Haffari
• Subjects: cs.CL, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.01364
• Pdf URL: https://arxiv.org/pdf/2402.01364
• Copy Paste: [[2402.01364]] Continual Learning for Large Language Models: A Survey(https://arxiv.org/abs/2402.01364)
• Keywords: large language model
• Abstract: Large language models (LLMs) are not amenable to frequent re-training, due to high training costs arising from their massive scale. However, updates are necessary to endow LLMs with new skills and keep them up-to-date with rapidly evolving human knowledge. This paper surveys recent works on continual learning for LLMs. Due to the unique nature of LLMs, we catalog continue learning techniques in a novel multi-staged categorization scheme, involving continual pretraining, instruction tuning, and alignment. We contrast continual learning for LLMs with simpler adaptation methods used in smaller models, as well as with other enhancement strategies like retrieval-augmented generation and model editing. Moreover, informed by a discussion of benchmarks and evaluation, we identify several challenges and future work directions for this crucial task.

Title: LIR: Efficient Degradation Removal for Lightweight Image Restoration

• Authors: Dongqi Fan, Ting Yue, Xin Zhao, Liang Chang
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01368
• Pdf URL: https://arxiv.org/pdf/2402.01368
• Copy Paste: [[2402.01368]] LIR: Efficient Degradation Removal for Lightweight Image Restoration(https://arxiv.org/abs/2402.01368)
• Keywords: transformer
• Abstract: Recently, there have been significant advancements in Image Restoration based on CNN and transformer. However, the inherent characteristics of the Image Restoration task are often overlooked in many works. These works often focus on the basic block design and stack numerous basic blocks to the model, leading to redundant parameters and unnecessary computations and hindering the efficiency of the image restoration. In this paper, we propose a Lightweight Image Restoration network called LIR to efficiently remove degradation (blur, rain, noise, haze, etc.). A key component in LIR is the Efficient Adaptive Attention (EAA) Block, which is mainly composed of Adaptive Filters and Attention Blocks. It is capable of adaptively sharpening contours, removing degradation, and capturing global information in various image restoration scenes in an efficient and computation-friendly manner. In addition, through a simple structural design, LIR addresses the degradations existing in the local and global residual connections that are ignored by modern networks. Extensive experiments demonstrate that our LIR achieves comparable performance to state-of-the-art networks on most benchmarks with fewer parameters and computations. It is worth noting that our LIR produces better visual results than state-of-the-art networks that are more in line with the human aesthetic.

Title: Cheating Suffix: Targeted Attack to Text-To-Image Diffusion Models with Multi-Modal Priors

• Authors: Dingcheng Yang, Yang Bai, Xiaojun Jia, Yang Liu, Xiaochun Cao, Wenjian Yu
• Subjects: cs.LG, cs.CR, cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01369
• Pdf URL: https://arxiv.org/pdf/2402.01369
• Copy Paste: [[2402.01369]] Cheating Suffix: Targeted Attack to Text-To-Image Diffusion Models with Multi-Modal Priors(https://arxiv.org/abs/2402.01369)
• Keywords: attack, diffusion
• Abstract: Diffusion models have been widely deployed in various image generation tasks, demonstrating an extraordinary connection between image and text modalities. However, they face challenges of being maliciously exploited to generate harmful or sensitive images by appending a specific suffix to the original prompt. Existing works mainly focus on using single-modal information to conduct attacks, which fails to utilize multi-modal features and results in less than satisfactory performance. Integrating multi-modal priors (MMP), i.e. both text and image features, we propose a targeted attack method named MMP-Attack in this work. Specifically, the goal of MMP-Attack is to add a target object into the image content while simultaneously removing the original object. The MMP-Attack shows a notable advantage over existing works with superior universality and transferability, which can effectively attack commercial text-to-image (T2I) models such as DALL-E 3. To the best of our knowledge, this marks the first successful attempt of transfer-based attack to commercial T2I models. Our code is publicly available at \url{https://github.com/ydc123/MMP-Attack}.

Title: Dive into the Chasm: Probing the Gap between In- and Cross-Topic Generalization

• Authors: Andreas Waldis, Yufang Hou, Iryna Gurevych
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01375
• Pdf URL: https://arxiv.org/pdf/2402.01375
• Copy Paste: [[2402.01375]] Dive into the Chasm: Probing the Gap between In- and Cross-Topic Generalization(https://arxiv.org/abs/2402.01375)
• Keywords: robust
• Abstract: Pre-trained language models (LMs) perform well in In-Topic setups, where training and testing data come from the same topics. However, they face challenges in Cross-Topic scenarios where testing data is derived from distinct topics -- such as Gun Control. This study analyzes various LMs with three probing-based experiments to shed light on the reasons behind the In- vs. Cross-Topic generalization gap. Thereby, we demonstrate, for the first time, that generalization gaps and the robustness of the embedding space vary significantly across LMs. Additionally, we assess larger LMs and underscore the relevance of our analysis for recent models. Overall, diverse pre-training objectives, architectural regularization, or data deduplication contribute to more robust LMs and diminish generalization gaps. Our research contributes to a deeper understanding and comparison of language models across different generalization scenarios.

Title: LoTR: Low Tensor Rank Weight Adaptation

• Authors: Daniel Bershatsky, Daria Cherniuk, Talgat Daulbaev, Ivan Oseledets
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.01376
• Pdf URL: https://arxiv.org/pdf/2402.01376
• Copy Paste: [[2402.01376]] LoTR: Low Tensor Rank Weight Adaptation(https://arxiv.org/abs/2402.01376)
• Keywords: transformer, large language model
• Abstract: In this paper we generalize and extend an idea of low-rank adaptation (LoRA) of large language models (LLMs) based on Transformer architecture. Widely used LoRA-like methods of fine-tuning LLMs are based on matrix factorization of gradient update. We introduce LoTR, a novel approach for parameter-efficient fine-tuning of LLMs which represents a gradient update to parameters in a form of tensor decomposition. Low-rank adapter for each layer is constructed as a product of three matrices, and tensor structure arises from sharing left and right multipliers of this product among layers. Simultaneous compression of a sequence of layers with low-rank tensor representation allows LoTR to archive even better parameter efficiency then LoRA especially for deep models. Moreover, the core tensor does not depend on original weight dimension and can be made arbitrary small, which allows for extremely cheap and fast downstream fine-tuning.

Title: LLM-based NLG Evaluation: Current Status and Challenges

• Authors: Mingqi Gao, Xinyu Hu, Jie Ruan, Xiao Pu, Xiaojun Wan
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01383
• Pdf URL: https://arxiv.org/pdf/2402.01383
• Copy Paste: [[2402.01383]] LLM-based NLG Evaluation: Current Status and Challenges(https://arxiv.org/abs/2402.01383)
• Keywords: large language model
• Abstract: Evaluating natural language generation (NLG) is a vital but challenging problem in artificial intelligence. Traditional evaluation metrics mainly capturing content (e.g. n-gram) overlap between system outputs and references are far from satisfactory, and large language models (LLMs) such as ChatGPT have demonstrated great potential in NLG evaluation in recent years. Various automatic evaluation methods based on LLMs have been proposed, including metrics derived from LLMs, prompting LLMs, and fine-tuning LLMs with labeled evaluation data. In this survey, we first give a taxonomy of LLM-based NLG evaluation methods, and discuss their pros and cons, respectively. We also discuss human-LLM collaboration for NLG evaluation. Lastly, we discuss several open problems in this area and point out future research directions.

Title: ALERT-Transformer: Bridging Asynchronous and Synchronous Machine Learning for Real-Time Event-based Spatio-Temporal Data

• Authors: Carmen Martin-Turrero, Maxence Bouvier, Manuel Breitenstein, Pietro Zanuttigh, Vincent Parret
• Subjects: cs.CV, cs.LG, cs.NE
• Abstract URL: https://arxiv.org/abs/2402.01393
• Pdf URL: https://arxiv.org/pdf/2402.01393
• Copy Paste: [[2402.01393]] ALERT-Transformer: Bridging Asynchronous and Synchronous Machine Learning for Real-Time Event-based Spatio-Temporal Data(https://arxiv.org/abs/2402.01393)
• Keywords: transformer
• Abstract: We seek to enable classic processing of continuous ultra-sparse spatiotemporal data generated by event-based sensors with dense machine learning models. We propose a novel hybrid pipeline composed of asynchronous sensing and synchronous processing that combines several ideas: (1) an embedding based on PointNet models -- the ALERT module -- that can continuously integrate new and dismiss old events thanks to a leakage mechanism, (2) a flexible readout of the embedded data that allows to feed any downstream model with always up-to-date features at any sampling rate, (3) exploiting the input sparsity in a patch-based approach inspired by Vision Transformer to optimize the efficiency of the method. These embeddings are then processed by a transformer model trained for object and gesture recognition. Using this approach, we achieve performances at the state-of-the-art with a lower latency than competitors. We also demonstrate that our asynchronous model can operate at any desired sampling rate.

Title: A Probabilistic Model to explain Self-Supervised Representation Learning

• Authors: Alice Bizeul, Bernhard Schölkopf, Carl Allen
• Subjects: cs.LG, cs.AI, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.01399
• Pdf URL: https://arxiv.org/pdf/2402.01399
• Copy Paste: [[2402.01399]] A Probabilistic Model to explain Self-Supervised Representation Learning(https://arxiv.org/abs/2402.01399)
• Keywords: generative
• Abstract: Self-supervised learning (SSL) learns representations by leveraging an auxiliary unsupervised task, such as classifying semantically related samples, e.g. different data augmentations or modalities. Of the many approaches to SSL, contrastive methods, e.g. SimCLR, CLIP and VicREG, have gained attention for learning representations that achieve downstream performance close to that of supervised learning. However, a theoretical understanding of the mechanism behind these methods eludes. We propose a generative latent variable model for the data and show that several families of discriminative self-supervised algorithms, including contrastive methods, approximately induce its latent structure over representations, providing a unifying theoretical framework. We also justify links to mutual information and the use of a projection head. Fitting our model generatively, as SimVE, improves performance over previous VAE methods on common benchmarks (e.g. FashionMNIST, CIFAR10, CelebA), narrows the gap to discriminative methods on _content_ classification and, as our analysis predicts, outperforms them where _style_ information is required, taking a step toward task-agnostic representations.

Title: Climbing the Ladder of Interpretability with Counterfactual Concept Bottleneck Models

• Authors: Gabriele Dominici, Pietro Barbiero, Francesco Giannini, Martin Gjoreski, Giuseppe Marra, Marc Langheinrich
• Subjects: cs.LG, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01408
• Pdf URL: https://arxiv.org/pdf/2402.01408
• Copy Paste: [[2402.01408]] Climbing the Ladder of Interpretability with Counterfactual Concept Bottleneck Models(https://arxiv.org/abs/2402.01408)
• Keywords: interpretability
• Abstract: Current deep learning models are not designed to simultaneously address three fundamental questions: predict class labels to solve a given classification task (the "What?"), explain task predictions (the "Why?"), and imagine alternative scenarios that could result in different predictions (the "What if?"). The inability to answer these questions represents a crucial gap in deploying reliable AI agents, calibrating human trust, and deepening human-machine interaction. To bridge this gap, we introduce CounterFactual Concept Bottleneck Models (CF-CBMs), a class of models designed to efficiently address the above queries all at once without the need to run post-hoc searches. Our results show that CF-CBMs produce: accurate predictions (the "What?"), simple explanations for task predictions (the "Why?"), and interpretable counterfactuals (the "What if?"). CF-CBMs can also sample or estimate the most probable counterfactual to: (i) explain the effect of concept interventions on tasks, (ii) show users how to get a desired class label, and (iii) propose concept interventions via "task-driven" interventions.

Title: XAI for Skin Cancer Detection with Prototypes and Non-Expert Supervision

• Authors: Miguel Correia, Alceu Bissoto, Carlos Santiago, Catarina Barata
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.01410
• Pdf URL: https://arxiv.org/pdf/2402.01410
• Copy Paste: [[2402.01410]] XAI for Skin Cancer Detection with Prototypes and Non-Expert Supervision(https://arxiv.org/abs/2402.01410)
• Keywords: interpretability, segmentation
• Abstract: Skin cancer detection through dermoscopy image analysis is a critical task. However, existing models used for this purpose often lack interpretability and reliability, raising the concern of physicians due to their black-box nature. In this paper, we propose a novel approach for the diagnosis of melanoma using an interpretable prototypical-part model. We introduce a guided supervision based on non-expert feedback through the incorporation of: 1) binary masks, obtained automatically using a segmentation network; and 2) user-refined prototypes. These two distinct information pathways aim to ensure that the learned prototypes correspond to relevant areas within the skin lesion, excluding confounding factors beyond its boundaries. Experimental results demonstrate that, even without expert supervision, our approach achieves superior performance and generalization compared to non-interpretable models.

Title: Sequence Shortening for Context-Aware Machine Translation

• Authors: Paweł Mąka, Yusuf Can Semerci, Jan Scholtes, Gerasimos Spanakis
• Subjects: cs.CL, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.01416
• Pdf URL: https://arxiv.org/pdf/2402.01416
• Copy Paste: [[2402.01416]] Sequence Shortening for Context-Aware Machine Translation(https://arxiv.org/abs/2402.01416)
• Keywords: interpretability
• Abstract: Context-aware Machine Translation aims to improve translations of sentences by incorporating surrounding sentences as context. Towards this task, two main architectures have been applied, namely single-encoder (based on concatenation) and multi-encoder models. In this study, we show that a special case of multi-encoder architecture, where the latent representation of the source sentence is cached and reused as the context in the next step, achieves higher accuracy on the contrastive datasets (where the models have to rank the correct translation among the provided sentences) and comparable BLEU and COMET scores as the single- and multi-encoder approaches. Furthermore, we investigate the application of Sequence Shortening to the cached representations. We test three pooling-based shortening techniques and introduce two novel methods - Latent Grouping and Latent Selecting, where the network learns to group tokens or selects the tokens to be cached as context. Our experiments show that the two methods achieve competitive BLEU and COMET scores and accuracies on the contrastive datasets to the other tested methods while potentially allowing for higher interpretability and reducing the growth of memory requirements with increased context size.

Title: EmoSpeaker: One-shot Fine-grained Emotion-Controlled Talking Face Generation

• Authors: Guanwen Feng, Haoran Cheng, Yunan Li, Zhiyuan Ma, Chaoneng Li, Zhihao Qian, Qiguang Miao, Chi-Man Pun
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01422
• Pdf URL: https://arxiv.org/pdf/2402.01422
• Copy Paste: [[2402.01422]] EmoSpeaker: One-shot Fine-grained Emotion-Controlled Talking Face Generation(https://arxiv.org/abs/2402.01422)
• Keywords: generative
• Abstract: Implementing fine-grained emotion control is crucial for emotion generation tasks because it enhances the expressive capability of the generative model, allowing it to accurately and comprehensively capture and express various nuanced emotional states, thereby improving the emotional quality and personalization of generated content. Generating fine-grained facial animations that accurately portray emotional expressions using only a portrait and an audio recording presents a challenge. In order to address this challenge, we propose a visual attribute-guided audio decoupler. This enables the obtention of content vectors solely related to the audio content, enhancing the stability of subsequent lip movement coefficient predictions. To achieve more precise emotional expression, we introduce a fine-grained emotion coefficient prediction module. Additionally, we propose an emotion intensity control method using a fine-grained emotion matrix. Through these, effective control over emotional expression in the generated videos and finer classification of emotion intensity are accomplished. Subsequently, a series of 3DMM coefficient generation networks are designed to predict 3D coefficients, followed by the utilization of a rendering network to generate the final video. Our experimental results demonstrate that our proposed method, EmoSpeaker, outperforms existing emotional talking face generation methods in terms of expression variation and lip synchronization. Project page: https://peterfanfan.github.io/EmoSpeaker/

Title: Different Tastes of Entities: Investigating Human Label Variation in Named Entity Annotations

• Authors: Siyao Peng, Zihang Sun, Sebastian Loftus, Barbara Plank
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01423
• Pdf URL: https://arxiv.org/pdf/2402.01423
• Copy Paste: [[2402.01423]] Different Tastes of Entities: Investigating Human Label Variation in Named Entity Annotations(https://arxiv.org/abs/2402.01423)
• Keywords: extraction
• Abstract: Named Entity Recognition (NER) is a key information extraction task with a long-standing tradition. While recent studies address and aim to correct annotation errors via re-labeling efforts, little is known about the sources of human label variation, such as text ambiguity, annotation error, or guideline divergence. This is especially the case for high-quality datasets and beyond English CoNLL03. This paper studies disagreements in expert-annotated named entity datasets for three languages: English, Danish, and Bavarian. We show that text ambiguity and artificial guideline changes are dominant factors for diverse annotations among high-quality revisions. We survey student annotations on a subset of difficult entities and substantiate the feasibility and necessity of manifold annotations for understanding named entity ambiguities from a distributional perspective.

Title: From Words to Molecules: A Survey of Large Language Models in Chemistry

• Authors: Chang Liao, Yemin Yu, Yu Mei, Ying Wei
• Subjects: cs.LG, cs.AI, q-bio.BM, q-bio.QM
• Abstract URL: https://arxiv.org/abs/2402.01439
• Pdf URL: https://arxiv.org/pdf/2402.01439
• Copy Paste: [[2402.01439]] From Words to Molecules: A Survey of Large Language Models in Chemistry(https://arxiv.org/abs/2402.01439)
• Keywords: interpretability, large language model
• Abstract: In recent years, Large Language Models (LLMs) have achieved significant success in natural language processing (NLP) and various interdisciplinary areas. However, applying LLMs to chemistry is a complex task that requires specialized domain knowledge. This paper provides a thorough exploration of the nuanced methodologies employed in integrating LLMs into the field of chemistry, delving into the complexities and innovations at this interdisciplinary juncture. Specifically, our analysis begins with examining how molecular information is fed into LLMs through various representation and tokenization methods. We then categorize chemical LLMs into three distinct groups based on the domain and modality of their input data, and discuss approaches for integrating these inputs for LLMs. Furthermore, this paper delves into the pretraining objectives with adaptations to chemical LLMs. After that, we explore the diverse applications of LLMs in chemistry, including novel paradigms for their application in chemistry tasks. Finally, we identify promising research directions, including further integration with chemical knowledge, advancements in continual learning, and improvements in model interpretability, paving the way for groundbreaking developments in the field.

Title: Integrating Large Language Models in Causal Discovery: A Statistical Causal Approach

• Authors: Masayuki Takayama, Tadahisa Okuda, Thong Pham, Tatsuyoshi Ikenoue, Shingo Fukuma, Shohei Shimizu, Akiyoshi Sannai
• Subjects: cs.LG, cs.AI, stat.ME, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.01454
• Pdf URL: https://arxiv.org/pdf/2402.01454
• Copy Paste: [[2402.01454]] Integrating Large Language Models in Causal Discovery: A Statistical Causal Approach(https://arxiv.org/abs/2402.01454)
• Keywords: large language model
• Abstract: In practical statistical causal discovery (SCD), embedding domain expert knowledge as constraints into the algorithm is widely accepted as significant for creating consistent meaningful causal models, despite the recognized challenges in systematic acquisition of the background knowledge. To overcome these challenges, this paper proposes a novel methodology for causal inference, in which SCD methods and knowledge based causal inference (KBCI) with a large language model (LLM) are synthesized through "statistical causal prompting (SCP)" for LLMs and prior knowledge augmentation for SCD. Experiments have revealed that GPT-4 can cause the output of the LLM-KBCI and the SCD result with prior knowledge from LLM-KBCI to approach the ground truth, and that the SCD result can be further improved, if GPT-4 undergoes SCP. Furthermore, it has been clarified that an LLM can improve SCD with its background knowledge, even if the LLM does not contain information on the dataset. The proposed approach can thus address challenges such as dataset biases and limitations, illustrating the potential of LLMs to improve data-driven causal inference across diverse scientific domains.

Title: Convolution kernel adaptation to calibrated fisheye

• Authors: Bruno Berenguel-Baeta, Maria Santos-Villafranca, Jesus Bermudez-Cameo, Alejandro Perez-Yus, Jose J. Guerrero
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01456
• Pdf URL: https://arxiv.org/pdf/2402.01456
• Copy Paste: [[2402.01456]] Convolution kernel adaptation to calibrated fisheye(https://arxiv.org/abs/2402.01456)
• Keywords: segmentation
• Abstract: Convolution kernels are the basic structural component of convolutional neural networks (CNNs). In the last years there has been a growing interest in fisheye cameras for many applications. However, the radially symmetric projection model of these cameras produces high distortions that affect the performance of CNNs, especially when the field of view is very large. In this work, we tackle this problem by proposing a method that leverages the calibration of cameras to deform the convolution kernel accordingly and adapt to the distortion. That way, the receptive field of the convolution is similar to standard convolutions in perspective images, allowing us to take advantage of pre-trained networks in large perspective datasets. We show how, with just a brief fine-tuning stage in a small dataset, we improve the performance of the network for the calibrated fisheye with respect to standard convolutions in depth estimation and semantic segmentation.

Title: Visual Gyroscope: Combination of Deep Learning Features and Direct Alignment for Panoramic Stabilization

• Authors: Bruno Berenguel-Baeta, Antoine N. Andre, Guillaume Caron, Jesus Bermudez-Cameo, Jose J. Guerrero
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01461
• Pdf URL: https://arxiv.org/pdf/2402.01461
• Copy Paste: [[2402.01461]] Visual Gyroscope: Combination of Deep Learning Features and Direct Alignment for Panoramic Stabilization(https://arxiv.org/abs/2402.01461)
• Keywords: robust
• Abstract: In this article we present a visual gyroscope based on equirectangular panoramas. We propose a new pipeline where we take advantage of combining three different methods to obtain a robust and accurate estimation of the attitude of the camera. We quantitatively and qualitatively validate our method on two image sequences taken with a $360^\circ$ dual-fisheye camera mounted on different aerial vehicles.

Title: AMOR: A Recipe for Building Adaptable Modular Knowledge Agents Through Process Feedback

• Authors: Jian Guan, Wei Wu, Zujie Wen, Peng Xu, Hongning Wang, Minlie Huang
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01469
• Pdf URL: https://arxiv.org/pdf/2402.01469
• Copy Paste: [[2402.01469]] AMOR: A Recipe for Building Adaptable Modular Knowledge Agents Through Process Feedback(https://arxiv.org/abs/2402.01469)
• Keywords: large language model
• Abstract: The notable success of large language models (LLMs) has sparked an upsurge in building language agents to complete various complex tasks. We present AMOR, an agent framework based on open-source LLMs, which reasons with external knowledge bases and adapts to specific domains through human supervision to the reasoning process. AMOR builds reasoning logic over a finite state machine (FSM) that solves problems through autonomous executions and transitions over disentangled modules. This allows humans to provide direct feedback to the individual modules, and thus naturally forms process supervision. Based on this reasoning and feedback framework, we develop AMOR through two-stage fine-tuning: warm-up and adaptation. The former fine-tunes the LLM with examples automatically constructed from various public datasets and enables AMOR to generalize across different knowledge environments, while the latter tailors AMOR to specific domains using process feedback. Extensive experiments across multiple domains demonstrate the advantage of AMOR to strong baselines, thanks to its FSM-based reasoning and process feedback mechanism.

Title: Synthetic Data for the Mitigation of Demographic Biases in Face Recognition

• Authors: Pietro Melzi, Christian Rathgeb, Ruben Tolosana, Ruben Vera-Rodriguez, Aythami Morales, Dominik Lawatsch, Florian Domin, Maxim Schaubert
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01472
• Pdf URL: https://arxiv.org/pdf/2402.01472
• Copy Paste: [[2402.01472]] Synthetic Data for the Mitigation of Demographic Biases in Face Recognition(https://arxiv.org/abs/2402.01472)
• Keywords: fair
• Abstract: This study investigates the possibility of mitigating the demographic biases that affect face recognition technologies through the use of synthetic data. Demographic biases have the potential to impact individuals from specific demographic groups, and can be identified by observing disparate performance of face recognition systems across demographic groups. They primarily arise from the unequal representations of demographic groups in the training data. In recent times, synthetic data have emerged as a solution to some problems that affect face recognition systems. In particular, during the generation process it is possible to specify the desired demographic and facial attributes of images, in order to control the demographic distribution of the synthesized dataset, and fairly represent the different demographic groups. We propose to fine-tune with synthetic data existing face recognition systems that present some demographic biases. We use synthetic datasets generated with GANDiffFace, a novel framework able to synthesize datasets for face recognition with controllable demographic distribution and realistic intra-class variations. We consider multiple datasets representing different demographic groups for training and evaluation. Also, we fine-tune different face recognition systems, and evaluate their demographic fairness with different metrics. Our results support the proposed approach and the use of synthetic data to mitigate demographic biases in face recognition.

Title: Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processes

• Authors: Yingyi Chen, Qinghua Tao, Francesco Tonin, Johan A.K. Suykens
• Subjects: cs.LG, cs.AI, cs.CV, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.01476
• Pdf URL: https://arxiv.org/pdf/2402.01476
• Copy Paste: [[2402.01476]] Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processes(https://arxiv.org/abs/2402.01476)
• Keywords: transformer
• Abstract: While the great capability of Transformers significantly boosts prediction accuracy, it could also yield overconfident predictions and require calibrated uncertainty estimation, which can be commonly tackled by Gaussian processes (GPs). Existing works apply GPs with symmetric kernels under variational inference to the attention kernel; however, omitting the fact that attention kernels are in essence asymmetric. Moreover, the complexity of deriving the GP posteriors remains high for large-scale data. In this work, we propose Kernel-Eigen Pair Sparse Variational Gaussian Processes (KEP-SVGP) for building uncertainty-aware self-attention where the asymmetry of attention kernels is tackled by Kernel SVD (KSVD) and a reduced complexity is acquired. Through KEP-SVGP, i) the SVGP pair induced by the two sets of singular vectors from KSVD w.r.t. the attention kernel fully characterizes the asymmetry; ii) using only a small set of adjoint eigenfunctions from KSVD, the derivation of SVGP posteriors can be based on the inversion of a diagonal matrix containing singular values, contributing to a reduction in time complexity; iii) an evidence lower bound is derived so that variational parameters can be optimized towards this objective. Experiments verify our excellent performances and efficiency on in-distribution, distribution-shift and out-of-distribution benchmarks.

Title: A Comparative Analysis of Conversational Large Language Models in Knowledge-Based Text Generation

• Authors: Phillip Schneider, Manuel Klettner, Elena Simperl, Florian Matthes
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01495
• Pdf URL: https://arxiv.org/pdf/2402.01495
• Copy Paste: [[2402.01495]] A Comparative Analysis of Conversational Large Language Models in Knowledge-Based Text Generation(https://arxiv.org/abs/2402.01495)
• Keywords: large language model
• Abstract: Generating natural language text from graph-structured data is essential for conversational information seeking. Semantic triples derived from knowledge graphs can serve as a valuable source for grounding responses from conversational agents by providing a factual basis for the information they communicate. This is especially relevant in the context of large language models, which offer great potential for conversational interaction but are prone to hallucinating, omitting, or producing conflicting information. In this study, we conduct an empirical analysis of conversational large language models in generating natural language text from semantic triples. We compare four large language models of varying sizes with different prompting techniques. Through a series of benchmark experiments on the WebNLG dataset, we analyze the models' performance and identify the most common issues in the generated predictions. Our findings show that the capabilities of large language models in triple verbalization can be significantly improved through few-shot prompting, post-processing, and efficient fine-tuning techniques, particularly for smaller models that exhibit lower zero-shot performance.

Title: Mapping the Multiverse of Latent Representations

• Authors: Jeremy Wayland, Corinna Coupette, Bastian Rieck
• Subjects: cs.LG, math.AT, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.01514
• Pdf URL: https://arxiv.org/pdf/2402.01514
• Copy Paste: [[2402.01514]] Mapping the Multiverse of Latent Representations(https://arxiv.org/abs/2402.01514)
• Keywords: robust
• Abstract: Echoing recent calls to counter reliability and robustness concerns in machine learning via multiverse analysis, we present PRESTO, a principled framework for mapping the multiverse of machine-learning models that rely on latent representations. Although such models enjoy widespread adoption, the variability in their embeddings remains poorly understood, resulting in unnecessary complexity and untrustworthy representations. Our framework uses persistent homology to characterize the latent spaces arising from different combinations of diverse machine-learning methods, (hyper)parameter configurations, and datasets, allowing us to measure their pairwise (dis)similarity and statistically reason about their distributions. As we demonstrate both theoretically and empirically, our pipeline preserves desirable properties of collections of latent representations, and it can be leveraged to perform sensitivity analysis, detect anomalous embeddings, or efficiently and effectively navigate hyperparameter search spaces.

Title: Cross-view Masked Diffusion Transformers for Person Image Synthesis

• Authors: Trung X. Pham, Zhang Kang, Chang D. Yoo
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01516
• Pdf URL: https://arxiv.org/pdf/2402.01516
• Copy Paste: [[2402.01516]] Cross-view Masked Diffusion Transformers for Person Image Synthesis(https://arxiv.org/abs/2402.01516)
• Keywords: diffusion, transformer
• Abstract: We present X-MDPT (Cross-view Masked Diffusion Prediction Transformers), a novel diffusion model designed for pose-guided human image generation. X-MDPT distinguishes itself by employing masked diffusion transformers that operate on latent patches, a departure from the commonly-used Unet structures in existing works. The model comprises three key modules: 1) a denoising diffusion Transformer, 2) an aggregation network that consolidates conditions into a single vector for the diffusion process, and 3) a mask cross-prediction module that enhances representation learning with semantic information from the reference image. X-MDPT demonstrates scalability, improving FID, SSIM, and LPIPS with larger models. Despite its simple design, our model outperforms state-of-the-art approaches on the DeepFashion dataset while exhibiting efficiency in terms of training parameters, training time, and inference speed. Our compact 33MB model achieves an FID of 7.42, surpassing a prior Unet latent diffusion approach (FID 8.07) using only $11\times$ fewer parameters. Our best model surpasses the pixel-based diffusion with $\frac{2}{3}$ of the parameters and achieves $5.43 \times$ faster inference.

Title: K-Level Reasoning with Large Language Models

• Authors: Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Yan Xia, Man Lan, Furu Wei
• Subjects: cs.CL, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01521
• Pdf URL: https://arxiv.org/pdf/2402.01521
• Copy Paste: [[2402.01521]] K-Level Reasoning with Large Language Models(https://arxiv.org/abs/2402.01521)
• Keywords: robust, large language model
• Abstract: While Large Language Models (LLMs) have demonstrated their proficiency in complex reasoning tasks, their performance in dynamic, interactive, and competitive scenarios - such as business strategy and stock market analysis - remains underexplored. To bridge this gap, we formally explore the dynamic reasoning capabilities of LLMs for decision-making in rapidly evolving environments. We introduce two game theory-based pilot challenges that mirror the complexities of real-world dynamic decision-making. These challenges are well-defined, enabling clear, controllable, and precise evaluation of LLMs' dynamic reasoning abilities. Through extensive experiments, we find that existing reasoning methods tend to falter in dynamic settings that require k-level thinking - a key concept not tackled by previous works. To address this, we propose a novel reasoning approach for LLMs, named "K-Level Reasoning". This approach adopts the perspective of rivals to recursively employ k-level thinking based on available historical information, which significantly improves the prediction accuracy of rivals' subsequent moves and informs more strategic decision-making. This research not only sets a robust quantitative benchmark for the assessment of dynamic reasoning but also markedly enhances the proficiency of LLMs in dynamic contexts.

Title: Decoding Speculative Decoding

• Authors: Minghao Yan, Saurabh Agarwal, Shivaram Venkataraman
• Subjects: cs.LG, cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01528
• Pdf URL: https://arxiv.org/pdf/2402.01528
• Copy Paste: [[2402.01528]] Decoding Speculative Decoding(https://arxiv.org/abs/2402.01528)
• Keywords: large language model
• Abstract: Speculative Decoding is a widely used technique to speed up inference for Large Language Models (LLMs) without modifying its outcome. When performing inference on an LLM, speculative decoding uses a smaller draft model which generates speculative tokens and then uses the target LLM to verify those draft tokens. The speedup provided by speculative decoding heavily depends on the choice of the draft model. It has been widely suggested to select a draft model that provides a high probability of the generated token being accepted by the LLM to achieve the highest throughput. However, our experiments indicate the contrary with throughput diminishing as the probability of generated tokens to be accepted by the target model increases. To understand this phenomenon, we perform extensive experiments to characterize the different factors that affect speculative decoding and how those factors interact and affect the speedups. Based on our experiments we describe an analytical model which can be used to decide the right draft model for a given workload. Further, using our insights we design a new draft model for LLaMA-65B which can provide 30% higher throughput than existing draft models.

Title: Closing the Gap in Human Behavior Analysis: A Pipeline for Synthesizing Trimodal Data

• Authors: Christian Stippel, Thomas Heitzinger, Rafael Sterzinger, Martin Kampel
• Subjects: cs.CV, cs.AI, cs.LG
• Abstract URL: https://arxiv.org/abs/2402.01537
• Pdf URL: https://arxiv.org/pdf/2402.01537
• Copy Paste: [[2402.01537]] Closing the Gap in Human Behavior Analysis: A Pipeline for Synthesizing Trimodal Data(https://arxiv.org/abs/2402.01537)
• Keywords: privacy, generative, segmentation
• Abstract: In pervasive machine learning, especially in Human Behavior Analysis (HBA), RGB has been the primary modality due to its accessibility and richness of information. However, linked with its benefits are challenges, including sensitivity to lighting conditions and privacy concerns. One possibility to overcome these vulnerabilities is to resort to different modalities. For instance, thermal is particularly adept at accentuating human forms, while depth adds crucial contextual layers. Despite their known benefits, only a few HBA-specific datasets that integrate these modalities exist. To address this shortage, our research introduces a novel generative technique for creating trimodal, i.e., RGB, thermal, and depth, human-focused datasets. This technique capitalizes on human segmentation masks derived from RGB images, combined with thermal and depth backgrounds that are sourced automatically. With these two ingredients, we synthesize depth and thermal counterparts from existing RGB data utilizing conditional image-to-image translation. By employing this approach, we generate trimodal data that can be leveraged to train models for settings with limited data, bad lightning conditions, or privacy-sensitive areas.

Title: Privacy-Preserving Distributed Learning for Residential Short-Term Load Forecasting

• Authors: Yi Dong, Yingjie Wang, Mariana Gama, Mustafa A. Mustafa, Geert Deconinck, Xiaowei Huang
• Subjects: cs.LG, cs.AI, cs.CR, cs.DC, cs.MA, eess.SY
• Abstract URL: https://arxiv.org/abs/2402.01546
• Pdf URL: https://arxiv.org/pdf/2402.01546
• Copy Paste: [[2402.01546]] Privacy-Preserving Distributed Learning for Residential Short-Term Load Forecasting(https://arxiv.org/abs/2402.01546)
• Keywords: secure, security, privacy, attack, robust, federate
• Abstract: In the realm of power systems, the increasing involvement of residential users in load forecasting applications has heightened concerns about data privacy. Specifically, the load data can inadvertently reveal the daily routines of residential users, thereby posing a risk to their property security. While federated learning (FL) has been employed to safeguard user privacy by enabling model training without the exchange of raw data, these FL models have shown vulnerabilities to emerging attack techniques, such as Deep Leakage from Gradients and poisoning attacks. To counteract these, we initially employ a Secure-Aggregation (SecAgg) algorithm that leverages multiparty computation cryptographic techniques to mitigate the risk of gradient leakage. However, the introduction of SecAgg necessitates the deployment of additional sub-center servers for executing the multiparty computation protocol, thereby escalating computational complexity and reducing system robustness, especially in scenarios where one or more sub-centers are unavailable. To address these challenges, we introduce a Markovian Switching-based distributed training framework, the convergence of which is substantiated through rigorous theoretical analysis. The Distributed Markovian Switching (DMS) topology shows strong robustness towards the poisoning attacks as well. Case studies employing real-world power system load data validate the efficacy of our proposed algorithm. It not only significantly minimizes communication complexity but also maintains accuracy levels comparable to traditional FL methods, thereby enhancing the scalability of our load forecasting algorithm.

Title: Hardware Trojans in Quantum Circuits, Their Impacts, and Defense

• Authors: Rupshali Roy, Subrata Das, Swaroop Ghosh
• Subjects: cs.CR
• Abstract URL: https://arxiv.org/abs/2402.01552
• Pdf URL: https://arxiv.org/pdf/2402.01552
• Copy Paste: [[2402.01552]] Hardware Trojans in Quantum Circuits, Their Impacts, and Defense(https://arxiv.org/abs/2402.01552)
• Keywords: security, defense
• Abstract: The reliability of the outcome of a quantum circuit in near-term noisy quantum computers depends on the gate count and depth for a given problem. Circuits with a short depth and lower gate count can yield the correct solution more often than the variant with a higher gate count and depth. To work successfully for Noisy Intermediate Scale Quantum (NISQ) computers, quantum circuits need to be optimized efficiently using a compiler that decomposes high-level gates to native gates of the hardware. Many 3rd party compilers are being developed for lower compilation time, reduced circuit depth, and lower gate count for large quantum circuits. Such compilers, or even a specific release version of a compiler that is otherwise trustworthy, may be unreliable and give rise to security risks such as insertion of a quantum trojan during compilation that evades detection due to the lack of a golden/Oracle model in quantum computing. Trojans may corrupt the functionality to give flipped probabilities of basis states, or result in a lower probability of correct basis states in the output. In this paper, we investigate and discuss the impact of a single qubit Trojan (we have chosen a Hadamard gate and a NOT gate) inserted one at a time at various locations in benchmark quantum circuits without changing the the depth of the circuit. Results indicate an average of 16.18% degradation for the Hadamard Trojan without noise, and 7.78% with noise. For the NOT Trojan (with noise) there is 14.6% degradation over all possible inputs. We then discuss the detection of such Trojans in a quantum circuit using CNN-based classifier achieving an accuracy of 90%.

Title: Boximator: Generating Rich and Controllable Motions for Video Synthesis

• Authors: Jiawei Wang, Yuchen Zhang, Jiaxin Zou, Yan Zeng, Guoqiang Wei, Liping Yuan, Hang Li
• Subjects: cs.CV, cs.AI
• Abstract URL: https://arxiv.org/abs/2402.01566
• Pdf URL: https://arxiv.org/pdf/2402.01566
• Copy Paste: [[2402.01566]] Boximator: Generating Rich and Controllable Motions for Video Synthesis(https://arxiv.org/abs/2402.01566)
• Keywords: robust, diffusion
• Abstract: Generating rich and controllable motion is a pivotal challenge in video synthesis. We propose Boximator, a new approach for fine-grained motion control. Boximator introduces two constraint types: hard box and soft box. Users select objects in the conditional frame using hard boxes and then use either type of boxes to roughly or rigorously define the object's position, shape, or motion path in future frames. Boximator functions as a plug-in for existing video diffusion models. Its training process preserves the base model's knowledge by freezing the original weights and training only the control module. To address training challenges, we introduce a novel self-tracking technique that greatly simplifies the learning of box-object correlations. Empirically, Boximator achieves state-of-the-art video quality (FVD) scores, improving on two base models, and further enhanced after incorporating box constraints. Its robust motion controllability is validated by drastic increases in the bounding box alignment metric. Human evaluation also shows that users favor Boximator generation results over the base model.

Title: NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties

• Authors: Jingyuan Sun, Mingxiao Li, Zijiao Chen, Marie-Francine Moens
• Subjects: cs.CV
• Abstract URL: https://arxiv.org/abs/2402.01590
• Pdf URL: https://arxiv.org/pdf/2402.01590
• Copy Paste: [[2402.01590]] NeuroCine: Decoding Vivid Video Sequences from Human Brain Activties(https://arxiv.org/abs/2402.01590)
• Keywords: interpretability, diffusion
• Abstract: In the pursuit to understand the intricacies of human brain's visual processing, reconstructing dynamic visual experiences from brain activities emerges as a challenging yet fascinating endeavor. While recent advancements have achieved success in reconstructing static images from non-invasive brain recordings, the domain of translating continuous brain activities into video format remains underexplored. In this work, we introduce NeuroCine, a novel dual-phase framework to targeting the inherent challenges of decoding fMRI data, such as noises, spatial redundancy and temporal lags. This framework proposes spatial masking and temporal interpolation-based augmentation for contrastive learning fMRI representations and a diffusion model enhanced by dependent prior noise for video generation. Tested on a publicly available fMRI dataset, our method shows promising results, outperforming the previous state-of-the-art models by a notable margin of ${20.97\%}$, ${31.00\%}$ and ${12.30\%}$ respectively on decoding the brain activities of three subjects in the fMRI dataset, as measured by SSIM. Additionally, our attention analysis suggests that the model aligns with existing brain structures and functions, indicating its biological plausibility and interpretability.

Title: L2G2G: a Scalable Local-to-Global Network Embedding with Graph Autoencoders

• Authors: Ruikang Ouyang, Andrew Elliott, Stratis Limnios, Mihai Cucuringu, Gesine Reinert
• Subjects: cs.LG, cs.AI, cs.SI, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.01614
• Pdf URL: https://arxiv.org/pdf/2402.01614
• Copy Paste: [[2402.01614]] L2G2G: a Scalable Local-to-Global Network Embedding with Graph Autoencoders(https://arxiv.org/abs/2402.01614)
• Keywords: fair
• Abstract: For analysing real-world networks, graph representation learning is a popular tool. These methods, such as a graph autoencoder (GAE), typically rely on low-dimensional representations, also called embeddings, which are obtained through minimising a loss function; these embeddings are used with a decoder for downstream tasks such as node classification and edge prediction. While GAEs tend to be fairly accurate, they suffer from scalability issues. For improved speed, a Local2Global approach, which combines graph patch embeddings based on eigenvector synchronisation, was shown to be fast and achieve good accuracy. Here we propose L2G2G, a Local2Global method which improves GAE accuracy without sacrificing scalability. This improvement is achieved by dynamically synchronising the latent node representations, while training the GAEs. It also benefits from the decoder computing an only local patch loss. Hence, aligning the local embeddings in each epoch utilises more information from the graph than a single post-training alignment does, while maintaining scalability. We illustrate on synthetic benchmarks, as well as real-world examples, that L2G2G achieves higher accuracy than the standard Local2Global approach and scales efficiently on the larger data sets. We find that for large and dense networks, it even outperforms the slow, but assumed more accurate, GAEs.

Title: Style Vectors for Steering Generative Large Language Model

• Authors: Kai Konen, Sophie Jentzsch, Diaoulé Diallo, Peer Schütt, Oliver Bensch, Roxanne El Baff, Dominik Opitz, Tobias Hecking
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01618
• Pdf URL: https://arxiv.org/pdf/2402.01618
• Copy Paste: [[2402.01618]] Style Vectors for Steering Generative Large Language Model(https://arxiv.org/abs/2402.01618)
• Keywords: generative, large language model
• Abstract: This research explores strategies for steering the output of large language models (LLMs) towards specific styles, such as sentiment, emotion, or writing style, by adding style vectors to the activations of hidden layers during text generation. We show that style vectors can be simply computed from recorded layer activations for input texts in a specific style in contrast to more complex training-based approaches. Through a series of experiments, we demonstrate the effectiveness of activation engineering using such style vectors to influence the style of generated text in a nuanced and parameterisable way, distinguishing it from prompt engineering. The presented research constitutes a significant step towards developing more adaptive and effective AI-empowered interactive systems.

Title: KB-Plugin: A Plug-and-play Framework for Large Language Models to Induce Programs over Low-resourced Knowledge Bases

• Authors: Jiajie Zhang, Shulin Cao, Linmei Hu, Ling Feng, Lei Hou, Juanzi Li
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01619
• Pdf URL: https://arxiv.org/pdf/2402.01619
• Copy Paste: [[2402.01619]] KB-Plugin: A Plug-and-play Framework for Large Language Models to Induce Programs over Low-resourced Knowledge Bases(https://arxiv.org/abs/2402.01619)
• Keywords: large language model
• Abstract: Program induction (PI) has become a promising paradigm for using knowledge bases (KBs) to help large language models (LLMs) answer complex knowledge-intensive questions. Nonetheless, PI typically relies on a large number of parallel question-program pairs to make the LLM aware of the schema of the given KB, and is thus challenging for many low-resourced KBs that lack annotated data. To this end, we propose KB-Plugin, a plug-and-play framework that enables LLMs to induce programs over any low-resourced KB. Firstly, KB-Plugin adopts self-supervised learning to encode the detailed schema information of a given KB into a pluggable module, namely schema plugin. Secondly, KB-Plugin utilizes abundant annotated data from a rich-resourced KB to train another pluggable module, namely PI plugin, which can help the LLM extract question-relevant schema information from the schema plugin of any KB and utilize this information to induce programs over this KB. Experiments on five heterogeneous KBQA datasets show that KB-Plugin achieves better or comparable performance with 25$\times$ smaller backbone LLM compared to SoTA PI methods for low-resourced KBs, and even approaches the performance of supervised methods. Our code and data are available at https://github.com/THU-KEG/KB-Plugin.

Title: MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Models

• Authors: Justin Chih-Yao Chen, Swarnadeep Saha, Elias Stengel-Eskin, Mohit Bansal
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01620
• Pdf URL: https://arxiv.org/pdf/2402.01620
• Copy Paste: [[2402.01620]] MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Models(https://arxiv.org/abs/2402.01620)
• Keywords: large language model
• Abstract: Multi-agent interactions between Large Language Model (LLM) agents have shown major improvements on diverse reasoning tasks. However, these involve long generations from multiple models across several rounds, making them expensive. Moreover, these multi-agent approaches fail to provide a final, single model for efficient inference. To address this, we introduce MAGDi, a new method for structured distillation of the reasoning interactions between multiple LLMs into smaller LMs. MAGDi teaches smaller models by representing multi-agent interactions as graphs, augmenting a base student model with a graph encoder, and distilling knowledge using three objective functions: next-token prediction, a contrastive loss between correct and incorrect reasoning, and a graph-based objective to model the interaction structure. Experiments on seven widely-used commonsense and math reasoning benchmarks show that MAGDi improves the reasoning capabilities of smaller models, outperforming several methods that distill from a single teacher and multiple teachers. Moreover, MAGDi also demonstrates an order of magnitude higher efficiency over its teachers. We conduct extensive analyses to show that MAGDi (1) enhances the generalizability to out-of-domain tasks, (2) scales positively with the size and strength of the base student model, and (3) obtains larger improvements (via our multi-teacher training) when applying self-consistency - an inference technique that relies on model diversity.

Title: Stochastic Two Points Method for Deep Model Zeroth-order Optimization

• Authors: Yijiang Pang, Jiayu Zhou
• Subjects: cs.LG
• Abstract URL: https://arxiv.org/abs/2402.01621
• Pdf URL: https://arxiv.org/pdf/2402.01621
• Copy Paste: [[2402.01621]] Stochastic Two Points Method for Deep Model Zeroth-order Optimization(https://arxiv.org/abs/2402.01621)
• Keywords: large language model
• Abstract: Large foundation models, such as large language models, have performed exceptionally well in various application scenarios. Building or fully fine-tuning such large models is usually prohibitive due to either hardware budget or lack of access to backpropagation. The zeroth-order methods offer a promising direction for tackling this challenge, where only forward passes are needed to update the model. This paper introduces an efficient Stochastic Two-Point (S2P) approach within the gradient-free regime. We present the theoretical convergence properties of S2P under the general and relaxed smoothness assumptions. The theoretical properties also shed light on a faster and more stable S2P variant, Accelerated S2P (AS2P), through exploiting our new convergence properties that better represent the dynamics of deep models in training. Our comprehensive empirical results show that AS2P is highly effective in optimizing objectives for large deep models, including language models, and outperforms standard methods across various model types and scales, with 2 $\times$ speed-up in training over most conducted tasks.

Title: TravelPlanner: A Benchmark for Real-World Planning with Language Agents

• Authors: Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, Yu Su
• Subjects: cs.CL
• Abstract URL: https://arxiv.org/abs/2402.01622
• Pdf URL: https://arxiv.org/pdf/2402.01622
• Copy Paste: [[2402.01622]] TravelPlanner: A Benchmark for Real-World Planning with Language Agents(https://arxiv.org/abs/2402.01622)
• Keywords: large language model
• Abstract: Planning has been part of the core pursuit for artificial intelligence since its conception, but earlier AI agents mostly focused on constrained settings because many of the cognitive substrates necessary for human-level planning have been lacking. Recently, language agents powered by large language models (LLMs) have shown interesting capabilities such as tool use and reasoning. Are these language agents capable of planning in more complex settings that are out of the reach of prior AI agents? To advance this investigation, we propose TravelPlanner, a new planning benchmark that focuses on travel planning, a common real-world planning scenario. It provides a rich sandbox environment, various tools for accessing nearly four million data records, and 1,225 meticulously curated planning intents and reference plans. Comprehensive evaluations show that the current language agents are not yet capable of handling such complex planning tasks-even GPT-4 only achieves a success rate of 0.6%. Language agents struggle to stay on task, use the right tools to collect information, or keep track of multiple constraints. However, we note that the mere possibility for language agents to tackle such a complex problem is in itself non-trivial progress. TravelPlanner provides a challenging yet meaningful testbed for future language agents.

Title: Beyond Lengthscales: No-regret Bayesian Optimisation With Unknown Hyperparameters Of Any Type

• Authors: Juliusz Ziomek, Masaki Adachi, Michael A. Osborne
• Subjects: cs.LG, stat.ML
• Abstract URL: https://arxiv.org/abs/2402.01632
• Pdf URL: https://arxiv.org/pdf/2402.01632
• Copy Paste: [[2402.01632]] Beyond Lengthscales: No-regret Bayesian Optimisation With Unknown Hyperparameters Of Any Type(https://arxiv.org/abs/2402.01632)
• Keywords: robust
• Abstract: Bayesian optimisation requires fitting a Gaussian process model, which in turn requires specifying hyperparameters - most of the theoretical literature assumes those hyperparameters are known. The commonly used maximum likelihood estimator for hyperparameters of the Gaussian process is consistent only if the data fills the space uniformly, which does not have to be the case in Bayesian optimisation. Since no guarantees exist regarding the correctness of hyperparameter estimation, and those hyperparameters can significantly affect the Gaussian process fit, theoretical analysis of Bayesian optimisation with unknown hyperparameters is very challenging. Previously proposed algorithms with the no-regret property were only able to handle the special case of unknown lengthscales, reproducing kernel Hilbert space norm and applied only to the frequentist case. We propose a novel algorithm, HE-GP-UCB, which is the first algorithm enjoying the no-regret property in the case of unknown hyperparameters of arbitrary form, and which supports both Bayesian and frequentist settings. Our proof idea is novel and can easily be extended to other variants of Bayesian optimisation. We show this by extending our algorithm to the adversarially robust optimisation setting under unknown hyperparameters. Finally, we empirically evaluate our algorithm on a set of toy problems and show that it can outperform the maximum likelihood estimator.